Collect LXC filesystems without pmxcfs or LXC socket IPC (#2562)

Fixes #2511.

**Problem.** The typed helper runs with `PrivateNetwork=true`. `pct
list`, `pct df` and `lxc-info` reach pmxcfs and the LXC monitor through
abstract Unix sockets, which belong to the network namespace, so inside
the helper they always fail. Least-privilege installs using the helper
therefore report `provider_unavailable` for `proxmox.lxc_filesystems` on
every report, and never get per-mount-point guest usage. The Proxmox API
only reports the root filesystem. Reproduced on PVE 9.2.18: `pct list`
works normally and fails with `ipcc_send_rec ... Connection refused`
under `PrivateNetwork=yes`.

The fix keeps the helper's hardening exactly as it is.

**Discovery without commands**
(`internal/hostagent/proxmox_lxc_running.go`)
- Lists `/etc/pve/lxc/*.conf` through the pmxcfs mount, which is a file
read rather than IPC.
- Finds each guest's init in `/sys/fs/cgroup/lxc/<vmid>`: `NSpid`
exactly `<pid> 1`, and the process's own cgroup inside `lxc/<vmid>`.
- Names guests by `hostname` or `CT<vmid>`, as `pct list` and the API
do, so monitoring's exact name match keeps working.
- Hands the whole node back to `pct list` whenever it cannot establish a
guest's state: no cgroup v2, an unreadable listing, config, cgroup or
process, a search limit, or cancellation. A guest it could not read
never silently disappears from a healthy-looking inventory.

**Reads through the shared filesystem prober**
(`internal/filesystemprobe`), replacing the separate plain-`statfs`
path.
- New `lxc` requests. The prober pins `/proc/<pid>`, requires the exact
`lxc/<vmid>` cgroup and the guest's init before and after reading, so
the PID 1 of a namespace nested inside the guest (Docker in LXC) cannot
stand in. It resolves every mount inside the guest root.
- `MountsOnly` reports a declared mount that isn't mounted (same device
as its parent) as `ErrNotMounted`, instead of copying the parent's
numbers.
- One bounded probe per guest is kept across the process, so a guest's
hung FUSE mount cannot stack probes in the root helper.
- The helper's `RestrictSUIDSGID=true` makes systemd answer `openat2`
with ENOSYS, because seccomp cannot inspect its flags. The prober now
falls back to a component-by-component walk with `O_NOFOLLOW` that
refuses every symbolic link instead of resolving it. Docker and Podman
reads get the same fallback.

**Real-node verification.** On a throwaway PVE 9.2.18 node, run under
the helper unit's exact sandbox properties (copied from `install.sh`):
- An unprivileged guest (rootfs plus `mp0`) and a privileged guest
returned readings matching `pct df`.
- A stopped guest was omitted.
- The result was not degraded.
- The prober's Linux tests passed as root on the node.

A first version that resolved paths only with `openat2` failed inside
the sandbox; that is what found the ENOSYS case. The throwaway
containers, template and binaries were removed afterwards. A live nested
PID namespace could not be reproduced in that guest, so that case is
covered by unit tests only.

**Tests**
- `TestHelperProxmoxLXCFilesystemsNeedNoPctOrLXCSockets`: the collector
the helper's provider runs, with every command failing, must return a
complete inventory. It fails without this change.
- `TestDiscoverRunningProxmoxLXCContainersFindsInitWithoutSockets`:
covers a nested init listed first, systemd guests and stopped guests.
-
`TestDiscoverRunningProxmoxLXCContainersLeavesUnestablishedStateToPct`:
covers cgroup v1, an unreadable config, a reused PID, an unreadable
`cgroup.procs`, a cgroup tree beyond the search limit, and cancellation.
- `TestProxmoxLXCIdentityIsExact`,
`TestObserverRejectsInexactProxmoxVMIDs`,
`TestMountsOnlyDetectsUnmountedPathsWithinRoot`,
`TestNonInitProcessCannotSupplyProxmoxGuestCapacity` and
`TestComponentWalkRefusesSymlinksAndStaysBeneathRoot`: cover the
prober's new identity, mount and fallback rules.
- `go test ./internal/hostagent ./internal/filesystemprobe
./internal/dockeragent ./cmd/pulse-agent-helper` passes, and `go vet` is
clean for Linux.

The `AGENT_SECURITY.md` capability row and the agent-lifecycle and
monitoring contracts are updated. The row still says this is not full
qualification: one PVE version, and no timeout or helper-loss evidence.

**Overlap with #2551.** #2551 touches the same collector,
`privilege_helper_client*.go` and the same three docs. It hardens `pct
list` parsing and adds v2 partial inventories (#2513), but still starts
from `pct list`, so inside the helper it would keep failing at that
first command. The two are complementary:
- #2551's stricter `pct list` parsing applies to the fallback path here.
- Its partial-inventory reporting wraps the same collection loop.

Whichever lands second needs a textual rebase in
`collectProxmoxLXCFilesystemsResult` and the docs.
This commit is contained in:
rcourtman
2026-10-06 11:56:04 +01:00
committed by GitHub
14 changed files with 1076 additions and 195 deletions
+1 -1
View File
@@ -279,7 +279,7 @@ or generic command path.
| Proxmox VE/PBS/PMG inventory, status, storage, and ordinary metrics | API-only connection with a narrowly scoped token; no host agent | No collector, helper, or runner authority is required for this data | Supported independently of the safe host-agent profile; it does not prove host-local SMART, LXC filesystem, or action parity | `agent-lifecycle`: keep API permissions and returned telemetry covered by provider tests |
| Standard Linux systemd host telemetry and collector update | Unprivileged `pulse-agent` plus the root-owned typed helper | Core `/proc`, filesystem, network, RAID, and hwmon telemetry stays in the collector; signed updates cross the fixed helper protocol and commit only after an authoritative report | **Qualified on disposable Ubuntu 24.04.4 arm64 at committed main `22fd662fb794f63efb9d3ca2158de73c4e07e1b8`** across the schema-v6 twenty-scenario run. It covers install, migration, explicit/automatic rollback, helper protocol and resource boundaries, effective-unit override rejection, reporting continuity, authoritative executable-digest commit, watchdog rollback, helper-restart recovery, and last-known-good restoration. It remains opt-in pending exact-RC reproduction, representative provider/appliance parity, and external review | `deployment-installability` and `security-privacy`: reproduce the same helper transaction from trusted release artifacts on representative hosts and accept the external boundary review |
| Linux SMART telemetry | `smart.snapshot` through the no-network helper; no caller-selected device or arguments | Implemented, unqualified on representative physical disks. Helper failure omits/degrades SMART only; the collector does not retry as root | Does not yet justify SMART parity or a default change | `agent-lifecycle`: record live SATA, SAS/controller, USB bridge, and NVMe evidence, including standby, permission failure, timeout, and partial-data cases |
| Proxmox node-local LXC filesystem telemetry | `proxmox.lxc_filesystems` through the no-network helper using fixed bounded `pct` operations | Implemented, unqualified on a representative PVE node. Helper failure omits/degrades this snapshot only | Does not yet justify Proxmox host-agent parity or a default change | `agent-lifecycle`: record live running/stopped LXC, mount, timeout, output-bound, and helper-loss behavior on supported PVE versions |
| Proxmox node-local LXC filesystem telemetry | `proxmox.lxc_filesystems` through the no-network helper. Configs are read through the pmxcfs mount, running guests and their init processes come from cgroup v2 and `/proc`, and usage comes from the shared filesystem prober, which pins the guest's init, re-checks its identity, and resolves each mount inside the guest root (openat2, or a symlink-refusing component walk where the helper's seccomp filter blocks openat2); the caller selects no VMID, path, or command. `pct` and `lxc-info` cannot work inside the helper, because pmxcfs and the LXC monitor listen on abstract Unix sockets that belong to the host network namespace (#2511); they remain the fallback whenever discovery cannot establish a guest's state, on cgroup v1 hosts, for root and `--grant-pct` installs, and per container when the prober read fails | Implemented. Exercised on a disposable PVE 9.2.18 node under the helper unit's sandbox properties, with running unprivileged (rootfs plus a mount point) and privileged guests and a stopped one; readings matched `pct df`. This is one PVE version, with no timeout, output-bound, helper-loss, or live nested-namespace evidence. Helper failure omits/degrades this snapshot only | Does not yet justify Proxmox host-agent parity or a default change | `agent-lifecycle`: record timeout, output-bound, and helper-loss behavior across supported PVE versions |
| Rootful Docker or Podman inventory | No direct collector access to a root-equivalent daemon socket | Implemented as a typed-helper summary-only fallback. Migration preserves container ID/name/image/state/status/creation inventory and marks the report `typed-helper-summary`; stats, secondary inventories, update checks, and actions remain unavailable | The standalone rootless packet exercises live same-family rootful Docker and Podman helper fallback plus summary-inventory semantic parity during socket loss. It does not qualify the complete standalone rootful fresh-install, migration, restart, helper-loss/recovery, and bounds matrix. Full rootful parity remains an explicit default blocker; the legacy/root profile is not safe-profile evidence | `agent-lifecycle`: record the remaining standalone lifecycle matrix on representative rootful Docker and Podman and decide explicitly whether reduced telemetry is sufficient |
| Collector-owned rootless Docker or Podman | Direct access only to one usable runtime socket owned by the `pulse-agent` UID whose daemon also attests rootless mode | Qualified locally for the exact source-bound monitoring path. Discovery is deferred until the collector account exists and accepts exactly one readable/writable, non-symlink socket owned by that UID across Docker and Podman. Cross-user, root-owned, remote, unreadable, non-rootless-daemon, or ambiguous endpoints are rejected. An installer-owned exact pin survives daemon loss and update; the running collector can move between direct rootless monitoring and rootful typed-helper summary without regaining action authority | **Qualified on distinct disposable Ubuntu 24.04/systemd 255 arm64 hosts at commit `60041ad9e60c282c892f944e04f777b874991a5d`.** All eleven canonical scenarios passed for Docker and all eleven for Podman. Receipt SHA-256: `7a116d63ab0cd1560482165055f0a8c9158ce0a8333a27bd9e8256582a52dfb5`; local attestation SHA-256: `566279ecd7d7dfa89c92f24243e9fcd5ae3e295d4284bd559d4be42f1e83b3b5`. This is local opt-in artifact-bound self-attestation, not published-RC provenance, action/update qualification, external review, or authorization for a default change | `deployment-installability`: reproduce the exact packet from published release artifacts; separately qualify any desired rootless image-update/action surface; retain the opt-in default until the remaining provider, appliance, and review conditions pass |
| Separate runner package update and package-cache cleanup | Root-owned `pulse-agent-runner`, host-bound action credential, typed request, postcondition, and durable receipt | The schema-v6 committed-main systemd receipt records a real verified apt-cache mutation, stale-fingerprint refusal, replay, nonce-bound readiness, effective-unit override rejection, exact credential rotation, bodyless self-revocation, and collector/helper continuity. A separate production Router regression exercises HTTPS issuance, WSS admission, encrypted token persistence, failed-rotation rollback, exact socket invalidation, two server restarts, old-secret rejection, and durable self-revoke | Qualified for the exercised systemd fixture paths and focused production Router lifecycle. Focused installer tests also cover atomic pending cancellation and fail-closed credential retention. The evidence does not cover every runner operation, representative providers, or an exact release candidate | `agent-lifecycle` and `api-contracts`: reproduce the combined systemd and production Router path from trusted RC artifacts with representative package-update success/failure/cancellation evidence |
@@ -572,11 +572,32 @@ standby evidence. Enumeration and each device probe have independent bounded
deadlines so one slow disk cannot consume the complete SMART pass, while
command errors retain bounded stderr for compatibility diagnosis.
On a PVE node, the Unified Agent may also report the mounted filesystem
capacity of running LXCs. The collector uses the fixed local `pct list` command
to admit only node-local running guests, reads each admitted guest's main
`/etc/pve/lxc/<vmid>.conf` mount declarations, resolves its init PID through
the fixed `lxc-info -n <vmid> -p` shape, and probes usage through the Linux
`/proc/<pid>/root` namespace. A configured path that resolves to the same
capacity of running LXCs. On a cgroup v2 node the collector admits node-local
guests from the `/etc/pve/lxc/*.conf` listing and admits only those with a live
init process: a PID in `/sys/fs/cgroup/lxc/<vmid>`'s tree that is PID 1 of the
namespace directly below the host's (`NSpid` exactly `<pid> 1`) and whose own
cgroup is inside `lxc/<vmid>`. It names each by its configured hostname or
`CT<vmid>`, as `pct list` does, reads its main mount declarations, and reads
usage through the shared `internal/filesystemprobe` observer, which pins the
process, re-checks that identity, resolves each mount inside the guest root
and holds one bounded in-flight probe per guest across the process. No command
runs on this path. `pct list`, `pct df` and `lxc-info` reach pmxcfs and the
LXC monitor through abstract Unix sockets scoped to the host network
namespace, so inside the `PrivateNetwork` typed helper they always fail, and
least-privilege installs using that helper reported `provider_unavailable`
on every report (#2511). Discovery hands the whole node back to `pct list`
whenever it cannot establish a guest's state (no cgroup v2, an unreadable
listing, config, cgroup or process, a search limit, or cancellation); only a
guest without a cgroup counts as stopped. The fixed `lxc-info -n <vmid> -p`
PID lookup remains for that path, and `pct df` remains the per-container
fallback, including through the `--grant-pct` wrapper.
`TestHelperProxmoxLXCFilesystemsNeedNoPctOrLXCSockets` drives the collector
the helper's provider runs with every command failing and requires a complete
inventory; `TestDiscoverRunningProxmoxLXCContainersLeavesUnestablishedStateToPct`
pins the fallbacks. On a disposable PVE 9.2.18 node, under the helper unit's
sandbox properties, it returned unprivileged (rootfs plus mount point) and
privileged guests matching `pct df` and omitted a stopped one; a nested PID
namespace was not reproduced there and is covered by tests only. A configured path that resolves to the same
device as its parent is not a live mount and must be omitted instead of
borrowing the parent's capacity. Least-privilege installs that cannot read the
config, resolve the PID, or traverse `/proc` may fall back to the fixed
@@ -844,6 +844,27 @@ tests, container collection boundary tests, ingestion regression and the
opt-in owned tmpfs collector fixture. Installed-agent and model qualification
are tracked separately in PATROL_ASSISTANT_CUSTOMER_JOURNEY.md.
The same observer reads Proxmox LXC guests for the host agent and its typed
helper (#2511). An `lxc` request names the decimal VMID; the pinned process
must sit in that guest's exact `lxc/<vmid>` cgroup and be its init (`NSpid` is
exactly `<pid> 1`, so the PID 1 of a namespace nested inside the guest, such
as Docker in LXC, cannot stand in for it), checked before and after the read.
With `MountsOnly`, a configured mountpoint sharing its parent's device reports
`ErrNotMounted` instead of the parent's counters. Docker and Podman identity
rules are unchanged. Confined resolution prefers openat2 with `RESOLVE_IN_ROOT`;
when the kernel or a systemd seccomp filter answers ENOSYS (the typed helper's
`RestrictSUIDSGID=true` blocks openat2 because its flags cannot be inspected),
the clean path is walked component by component with `O_NOFOLLOW` and any
symbolic link is refused, so neither path resolves outside the guest root.
Docker and Podman reads take the same fallback, so a collector under such a
filter now gets confined counters instead of unavailable evidence.
`TestComponentWalkRefusesSymlinksAndStaysBeneathRoot`,
`TestMountsOnlyDetectsUnmountedPathsWithinRoot`,
`TestNonInitProcessCannotSupplyProxmoxGuestCapacity` and
`TestProxmoxLXCIdentityIsExact` pin these rules; the Linux tests also passed
as root on a disposable PVE 9.2.18 node, where the helper-sandboxed collector
returned readings matching `pct df` through the component walk.
Docker mount collection preserves both native `Mounts` records and entries
reported only in `HostConfig.Tmpfs`. Existing reported destinations remain
authoritative. Additional tmpfs destinations are ordered deterministically,
@@ -279,7 +279,7 @@ or generic command path.
| Proxmox VE/PBS/PMG inventory, status, storage, and ordinary metrics | API-only connection with a narrowly scoped token; no host agent | No collector, helper, or runner authority is required for this data | Supported independently of the safe host-agent profile; it does not prove host-local SMART, LXC filesystem, or action parity | `agent-lifecycle`: keep API permissions and returned telemetry covered by provider tests |
| Standard Linux systemd host telemetry and collector update | Unprivileged `pulse-agent` plus the root-owned typed helper | Core `/proc`, filesystem, network, RAID, and hwmon telemetry stays in the collector; signed updates cross the fixed helper protocol and commit only after an authoritative report | **Qualified on disposable Ubuntu 24.04.4 arm64 at committed main `22fd662fb794f63efb9d3ca2158de73c4e07e1b8`** across the schema-v6 twenty-scenario run. It covers install, migration, explicit/automatic rollback, helper protocol and resource boundaries, effective-unit override rejection, reporting continuity, authoritative executable-digest commit, watchdog rollback, helper-restart recovery, and last-known-good restoration. It remains opt-in pending exact-RC reproduction, representative provider/appliance parity, and external review | `deployment-installability` and `security-privacy`: reproduce the same helper transaction from trusted release artifacts on representative hosts and accept the external boundary review |
| Linux SMART telemetry | `smart.snapshot` through the no-network helper; no caller-selected device or arguments | Implemented, unqualified on representative physical disks. Helper failure omits/degrades SMART only; the collector does not retry as root | Does not yet justify SMART parity or a default change | `agent-lifecycle`: record live SATA, SAS/controller, USB bridge, and NVMe evidence, including standby, permission failure, timeout, and partial-data cases |
| Proxmox node-local LXC filesystem telemetry | `proxmox.lxc_filesystems` through the no-network helper using fixed bounded `pct` operations | Implemented, unqualified on a representative PVE node. Helper failure omits/degrades this snapshot only | Does not yet justify Proxmox host-agent parity or a default change | `agent-lifecycle`: record live running/stopped LXC, mount, timeout, output-bound, and helper-loss behavior on supported PVE versions |
| Proxmox node-local LXC filesystem telemetry | `proxmox.lxc_filesystems` through the no-network helper. Configs are read through the pmxcfs mount, running guests and their init processes come from cgroup v2 and `/proc`, and usage comes from the shared filesystem prober, which pins the guest's init, re-checks its identity, and resolves each mount inside the guest root (openat2, or a symlink-refusing component walk where the helper's seccomp filter blocks openat2); the caller selects no VMID, path, or command. `pct` and `lxc-info` cannot work inside the helper, because pmxcfs and the LXC monitor listen on abstract Unix sockets that belong to the host network namespace (#2511); they remain the fallback whenever discovery cannot establish a guest's state, on cgroup v1 hosts, for root and `--grant-pct` installs, and per container when the prober read fails | Implemented. Exercised on a disposable PVE 9.2.18 node under the helper unit's sandbox properties, with running unprivileged (rootfs plus a mount point) and privileged guests and a stopped one; readings matched `pct df`. This is one PVE version, with no timeout, output-bound, helper-loss, or live nested-namespace evidence. Helper failure omits/degrades this snapshot only | Does not yet justify Proxmox host-agent parity or a default change | `agent-lifecycle`: record timeout, output-bound, and helper-loss behavior across supported PVE versions |
| Rootful Docker or Podman inventory | No direct collector access to a root-equivalent daemon socket | Implemented as a typed-helper summary-only fallback. Migration preserves container ID/name/image/state/status/creation inventory and marks the report `typed-helper-summary`; stats, secondary inventories, update checks, and actions remain unavailable | The standalone rootless packet exercises live same-family rootful Docker and Podman helper fallback plus summary-inventory semantic parity during socket loss. It does not qualify the complete standalone rootful fresh-install, migration, restart, helper-loss/recovery, and bounds matrix. Full rootful parity remains an explicit default blocker; the legacy/root profile is not safe-profile evidence | `agent-lifecycle`: record the remaining standalone lifecycle matrix on representative rootful Docker and Podman and decide explicitly whether reduced telemetry is sufficient |
| Collector-owned rootless Docker or Podman | Direct access only to one usable runtime socket owned by the `pulse-agent` UID whose daemon also attests rootless mode | Qualified locally for the exact source-bound monitoring path. Discovery is deferred until the collector account exists and accepts exactly one readable/writable, non-symlink socket owned by that UID across Docker and Podman. Cross-user, root-owned, remote, unreadable, non-rootless-daemon, or ambiguous endpoints are rejected. An installer-owned exact pin survives daemon loss and update; the running collector can move between direct rootless monitoring and rootful typed-helper summary without regaining action authority | **Qualified on distinct disposable Ubuntu 24.04/systemd 255 arm64 hosts at commit `60041ad9e60c282c892f944e04f777b874991a5d`.** All eleven canonical scenarios passed for Docker and all eleven for Podman. Receipt SHA-256: `7a116d63ab0cd1560482165055f0a8c9158ce0a8333a27bd9e8256582a52dfb5`; local attestation SHA-256: `566279ecd7d7dfa89c92f24243e9fcd5ae3e295d4284bd559d4be42f1e83b3b5`. This is local opt-in artifact-bound self-attestation, not published-RC provenance, action/update qualification, external review, or authorization for a default change | `deployment-installability`: reproduce the exact packet from published release artifacts; separately qualify any desired rootless image-update/action surface; retain the opt-in default until the remaining provider, appliance, and review conditions pass |
| Separate runner package update and package-cache cleanup | Root-owned `pulse-agent-runner`, host-bound action credential, typed request, postcondition, and durable receipt | The schema-v6 committed-main systemd receipt records a real verified apt-cache mutation, stale-fingerprint refusal, replay, nonce-bound readiness, effective-unit override rejection, exact credential rotation, bodyless self-revocation, and collector/helper continuity. A separate production Router regression exercises HTTPS issuance, WSS admission, encrypted token persistence, failed-rotation rollback, exact socket invalidation, two server restarts, old-secret rejection, and durable self-revoke | Qualified for the exercised systemd fixture paths and focused production Router lifecycle. Focused installer tests also cover atomic pending cancellation and fail-closed credential retention. The evidence does not cover every runner operation, representative providers, or an exact release candidate | `agent-lifecycle` and `api-contracts`: reproduce the combined systemd and production Router path from trusted RC artifacts with representative package-update success/failure/cancellation evidence |
+73 -6
View File
@@ -7,6 +7,7 @@ import (
"errors"
"fmt"
"path"
"strconv"
"strings"
"sync"
"time"
@@ -24,8 +25,24 @@ const (
type ContainerRequest struct {
PID int
ContainerID string
// Runtime is docker or podman (ContainerID is the 64-hex ID) or lxc, a
// Proxmox container (ContainerID is the decimal VMID and PID must be the
// container's init process).
Runtime string
Mountpoints []string
// MountsOnly reports a mountpoint that shares its parent's device as
// ErrNotMounted instead of returning the parent filesystem's counters. A
// Proxmox config can declare a mount point the running guest never mounted.
MountsOnly bool
}
// ErrNotMounted marks an observation whose path is not a distinct mounted
// filesystem inside the container. Callers test it with IsNotMounted.
var ErrNotMounted = errors.New("path is not a mounted filesystem in the container")
// IsNotMounted reports whether an observation was rejected as ErrNotMounted.
func IsNotMounted(observation filesystem.Observation) bool {
return observation.Error == ErrNotMounted.Error()
}
// Observer bounds potentially uninterruptible kernel reads. A timed-out probe
@@ -101,12 +118,18 @@ func validateRequest(req ContainerRequest) error {
if req.PID <= 0 {
return errors.New("container process is unavailable")
}
if len(req.ContainerID) != 64 || strings.IndexFunc(req.ContainerID, func(r rune) bool {
return !(r >= '0' && r <= '9' || r >= 'a' && r <= 'f')
}) >= 0 {
return errors.New("filesystem observation requires an exact container ID")
}
if req.Runtime != "docker" && req.Runtime != "podman" {
switch req.Runtime {
case "docker", "podman":
if len(req.ContainerID) != 64 || strings.IndexFunc(req.ContainerID, func(r rune) bool {
return !(r >= '0' && r <= '9' || r >= 'a' && r <= 'f')
}) >= 0 {
return errors.New("filesystem observation requires an exact container ID")
}
case "lxc":
if !validLXCVMID(req.ContainerID) {
return errors.New("filesystem observation requires an exact Proxmox VMID")
}
default:
return errors.New("unsupported container runtime")
}
if len(req.Mountpoints) == 0 || len(req.Mountpoints) > maxMountpoints {
@@ -122,9 +145,53 @@ func validateRequest(req ContainerRequest) error {
return nil
}
func validLXCVMID(id string) bool {
vmid, err := strconv.Atoi(id)
return err == nil && vmid >= 100 && vmid <= 999999999 && strconv.Itoa(vmid) == id
}
// CgroupMatches reports whether a /proc/<pid>/cgroup body places the process
// inside the given container, with the same exact-identity rules the observer
// applies to its pinned process.
func CgroupMatches(raw, runtime, containerID string) bool {
return cgroupMatches(raw, ContainerRequest{Runtime: runtime, ContainerID: containerID})
}
// IsNamespaceInit reports whether a /proc/<pid>/status body describes pid as
// PID 1 of a PID namespace one level below the reader's: NSpid is exactly
// "<pid> 1". A process that is PID 1 of a namespace nested inside the
// container (Docker inside an LXC guest) lists three or more PIDs and is not
// the container's init.
func IsNamespaceInit(status string, pid int) bool {
for _, line := range strings.Split(status, "\n") {
rest, ok := strings.CutPrefix(line, "NSpid:")
if !ok {
continue
}
fields := strings.Fields(rest)
return len(fields) == 2 && fields[0] == strconv.Itoa(pid) && fields[1] == "1"
}
return false
}
// cgroupMatches accepts an exact runtime-owned cgroup component, never a
// substring, container name, short ID, or unrelated process with the same PID.
func cgroupMatches(raw string, req ContainerRequest) bool {
if req.Runtime == "lxc" {
// pve-container places a guest under lxc/<vmid> (payload in ns/).
// lxc.monitor/<vmid> and other VMIDs sharing a prefix never match.
for _, line := range strings.Split(raw, "\n") {
parts := strings.SplitN(line, ":", 3)
if len(parts) != 3 {
continue
}
components := strings.Split(parts[2], "/")
if len(components) >= 3 && components[0] == "" && components[1] == "lxc" && components[2] == req.ContainerID {
return true
}
}
return false
}
scope := "docker-" + req.ContainerID + ".scope"
if req.Runtime == "podman" {
scope = "libpod-" + req.ContainerID + ".scope"
+118 -6
View File
@@ -9,7 +9,9 @@ import (
"io"
"math"
"os"
"path"
"strconv"
"strings"
"time"
"github.com/rcourtman/pulse-go-rewrite/pkg/agents/filesystem"
@@ -25,7 +27,7 @@ func observeContainer(ctx context.Context, req ContainerRequest) ([]filesystem.O
return nil, fmt.Errorf("open container process: %w", err)
}
defer unix.Close(proc)
if err := verifyCgroup(proc, req); err != nil {
if err := verifyIdentity(proc, req); err != nil {
return nil, err
}
root, err := unix.Openat(proc, "root", unix.O_PATH|unix.O_DIRECTORY|unix.O_CLOEXEC, 0)
@@ -33,7 +35,7 @@ func observeContainer(ctx context.Context, req ContainerRequest) ([]filesystem.O
return nil, fmt.Errorf("open container filesystem namespace: %w", err)
}
defer unix.Close(root)
if err := verifyCgroup(proc, req); err != nil {
if err := verifyIdentity(proc, req); err != nil {
return nil, err
}
observations := make([]filesystem.Observation, 0, len(req.Mountpoints))
@@ -42,6 +44,14 @@ func observeContainer(ctx context.Context, req ContainerRequest) ([]filesystem.O
return nil, err
}
observation := filesystem.Observation{Mountpoint: mount, Source: Source}
if req.MountsOnly {
if mounted, mountErr := mountedWithinRoot(root, mount); mountErr == nil && !mounted {
observation.ObservedAt = time.Now().UTC()
observation.Error = ErrNotMounted.Error()
observations = append(observations, observation)
continue
}
}
observation.Type, observation.Usage, err = observePath(root, mount)
observation.ObservedAt = time.Now().UTC()
if err != nil {
@@ -49,12 +59,70 @@ func observeContainer(ctx context.Context, req ContainerRequest) ([]filesystem.O
}
observations = append(observations, observation)
}
if err := verifyCgroup(proc, req); err != nil {
if err := verifyIdentity(proc, req); err != nil {
return nil, err
}
return observations, nil
}
func verifyIdentity(proc int, req ContainerRequest) error {
if err := verifyCgroup(proc, req); err != nil {
return err
}
if req.Runtime == "lxc" {
return verifyNamespaceInit(proc, req.PID)
}
return nil
}
// verifyNamespaceInit requires the pinned process to be the guest's init, so a
// process of a namespace nested inside the guest cannot supply its own root.
func verifyNamespaceInit(proc, pid int) error {
fd, err := unix.Openat(proc, "status", unix.O_RDONLY|unix.O_CLOEXEC|unix.O_NOFOLLOW, 0)
if err != nil {
return fmt.Errorf("read container process status: %w", err)
}
file := os.NewFile(uintptr(fd), "status")
defer file.Close()
raw, err := io.ReadAll(io.LimitReader(file, 65537))
if err != nil || len(raw) > 65536 {
return errors.New("container process status is unavailable")
}
if !IsNamespaceInit(string(raw), pid) {
return errors.New("process is not the container's init")
}
return nil
}
// mountedWithinRoot reports whether mount is a different filesystem from its
// parent directory, both resolved within the resource root.
func mountedWithinRoot(root int, mount string) (bool, error) {
if mount == "/" {
return true, nil
}
device := func(target string) (uint64, error) {
fd, err := openInRoot(root, target)
if err != nil {
return 0, err
}
defer unix.Close(fd)
var stat unix.Stat_t
if err := unix.Fstat(fd, &stat); err != nil {
return 0, err
}
return uint64(stat.Dev), nil
}
own, err := device(mount)
if err != nil {
return false, err
}
parent, err := device(path.Dir(mount))
if err != nil {
return false, err
}
return own != parent, nil
}
func verifyCgroup(proc int, req ContainerRequest) error {
fd, err := unix.Openat(proc, "cgroup", unix.O_RDONLY|unix.O_CLOEXEC|unix.O_NOFOLLOW, 0)
if err != nil {
@@ -72,13 +140,57 @@ func verifyCgroup(proc int, req ContainerRequest) error {
return nil
}
func observePath(root int, mount string) (string, *filesystem.Usage, error) {
// Absolute symlinks stay within the resource root. Magic links cannot escape
// it. Unsupported kernels fail rather than falling back to unsafe resolution.
// openInRoot opens mount beneath root without ever leaving it. With openat2,
// absolute symlinks stay within the resource root and magic links cannot
// escape it. systemd answers openat2 with ENOSYS under RestrictSUIDSGID,
// because seccomp cannot inspect its flags, and the typed helper sets that, so
// there the clean path is walked one component at a time and any symbolic link
// is refused rather than resolved. Neither path falls back to unconfined
// resolution.
func openInRoot(root int, mount string) (int, error) {
fd, err := unix.Openat2(root, mount, &unix.OpenHow{
Flags: unix.O_PATH | unix.O_CLOEXEC,
Resolve: unix.RESOLVE_IN_ROOT | unix.RESOLVE_NO_MAGICLINKS,
})
if !errors.Is(err, unix.ENOSYS) {
return fd, err
}
return openBeneathWithoutSymlinks(root, mount)
}
func openBeneathWithoutSymlinks(root int, mount string) (int, error) {
if !path.IsAbs(mount) || path.Clean(mount) != mount {
return -1, errors.New("mount path must be clean and absolute")
}
current, err := unix.Dup(root)
if err != nil {
return -1, err
}
for _, component := range strings.Split(strings.TrimPrefix(mount, "/"), "/") {
if component == "" {
continue
}
next, err := unix.Openat(current, component, unix.O_PATH|unix.O_NOFOLLOW|unix.O_CLOEXEC, 0)
unix.Close(current)
if err != nil {
return -1, err
}
var stat unix.Stat_t
if err := unix.Fstat(next, &stat); err != nil {
unix.Close(next)
return -1, err
}
if stat.Mode&unix.S_IFMT == unix.S_IFLNK {
unix.Close(next)
return -1, errors.New("mount path crosses a symbolic link")
}
current = next
}
return current, nil
}
func observePath(root int, mount string) (string, *filesystem.Usage, error) {
fd, err := openInRoot(root, mount)
if err != nil {
return "", nil, fmt.Errorf("open mount within resource namespace: %w", err)
}
@@ -81,3 +81,68 @@ func TestWrongProcessCannotSupplyContainerCapacity(t *testing.T) {
t.Fatalf("unrelated process supplied capacity: %+v %v", got, err)
}
}
// A configured mountpoint that shares its parent's device is not mounted in
// the guest; reporting it would copy the parent filesystem's counters.
func TestMountsOnlyDetectsUnmountedPathsWithinRoot(t *testing.T) {
dir := t.TempDir()
if err := os.Mkdir(filepath.Join(dir, "data"), 0700); err != nil {
t.Fatal(err)
}
fd, err := unix.Open(dir, unix.O_PATH|unix.O_DIRECTORY|unix.O_CLOEXEC, 0)
if err != nil {
t.Fatal(err)
}
defer unix.Close(fd)
if mounted, err := mountedWithinRoot(fd, "/"); err != nil || !mounted {
t.Fatalf("root reported unmounted: %v %v", mounted, err)
}
if mounted, err := mountedWithinRoot(fd, "/data"); err != nil || mounted {
t.Fatalf("plain directory reported as a mount: %v %v", mounted, err)
}
if _, err := mountedWithinRoot(fd, "/missing"); err == nil {
t.Fatal("missing path reported a mount state")
}
}
// The test process is not PID 1 of a namespace, so it cannot supply a Proxmox
// guest's filesystems even if a request names it.
func TestNonInitProcessCannotSupplyProxmoxGuestCapacity(t *testing.T) {
r := ContainerRequest{PID: os.Getpid(), ContainerID: "126", Runtime: "lxc", Mountpoints: []string{"/"}, MountsOnly: true}
if got, err := observeContainer(context.Background(), r); err == nil || got != nil {
t.Fatalf("non-init process supplied capacity: %+v %v", got, err)
}
}
// Inside the typed helper systemd blocks openat2 (RestrictSUIDSGID), so mounts
// are resolved component by component. That walk must stay beneath the root
// by refusing every symbolic link, including ones that would stay inside.
func TestComponentWalkRefusesSymlinksAndStaysBeneathRoot(t *testing.T) {
dir := t.TempDir()
if err := os.MkdirAll(filepath.Join(dir, "srv", "data"), 0700); err != nil {
t.Fatal(err)
}
for name, target := range map[string]string{"inside-link": "/srv", "outside": "/etc", "relative-up": "../../.."} {
if err := os.Symlink(target, filepath.Join(dir, name)); err != nil {
t.Fatal(err)
}
}
root, err := unix.Open(dir, unix.O_PATH|unix.O_DIRECTORY|unix.O_CLOEXEC, 0)
if err != nil {
t.Fatal(err)
}
defer unix.Close(root)
for _, mount := range []string{"/", "/srv", "/srv/data"} {
fd, err := openBeneathWithoutSymlinks(root, mount)
if err != nil {
t.Fatalf("plain path %s refused: %v", mount, err)
}
unix.Close(fd)
}
for _, mount := range []string{"/inside-link", "/inside-link/data", "/outside", "/outside/passwd", "/relative-up", "/srv/../etc", "srv", "/proc/self/root"} {
if fd, err := openBeneathWithoutSymlinks(root, mount); err == nil {
unix.Close(fd)
t.Fatalf("%s resolved through the component walk", mount)
}
}
}
+54
View File
@@ -79,3 +79,57 @@ func TestObserverRejectsInvalidCoordinatesBeforeNativeRead(t *testing.T) {
t.Fatal(err)
}
}
// A Proxmox guest is identified by its exact lxc/<vmid> cgroup and its init
// process. Neither another VMID with the same prefix, LXC's monitor cgroup,
// nor the PID 1 of a namespace nested inside the guest (Docker in LXC) may
// stand in for it.
func TestProxmoxLXCIdentityIsExact(t *testing.T) {
for raw, want := range map[string]bool{
"0::/lxc/126/ns": true,
"0::/lxc/126/ns/init.scope": true,
"0::/lxc/126": true,
"0::/lxc/1260/ns": false,
"0::/lxc/12/ns": false,
"0::/lxc.monitor/126": false,
"0::/system.slice/lxc/126/ns": false,
"0::/docker/" + strings.Repeat("a", 64): false,
} {
if got := CgroupMatches(raw, "lxc", "126"); got != want {
t.Errorf("CgroupMatches(%q) = %v, want %v", raw, got, want)
}
}
for status, want := range map[string]bool{
"Name:\tinit\nNSpid:\t3019253\t1\n": true,
"NSpid:\t3019253\t413\n": false,
"NSpid:\t3019253\t420\t1\n": false, // nested namespace init
"NSpid:\t1\n": false, // the host's own init
"NSpid:\t999\t1\n": false, // another process's line
"Name:\tinit\n": false,
} {
if got := IsNamespaceInit(status, 3019253); got != want {
t.Errorf("IsNamespaceInit(%q) = %v, want %v", status, got, want)
}
}
}
func TestObserverRejectsInexactProxmoxVMIDs(t *testing.T) {
o := &Observer{probe: func(context.Context, ContainerRequest) ([]filesystem.Observation, error) {
t.Error("invalid coordinates reached native read")
return nil, nil
}}
for _, id := range []string{"99", "0126", "126a", "-126", "1000000000", ""} {
r := ContainerRequest{PID: 42, ContainerID: id, Runtime: "lxc", Mountpoints: []string{"/"}}
if _, err := o.Observe(context.Background(), r); err == nil {
t.Fatalf("accepted VMID %q", id)
}
}
accepted := false
o.probe = func(context.Context, ContainerRequest) ([]filesystem.Observation, error) {
accepted = true
return nil, nil
}
if _, err := o.Observe(context.Background(), ContainerRequest{PID: 42, ContainerID: "126", Runtime: "lxc", Mountpoints: []string{"/"}}); err != nil || !accepted {
t.Fatalf("exact VMID rejected: %v", err)
}
}
+20 -6
View File
@@ -7,8 +7,10 @@ import (
"os"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/filesystemprobe"
"github.com/rcourtman/pulse-go-rewrite/internal/hostmetrics"
"github.com/rcourtman/pulse-go-rewrite/internal/sensors"
"github.com/rcourtman/pulse-go-rewrite/pkg/agents/filesystem"
agentshost "github.com/rcourtman/pulse-go-rewrite/pkg/agents/host"
gohost "github.com/shirou/gopsutil/v4/host"
)
@@ -44,15 +46,27 @@ type mockCollector struct {
name string,
arg ...string,
) (string, error)
lookPathFn func(file string) (string, error)
filesystemUsageFn func(path string) (hostFilesystemUsage, error)
lookPathFn func(file string) (string, error)
observeLXCFn func(ctx context.Context, req filesystemprobe.ContainerRequest) ([]filesystem.Observation, error)
readDirFn func(name string) ([]os.DirEntry, error)
}
func (m *mockCollector) FilesystemUsage(path string) (hostFilesystemUsage, error) {
if m.filesystemUsageFn != nil {
return m.filesystemUsageFn(path)
// ReadDir satisfies proxmoxLXCDirLister. Without readDirFn it reports the
// directory missing, so collectors keep their pct path unless a test opts in.
func (m *mockCollector) ReadDir(name string) ([]os.DirEntry, error) {
if m.readDirFn != nil {
return m.readDirFn(name)
}
return hostFilesystemUsage{}, os.ErrNotExist
return nil, os.ErrNotExist
}
// ObserveProxmoxLXC satisfies proxmoxLXCObserver. Without observeLXCFn the
// probe fails, as it does off Linux, so collectors fall back to pct df.
func (m *mockCollector) ObserveProxmoxLXC(ctx context.Context, req filesystemprobe.ContainerRequest) ([]filesystem.Observation, error) {
if m.observeLXCFn != nil {
return m.observeLXCFn(ctx, req)
}
return nil, os.ErrNotExist
}
func (m *mockCollector) HostInfo(ctx context.Context) (*gohost.InfoStat, error) {
@@ -412,3 +412,44 @@ func requirePrivilegeHelperModuleStatus(t *testing.T, statuses []agentshost.Modu
t.Fatalf("typed privilege helper module missing from %+v", statuses)
return agentshost.ModuleStatus{}
}
// The typed helper runs with PrivateNetwork=true, where pct and lxc-info cannot
// reach pmxcfs or the LXC monitor: their abstract Unix sockets belong to the
// host network namespace (#2511). This drives the collector the helper's
// provider runs, with every command failing as it does there: it must still
// return a complete inventory from configs, cgroups and /proc alone. A
// degraded result is what the provider turns into provider_unavailable.
func TestHelperProxmoxLXCFilesystemsNeedNoPctOrLXCSockets(t *testing.T) {
h := newFakeProxmoxLXCHost()
h.config(126, "hostname: qual2511\nrootfs: delly2-lvm:vm-126-disk-0,size=1G\nmp0: delly2-lvm:vm-126-disk-1,mp=/srv/data,size=1G\n")
h.cgroup("126/ns", 3019253)
h.proc(3019253, "3019253\t1", "/lxc/126/ns")
h.config(127, "hostname: qual2511-priv\nrootfs: delly2-lvm:vm-127-disk-0,size=1G\n")
h.cgroup("127/ns", 3021000)
h.proc(3021000, "3021000\t1", "/lxc/127/ns")
h.config(128, "hostname: stopped\nrootfs: delly2-lvm:vm-128-disk-0,size=1G\n")
// Counters observed on a real PVE 9.2 node for the same layout.
h.observations[3019253] = map[string]lxcObservation{
"/": lxcUsage(1020702720, 12021760, 938217472),
"/srv/data": lxcUsage(1020702720, 290816, 949948416),
}
h.observations[3021000] = map[string]lxcObservation{"/": lxcUsage(1020702720, 12017664, 938221568)}
collector, commands := h.collector(t)
agent := &Agent{logger: zerolog.Nop(), collector: collector}
result := agent.collectProxmoxLXCFilesystemsResult(context.Background())
if !result.Applicable || result.Degraded || result.FailedContainers != 0 {
t.Fatalf("collection = applicable %v, degraded %v, failed %d; want a complete inventory", result.Applicable, result.Degraded, result.FailedContainers)
}
if *commands != 0 {
t.Fatalf("collection ran %d pct/lxc-info commands; none can work inside the helper", *commands)
}
got := result.Inventory.Containers
if len(got) != 2 || got[0].VMID != 126 || got[0].Name != "qual2511" || got[1].VMID != 127 || got[1].Name != "qual2511-priv" {
t.Fatalf("containers = %+v, want the two running ones by their pct names", got)
}
if len(got[0].Disks) != 2 || got[0].Disks[1].Mountpoint != "/srv/data" || got[0].Disks[1].UsedBytes != 290816 {
t.Fatalf("126 disks = %+v, want rootfs and the mp0 mount with its own usage", got[0].Disks)
}
}
+111 -95
View File
@@ -15,6 +15,7 @@ import (
"time"
"unicode/utf8"
"github.com/rcourtman/pulse-go-rewrite/internal/filesystemprobe"
agentshost "github.com/rcourtman/pulse-go-rewrite/pkg/agents/host"
"github.com/rs/zerolog"
)
@@ -39,12 +40,15 @@ const (
proxmoxLXCCollectionTimeout = 20 * time.Second
proxmoxLXCMaxContainers = 128
proxmoxLXCMaxDisks = 257
proxmoxLXCMaxListOutputBytes = 64 * 1024
proxmoxLXCMaxDFOutputBytes = 64 * 1024
proxmoxLXCMaxConfigBytes = 64 * 1024
proxmoxLXCMaxPIDOutputBytes = 4 * 1024
proxmoxLXCMaxNameBytes = 256
proxmoxLXCMaxLabelBytes = 512
// proxmoxLXCMaxObservedMounts is the shared filesystem prober's per-request
// mountpoint limit.
proxmoxLXCMaxObservedMounts = 128
proxmoxLXCMaxListOutputBytes = 64 * 1024
proxmoxLXCMaxDFOutputBytes = 64 * 1024
proxmoxLXCMaxConfigBytes = 64 * 1024
proxmoxLXCMaxPIDOutputBytes = 4 * 1024
proxmoxLXCMaxNameBytes = 256
proxmoxLXCMaxLabelBytes = 512
)
// proxmoxLXCConfigDir is a pmxcfs symlink to the local node's container
@@ -54,6 +58,9 @@ const proxmoxLXCConfigDir = "/etc/pve/lxc"
type proxmoxLXCRunningContainer struct {
VMID int
Name string
// PID is the container's init process when discovery found it without
// lxc-info; 0 means unknown.
PID int
}
// ProxmoxLXCFilesystemCollectionResult describes applicability separately from
@@ -68,21 +75,6 @@ type ProxmoxLXCFilesystemCollectionResult struct {
FailedContainers int
}
// hostFilesystemUsage carries statfs-derived usage for one path plus the
// st_dev identity needed to tell a real mount from its parent filesystem.
type hostFilesystemUsage struct {
TotalBytes int64
UsedBytes int64
AvailBytes int64
Device uint64
}
// filesystemUsageProber is an optional collector capability. It exists on
// Linux builds of the default collector; mocks opt in for tests.
type filesystemUsageProber interface {
FilesystemUsage(path string) (hostFilesystemUsage, error)
}
type proxmoxLXCConfigMount struct {
Key string
Volume string
@@ -142,30 +134,38 @@ func (a *Agent) collectProxmoxLXCFilesystemsResult(ctx context.Context) ProxmoxL
}
result.Applicable = true
observer, _ := a.collector.(proxmoxLXCObserver)
listCtx, cancelList := context.WithTimeout(ctx, proxmoxLXCQueryTimeout)
listOutput, err := collectCommandOutputLimited(
listCtx,
a.collector,
proxmoxLXCMaxListOutputBytes,
pctPath,
"list",
)
containers, discovered := []proxmoxLXCRunningContainer(nil), false
if observer != nil {
containers, discovered = a.discoverRunningProxmoxLXCContainers(listCtx)
}
if !discovered {
listOutput, listErr := collectCommandOutputLimited(
listCtx,
a.collector,
proxmoxLXCMaxListOutputBytes,
pctPath,
"list",
)
if listErr != nil {
cancelList()
a.logger.Debug().Err(listErr).Msg("Failed to list local Proxmox LXC containers")
result.Degraded = true
return result
}
containers, err = parseProxmoxLXCRunningContainers(listOutput)
if err != nil {
cancelList()
a.logger.Debug().Err(err).Msg("Failed to parse local Proxmox LXC container list")
result.Degraded = true
return result
}
}
cancelList()
if err != nil {
a.logger.Debug().Err(err).Msg("Failed to list local Proxmox LXC containers")
result.Degraded = true
return result
}
containers, err := parseProxmoxLXCRunningContainers(listOutput)
if err != nil {
a.logger.Debug().Err(err).Msg("Failed to parse local Proxmox LXC container list")
result.Degraded = true
return result
}
prober, _ := a.collector.(filesystemUsageProber)
lxcInfoPath := ""
if prober != nil {
if observer != nil && !discovered {
if resolved, lookErr := a.collector.LookPath("lxc-info"); lookErr == nil {
lxcInfoPath = resolved
}
@@ -184,7 +184,7 @@ func (a *Agent) collectProxmoxLXCFilesystemsResult(ctx context.Context) ProxmoxL
continue
}
containerCtx, cancelContainer := context.WithTimeout(collectionCtx, proxmoxLXCContainerQueryTimeout)
disks, collectionErr := a.collectProxmoxLXCContainerDisks(containerCtx, prober, lxcInfoPath, pctPath, container.VMID)
disks, collectionErr := a.collectProxmoxLXCContainerDisks(containerCtx, observer, lxcInfoPath, pctPath, container)
cancelContainer()
if collectionErr != nil {
result.FailedContainers++
@@ -209,13 +209,14 @@ func (a *Agent) collectProxmoxLXCFilesystemsResult(ctx context.Context) ProxmoxL
func (a *Agent) collectProxmoxLXCContainerDisks(
ctx context.Context,
prober filesystemUsageProber,
observer proxmoxLXCObserver,
lxcInfoPath string,
pctPath string,
vmid int,
container proxmoxLXCRunningContainer,
) ([]agentshost.Disk, error) {
if prober != nil && lxcInfoPath != "" {
disks, fastErr := a.collectProxmoxLXCContainerDisksFast(ctx, prober, lxcInfoPath, vmid)
vmid := container.VMID
if observer != nil && (container.PID > 0 || lxcInfoPath != "") {
disks, fastErr := a.collectProxmoxLXCContainerDisksFast(ctx, observer, lxcInfoPath, container)
if fastErr == nil {
return disks, nil
}
@@ -259,10 +260,11 @@ func (a *Agent) collectProxmoxLXCContainerDisks(
// numbers — so it is dropped rather than invented.
func (a *Agent) collectProxmoxLXCContainerDisksFast(
ctx context.Context,
prober filesystemUsageProber,
observer proxmoxLXCObserver,
lxcInfoPath string,
vmid int,
container proxmoxLXCRunningContainer,
) ([]agentshost.Disk, error) {
vmid := container.VMID
raw, err := a.collector.ReadFile(proxmoxLXCConfigPath(vmid))
if err != nil {
return nil, fmt.Errorf("read container config: %w", err)
@@ -275,59 +277,76 @@ func (a *Agent) collectProxmoxLXCContainerDisksFast(
return nil, errors.New("no filesystem mounts in container config")
}
pidOutput, err := collectCommandOutputLimited(
ctx,
a.collector,
proxmoxLXCMaxPIDOutputBytes,
lxcInfoPath,
"-n",
strconv.Itoa(vmid),
"-p",
)
if err != nil {
return nil, fmt.Errorf("resolve container pid: %w", err)
}
pid, err := parseLXCInfoPID(pidOutput)
if err != nil {
return nil, err
pid := container.PID
if pid <= 0 {
pidOutput, pidErr := collectCommandOutputLimited(
ctx,
a.collector,
proxmoxLXCMaxPIDOutputBytes,
lxcInfoPath,
"-n",
strconv.Itoa(vmid),
"-p",
)
if pidErr != nil {
return nil, fmt.Errorf("resolve container pid: %w", pidErr)
}
pid, err = parseLXCInfoPID(pidOutput)
if err != nil {
return nil, err
}
}
procRoot := fmt.Sprintf("/proc/%d/root", pid)
result := make([]agentshost.Disk, 0, len(mounts))
// The shared prober pins /proc/<pid>, requires it to be this guest's init
// in lxc/<vmid>, resolves every mount inside the guest's root (an absolute
// symlink cannot reach the host), and bounds a read stuck on a hung guest
// filesystem without starting another for the same guest.
byPath := make(map[string]proxmoxLXCConfigMount, len(mounts))
mountpoints := make([]string, 0, len(mounts))
for _, mount := range mounts {
if ctx.Err() != nil {
return nil, ctx.Err()
if _, seen := byPath[mount.Path]; seen || len(mountpoints) == proxmoxLXCMaxObservedMounts {
continue
}
target := path.Join(procRoot, mount.Path)
usage, usageErr := prober.FilesystemUsage(target)
if usageErr != nil {
return nil, fmt.Errorf("probe %s filesystem usage: %w", mount.Key, usageErr)
byPath[mount.Path] = mount
mountpoints = append(mountpoints, mount.Path)
}
observations, err := observer.ObserveProxmoxLXC(ctx, filesystemprobe.ContainerRequest{
PID: pid,
ContainerID: strconv.Itoa(vmid),
Runtime: "lxc",
Mountpoints: mountpoints,
MountsOnly: true,
})
if err != nil {
return nil, fmt.Errorf("observe container filesystems: %w", err)
}
result := make([]agentshost.Disk, 0, len(observations))
for _, observation := range observations {
mount, known := byPath[observation.Mountpoint]
if !known {
return nil, fmt.Errorf("observation for unrequested mount %q", observation.Mountpoint)
}
if usage.TotalBytes <= 0 {
if filesystemprobe.IsNotMounted(observation) {
// Declared in the config but not mounted in the running guest; its
// path would report the parent filesystem's numbers.
continue
}
if observation.Error != "" || observation.Usage == nil {
return nil, fmt.Errorf("probe %s filesystem usage: %s", mount.Key, observation.Error)
}
usage := observation.Usage
if usage.CapacityBytes == 0 || usage.CapacityBytes > math.MaxInt64 {
return nil, fmt.Errorf("probe %s filesystem usage: invalid total bytes", mount.Key)
}
if mount.Path != "/" {
parent, parentErr := prober.FilesystemUsage(path.Dir(target))
if parentErr != nil {
return nil, fmt.Errorf("probe %s parent filesystem usage: %w", mount.Key, parentErr)
}
if parent.Device == usage.Device {
continue
}
}
used := usage.UsedBytes
total := int64(usage.CapacityBytes)
used := total - int64(usage.FreeBytes)
if used < 0 {
used = 0
}
if used > usage.TotalBytes {
used = usage.TotalBytes
}
avail := usage.AvailBytes
if avail < 0 {
avail = 0
}
if avail > usage.TotalBytes {
avail = usage.TotalBytes
avail := int64(usage.AvailableBytes)
if avail > total {
avail = total
}
usagePct := 0.0
if denominator := used + avail; denominator > 0 {
@@ -340,14 +359,11 @@ func (a *Agent) collectProxmoxLXCContainerDisksFast(
Device: mount.Volume,
Mountpoint: mount.Path,
Type: mount.Key,
TotalBytes: usage.TotalBytes,
TotalBytes: total,
UsedBytes: used,
FreeBytes: avail,
Usage: usagePct,
})
if len(result) == proxmoxLXCMaxDisks {
break
}
}
if len(result) == 0 {
return nil, errors.New("no mounted container filesystems resolved")
@@ -1,41 +0,0 @@
//go:build linux
package hostagent
import (
"fmt"
"syscall"
)
// FilesystemUsage satisfies filesystemUsageProber on Linux. Statfs through a
// /proc/<pid>/root path traverses into that process's mount namespace, so the
// numbers match what df reports inside the container.
func (c *defaultCollector) FilesystemUsage(path string) (hostFilesystemUsage, error) {
var fs syscall.Statfs_t
if err := syscall.Statfs(path, &fs); err != nil {
return hostFilesystemUsage{}, fmt.Errorf("statfs %s: %w", path, err)
}
var st syscall.Stat_t
if err := syscall.Stat(path, &st); err != nil {
return hostFilesystemUsage{}, fmt.Errorf("stat %s: %w", path, err)
}
blockSize := int64(fs.Frsize)
if blockSize <= 0 {
blockSize = int64(fs.Bsize)
}
if blockSize <= 0 {
return hostFilesystemUsage{}, fmt.Errorf("statfs %s: invalid block size", path)
}
total := int64(fs.Blocks) * blockSize
free := int64(fs.Bfree) * blockSize
used := total - free
if used < 0 {
used = 0
}
return hostFilesystemUsage{
TotalBytes: total,
UsedBytes: used,
AvailBytes: int64(fs.Bavail) * blockSize,
Device: uint64(st.Dev),
}, nil
}
@@ -4,11 +4,16 @@ import (
"context"
"errors"
"fmt"
"io/fs"
"os"
"path"
"strconv"
"strings"
"testing"
"time"
"github.com/rcourtman/pulse-go-rewrite/internal/filesystemprobe"
"github.com/rcourtman/pulse-go-rewrite/pkg/agents/filesystem"
"github.com/rs/zerolog"
)
@@ -153,16 +158,15 @@ mp2: tank:snap-only,mp=/snap-only,size=5G
`,
"/etc/pve/lxc/101.conf": "rootfs: local-lvm:vm-101-disk-0,size=4G\n",
}
usageByPath := map[string]hostFilesystemUsage{
"/proc/4242/root": {TotalBytes: 8 << 30, UsedBytes: 2 << 30, AvailBytes: 6 << 30, Device: 11},
"/proc/4242/root/srv": {TotalBytes: 8 << 30, UsedBytes: 2 << 30, AvailBytes: 6 << 30, Device: 11},
"/proc/4242/root/srv/data": {TotalBytes: 1 << 40, UsedBytes: 512 << 30, AvailBytes: 512 << 30, Device: 12},
// mp1 is configured but not mounted in the running container, so the
// path resolves to the rootfs device and must be dropped, not reported
// with the parent's numbers.
"/proc/4242/root/mnt": {TotalBytes: 8 << 30, UsedBytes: 2 << 30, AvailBytes: 6 << 30, Device: 11},
"/proc/4242/root/mnt/missing": {TotalBytes: 8 << 30, UsedBytes: 2 << 30, AvailBytes: 6 << 30, Device: 11},
"/proc/5252/root": {TotalBytes: 4 << 30, UsedBytes: 1 << 30, AvailBytes: 3 << 30, Device: 21},
observations := map[int]map[string]lxcObservation{
4242: {
"/": lxcUsage(8<<30, 2<<30, 6<<30),
"/srv/data": lxcUsage(1<<40, 512<<30, 512<<30),
// mp1 is configured but not mounted in the running container, so
// the prober rejects it instead of reporting the parent's numbers.
"/mnt/missing": {notMounted: true},
},
5252: {"/": lxcUsage(4<<30, 1<<30, 3<<30)},
}
collector := &mockCollector{
goos: "linux",
@@ -182,13 +186,7 @@ mp2: tank:snap-only,mp=/snap-only,size=5G
}
return []byte(config), nil
},
filesystemUsageFn: func(path string) (hostFilesystemUsage, error) {
usage, ok := usageByPath[path]
if !ok {
t.Fatalf("unexpected statfs path %q", path)
}
return usage, nil
},
observeLXCFn: fakeLXCObserve(t, observations),
commandCombinedOutputLimitedFn: func(
_ context.Context,
_ int,
@@ -267,12 +265,9 @@ func TestCollectProxmoxLXCFilesystemsFallsBackToPctDFPerContainer(t *testing.T)
// back to pct df.
return nil, os.ErrPermission
},
filesystemUsageFn: func(path string) (hostFilesystemUsage, error) {
if path != "/proc/4242/root" {
t.Fatalf("unexpected statfs path %q", path)
}
return hostFilesystemUsage{TotalBytes: 8 << 30, UsedBytes: 2 << 30, AvailBytes: 6 << 30, Device: 11}, nil
},
observeLXCFn: fakeLXCObserve(t, map[int]map[string]lxcObservation{
4242: {"/": lxcUsage(8<<30, 2<<30, 6<<30)},
}),
commandCombinedOutputLimitedFn: func(
_ context.Context,
_ int,
@@ -345,17 +340,12 @@ mp0 tank:subvol-100-disk-0 1.0T 512.0G 512.0G 50.0 /srv/data
readFileFn: func(string) ([]byte, error) {
return []byte("rootfs: local-lvm:vm-100-disk-0,size=8G\nmp0: tank:subvol-100-disk-0,mp=/srv/data,size=1T\n"), nil
},
filesystemUsageFn: func(target string) (hostFilesystemUsage, error) {
switch target {
case "/proc/4242/root":
return hostFilesystemUsage{TotalBytes: 8 << 30, UsedBytes: 2 << 30, AvailBytes: 6 << 30, Device: 11}, nil
case "/proc/4242/root/srv/data":
return hostFilesystemUsage{}, errors.New("statfs access failed")
default:
t.Fatalf("unexpected statfs path %q", target)
return hostFilesystemUsage{}, nil
}
},
observeLXCFn: fakeLXCObserve(t, map[int]map[string]lxcObservation{
4242: {
"/": lxcUsage(8<<30, 2<<30, 6<<30),
"/srv/data": {err: "read filesystem counters: permission denied"},
},
}),
commandCombinedOutputLimitedFn: func(_ context.Context, _ int, name string, args ...string) (string, error) {
joined := strings.Join(args, " ")
switch {
@@ -469,3 +459,281 @@ mp5 pool:hot 10.0G 1.0G 9.0G 101.0 /hot
t.Fatal("expected oversized pct df error")
}
}
type fakeLXCDirEntry struct {
name string
dir bool
}
func (e fakeLXCDirEntry) Name() string { return e.name }
func (e fakeLXCDirEntry) IsDir() bool { return e.dir }
func (e fakeLXCDirEntry) Type() fs.FileMode {
if e.dir {
return fs.ModeDir
}
return 0
}
func (e fakeLXCDirEntry) Info() (fs.FileInfo, error) { return nil, fs.ErrNotExist }
// fakeProxmoxLXCHost models what socket-free discovery reads on a cgroup v2
// Proxmox node. The layout follows a real PVE 9.2 node: configs under
// /etc/pve/lxc, the payload under /sys/fs/cgroup/lxc/<vmid>/ns, and the
// container init identified by an NSpid ending in 1.
type fakeProxmoxLXCHost struct {
files map[string]string
dirs map[string]map[string]bool
observations map[int]map[string]lxcObservation
}
func newFakeProxmoxLXCHost() *fakeProxmoxLXCHost {
h := &fakeProxmoxLXCHost{
files: map[string]string{proxmoxLXCCgroupV2Marker: "cpuset cpu io memory pids\n"},
dirs: map[string]map[string]bool{},
observations: map[int]map[string]lxcObservation{},
}
h.dir(proxmoxLXCConfigDir)
return h
}
func (h *fakeProxmoxLXCHost) dir(name string) {
if _, ok := h.dirs[name]; !ok {
h.dirs[name] = map[string]bool{}
}
}
func (h *fakeProxmoxLXCHost) config(vmid int, content string) {
h.dirs[proxmoxLXCConfigDir][strconv.Itoa(vmid)+".conf"] = false
h.files[proxmoxLXCConfigPath(vmid)] = content
}
// cgroup registers a cgroup directory (relative to lxc/) and its processes.
func (h *fakeProxmoxLXCHost) cgroup(rel string, pids ...int) {
full := path.Join(proxmoxLXCCgroupRoot, rel)
for child := full; child != proxmoxLXCCgroupRoot; child = path.Dir(child) {
h.dir(child)
// Every cgroup directory has a cgroup.procs, empty when its
// processes live in child cgroups.
if _, ok := h.files[path.Join(child, "cgroup.procs")]; !ok {
h.files[path.Join(child, "cgroup.procs")] = ""
}
parent := path.Dir(child)
h.dir(parent)
h.dirs[parent][path.Base(child)] = true
}
fields := make([]string, 0, len(pids))
for _, pid := range pids {
fields = append(fields, strconv.Itoa(pid))
}
h.files[path.Join(full, "cgroup.procs")] = strings.Join(fields, "\n") + "\n"
}
func (h *fakeProxmoxLXCHost) proc(pid int, nspid, cgroup string) {
h.files["/proc/"+strconv.Itoa(pid)+"/status"] = "Name:\tinit\nNSpid:\t" + nspid + "\n"
h.files["/proc/"+strconv.Itoa(pid)+"/cgroup"] = "0::" + cgroup + "\n"
}
// collector serves the fake host. pct and lxc-info exist but every call fails
// the way they fail inside the PrivateNetwork helper.
func (h *fakeProxmoxLXCHost) collector(t *testing.T) (*mockCollector, *int) {
t.Helper()
commands := 0
return &mockCollector{
goos: "linux",
lookPathFn: func(file string) (string, error) {
switch file {
case "pct":
return "/usr/sbin/pct", nil
case "lxc-info":
return "/usr/bin/lxc-info", nil
}
return "", os.ErrNotExist
},
statFn: func(name string) (os.FileInfo, error) {
if _, ok := h.files[name]; ok {
return nil, nil
}
if _, ok := h.dirs[name]; ok {
return nil, nil
}
return nil, os.ErrNotExist
},
readFileFn: func(name string) ([]byte, error) {
content, ok := h.files[name]
if !ok {
return nil, os.ErrNotExist
}
return []byte(content), nil
},
readDirFn: func(name string) ([]os.DirEntry, error) {
children, ok := h.dirs[name]
if !ok {
return nil, os.ErrNotExist
}
entries := make([]os.DirEntry, 0, len(children))
for child, isDir := range children {
entries = append(entries, fakeLXCDirEntry{name: child, dir: isDir})
}
return entries, nil
},
observeLXCFn: fakeLXCObserve(t, h.observations),
commandCombinedOutputLimitedFn: func(_ context.Context, _ int, name string, args ...string) (string, error) {
commands++
return "ipcc_send_rec[1] failed: Connection refused", errors.New("exit status 2")
},
}, &commands
}
func TestProxmoxLXCConfigNameMatchesPctList(t *testing.T) {
cases := []struct {
config string
want string
}{
{"#managed\narch: amd64\nhostname: web\nrootfs: x,size=1G\n", "web"},
{"arch: amd64\nrootfs: x,size=1G\n", "CT126"},
{"hostname: \n", "CT126"},
{"arch: amd64\n\n[snap]\nhostname: old\n", "CT126"},
}
for _, tc := range cases {
if got := proxmoxLXCConfigName(tc.config, 126); got != tc.want {
t.Errorf("proxmoxLXCConfigName(%q) = %q, want %q", tc.config, got, tc.want)
}
}
}
func TestDiscoverRunningProxmoxLXCContainersFindsInitWithoutSockets(t *testing.T) {
h := newFakeProxmoxLXCHost()
// Alpine guest running Docker: a nested container's PID 1 shares the
// guest's cgroup listing and comes first, but is not the guest's init.
h.config(126, "hostname: qual2511\nrootfs: lvm:vm-126-disk-0,size=1G\n")
h.cgroup("126")
h.cgroup("126/ns", 3019900, 3019673, 3019253)
h.cgroup("126/ns/openrc.syslog", 3019638)
h.proc(3019900, "3019900\t420\t1", "/lxc/126/ns")
h.proc(3019673, "3019673\t413", "/lxc/126/ns")
h.proc(3019253, "3019253\t1", "/lxc/126/ns")
// systemd guest: init in ns/init.scope, services elsewhere.
h.config(127, "rootfs: lvm:vm-127-disk-0,size=1G\n")
h.cgroup("127/ns/system.slice/cron.service", 4100)
h.cgroup("127/ns/init.scope", 4000)
h.proc(4100, "4100\t88", "/lxc/127/ns/system.slice/cron.service")
h.proc(4000, "4000\t1", "/lxc/127/ns/init.scope")
// Stopped: config only, no cgroup.
h.config(128, "hostname: stopped\nrootfs: lvm:vm-128-disk-0,size=1G\n")
collector, commands := h.collector(t)
agent := &Agent{logger: zerolog.Nop(), collector: collector}
got, ok := agent.discoverRunningProxmoxLXCContainers(context.Background())
if !ok {
t.Fatal("discovery was not applicable on a cgroup v2 Proxmox host")
}
want := []proxmoxLXCRunningContainer{
{VMID: 126, Name: "qual2511", PID: 3019253},
{VMID: 127, Name: "CT127", PID: 4000},
}
if fmt.Sprint(got) != fmt.Sprint(want) {
t.Fatalf("discovered %+v, want %+v", got, want)
}
if *commands != 0 {
t.Fatalf("discovery ran %d commands, want none", *commands)
}
}
// Discovery must never turn a running guest it could not read into a smaller,
// healthy-looking inventory. Whenever a guest's state is not established it
// hands the whole node back to pct, which works for root installs and is
// reported as degraded inside the helper.
func TestDiscoverRunningProxmoxLXCContainersLeavesUnestablishedStateToPct(t *testing.T) {
running := func() *fakeProxmoxLXCHost {
h := newFakeProxmoxLXCHost()
h.config(126, "hostname: web\n")
h.cgroup("126/ns", 10)
h.proc(10, "10\t1", "/lxc/126/ns")
return h
}
cases := map[string]func(h *fakeProxmoxLXCHost) context.Context{
"cgroup v1 host": func(h *fakeProxmoxLXCHost) context.Context {
delete(h.files, proxmoxLXCCgroupV2Marker)
return context.Background()
},
"unreadable config of a running guest": func(h *fakeProxmoxLXCHost) context.Context {
delete(h.files, proxmoxLXCConfigPath(126))
return context.Background()
},
"guest cgroup without an init (a reused PID from another guest)": func(h *fakeProxmoxLXCHost) context.Context {
h.proc(10, "10\t1", "/lxc/130/ns")
return context.Background()
},
"unreadable cgroup.procs": func(h *fakeProxmoxLXCHost) context.Context {
delete(h.files, path.Join(proxmoxLXCCgroupRoot, "126/ns/cgroup.procs"))
return context.Background()
},
"guest-created cgroup tree beyond the search limit": func(h *fakeProxmoxLXCHost) context.Context {
h.proc(10, "10\t2", "/lxc/126/ns")
for i := 0; i < proxmoxLXCMaxCgroupDirs; i++ {
h.cgroup("126/ns/c" + strconv.Itoa(i))
}
return context.Background()
},
"cancelled collection": func(*fakeProxmoxLXCHost) context.Context {
ctx, cancel := context.WithCancel(context.Background())
cancel()
return ctx
},
}
for name, mutate := range cases {
t.Run(name, func(t *testing.T) {
h := running()
ctx := mutate(h)
collector, _ := h.collector(t)
got, ok := (&Agent{logger: zerolog.Nop(), collector: collector}).discoverRunningProxmoxLXCContainers(ctx)
if ok {
t.Fatalf("discovery reported %+v as complete", got)
}
})
}
}
// lxcObservation is one mount's answer from the fake filesystem prober.
type lxcObservation struct {
usage *filesystem.Usage
notMounted bool
err string
}
func lxcUsage(total, used, avail uint64) lxcObservation {
return lxcObservation{usage: &filesystem.Usage{CapacityBytes: total, FreeBytes: total - used, AvailableBytes: avail}}
}
// fakeLXCObserve answers prober requests from per-PID mount tables. A PID the
// table does not know is not the guest's init, as the real prober would find.
func fakeLXCObserve(t *testing.T, byPID map[int]map[string]lxcObservation) func(context.Context, filesystemprobe.ContainerRequest) ([]filesystem.Observation, error) {
t.Helper()
return func(_ context.Context, req filesystemprobe.ContainerRequest) ([]filesystem.Observation, error) {
if req.Runtime != "lxc" || !req.MountsOnly || req.ContainerID == "" {
t.Errorf("prober request = %+v, want an exact lxc request with MountsOnly", req)
}
mounts, ok := byPID[req.PID]
if !ok {
return nil, errors.New("process is not the container's init")
}
out := make([]filesystem.Observation, 0, len(req.Mountpoints))
for _, mountpoint := range req.Mountpoints {
answer, known := mounts[mountpoint]
if !known {
t.Errorf("unexpected mount %q for pid %d", mountpoint, req.PID)
continue
}
observation := filesystem.Observation{Mountpoint: mountpoint, Source: filesystemprobe.Source}
switch {
case answer.notMounted:
observation.Error = filesystemprobe.ErrNotMounted.Error()
case answer.err != "":
observation.Error = answer.err
default:
observation.Usage = answer.usage
}
out = append(out, observation)
}
return out, nil
}
}
+243
View File
@@ -0,0 +1,243 @@
package hostagent
import (
"context"
"errors"
"os"
"path"
"sort"
"strconv"
"strings"
"github.com/rcourtman/pulse-go-rewrite/internal/filesystemprobe"
"github.com/rcourtman/pulse-go-rewrite/pkg/agents/filesystem"
)
// Socket-free discovery of the node's running containers (#2511).
//
// pct list, pct df and lxc-info reach pmxcfs and the LXC monitor through
// abstract Unix sockets, which belong to a network namespace. The typed helper
// runs with PrivateNetwork=true, so every one of them fails there. Container
// configs are ordinary reads through the pmxcfs FUSE mount, and a running
// container's init process is visible in its cgroup and in /proc, so the helper
// can find what it needs without leaving its sandbox. Usage is then read by
// the shared filesystem prober, which pins the process, re-checks its identity
// and resolves every path inside the container's root.
const (
// proxmoxLXCCgroupRoot is where pve-container places each container's
// cgroup (lxc/<vmid>, with the payload under ns/) on a cgroup v2 host.
proxmoxLXCCgroupRoot = "/sys/fs/cgroup/lxc"
proxmoxLXCCgroupV2Marker = "/sys/fs/cgroup/cgroup.controllers"
proxmoxLXCMaxCgroupDirs = 64
proxmoxLXCMaxCgroupPIDs = 4096
proxmoxLXCMaxCgroupProcsBytes = 64 * 1024
proxmoxLXCMaxProcStatusBytes = 64 * 1024
proxmoxLXCMaxProcCgroupBytes = 64 * 1024
proxmoxLXCMaxConfigEntries = 4096
)
// proxmoxLXCDirLister is an optional collector capability, like
// proxmoxLXCObserver: the default collector lists directories and test
// collectors opt in.
type proxmoxLXCDirLister interface {
ReadDir(name string) ([]os.DirEntry, error)
}
// proxmoxLXCObserver reads a running container's filesystem counters through
// its pinned init process.
type proxmoxLXCObserver interface {
ObserveProxmoxLXC(ctx context.Context, req filesystemprobe.ContainerRequest) ([]filesystem.Observation, error)
}
// proxmoxLXCFilesystemObserver is shared by every collection in the process.
// A probe stuck on a guest's hung filesystem keeps its slot here, so the
// typed helper never stacks a second probe on that guest.
var proxmoxLXCFilesystemObserver filesystemprobe.Observer
func (c *defaultCollector) ReadDir(name string) ([]os.DirEntry, error) {
return os.ReadDir(name)
}
func (c *defaultCollector) ObserveProxmoxLXC(ctx context.Context, req filesystemprobe.ContainerRequest) ([]filesystem.Observation, error) {
return proxmoxLXCFilesystemObserver.Observe(ctx, req)
}
type proxmoxLXCInitState int
const (
proxmoxLXCStopped proxmoxLXCInitState = iota
proxmoxLXCRunning
proxmoxLXCUnknown
)
// discoverRunningProxmoxLXCContainers lists this node's running containers
// with their init PIDs. ok is false when the host cannot be read this way or
// any container's state cannot be established (no directory listing, no
// cgroup v2 hierarchy, an unreadable config or cgroup, a search limit, or
// cancellation), so the caller uses pct instead of reporting an inventory that
// silently misses a running guest. Only a container with no cgroup at all is
// stopped and left out, as pct list leaves it out.
func (a *Agent) discoverRunningProxmoxLXCContainers(ctx context.Context) ([]proxmoxLXCRunningContainer, bool) {
lister, isLister := a.collector.(proxmoxLXCDirLister)
if !isLister {
return nil, false
}
if _, err := a.collector.Stat(proxmoxLXCCgroupV2Marker); err != nil {
return nil, false
}
entries, err := lister.ReadDir(proxmoxLXCConfigDir)
if err != nil {
a.logger.Debug().Err(err).Msg("Cannot list Proxmox LXC configs; using pct")
return nil, false
}
if len(entries) > proxmoxLXCMaxConfigEntries {
return nil, false
}
vmids := make([]int, 0, len(entries))
for _, entry := range entries {
name := entry.Name()
if !strings.HasSuffix(name, ".conf") {
continue
}
vmid, convErr := strconv.Atoi(strings.TrimSuffix(name, ".conf"))
if convErr != nil || vmid < 100 || vmid > 999999999 {
continue
}
vmids = append(vmids, vmid)
}
sort.Ints(vmids)
running := make([]proxmoxLXCRunningContainer, 0, len(vmids))
for _, vmid := range vmids {
if ctx.Err() != nil {
return nil, false
}
pid, state := a.proxmoxLXCInitPID(ctx, lister, vmid)
switch state {
case proxmoxLXCStopped:
continue
case proxmoxLXCUnknown:
a.logger.Debug().Int("vmid", vmid).Msg("Cannot establish a Proxmox LXC init process; using pct")
return nil, false
}
raw, readErr := a.collector.ReadFile(proxmoxLXCConfigPath(vmid))
if readErr != nil || len(raw) > proxmoxLXCMaxConfigBytes {
// Without its config a running container has no trustworthy name or
// mounts; leave the whole node to pct rather than guess.
a.logger.Debug().Err(readErr).Int("vmid", vmid).Msg("Cannot read a running Proxmox LXC config; using pct")
return nil, false
}
running = append(running, proxmoxLXCRunningContainer{VMID: vmid, Name: proxmoxLXCConfigName(string(raw), vmid), PID: pid})
if len(running) == proxmoxLXCMaxContainers {
break
}
}
return running, true
}
// proxmoxLXCInitPID finds the container's init process: the process in the
// container's cgroup tree that is PID 1 of the namespace directly below the
// host's and whose own cgroup is inside lxc/<vmid>. The filesystem prober
// re-checks both on the pinned process before reading anything.
func (a *Agent) proxmoxLXCInitPID(ctx context.Context, lister proxmoxLXCDirLister, vmid int) (int, proxmoxLXCInitState) {
root := path.Join(proxmoxLXCCgroupRoot, strconv.Itoa(vmid))
if _, err := lister.ReadDir(root); err != nil {
if errors.Is(err, os.ErrNotExist) {
return 0, proxmoxLXCStopped
}
return 0, proxmoxLXCUnknown
}
queue := []string{root}
queued := 1
checked := 0
for len(queue) > 0 {
if ctx.Err() != nil {
return 0, proxmoxLXCUnknown
}
dir := queue[0]
queue = queue[1:]
raw, err := a.collector.ReadFile(path.Join(dir, "cgroup.procs"))
if err != nil || len(raw) > proxmoxLXCMaxCgroupProcsBytes {
return 0, proxmoxLXCUnknown
}
for _, field := range strings.Fields(string(raw)) {
if checked == proxmoxLXCMaxCgroupPIDs {
return 0, proxmoxLXCUnknown
}
checked++
pid, convErr := strconv.Atoi(field)
if convErr != nil || pid <= 0 {
continue
}
if a.isProxmoxLXCInit(pid, vmid) {
return pid, proxmoxLXCRunning
}
}
entries, err := lister.ReadDir(dir)
if err != nil {
return 0, proxmoxLXCUnknown
}
children := make([]string, 0, len(entries))
for _, entry := range entries {
if entry.IsDir() {
children = append(children, entry.Name())
}
}
// systemd guests run init in init.scope; look there first.
sort.SliceStable(children, func(i, j int) bool {
return children[i] == "init.scope" && children[j] != "init.scope"
})
for _, child := range children {
if queued == proxmoxLXCMaxCgroupDirs {
// A guest can create cgroups in its delegated subtree; give up
// rather than walk an unbounded tree.
return 0, proxmoxLXCUnknown
}
queued++
queue = append(queue, path.Join(dir, child))
}
}
// A cgroup without an init is a guest starting or stopping.
return 0, proxmoxLXCUnknown
}
func (a *Agent) isProxmoxLXCInit(pid, vmid int) bool {
status, err := a.collector.ReadFile("/proc/" + strconv.Itoa(pid) + "/status")
if err != nil || len(status) > proxmoxLXCMaxProcStatusBytes {
return false
}
if !filesystemprobe.IsNamespaceInit(string(status), pid) {
return false
}
cgroup, err := a.collector.ReadFile("/proc/" + strconv.Itoa(pid) + "/cgroup")
if err != nil || len(cgroup) > proxmoxLXCMaxProcCgroupBytes {
return false
}
return filesystemprobe.CgroupMatches(string(cgroup), "lxc", strconv.Itoa(vmid))
}
// proxmoxLXCConfigName is the name pct list and the Proxmox API report: the
// configured hostname, or CT<vmid> when none is set. Pulse matches agent
// readings to API containers by this name.
func proxmoxLXCConfigName(config string, vmid int) string {
for _, rawLine := range strings.Split(config, "\n") {
line := strings.TrimSpace(rawLine)
if strings.HasPrefix(line, "[") {
break
}
value, ok := strings.CutPrefix(line, "hostname:")
if !ok {
continue
}
if name := strings.TrimSpace(value); name != "" && safeProxmoxLXCText(name, proxmoxLXCMaxNameBytes) {
return name
}
break
}
return "CT" + strconv.Itoa(vmid)
}