mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-10-23 20:25:56 +00:00
Collect LXC filesystems without pmxcfs or LXC socket IPC (#2562)
Fixes #2511. **Problem.** The typed helper runs with `PrivateNetwork=true`. `pct list`, `pct df` and `lxc-info` reach pmxcfs and the LXC monitor through abstract Unix sockets, which belong to the network namespace, so inside the helper they always fail. Least-privilege installs using the helper therefore report `provider_unavailable` for `proxmox.lxc_filesystems` on every report, and never get per-mount-point guest usage. The Proxmox API only reports the root filesystem. Reproduced on PVE 9.2.18: `pct list` works normally and fails with `ipcc_send_rec ... Connection refused` under `PrivateNetwork=yes`. The fix keeps the helper's hardening exactly as it is. **Discovery without commands** (`internal/hostagent/proxmox_lxc_running.go`) - Lists `/etc/pve/lxc/*.conf` through the pmxcfs mount, which is a file read rather than IPC. - Finds each guest's init in `/sys/fs/cgroup/lxc/<vmid>`: `NSpid` exactly `<pid> 1`, and the process's own cgroup inside `lxc/<vmid>`. - Names guests by `hostname` or `CT<vmid>`, as `pct list` and the API do, so monitoring's exact name match keeps working. - Hands the whole node back to `pct list` whenever it cannot establish a guest's state: no cgroup v2, an unreadable listing, config, cgroup or process, a search limit, or cancellation. A guest it could not read never silently disappears from a healthy-looking inventory. **Reads through the shared filesystem prober** (`internal/filesystemprobe`), replacing the separate plain-`statfs` path. - New `lxc` requests. The prober pins `/proc/<pid>`, requires the exact `lxc/<vmid>` cgroup and the guest's init before and after reading, so the PID 1 of a namespace nested inside the guest (Docker in LXC) cannot stand in. It resolves every mount inside the guest root. - `MountsOnly` reports a declared mount that isn't mounted (same device as its parent) as `ErrNotMounted`, instead of copying the parent's numbers. - One bounded probe per guest is kept across the process, so a guest's hung FUSE mount cannot stack probes in the root helper. - The helper's `RestrictSUIDSGID=true` makes systemd answer `openat2` with ENOSYS, because seccomp cannot inspect its flags. The prober now falls back to a component-by-component walk with `O_NOFOLLOW` that refuses every symbolic link instead of resolving it. Docker and Podman reads get the same fallback. **Real-node verification.** On a throwaway PVE 9.2.18 node, run under the helper unit's exact sandbox properties (copied from `install.sh`): - An unprivileged guest (rootfs plus `mp0`) and a privileged guest returned readings matching `pct df`. - A stopped guest was omitted. - The result was not degraded. - The prober's Linux tests passed as root on the node. A first version that resolved paths only with `openat2` failed inside the sandbox; that is what found the ENOSYS case. The throwaway containers, template and binaries were removed afterwards. A live nested PID namespace could not be reproduced in that guest, so that case is covered by unit tests only. **Tests** - `TestHelperProxmoxLXCFilesystemsNeedNoPctOrLXCSockets`: the collector the helper's provider runs, with every command failing, must return a complete inventory. It fails without this change. - `TestDiscoverRunningProxmoxLXCContainersFindsInitWithoutSockets`: covers a nested init listed first, systemd guests and stopped guests. - `TestDiscoverRunningProxmoxLXCContainersLeavesUnestablishedStateToPct`: covers cgroup v1, an unreadable config, a reused PID, an unreadable `cgroup.procs`, a cgroup tree beyond the search limit, and cancellation. - `TestProxmoxLXCIdentityIsExact`, `TestObserverRejectsInexactProxmoxVMIDs`, `TestMountsOnlyDetectsUnmountedPathsWithinRoot`, `TestNonInitProcessCannotSupplyProxmoxGuestCapacity` and `TestComponentWalkRefusesSymlinksAndStaysBeneathRoot`: cover the prober's new identity, mount and fallback rules. - `go test ./internal/hostagent ./internal/filesystemprobe ./internal/dockeragent ./cmd/pulse-agent-helper` passes, and `go vet` is clean for Linux. The `AGENT_SECURITY.md` capability row and the agent-lifecycle and monitoring contracts are updated. The row still says this is not full qualification: one PVE version, and no timeout or helper-loss evidence. **Overlap with #2551.** #2551 touches the same collector, `privilege_helper_client*.go` and the same three docs. It hardens `pct list` parsing and adds v2 partial inventories (#2513), but still starts from `pct list`, so inside the helper it would keep failing at that first command. The two are complementary: - #2551's stricter `pct list` parsing applies to the fallback path here. - Its partial-inventory reporting wraps the same collection loop. Whichever lands second needs a textual rebase in `collectProxmoxLXCFilesystemsResult` and the docs.
This commit is contained in:
@@ -279,7 +279,7 @@ or generic command path.
|
||||
| Proxmox VE/PBS/PMG inventory, status, storage, and ordinary metrics | API-only connection with a narrowly scoped token; no host agent | No collector, helper, or runner authority is required for this data | Supported independently of the safe host-agent profile; it does not prove host-local SMART, LXC filesystem, or action parity | `agent-lifecycle`: keep API permissions and returned telemetry covered by provider tests |
|
||||
| Standard Linux systemd host telemetry and collector update | Unprivileged `pulse-agent` plus the root-owned typed helper | Core `/proc`, filesystem, network, RAID, and hwmon telemetry stays in the collector; signed updates cross the fixed helper protocol and commit only after an authoritative report | **Qualified on disposable Ubuntu 24.04.4 arm64 at committed main `22fd662fb794f63efb9d3ca2158de73c4e07e1b8`** across the schema-v6 twenty-scenario run. It covers install, migration, explicit/automatic rollback, helper protocol and resource boundaries, effective-unit override rejection, reporting continuity, authoritative executable-digest commit, watchdog rollback, helper-restart recovery, and last-known-good restoration. It remains opt-in pending exact-RC reproduction, representative provider/appliance parity, and external review | `deployment-installability` and `security-privacy`: reproduce the same helper transaction from trusted release artifacts on representative hosts and accept the external boundary review |
|
||||
| Linux SMART telemetry | `smart.snapshot` through the no-network helper; no caller-selected device or arguments | Implemented, unqualified on representative physical disks. Helper failure omits/degrades SMART only; the collector does not retry as root | Does not yet justify SMART parity or a default change | `agent-lifecycle`: record live SATA, SAS/controller, USB bridge, and NVMe evidence, including standby, permission failure, timeout, and partial-data cases |
|
||||
| Proxmox node-local LXC filesystem telemetry | `proxmox.lxc_filesystems` through the no-network helper using fixed bounded `pct` operations | Implemented, unqualified on a representative PVE node. Helper failure omits/degrades this snapshot only | Does not yet justify Proxmox host-agent parity or a default change | `agent-lifecycle`: record live running/stopped LXC, mount, timeout, output-bound, and helper-loss behavior on supported PVE versions |
|
||||
| Proxmox node-local LXC filesystem telemetry | `proxmox.lxc_filesystems` through the no-network helper. Configs are read through the pmxcfs mount, running guests and their init processes come from cgroup v2 and `/proc`, and usage comes from the shared filesystem prober, which pins the guest's init, re-checks its identity, and resolves each mount inside the guest root (openat2, or a symlink-refusing component walk where the helper's seccomp filter blocks openat2); the caller selects no VMID, path, or command. `pct` and `lxc-info` cannot work inside the helper, because pmxcfs and the LXC monitor listen on abstract Unix sockets that belong to the host network namespace (#2511); they remain the fallback whenever discovery cannot establish a guest's state, on cgroup v1 hosts, for root and `--grant-pct` installs, and per container when the prober read fails | Implemented. Exercised on a disposable PVE 9.2.18 node under the helper unit's sandbox properties, with running unprivileged (rootfs plus a mount point) and privileged guests and a stopped one; readings matched `pct df`. This is one PVE version, with no timeout, output-bound, helper-loss, or live nested-namespace evidence. Helper failure omits/degrades this snapshot only | Does not yet justify Proxmox host-agent parity or a default change | `agent-lifecycle`: record timeout, output-bound, and helper-loss behavior across supported PVE versions |
|
||||
| Rootful Docker or Podman inventory | No direct collector access to a root-equivalent daemon socket | Implemented as a typed-helper summary-only fallback. Migration preserves container ID/name/image/state/status/creation inventory and marks the report `typed-helper-summary`; stats, secondary inventories, update checks, and actions remain unavailable | The standalone rootless packet exercises live same-family rootful Docker and Podman helper fallback plus summary-inventory semantic parity during socket loss. It does not qualify the complete standalone rootful fresh-install, migration, restart, helper-loss/recovery, and bounds matrix. Full rootful parity remains an explicit default blocker; the legacy/root profile is not safe-profile evidence | `agent-lifecycle`: record the remaining standalone lifecycle matrix on representative rootful Docker and Podman and decide explicitly whether reduced telemetry is sufficient |
|
||||
| Collector-owned rootless Docker or Podman | Direct access only to one usable runtime socket owned by the `pulse-agent` UID whose daemon also attests rootless mode | Qualified locally for the exact source-bound monitoring path. Discovery is deferred until the collector account exists and accepts exactly one readable/writable, non-symlink socket owned by that UID across Docker and Podman. Cross-user, root-owned, remote, unreadable, non-rootless-daemon, or ambiguous endpoints are rejected. An installer-owned exact pin survives daemon loss and update; the running collector can move between direct rootless monitoring and rootful typed-helper summary without regaining action authority | **Qualified on distinct disposable Ubuntu 24.04/systemd 255 arm64 hosts at commit `60041ad9e60c282c892f944e04f777b874991a5d`.** All eleven canonical scenarios passed for Docker and all eleven for Podman. Receipt SHA-256: `7a116d63ab0cd1560482165055f0a8c9158ce0a8333a27bd9e8256582a52dfb5`; local attestation SHA-256: `566279ecd7d7dfa89c92f24243e9fcd5ae3e295d4284bd559d4be42f1e83b3b5`. This is local opt-in artifact-bound self-attestation, not published-RC provenance, action/update qualification, external review, or authorization for a default change | `deployment-installability`: reproduce the exact packet from published release artifacts; separately qualify any desired rootless image-update/action surface; retain the opt-in default until the remaining provider, appliance, and review conditions pass |
|
||||
| Separate runner package update and package-cache cleanup | Root-owned `pulse-agent-runner`, host-bound action credential, typed request, postcondition, and durable receipt | The schema-v6 committed-main systemd receipt records a real verified apt-cache mutation, stale-fingerprint refusal, replay, nonce-bound readiness, effective-unit override rejection, exact credential rotation, bodyless self-revocation, and collector/helper continuity. A separate production Router regression exercises HTTPS issuance, WSS admission, encrypted token persistence, failed-rotation rollback, exact socket invalidation, two server restarts, old-secret rejection, and durable self-revoke | Qualified for the exercised systemd fixture paths and focused production Router lifecycle. Focused installer tests also cover atomic pending cancellation and fail-closed credential retention. The evidence does not cover every runner operation, representative providers, or an exact release candidate | `agent-lifecycle` and `api-contracts`: reproduce the combined systemd and production Router path from trusted RC artifacts with representative package-update success/failure/cancellation evidence |
|
||||
|
||||
@@ -572,11 +572,32 @@ standby evidence. Enumeration and each device probe have independent bounded
|
||||
deadlines so one slow disk cannot consume the complete SMART pass, while
|
||||
command errors retain bounded stderr for compatibility diagnosis.
|
||||
On a PVE node, the Unified Agent may also report the mounted filesystem
|
||||
capacity of running LXCs. The collector uses the fixed local `pct list` command
|
||||
to admit only node-local running guests, reads each admitted guest's main
|
||||
`/etc/pve/lxc/<vmid>.conf` mount declarations, resolves its init PID through
|
||||
the fixed `lxc-info -n <vmid> -p` shape, and probes usage through the Linux
|
||||
`/proc/<pid>/root` namespace. A configured path that resolves to the same
|
||||
capacity of running LXCs. On a cgroup v2 node the collector admits node-local
|
||||
guests from the `/etc/pve/lxc/*.conf` listing and admits only those with a live
|
||||
init process: a PID in `/sys/fs/cgroup/lxc/<vmid>`'s tree that is PID 1 of the
|
||||
namespace directly below the host's (`NSpid` exactly `<pid> 1`) and whose own
|
||||
cgroup is inside `lxc/<vmid>`. It names each by its configured hostname or
|
||||
`CT<vmid>`, as `pct list` does, reads its main mount declarations, and reads
|
||||
usage through the shared `internal/filesystemprobe` observer, which pins the
|
||||
process, re-checks that identity, resolves each mount inside the guest root
|
||||
and holds one bounded in-flight probe per guest across the process. No command
|
||||
runs on this path. `pct list`, `pct df` and `lxc-info` reach pmxcfs and the
|
||||
LXC monitor through abstract Unix sockets scoped to the host network
|
||||
namespace, so inside the `PrivateNetwork` typed helper they always fail, and
|
||||
least-privilege installs using that helper reported `provider_unavailable`
|
||||
on every report (#2511). Discovery hands the whole node back to `pct list`
|
||||
whenever it cannot establish a guest's state (no cgroup v2, an unreadable
|
||||
listing, config, cgroup or process, a search limit, or cancellation); only a
|
||||
guest without a cgroup counts as stopped. The fixed `lxc-info -n <vmid> -p`
|
||||
PID lookup remains for that path, and `pct df` remains the per-container
|
||||
fallback, including through the `--grant-pct` wrapper.
|
||||
`TestHelperProxmoxLXCFilesystemsNeedNoPctOrLXCSockets` drives the collector
|
||||
the helper's provider runs with every command failing and requires a complete
|
||||
inventory; `TestDiscoverRunningProxmoxLXCContainersLeavesUnestablishedStateToPct`
|
||||
pins the fallbacks. On a disposable PVE 9.2.18 node, under the helper unit's
|
||||
sandbox properties, it returned unprivileged (rootfs plus mount point) and
|
||||
privileged guests matching `pct df` and omitted a stopped one; a nested PID
|
||||
namespace was not reproduced there and is covered by tests only. A configured path that resolves to the same
|
||||
device as its parent is not a live mount and must be omitted instead of
|
||||
borrowing the parent's capacity. Least-privilege installs that cannot read the
|
||||
config, resolve the PID, or traverse `/proc` may fall back to the fixed
|
||||
|
||||
@@ -844,6 +844,27 @@ tests, container collection boundary tests, ingestion regression and the
|
||||
opt-in owned tmpfs collector fixture. Installed-agent and model qualification
|
||||
are tracked separately in PATROL_ASSISTANT_CUSTOMER_JOURNEY.md.
|
||||
|
||||
The same observer reads Proxmox LXC guests for the host agent and its typed
|
||||
helper (#2511). An `lxc` request names the decimal VMID; the pinned process
|
||||
must sit in that guest's exact `lxc/<vmid>` cgroup and be its init (`NSpid` is
|
||||
exactly `<pid> 1`, so the PID 1 of a namespace nested inside the guest, such
|
||||
as Docker in LXC, cannot stand in for it), checked before and after the read.
|
||||
With `MountsOnly`, a configured mountpoint sharing its parent's device reports
|
||||
`ErrNotMounted` instead of the parent's counters. Docker and Podman identity
|
||||
rules are unchanged. Confined resolution prefers openat2 with `RESOLVE_IN_ROOT`;
|
||||
when the kernel or a systemd seccomp filter answers ENOSYS (the typed helper's
|
||||
`RestrictSUIDSGID=true` blocks openat2 because its flags cannot be inspected),
|
||||
the clean path is walked component by component with `O_NOFOLLOW` and any
|
||||
symbolic link is refused, so neither path resolves outside the guest root.
|
||||
Docker and Podman reads take the same fallback, so a collector under such a
|
||||
filter now gets confined counters instead of unavailable evidence.
|
||||
`TestComponentWalkRefusesSymlinksAndStaysBeneathRoot`,
|
||||
`TestMountsOnlyDetectsUnmountedPathsWithinRoot`,
|
||||
`TestNonInitProcessCannotSupplyProxmoxGuestCapacity` and
|
||||
`TestProxmoxLXCIdentityIsExact` pin these rules; the Linux tests also passed
|
||||
as root on a disposable PVE 9.2.18 node, where the helper-sandboxed collector
|
||||
returned readings matching `pct df` through the component walk.
|
||||
|
||||
Docker mount collection preserves both native `Mounts` records and entries
|
||||
reported only in `HostConfig.Tmpfs`. Existing reported destinations remain
|
||||
authoritative. Additional tmpfs destinations are ordered deterministically,
|
||||
|
||||
@@ -279,7 +279,7 @@ or generic command path.
|
||||
| Proxmox VE/PBS/PMG inventory, status, storage, and ordinary metrics | API-only connection with a narrowly scoped token; no host agent | No collector, helper, or runner authority is required for this data | Supported independently of the safe host-agent profile; it does not prove host-local SMART, LXC filesystem, or action parity | `agent-lifecycle`: keep API permissions and returned telemetry covered by provider tests |
|
||||
| Standard Linux systemd host telemetry and collector update | Unprivileged `pulse-agent` plus the root-owned typed helper | Core `/proc`, filesystem, network, RAID, and hwmon telemetry stays in the collector; signed updates cross the fixed helper protocol and commit only after an authoritative report | **Qualified on disposable Ubuntu 24.04.4 arm64 at committed main `22fd662fb794f63efb9d3ca2158de73c4e07e1b8`** across the schema-v6 twenty-scenario run. It covers install, migration, explicit/automatic rollback, helper protocol and resource boundaries, effective-unit override rejection, reporting continuity, authoritative executable-digest commit, watchdog rollback, helper-restart recovery, and last-known-good restoration. It remains opt-in pending exact-RC reproduction, representative provider/appliance parity, and external review | `deployment-installability` and `security-privacy`: reproduce the same helper transaction from trusted release artifacts on representative hosts and accept the external boundary review |
|
||||
| Linux SMART telemetry | `smart.snapshot` through the no-network helper; no caller-selected device or arguments | Implemented, unqualified on representative physical disks. Helper failure omits/degrades SMART only; the collector does not retry as root | Does not yet justify SMART parity or a default change | `agent-lifecycle`: record live SATA, SAS/controller, USB bridge, and NVMe evidence, including standby, permission failure, timeout, and partial-data cases |
|
||||
| Proxmox node-local LXC filesystem telemetry | `proxmox.lxc_filesystems` through the no-network helper using fixed bounded `pct` operations | Implemented, unqualified on a representative PVE node. Helper failure omits/degrades this snapshot only | Does not yet justify Proxmox host-agent parity or a default change | `agent-lifecycle`: record live running/stopped LXC, mount, timeout, output-bound, and helper-loss behavior on supported PVE versions |
|
||||
| Proxmox node-local LXC filesystem telemetry | `proxmox.lxc_filesystems` through the no-network helper. Configs are read through the pmxcfs mount, running guests and their init processes come from cgroup v2 and `/proc`, and usage comes from the shared filesystem prober, which pins the guest's init, re-checks its identity, and resolves each mount inside the guest root (openat2, or a symlink-refusing component walk where the helper's seccomp filter blocks openat2); the caller selects no VMID, path, or command. `pct` and `lxc-info` cannot work inside the helper, because pmxcfs and the LXC monitor listen on abstract Unix sockets that belong to the host network namespace (#2511); they remain the fallback whenever discovery cannot establish a guest's state, on cgroup v1 hosts, for root and `--grant-pct` installs, and per container when the prober read fails | Implemented. Exercised on a disposable PVE 9.2.18 node under the helper unit's sandbox properties, with running unprivileged (rootfs plus a mount point) and privileged guests and a stopped one; readings matched `pct df`. This is one PVE version, with no timeout, output-bound, helper-loss, or live nested-namespace evidence. Helper failure omits/degrades this snapshot only | Does not yet justify Proxmox host-agent parity or a default change | `agent-lifecycle`: record timeout, output-bound, and helper-loss behavior across supported PVE versions |
|
||||
| Rootful Docker or Podman inventory | No direct collector access to a root-equivalent daemon socket | Implemented as a typed-helper summary-only fallback. Migration preserves container ID/name/image/state/status/creation inventory and marks the report `typed-helper-summary`; stats, secondary inventories, update checks, and actions remain unavailable | The standalone rootless packet exercises live same-family rootful Docker and Podman helper fallback plus summary-inventory semantic parity during socket loss. It does not qualify the complete standalone rootful fresh-install, migration, restart, helper-loss/recovery, and bounds matrix. Full rootful parity remains an explicit default blocker; the legacy/root profile is not safe-profile evidence | `agent-lifecycle`: record the remaining standalone lifecycle matrix on representative rootful Docker and Podman and decide explicitly whether reduced telemetry is sufficient |
|
||||
| Collector-owned rootless Docker or Podman | Direct access only to one usable runtime socket owned by the `pulse-agent` UID whose daemon also attests rootless mode | Qualified locally for the exact source-bound monitoring path. Discovery is deferred until the collector account exists and accepts exactly one readable/writable, non-symlink socket owned by that UID across Docker and Podman. Cross-user, root-owned, remote, unreadable, non-rootless-daemon, or ambiguous endpoints are rejected. An installer-owned exact pin survives daemon loss and update; the running collector can move between direct rootless monitoring and rootful typed-helper summary without regaining action authority | **Qualified on distinct disposable Ubuntu 24.04/systemd 255 arm64 hosts at commit `60041ad9e60c282c892f944e04f777b874991a5d`.** All eleven canonical scenarios passed for Docker and all eleven for Podman. Receipt SHA-256: `7a116d63ab0cd1560482165055f0a8c9158ce0a8333a27bd9e8256582a52dfb5`; local attestation SHA-256: `566279ecd7d7dfa89c92f24243e9fcd5ae3e295d4284bd559d4be42f1e83b3b5`. This is local opt-in artifact-bound self-attestation, not published-RC provenance, action/update qualification, external review, or authorization for a default change | `deployment-installability`: reproduce the exact packet from published release artifacts; separately qualify any desired rootless image-update/action surface; retain the opt-in default until the remaining provider, appliance, and review conditions pass |
|
||||
| Separate runner package update and package-cache cleanup | Root-owned `pulse-agent-runner`, host-bound action credential, typed request, postcondition, and durable receipt | The schema-v6 committed-main systemd receipt records a real verified apt-cache mutation, stale-fingerprint refusal, replay, nonce-bound readiness, effective-unit override rejection, exact credential rotation, bodyless self-revocation, and collector/helper continuity. A separate production Router regression exercises HTTPS issuance, WSS admission, encrypted token persistence, failed-rotation rollback, exact socket invalidation, two server restarts, old-secret rejection, and durable self-revoke | Qualified for the exercised systemd fixture paths and focused production Router lifecycle. Focused installer tests also cover atomic pending cancellation and fail-closed credential retention. The evidence does not cover every runner operation, representative providers, or an exact release candidate | `agent-lifecycle` and `api-contracts`: reproduce the combined systemd and production Router path from trusted RC artifacts with representative package-update success/failure/cancellation evidence |
|
||||
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"path"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
@@ -24,8 +25,24 @@ const (
|
||||
type ContainerRequest struct {
|
||||
PID int
|
||||
ContainerID string
|
||||
// Runtime is docker or podman (ContainerID is the 64-hex ID) or lxc, a
|
||||
// Proxmox container (ContainerID is the decimal VMID and PID must be the
|
||||
// container's init process).
|
||||
Runtime string
|
||||
Mountpoints []string
|
||||
// MountsOnly reports a mountpoint that shares its parent's device as
|
||||
// ErrNotMounted instead of returning the parent filesystem's counters. A
|
||||
// Proxmox config can declare a mount point the running guest never mounted.
|
||||
MountsOnly bool
|
||||
}
|
||||
|
||||
// ErrNotMounted marks an observation whose path is not a distinct mounted
|
||||
// filesystem inside the container. Callers test it with IsNotMounted.
|
||||
var ErrNotMounted = errors.New("path is not a mounted filesystem in the container")
|
||||
|
||||
// IsNotMounted reports whether an observation was rejected as ErrNotMounted.
|
||||
func IsNotMounted(observation filesystem.Observation) bool {
|
||||
return observation.Error == ErrNotMounted.Error()
|
||||
}
|
||||
|
||||
// Observer bounds potentially uninterruptible kernel reads. A timed-out probe
|
||||
@@ -101,12 +118,18 @@ func validateRequest(req ContainerRequest) error {
|
||||
if req.PID <= 0 {
|
||||
return errors.New("container process is unavailable")
|
||||
}
|
||||
if len(req.ContainerID) != 64 || strings.IndexFunc(req.ContainerID, func(r rune) bool {
|
||||
return !(r >= '0' && r <= '9' || r >= 'a' && r <= 'f')
|
||||
}) >= 0 {
|
||||
return errors.New("filesystem observation requires an exact container ID")
|
||||
}
|
||||
if req.Runtime != "docker" && req.Runtime != "podman" {
|
||||
switch req.Runtime {
|
||||
case "docker", "podman":
|
||||
if len(req.ContainerID) != 64 || strings.IndexFunc(req.ContainerID, func(r rune) bool {
|
||||
return !(r >= '0' && r <= '9' || r >= 'a' && r <= 'f')
|
||||
}) >= 0 {
|
||||
return errors.New("filesystem observation requires an exact container ID")
|
||||
}
|
||||
case "lxc":
|
||||
if !validLXCVMID(req.ContainerID) {
|
||||
return errors.New("filesystem observation requires an exact Proxmox VMID")
|
||||
}
|
||||
default:
|
||||
return errors.New("unsupported container runtime")
|
||||
}
|
||||
if len(req.Mountpoints) == 0 || len(req.Mountpoints) > maxMountpoints {
|
||||
@@ -122,9 +145,53 @@ func validateRequest(req ContainerRequest) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func validLXCVMID(id string) bool {
|
||||
vmid, err := strconv.Atoi(id)
|
||||
return err == nil && vmid >= 100 && vmid <= 999999999 && strconv.Itoa(vmid) == id
|
||||
}
|
||||
|
||||
// CgroupMatches reports whether a /proc/<pid>/cgroup body places the process
|
||||
// inside the given container, with the same exact-identity rules the observer
|
||||
// applies to its pinned process.
|
||||
func CgroupMatches(raw, runtime, containerID string) bool {
|
||||
return cgroupMatches(raw, ContainerRequest{Runtime: runtime, ContainerID: containerID})
|
||||
}
|
||||
|
||||
// IsNamespaceInit reports whether a /proc/<pid>/status body describes pid as
|
||||
// PID 1 of a PID namespace one level below the reader's: NSpid is exactly
|
||||
// "<pid> 1". A process that is PID 1 of a namespace nested inside the
|
||||
// container (Docker inside an LXC guest) lists three or more PIDs and is not
|
||||
// the container's init.
|
||||
func IsNamespaceInit(status string, pid int) bool {
|
||||
for _, line := range strings.Split(status, "\n") {
|
||||
rest, ok := strings.CutPrefix(line, "NSpid:")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
fields := strings.Fields(rest)
|
||||
return len(fields) == 2 && fields[0] == strconv.Itoa(pid) && fields[1] == "1"
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// cgroupMatches accepts an exact runtime-owned cgroup component, never a
|
||||
// substring, container name, short ID, or unrelated process with the same PID.
|
||||
func cgroupMatches(raw string, req ContainerRequest) bool {
|
||||
if req.Runtime == "lxc" {
|
||||
// pve-container places a guest under lxc/<vmid> (payload in ns/).
|
||||
// lxc.monitor/<vmid> and other VMIDs sharing a prefix never match.
|
||||
for _, line := range strings.Split(raw, "\n") {
|
||||
parts := strings.SplitN(line, ":", 3)
|
||||
if len(parts) != 3 {
|
||||
continue
|
||||
}
|
||||
components := strings.Split(parts[2], "/")
|
||||
if len(components) >= 3 && components[0] == "" && components[1] == "lxc" && components[2] == req.ContainerID {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
scope := "docker-" + req.ContainerID + ".scope"
|
||||
if req.Runtime == "podman" {
|
||||
scope = "libpod-" + req.ContainerID + ".scope"
|
||||
|
||||
@@ -9,7 +9,9 @@ import (
|
||||
"io"
|
||||
"math"
|
||||
"os"
|
||||
"path"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/agents/filesystem"
|
||||
@@ -25,7 +27,7 @@ func observeContainer(ctx context.Context, req ContainerRequest) ([]filesystem.O
|
||||
return nil, fmt.Errorf("open container process: %w", err)
|
||||
}
|
||||
defer unix.Close(proc)
|
||||
if err := verifyCgroup(proc, req); err != nil {
|
||||
if err := verifyIdentity(proc, req); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
root, err := unix.Openat(proc, "root", unix.O_PATH|unix.O_DIRECTORY|unix.O_CLOEXEC, 0)
|
||||
@@ -33,7 +35,7 @@ func observeContainer(ctx context.Context, req ContainerRequest) ([]filesystem.O
|
||||
return nil, fmt.Errorf("open container filesystem namespace: %w", err)
|
||||
}
|
||||
defer unix.Close(root)
|
||||
if err := verifyCgroup(proc, req); err != nil {
|
||||
if err := verifyIdentity(proc, req); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
observations := make([]filesystem.Observation, 0, len(req.Mountpoints))
|
||||
@@ -42,6 +44,14 @@ func observeContainer(ctx context.Context, req ContainerRequest) ([]filesystem.O
|
||||
return nil, err
|
||||
}
|
||||
observation := filesystem.Observation{Mountpoint: mount, Source: Source}
|
||||
if req.MountsOnly {
|
||||
if mounted, mountErr := mountedWithinRoot(root, mount); mountErr == nil && !mounted {
|
||||
observation.ObservedAt = time.Now().UTC()
|
||||
observation.Error = ErrNotMounted.Error()
|
||||
observations = append(observations, observation)
|
||||
continue
|
||||
}
|
||||
}
|
||||
observation.Type, observation.Usage, err = observePath(root, mount)
|
||||
observation.ObservedAt = time.Now().UTC()
|
||||
if err != nil {
|
||||
@@ -49,12 +59,70 @@ func observeContainer(ctx context.Context, req ContainerRequest) ([]filesystem.O
|
||||
}
|
||||
observations = append(observations, observation)
|
||||
}
|
||||
if err := verifyCgroup(proc, req); err != nil {
|
||||
if err := verifyIdentity(proc, req); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return observations, nil
|
||||
}
|
||||
|
||||
func verifyIdentity(proc int, req ContainerRequest) error {
|
||||
if err := verifyCgroup(proc, req); err != nil {
|
||||
return err
|
||||
}
|
||||
if req.Runtime == "lxc" {
|
||||
return verifyNamespaceInit(proc, req.PID)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// verifyNamespaceInit requires the pinned process to be the guest's init, so a
|
||||
// process of a namespace nested inside the guest cannot supply its own root.
|
||||
func verifyNamespaceInit(proc, pid int) error {
|
||||
fd, err := unix.Openat(proc, "status", unix.O_RDONLY|unix.O_CLOEXEC|unix.O_NOFOLLOW, 0)
|
||||
if err != nil {
|
||||
return fmt.Errorf("read container process status: %w", err)
|
||||
}
|
||||
file := os.NewFile(uintptr(fd), "status")
|
||||
defer file.Close()
|
||||
raw, err := io.ReadAll(io.LimitReader(file, 65537))
|
||||
if err != nil || len(raw) > 65536 {
|
||||
return errors.New("container process status is unavailable")
|
||||
}
|
||||
if !IsNamespaceInit(string(raw), pid) {
|
||||
return errors.New("process is not the container's init")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// mountedWithinRoot reports whether mount is a different filesystem from its
|
||||
// parent directory, both resolved within the resource root.
|
||||
func mountedWithinRoot(root int, mount string) (bool, error) {
|
||||
if mount == "/" {
|
||||
return true, nil
|
||||
}
|
||||
device := func(target string) (uint64, error) {
|
||||
fd, err := openInRoot(root, target)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer unix.Close(fd)
|
||||
var stat unix.Stat_t
|
||||
if err := unix.Fstat(fd, &stat); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return uint64(stat.Dev), nil
|
||||
}
|
||||
own, err := device(mount)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
parent, err := device(path.Dir(mount))
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
return own != parent, nil
|
||||
}
|
||||
|
||||
func verifyCgroup(proc int, req ContainerRequest) error {
|
||||
fd, err := unix.Openat(proc, "cgroup", unix.O_RDONLY|unix.O_CLOEXEC|unix.O_NOFOLLOW, 0)
|
||||
if err != nil {
|
||||
@@ -72,13 +140,57 @@ func verifyCgroup(proc int, req ContainerRequest) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func observePath(root int, mount string) (string, *filesystem.Usage, error) {
|
||||
// Absolute symlinks stay within the resource root. Magic links cannot escape
|
||||
// it. Unsupported kernels fail rather than falling back to unsafe resolution.
|
||||
// openInRoot opens mount beneath root without ever leaving it. With openat2,
|
||||
// absolute symlinks stay within the resource root and magic links cannot
|
||||
// escape it. systemd answers openat2 with ENOSYS under RestrictSUIDSGID,
|
||||
// because seccomp cannot inspect its flags, and the typed helper sets that, so
|
||||
// there the clean path is walked one component at a time and any symbolic link
|
||||
// is refused rather than resolved. Neither path falls back to unconfined
|
||||
// resolution.
|
||||
func openInRoot(root int, mount string) (int, error) {
|
||||
fd, err := unix.Openat2(root, mount, &unix.OpenHow{
|
||||
Flags: unix.O_PATH | unix.O_CLOEXEC,
|
||||
Resolve: unix.RESOLVE_IN_ROOT | unix.RESOLVE_NO_MAGICLINKS,
|
||||
})
|
||||
if !errors.Is(err, unix.ENOSYS) {
|
||||
return fd, err
|
||||
}
|
||||
return openBeneathWithoutSymlinks(root, mount)
|
||||
}
|
||||
|
||||
func openBeneathWithoutSymlinks(root int, mount string) (int, error) {
|
||||
if !path.IsAbs(mount) || path.Clean(mount) != mount {
|
||||
return -1, errors.New("mount path must be clean and absolute")
|
||||
}
|
||||
current, err := unix.Dup(root)
|
||||
if err != nil {
|
||||
return -1, err
|
||||
}
|
||||
for _, component := range strings.Split(strings.TrimPrefix(mount, "/"), "/") {
|
||||
if component == "" {
|
||||
continue
|
||||
}
|
||||
next, err := unix.Openat(current, component, unix.O_PATH|unix.O_NOFOLLOW|unix.O_CLOEXEC, 0)
|
||||
unix.Close(current)
|
||||
if err != nil {
|
||||
return -1, err
|
||||
}
|
||||
var stat unix.Stat_t
|
||||
if err := unix.Fstat(next, &stat); err != nil {
|
||||
unix.Close(next)
|
||||
return -1, err
|
||||
}
|
||||
if stat.Mode&unix.S_IFMT == unix.S_IFLNK {
|
||||
unix.Close(next)
|
||||
return -1, errors.New("mount path crosses a symbolic link")
|
||||
}
|
||||
current = next
|
||||
}
|
||||
return current, nil
|
||||
}
|
||||
|
||||
func observePath(root int, mount string) (string, *filesystem.Usage, error) {
|
||||
fd, err := openInRoot(root, mount)
|
||||
if err != nil {
|
||||
return "", nil, fmt.Errorf("open mount within resource namespace: %w", err)
|
||||
}
|
||||
|
||||
@@ -81,3 +81,68 @@ func TestWrongProcessCannotSupplyContainerCapacity(t *testing.T) {
|
||||
t.Fatalf("unrelated process supplied capacity: %+v %v", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
// A configured mountpoint that shares its parent's device is not mounted in
|
||||
// the guest; reporting it would copy the parent filesystem's counters.
|
||||
func TestMountsOnlyDetectsUnmountedPathsWithinRoot(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
if err := os.Mkdir(filepath.Join(dir, "data"), 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
fd, err := unix.Open(dir, unix.O_PATH|unix.O_DIRECTORY|unix.O_CLOEXEC, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer unix.Close(fd)
|
||||
if mounted, err := mountedWithinRoot(fd, "/"); err != nil || !mounted {
|
||||
t.Fatalf("root reported unmounted: %v %v", mounted, err)
|
||||
}
|
||||
if mounted, err := mountedWithinRoot(fd, "/data"); err != nil || mounted {
|
||||
t.Fatalf("plain directory reported as a mount: %v %v", mounted, err)
|
||||
}
|
||||
if _, err := mountedWithinRoot(fd, "/missing"); err == nil {
|
||||
t.Fatal("missing path reported a mount state")
|
||||
}
|
||||
}
|
||||
|
||||
// The test process is not PID 1 of a namespace, so it cannot supply a Proxmox
|
||||
// guest's filesystems even if a request names it.
|
||||
func TestNonInitProcessCannotSupplyProxmoxGuestCapacity(t *testing.T) {
|
||||
r := ContainerRequest{PID: os.Getpid(), ContainerID: "126", Runtime: "lxc", Mountpoints: []string{"/"}, MountsOnly: true}
|
||||
if got, err := observeContainer(context.Background(), r); err == nil || got != nil {
|
||||
t.Fatalf("non-init process supplied capacity: %+v %v", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
// Inside the typed helper systemd blocks openat2 (RestrictSUIDSGID), so mounts
|
||||
// are resolved component by component. That walk must stay beneath the root
|
||||
// by refusing every symbolic link, including ones that would stay inside.
|
||||
func TestComponentWalkRefusesSymlinksAndStaysBeneathRoot(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
if err := os.MkdirAll(filepath.Join(dir, "srv", "data"), 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for name, target := range map[string]string{"inside-link": "/srv", "outside": "/etc", "relative-up": "../../.."} {
|
||||
if err := os.Symlink(target, filepath.Join(dir, name)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
root, err := unix.Open(dir, unix.O_PATH|unix.O_DIRECTORY|unix.O_CLOEXEC, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer unix.Close(root)
|
||||
for _, mount := range []string{"/", "/srv", "/srv/data"} {
|
||||
fd, err := openBeneathWithoutSymlinks(root, mount)
|
||||
if err != nil {
|
||||
t.Fatalf("plain path %s refused: %v", mount, err)
|
||||
}
|
||||
unix.Close(fd)
|
||||
}
|
||||
for _, mount := range []string{"/inside-link", "/inside-link/data", "/outside", "/outside/passwd", "/relative-up", "/srv/../etc", "srv", "/proc/self/root"} {
|
||||
if fd, err := openBeneathWithoutSymlinks(root, mount); err == nil {
|
||||
unix.Close(fd)
|
||||
t.Fatalf("%s resolved through the component walk", mount)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -79,3 +79,57 @@ func TestObserverRejectsInvalidCoordinatesBeforeNativeRead(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// A Proxmox guest is identified by its exact lxc/<vmid> cgroup and its init
|
||||
// process. Neither another VMID with the same prefix, LXC's monitor cgroup,
|
||||
// nor the PID 1 of a namespace nested inside the guest (Docker in LXC) may
|
||||
// stand in for it.
|
||||
func TestProxmoxLXCIdentityIsExact(t *testing.T) {
|
||||
for raw, want := range map[string]bool{
|
||||
"0::/lxc/126/ns": true,
|
||||
"0::/lxc/126/ns/init.scope": true,
|
||||
"0::/lxc/126": true,
|
||||
"0::/lxc/1260/ns": false,
|
||||
"0::/lxc/12/ns": false,
|
||||
"0::/lxc.monitor/126": false,
|
||||
"0::/system.slice/lxc/126/ns": false,
|
||||
"0::/docker/" + strings.Repeat("a", 64): false,
|
||||
} {
|
||||
if got := CgroupMatches(raw, "lxc", "126"); got != want {
|
||||
t.Errorf("CgroupMatches(%q) = %v, want %v", raw, got, want)
|
||||
}
|
||||
}
|
||||
for status, want := range map[string]bool{
|
||||
"Name:\tinit\nNSpid:\t3019253\t1\n": true,
|
||||
"NSpid:\t3019253\t413\n": false,
|
||||
"NSpid:\t3019253\t420\t1\n": false, // nested namespace init
|
||||
"NSpid:\t1\n": false, // the host's own init
|
||||
"NSpid:\t999\t1\n": false, // another process's line
|
||||
"Name:\tinit\n": false,
|
||||
} {
|
||||
if got := IsNamespaceInit(status, 3019253); got != want {
|
||||
t.Errorf("IsNamespaceInit(%q) = %v, want %v", status, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestObserverRejectsInexactProxmoxVMIDs(t *testing.T) {
|
||||
o := &Observer{probe: func(context.Context, ContainerRequest) ([]filesystem.Observation, error) {
|
||||
t.Error("invalid coordinates reached native read")
|
||||
return nil, nil
|
||||
}}
|
||||
for _, id := range []string{"99", "0126", "126a", "-126", "1000000000", ""} {
|
||||
r := ContainerRequest{PID: 42, ContainerID: id, Runtime: "lxc", Mountpoints: []string{"/"}}
|
||||
if _, err := o.Observe(context.Background(), r); err == nil {
|
||||
t.Fatalf("accepted VMID %q", id)
|
||||
}
|
||||
}
|
||||
accepted := false
|
||||
o.probe = func(context.Context, ContainerRequest) ([]filesystem.Observation, error) {
|
||||
accepted = true
|
||||
return nil, nil
|
||||
}
|
||||
if _, err := o.Observe(context.Background(), ContainerRequest{PID: 42, ContainerID: "126", Runtime: "lxc", Mountpoints: []string{"/"}}); err != nil || !accepted {
|
||||
t.Fatalf("exact VMID rejected: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -7,8 +7,10 @@ import (
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/filesystemprobe"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/hostmetrics"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/sensors"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/agents/filesystem"
|
||||
agentshost "github.com/rcourtman/pulse-go-rewrite/pkg/agents/host"
|
||||
gohost "github.com/shirou/gopsutil/v4/host"
|
||||
)
|
||||
@@ -44,15 +46,27 @@ type mockCollector struct {
|
||||
name string,
|
||||
arg ...string,
|
||||
) (string, error)
|
||||
lookPathFn func(file string) (string, error)
|
||||
filesystemUsageFn func(path string) (hostFilesystemUsage, error)
|
||||
lookPathFn func(file string) (string, error)
|
||||
observeLXCFn func(ctx context.Context, req filesystemprobe.ContainerRequest) ([]filesystem.Observation, error)
|
||||
readDirFn func(name string) ([]os.DirEntry, error)
|
||||
}
|
||||
|
||||
func (m *mockCollector) FilesystemUsage(path string) (hostFilesystemUsage, error) {
|
||||
if m.filesystemUsageFn != nil {
|
||||
return m.filesystemUsageFn(path)
|
||||
// ReadDir satisfies proxmoxLXCDirLister. Without readDirFn it reports the
|
||||
// directory missing, so collectors keep their pct path unless a test opts in.
|
||||
func (m *mockCollector) ReadDir(name string) ([]os.DirEntry, error) {
|
||||
if m.readDirFn != nil {
|
||||
return m.readDirFn(name)
|
||||
}
|
||||
return hostFilesystemUsage{}, os.ErrNotExist
|
||||
return nil, os.ErrNotExist
|
||||
}
|
||||
|
||||
// ObserveProxmoxLXC satisfies proxmoxLXCObserver. Without observeLXCFn the
|
||||
// probe fails, as it does off Linux, so collectors fall back to pct df.
|
||||
func (m *mockCollector) ObserveProxmoxLXC(ctx context.Context, req filesystemprobe.ContainerRequest) ([]filesystem.Observation, error) {
|
||||
if m.observeLXCFn != nil {
|
||||
return m.observeLXCFn(ctx, req)
|
||||
}
|
||||
return nil, os.ErrNotExist
|
||||
}
|
||||
|
||||
func (m *mockCollector) HostInfo(ctx context.Context) (*gohost.InfoStat, error) {
|
||||
|
||||
@@ -412,3 +412,44 @@ func requirePrivilegeHelperModuleStatus(t *testing.T, statuses []agentshost.Modu
|
||||
t.Fatalf("typed privilege helper module missing from %+v", statuses)
|
||||
return agentshost.ModuleStatus{}
|
||||
}
|
||||
|
||||
// The typed helper runs with PrivateNetwork=true, where pct and lxc-info cannot
|
||||
// reach pmxcfs or the LXC monitor: their abstract Unix sockets belong to the
|
||||
// host network namespace (#2511). This drives the collector the helper's
|
||||
// provider runs, with every command failing as it does there: it must still
|
||||
// return a complete inventory from configs, cgroups and /proc alone. A
|
||||
// degraded result is what the provider turns into provider_unavailable.
|
||||
func TestHelperProxmoxLXCFilesystemsNeedNoPctOrLXCSockets(t *testing.T) {
|
||||
h := newFakeProxmoxLXCHost()
|
||||
h.config(126, "hostname: qual2511\nrootfs: delly2-lvm:vm-126-disk-0,size=1G\nmp0: delly2-lvm:vm-126-disk-1,mp=/srv/data,size=1G\n")
|
||||
h.cgroup("126/ns", 3019253)
|
||||
h.proc(3019253, "3019253\t1", "/lxc/126/ns")
|
||||
h.config(127, "hostname: qual2511-priv\nrootfs: delly2-lvm:vm-127-disk-0,size=1G\n")
|
||||
h.cgroup("127/ns", 3021000)
|
||||
h.proc(3021000, "3021000\t1", "/lxc/127/ns")
|
||||
h.config(128, "hostname: stopped\nrootfs: delly2-lvm:vm-128-disk-0,size=1G\n")
|
||||
// Counters observed on a real PVE 9.2 node for the same layout.
|
||||
h.observations[3019253] = map[string]lxcObservation{
|
||||
"/": lxcUsage(1020702720, 12021760, 938217472),
|
||||
"/srv/data": lxcUsage(1020702720, 290816, 949948416),
|
||||
}
|
||||
h.observations[3021000] = map[string]lxcObservation{"/": lxcUsage(1020702720, 12017664, 938221568)}
|
||||
|
||||
collector, commands := h.collector(t)
|
||||
agent := &Agent{logger: zerolog.Nop(), collector: collector}
|
||||
result := agent.collectProxmoxLXCFilesystemsResult(context.Background())
|
||||
|
||||
if !result.Applicable || result.Degraded || result.FailedContainers != 0 {
|
||||
t.Fatalf("collection = applicable %v, degraded %v, failed %d; want a complete inventory", result.Applicable, result.Degraded, result.FailedContainers)
|
||||
}
|
||||
if *commands != 0 {
|
||||
t.Fatalf("collection ran %d pct/lxc-info commands; none can work inside the helper", *commands)
|
||||
}
|
||||
got := result.Inventory.Containers
|
||||
if len(got) != 2 || got[0].VMID != 126 || got[0].Name != "qual2511" || got[1].VMID != 127 || got[1].Name != "qual2511-priv" {
|
||||
t.Fatalf("containers = %+v, want the two running ones by their pct names", got)
|
||||
}
|
||||
if len(got[0].Disks) != 2 || got[0].Disks[1].Mountpoint != "/srv/data" || got[0].Disks[1].UsedBytes != 290816 {
|
||||
t.Fatalf("126 disks = %+v, want rootfs and the mp0 mount with its own usage", got[0].Disks)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -15,6 +15,7 @@ import (
|
||||
"time"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/filesystemprobe"
|
||||
agentshost "github.com/rcourtman/pulse-go-rewrite/pkg/agents/host"
|
||||
"github.com/rs/zerolog"
|
||||
)
|
||||
@@ -39,12 +40,15 @@ const (
|
||||
proxmoxLXCCollectionTimeout = 20 * time.Second
|
||||
proxmoxLXCMaxContainers = 128
|
||||
proxmoxLXCMaxDisks = 257
|
||||
proxmoxLXCMaxListOutputBytes = 64 * 1024
|
||||
proxmoxLXCMaxDFOutputBytes = 64 * 1024
|
||||
proxmoxLXCMaxConfigBytes = 64 * 1024
|
||||
proxmoxLXCMaxPIDOutputBytes = 4 * 1024
|
||||
proxmoxLXCMaxNameBytes = 256
|
||||
proxmoxLXCMaxLabelBytes = 512
|
||||
// proxmoxLXCMaxObservedMounts is the shared filesystem prober's per-request
|
||||
// mountpoint limit.
|
||||
proxmoxLXCMaxObservedMounts = 128
|
||||
proxmoxLXCMaxListOutputBytes = 64 * 1024
|
||||
proxmoxLXCMaxDFOutputBytes = 64 * 1024
|
||||
proxmoxLXCMaxConfigBytes = 64 * 1024
|
||||
proxmoxLXCMaxPIDOutputBytes = 4 * 1024
|
||||
proxmoxLXCMaxNameBytes = 256
|
||||
proxmoxLXCMaxLabelBytes = 512
|
||||
)
|
||||
|
||||
// proxmoxLXCConfigDir is a pmxcfs symlink to the local node's container
|
||||
@@ -54,6 +58,9 @@ const proxmoxLXCConfigDir = "/etc/pve/lxc"
|
||||
type proxmoxLXCRunningContainer struct {
|
||||
VMID int
|
||||
Name string
|
||||
// PID is the container's init process when discovery found it without
|
||||
// lxc-info; 0 means unknown.
|
||||
PID int
|
||||
}
|
||||
|
||||
// ProxmoxLXCFilesystemCollectionResult describes applicability separately from
|
||||
@@ -68,21 +75,6 @@ type ProxmoxLXCFilesystemCollectionResult struct {
|
||||
FailedContainers int
|
||||
}
|
||||
|
||||
// hostFilesystemUsage carries statfs-derived usage for one path plus the
|
||||
// st_dev identity needed to tell a real mount from its parent filesystem.
|
||||
type hostFilesystemUsage struct {
|
||||
TotalBytes int64
|
||||
UsedBytes int64
|
||||
AvailBytes int64
|
||||
Device uint64
|
||||
}
|
||||
|
||||
// filesystemUsageProber is an optional collector capability. It exists on
|
||||
// Linux builds of the default collector; mocks opt in for tests.
|
||||
type filesystemUsageProber interface {
|
||||
FilesystemUsage(path string) (hostFilesystemUsage, error)
|
||||
}
|
||||
|
||||
type proxmoxLXCConfigMount struct {
|
||||
Key string
|
||||
Volume string
|
||||
@@ -142,30 +134,38 @@ func (a *Agent) collectProxmoxLXCFilesystemsResult(ctx context.Context) ProxmoxL
|
||||
}
|
||||
result.Applicable = true
|
||||
|
||||
observer, _ := a.collector.(proxmoxLXCObserver)
|
||||
listCtx, cancelList := context.WithTimeout(ctx, proxmoxLXCQueryTimeout)
|
||||
listOutput, err := collectCommandOutputLimited(
|
||||
listCtx,
|
||||
a.collector,
|
||||
proxmoxLXCMaxListOutputBytes,
|
||||
pctPath,
|
||||
"list",
|
||||
)
|
||||
containers, discovered := []proxmoxLXCRunningContainer(nil), false
|
||||
if observer != nil {
|
||||
containers, discovered = a.discoverRunningProxmoxLXCContainers(listCtx)
|
||||
}
|
||||
if !discovered {
|
||||
listOutput, listErr := collectCommandOutputLimited(
|
||||
listCtx,
|
||||
a.collector,
|
||||
proxmoxLXCMaxListOutputBytes,
|
||||
pctPath,
|
||||
"list",
|
||||
)
|
||||
if listErr != nil {
|
||||
cancelList()
|
||||
a.logger.Debug().Err(listErr).Msg("Failed to list local Proxmox LXC containers")
|
||||
result.Degraded = true
|
||||
return result
|
||||
}
|
||||
containers, err = parseProxmoxLXCRunningContainers(listOutput)
|
||||
if err != nil {
|
||||
cancelList()
|
||||
a.logger.Debug().Err(err).Msg("Failed to parse local Proxmox LXC container list")
|
||||
result.Degraded = true
|
||||
return result
|
||||
}
|
||||
}
|
||||
cancelList()
|
||||
if err != nil {
|
||||
a.logger.Debug().Err(err).Msg("Failed to list local Proxmox LXC containers")
|
||||
result.Degraded = true
|
||||
return result
|
||||
}
|
||||
containers, err := parseProxmoxLXCRunningContainers(listOutput)
|
||||
if err != nil {
|
||||
a.logger.Debug().Err(err).Msg("Failed to parse local Proxmox LXC container list")
|
||||
result.Degraded = true
|
||||
return result
|
||||
}
|
||||
|
||||
prober, _ := a.collector.(filesystemUsageProber)
|
||||
lxcInfoPath := ""
|
||||
if prober != nil {
|
||||
if observer != nil && !discovered {
|
||||
if resolved, lookErr := a.collector.LookPath("lxc-info"); lookErr == nil {
|
||||
lxcInfoPath = resolved
|
||||
}
|
||||
@@ -184,7 +184,7 @@ func (a *Agent) collectProxmoxLXCFilesystemsResult(ctx context.Context) ProxmoxL
|
||||
continue
|
||||
}
|
||||
containerCtx, cancelContainer := context.WithTimeout(collectionCtx, proxmoxLXCContainerQueryTimeout)
|
||||
disks, collectionErr := a.collectProxmoxLXCContainerDisks(containerCtx, prober, lxcInfoPath, pctPath, container.VMID)
|
||||
disks, collectionErr := a.collectProxmoxLXCContainerDisks(containerCtx, observer, lxcInfoPath, pctPath, container)
|
||||
cancelContainer()
|
||||
if collectionErr != nil {
|
||||
result.FailedContainers++
|
||||
@@ -209,13 +209,14 @@ func (a *Agent) collectProxmoxLXCFilesystemsResult(ctx context.Context) ProxmoxL
|
||||
|
||||
func (a *Agent) collectProxmoxLXCContainerDisks(
|
||||
ctx context.Context,
|
||||
prober filesystemUsageProber,
|
||||
observer proxmoxLXCObserver,
|
||||
lxcInfoPath string,
|
||||
pctPath string,
|
||||
vmid int,
|
||||
container proxmoxLXCRunningContainer,
|
||||
) ([]agentshost.Disk, error) {
|
||||
if prober != nil && lxcInfoPath != "" {
|
||||
disks, fastErr := a.collectProxmoxLXCContainerDisksFast(ctx, prober, lxcInfoPath, vmid)
|
||||
vmid := container.VMID
|
||||
if observer != nil && (container.PID > 0 || lxcInfoPath != "") {
|
||||
disks, fastErr := a.collectProxmoxLXCContainerDisksFast(ctx, observer, lxcInfoPath, container)
|
||||
if fastErr == nil {
|
||||
return disks, nil
|
||||
}
|
||||
@@ -259,10 +260,11 @@ func (a *Agent) collectProxmoxLXCContainerDisks(
|
||||
// numbers — so it is dropped rather than invented.
|
||||
func (a *Agent) collectProxmoxLXCContainerDisksFast(
|
||||
ctx context.Context,
|
||||
prober filesystemUsageProber,
|
||||
observer proxmoxLXCObserver,
|
||||
lxcInfoPath string,
|
||||
vmid int,
|
||||
container proxmoxLXCRunningContainer,
|
||||
) ([]agentshost.Disk, error) {
|
||||
vmid := container.VMID
|
||||
raw, err := a.collector.ReadFile(proxmoxLXCConfigPath(vmid))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read container config: %w", err)
|
||||
@@ -275,59 +277,76 @@ func (a *Agent) collectProxmoxLXCContainerDisksFast(
|
||||
return nil, errors.New("no filesystem mounts in container config")
|
||||
}
|
||||
|
||||
pidOutput, err := collectCommandOutputLimited(
|
||||
ctx,
|
||||
a.collector,
|
||||
proxmoxLXCMaxPIDOutputBytes,
|
||||
lxcInfoPath,
|
||||
"-n",
|
||||
strconv.Itoa(vmid),
|
||||
"-p",
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("resolve container pid: %w", err)
|
||||
}
|
||||
pid, err := parseLXCInfoPID(pidOutput)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
pid := container.PID
|
||||
if pid <= 0 {
|
||||
pidOutput, pidErr := collectCommandOutputLimited(
|
||||
ctx,
|
||||
a.collector,
|
||||
proxmoxLXCMaxPIDOutputBytes,
|
||||
lxcInfoPath,
|
||||
"-n",
|
||||
strconv.Itoa(vmid),
|
||||
"-p",
|
||||
)
|
||||
if pidErr != nil {
|
||||
return nil, fmt.Errorf("resolve container pid: %w", pidErr)
|
||||
}
|
||||
pid, err = parseLXCInfoPID(pidOutput)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
procRoot := fmt.Sprintf("/proc/%d/root", pid)
|
||||
result := make([]agentshost.Disk, 0, len(mounts))
|
||||
// The shared prober pins /proc/<pid>, requires it to be this guest's init
|
||||
// in lxc/<vmid>, resolves every mount inside the guest's root (an absolute
|
||||
// symlink cannot reach the host), and bounds a read stuck on a hung guest
|
||||
// filesystem without starting another for the same guest.
|
||||
byPath := make(map[string]proxmoxLXCConfigMount, len(mounts))
|
||||
mountpoints := make([]string, 0, len(mounts))
|
||||
for _, mount := range mounts {
|
||||
if ctx.Err() != nil {
|
||||
return nil, ctx.Err()
|
||||
if _, seen := byPath[mount.Path]; seen || len(mountpoints) == proxmoxLXCMaxObservedMounts {
|
||||
continue
|
||||
}
|
||||
target := path.Join(procRoot, mount.Path)
|
||||
usage, usageErr := prober.FilesystemUsage(target)
|
||||
if usageErr != nil {
|
||||
return nil, fmt.Errorf("probe %s filesystem usage: %w", mount.Key, usageErr)
|
||||
byPath[mount.Path] = mount
|
||||
mountpoints = append(mountpoints, mount.Path)
|
||||
}
|
||||
observations, err := observer.ObserveProxmoxLXC(ctx, filesystemprobe.ContainerRequest{
|
||||
PID: pid,
|
||||
ContainerID: strconv.Itoa(vmid),
|
||||
Runtime: "lxc",
|
||||
Mountpoints: mountpoints,
|
||||
MountsOnly: true,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("observe container filesystems: %w", err)
|
||||
}
|
||||
|
||||
result := make([]agentshost.Disk, 0, len(observations))
|
||||
for _, observation := range observations {
|
||||
mount, known := byPath[observation.Mountpoint]
|
||||
if !known {
|
||||
return nil, fmt.Errorf("observation for unrequested mount %q", observation.Mountpoint)
|
||||
}
|
||||
if usage.TotalBytes <= 0 {
|
||||
if filesystemprobe.IsNotMounted(observation) {
|
||||
// Declared in the config but not mounted in the running guest; its
|
||||
// path would report the parent filesystem's numbers.
|
||||
continue
|
||||
}
|
||||
if observation.Error != "" || observation.Usage == nil {
|
||||
return nil, fmt.Errorf("probe %s filesystem usage: %s", mount.Key, observation.Error)
|
||||
}
|
||||
usage := observation.Usage
|
||||
if usage.CapacityBytes == 0 || usage.CapacityBytes > math.MaxInt64 {
|
||||
return nil, fmt.Errorf("probe %s filesystem usage: invalid total bytes", mount.Key)
|
||||
}
|
||||
if mount.Path != "/" {
|
||||
parent, parentErr := prober.FilesystemUsage(path.Dir(target))
|
||||
if parentErr != nil {
|
||||
return nil, fmt.Errorf("probe %s parent filesystem usage: %w", mount.Key, parentErr)
|
||||
}
|
||||
if parent.Device == usage.Device {
|
||||
continue
|
||||
}
|
||||
}
|
||||
used := usage.UsedBytes
|
||||
total := int64(usage.CapacityBytes)
|
||||
used := total - int64(usage.FreeBytes)
|
||||
if used < 0 {
|
||||
used = 0
|
||||
}
|
||||
if used > usage.TotalBytes {
|
||||
used = usage.TotalBytes
|
||||
}
|
||||
avail := usage.AvailBytes
|
||||
if avail < 0 {
|
||||
avail = 0
|
||||
}
|
||||
if avail > usage.TotalBytes {
|
||||
avail = usage.TotalBytes
|
||||
avail := int64(usage.AvailableBytes)
|
||||
if avail > total {
|
||||
avail = total
|
||||
}
|
||||
usagePct := 0.0
|
||||
if denominator := used + avail; denominator > 0 {
|
||||
@@ -340,14 +359,11 @@ func (a *Agent) collectProxmoxLXCContainerDisksFast(
|
||||
Device: mount.Volume,
|
||||
Mountpoint: mount.Path,
|
||||
Type: mount.Key,
|
||||
TotalBytes: usage.TotalBytes,
|
||||
TotalBytes: total,
|
||||
UsedBytes: used,
|
||||
FreeBytes: avail,
|
||||
Usage: usagePct,
|
||||
})
|
||||
if len(result) == proxmoxLXCMaxDisks {
|
||||
break
|
||||
}
|
||||
}
|
||||
if len(result) == 0 {
|
||||
return nil, errors.New("no mounted container filesystems resolved")
|
||||
|
||||
@@ -1,41 +0,0 @@
|
||||
//go:build linux
|
||||
|
||||
package hostagent
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"syscall"
|
||||
)
|
||||
|
||||
// FilesystemUsage satisfies filesystemUsageProber on Linux. Statfs through a
|
||||
// /proc/<pid>/root path traverses into that process's mount namespace, so the
|
||||
// numbers match what df reports inside the container.
|
||||
func (c *defaultCollector) FilesystemUsage(path string) (hostFilesystemUsage, error) {
|
||||
var fs syscall.Statfs_t
|
||||
if err := syscall.Statfs(path, &fs); err != nil {
|
||||
return hostFilesystemUsage{}, fmt.Errorf("statfs %s: %w", path, err)
|
||||
}
|
||||
var st syscall.Stat_t
|
||||
if err := syscall.Stat(path, &st); err != nil {
|
||||
return hostFilesystemUsage{}, fmt.Errorf("stat %s: %w", path, err)
|
||||
}
|
||||
blockSize := int64(fs.Frsize)
|
||||
if blockSize <= 0 {
|
||||
blockSize = int64(fs.Bsize)
|
||||
}
|
||||
if blockSize <= 0 {
|
||||
return hostFilesystemUsage{}, fmt.Errorf("statfs %s: invalid block size", path)
|
||||
}
|
||||
total := int64(fs.Blocks) * blockSize
|
||||
free := int64(fs.Bfree) * blockSize
|
||||
used := total - free
|
||||
if used < 0 {
|
||||
used = 0
|
||||
}
|
||||
return hostFilesystemUsage{
|
||||
TotalBytes: total,
|
||||
UsedBytes: used,
|
||||
AvailBytes: int64(fs.Bavail) * blockSize,
|
||||
Device: uint64(st.Dev),
|
||||
}, nil
|
||||
}
|
||||
@@ -4,11 +4,16 @@ import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/filesystemprobe"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/agents/filesystem"
|
||||
"github.com/rs/zerolog"
|
||||
)
|
||||
|
||||
@@ -153,16 +158,15 @@ mp2: tank:snap-only,mp=/snap-only,size=5G
|
||||
`,
|
||||
"/etc/pve/lxc/101.conf": "rootfs: local-lvm:vm-101-disk-0,size=4G\n",
|
||||
}
|
||||
usageByPath := map[string]hostFilesystemUsage{
|
||||
"/proc/4242/root": {TotalBytes: 8 << 30, UsedBytes: 2 << 30, AvailBytes: 6 << 30, Device: 11},
|
||||
"/proc/4242/root/srv": {TotalBytes: 8 << 30, UsedBytes: 2 << 30, AvailBytes: 6 << 30, Device: 11},
|
||||
"/proc/4242/root/srv/data": {TotalBytes: 1 << 40, UsedBytes: 512 << 30, AvailBytes: 512 << 30, Device: 12},
|
||||
// mp1 is configured but not mounted in the running container, so the
|
||||
// path resolves to the rootfs device and must be dropped, not reported
|
||||
// with the parent's numbers.
|
||||
"/proc/4242/root/mnt": {TotalBytes: 8 << 30, UsedBytes: 2 << 30, AvailBytes: 6 << 30, Device: 11},
|
||||
"/proc/4242/root/mnt/missing": {TotalBytes: 8 << 30, UsedBytes: 2 << 30, AvailBytes: 6 << 30, Device: 11},
|
||||
"/proc/5252/root": {TotalBytes: 4 << 30, UsedBytes: 1 << 30, AvailBytes: 3 << 30, Device: 21},
|
||||
observations := map[int]map[string]lxcObservation{
|
||||
4242: {
|
||||
"/": lxcUsage(8<<30, 2<<30, 6<<30),
|
||||
"/srv/data": lxcUsage(1<<40, 512<<30, 512<<30),
|
||||
// mp1 is configured but not mounted in the running container, so
|
||||
// the prober rejects it instead of reporting the parent's numbers.
|
||||
"/mnt/missing": {notMounted: true},
|
||||
},
|
||||
5252: {"/": lxcUsage(4<<30, 1<<30, 3<<30)},
|
||||
}
|
||||
collector := &mockCollector{
|
||||
goos: "linux",
|
||||
@@ -182,13 +186,7 @@ mp2: tank:snap-only,mp=/snap-only,size=5G
|
||||
}
|
||||
return []byte(config), nil
|
||||
},
|
||||
filesystemUsageFn: func(path string) (hostFilesystemUsage, error) {
|
||||
usage, ok := usageByPath[path]
|
||||
if !ok {
|
||||
t.Fatalf("unexpected statfs path %q", path)
|
||||
}
|
||||
return usage, nil
|
||||
},
|
||||
observeLXCFn: fakeLXCObserve(t, observations),
|
||||
commandCombinedOutputLimitedFn: func(
|
||||
_ context.Context,
|
||||
_ int,
|
||||
@@ -267,12 +265,9 @@ func TestCollectProxmoxLXCFilesystemsFallsBackToPctDFPerContainer(t *testing.T)
|
||||
// back to pct df.
|
||||
return nil, os.ErrPermission
|
||||
},
|
||||
filesystemUsageFn: func(path string) (hostFilesystemUsage, error) {
|
||||
if path != "/proc/4242/root" {
|
||||
t.Fatalf("unexpected statfs path %q", path)
|
||||
}
|
||||
return hostFilesystemUsage{TotalBytes: 8 << 30, UsedBytes: 2 << 30, AvailBytes: 6 << 30, Device: 11}, nil
|
||||
},
|
||||
observeLXCFn: fakeLXCObserve(t, map[int]map[string]lxcObservation{
|
||||
4242: {"/": lxcUsage(8<<30, 2<<30, 6<<30)},
|
||||
}),
|
||||
commandCombinedOutputLimitedFn: func(
|
||||
_ context.Context,
|
||||
_ int,
|
||||
@@ -345,17 +340,12 @@ mp0 tank:subvol-100-disk-0 1.0T 512.0G 512.0G 50.0 /srv/data
|
||||
readFileFn: func(string) ([]byte, error) {
|
||||
return []byte("rootfs: local-lvm:vm-100-disk-0,size=8G\nmp0: tank:subvol-100-disk-0,mp=/srv/data,size=1T\n"), nil
|
||||
},
|
||||
filesystemUsageFn: func(target string) (hostFilesystemUsage, error) {
|
||||
switch target {
|
||||
case "/proc/4242/root":
|
||||
return hostFilesystemUsage{TotalBytes: 8 << 30, UsedBytes: 2 << 30, AvailBytes: 6 << 30, Device: 11}, nil
|
||||
case "/proc/4242/root/srv/data":
|
||||
return hostFilesystemUsage{}, errors.New("statfs access failed")
|
||||
default:
|
||||
t.Fatalf("unexpected statfs path %q", target)
|
||||
return hostFilesystemUsage{}, nil
|
||||
}
|
||||
},
|
||||
observeLXCFn: fakeLXCObserve(t, map[int]map[string]lxcObservation{
|
||||
4242: {
|
||||
"/": lxcUsage(8<<30, 2<<30, 6<<30),
|
||||
"/srv/data": {err: "read filesystem counters: permission denied"},
|
||||
},
|
||||
}),
|
||||
commandCombinedOutputLimitedFn: func(_ context.Context, _ int, name string, args ...string) (string, error) {
|
||||
joined := strings.Join(args, " ")
|
||||
switch {
|
||||
@@ -469,3 +459,281 @@ mp5 pool:hot 10.0G 1.0G 9.0G 101.0 /hot
|
||||
t.Fatal("expected oversized pct df error")
|
||||
}
|
||||
}
|
||||
|
||||
type fakeLXCDirEntry struct {
|
||||
name string
|
||||
dir bool
|
||||
}
|
||||
|
||||
func (e fakeLXCDirEntry) Name() string { return e.name }
|
||||
func (e fakeLXCDirEntry) IsDir() bool { return e.dir }
|
||||
func (e fakeLXCDirEntry) Type() fs.FileMode {
|
||||
if e.dir {
|
||||
return fs.ModeDir
|
||||
}
|
||||
return 0
|
||||
}
|
||||
func (e fakeLXCDirEntry) Info() (fs.FileInfo, error) { return nil, fs.ErrNotExist }
|
||||
|
||||
// fakeProxmoxLXCHost models what socket-free discovery reads on a cgroup v2
|
||||
// Proxmox node. The layout follows a real PVE 9.2 node: configs under
|
||||
// /etc/pve/lxc, the payload under /sys/fs/cgroup/lxc/<vmid>/ns, and the
|
||||
// container init identified by an NSpid ending in 1.
|
||||
type fakeProxmoxLXCHost struct {
|
||||
files map[string]string
|
||||
dirs map[string]map[string]bool
|
||||
observations map[int]map[string]lxcObservation
|
||||
}
|
||||
|
||||
func newFakeProxmoxLXCHost() *fakeProxmoxLXCHost {
|
||||
h := &fakeProxmoxLXCHost{
|
||||
files: map[string]string{proxmoxLXCCgroupV2Marker: "cpuset cpu io memory pids\n"},
|
||||
dirs: map[string]map[string]bool{},
|
||||
observations: map[int]map[string]lxcObservation{},
|
||||
}
|
||||
h.dir(proxmoxLXCConfigDir)
|
||||
return h
|
||||
}
|
||||
|
||||
func (h *fakeProxmoxLXCHost) dir(name string) {
|
||||
if _, ok := h.dirs[name]; !ok {
|
||||
h.dirs[name] = map[string]bool{}
|
||||
}
|
||||
}
|
||||
|
||||
func (h *fakeProxmoxLXCHost) config(vmid int, content string) {
|
||||
h.dirs[proxmoxLXCConfigDir][strconv.Itoa(vmid)+".conf"] = false
|
||||
h.files[proxmoxLXCConfigPath(vmid)] = content
|
||||
}
|
||||
|
||||
// cgroup registers a cgroup directory (relative to lxc/) and its processes.
|
||||
func (h *fakeProxmoxLXCHost) cgroup(rel string, pids ...int) {
|
||||
full := path.Join(proxmoxLXCCgroupRoot, rel)
|
||||
for child := full; child != proxmoxLXCCgroupRoot; child = path.Dir(child) {
|
||||
h.dir(child)
|
||||
// Every cgroup directory has a cgroup.procs, empty when its
|
||||
// processes live in child cgroups.
|
||||
if _, ok := h.files[path.Join(child, "cgroup.procs")]; !ok {
|
||||
h.files[path.Join(child, "cgroup.procs")] = ""
|
||||
}
|
||||
parent := path.Dir(child)
|
||||
h.dir(parent)
|
||||
h.dirs[parent][path.Base(child)] = true
|
||||
}
|
||||
fields := make([]string, 0, len(pids))
|
||||
for _, pid := range pids {
|
||||
fields = append(fields, strconv.Itoa(pid))
|
||||
}
|
||||
h.files[path.Join(full, "cgroup.procs")] = strings.Join(fields, "\n") + "\n"
|
||||
}
|
||||
|
||||
func (h *fakeProxmoxLXCHost) proc(pid int, nspid, cgroup string) {
|
||||
h.files["/proc/"+strconv.Itoa(pid)+"/status"] = "Name:\tinit\nNSpid:\t" + nspid + "\n"
|
||||
h.files["/proc/"+strconv.Itoa(pid)+"/cgroup"] = "0::" + cgroup + "\n"
|
||||
}
|
||||
|
||||
// collector serves the fake host. pct and lxc-info exist but every call fails
|
||||
// the way they fail inside the PrivateNetwork helper.
|
||||
func (h *fakeProxmoxLXCHost) collector(t *testing.T) (*mockCollector, *int) {
|
||||
t.Helper()
|
||||
commands := 0
|
||||
return &mockCollector{
|
||||
goos: "linux",
|
||||
lookPathFn: func(file string) (string, error) {
|
||||
switch file {
|
||||
case "pct":
|
||||
return "/usr/sbin/pct", nil
|
||||
case "lxc-info":
|
||||
return "/usr/bin/lxc-info", nil
|
||||
}
|
||||
return "", os.ErrNotExist
|
||||
},
|
||||
statFn: func(name string) (os.FileInfo, error) {
|
||||
if _, ok := h.files[name]; ok {
|
||||
return nil, nil
|
||||
}
|
||||
if _, ok := h.dirs[name]; ok {
|
||||
return nil, nil
|
||||
}
|
||||
return nil, os.ErrNotExist
|
||||
},
|
||||
readFileFn: func(name string) ([]byte, error) {
|
||||
content, ok := h.files[name]
|
||||
if !ok {
|
||||
return nil, os.ErrNotExist
|
||||
}
|
||||
return []byte(content), nil
|
||||
},
|
||||
readDirFn: func(name string) ([]os.DirEntry, error) {
|
||||
children, ok := h.dirs[name]
|
||||
if !ok {
|
||||
return nil, os.ErrNotExist
|
||||
}
|
||||
entries := make([]os.DirEntry, 0, len(children))
|
||||
for child, isDir := range children {
|
||||
entries = append(entries, fakeLXCDirEntry{name: child, dir: isDir})
|
||||
}
|
||||
return entries, nil
|
||||
},
|
||||
observeLXCFn: fakeLXCObserve(t, h.observations),
|
||||
commandCombinedOutputLimitedFn: func(_ context.Context, _ int, name string, args ...string) (string, error) {
|
||||
commands++
|
||||
return "ipcc_send_rec[1] failed: Connection refused", errors.New("exit status 2")
|
||||
},
|
||||
}, &commands
|
||||
}
|
||||
|
||||
func TestProxmoxLXCConfigNameMatchesPctList(t *testing.T) {
|
||||
cases := []struct {
|
||||
config string
|
||||
want string
|
||||
}{
|
||||
{"#managed\narch: amd64\nhostname: web\nrootfs: x,size=1G\n", "web"},
|
||||
{"arch: amd64\nrootfs: x,size=1G\n", "CT126"},
|
||||
{"hostname: \n", "CT126"},
|
||||
{"arch: amd64\n\n[snap]\nhostname: old\n", "CT126"},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
if got := proxmoxLXCConfigName(tc.config, 126); got != tc.want {
|
||||
t.Errorf("proxmoxLXCConfigName(%q) = %q, want %q", tc.config, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiscoverRunningProxmoxLXCContainersFindsInitWithoutSockets(t *testing.T) {
|
||||
h := newFakeProxmoxLXCHost()
|
||||
// Alpine guest running Docker: a nested container's PID 1 shares the
|
||||
// guest's cgroup listing and comes first, but is not the guest's init.
|
||||
h.config(126, "hostname: qual2511\nrootfs: lvm:vm-126-disk-0,size=1G\n")
|
||||
h.cgroup("126")
|
||||
h.cgroup("126/ns", 3019900, 3019673, 3019253)
|
||||
h.cgroup("126/ns/openrc.syslog", 3019638)
|
||||
h.proc(3019900, "3019900\t420\t1", "/lxc/126/ns")
|
||||
h.proc(3019673, "3019673\t413", "/lxc/126/ns")
|
||||
h.proc(3019253, "3019253\t1", "/lxc/126/ns")
|
||||
// systemd guest: init in ns/init.scope, services elsewhere.
|
||||
h.config(127, "rootfs: lvm:vm-127-disk-0,size=1G\n")
|
||||
h.cgroup("127/ns/system.slice/cron.service", 4100)
|
||||
h.cgroup("127/ns/init.scope", 4000)
|
||||
h.proc(4100, "4100\t88", "/lxc/127/ns/system.slice/cron.service")
|
||||
h.proc(4000, "4000\t1", "/lxc/127/ns/init.scope")
|
||||
// Stopped: config only, no cgroup.
|
||||
h.config(128, "hostname: stopped\nrootfs: lvm:vm-128-disk-0,size=1G\n")
|
||||
|
||||
collector, commands := h.collector(t)
|
||||
agent := &Agent{logger: zerolog.Nop(), collector: collector}
|
||||
got, ok := agent.discoverRunningProxmoxLXCContainers(context.Background())
|
||||
if !ok {
|
||||
t.Fatal("discovery was not applicable on a cgroup v2 Proxmox host")
|
||||
}
|
||||
want := []proxmoxLXCRunningContainer{
|
||||
{VMID: 126, Name: "qual2511", PID: 3019253},
|
||||
{VMID: 127, Name: "CT127", PID: 4000},
|
||||
}
|
||||
if fmt.Sprint(got) != fmt.Sprint(want) {
|
||||
t.Fatalf("discovered %+v, want %+v", got, want)
|
||||
}
|
||||
if *commands != 0 {
|
||||
t.Fatalf("discovery ran %d commands, want none", *commands)
|
||||
}
|
||||
}
|
||||
|
||||
// Discovery must never turn a running guest it could not read into a smaller,
|
||||
// healthy-looking inventory. Whenever a guest's state is not established it
|
||||
// hands the whole node back to pct, which works for root installs and is
|
||||
// reported as degraded inside the helper.
|
||||
func TestDiscoverRunningProxmoxLXCContainersLeavesUnestablishedStateToPct(t *testing.T) {
|
||||
running := func() *fakeProxmoxLXCHost {
|
||||
h := newFakeProxmoxLXCHost()
|
||||
h.config(126, "hostname: web\n")
|
||||
h.cgroup("126/ns", 10)
|
||||
h.proc(10, "10\t1", "/lxc/126/ns")
|
||||
return h
|
||||
}
|
||||
cases := map[string]func(h *fakeProxmoxLXCHost) context.Context{
|
||||
"cgroup v1 host": func(h *fakeProxmoxLXCHost) context.Context {
|
||||
delete(h.files, proxmoxLXCCgroupV2Marker)
|
||||
return context.Background()
|
||||
},
|
||||
"unreadable config of a running guest": func(h *fakeProxmoxLXCHost) context.Context {
|
||||
delete(h.files, proxmoxLXCConfigPath(126))
|
||||
return context.Background()
|
||||
},
|
||||
"guest cgroup without an init (a reused PID from another guest)": func(h *fakeProxmoxLXCHost) context.Context {
|
||||
h.proc(10, "10\t1", "/lxc/130/ns")
|
||||
return context.Background()
|
||||
},
|
||||
"unreadable cgroup.procs": func(h *fakeProxmoxLXCHost) context.Context {
|
||||
delete(h.files, path.Join(proxmoxLXCCgroupRoot, "126/ns/cgroup.procs"))
|
||||
return context.Background()
|
||||
},
|
||||
"guest-created cgroup tree beyond the search limit": func(h *fakeProxmoxLXCHost) context.Context {
|
||||
h.proc(10, "10\t2", "/lxc/126/ns")
|
||||
for i := 0; i < proxmoxLXCMaxCgroupDirs; i++ {
|
||||
h.cgroup("126/ns/c" + strconv.Itoa(i))
|
||||
}
|
||||
return context.Background()
|
||||
},
|
||||
"cancelled collection": func(*fakeProxmoxLXCHost) context.Context {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
return ctx
|
||||
},
|
||||
}
|
||||
for name, mutate := range cases {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
h := running()
|
||||
ctx := mutate(h)
|
||||
collector, _ := h.collector(t)
|
||||
got, ok := (&Agent{logger: zerolog.Nop(), collector: collector}).discoverRunningProxmoxLXCContainers(ctx)
|
||||
if ok {
|
||||
t.Fatalf("discovery reported %+v as complete", got)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// lxcObservation is one mount's answer from the fake filesystem prober.
|
||||
type lxcObservation struct {
|
||||
usage *filesystem.Usage
|
||||
notMounted bool
|
||||
err string
|
||||
}
|
||||
|
||||
func lxcUsage(total, used, avail uint64) lxcObservation {
|
||||
return lxcObservation{usage: &filesystem.Usage{CapacityBytes: total, FreeBytes: total - used, AvailableBytes: avail}}
|
||||
}
|
||||
|
||||
// fakeLXCObserve answers prober requests from per-PID mount tables. A PID the
|
||||
// table does not know is not the guest's init, as the real prober would find.
|
||||
func fakeLXCObserve(t *testing.T, byPID map[int]map[string]lxcObservation) func(context.Context, filesystemprobe.ContainerRequest) ([]filesystem.Observation, error) {
|
||||
t.Helper()
|
||||
return func(_ context.Context, req filesystemprobe.ContainerRequest) ([]filesystem.Observation, error) {
|
||||
if req.Runtime != "lxc" || !req.MountsOnly || req.ContainerID == "" {
|
||||
t.Errorf("prober request = %+v, want an exact lxc request with MountsOnly", req)
|
||||
}
|
||||
mounts, ok := byPID[req.PID]
|
||||
if !ok {
|
||||
return nil, errors.New("process is not the container's init")
|
||||
}
|
||||
out := make([]filesystem.Observation, 0, len(req.Mountpoints))
|
||||
for _, mountpoint := range req.Mountpoints {
|
||||
answer, known := mounts[mountpoint]
|
||||
if !known {
|
||||
t.Errorf("unexpected mount %q for pid %d", mountpoint, req.PID)
|
||||
continue
|
||||
}
|
||||
observation := filesystem.Observation{Mountpoint: mountpoint, Source: filesystemprobe.Source}
|
||||
switch {
|
||||
case answer.notMounted:
|
||||
observation.Error = filesystemprobe.ErrNotMounted.Error()
|
||||
case answer.err != "":
|
||||
observation.Error = answer.err
|
||||
default:
|
||||
observation.Usage = answer.usage
|
||||
}
|
||||
out = append(out, observation)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,243 @@
|
||||
package hostagent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/filesystemprobe"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/agents/filesystem"
|
||||
)
|
||||
|
||||
// Socket-free discovery of the node's running containers (#2511).
|
||||
//
|
||||
// pct list, pct df and lxc-info reach pmxcfs and the LXC monitor through
|
||||
// abstract Unix sockets, which belong to a network namespace. The typed helper
|
||||
// runs with PrivateNetwork=true, so every one of them fails there. Container
|
||||
// configs are ordinary reads through the pmxcfs FUSE mount, and a running
|
||||
// container's init process is visible in its cgroup and in /proc, so the helper
|
||||
// can find what it needs without leaving its sandbox. Usage is then read by
|
||||
// the shared filesystem prober, which pins the process, re-checks its identity
|
||||
// and resolves every path inside the container's root.
|
||||
|
||||
const (
|
||||
// proxmoxLXCCgroupRoot is where pve-container places each container's
|
||||
// cgroup (lxc/<vmid>, with the payload under ns/) on a cgroup v2 host.
|
||||
proxmoxLXCCgroupRoot = "/sys/fs/cgroup/lxc"
|
||||
proxmoxLXCCgroupV2Marker = "/sys/fs/cgroup/cgroup.controllers"
|
||||
proxmoxLXCMaxCgroupDirs = 64
|
||||
proxmoxLXCMaxCgroupPIDs = 4096
|
||||
proxmoxLXCMaxCgroupProcsBytes = 64 * 1024
|
||||
proxmoxLXCMaxProcStatusBytes = 64 * 1024
|
||||
proxmoxLXCMaxProcCgroupBytes = 64 * 1024
|
||||
proxmoxLXCMaxConfigEntries = 4096
|
||||
)
|
||||
|
||||
// proxmoxLXCDirLister is an optional collector capability, like
|
||||
// proxmoxLXCObserver: the default collector lists directories and test
|
||||
// collectors opt in.
|
||||
type proxmoxLXCDirLister interface {
|
||||
ReadDir(name string) ([]os.DirEntry, error)
|
||||
}
|
||||
|
||||
// proxmoxLXCObserver reads a running container's filesystem counters through
|
||||
// its pinned init process.
|
||||
type proxmoxLXCObserver interface {
|
||||
ObserveProxmoxLXC(ctx context.Context, req filesystemprobe.ContainerRequest) ([]filesystem.Observation, error)
|
||||
}
|
||||
|
||||
// proxmoxLXCFilesystemObserver is shared by every collection in the process.
|
||||
// A probe stuck on a guest's hung filesystem keeps its slot here, so the
|
||||
// typed helper never stacks a second probe on that guest.
|
||||
var proxmoxLXCFilesystemObserver filesystemprobe.Observer
|
||||
|
||||
func (c *defaultCollector) ReadDir(name string) ([]os.DirEntry, error) {
|
||||
return os.ReadDir(name)
|
||||
}
|
||||
|
||||
func (c *defaultCollector) ObserveProxmoxLXC(ctx context.Context, req filesystemprobe.ContainerRequest) ([]filesystem.Observation, error) {
|
||||
return proxmoxLXCFilesystemObserver.Observe(ctx, req)
|
||||
}
|
||||
|
||||
type proxmoxLXCInitState int
|
||||
|
||||
const (
|
||||
proxmoxLXCStopped proxmoxLXCInitState = iota
|
||||
proxmoxLXCRunning
|
||||
proxmoxLXCUnknown
|
||||
)
|
||||
|
||||
// discoverRunningProxmoxLXCContainers lists this node's running containers
|
||||
// with their init PIDs. ok is false when the host cannot be read this way or
|
||||
// any container's state cannot be established (no directory listing, no
|
||||
// cgroup v2 hierarchy, an unreadable config or cgroup, a search limit, or
|
||||
// cancellation), so the caller uses pct instead of reporting an inventory that
|
||||
// silently misses a running guest. Only a container with no cgroup at all is
|
||||
// stopped and left out, as pct list leaves it out.
|
||||
func (a *Agent) discoverRunningProxmoxLXCContainers(ctx context.Context) ([]proxmoxLXCRunningContainer, bool) {
|
||||
lister, isLister := a.collector.(proxmoxLXCDirLister)
|
||||
if !isLister {
|
||||
return nil, false
|
||||
}
|
||||
if _, err := a.collector.Stat(proxmoxLXCCgroupV2Marker); err != nil {
|
||||
return nil, false
|
||||
}
|
||||
entries, err := lister.ReadDir(proxmoxLXCConfigDir)
|
||||
if err != nil {
|
||||
a.logger.Debug().Err(err).Msg("Cannot list Proxmox LXC configs; using pct")
|
||||
return nil, false
|
||||
}
|
||||
if len(entries) > proxmoxLXCMaxConfigEntries {
|
||||
return nil, false
|
||||
}
|
||||
|
||||
vmids := make([]int, 0, len(entries))
|
||||
for _, entry := range entries {
|
||||
name := entry.Name()
|
||||
if !strings.HasSuffix(name, ".conf") {
|
||||
continue
|
||||
}
|
||||
vmid, convErr := strconv.Atoi(strings.TrimSuffix(name, ".conf"))
|
||||
if convErr != nil || vmid < 100 || vmid > 999999999 {
|
||||
continue
|
||||
}
|
||||
vmids = append(vmids, vmid)
|
||||
}
|
||||
sort.Ints(vmids)
|
||||
|
||||
running := make([]proxmoxLXCRunningContainer, 0, len(vmids))
|
||||
for _, vmid := range vmids {
|
||||
if ctx.Err() != nil {
|
||||
return nil, false
|
||||
}
|
||||
pid, state := a.proxmoxLXCInitPID(ctx, lister, vmid)
|
||||
switch state {
|
||||
case proxmoxLXCStopped:
|
||||
continue
|
||||
case proxmoxLXCUnknown:
|
||||
a.logger.Debug().Int("vmid", vmid).Msg("Cannot establish a Proxmox LXC init process; using pct")
|
||||
return nil, false
|
||||
}
|
||||
raw, readErr := a.collector.ReadFile(proxmoxLXCConfigPath(vmid))
|
||||
if readErr != nil || len(raw) > proxmoxLXCMaxConfigBytes {
|
||||
// Without its config a running container has no trustworthy name or
|
||||
// mounts; leave the whole node to pct rather than guess.
|
||||
a.logger.Debug().Err(readErr).Int("vmid", vmid).Msg("Cannot read a running Proxmox LXC config; using pct")
|
||||
return nil, false
|
||||
}
|
||||
running = append(running, proxmoxLXCRunningContainer{VMID: vmid, Name: proxmoxLXCConfigName(string(raw), vmid), PID: pid})
|
||||
if len(running) == proxmoxLXCMaxContainers {
|
||||
break
|
||||
}
|
||||
}
|
||||
return running, true
|
||||
}
|
||||
|
||||
// proxmoxLXCInitPID finds the container's init process: the process in the
|
||||
// container's cgroup tree that is PID 1 of the namespace directly below the
|
||||
// host's and whose own cgroup is inside lxc/<vmid>. The filesystem prober
|
||||
// re-checks both on the pinned process before reading anything.
|
||||
func (a *Agent) proxmoxLXCInitPID(ctx context.Context, lister proxmoxLXCDirLister, vmid int) (int, proxmoxLXCInitState) {
|
||||
root := path.Join(proxmoxLXCCgroupRoot, strconv.Itoa(vmid))
|
||||
if _, err := lister.ReadDir(root); err != nil {
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return 0, proxmoxLXCStopped
|
||||
}
|
||||
return 0, proxmoxLXCUnknown
|
||||
}
|
||||
|
||||
queue := []string{root}
|
||||
queued := 1
|
||||
checked := 0
|
||||
for len(queue) > 0 {
|
||||
if ctx.Err() != nil {
|
||||
return 0, proxmoxLXCUnknown
|
||||
}
|
||||
dir := queue[0]
|
||||
queue = queue[1:]
|
||||
|
||||
raw, err := a.collector.ReadFile(path.Join(dir, "cgroup.procs"))
|
||||
if err != nil || len(raw) > proxmoxLXCMaxCgroupProcsBytes {
|
||||
return 0, proxmoxLXCUnknown
|
||||
}
|
||||
for _, field := range strings.Fields(string(raw)) {
|
||||
if checked == proxmoxLXCMaxCgroupPIDs {
|
||||
return 0, proxmoxLXCUnknown
|
||||
}
|
||||
checked++
|
||||
pid, convErr := strconv.Atoi(field)
|
||||
if convErr != nil || pid <= 0 {
|
||||
continue
|
||||
}
|
||||
if a.isProxmoxLXCInit(pid, vmid) {
|
||||
return pid, proxmoxLXCRunning
|
||||
}
|
||||
}
|
||||
|
||||
entries, err := lister.ReadDir(dir)
|
||||
if err != nil {
|
||||
return 0, proxmoxLXCUnknown
|
||||
}
|
||||
children := make([]string, 0, len(entries))
|
||||
for _, entry := range entries {
|
||||
if entry.IsDir() {
|
||||
children = append(children, entry.Name())
|
||||
}
|
||||
}
|
||||
// systemd guests run init in init.scope; look there first.
|
||||
sort.SliceStable(children, func(i, j int) bool {
|
||||
return children[i] == "init.scope" && children[j] != "init.scope"
|
||||
})
|
||||
for _, child := range children {
|
||||
if queued == proxmoxLXCMaxCgroupDirs {
|
||||
// A guest can create cgroups in its delegated subtree; give up
|
||||
// rather than walk an unbounded tree.
|
||||
return 0, proxmoxLXCUnknown
|
||||
}
|
||||
queued++
|
||||
queue = append(queue, path.Join(dir, child))
|
||||
}
|
||||
}
|
||||
// A cgroup without an init is a guest starting or stopping.
|
||||
return 0, proxmoxLXCUnknown
|
||||
}
|
||||
|
||||
func (a *Agent) isProxmoxLXCInit(pid, vmid int) bool {
|
||||
status, err := a.collector.ReadFile("/proc/" + strconv.Itoa(pid) + "/status")
|
||||
if err != nil || len(status) > proxmoxLXCMaxProcStatusBytes {
|
||||
return false
|
||||
}
|
||||
if !filesystemprobe.IsNamespaceInit(string(status), pid) {
|
||||
return false
|
||||
}
|
||||
cgroup, err := a.collector.ReadFile("/proc/" + strconv.Itoa(pid) + "/cgroup")
|
||||
if err != nil || len(cgroup) > proxmoxLXCMaxProcCgroupBytes {
|
||||
return false
|
||||
}
|
||||
return filesystemprobe.CgroupMatches(string(cgroup), "lxc", strconv.Itoa(vmid))
|
||||
}
|
||||
|
||||
// proxmoxLXCConfigName is the name pct list and the Proxmox API report: the
|
||||
// configured hostname, or CT<vmid> when none is set. Pulse matches agent
|
||||
// readings to API containers by this name.
|
||||
func proxmoxLXCConfigName(config string, vmid int) string {
|
||||
for _, rawLine := range strings.Split(config, "\n") {
|
||||
line := strings.TrimSpace(rawLine)
|
||||
if strings.HasPrefix(line, "[") {
|
||||
break
|
||||
}
|
||||
value, ok := strings.CutPrefix(line, "hostname:")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if name := strings.TrimSpace(value); name != "" && safeProxmoxLXCText(name, proxmoxLXCMaxNameBytes) {
|
||||
return name
|
||||
}
|
||||
break
|
||||
}
|
||||
return "CT" + strconv.Itoa(vmid)
|
||||
}
|
||||
Reference in New Issue
Block a user