mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-10-23 20:25:56 +00:00
Collect LXC filesystems without pmxcfs or LXC socket IPC
The typed helper runs with PrivateNetwork=true. pct list, pct df and lxc-info reach pmxcfs and the LXC monitor through abstract Unix sockets, which belong to the network namespace, so inside the helper they always fail. Least-privilege installs using the helper therefore reported provider_unavailable for proxmox.lxc_filesystems on every report and never got per-mount guest usage (#2511). Discover running guests without commands: list /etc/pve/lxc through the pmxcfs mount, find each guest's init in /sys/fs/cgroup/lxc/<vmid> (NSpid exactly "<pid> 1", cgroup inside lxc/<vmid>), and name it by hostname or CT<vmid> as pct list does. Discovery hands the whole node back to pct whenever a guest's state cannot be established, so a guest it could not read never disappears from a healthy-looking inventory. Read usage through the shared filesystem prober instead of a second, plain statfs path. It now accepts lxc requests: it pins the process, requires the guest's exact cgroup and init (a nested namespace's PID 1 cannot stand in), skips declared mounts that are not mounted, and keeps one bounded probe per guest. The helper's RestrictSUIDSGID makes systemd answer openat2 with ENOSYS, so the prober falls back to a component walk that refuses every symbolic link rather than resolving it; Docker and Podman reads gain the same fallback. Exercised on a disposable PVE 9.2.18 node under the helper unit's sandbox properties: unprivileged and privileged guests matched pct df, a stopped guest was omitted, and the prober's Linux tests passed as root. A second opinion reviewed the design and the first draft; this version addresses its findings.
This commit is contained in:
@@ -279,7 +279,7 @@ or generic command path.
|
||||
| Proxmox VE/PBS/PMG inventory, status, storage, and ordinary metrics | API-only connection with a narrowly scoped token; no host agent | No collector, helper, or runner authority is required for this data | Supported independently of the safe host-agent profile; it does not prove host-local SMART, LXC filesystem, or action parity | `agent-lifecycle`: keep API permissions and returned telemetry covered by provider tests |
|
||||
| Standard Linux systemd host telemetry and collector update | Unprivileged `pulse-agent` plus the root-owned typed helper | Core `/proc`, filesystem, network, RAID, and hwmon telemetry stays in the collector; signed updates cross the fixed helper protocol and commit only after an authoritative report | **Qualified on disposable Ubuntu 24.04.4 arm64 at committed main `22fd662fb794f63efb9d3ca2158de73c4e07e1b8`** across the schema-v6 twenty-scenario run. It covers install, migration, explicit/automatic rollback, helper protocol and resource boundaries, effective-unit override rejection, reporting continuity, authoritative executable-digest commit, watchdog rollback, helper-restart recovery, and last-known-good restoration. It remains opt-in pending exact-RC reproduction, representative provider/appliance parity, and external review | `deployment-installability` and `security-privacy`: reproduce the same helper transaction from trusted release artifacts on representative hosts and accept the external boundary review |
|
||||
| Linux SMART telemetry | `smart.snapshot` through the no-network helper; no caller-selected device or arguments | Implemented, unqualified on representative physical disks. Helper failure omits/degrades SMART only; the collector does not retry as root | Does not yet justify SMART parity or a default change | `agent-lifecycle`: record live SATA, SAS/controller, USB bridge, and NVMe evidence, including standby, permission failure, timeout, and partial-data cases |
|
||||
| Proxmox node-local LXC filesystem telemetry | `proxmox.lxc_filesystems` through the no-network helper using fixed bounded `pct` operations | Implemented, unqualified on a representative PVE node. Helper failure omits/degrades this snapshot only | Does not yet justify Proxmox host-agent parity or a default change | `agent-lifecycle`: record live running/stopped LXC, mount, timeout, output-bound, and helper-loss behavior on supported PVE versions |
|
||||
| Proxmox node-local LXC filesystem telemetry | `proxmox.lxc_filesystems` through the no-network helper. Configs are read through the pmxcfs mount, running guests and their init processes come from cgroup v2 and `/proc`, and usage comes from the shared filesystem prober, which pins the guest's init, re-checks its identity, and resolves each mount inside the guest root (openat2, or a symlink-refusing component walk where the helper's seccomp filter blocks openat2); the caller selects no VMID, path, or command. `pct` and `lxc-info` cannot work inside the helper, because pmxcfs and the LXC monitor listen on abstract Unix sockets that belong to the host network namespace (#2511); they remain the fallback whenever discovery cannot establish a guest's state, on cgroup v1 hosts, for root and `--grant-pct` installs, and per container when the prober read fails | Implemented. Exercised on a disposable PVE 9.2.18 node under the helper unit's sandbox properties, with running unprivileged (rootfs plus a mount point) and privileged guests and a stopped one; readings matched `pct df`. This is one PVE version, with no timeout, output-bound, helper-loss, or live nested-namespace evidence. Helper failure omits/degrades this snapshot only | Does not yet justify Proxmox host-agent parity or a default change | `agent-lifecycle`: record timeout, output-bound, and helper-loss behavior across supported PVE versions |
|
||||
| Rootful Docker or Podman inventory | No direct collector access to a root-equivalent daemon socket | Implemented as a typed-helper summary-only fallback. Migration preserves container ID/name/image/state/status/creation inventory and marks the report `typed-helper-summary`; stats, secondary inventories, update checks, and actions remain unavailable | The standalone rootless packet exercises live same-family rootful Docker and Podman helper fallback plus summary-inventory semantic parity during socket loss. It does not qualify the complete standalone rootful fresh-install, migration, restart, helper-loss/recovery, and bounds matrix. Full rootful parity remains an explicit default blocker; the legacy/root profile is not safe-profile evidence | `agent-lifecycle`: record the remaining standalone lifecycle matrix on representative rootful Docker and Podman and decide explicitly whether reduced telemetry is sufficient |
|
||||
| Collector-owned rootless Docker or Podman | Direct access only to one usable runtime socket owned by the `pulse-agent` UID whose daemon also attests rootless mode | Qualified locally for the exact source-bound monitoring path. Discovery is deferred until the collector account exists and accepts exactly one readable/writable, non-symlink socket owned by that UID across Docker and Podman. Cross-user, root-owned, remote, unreadable, non-rootless-daemon, or ambiguous endpoints are rejected. An installer-owned exact pin survives daemon loss and update; the running collector can move between direct rootless monitoring and rootful typed-helper summary without regaining action authority | **Qualified on distinct disposable Ubuntu 24.04/systemd 255 arm64 hosts at commit `60041ad9e60c282c892f944e04f777b874991a5d`.** All eleven canonical scenarios passed for Docker and all eleven for Podman. Receipt SHA-256: `7a116d63ab0cd1560482165055f0a8c9158ce0a8333a27bd9e8256582a52dfb5`; local attestation SHA-256: `566279ecd7d7dfa89c92f24243e9fcd5ae3e295d4284bd559d4be42f1e83b3b5`. This is local opt-in artifact-bound self-attestation, not published-RC provenance, action/update qualification, external review, or authorization for a default change | `deployment-installability`: reproduce the exact packet from published release artifacts; separately qualify any desired rootless image-update/action surface; retain the opt-in default until the remaining provider, appliance, and review conditions pass |
|
||||
| Separate runner package update and package-cache cleanup | Root-owned `pulse-agent-runner`, host-bound action credential, typed request, postcondition, and durable receipt | The schema-v6 committed-main systemd receipt records a real verified apt-cache mutation, stale-fingerprint refusal, replay, nonce-bound readiness, effective-unit override rejection, exact credential rotation, bodyless self-revocation, and collector/helper continuity. A separate production Router regression exercises HTTPS issuance, WSS admission, encrypted token persistence, failed-rotation rollback, exact socket invalidation, two server restarts, old-secret rejection, and durable self-revoke | Qualified for the exercised systemd fixture paths and focused production Router lifecycle. Focused installer tests also cover atomic pending cancellation and fail-closed credential retention. The evidence does not cover every runner operation, representative providers, or an exact release candidate | `agent-lifecycle` and `api-contracts`: reproduce the combined systemd and production Router path from trusted RC artifacts with representative package-update success/failure/cancellation evidence |
|
||||
|
||||
@@ -572,11 +572,32 @@ standby evidence. Enumeration and each device probe have independent bounded
|
||||
deadlines so one slow disk cannot consume the complete SMART pass, while
|
||||
command errors retain bounded stderr for compatibility diagnosis.
|
||||
On a PVE node, the Unified Agent may also report the mounted filesystem
|
||||
capacity of running LXCs. The collector uses the fixed local `pct list` command
|
||||
to admit only node-local running guests, reads each admitted guest's main
|
||||
`/etc/pve/lxc/<vmid>.conf` mount declarations, resolves its init PID through
|
||||
the fixed `lxc-info -n <vmid> -p` shape, and probes usage through the Linux
|
||||
`/proc/<pid>/root` namespace. A configured path that resolves to the same
|
||||
capacity of running LXCs. On a cgroup v2 node the collector admits node-local
|
||||
guests from the `/etc/pve/lxc/*.conf` listing and admits only those with a live
|
||||
init process: a PID in `/sys/fs/cgroup/lxc/<vmid>`'s tree that is PID 1 of the
|
||||
namespace directly below the host's (`NSpid` exactly `<pid> 1`) and whose own
|
||||
cgroup is inside `lxc/<vmid>`. It names each by its configured hostname or
|
||||
`CT<vmid>`, as `pct list` does, reads its main mount declarations, and reads
|
||||
usage through the shared `internal/filesystemprobe` observer, which pins the
|
||||
process, re-checks that identity, resolves each mount inside the guest root
|
||||
and holds one bounded in-flight probe per guest across the process. No command
|
||||
runs on this path. `pct list`, `pct df` and `lxc-info` reach pmxcfs and the
|
||||
LXC monitor through abstract Unix sockets scoped to the host network
|
||||
namespace, so inside the `PrivateNetwork` typed helper they always fail, and
|
||||
least-privilege installs using that helper reported `provider_unavailable`
|
||||
on every report (#2511). Discovery hands the whole node back to `pct list`
|
||||
whenever it cannot establish a guest's state (no cgroup v2, an unreadable
|
||||
listing, config, cgroup or process, a search limit, or cancellation); only a
|
||||
guest without a cgroup counts as stopped. The fixed `lxc-info -n <vmid> -p`
|
||||
PID lookup remains for that path, and `pct df` remains the per-container
|
||||
fallback, including through the `--grant-pct` wrapper.
|
||||
`TestHelperProxmoxLXCFilesystemsNeedNoPctOrLXCSockets` drives the collector
|
||||
the helper's provider runs with every command failing and requires a complete
|
||||
inventory; `TestDiscoverRunningProxmoxLXCContainersLeavesUnestablishedStateToPct`
|
||||
pins the fallbacks. On a disposable PVE 9.2.18 node, under the helper unit's
|
||||
sandbox properties, it returned unprivileged (rootfs plus mount point) and
|
||||
privileged guests matching `pct df` and omitted a stopped one; a nested PID
|
||||
namespace was not reproduced there and is covered by tests only. A configured path that resolves to the same
|
||||
device as its parent is not a live mount and must be omitted instead of
|
||||
borrowing the parent's capacity. Least-privilege installs that cannot read the
|
||||
config, resolve the PID, or traverse `/proc` may fall back to the fixed
|
||||
|
||||
@@ -844,6 +844,27 @@ tests, container collection boundary tests, ingestion regression and the
|
||||
opt-in owned tmpfs collector fixture. Installed-agent and model qualification
|
||||
are tracked separately in PATROL_ASSISTANT_CUSTOMER_JOURNEY.md.
|
||||
|
||||
The same observer reads Proxmox LXC guests for the host agent and its typed
|
||||
helper (#2511). An `lxc` request names the decimal VMID; the pinned process
|
||||
must sit in that guest's exact `lxc/<vmid>` cgroup and be its init (`NSpid` is
|
||||
exactly `<pid> 1`, so the PID 1 of a namespace nested inside the guest, such
|
||||
as Docker in LXC, cannot stand in for it), checked before and after the read.
|
||||
With `MountsOnly`, a configured mountpoint sharing its parent's device reports
|
||||
`ErrNotMounted` instead of the parent's counters. Docker and Podman identity
|
||||
rules are unchanged. Confined resolution prefers openat2 with `RESOLVE_IN_ROOT`;
|
||||
when the kernel or a systemd seccomp filter answers ENOSYS (the typed helper's
|
||||
`RestrictSUIDSGID=true` blocks openat2 because its flags cannot be inspected),
|
||||
the clean path is walked component by component with `O_NOFOLLOW` and any
|
||||
symbolic link is refused, so neither path resolves outside the guest root.
|
||||
Docker and Podman reads take the same fallback, so a collector under such a
|
||||
filter now gets confined counters instead of unavailable evidence.
|
||||
`TestComponentWalkRefusesSymlinksAndStaysBeneathRoot`,
|
||||
`TestMountsOnlyDetectsUnmountedPathsWithinRoot`,
|
||||
`TestNonInitProcessCannotSupplyProxmoxGuestCapacity` and
|
||||
`TestProxmoxLXCIdentityIsExact` pin these rules; the Linux tests also passed
|
||||
as root on a disposable PVE 9.2.18 node, where the helper-sandboxed collector
|
||||
returned readings matching `pct df` through the component walk.
|
||||
|
||||
Docker mount collection preserves both native `Mounts` records and entries
|
||||
reported only in `HostConfig.Tmpfs`. Existing reported destinations remain
|
||||
authoritative. Additional tmpfs destinations are ordered deterministically,
|
||||
|
||||
@@ -279,7 +279,7 @@ or generic command path.
|
||||
| Proxmox VE/PBS/PMG inventory, status, storage, and ordinary metrics | API-only connection with a narrowly scoped token; no host agent | No collector, helper, or runner authority is required for this data | Supported independently of the safe host-agent profile; it does not prove host-local SMART, LXC filesystem, or action parity | `agent-lifecycle`: keep API permissions and returned telemetry covered by provider tests |
|
||||
| Standard Linux systemd host telemetry and collector update | Unprivileged `pulse-agent` plus the root-owned typed helper | Core `/proc`, filesystem, network, RAID, and hwmon telemetry stays in the collector; signed updates cross the fixed helper protocol and commit only after an authoritative report | **Qualified on disposable Ubuntu 24.04.4 arm64 at committed main `22fd662fb794f63efb9d3ca2158de73c4e07e1b8`** across the schema-v6 twenty-scenario run. It covers install, migration, explicit/automatic rollback, helper protocol and resource boundaries, effective-unit override rejection, reporting continuity, authoritative executable-digest commit, watchdog rollback, helper-restart recovery, and last-known-good restoration. It remains opt-in pending exact-RC reproduction, representative provider/appliance parity, and external review | `deployment-installability` and `security-privacy`: reproduce the same helper transaction from trusted release artifacts on representative hosts and accept the external boundary review |
|
||||
| Linux SMART telemetry | `smart.snapshot` through the no-network helper; no caller-selected device or arguments | Implemented, unqualified on representative physical disks. Helper failure omits/degrades SMART only; the collector does not retry as root | Does not yet justify SMART parity or a default change | `agent-lifecycle`: record live SATA, SAS/controller, USB bridge, and NVMe evidence, including standby, permission failure, timeout, and partial-data cases |
|
||||
| Proxmox node-local LXC filesystem telemetry | `proxmox.lxc_filesystems` through the no-network helper using fixed bounded `pct` operations | Implemented, unqualified on a representative PVE node. Helper failure omits/degrades this snapshot only | Does not yet justify Proxmox host-agent parity or a default change | `agent-lifecycle`: record live running/stopped LXC, mount, timeout, output-bound, and helper-loss behavior on supported PVE versions |
|
||||
| Proxmox node-local LXC filesystem telemetry | `proxmox.lxc_filesystems` through the no-network helper. Configs are read through the pmxcfs mount, running guests and their init processes come from cgroup v2 and `/proc`, and usage comes from the shared filesystem prober, which pins the guest's init, re-checks its identity, and resolves each mount inside the guest root (openat2, or a symlink-refusing component walk where the helper's seccomp filter blocks openat2); the caller selects no VMID, path, or command. `pct` and `lxc-info` cannot work inside the helper, because pmxcfs and the LXC monitor listen on abstract Unix sockets that belong to the host network namespace (#2511); they remain the fallback whenever discovery cannot establish a guest's state, on cgroup v1 hosts, for root and `--grant-pct` installs, and per container when the prober read fails | Implemented. Exercised on a disposable PVE 9.2.18 node under the helper unit's sandbox properties, with running unprivileged (rootfs plus a mount point) and privileged guests and a stopped one; readings matched `pct df`. This is one PVE version, with no timeout, output-bound, helper-loss, or live nested-namespace evidence. Helper failure omits/degrades this snapshot only | Does not yet justify Proxmox host-agent parity or a default change | `agent-lifecycle`: record timeout, output-bound, and helper-loss behavior across supported PVE versions |
|
||||
| Rootful Docker or Podman inventory | No direct collector access to a root-equivalent daemon socket | Implemented as a typed-helper summary-only fallback. Migration preserves container ID/name/image/state/status/creation inventory and marks the report `typed-helper-summary`; stats, secondary inventories, update checks, and actions remain unavailable | The standalone rootless packet exercises live same-family rootful Docker and Podman helper fallback plus summary-inventory semantic parity during socket loss. It does not qualify the complete standalone rootful fresh-install, migration, restart, helper-loss/recovery, and bounds matrix. Full rootful parity remains an explicit default blocker; the legacy/root profile is not safe-profile evidence | `agent-lifecycle`: record the remaining standalone lifecycle matrix on representative rootful Docker and Podman and decide explicitly whether reduced telemetry is sufficient |
|
||||
| Collector-owned rootless Docker or Podman | Direct access only to one usable runtime socket owned by the `pulse-agent` UID whose daemon also attests rootless mode | Qualified locally for the exact source-bound monitoring path. Discovery is deferred until the collector account exists and accepts exactly one readable/writable, non-symlink socket owned by that UID across Docker and Podman. Cross-user, root-owned, remote, unreadable, non-rootless-daemon, or ambiguous endpoints are rejected. An installer-owned exact pin survives daemon loss and update; the running collector can move between direct rootless monitoring and rootful typed-helper summary without regaining action authority | **Qualified on distinct disposable Ubuntu 24.04/systemd 255 arm64 hosts at commit `60041ad9e60c282c892f944e04f777b874991a5d`.** All eleven canonical scenarios passed for Docker and all eleven for Podman. Receipt SHA-256: `7a116d63ab0cd1560482165055f0a8c9158ce0a8333a27bd9e8256582a52dfb5`; local attestation SHA-256: `566279ecd7d7dfa89c92f24243e9fcd5ae3e295d4284bd559d4be42f1e83b3b5`. This is local opt-in artifact-bound self-attestation, not published-RC provenance, action/update qualification, external review, or authorization for a default change | `deployment-installability`: reproduce the exact packet from published release artifacts; separately qualify any desired rootless image-update/action surface; retain the opt-in default until the remaining provider, appliance, and review conditions pass |
|
||||
| Separate runner package update and package-cache cleanup | Root-owned `pulse-agent-runner`, host-bound action credential, typed request, postcondition, and durable receipt | The schema-v6 committed-main systemd receipt records a real verified apt-cache mutation, stale-fingerprint refusal, replay, nonce-bound readiness, effective-unit override rejection, exact credential rotation, bodyless self-revocation, and collector/helper continuity. A separate production Router regression exercises HTTPS issuance, WSS admission, encrypted token persistence, failed-rotation rollback, exact socket invalidation, two server restarts, old-secret rejection, and durable self-revoke | Qualified for the exercised systemd fixture paths and focused production Router lifecycle. Focused installer tests also cover atomic pending cancellation and fail-closed credential retention. The evidence does not cover every runner operation, representative providers, or an exact release candidate | `agent-lifecycle` and `api-contracts`: reproduce the combined systemd and production Router path from trusted RC artifacts with representative package-update success/failure/cancellation evidence |
|
||||
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"path"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
@@ -24,8 +25,24 @@ const (
|
||||
type ContainerRequest struct {
|
||||
PID int
|
||||
ContainerID string
|
||||
// Runtime is docker or podman (ContainerID is the 64-hex ID) or lxc, a
|
||||
// Proxmox container (ContainerID is the decimal VMID and PID must be the
|
||||
// container's init process).
|
||||
Runtime string
|
||||
Mountpoints []string
|
||||
// MountsOnly reports a mountpoint that shares its parent's device as
|
||||
// ErrNotMounted instead of returning the parent filesystem's counters. A
|
||||
// Proxmox config can declare a mount point the running guest never mounted.
|
||||
MountsOnly bool
|
||||
}
|
||||
|
||||
// ErrNotMounted marks an observation whose path is not a distinct mounted
|
||||
// filesystem inside the container. Callers test it with IsNotMounted.
|
||||
var ErrNotMounted = errors.New("path is not a mounted filesystem in the container")
|
||||
|
||||
// IsNotMounted reports whether an observation was rejected as ErrNotMounted.
|
||||
func IsNotMounted(observation filesystem.Observation) bool {
|
||||
return observation.Error == ErrNotMounted.Error()
|
||||
}
|
||||
|
||||
// Observer bounds potentially uninterruptible kernel reads. A timed-out probe
|
||||
@@ -101,12 +118,18 @@ func validateRequest(req ContainerRequest) error {
|
||||
if req.PID <= 0 {
|
||||
return errors.New("container process is unavailable")
|
||||
}
|
||||
if len(req.ContainerID) != 64 || strings.IndexFunc(req.ContainerID, func(r rune) bool {
|
||||
return !(r >= '0' && r <= '9' || r >= 'a' && r <= 'f')
|
||||
}) >= 0 {
|
||||
return errors.New("filesystem observation requires an exact container ID")
|
||||
}
|
||||
if req.Runtime != "docker" && req.Runtime != "podman" {
|
||||
switch req.Runtime {
|
||||
case "docker", "podman":
|
||||
if len(req.ContainerID) != 64 || strings.IndexFunc(req.ContainerID, func(r rune) bool {
|
||||
return !(r >= '0' && r <= '9' || r >= 'a' && r <= 'f')
|
||||
}) >= 0 {
|
||||
return errors.New("filesystem observation requires an exact container ID")
|
||||
}
|
||||
case "lxc":
|
||||
if !validLXCVMID(req.ContainerID) {
|
||||
return errors.New("filesystem observation requires an exact Proxmox VMID")
|
||||
}
|
||||
default:
|
||||
return errors.New("unsupported container runtime")
|
||||
}
|
||||
if len(req.Mountpoints) == 0 || len(req.Mountpoints) > maxMountpoints {
|
||||
@@ -122,9 +145,53 @@ func validateRequest(req ContainerRequest) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func validLXCVMID(id string) bool {
|
||||
vmid, err := strconv.Atoi(id)
|
||||
return err == nil && vmid >= 100 && vmid <= 999999999 && strconv.Itoa(vmid) == id
|
||||
}
|
||||
|
||||
// CgroupMatches reports whether a /proc/<pid>/cgroup body places the process
|
||||
// inside the given container, with the same exact-identity rules the observer
|
||||
// applies to its pinned process.
|
||||
func CgroupMatches(raw, runtime, containerID string) bool {
|
||||
return cgroupMatches(raw, ContainerRequest{Runtime: runtime, ContainerID: containerID})
|
||||
}
|
||||
|
||||
// IsNamespaceInit reports whether a /proc/<pid>/status body describes pid as
|
||||
// PID 1 of a PID namespace one level below the reader's: NSpid is exactly
|
||||
// "<pid> 1". A process that is PID 1 of a namespace nested inside the
|
||||
// container (Docker inside an LXC guest) lists three or more PIDs and is not
|
||||
// the container's init.
|
||||
func IsNamespaceInit(status string, pid int) bool {
|
||||
for _, line := range strings.Split(status, "\n") {
|
||||
rest, ok := strings.CutPrefix(line, "NSpid:")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
fields := strings.Fields(rest)
|
||||
return len(fields) == 2 && fields[0] == strconv.Itoa(pid) && fields[1] == "1"
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// cgroupMatches accepts an exact runtime-owned cgroup component, never a
|
||||
// substring, container name, short ID, or unrelated process with the same PID.
|
||||
func cgroupMatches(raw string, req ContainerRequest) bool {
|
||||
if req.Runtime == "lxc" {
|
||||
// pve-container places a guest under lxc/<vmid> (payload in ns/).
|
||||
// lxc.monitor/<vmid> and other VMIDs sharing a prefix never match.
|
||||
for _, line := range strings.Split(raw, "\n") {
|
||||
parts := strings.SplitN(line, ":", 3)
|
||||
if len(parts) != 3 {
|
||||
continue
|
||||
}
|
||||
components := strings.Split(parts[2], "/")
|
||||
if len(components) >= 3 && components[0] == "" && components[1] == "lxc" && components[2] == req.ContainerID {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
scope := "docker-" + req.ContainerID + ".scope"
|
||||
if req.Runtime == "podman" {
|
||||
scope = "libpod-" + req.ContainerID + ".scope"
|
||||
|
||||
@@ -9,7 +9,9 @@ import (
|
||||
"io"
|
||||
"math"
|
||||
"os"
|
||||
"path"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/agents/filesystem"
|
||||
@@ -25,7 +27,7 @@ func observeContainer(ctx context.Context, req ContainerRequest) ([]filesystem.O
|
||||
return nil, fmt.Errorf("open container process: %w", err)
|
||||
}
|
||||
defer unix.Close(proc)
|
||||
if err := verifyCgroup(proc, req); err != nil {
|
||||
if err := verifyIdentity(proc, req); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
root, err := unix.Openat(proc, "root", unix.O_PATH|unix.O_DIRECTORY|unix.O_CLOEXEC, 0)
|
||||
@@ -33,7 +35,7 @@ func observeContainer(ctx context.Context, req ContainerRequest) ([]filesystem.O
|
||||
return nil, fmt.Errorf("open container filesystem namespace: %w", err)
|
||||
}
|
||||
defer unix.Close(root)
|
||||
if err := verifyCgroup(proc, req); err != nil {
|
||||
if err := verifyIdentity(proc, req); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
observations := make([]filesystem.Observation, 0, len(req.Mountpoints))
|
||||
@@ -42,6 +44,14 @@ func observeContainer(ctx context.Context, req ContainerRequest) ([]filesystem.O
|
||||
return nil, err
|
||||
}
|
||||
observation := filesystem.Observation{Mountpoint: mount, Source: Source}
|
||||
if req.MountsOnly {
|
||||
if mounted, mountErr := mountedWithinRoot(root, mount); mountErr == nil && !mounted {
|
||||
observation.ObservedAt = time.Now().UTC()
|
||||
observation.Error = ErrNotMounted.Error()
|
||||
observations = append(observations, observation)
|
||||
continue
|
||||
}
|
||||
}
|
||||
observation.Type, observation.Usage, err = observePath(root, mount)
|
||||
observation.ObservedAt = time.Now().UTC()
|
||||
if err != nil {
|
||||
@@ -49,12 +59,70 @@ func observeContainer(ctx context.Context, req ContainerRequest) ([]filesystem.O
|
||||
}
|
||||
observations = append(observations, observation)
|
||||
}
|
||||
if err := verifyCgroup(proc, req); err != nil {
|
||||
if err := verifyIdentity(proc, req); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return observations, nil
|
||||
}
|
||||
|
||||
func verifyIdentity(proc int, req ContainerRequest) error {
|
||||
if err := verifyCgroup(proc, req); err != nil {
|
||||
return err
|
||||
}
|
||||
if req.Runtime == "lxc" {
|
||||
return verifyNamespaceInit(proc, req.PID)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// verifyNamespaceInit requires the pinned process to be the guest's init, so a
|
||||
// process of a namespace nested inside the guest cannot supply its own root.
|
||||
func verifyNamespaceInit(proc, pid int) error {
|
||||
fd, err := unix.Openat(proc, "status", unix.O_RDONLY|unix.O_CLOEXEC|unix.O_NOFOLLOW, 0)
|
||||
if err != nil {
|
||||
return fmt.Errorf("read container process status: %w", err)
|
||||
}
|
||||
file := os.NewFile(uintptr(fd), "status")
|
||||
defer file.Close()
|
||||
raw, err := io.ReadAll(io.LimitReader(file, 65537))
|
||||
if err != nil || len(raw) > 65536 {
|
||||
return errors.New("container process status is unavailable")
|
||||
}
|
||||
if !IsNamespaceInit(string(raw), pid) {
|
||||
return errors.New("process is not the container's init")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// mountedWithinRoot reports whether mount is a different filesystem from its
|
||||
// parent directory, both resolved within the resource root.
|
||||
func mountedWithinRoot(root int, mount string) (bool, error) {
|
||||
if mount == "/" {
|
||||
return true, nil
|
||||
}
|
||||
device := func(target string) (uint64, error) {
|
||||
fd, err := openInRoot(root, target)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer unix.Close(fd)
|
||||
var stat unix.Stat_t
|
||||
if err := unix.Fstat(fd, &stat); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return uint64(stat.Dev), nil
|
||||
}
|
||||
own, err := device(mount)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
parent, err := device(path.Dir(mount))
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
return own != parent, nil
|
||||
}
|
||||
|
||||
func verifyCgroup(proc int, req ContainerRequest) error {
|
||||
fd, err := unix.Openat(proc, "cgroup", unix.O_RDONLY|unix.O_CLOEXEC|unix.O_NOFOLLOW, 0)
|
||||
if err != nil {
|
||||
@@ -72,13 +140,57 @@ func verifyCgroup(proc int, req ContainerRequest) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func observePath(root int, mount string) (string, *filesystem.Usage, error) {
|
||||
// Absolute symlinks stay within the resource root. Magic links cannot escape
|
||||
// it. Unsupported kernels fail rather than falling back to unsafe resolution.
|
||||
// openInRoot opens mount beneath root without ever leaving it. With openat2,
|
||||
// absolute symlinks stay within the resource root and magic links cannot
|
||||
// escape it. systemd answers openat2 with ENOSYS under RestrictSUIDSGID,
|
||||
// because seccomp cannot inspect its flags, and the typed helper sets that, so
|
||||
// there the clean path is walked one component at a time and any symbolic link
|
||||
// is refused rather than resolved. Neither path falls back to unconfined
|
||||
// resolution.
|
||||
func openInRoot(root int, mount string) (int, error) {
|
||||
fd, err := unix.Openat2(root, mount, &unix.OpenHow{
|
||||
Flags: unix.O_PATH | unix.O_CLOEXEC,
|
||||
Resolve: unix.RESOLVE_IN_ROOT | unix.RESOLVE_NO_MAGICLINKS,
|
||||
})
|
||||
if !errors.Is(err, unix.ENOSYS) {
|
||||
return fd, err
|
||||
}
|
||||
return openBeneathWithoutSymlinks(root, mount)
|
||||
}
|
||||
|
||||
func openBeneathWithoutSymlinks(root int, mount string) (int, error) {
|
||||
if !path.IsAbs(mount) || path.Clean(mount) != mount {
|
||||
return -1, errors.New("mount path must be clean and absolute")
|
||||
}
|
||||
current, err := unix.Dup(root)
|
||||
if err != nil {
|
||||
return -1, err
|
||||
}
|
||||
for _, component := range strings.Split(strings.TrimPrefix(mount, "/"), "/") {
|
||||
if component == "" {
|
||||
continue
|
||||
}
|
||||
next, err := unix.Openat(current, component, unix.O_PATH|unix.O_NOFOLLOW|unix.O_CLOEXEC, 0)
|
||||
unix.Close(current)
|
||||
if err != nil {
|
||||
return -1, err
|
||||
}
|
||||
var stat unix.Stat_t
|
||||
if err := unix.Fstat(next, &stat); err != nil {
|
||||
unix.Close(next)
|
||||
return -1, err
|
||||
}
|
||||
if stat.Mode&unix.S_IFMT == unix.S_IFLNK {
|
||||
unix.Close(next)
|
||||
return -1, errors.New("mount path crosses a symbolic link")
|
||||
}
|
||||
current = next
|
||||
}
|
||||
return current, nil
|
||||
}
|
||||
|
||||
func observePath(root int, mount string) (string, *filesystem.Usage, error) {
|
||||
fd, err := openInRoot(root, mount)
|
||||
if err != nil {
|
||||
return "", nil, fmt.Errorf("open mount within resource namespace: %w", err)
|
||||
}
|
||||
|
||||
@@ -81,3 +81,68 @@ func TestWrongProcessCannotSupplyContainerCapacity(t *testing.T) {
|
||||
t.Fatalf("unrelated process supplied capacity: %+v %v", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
// A configured mountpoint that shares its parent's device is not mounted in
|
||||
// the guest; reporting it would copy the parent filesystem's counters.
|
||||
func TestMountsOnlyDetectsUnmountedPathsWithinRoot(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
if err := os.Mkdir(filepath.Join(dir, "data"), 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
fd, err := unix.Open(dir, unix.O_PATH|unix.O_DIRECTORY|unix.O_CLOEXEC, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer unix.Close(fd)
|
||||
if mounted, err := mountedWithinRoot(fd, "/"); err != nil || !mounted {
|
||||
t.Fatalf("root reported unmounted: %v %v", mounted, err)
|
||||
}
|
||||
if mounted, err := mountedWithinRoot(fd, "/data"); err != nil || mounted {
|
||||
t.Fatalf("plain directory reported as a mount: %v %v", mounted, err)
|
||||
}
|
||||
if _, err := mountedWithinRoot(fd, "/missing"); err == nil {
|
||||
t.Fatal("missing path reported a mount state")
|
||||
}
|
||||
}
|
||||
|
||||
// The test process is not PID 1 of a namespace, so it cannot supply a Proxmox
|
||||
// guest's filesystems even if a request names it.
|
||||
func TestNonInitProcessCannotSupplyProxmoxGuestCapacity(t *testing.T) {
|
||||
r := ContainerRequest{PID: os.Getpid(), ContainerID: "126", Runtime: "lxc", Mountpoints: []string{"/"}, MountsOnly: true}
|
||||
if got, err := observeContainer(context.Background(), r); err == nil || got != nil {
|
||||
t.Fatalf("non-init process supplied capacity: %+v %v", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
// Inside the typed helper systemd blocks openat2 (RestrictSUIDSGID), so mounts
|
||||
// are resolved component by component. That walk must stay beneath the root
|
||||
// by refusing every symbolic link, including ones that would stay inside.
|
||||
func TestComponentWalkRefusesSymlinksAndStaysBeneathRoot(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
if err := os.MkdirAll(filepath.Join(dir, "srv", "data"), 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for name, target := range map[string]string{"inside-link": "/srv", "outside": "/etc", "relative-up": "../../.."} {
|
||||
if err := os.Symlink(target, filepath.Join(dir, name)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
root, err := unix.Open(dir, unix.O_PATH|unix.O_DIRECTORY|unix.O_CLOEXEC, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer unix.Close(root)
|
||||
for _, mount := range []string{"/", "/srv", "/srv/data"} {
|
||||
fd, err := openBeneathWithoutSymlinks(root, mount)
|
||||
if err != nil {
|
||||
t.Fatalf("plain path %s refused: %v", mount, err)
|
||||
}
|
||||
unix.Close(fd)
|
||||
}
|
||||
for _, mount := range []string{"/inside-link", "/inside-link/data", "/outside", "/outside/passwd", "/relative-up", "/srv/../etc", "srv", "/proc/self/root"} {
|
||||
if fd, err := openBeneathWithoutSymlinks(root, mount); err == nil {
|
||||
unix.Close(fd)
|
||||
t.Fatalf("%s resolved through the component walk", mount)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -79,3 +79,57 @@ func TestObserverRejectsInvalidCoordinatesBeforeNativeRead(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// A Proxmox guest is identified by its exact lxc/<vmid> cgroup and its init
|
||||
// process. Neither another VMID with the same prefix, LXC's monitor cgroup,
|
||||
// nor the PID 1 of a namespace nested inside the guest (Docker in LXC) may
|
||||
// stand in for it.
|
||||
func TestProxmoxLXCIdentityIsExact(t *testing.T) {
|
||||
for raw, want := range map[string]bool{
|
||||
"0::/lxc/126/ns": true,
|
||||
"0::/lxc/126/ns/init.scope": true,
|
||||
"0::/lxc/126": true,
|
||||
"0::/lxc/1260/ns": false,
|
||||
"0::/lxc/12/ns": false,
|
||||
"0::/lxc.monitor/126": false,
|
||||
"0::/system.slice/lxc/126/ns": false,
|
||||
"0::/docker/" + strings.Repeat("a", 64): false,
|
||||
} {
|
||||
if got := CgroupMatches(raw, "lxc", "126"); got != want {
|
||||
t.Errorf("CgroupMatches(%q) = %v, want %v", raw, got, want)
|
||||
}
|
||||
}
|
||||
for status, want := range map[string]bool{
|
||||
"Name:\tinit\nNSpid:\t3019253\t1\n": true,
|
||||
"NSpid:\t3019253\t413\n": false,
|
||||
"NSpid:\t3019253\t420\t1\n": false, // nested namespace init
|
||||
"NSpid:\t1\n": false, // the host's own init
|
||||
"NSpid:\t999\t1\n": false, // another process's line
|
||||
"Name:\tinit\n": false,
|
||||
} {
|
||||
if got := IsNamespaceInit(status, 3019253); got != want {
|
||||
t.Errorf("IsNamespaceInit(%q) = %v, want %v", status, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestObserverRejectsInexactProxmoxVMIDs(t *testing.T) {
|
||||
o := &Observer{probe: func(context.Context, ContainerRequest) ([]filesystem.Observation, error) {
|
||||
t.Error("invalid coordinates reached native read")
|
||||
return nil, nil
|
||||
}}
|
||||
for _, id := range []string{"99", "0126", "126a", "-126", "1000000000", ""} {
|
||||
r := ContainerRequest{PID: 42, ContainerID: id, Runtime: "lxc", Mountpoints: []string{"/"}}
|
||||
if _, err := o.Observe(context.Background(), r); err == nil {
|
||||
t.Fatalf("accepted VMID %q", id)
|
||||
}
|
||||
}
|
||||
accepted := false
|
||||
o.probe = func(context.Context, ContainerRequest) ([]filesystem.Observation, error) {
|
||||
accepted = true
|
||||
return nil, nil
|
||||
}
|
||||
if _, err := o.Observe(context.Background(), ContainerRequest{PID: 42, ContainerID: "126", Runtime: "lxc", Mountpoints: []string{"/"}}); err != nil || !accepted {
|
||||
t.Fatalf("exact VMID rejected: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -7,8 +7,10 @@ import (
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/filesystemprobe"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/hostmetrics"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/sensors"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/agents/filesystem"
|
||||
agentshost "github.com/rcourtman/pulse-go-rewrite/pkg/agents/host"
|
||||
gohost "github.com/shirou/gopsutil/v4/host"
|
||||
)
|
||||
@@ -44,15 +46,27 @@ type mockCollector struct {
|
||||
name string,
|
||||
arg ...string,
|
||||
) (string, error)
|
||||
lookPathFn func(file string) (string, error)
|
||||
filesystemUsageFn func(path string) (hostFilesystemUsage, error)
|
||||
lookPathFn func(file string) (string, error)
|
||||
observeLXCFn func(ctx context.Context, req filesystemprobe.ContainerRequest) ([]filesystem.Observation, error)
|
||||
readDirFn func(name string) ([]os.DirEntry, error)
|
||||
}
|
||||
|
||||
func (m *mockCollector) FilesystemUsage(path string) (hostFilesystemUsage, error) {
|
||||
if m.filesystemUsageFn != nil {
|
||||
return m.filesystemUsageFn(path)
|
||||
// ReadDir satisfies proxmoxLXCDirLister. Without readDirFn it reports the
|
||||
// directory missing, so collectors keep their pct path unless a test opts in.
|
||||
func (m *mockCollector) ReadDir(name string) ([]os.DirEntry, error) {
|
||||
if m.readDirFn != nil {
|
||||
return m.readDirFn(name)
|
||||
}
|
||||
return hostFilesystemUsage{}, os.ErrNotExist
|
||||
return nil, os.ErrNotExist
|
||||
}
|
||||
|
||||
// ObserveProxmoxLXC satisfies proxmoxLXCObserver. Without observeLXCFn the
|
||||
// probe fails, as it does off Linux, so collectors fall back to pct df.
|
||||
func (m *mockCollector) ObserveProxmoxLXC(ctx context.Context, req filesystemprobe.ContainerRequest) ([]filesystem.Observation, error) {
|
||||
if m.observeLXCFn != nil {
|
||||
return m.observeLXCFn(ctx, req)
|
||||
}
|
||||
return nil, os.ErrNotExist
|
||||
}
|
||||
|
||||
func (m *mockCollector) HostInfo(ctx context.Context) (*gohost.InfoStat, error) {
|
||||
|
||||
@@ -412,3 +412,44 @@ func requirePrivilegeHelperModuleStatus(t *testing.T, statuses []agentshost.Modu
|
||||
t.Fatalf("typed privilege helper module missing from %+v", statuses)
|
||||
return agentshost.ModuleStatus{}
|
||||
}
|
||||
|
||||
// The typed helper runs with PrivateNetwork=true, where pct and lxc-info cannot
|
||||
// reach pmxcfs or the LXC monitor: their abstract Unix sockets belong to the
|
||||
// host network namespace (#2511). This drives the collector the helper's
|
||||
// provider runs, with every command failing as it does there: it must still
|
||||
// return a complete inventory from configs, cgroups and /proc alone. A
|
||||
// degraded result is what the provider turns into provider_unavailable.
|
||||
func TestHelperProxmoxLXCFilesystemsNeedNoPctOrLXCSockets(t *testing.T) {
|
||||
h := newFakeProxmoxLXCHost()
|
||||
h.config(126, "hostname: qual2511\nrootfs: delly2-lvm:vm-126-disk-0,size=1G\nmp0: delly2-lvm:vm-126-disk-1,mp=/srv/data,size=1G\n")
|
||||
h.cgroup("126/ns", 3019253)
|
||||
h.proc(3019253, "3019253\t1", "/lxc/126/ns")
|
||||
h.config(127, "hostname: qual2511-priv\nrootfs: delly2-lvm:vm-127-disk-0,size=1G\n")
|
||||
h.cgroup("127/ns", 3021000)
|
||||
h.proc(3021000, "3021000\t1", "/lxc/127/ns")
|
||||
h.config(128, "hostname: stopped\nrootfs: delly2-lvm:vm-128-disk-0,size=1G\n")
|
||||
// Counters observed on a real PVE 9.2 node for the same layout.
|
||||
h.observations[3019253] = map[string]lxcObservation{
|
||||
"/": lxcUsage(1020702720, 12021760, 938217472),
|
||||
"/srv/data": lxcUsage(1020702720, 290816, 949948416),
|
||||
}
|
||||
h.observations[3021000] = map[string]lxcObservation{"/": lxcUsage(1020702720, 12017664, 938221568)}
|
||||
|
||||
collector, commands := h.collector(t)
|
||||
agent := &Agent{logger: zerolog.Nop(), collector: collector}
|
||||
result := agent.collectProxmoxLXCFilesystemsResult(context.Background())
|
||||
|
||||
if !result.Applicable || result.Degraded || result.FailedContainers != 0 {
|
||||
t.Fatalf("collection = applicable %v, degraded %v, failed %d; want a complete inventory", result.Applicable, result.Degraded, result.FailedContainers)
|
||||
}
|
||||
if *commands != 0 {
|
||||
t.Fatalf("collection ran %d pct/lxc-info commands; none can work inside the helper", *commands)
|
||||
}
|
||||
got := result.Inventory.Containers
|
||||
if len(got) != 2 || got[0].VMID != 126 || got[0].Name != "qual2511" || got[1].VMID != 127 || got[1].Name != "qual2511-priv" {
|
||||
t.Fatalf("containers = %+v, want the two running ones by their pct names", got)
|
||||
}
|
||||
if len(got[0].Disks) != 2 || got[0].Disks[1].Mountpoint != "/srv/data" || got[0].Disks[1].UsedBytes != 290816 {
|
||||
t.Fatalf("126 disks = %+v, want rootfs and the mp0 mount with its own usage", got[0].Disks)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -15,6 +15,7 @@ import (
|
||||
"time"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/filesystemprobe"
|
||||
agentshost "github.com/rcourtman/pulse-go-rewrite/pkg/agents/host"
|
||||
"github.com/rs/zerolog"
|
||||
)
|
||||
@@ -39,12 +40,15 @@ const (
|
||||
proxmoxLXCCollectionTimeout = 20 * time.Second
|
||||
proxmoxLXCMaxContainers = 128
|
||||
proxmoxLXCMaxDisks = 257
|
||||
proxmoxLXCMaxListOutputBytes = 64 * 1024
|
||||
proxmoxLXCMaxDFOutputBytes = 64 * 1024
|
||||
proxmoxLXCMaxConfigBytes = 64 * 1024
|
||||
proxmoxLXCMaxPIDOutputBytes = 4 * 1024
|
||||
proxmoxLXCMaxNameBytes = 256
|
||||
proxmoxLXCMaxLabelBytes = 512
|
||||
// proxmoxLXCMaxObservedMounts is the shared filesystem prober's per-request
|
||||
// mountpoint limit.
|
||||
proxmoxLXCMaxObservedMounts = 128
|
||||
proxmoxLXCMaxListOutputBytes = 64 * 1024
|
||||
proxmoxLXCMaxDFOutputBytes = 64 * 1024
|
||||
proxmoxLXCMaxConfigBytes = 64 * 1024
|
||||
proxmoxLXCMaxPIDOutputBytes = 4 * 1024
|
||||
proxmoxLXCMaxNameBytes = 256
|
||||
proxmoxLXCMaxLabelBytes = 512
|
||||
)
|
||||
|
||||
// proxmoxLXCConfigDir is a pmxcfs symlink to the local node's container
|
||||
@@ -54,6 +58,9 @@ const proxmoxLXCConfigDir = "/etc/pve/lxc"
|
||||
type proxmoxLXCRunningContainer struct {
|
||||
VMID int
|
||||
Name string
|
||||
// PID is the container's init process when discovery found it without
|
||||
// lxc-info; 0 means unknown.
|
||||
PID int
|
||||
}
|
||||
|
||||
// ProxmoxLXCFilesystemCollectionResult describes applicability separately from
|
||||
@@ -68,21 +75,6 @@ type ProxmoxLXCFilesystemCollectionResult struct {
|
||||
FailedContainers int
|
||||
}
|
||||
|
||||
// hostFilesystemUsage carries statfs-derived usage for one path plus the
|
||||
// st_dev identity needed to tell a real mount from its parent filesystem.
|
||||
type hostFilesystemUsage struct {
|
||||
TotalBytes int64
|
||||
UsedBytes int64
|
||||
AvailBytes int64
|
||||
Device uint64
|
||||
}
|
||||
|
||||
// filesystemUsageProber is an optional collector capability. It exists on
|
||||
// Linux builds of the default collector; mocks opt in for tests.
|
||||
type filesystemUsageProber interface {
|
||||
FilesystemUsage(path string) (hostFilesystemUsage, error)
|
||||
}
|
||||
|
||||
type proxmoxLXCConfigMount struct {
|
||||
Key string
|
||||
Volume string
|
||||
@@ -142,30 +134,38 @@ func (a *Agent) collectProxmoxLXCFilesystemsResult(ctx context.Context) ProxmoxL
|
||||
}
|
||||
result.Applicable = true
|
||||
|
||||
observer, _ := a.collector.(proxmoxLXCObserver)
|
||||
listCtx, cancelList := context.WithTimeout(ctx, proxmoxLXCQueryTimeout)
|
||||
listOutput, err := collectCommandOutputLimited(
|
||||
listCtx,
|
||||
a.collector,
|
||||
proxmoxLXCMaxListOutputBytes,
|
||||
pctPath,
|
||||
"list",
|
||||
)
|
||||
containers, discovered := []proxmoxLXCRunningContainer(nil), false
|
||||
if observer != nil {
|
||||
containers, discovered = a.discoverRunningProxmoxLXCContainers(listCtx)
|
||||
}
|
||||
if !discovered {
|
||||
listOutput, listErr := collectCommandOutputLimited(
|
||||
listCtx,
|
||||
a.collector,
|
||||
proxmoxLXCMaxListOutputBytes,
|
||||
pctPath,
|
||||
"list",
|
||||
)
|
||||
if listErr != nil {
|
||||
cancelList()
|
||||
a.logger.Debug().Err(listErr).Msg("Failed to list local Proxmox LXC containers")
|
||||
result.Degraded = true
|
||||
return result
|
||||
}
|
||||
containers, err = parseProxmoxLXCRunningContainers(listOutput)
|
||||
if err != nil {
|
||||
cancelList()
|
||||
a.logger.Debug().Err(err).Msg("Failed to parse local Proxmox LXC container list")
|
||||
result.Degraded = true
|
||||
return result
|
||||
}
|
||||
}
|
||||
cancelList()
|
||||
if err != nil {
|
||||
a.logger.Debug().Err(err).Msg("Failed to list local Proxmox LXC containers")
|
||||
result.Degraded = true
|
||||
return result
|
||||
}
|
||||
containers, err := parseProxmoxLXCRunningContainers(listOutput)
|
||||
if err != nil {
|
||||
a.logger.Debug().Err(err).Msg("Failed to parse local Proxmox LXC container list")
|
||||
result.Degraded = true
|
||||
return result
|
||||
}
|
||||
|
||||
prober, _ := a.collector.(filesystemUsageProber)
|
||||
lxcInfoPath := ""
|
||||
if prober != nil {
|
||||
if observer != nil && !discovered {
|
||||
if resolved, lookErr := a.collector.LookPath("lxc-info"); lookErr == nil {
|
||||
lxcInfoPath = resolved
|
||||
}
|
||||
@@ -184,7 +184,7 @@ func (a *Agent) collectProxmoxLXCFilesystemsResult(ctx context.Context) ProxmoxL
|
||||
continue
|
||||
}
|
||||
containerCtx, cancelContainer := context.WithTimeout(collectionCtx, proxmoxLXCContainerQueryTimeout)
|
||||
disks, collectionErr := a.collectProxmoxLXCContainerDisks(containerCtx, prober, lxcInfoPath, pctPath, container.VMID)
|
||||
disks, collectionErr := a.collectProxmoxLXCContainerDisks(containerCtx, observer, lxcInfoPath, pctPath, container)
|
||||
cancelContainer()
|
||||
if collectionErr != nil {
|
||||
result.FailedContainers++
|
||||
@@ -209,13 +209,14 @@ func (a *Agent) collectProxmoxLXCFilesystemsResult(ctx context.Context) ProxmoxL
|
||||
|
||||
func (a *Agent) collectProxmoxLXCContainerDisks(
|
||||
ctx context.Context,
|
||||
prober filesystemUsageProber,
|
||||
observer proxmoxLXCObserver,
|
||||
lxcInfoPath string,
|
||||
pctPath string,
|
||||
vmid int,
|
||||
container proxmoxLXCRunningContainer,
|
||||
) ([]agentshost.Disk, error) {
|
||||
if prober != nil && lxcInfoPath != "" {
|
||||
disks, fastErr := a.collectProxmoxLXCContainerDisksFast(ctx, prober, lxcInfoPath, vmid)
|
||||
vmid := container.VMID
|
||||
if observer != nil && (container.PID > 0 || lxcInfoPath != "") {
|
||||
disks, fastErr := a.collectProxmoxLXCContainerDisksFast(ctx, observer, lxcInfoPath, container)
|
||||
if fastErr == nil {
|
||||
return disks, nil
|
||||
}
|
||||
@@ -259,10 +260,11 @@ func (a *Agent) collectProxmoxLXCContainerDisks(
|
||||
// numbers — so it is dropped rather than invented.
|
||||
func (a *Agent) collectProxmoxLXCContainerDisksFast(
|
||||
ctx context.Context,
|
||||
prober filesystemUsageProber,
|
||||
observer proxmoxLXCObserver,
|
||||
lxcInfoPath string,
|
||||
vmid int,
|
||||
container proxmoxLXCRunningContainer,
|
||||
) ([]agentshost.Disk, error) {
|
||||
vmid := container.VMID
|
||||
raw, err := a.collector.ReadFile(proxmoxLXCConfigPath(vmid))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read container config: %w", err)
|
||||
@@ -275,59 +277,76 @@ func (a *Agent) collectProxmoxLXCContainerDisksFast(
|
||||
return nil, errors.New("no filesystem mounts in container config")
|
||||
}
|
||||
|
||||
pidOutput, err := collectCommandOutputLimited(
|
||||
ctx,
|
||||
a.collector,
|
||||
proxmoxLXCMaxPIDOutputBytes,
|
||||
lxcInfoPath,
|
||||
"-n",
|
||||
strconv.Itoa(vmid),
|
||||
"-p",
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("resolve container pid: %w", err)
|
||||
}
|
||||
pid, err := parseLXCInfoPID(pidOutput)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
pid := container.PID
|
||||
if pid <= 0 {
|
||||
pidOutput, pidErr := collectCommandOutputLimited(
|
||||
ctx,
|
||||
a.collector,
|
||||
proxmoxLXCMaxPIDOutputBytes,
|
||||
lxcInfoPath,
|
||||
"-n",
|
||||
strconv.Itoa(vmid),
|
||||
"-p",
|
||||
)
|
||||
if pidErr != nil {
|
||||
return nil, fmt.Errorf("resolve container pid: %w", pidErr)
|
||||
}
|
||||
pid, err = parseLXCInfoPID(pidOutput)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
procRoot := fmt.Sprintf("/proc/%d/root", pid)
|
||||
result := make([]agentshost.Disk, 0, len(mounts))
|
||||
// The shared prober pins /proc/<pid>, requires it to be this guest's init
|
||||
// in lxc/<vmid>, resolves every mount inside the guest's root (an absolute
|
||||
// symlink cannot reach the host), and bounds a read stuck on a hung guest
|
||||
// filesystem without starting another for the same guest.
|
||||
byPath := make(map[string]proxmoxLXCConfigMount, len(mounts))
|
||||
mountpoints := make([]string, 0, len(mounts))
|
||||
for _, mount := range mounts {
|
||||
if ctx.Err() != nil {
|
||||
return nil, ctx.Err()
|
||||
if _, seen := byPath[mount.Path]; seen || len(mountpoints) == proxmoxLXCMaxObservedMounts {
|
||||
continue
|
||||
}
|
||||
target := path.Join(procRoot, mount.Path)
|
||||
usage, usageErr := prober.FilesystemUsage(target)
|
||||
if usageErr != nil {
|
||||
return nil, fmt.Errorf("probe %s filesystem usage: %w", mount.Key, usageErr)
|
||||
byPath[mount.Path] = mount
|
||||
mountpoints = append(mountpoints, mount.Path)
|
||||
}
|
||||
observations, err := observer.ObserveProxmoxLXC(ctx, filesystemprobe.ContainerRequest{
|
||||
PID: pid,
|
||||
ContainerID: strconv.Itoa(vmid),
|
||||
Runtime: "lxc",
|
||||
Mountpoints: mountpoints,
|
||||
MountsOnly: true,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("observe container filesystems: %w", err)
|
||||
}
|
||||
|
||||
result := make([]agentshost.Disk, 0, len(observations))
|
||||
for _, observation := range observations {
|
||||
mount, known := byPath[observation.Mountpoint]
|
||||
if !known {
|
||||
return nil, fmt.Errorf("observation for unrequested mount %q", observation.Mountpoint)
|
||||
}
|
||||
if usage.TotalBytes <= 0 {
|
||||
if filesystemprobe.IsNotMounted(observation) {
|
||||
// Declared in the config but not mounted in the running guest; its
|
||||
// path would report the parent filesystem's numbers.
|
||||
continue
|
||||
}
|
||||
if observation.Error != "" || observation.Usage == nil {
|
||||
return nil, fmt.Errorf("probe %s filesystem usage: %s", mount.Key, observation.Error)
|
||||
}
|
||||
usage := observation.Usage
|
||||
if usage.CapacityBytes == 0 || usage.CapacityBytes > math.MaxInt64 {
|
||||
return nil, fmt.Errorf("probe %s filesystem usage: invalid total bytes", mount.Key)
|
||||
}
|
||||
if mount.Path != "/" {
|
||||
parent, parentErr := prober.FilesystemUsage(path.Dir(target))
|
||||
if parentErr != nil {
|
||||
return nil, fmt.Errorf("probe %s parent filesystem usage: %w", mount.Key, parentErr)
|
||||
}
|
||||
if parent.Device == usage.Device {
|
||||
continue
|
||||
}
|
||||
}
|
||||
used := usage.UsedBytes
|
||||
total := int64(usage.CapacityBytes)
|
||||
used := total - int64(usage.FreeBytes)
|
||||
if used < 0 {
|
||||
used = 0
|
||||
}
|
||||
if used > usage.TotalBytes {
|
||||
used = usage.TotalBytes
|
||||
}
|
||||
avail := usage.AvailBytes
|
||||
if avail < 0 {
|
||||
avail = 0
|
||||
}
|
||||
if avail > usage.TotalBytes {
|
||||
avail = usage.TotalBytes
|
||||
avail := int64(usage.AvailableBytes)
|
||||
if avail > total {
|
||||
avail = total
|
||||
}
|
||||
usagePct := 0.0
|
||||
if denominator := used + avail; denominator > 0 {
|
||||
@@ -340,14 +359,11 @@ func (a *Agent) collectProxmoxLXCContainerDisksFast(
|
||||
Device: mount.Volume,
|
||||
Mountpoint: mount.Path,
|
||||
Type: mount.Key,
|
||||
TotalBytes: usage.TotalBytes,
|
||||
TotalBytes: total,
|
||||
UsedBytes: used,
|
||||
FreeBytes: avail,
|
||||
Usage: usagePct,
|
||||
})
|
||||
if len(result) == proxmoxLXCMaxDisks {
|
||||
break
|
||||
}
|
||||
}
|
||||
if len(result) == 0 {
|
||||
return nil, errors.New("no mounted container filesystems resolved")
|
||||
|
||||
@@ -1,41 +0,0 @@
|
||||
//go:build linux
|
||||
|
||||
package hostagent
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"syscall"
|
||||
)
|
||||
|
||||
// FilesystemUsage satisfies filesystemUsageProber on Linux. Statfs through a
|
||||
// /proc/<pid>/root path traverses into that process's mount namespace, so the
|
||||
// numbers match what df reports inside the container.
|
||||
func (c *defaultCollector) FilesystemUsage(path string) (hostFilesystemUsage, error) {
|
||||
var fs syscall.Statfs_t
|
||||
if err := syscall.Statfs(path, &fs); err != nil {
|
||||
return hostFilesystemUsage{}, fmt.Errorf("statfs %s: %w", path, err)
|
||||
}
|
||||
var st syscall.Stat_t
|
||||
if err := syscall.Stat(path, &st); err != nil {
|
||||
return hostFilesystemUsage{}, fmt.Errorf("stat %s: %w", path, err)
|
||||
}
|
||||
blockSize := int64(fs.Frsize)
|
||||
if blockSize <= 0 {
|
||||
blockSize = int64(fs.Bsize)
|
||||
}
|
||||
if blockSize <= 0 {
|
||||
return hostFilesystemUsage{}, fmt.Errorf("statfs %s: invalid block size", path)
|
||||
}
|
||||
total := int64(fs.Blocks) * blockSize
|
||||
free := int64(fs.Bfree) * blockSize
|
||||
used := total - free
|
||||
if used < 0 {
|
||||
used = 0
|
||||
}
|
||||
return hostFilesystemUsage{
|
||||
TotalBytes: total,
|
||||
UsedBytes: used,
|
||||
AvailBytes: int64(fs.Bavail) * blockSize,
|
||||
Device: uint64(st.Dev),
|
||||
}, nil
|
||||
}
|
||||
@@ -4,11 +4,16 @@ import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/filesystemprobe"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/agents/filesystem"
|
||||
"github.com/rs/zerolog"
|
||||
)
|
||||
|
||||
@@ -153,16 +158,15 @@ mp2: tank:snap-only,mp=/snap-only,size=5G
|
||||
`,
|
||||
"/etc/pve/lxc/101.conf": "rootfs: local-lvm:vm-101-disk-0,size=4G\n",
|
||||
}
|
||||
usageByPath := map[string]hostFilesystemUsage{
|
||||
"/proc/4242/root": {TotalBytes: 8 << 30, UsedBytes: 2 << 30, AvailBytes: 6 << 30, Device: 11},
|
||||
"/proc/4242/root/srv": {TotalBytes: 8 << 30, UsedBytes: 2 << 30, AvailBytes: 6 << 30, Device: 11},
|
||||
"/proc/4242/root/srv/data": {TotalBytes: 1 << 40, UsedBytes: 512 << 30, AvailBytes: 512 << 30, Device: 12},
|
||||
// mp1 is configured but not mounted in the running container, so the
|
||||
// path resolves to the rootfs device and must be dropped, not reported
|
||||
// with the parent's numbers.
|
||||
"/proc/4242/root/mnt": {TotalBytes: 8 << 30, UsedBytes: 2 << 30, AvailBytes: 6 << 30, Device: 11},
|
||||
"/proc/4242/root/mnt/missing": {TotalBytes: 8 << 30, UsedBytes: 2 << 30, AvailBytes: 6 << 30, Device: 11},
|
||||
"/proc/5252/root": {TotalBytes: 4 << 30, UsedBytes: 1 << 30, AvailBytes: 3 << 30, Device: 21},
|
||||
observations := map[int]map[string]lxcObservation{
|
||||
4242: {
|
||||
"/": lxcUsage(8<<30, 2<<30, 6<<30),
|
||||
"/srv/data": lxcUsage(1<<40, 512<<30, 512<<30),
|
||||
// mp1 is configured but not mounted in the running container, so
|
||||
// the prober rejects it instead of reporting the parent's numbers.
|
||||
"/mnt/missing": {notMounted: true},
|
||||
},
|
||||
5252: {"/": lxcUsage(4<<30, 1<<30, 3<<30)},
|
||||
}
|
||||
collector := &mockCollector{
|
||||
goos: "linux",
|
||||
@@ -182,13 +186,7 @@ mp2: tank:snap-only,mp=/snap-only,size=5G
|
||||
}
|
||||
return []byte(config), nil
|
||||
},
|
||||
filesystemUsageFn: func(path string) (hostFilesystemUsage, error) {
|
||||
usage, ok := usageByPath[path]
|
||||
if !ok {
|
||||
t.Fatalf("unexpected statfs path %q", path)
|
||||
}
|
||||
return usage, nil
|
||||
},
|
||||
observeLXCFn: fakeLXCObserve(t, observations),
|
||||
commandCombinedOutputLimitedFn: func(
|
||||
_ context.Context,
|
||||
_ int,
|
||||
@@ -267,12 +265,9 @@ func TestCollectProxmoxLXCFilesystemsFallsBackToPctDFPerContainer(t *testing.T)
|
||||
// back to pct df.
|
||||
return nil, os.ErrPermission
|
||||
},
|
||||
filesystemUsageFn: func(path string) (hostFilesystemUsage, error) {
|
||||
if path != "/proc/4242/root" {
|
||||
t.Fatalf("unexpected statfs path %q", path)
|
||||
}
|
||||
return hostFilesystemUsage{TotalBytes: 8 << 30, UsedBytes: 2 << 30, AvailBytes: 6 << 30, Device: 11}, nil
|
||||
},
|
||||
observeLXCFn: fakeLXCObserve(t, map[int]map[string]lxcObservation{
|
||||
4242: {"/": lxcUsage(8<<30, 2<<30, 6<<30)},
|
||||
}),
|
||||
commandCombinedOutputLimitedFn: func(
|
||||
_ context.Context,
|
||||
_ int,
|
||||
@@ -345,17 +340,12 @@ mp0 tank:subvol-100-disk-0 1.0T 512.0G 512.0G 50.0 /srv/data
|
||||
readFileFn: func(string) ([]byte, error) {
|
||||
return []byte("rootfs: local-lvm:vm-100-disk-0,size=8G\nmp0: tank:subvol-100-disk-0,mp=/srv/data,size=1T\n"), nil
|
||||
},
|
||||
filesystemUsageFn: func(target string) (hostFilesystemUsage, error) {
|
||||
switch target {
|
||||
case "/proc/4242/root":
|
||||
return hostFilesystemUsage{TotalBytes: 8 << 30, UsedBytes: 2 << 30, AvailBytes: 6 << 30, Device: 11}, nil
|
||||
case "/proc/4242/root/srv/data":
|
||||
return hostFilesystemUsage{}, errors.New("statfs access failed")
|
||||
default:
|
||||
t.Fatalf("unexpected statfs path %q", target)
|
||||
return hostFilesystemUsage{}, nil
|
||||
}
|
||||
},
|
||||
observeLXCFn: fakeLXCObserve(t, map[int]map[string]lxcObservation{
|
||||
4242: {
|
||||
"/": lxcUsage(8<<30, 2<<30, 6<<30),
|
||||
"/srv/data": {err: "read filesystem counters: permission denied"},
|
||||
},
|
||||
}),
|
||||
commandCombinedOutputLimitedFn: func(_ context.Context, _ int, name string, args ...string) (string, error) {
|
||||
joined := strings.Join(args, " ")
|
||||
switch {
|
||||
@@ -469,3 +459,281 @@ mp5 pool:hot 10.0G 1.0G 9.0G 101.0 /hot
|
||||
t.Fatal("expected oversized pct df error")
|
||||
}
|
||||
}
|
||||
|
||||
type fakeLXCDirEntry struct {
|
||||
name string
|
||||
dir bool
|
||||
}
|
||||
|
||||
func (e fakeLXCDirEntry) Name() string { return e.name }
|
||||
func (e fakeLXCDirEntry) IsDir() bool { return e.dir }
|
||||
func (e fakeLXCDirEntry) Type() fs.FileMode {
|
||||
if e.dir {
|
||||
return fs.ModeDir
|
||||
}
|
||||
return 0
|
||||
}
|
||||
func (e fakeLXCDirEntry) Info() (fs.FileInfo, error) { return nil, fs.ErrNotExist }
|
||||
|
||||
// fakeProxmoxLXCHost models what socket-free discovery reads on a cgroup v2
|
||||
// Proxmox node. The layout follows a real PVE 9.2 node: configs under
|
||||
// /etc/pve/lxc, the payload under /sys/fs/cgroup/lxc/<vmid>/ns, and the
|
||||
// container init identified by an NSpid ending in 1.
|
||||
type fakeProxmoxLXCHost struct {
|
||||
files map[string]string
|
||||
dirs map[string]map[string]bool
|
||||
observations map[int]map[string]lxcObservation
|
||||
}
|
||||
|
||||
func newFakeProxmoxLXCHost() *fakeProxmoxLXCHost {
|
||||
h := &fakeProxmoxLXCHost{
|
||||
files: map[string]string{proxmoxLXCCgroupV2Marker: "cpuset cpu io memory pids\n"},
|
||||
dirs: map[string]map[string]bool{},
|
||||
observations: map[int]map[string]lxcObservation{},
|
||||
}
|
||||
h.dir(proxmoxLXCConfigDir)
|
||||
return h
|
||||
}
|
||||
|
||||
func (h *fakeProxmoxLXCHost) dir(name string) {
|
||||
if _, ok := h.dirs[name]; !ok {
|
||||
h.dirs[name] = map[string]bool{}
|
||||
}
|
||||
}
|
||||
|
||||
func (h *fakeProxmoxLXCHost) config(vmid int, content string) {
|
||||
h.dirs[proxmoxLXCConfigDir][strconv.Itoa(vmid)+".conf"] = false
|
||||
h.files[proxmoxLXCConfigPath(vmid)] = content
|
||||
}
|
||||
|
||||
// cgroup registers a cgroup directory (relative to lxc/) and its processes.
|
||||
func (h *fakeProxmoxLXCHost) cgroup(rel string, pids ...int) {
|
||||
full := path.Join(proxmoxLXCCgroupRoot, rel)
|
||||
for child := full; child != proxmoxLXCCgroupRoot; child = path.Dir(child) {
|
||||
h.dir(child)
|
||||
// Every cgroup directory has a cgroup.procs, empty when its
|
||||
// processes live in child cgroups.
|
||||
if _, ok := h.files[path.Join(child, "cgroup.procs")]; !ok {
|
||||
h.files[path.Join(child, "cgroup.procs")] = ""
|
||||
}
|
||||
parent := path.Dir(child)
|
||||
h.dir(parent)
|
||||
h.dirs[parent][path.Base(child)] = true
|
||||
}
|
||||
fields := make([]string, 0, len(pids))
|
||||
for _, pid := range pids {
|
||||
fields = append(fields, strconv.Itoa(pid))
|
||||
}
|
||||
h.files[path.Join(full, "cgroup.procs")] = strings.Join(fields, "\n") + "\n"
|
||||
}
|
||||
|
||||
func (h *fakeProxmoxLXCHost) proc(pid int, nspid, cgroup string) {
|
||||
h.files["/proc/"+strconv.Itoa(pid)+"/status"] = "Name:\tinit\nNSpid:\t" + nspid + "\n"
|
||||
h.files["/proc/"+strconv.Itoa(pid)+"/cgroup"] = "0::" + cgroup + "\n"
|
||||
}
|
||||
|
||||
// collector serves the fake host. pct and lxc-info exist but every call fails
|
||||
// the way they fail inside the PrivateNetwork helper.
|
||||
func (h *fakeProxmoxLXCHost) collector(t *testing.T) (*mockCollector, *int) {
|
||||
t.Helper()
|
||||
commands := 0
|
||||
return &mockCollector{
|
||||
goos: "linux",
|
||||
lookPathFn: func(file string) (string, error) {
|
||||
switch file {
|
||||
case "pct":
|
||||
return "/usr/sbin/pct", nil
|
||||
case "lxc-info":
|
||||
return "/usr/bin/lxc-info", nil
|
||||
}
|
||||
return "", os.ErrNotExist
|
||||
},
|
||||
statFn: func(name string) (os.FileInfo, error) {
|
||||
if _, ok := h.files[name]; ok {
|
||||
return nil, nil
|
||||
}
|
||||
if _, ok := h.dirs[name]; ok {
|
||||
return nil, nil
|
||||
}
|
||||
return nil, os.ErrNotExist
|
||||
},
|
||||
readFileFn: func(name string) ([]byte, error) {
|
||||
content, ok := h.files[name]
|
||||
if !ok {
|
||||
return nil, os.ErrNotExist
|
||||
}
|
||||
return []byte(content), nil
|
||||
},
|
||||
readDirFn: func(name string) ([]os.DirEntry, error) {
|
||||
children, ok := h.dirs[name]
|
||||
if !ok {
|
||||
return nil, os.ErrNotExist
|
||||
}
|
||||
entries := make([]os.DirEntry, 0, len(children))
|
||||
for child, isDir := range children {
|
||||
entries = append(entries, fakeLXCDirEntry{name: child, dir: isDir})
|
||||
}
|
||||
return entries, nil
|
||||
},
|
||||
observeLXCFn: fakeLXCObserve(t, h.observations),
|
||||
commandCombinedOutputLimitedFn: func(_ context.Context, _ int, name string, args ...string) (string, error) {
|
||||
commands++
|
||||
return "ipcc_send_rec[1] failed: Connection refused", errors.New("exit status 2")
|
||||
},
|
||||
}, &commands
|
||||
}
|
||||
|
||||
func TestProxmoxLXCConfigNameMatchesPctList(t *testing.T) {
|
||||
cases := []struct {
|
||||
config string
|
||||
want string
|
||||
}{
|
||||
{"#managed\narch: amd64\nhostname: web\nrootfs: x,size=1G\n", "web"},
|
||||
{"arch: amd64\nrootfs: x,size=1G\n", "CT126"},
|
||||
{"hostname: \n", "CT126"},
|
||||
{"arch: amd64\n\n[snap]\nhostname: old\n", "CT126"},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
if got := proxmoxLXCConfigName(tc.config, 126); got != tc.want {
|
||||
t.Errorf("proxmoxLXCConfigName(%q) = %q, want %q", tc.config, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiscoverRunningProxmoxLXCContainersFindsInitWithoutSockets(t *testing.T) {
|
||||
h := newFakeProxmoxLXCHost()
|
||||
// Alpine guest running Docker: a nested container's PID 1 shares the
|
||||
// guest's cgroup listing and comes first, but is not the guest's init.
|
||||
h.config(126, "hostname: qual2511\nrootfs: lvm:vm-126-disk-0,size=1G\n")
|
||||
h.cgroup("126")
|
||||
h.cgroup("126/ns", 3019900, 3019673, 3019253)
|
||||
h.cgroup("126/ns/openrc.syslog", 3019638)
|
||||
h.proc(3019900, "3019900\t420\t1", "/lxc/126/ns")
|
||||
h.proc(3019673, "3019673\t413", "/lxc/126/ns")
|
||||
h.proc(3019253, "3019253\t1", "/lxc/126/ns")
|
||||
// systemd guest: init in ns/init.scope, services elsewhere.
|
||||
h.config(127, "rootfs: lvm:vm-127-disk-0,size=1G\n")
|
||||
h.cgroup("127/ns/system.slice/cron.service", 4100)
|
||||
h.cgroup("127/ns/init.scope", 4000)
|
||||
h.proc(4100, "4100\t88", "/lxc/127/ns/system.slice/cron.service")
|
||||
h.proc(4000, "4000\t1", "/lxc/127/ns/init.scope")
|
||||
// Stopped: config only, no cgroup.
|
||||
h.config(128, "hostname: stopped\nrootfs: lvm:vm-128-disk-0,size=1G\n")
|
||||
|
||||
collector, commands := h.collector(t)
|
||||
agent := &Agent{logger: zerolog.Nop(), collector: collector}
|
||||
got, ok := agent.discoverRunningProxmoxLXCContainers(context.Background())
|
||||
if !ok {
|
||||
t.Fatal("discovery was not applicable on a cgroup v2 Proxmox host")
|
||||
}
|
||||
want := []proxmoxLXCRunningContainer{
|
||||
{VMID: 126, Name: "qual2511", PID: 3019253},
|
||||
{VMID: 127, Name: "CT127", PID: 4000},
|
||||
}
|
||||
if fmt.Sprint(got) != fmt.Sprint(want) {
|
||||
t.Fatalf("discovered %+v, want %+v", got, want)
|
||||
}
|
||||
if *commands != 0 {
|
||||
t.Fatalf("discovery ran %d commands, want none", *commands)
|
||||
}
|
||||
}
|
||||
|
||||
// Discovery must never turn a running guest it could not read into a smaller,
|
||||
// healthy-looking inventory. Whenever a guest's state is not established it
|
||||
// hands the whole node back to pct, which works for root installs and is
|
||||
// reported as degraded inside the helper.
|
||||
func TestDiscoverRunningProxmoxLXCContainersLeavesUnestablishedStateToPct(t *testing.T) {
|
||||
running := func() *fakeProxmoxLXCHost {
|
||||
h := newFakeProxmoxLXCHost()
|
||||
h.config(126, "hostname: web\n")
|
||||
h.cgroup("126/ns", 10)
|
||||
h.proc(10, "10\t1", "/lxc/126/ns")
|
||||
return h
|
||||
}
|
||||
cases := map[string]func(h *fakeProxmoxLXCHost) context.Context{
|
||||
"cgroup v1 host": func(h *fakeProxmoxLXCHost) context.Context {
|
||||
delete(h.files, proxmoxLXCCgroupV2Marker)
|
||||
return context.Background()
|
||||
},
|
||||
"unreadable config of a running guest": func(h *fakeProxmoxLXCHost) context.Context {
|
||||
delete(h.files, proxmoxLXCConfigPath(126))
|
||||
return context.Background()
|
||||
},
|
||||
"guest cgroup without an init (a reused PID from another guest)": func(h *fakeProxmoxLXCHost) context.Context {
|
||||
h.proc(10, "10\t1", "/lxc/130/ns")
|
||||
return context.Background()
|
||||
},
|
||||
"unreadable cgroup.procs": func(h *fakeProxmoxLXCHost) context.Context {
|
||||
delete(h.files, path.Join(proxmoxLXCCgroupRoot, "126/ns/cgroup.procs"))
|
||||
return context.Background()
|
||||
},
|
||||
"guest-created cgroup tree beyond the search limit": func(h *fakeProxmoxLXCHost) context.Context {
|
||||
h.proc(10, "10\t2", "/lxc/126/ns")
|
||||
for i := 0; i < proxmoxLXCMaxCgroupDirs; i++ {
|
||||
h.cgroup("126/ns/c" + strconv.Itoa(i))
|
||||
}
|
||||
return context.Background()
|
||||
},
|
||||
"cancelled collection": func(*fakeProxmoxLXCHost) context.Context {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
return ctx
|
||||
},
|
||||
}
|
||||
for name, mutate := range cases {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
h := running()
|
||||
ctx := mutate(h)
|
||||
collector, _ := h.collector(t)
|
||||
got, ok := (&Agent{logger: zerolog.Nop(), collector: collector}).discoverRunningProxmoxLXCContainers(ctx)
|
||||
if ok {
|
||||
t.Fatalf("discovery reported %+v as complete", got)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// lxcObservation is one mount's answer from the fake filesystem prober.
|
||||
type lxcObservation struct {
|
||||
usage *filesystem.Usage
|
||||
notMounted bool
|
||||
err string
|
||||
}
|
||||
|
||||
func lxcUsage(total, used, avail uint64) lxcObservation {
|
||||
return lxcObservation{usage: &filesystem.Usage{CapacityBytes: total, FreeBytes: total - used, AvailableBytes: avail}}
|
||||
}
|
||||
|
||||
// fakeLXCObserve answers prober requests from per-PID mount tables. A PID the
|
||||
// table does not know is not the guest's init, as the real prober would find.
|
||||
func fakeLXCObserve(t *testing.T, byPID map[int]map[string]lxcObservation) func(context.Context, filesystemprobe.ContainerRequest) ([]filesystem.Observation, error) {
|
||||
t.Helper()
|
||||
return func(_ context.Context, req filesystemprobe.ContainerRequest) ([]filesystem.Observation, error) {
|
||||
if req.Runtime != "lxc" || !req.MountsOnly || req.ContainerID == "" {
|
||||
t.Errorf("prober request = %+v, want an exact lxc request with MountsOnly", req)
|
||||
}
|
||||
mounts, ok := byPID[req.PID]
|
||||
if !ok {
|
||||
return nil, errors.New("process is not the container's init")
|
||||
}
|
||||
out := make([]filesystem.Observation, 0, len(req.Mountpoints))
|
||||
for _, mountpoint := range req.Mountpoints {
|
||||
answer, known := mounts[mountpoint]
|
||||
if !known {
|
||||
t.Errorf("unexpected mount %q for pid %d", mountpoint, req.PID)
|
||||
continue
|
||||
}
|
||||
observation := filesystem.Observation{Mountpoint: mountpoint, Source: filesystemprobe.Source}
|
||||
switch {
|
||||
case answer.notMounted:
|
||||
observation.Error = filesystemprobe.ErrNotMounted.Error()
|
||||
case answer.err != "":
|
||||
observation.Error = answer.err
|
||||
default:
|
||||
observation.Usage = answer.usage
|
||||
}
|
||||
out = append(out, observation)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,243 @@
|
||||
package hostagent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/filesystemprobe"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/agents/filesystem"
|
||||
)
|
||||
|
||||
// Socket-free discovery of the node's running containers (#2511).
|
||||
//
|
||||
// pct list, pct df and lxc-info reach pmxcfs and the LXC monitor through
|
||||
// abstract Unix sockets, which belong to a network namespace. The typed helper
|
||||
// runs with PrivateNetwork=true, so every one of them fails there. Container
|
||||
// configs are ordinary reads through the pmxcfs FUSE mount, and a running
|
||||
// container's init process is visible in its cgroup and in /proc, so the helper
|
||||
// can find what it needs without leaving its sandbox. Usage is then read by
|
||||
// the shared filesystem prober, which pins the process, re-checks its identity
|
||||
// and resolves every path inside the container's root.
|
||||
|
||||
const (
|
||||
// proxmoxLXCCgroupRoot is where pve-container places each container's
|
||||
// cgroup (lxc/<vmid>, with the payload under ns/) on a cgroup v2 host.
|
||||
proxmoxLXCCgroupRoot = "/sys/fs/cgroup/lxc"
|
||||
proxmoxLXCCgroupV2Marker = "/sys/fs/cgroup/cgroup.controllers"
|
||||
proxmoxLXCMaxCgroupDirs = 64
|
||||
proxmoxLXCMaxCgroupPIDs = 4096
|
||||
proxmoxLXCMaxCgroupProcsBytes = 64 * 1024
|
||||
proxmoxLXCMaxProcStatusBytes = 64 * 1024
|
||||
proxmoxLXCMaxProcCgroupBytes = 64 * 1024
|
||||
proxmoxLXCMaxConfigEntries = 4096
|
||||
)
|
||||
|
||||
// proxmoxLXCDirLister is an optional collector capability, like
|
||||
// proxmoxLXCObserver: the default collector lists directories and test
|
||||
// collectors opt in.
|
||||
type proxmoxLXCDirLister interface {
|
||||
ReadDir(name string) ([]os.DirEntry, error)
|
||||
}
|
||||
|
||||
// proxmoxLXCObserver reads a running container's filesystem counters through
|
||||
// its pinned init process.
|
||||
type proxmoxLXCObserver interface {
|
||||
ObserveProxmoxLXC(ctx context.Context, req filesystemprobe.ContainerRequest) ([]filesystem.Observation, error)
|
||||
}
|
||||
|
||||
// proxmoxLXCFilesystemObserver is shared by every collection in the process.
|
||||
// A probe stuck on a guest's hung filesystem keeps its slot here, so the
|
||||
// typed helper never stacks a second probe on that guest.
|
||||
var proxmoxLXCFilesystemObserver filesystemprobe.Observer
|
||||
|
||||
func (c *defaultCollector) ReadDir(name string) ([]os.DirEntry, error) {
|
||||
return os.ReadDir(name)
|
||||
}
|
||||
|
||||
func (c *defaultCollector) ObserveProxmoxLXC(ctx context.Context, req filesystemprobe.ContainerRequest) ([]filesystem.Observation, error) {
|
||||
return proxmoxLXCFilesystemObserver.Observe(ctx, req)
|
||||
}
|
||||
|
||||
type proxmoxLXCInitState int
|
||||
|
||||
const (
|
||||
proxmoxLXCStopped proxmoxLXCInitState = iota
|
||||
proxmoxLXCRunning
|
||||
proxmoxLXCUnknown
|
||||
)
|
||||
|
||||
// discoverRunningProxmoxLXCContainers lists this node's running containers
|
||||
// with their init PIDs. ok is false when the host cannot be read this way or
|
||||
// any container's state cannot be established (no directory listing, no
|
||||
// cgroup v2 hierarchy, an unreadable config or cgroup, a search limit, or
|
||||
// cancellation), so the caller uses pct instead of reporting an inventory that
|
||||
// silently misses a running guest. Only a container with no cgroup at all is
|
||||
// stopped and left out, as pct list leaves it out.
|
||||
func (a *Agent) discoverRunningProxmoxLXCContainers(ctx context.Context) ([]proxmoxLXCRunningContainer, bool) {
|
||||
lister, isLister := a.collector.(proxmoxLXCDirLister)
|
||||
if !isLister {
|
||||
return nil, false
|
||||
}
|
||||
if _, err := a.collector.Stat(proxmoxLXCCgroupV2Marker); err != nil {
|
||||
return nil, false
|
||||
}
|
||||
entries, err := lister.ReadDir(proxmoxLXCConfigDir)
|
||||
if err != nil {
|
||||
a.logger.Debug().Err(err).Msg("Cannot list Proxmox LXC configs; using pct")
|
||||
return nil, false
|
||||
}
|
||||
if len(entries) > proxmoxLXCMaxConfigEntries {
|
||||
return nil, false
|
||||
}
|
||||
|
||||
vmids := make([]int, 0, len(entries))
|
||||
for _, entry := range entries {
|
||||
name := entry.Name()
|
||||
if !strings.HasSuffix(name, ".conf") {
|
||||
continue
|
||||
}
|
||||
vmid, convErr := strconv.Atoi(strings.TrimSuffix(name, ".conf"))
|
||||
if convErr != nil || vmid < 100 || vmid > 999999999 {
|
||||
continue
|
||||
}
|
||||
vmids = append(vmids, vmid)
|
||||
}
|
||||
sort.Ints(vmids)
|
||||
|
||||
running := make([]proxmoxLXCRunningContainer, 0, len(vmids))
|
||||
for _, vmid := range vmids {
|
||||
if ctx.Err() != nil {
|
||||
return nil, false
|
||||
}
|
||||
pid, state := a.proxmoxLXCInitPID(ctx, lister, vmid)
|
||||
switch state {
|
||||
case proxmoxLXCStopped:
|
||||
continue
|
||||
case proxmoxLXCUnknown:
|
||||
a.logger.Debug().Int("vmid", vmid).Msg("Cannot establish a Proxmox LXC init process; using pct")
|
||||
return nil, false
|
||||
}
|
||||
raw, readErr := a.collector.ReadFile(proxmoxLXCConfigPath(vmid))
|
||||
if readErr != nil || len(raw) > proxmoxLXCMaxConfigBytes {
|
||||
// Without its config a running container has no trustworthy name or
|
||||
// mounts; leave the whole node to pct rather than guess.
|
||||
a.logger.Debug().Err(readErr).Int("vmid", vmid).Msg("Cannot read a running Proxmox LXC config; using pct")
|
||||
return nil, false
|
||||
}
|
||||
running = append(running, proxmoxLXCRunningContainer{VMID: vmid, Name: proxmoxLXCConfigName(string(raw), vmid), PID: pid})
|
||||
if len(running) == proxmoxLXCMaxContainers {
|
||||
break
|
||||
}
|
||||
}
|
||||
return running, true
|
||||
}
|
||||
|
||||
// proxmoxLXCInitPID finds the container's init process: the process in the
|
||||
// container's cgroup tree that is PID 1 of the namespace directly below the
|
||||
// host's and whose own cgroup is inside lxc/<vmid>. The filesystem prober
|
||||
// re-checks both on the pinned process before reading anything.
|
||||
func (a *Agent) proxmoxLXCInitPID(ctx context.Context, lister proxmoxLXCDirLister, vmid int) (int, proxmoxLXCInitState) {
|
||||
root := path.Join(proxmoxLXCCgroupRoot, strconv.Itoa(vmid))
|
||||
if _, err := lister.ReadDir(root); err != nil {
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return 0, proxmoxLXCStopped
|
||||
}
|
||||
return 0, proxmoxLXCUnknown
|
||||
}
|
||||
|
||||
queue := []string{root}
|
||||
queued := 1
|
||||
checked := 0
|
||||
for len(queue) > 0 {
|
||||
if ctx.Err() != nil {
|
||||
return 0, proxmoxLXCUnknown
|
||||
}
|
||||
dir := queue[0]
|
||||
queue = queue[1:]
|
||||
|
||||
raw, err := a.collector.ReadFile(path.Join(dir, "cgroup.procs"))
|
||||
if err != nil || len(raw) > proxmoxLXCMaxCgroupProcsBytes {
|
||||
return 0, proxmoxLXCUnknown
|
||||
}
|
||||
for _, field := range strings.Fields(string(raw)) {
|
||||
if checked == proxmoxLXCMaxCgroupPIDs {
|
||||
return 0, proxmoxLXCUnknown
|
||||
}
|
||||
checked++
|
||||
pid, convErr := strconv.Atoi(field)
|
||||
if convErr != nil || pid <= 0 {
|
||||
continue
|
||||
}
|
||||
if a.isProxmoxLXCInit(pid, vmid) {
|
||||
return pid, proxmoxLXCRunning
|
||||
}
|
||||
}
|
||||
|
||||
entries, err := lister.ReadDir(dir)
|
||||
if err != nil {
|
||||
return 0, proxmoxLXCUnknown
|
||||
}
|
||||
children := make([]string, 0, len(entries))
|
||||
for _, entry := range entries {
|
||||
if entry.IsDir() {
|
||||
children = append(children, entry.Name())
|
||||
}
|
||||
}
|
||||
// systemd guests run init in init.scope; look there first.
|
||||
sort.SliceStable(children, func(i, j int) bool {
|
||||
return children[i] == "init.scope" && children[j] != "init.scope"
|
||||
})
|
||||
for _, child := range children {
|
||||
if queued == proxmoxLXCMaxCgroupDirs {
|
||||
// A guest can create cgroups in its delegated subtree; give up
|
||||
// rather than walk an unbounded tree.
|
||||
return 0, proxmoxLXCUnknown
|
||||
}
|
||||
queued++
|
||||
queue = append(queue, path.Join(dir, child))
|
||||
}
|
||||
}
|
||||
// A cgroup without an init is a guest starting or stopping.
|
||||
return 0, proxmoxLXCUnknown
|
||||
}
|
||||
|
||||
func (a *Agent) isProxmoxLXCInit(pid, vmid int) bool {
|
||||
status, err := a.collector.ReadFile("/proc/" + strconv.Itoa(pid) + "/status")
|
||||
if err != nil || len(status) > proxmoxLXCMaxProcStatusBytes {
|
||||
return false
|
||||
}
|
||||
if !filesystemprobe.IsNamespaceInit(string(status), pid) {
|
||||
return false
|
||||
}
|
||||
cgroup, err := a.collector.ReadFile("/proc/" + strconv.Itoa(pid) + "/cgroup")
|
||||
if err != nil || len(cgroup) > proxmoxLXCMaxProcCgroupBytes {
|
||||
return false
|
||||
}
|
||||
return filesystemprobe.CgroupMatches(string(cgroup), "lxc", strconv.Itoa(vmid))
|
||||
}
|
||||
|
||||
// proxmoxLXCConfigName is the name pct list and the Proxmox API report: the
|
||||
// configured hostname, or CT<vmid> when none is set. Pulse matches agent
|
||||
// readings to API containers by this name.
|
||||
func proxmoxLXCConfigName(config string, vmid int) string {
|
||||
for _, rawLine := range strings.Split(config, "\n") {
|
||||
line := strings.TrimSpace(rawLine)
|
||||
if strings.HasPrefix(line, "[") {
|
||||
break
|
||||
}
|
||||
value, ok := strings.CutPrefix(line, "hostname:")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if name := strings.TrimSpace(value); name != "" && safeProxmoxLXCText(name, proxmoxLXCMaxNameBytes) {
|
||||
return name
|
||||
}
|
||||
break
|
||||
}
|
||||
return "CT" + strconv.Itoa(vmid)
|
||||
}
|
||||
Reference in New Issue
Block a user