mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-10-24 04:33:17 +00:00
Give Docker containers and services without IDs their own alert refs (#2629)
Follow-up to #2620, which documented both conflations below as residuals. ## Problem Two Docker alert producers wrote names into identity slots: - A container reported without an ID alerted under its Docker host's own reference, `docker:<host>`. Its alerts showed on the host, joined the host's resource history (which #2620 binds to the host), were never cleared when the container left the report (cleanup matches `docker:<host>/`), and Patrol registered `docker:<host>` on both the host and the container, so the host's scope alias was ambiguous. - A Swarm service without an ID put its normalized name in the `/service/<ID>` slot, so a name equal to another service's Swarm ID shared that service's alerts and history. #2620's resolver had to treat every service in such a cluster as a conflict. Pulse's own agent always sends Docker and Swarm IDs, so this needs a malformed or third-party report. The registry still shows those containers and services. ## Change - `internal/alerts`: `DockerHostResourceID`, `DockerContainerResourceID(host, id, name)` and `DockerServiceResourceID` are the only builders. An ID-less container alerts under `docker:<host>/name:<name>` and an ID-less service under `docker:<host>/service/name:<normalized name>`. Docker and Swarm IDs never contain a colon, so neither can equal a host or ID reference. One with neither an ID nor a name raises no alert. - The container and service policy switches classify by the first path segment after the host, so a container name containing `/service/` stays a container. - Report cleanup, host removal and the host's offline alert also clear a pre-upgrade container alert left on exactly `docker:<host>` (recorded host must match, container ID empty). A pending image update recorded there keeps its age when the first report still shows it pending. - Canonical state IDs were split at the first `::`, but colliding Docker hosts get IDs like `<base>::<suffix>`. Every container alert of such a host was rewritten to `docker:<base>`, the other host's reference: it joined that host's history and missed its own host's cleanup. `splitCanonicalStateID` now splits before the alert's recorded spec ID, else after its resource ID, so alerts saved the old way also recover on restore. - Patrol registers the same builders' output as scope aliases, with the container's name. - History: a service slot containing `:` is a name reference (never binds, never retried), and the cluster-wide conflict for ID-less services is gone. A real service ID reference now binds even when its cluster also has an ID-less service. - Contract notes in alerts, ai-runtime, monitoring and unified-resources. Remaining legacy boundary (in the unified-resources contract): rows journaled before this change keep their old reference. An ID-less container's old rows stay in its host's history, and an ID-less service's old rows bind only to a service whose ID equals its normalized name. ## Proof - `TestDockerAlertsWithoutIDsUseTheirOwnReferences` (real `CheckDockerHost`: fired and active references, removal cleanup, pre-upgrade leftover on report and offline, config-save container/service switches). - `TestDockerHostIDWithSeparatorKeepsItsAlertReferences`, `TestDockerAlertsRestoredWithCutShortReferencesRecover` (writes `active-alerts.json`, restores through `LoadActiveAlerts`) and `TestDockerContainerWithoutIDKeepsPendingUpdateAgeAcrossUpgrade`. - `TestOwnerAlertTimelinesUseCanonicalHistoryIdentity`: real producers into the SQLite history. The ID-less container and the service named like `web`'s Swarm ID stay out of the host's and `web`'s histories. - `TestResolvePatrolScopeResolvesDockerAlertResourceIDs`: the host's alias stays unambiguous, and the ID-less container's name reference resolves to the container in the unified read state. - `TestHistoryIdentityMonitorAdapterResolvesDockerHostAndServiceReferences`: the ID reference binds before and after the service leaves inventory, and name references never bind or retry. - Mutation check: putting back each old behavior fails at least one of these tests. Covered: host fallback, unmarked service name, no skip for unidentifiable containers or services, each legacy clause and its host check, the update-age carry-over, first-`::` splitting with and without the spec rule, substring classification, the Patrol alias without the name, and the resolver accepting a `name:` slot.
This commit is contained in:
@@ -6543,6 +6543,20 @@ Patrol scope payloads. API-backed TrueNAS systems may still keep `truenas`
|
||||
platform metadata and separate run-history coverage counts, but AI resource
|
||||
type fields must normalize to canonical `agent` once they cross the governed
|
||||
runtime boundary.
|
||||
Patrol scope resolution registers the alert subsystem's Docker references as
|
||||
aliases, built by the same `internal/alerts` builders rather than formatted
|
||||
locally: `DockerHostResourceID` on the Docker host record, and on each
|
||||
container record `dockerAlertScopeAlias`, which passes the container's name so
|
||||
a container reported without an ID answers to its marked name reference
|
||||
(`docker:<host ID>/name:<name>`) and never to its host's reference.
|
||||
`dockerServiceAlertScopeAliases` registers service references, including
|
||||
`docker:<host ID>/service/name:<name>` for a service without an ID, on the
|
||||
owning host. The legacy snapshot record of a container without an ID has no ID
|
||||
to scope by, so only the unified read state can scope its alerts. Proof:
|
||||
`TestResolvePatrolScopeResolvesDockerAlertResourceIDs` in
|
||||
`internal/ai/patrol_run_test.go` and
|
||||
`TestDockerServiceAlertScopeAliasesDeriveIDFromName` in
|
||||
`internal/ai/patrol_state_scope_alias_test.go`.
|
||||
The same governed-context rule also applies to the main unified AI resource
|
||||
overview: infrastructure, workload, alert-label, and top-consumer summaries
|
||||
must not leak raw resource names, cluster labels, IP addresses, or unresolved
|
||||
|
||||
@@ -1477,6 +1477,45 @@ short-ID, unified-hash, and slash-tail forms as trailing lookup candidates).
|
||||
Container override work must not reintroduce a runtime-container-ID
|
||||
persistence key.
|
||||
|
||||
Docker alert references come only from the builders in
|
||||
`internal/alerts/docker.go`: a host's own alert `docker:<host ID>`
|
||||
(`DockerHostResourceID`, used only by its connectivity alert), a container
|
||||
`docker:<host ID>/<container ID>` (`DockerContainerResourceID`), and a Swarm
|
||||
service `docker:<host ID>/service/<service ID>` (`DockerServiceResourceID`).
|
||||
A container or service reported without an ID alerts under its name, marked in
|
||||
the ID position (`docker:<host ID>/name:<name>`,
|
||||
`docker:<host ID>/service/name:<normalized name>`). Docker and Swarm IDs never
|
||||
contain a colon, so a name reference never takes its host's reference or
|
||||
another service's ID reference, and one with neither an ID nor a name raises
|
||||
no alert. Report cleanup, host removal and the host's offline alert firing
|
||||
clear children by the `docker:<host ID>/` prefix; the container and service
|
||||
policy switches classify by the first path segment after the host
|
||||
(`dockerAlertResourcePath`), so a container name containing `/service/` stays
|
||||
a container; and Patrol's scope aliases and resource history parse the same
|
||||
forms. Service overrides key on the same reference, so an override written for
|
||||
a service without an ID under its old unmarked name no longer applies. A
|
||||
reference can contain the canonical state separator (a Docker host
|
||||
disambiguated as `<base>::<suffix>`), so `splitCanonicalStateID` splits a
|
||||
state ID before the alert's recorded spec ID, or else after its resource ID,
|
||||
and only without either at the first `::`. Splitting at the first `::` gave
|
||||
such a host's alerts the reference of the host named `<base>`; an alert saved
|
||||
that way recovers its reference on restore because its spec ID is intact.
|
||||
Before name references existed, a container without an ID alerted under its
|
||||
host's reference. Only the connectivity alert belongs there, so the host's
|
||||
next report, or its offline alert firing, clears any such alert
|
||||
left active whose recorded host matches and whose container ID is empty. A
|
||||
pending image update recorded there keeps its age under the container's new
|
||||
reference when that first report still shows the update pending; a first
|
||||
report with missing or failed update evidence clears it, and the delay starts
|
||||
again on the next positive report. Proof:
|
||||
`TestDockerAlertsWithoutIDsUseTheirOwnReferences`,
|
||||
`TestDockerHostIDWithSeparatorKeepsItsAlertReferences`,
|
||||
`TestDockerAlertsRestoredWithCutShortReferencesRecover`,
|
||||
`TestDockerContainerWithoutIDKeepsPendingUpdateAgeAcrossUpgrade` and
|
||||
`TestDockerContainerResourceID` in `internal/alerts/alerts_test.go`, and
|
||||
`TestOwnerAlertTimelinesUseCanonicalHistoryIdentity` in
|
||||
`internal/monitoring/monitor_alert_handling_test.go`.
|
||||
|
||||
Backup orphan evaluation is also inventory-scoped. The alerts runtime may
|
||||
evaluate recovery rollups for backup age, but unresolved Proxmox PVE backup
|
||||
subjects must not be treated as orphaned until monitoring has supplied the
|
||||
|
||||
@@ -1065,14 +1065,15 @@ events (ZFS pools and devices, host filesystems, disks, RAID arrays, sensors
|
||||
and the Unraid array), pass through the same writer, which binds them to the
|
||||
Docker host, Swarm service, storage or host named in the unified-resources
|
||||
history identity clause. Container names and shortened container IDs still
|
||||
bind nothing. `CheckDockerHost` gives a container without an ID its host's
|
||||
reference, so that container's events join the host's history. A producer
|
||||
bind nothing, and neither does the marked name a container or Swarm service
|
||||
reported without an ID alerts under, so its events join neither its host's
|
||||
history nor that of a service whose ID it is named like. A producer
|
||||
that changes one of these reference shapes must change
|
||||
`dockerHostHistoryReference` or `historySubResourceOwner` with it.
|
||||
`TestOwnerAlertTimelinesUseCanonicalHistoryIdentity` in
|
||||
`internal/monitoring/monitor_alert_handling_test.go` pins the Docker host,
|
||||
Swarm service, ZFS pool and device, and host filesystem shapes against the
|
||||
real alert manager.
|
||||
Swarm service, ID-less container and service, ZFS pool and device, and host
|
||||
filesystem shapes against the real alert manager.
|
||||
|
||||
TrueNAS native alert projection preserves the trimmed, uppercase provider level in ResourceIncident.NativeSeverity. INFO and NOTICE retain the same canonical monitor risk; consumers must not lose their distinct actionability when projecting provider evidence. Native CRITICAL, ALERT, and EMERGENCY all project to canonical critical severity; EMERGENCY must not be discarded as unknown or make a still-active condition appear recovered. WARNING remains warning, and INFO and NOTICE remain informational at this projection boundary.
|
||||
|
||||
|
||||
@@ -5842,17 +5842,20 @@ Docker host alerts (`docker:<host ID>`) and Swarm service alerts
|
||||
Docker source identities: the host ID is the Docker host's source ID, and a
|
||||
service reference names the service with that exact ID in the host's Swarm
|
||||
cluster, whichever manager reported it. A hostname in the host position is
|
||||
only ever looked up as a Docker host source ID. A service without an ID is
|
||||
referenced by its normalized name in the same place, so while its cluster has
|
||||
such a service every service reference there is a conflict. Alert references
|
||||
do not record whether they came from an ID, so a row journaled by an ID-less
|
||||
service can still join another service's history if a later generation or
|
||||
event binds the same reference; Swarm always reports service IDs, so that needs a malformed
|
||||
report naming a service exactly like another service's ID. Hostless
|
||||
`docker-service:<name>` references, container names and shortened container
|
||||
IDs never bind and are never retried against later inventory. A container
|
||||
without an ID alerts under its host's reference (`alerts.DockerResourceID`), so
|
||||
its events join that host's history, never another container's.
|
||||
only ever looked up as a Docker host source ID. The registry keys a service
|
||||
without an ID by its raw name, so the service entry must also carry exactly
|
||||
the referenced ID. A container or service reported without an ID alerts under
|
||||
its name, marked in the ID position (`docker:<host ID>/name:<name>` and
|
||||
`docker:<host ID>/service/name:<name>`, from `alerts.DockerContainerResourceID`
|
||||
and `alerts.DockerServiceResourceID`). Docker and Swarm IDs never contain a
|
||||
colon, so such a reference never equals its host's reference or another
|
||||
service's ID reference, and `dockerHostHistoryReference` treats a service slot
|
||||
with a colon as a name. These name references, hostless `docker-service:`
|
||||
references, container names and shortened container IDs never bind and are
|
||||
never retried against later inventory. Rows journaled before name references
|
||||
existed keep the reference they were written under: an ID-less container's
|
||||
rows stay in its host's history, and an ID-less service's rows bind only to a
|
||||
service in that cluster whose ID equals its normalized name.
|
||||
Sub-resource alert references name no resource of their own and bind to the
|
||||
owner whose durable ID they carry (`historySubResourceOwner`): ZFS pool and
|
||||
device alerts (`<storage ID>/zfs-pool:<pool>[/device:<device>]`) to the
|
||||
|
||||
@@ -149,12 +149,18 @@ func TestResolvePatrolScopeCanonicalIDOutranksAliasCollision(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestResolvePatrolScopeResolvesDockerAlertResourceIDs(t *testing.T) {
|
||||
// A container reported without an ID alerts under its own name reference,
|
||||
// so the host's reference stays unambiguous.
|
||||
snapshot := models.StateSnapshot{DockerHosts: []models.DockerHost{{
|
||||
ID: "nas", Hostname: "nas",
|
||||
Containers: []models.DockerContainer{{ID: "abc123def456", Name: "dockhand-hawser-updater-nas", State: "exited"}},
|
||||
Containers: []models.DockerContainer{
|
||||
{ID: "abc123def456", Name: "dockhand-hawser-updater-nas", State: "exited"},
|
||||
{Name: "legacy-sidecar", State: "exited"},
|
||||
},
|
||||
}}}
|
||||
containerAlertID := alerts.DockerResourceID("nas", "abc123def456")
|
||||
hostAlertID := alerts.DockerResourceID("nas", "")
|
||||
containerAlertID := alerts.DockerContainerResourceID("nas", "abc123def456", "dockhand-hawser-updater-nas")
|
||||
namedContainerAlertID := alerts.DockerContainerResourceID("nas", "", "legacy-sidecar")
|
||||
hostAlertID := alerts.DockerHostResourceID("nas")
|
||||
|
||||
snapshotState := newPatrolRuntimeState(snapshot)
|
||||
registry := unifiedresources.NewRegistry(nil)
|
||||
@@ -165,7 +171,12 @@ func TestResolvePatrolScopeResolvesDockerAlertResourceIDs(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
requested string
|
||||
resourceType string
|
||||
}{{containerAlertID, "app-container"}, {hostAlertID, "docker-host"}} {
|
||||
}{{containerAlertID, "app-container"}, {namedContainerAlertID, "app-container"}, {hostAlertID, "docker-host"}} {
|
||||
if tc.requested == namedContainerAlertID && state.readState == nil {
|
||||
// The legacy snapshot record of a container without an ID has
|
||||
// no ID to scope by; only the unified read state can.
|
||||
continue
|
||||
}
|
||||
resolved, resolution := resolvePatrolScopeState(state, PatrolScope{ResourceIDs: []string{tc.requested}})
|
||||
if len(resolution.UnmatchedResourceIDs) != 0 || len(resolution.AmbiguousResourceIDs) != 0 {
|
||||
t.Fatalf("alert resource ID %q did not resolve exactly: %+v", tc.requested, resolution)
|
||||
|
||||
+19
-14
@@ -9,32 +9,37 @@ import (
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/unifiedresources"
|
||||
)
|
||||
|
||||
// dockerAlertScopeAlias mirrors the alert subsystem's docker resource IDs
|
||||
// ("docker:<host>" and "docker:<host>/<container>") so an alert's resource ID
|
||||
// resolves to the patrol runtime record for that host or container (#1699).
|
||||
// Returns "" without a host part rather than the shared "docker:unknown"
|
||||
// fallback, which would alias unrelated records together.
|
||||
func dockerAlertScopeAlias(hostID, containerID string) string {
|
||||
// dockerAlertScopeAlias mirrors the alert subsystem's Docker container
|
||||
// resource IDs ("docker:<host>/<container ID>", or "docker:<host>/name:<name>"
|
||||
// for a container without an ID) so an alert's resource ID resolves to the
|
||||
// patrol runtime record for that container (#1699). A host's own records use
|
||||
// alerts.DockerHostResourceID. Returns "" without a host part rather than a
|
||||
// hostless form, which would alias unrelated records together.
|
||||
func dockerAlertScopeAlias(hostID, containerID, containerName string) string {
|
||||
if strings.TrimSpace(hostID) == "" {
|
||||
return ""
|
||||
}
|
||||
return alerts.DockerResourceID(hostID, containerID)
|
||||
return alerts.DockerContainerResourceID(hostID, containerID, containerName)
|
||||
}
|
||||
|
||||
// dockerServiceAlertScopeAliases mirrors the alert subsystem's Swarm service
|
||||
// resource IDs ("docker:<host>/service/<serviceID>") onto the owning Docker
|
||||
// resource IDs ("docker:<host>/service/<serviceID>", or
|
||||
// "docker:<host>/service/name:<name>" without an ID) onto the owning Docker
|
||||
// host record. Patrol has no service-level analysis, and a service's tasks run
|
||||
// as containers on the host, so the host is the smallest unit Patrol can
|
||||
// investigate for a service alert (#1699). Hosts without an ID are skipped for
|
||||
// the same anti-aliasing reason as dockerAlertScopeAlias: the host-less
|
||||
// "docker-service:<name>" fallback could collide across hosts.
|
||||
// "docker-service:<name>" fallback could collide across hosts. A service with
|
||||
// neither an ID nor a name raises no alert, so it has no alias.
|
||||
func dockerServiceAlertScopeAliases(hostID string, services []models.DockerService) []string {
|
||||
if strings.TrimSpace(hostID) == "" || len(services) == 0 {
|
||||
return nil
|
||||
}
|
||||
aliases := make([]string, 0, len(services))
|
||||
for _, service := range services {
|
||||
aliases = append(aliases, alerts.DockerServiceResourceID(hostID, service.ID, service.Name))
|
||||
if alias := alerts.DockerServiceResourceID(hostID, service.ID, service.Name); alias != "" {
|
||||
aliases = append(aliases, alias)
|
||||
}
|
||||
}
|
||||
return aliases
|
||||
}
|
||||
@@ -263,14 +268,14 @@ func patrolVisitRuntimeResources(s patrolRuntimeState, visit func(patrolRuntimeR
|
||||
}
|
||||
}
|
||||
for _, dh := range rs.DockerHosts() {
|
||||
aliases := append([]string{dh.Name(), dh.Hostname(), dockerAlertScopeAlias(dh.HostSourceID(), "")},
|
||||
aliases := append([]string{dh.Name(), dh.Hostname(), alerts.DockerHostResourceID(dh.HostSourceID())},
|
||||
dockerServiceAlertScopeAliases(dh.HostSourceID(), dh.Services())...)
|
||||
if !emit(patrolRuntimeResourceDockerHost, []string{dh.ID(), dh.HostSourceID()}, aliases...) {
|
||||
return
|
||||
}
|
||||
}
|
||||
for _, dc := range rs.DockerContainers() {
|
||||
if !emit(patrolRuntimeResourceDockerItem, []string{dc.ID(), dc.ContainerID()}, dc.Name(), dockerAlertScopeAlias(dc.HostSourceID(), dc.ContainerID())) {
|
||||
if !emit(patrolRuntimeResourceDockerItem, []string{dc.ID(), dc.ContainerID()}, dc.Name(), dockerAlertScopeAlias(dc.HostSourceID(), dc.ContainerID(), dc.Name())) {
|
||||
return
|
||||
}
|
||||
}
|
||||
@@ -316,13 +321,13 @@ func patrolVisitRuntimeResources(s patrolRuntimeState, visit func(patrolRuntimeR
|
||||
}
|
||||
}
|
||||
for _, dh := range s.DockerHosts {
|
||||
aliases := append([]string{dh.DisplayName, dh.CustomDisplayName, dh.Hostname, dockerAlertScopeAlias(dh.ID, "")},
|
||||
aliases := append([]string{dh.DisplayName, dh.CustomDisplayName, dh.Hostname, alerts.DockerHostResourceID(dh.ID)},
|
||||
dockerServiceAlertScopeAliases(dh.ID, dh.Services)...)
|
||||
if !emit(patrolRuntimeResourceDockerHost, []string{dh.ID}, aliases...) {
|
||||
return
|
||||
}
|
||||
for _, dc := range dh.Containers {
|
||||
if !emit(patrolRuntimeResourceDockerItem, []string{dc.ID}, dc.Name, dockerAlertScopeAlias(dh.ID, dc.ID)) {
|
||||
if !emit(patrolRuntimeResourceDockerItem, []string{dc.ID}, dc.Name, dockerAlertScopeAlias(dh.ID, dc.ID, dc.Name)) {
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
@@ -78,16 +78,17 @@ func TestDockerServiceAlertScopeAliasesWithoutServices(t *testing.T) {
|
||||
}
|
||||
|
||||
// TestDockerServiceAlertScopeAliasesDeriveIDFromName covers services that carry
|
||||
// no ID, where the canonical builder derives one from the service name. The
|
||||
// alias has to follow that derivation rather than formatting its own string.
|
||||
// no ID, where the canonical builder references the marked service name. The
|
||||
// alias has to follow that derivation rather than formatting its own string,
|
||||
// and a service with neither an ID nor a name raises no alert, so no alias.
|
||||
func TestDockerServiceAlertScopeAliasesDeriveIDFromName(t *testing.T) {
|
||||
got := dockerServiceAlertScopeAliases("host-1", []models.DockerService{{Name: "Web API"}})
|
||||
got := dockerServiceAlertScopeAliases("host-1", []models.DockerService{{Name: "Web API"}, {Name: " "}})
|
||||
|
||||
want := alerts.DockerServiceResourceID("host-1", "", "Web API")
|
||||
if len(got) != 1 || got[0] != want {
|
||||
t.Fatalf("aliases = %v, want [%q]", got, want)
|
||||
}
|
||||
if got[0] != "docker:host-1/service/web-api" {
|
||||
t.Errorf("alias = %q, want %q", got[0], "docker:host-1/service/web-api")
|
||||
if got[0] != "docker:host-1/service/name:web-api" {
|
||||
t.Errorf("alias = %q, want %q", got[0], "docker:host-1/service/name:web-api")
|
||||
}
|
||||
}
|
||||
|
||||
+315
-58
@@ -2720,7 +2720,7 @@ func TestCheckDockerHostIgnoresContainersByPrefix(t *testing.T) {
|
||||
Containers: []models.DockerContainer{container},
|
||||
}
|
||||
|
||||
resourceID := DockerResourceID(host.ID, container.ID)
|
||||
resourceID := DockerContainerResourceID(host.ID, container.ID, "")
|
||||
alertID := fmt.Sprintf("docker-container-state-%s", resourceID)
|
||||
|
||||
// Run twice to satisfy the confirmation threshold when not ignored
|
||||
@@ -2759,7 +2759,7 @@ func TestCheckDockerHostContainerCPUUsesCapacityNormalizedPercent(t *testing.T)
|
||||
CPUPercent: 240,
|
||||
}},
|
||||
}
|
||||
alertID := canonicalMetricStateID(DockerResourceID(host.ID, "container-1"), "cpu")
|
||||
alertID := canonicalMetricStateID(DockerContainerResourceID(host.ID, "container-1", ""), "cpu")
|
||||
|
||||
m.CheckDockerHost(host)
|
||||
m.mu.RLock()
|
||||
@@ -3058,7 +3058,7 @@ func TestDockerContainerStateUsesDockerDefaults(t *testing.T) {
|
||||
m.CheckDockerHost(host)
|
||||
m.CheckDockerHost(host)
|
||||
|
||||
resourceID := DockerResourceID(host.ID, container.ID)
|
||||
resourceID := DockerContainerResourceID(host.ID, container.ID, "")
|
||||
alertID := fmt.Sprintf("docker-container-state-%s", resourceID)
|
||||
alert, exists := testLookupActiveAlert(t, m, alertID)
|
||||
if !exists {
|
||||
@@ -3091,7 +3091,7 @@ func TestDockerContainerStateRespectsDisableDefault(t *testing.T) {
|
||||
m.CheckDockerHost(host)
|
||||
m.CheckDockerHost(host)
|
||||
|
||||
resourceID := DockerResourceID(host.ID, container.ID)
|
||||
resourceID := DockerContainerResourceID(host.ID, container.ID, "")
|
||||
alertID := fmt.Sprintf("docker-container-state-%s", resourceID)
|
||||
if testHasActiveAlert(t, m, alertID) {
|
||||
t.Fatalf("did not expect docker container state alert when defaults disable connectivity")
|
||||
@@ -3121,7 +3121,7 @@ func TestDockerContainerMemoryLimitHysteresis(t *testing.T) {
|
||||
|
||||
m.CheckDockerHost(hostHigh)
|
||||
|
||||
resourceID := DockerResourceID(hostID, containerID)
|
||||
resourceID := DockerContainerResourceID(hostID, containerID, "")
|
||||
alertID := fmt.Sprintf("docker-container-memory-limit-%s", resourceID)
|
||||
alert, exists := testLookupActiveAlert(t, m, alertID)
|
||||
if !exists {
|
||||
@@ -3190,7 +3190,7 @@ func TestDockerContainerDiskUsageAlert(t *testing.T) {
|
||||
|
||||
m.CheckDockerHost(host)
|
||||
|
||||
resourceID := DockerResourceID(host.ID, host.Containers[0].ID)
|
||||
resourceID := DockerContainerResourceID(host.ID, host.Containers[0].ID, "")
|
||||
alertID := canonicalMetricStateID(resourceID, "disk")
|
||||
alert, exists := testLookupActiveAlert(t, m, alertID)
|
||||
if !exists {
|
||||
@@ -3782,7 +3782,7 @@ func TestCheckDockerHostIgnoredPrefixClearsExistingAlerts(t *testing.T) {
|
||||
Hostname: "docker-host.local",
|
||||
Containers: []models.DockerContainer{container},
|
||||
}
|
||||
resourceID := DockerResourceID(host.ID, container.ID)
|
||||
resourceID := DockerContainerResourceID(host.ID, container.ID, "")
|
||||
stateAlertID := fmt.Sprintf("docker-container-state-%s", resourceID)
|
||||
healthAlertID := fmt.Sprintf("docker-container-health-%s", resourceID)
|
||||
restartAlertID := fmt.Sprintf("docker-container-restart-loop-%s", resourceID)
|
||||
@@ -3954,20 +3954,25 @@ func TestDockerContainerDisplayName(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestDockerResourceID(t *testing.T) {
|
||||
func TestDockerContainerResourceID(t *testing.T) {
|
||||
// t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
hostID string
|
||||
containerID string
|
||||
want string
|
||||
name string
|
||||
hostID string
|
||||
containerID string
|
||||
containerName string
|
||||
want string
|
||||
}{
|
||||
{name: "both ids present", hostID: "host1", containerID: "abc", want: "docker:host1/abc"},
|
||||
{name: "both ids present", hostID: "host1", containerID: "abc", containerName: "web", want: "docker:host1/abc"},
|
||||
{name: "trims ids", hostID: " host1 ", containerID: " abc ", want: "docker:host1/abc"},
|
||||
{name: "missing host id", hostID: "", containerID: "abc", want: "docker:container/abc"},
|
||||
{name: "missing container id", hostID: "host1", containerID: "", want: "docker:host1"},
|
||||
{name: "both missing", hostID: "", containerID: "", want: "docker:unknown"},
|
||||
// A container without an ID never takes its host's reference.
|
||||
{name: "missing container id uses the marked name", hostID: "host1", containerID: "", containerName: " /web ", want: "docker:host1/name:web"},
|
||||
{name: "a name never equals an ID reference", hostID: "host1", containerID: "", containerName: "abc", want: "docker:host1/name:abc"},
|
||||
{name: "no id and no name", hostID: "host1", containerID: "", containerName: " / ", want: ""},
|
||||
{name: "a name without a host identifies nothing", hostID: "", containerID: "", containerName: "web", want: ""},
|
||||
{name: "all missing", want: ""},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
@@ -3975,11 +3980,255 @@ func TestDockerResourceID(t *testing.T) {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
// t.Parallel()
|
||||
|
||||
if got := DockerResourceID(tc.hostID, tc.containerID); got != tc.want {
|
||||
t.Fatalf("DockerResourceID(%q, %q) = %q, want %q", tc.hostID, tc.containerID, got, tc.want)
|
||||
if got := DockerContainerResourceID(tc.hostID, tc.containerID, tc.containerName); got != tc.want {
|
||||
t.Fatalf("DockerContainerResourceID(%q, %q, %q) = %q, want %q", tc.hostID, tc.containerID, tc.containerName, got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
if got := DockerHostResourceID(" host1 "); got != "docker:host1" {
|
||||
t.Fatalf("DockerHostResourceID = %q, want docker:host1", got)
|
||||
}
|
||||
if got := DockerHostResourceID(" "); got != "" {
|
||||
t.Fatalf("DockerHostResourceID without a host = %q, want empty", got)
|
||||
}
|
||||
}
|
||||
|
||||
// Containers and Swarm services reported without an ID alert under their own
|
||||
// marked name, never under the host's reference or another service's ID
|
||||
// reference, so cleanup, removal and the policy switches reach them like any
|
||||
// other container or service.
|
||||
func TestDockerAlertsWithoutIDsUseTheirOwnReferences(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
config := m.GetConfig()
|
||||
config.Enabled = true
|
||||
config.TimeThresholds = map[string]int{"all": 0}
|
||||
config.SuppressionWindow = 0
|
||||
m.UpdateConfig(config)
|
||||
|
||||
containerID := strings.Repeat("c", 64)
|
||||
serviceID := "x7k2m9q4w1e8r5t3y6u0i2o4p"
|
||||
host := models.DockerHost{ID: "h1", Hostname: "tower", DisplayName: "tower", Status: "online", LastSeen: time.Now(),
|
||||
Containers: []models.DockerContainer{
|
||||
{ID: containerID, Name: "web", State: "running", Health: "unhealthy"},
|
||||
{Name: "/sidecar", State: "running", Health: "unhealthy"},
|
||||
// Docker forbids slashes in names, but a report can carry anything.
|
||||
{Name: "jobs/service/runner", State: "running", Health: "unhealthy"},
|
||||
{State: "running", Health: "unhealthy"}, // no identity at all
|
||||
},
|
||||
Services: []models.DockerService{
|
||||
{ID: serviceID, Name: "api", DesiredTasks: 2},
|
||||
{Name: strings.ToUpper(serviceID), DesiredTasks: 2},
|
||||
{DesiredTasks: 2},
|
||||
}}
|
||||
refs := func() map[string]bool {
|
||||
got := map[string]bool{}
|
||||
for _, alert := range m.GetActiveAlerts() {
|
||||
got[alert.ResourceID] = true
|
||||
}
|
||||
return got
|
||||
}
|
||||
var firedMu sync.Mutex
|
||||
fired := map[string]bool{}
|
||||
m.SubscribeLifecycleCallback(func(event LifecycleEvent) {
|
||||
if event.Type == eventlog.TypeFired && event.Alert != nil {
|
||||
firedMu.Lock()
|
||||
fired[event.Alert.ResourceID] = true
|
||||
firedMu.Unlock()
|
||||
}
|
||||
})
|
||||
m.CheckDockerHost(host)
|
||||
want := map[string]bool{
|
||||
"docker:h1/" + containerID: true,
|
||||
"docker:h1/name:sidecar": true,
|
||||
"docker:h1/name:jobs/service/runner": true,
|
||||
"docker:h1/service/" + serviceID: true,
|
||||
"docker:h1/service/name:" + serviceID: true,
|
||||
}
|
||||
if got := refs(); !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("alert references = %v, want %v", got, want)
|
||||
}
|
||||
firedMu.Lock()
|
||||
if !reflect.DeepEqual(fired, want) {
|
||||
t.Fatalf("fired alert references = %v, want %v", fired, want)
|
||||
}
|
||||
firedMu.Unlock()
|
||||
|
||||
// A container that leaves the report clears under its own reference.
|
||||
reported := host
|
||||
reported.Containers = host.Containers[:1]
|
||||
m.CheckDockerHost(reported)
|
||||
if got := refs(); got["docker:h1/name:sidecar"] || got["docker:h1/name:jobs/service/runner"] {
|
||||
t.Fatalf("alerts of removed containers without an ID survived the report: %v", got)
|
||||
}
|
||||
|
||||
// Before this reference existed such a container alerted under its host's.
|
||||
// No producer refreshes those, so the next report clears them.
|
||||
legacy := func() {
|
||||
m.mu.Lock()
|
||||
m.activeAlerts["legacy"] = &Alert{ID: "legacy", Type: "docker-container-health", ResourceID: "docker:h1", StartTime: time.Now(), LastSeen: time.Now(),
|
||||
Metadata: map[string]interface{}{"hostId": "h1", "containerId": "", "containerName": "sidecar"}}
|
||||
m.mu.Unlock()
|
||||
}
|
||||
legacy()
|
||||
m.CheckDockerHost(host)
|
||||
if refs()["docker:h1"] {
|
||||
t.Fatal("a pre-upgrade container alert on the host's reference survived the host's report")
|
||||
}
|
||||
// Going offline clears it too, and keeps the host's own alert.
|
||||
legacy()
|
||||
for range 3 {
|
||||
m.HandleDockerHostOffline(host)
|
||||
}
|
||||
offline := 0
|
||||
for _, alert := range m.GetActiveAlerts() {
|
||||
if alert.ResourceID == "docker:h1" {
|
||||
if alert.Type != "docker-host-offline" {
|
||||
t.Fatalf("a pre-upgrade container alert on the host's reference survived going offline: %+v", alert)
|
||||
}
|
||||
offline++
|
||||
}
|
||||
}
|
||||
if offline != 1 {
|
||||
t.Fatalf("host offline alerts = %d, want 1", offline)
|
||||
}
|
||||
|
||||
// The policy switches classify the name references by kind.
|
||||
m.CheckDockerHost(host)
|
||||
config = m.GetConfig()
|
||||
config.DisableAllDockerContainers = true
|
||||
m.UpdateConfig(config)
|
||||
if got := refs(); len(got) != 2 || !got["docker:h1/service/"+serviceID] || !got["docker:h1/service/name:"+serviceID] {
|
||||
t.Fatalf("after disabling containers, alert references = %v", got)
|
||||
}
|
||||
config.DisableAllDockerServices = true
|
||||
m.UpdateConfig(config)
|
||||
if got := refs(); len(got) != 0 {
|
||||
t.Fatalf("after disabling services, alert references = %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A Docker host disambiguated as "<base>::<suffix>" carries the canonical
|
||||
// state separator in its ID. Its alerts keep their full reference, so they
|
||||
// never take the reference of the host named "<base>", and a restored alert
|
||||
// whose reference was cut short there is not that host's to clear.
|
||||
func TestDockerHostIDWithSeparatorKeepsItsAlertReferences(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
config := m.GetConfig()
|
||||
config.Enabled = true
|
||||
config.TimeThresholds = map[string]int{"all": 0}
|
||||
m.UpdateConfig(config)
|
||||
containerID := strings.Repeat("d", 64)
|
||||
report := func(hostID string) models.DockerHost {
|
||||
return models.DockerHost{ID: hostID, Hostname: "clone", LastSeen: time.Now(),
|
||||
Containers: []models.DockerContainer{{ID: containerID, Name: "web", State: "running", Health: "unhealthy"}}}
|
||||
}
|
||||
m.CheckDockerHost(report("base::suffix"))
|
||||
m.CheckDockerHost(report("base"))
|
||||
got := map[string]string{}
|
||||
for _, alert := range m.GetActiveAlerts() {
|
||||
got[alert.Metadata["hostId"].(string)] = alert.ResourceID
|
||||
}
|
||||
want := map[string]string{"base": "docker:base/" + containerID, "base::suffix": "docker:base::suffix/" + containerID}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("alert references by host = %v, want %v", got, want)
|
||||
}
|
||||
|
||||
restored := &Alert{ID: "docker:base::suffix/x::docker:base::suffix/x-health", Type: "docker-container-health", ResourceID: "docker:base",
|
||||
StartTime: time.Now(), LastSeen: time.Now(), Metadata: map[string]interface{}{"hostId": "base::suffix", "containerId": "x"}}
|
||||
m.mu.Lock()
|
||||
m.activeAlerts[restored.ID] = restored
|
||||
m.mu.Unlock()
|
||||
m.CheckDockerHost(report("base"))
|
||||
m.mu.RLock()
|
||||
_, kept := m.activeAlerts[restored.ID]
|
||||
m.mu.RUnlock()
|
||||
if !kept {
|
||||
t.Fatal("host base cleared another host's restored alert as its own pre-upgrade container alert")
|
||||
}
|
||||
}
|
||||
|
||||
// Alerts saved before state IDs split after their own spec carry a resource
|
||||
// ID cut short at the first "::" of a host ID like "<base>::<suffix>". They
|
||||
// recover their reference on restore, so the owning host's report clears them
|
||||
// and carries the age of a pending update.
|
||||
func TestDockerAlertsRestoredWithCutShortReferencesRecover(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
config := m.GetConfig()
|
||||
config.Enabled = true
|
||||
config.TimeThresholds = map[string]int{"all": 0}
|
||||
config.DockerDefaults.UpdateAlertDelayHours = 24
|
||||
m.UpdateConfig(config)
|
||||
hostRef := "docker:base::suffix"
|
||||
containerRef := hostRef + "/" + strings.Repeat("e", 64)
|
||||
startedAt := time.Now().Add(-72 * time.Hour).Truncate(time.Second)
|
||||
saved := []*Alert{
|
||||
{ID: buildCanonicalStateID(hostRef, hostRef+"-image-update"), CanonicalSpecID: hostRef + "-image-update", Type: "docker-container-update",
|
||||
ResourceID: "docker:base", Level: AlertLevelWarning, StartTime: startedAt, LastSeen: time.Now(),
|
||||
Metadata: map[string]interface{}{"hostId": "base::suffix", "containerId": "", "containerName": "sidecar"}},
|
||||
{ID: buildCanonicalStateID(containerRef, containerRef+"-health"), CanonicalSpecID: containerRef + "-health", Type: "docker-container-health",
|
||||
ResourceID: "docker:base", Level: AlertLevelCritical, StartTime: startedAt, LastSeen: time.Now(),
|
||||
Metadata: map[string]interface{}{"hostId": "base::suffix", "containerId": strings.Repeat("e", 64), "containerName": "gone"}},
|
||||
}
|
||||
data, err := json.Marshal(saved)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.MkdirAll(m.getAlertsDir(), alertsDirPerm); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(m.getAlertsDir(), "active-alerts.json"), data, alertsFilePerm); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := m.LoadActiveAlerts(); err != nil {
|
||||
t.Fatalf("LoadActiveAlerts() error = %v", err)
|
||||
}
|
||||
restored := map[string]bool{}
|
||||
for _, alert := range m.GetActiveAlerts() {
|
||||
restored[alert.ResourceID] = true
|
||||
}
|
||||
if want := map[string]bool{hostRef: true, containerRef: true}; !reflect.DeepEqual(restored, want) {
|
||||
t.Fatalf("restored references = %v, want %v", restored, want)
|
||||
}
|
||||
|
||||
m.CheckDockerHost(models.DockerHost{ID: "base::suffix", Hostname: "clone", LastSeen: time.Now(), Containers: []models.DockerContainer{{
|
||||
Name: "sidecar", State: "running",
|
||||
UpdateStatus: &models.DockerContainerUpdateStatus{UpdateAvailable: true, CurrentDigest: "sha256:a", LatestDigest: "sha256:b", LastChecked: time.Now()},
|
||||
}}})
|
||||
active := m.GetActiveAlerts()
|
||||
if len(active) != 1 || active[0].ResourceID != hostRef+"/name:sidecar" || !active[0].StartTime.Equal(startedAt) {
|
||||
t.Fatalf("active alerts = %+v, want only the update on %s/name:sidecar started %s", active, hostRef, startedAt)
|
||||
}
|
||||
}
|
||||
|
||||
// Before a container without an ID had a reference of its own, its pending
|
||||
// image update was recorded under its host's reference. The first report
|
||||
// after the upgrade keeps that age instead of restarting the delay.
|
||||
func TestDockerContainerWithoutIDKeepsPendingUpdateAgeAcrossUpgrade(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
config := m.GetConfig()
|
||||
config.Enabled = true
|
||||
config.DockerDefaults.UpdateAlertDelayHours = 24
|
||||
m.UpdateConfig(config)
|
||||
startedAt := time.Now().Add(-72 * time.Hour).Truncate(time.Second)
|
||||
legacyID := buildCanonicalStateID("docker:h1", "docker:h1-image-update")
|
||||
m.mu.Lock()
|
||||
m.activeAlerts[legacyID] = &Alert{ID: legacyID, Type: "docker-container-update", ResourceID: "docker:h1", StartTime: startedAt, LastSeen: time.Now(),
|
||||
Metadata: map[string]interface{}{"hostId": "h1", "containerId": "", "containerName": "sidecar"}}
|
||||
m.mu.Unlock()
|
||||
|
||||
m.CheckDockerHost(models.DockerHost{ID: "h1", Hostname: "tower", LastSeen: time.Now(), Containers: []models.DockerContainer{{
|
||||
Name: "sidecar", State: "running", Image: "nginx:latest",
|
||||
UpdateStatus: &models.DockerContainerUpdateStatus{UpdateAvailable: true, CurrentDigest: "sha256:a", LatestDigest: "sha256:b", LastChecked: time.Now()},
|
||||
}}})
|
||||
var updates []Alert
|
||||
for _, alert := range m.GetActiveAlerts() {
|
||||
if alert.Type == "docker-container-update" {
|
||||
updates = append(updates, alert)
|
||||
}
|
||||
}
|
||||
if len(updates) != 1 || updates[0].ResourceID != "docker:h1/name:sidecar" || !updates[0].StartTime.Equal(startedAt) {
|
||||
t.Fatalf("update alerts = %+v, want one on docker:h1/name:sidecar started %s", updates, startedAt)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHasKnownFirmwareBug(t *testing.T) {
|
||||
@@ -6127,81 +6376,89 @@ func TestDockerServiceResourceID(t *testing.T) {
|
||||
expected: "docker-service:svc-123",
|
||||
},
|
||||
{
|
||||
name: "derives ID from service name when ID empty",
|
||||
name: "marks the derived name when ID empty",
|
||||
hostID: "host-1",
|
||||
serviceID: "",
|
||||
serviceName: "My Service",
|
||||
expected: "docker:host-1/service/my-service",
|
||||
expected: "docker:host-1/service/name:my-service",
|
||||
},
|
||||
{
|
||||
name: "special chars in name replaced with dash",
|
||||
hostID: "host-1",
|
||||
serviceID: "",
|
||||
serviceName: "my/service:v1.0",
|
||||
expected: "docker:host-1/service/my-service-v1-0",
|
||||
expected: "docker:host-1/service/name:my-service-v1-0",
|
||||
},
|
||||
{
|
||||
name: "backslash and colon replaced",
|
||||
hostID: "host-1",
|
||||
serviceID: "",
|
||||
serviceName: "path\\to:service",
|
||||
expected: "docker:host-1/service/path-to-service",
|
||||
expected: "docker:host-1/service/name:path-to-service",
|
||||
},
|
||||
{
|
||||
name: "preserves alphanumeric and underscore",
|
||||
hostID: "host-1",
|
||||
serviceID: "",
|
||||
serviceName: "my_service_123",
|
||||
expected: "docker:host-1/service/my_service_123",
|
||||
expected: "docker:host-1/service/name:my_service_123",
|
||||
},
|
||||
{
|
||||
name: "preserves hyphens",
|
||||
hostID: "host-1",
|
||||
serviceID: "",
|
||||
serviceName: "my-service-name",
|
||||
expected: "docker:host-1/service/my-service-name",
|
||||
expected: "docker:host-1/service/name:my-service-name",
|
||||
},
|
||||
{
|
||||
name: "trims leading/trailing dashes and underscores",
|
||||
hostID: "host-1",
|
||||
serviceID: "",
|
||||
serviceName: "---my-service___",
|
||||
expected: "docker:host-1/service/my-service",
|
||||
expected: "docker:host-1/service/name:my-service",
|
||||
},
|
||||
{
|
||||
name: "truncates long derived ID to 32 chars",
|
||||
hostID: "host-1",
|
||||
serviceID: "",
|
||||
serviceName: "this-is-a-very-long-service-name-that-exceeds-the-limit",
|
||||
expected: "docker:host-1/service/this-is-a-very-long-service-name",
|
||||
expected: "docker:host-1/service/name:this-is-a-very-long-service-name",
|
||||
},
|
||||
{
|
||||
name: "uses 'service' when name is all special chars",
|
||||
name: "no identity when name is all special chars",
|
||||
hostID: "host-1",
|
||||
serviceID: "",
|
||||
serviceName: "!!!@@@###",
|
||||
expected: "docker:host-1/service/service",
|
||||
expected: "",
|
||||
},
|
||||
{
|
||||
name: "uses 'service' when both ID and name empty",
|
||||
name: "no identity when both ID and name empty",
|
||||
hostID: "host-1",
|
||||
serviceID: "",
|
||||
serviceName: "",
|
||||
expected: "docker:host-1/service/service",
|
||||
expected: "",
|
||||
},
|
||||
{
|
||||
name: "uses 'service' when both ID and name whitespace",
|
||||
name: "no identity when both ID and name whitespace",
|
||||
hostID: "host-1",
|
||||
serviceID: " ",
|
||||
serviceName: " ",
|
||||
expected: "docker:host-1/service/service",
|
||||
expected: "",
|
||||
},
|
||||
{
|
||||
name: "no host and derived name",
|
||||
hostID: "",
|
||||
serviceID: "",
|
||||
serviceName: "webserver",
|
||||
expected: "docker-service:webserver",
|
||||
expected: "docker-service:name:webserver",
|
||||
},
|
||||
{
|
||||
// A service named like another service's Swarm ID keeps its own reference.
|
||||
name: "a name never equals an ID reference",
|
||||
hostID: "host-1",
|
||||
serviceID: "",
|
||||
serviceName: "X7K2M9Q4W1E8R5T3Y6U0I2O4P",
|
||||
expected: "docker:host-1/service/name:x7k2m9q4w1e8r5t3y6u0i2o4p",
|
||||
},
|
||||
}
|
||||
|
||||
@@ -13154,7 +13411,7 @@ func TestDockerContainerHealthAlert(t *testing.T) {
|
||||
|
||||
m.CheckDockerHost(host)
|
||||
|
||||
resourceID := DockerResourceID(host.ID, host.Containers[0].ID)
|
||||
resourceID := DockerContainerResourceID(host.ID, host.Containers[0].ID, "")
|
||||
alertID := fmt.Sprintf("docker-container-health-%s", resourceID)
|
||||
if testHasActiveAlert(t, m, alertID) {
|
||||
t.Fatal("expected no health alert for healthy container")
|
||||
@@ -13181,7 +13438,7 @@ func TestDockerContainerHealthAlert(t *testing.T) {
|
||||
|
||||
m.CheckDockerHost(host)
|
||||
|
||||
resourceID := DockerResourceID(host.ID, host.Containers[0].ID)
|
||||
resourceID := DockerContainerResourceID(host.ID, host.Containers[0].ID, "")
|
||||
alertID := fmt.Sprintf("docker-container-health-%s", resourceID)
|
||||
if testHasActiveAlert(t, m, alertID) {
|
||||
t.Fatal("expected no health alert for container with empty health")
|
||||
@@ -13208,7 +13465,7 @@ func TestDockerContainerHealthAlert(t *testing.T) {
|
||||
|
||||
m.CheckDockerHost(host)
|
||||
|
||||
resourceID := DockerResourceID(host.ID, host.Containers[0].ID)
|
||||
resourceID := DockerContainerResourceID(host.ID, host.Containers[0].ID, "")
|
||||
alertID := fmt.Sprintf("docker-container-health-%s", resourceID)
|
||||
if testHasActiveAlert(t, m, alertID) {
|
||||
t.Fatal("expected no health alert for container with none health")
|
||||
@@ -13235,7 +13492,7 @@ func TestDockerContainerHealthAlert(t *testing.T) {
|
||||
|
||||
m.CheckDockerHost(host)
|
||||
|
||||
resourceID := DockerResourceID(host.ID, host.Containers[0].ID)
|
||||
resourceID := DockerContainerResourceID(host.ID, host.Containers[0].ID, "")
|
||||
alertID := fmt.Sprintf("docker-container-health-%s", resourceID)
|
||||
if testHasActiveAlert(t, m, alertID) {
|
||||
t.Fatal("expected no health alert for starting container")
|
||||
@@ -13262,7 +13519,7 @@ func TestDockerContainerHealthAlert(t *testing.T) {
|
||||
|
||||
m.CheckDockerHost(host)
|
||||
|
||||
resourceID := DockerResourceID(host.ID, host.Containers[0].ID)
|
||||
resourceID := DockerContainerResourceID(host.ID, host.Containers[0].ID, "")
|
||||
alertID := fmt.Sprintf("docker-container-health-%s", resourceID)
|
||||
alert, exists := testLookupActiveAlert(t, m, alertID)
|
||||
if !exists {
|
||||
@@ -13302,7 +13559,7 @@ func TestDockerContainerHealthAlert(t *testing.T) {
|
||||
|
||||
m.CheckDockerHost(host)
|
||||
|
||||
resourceID := DockerResourceID(host.ID, host.Containers[0].ID)
|
||||
resourceID := DockerContainerResourceID(host.ID, host.Containers[0].ID, "")
|
||||
alertID := fmt.Sprintf("docker-container-health-%s", resourceID)
|
||||
alert, exists := testLookupActiveAlert(t, m, alertID)
|
||||
if !exists {
|
||||
@@ -13333,7 +13590,7 @@ func TestDockerContainerHealthAlert(t *testing.T) {
|
||||
|
||||
m.CheckDockerHost(host)
|
||||
|
||||
resourceID := DockerResourceID(host.ID, host.Containers[0].ID)
|
||||
resourceID := DockerContainerResourceID(host.ID, host.Containers[0].ID, "")
|
||||
alertID := fmt.Sprintf("docker-container-health-%s", resourceID)
|
||||
if !testHasActiveAlert(t, m, alertID) {
|
||||
t.Fatal("expected running unhealthy container to raise a health alert")
|
||||
@@ -13376,7 +13633,7 @@ func TestDockerContainerHealthAlert(t *testing.T) {
|
||||
|
||||
m.CheckDockerHost(hostUnhealthy)
|
||||
|
||||
resourceID := DockerResourceID(hostID, containerID)
|
||||
resourceID := DockerContainerResourceID(hostID, containerID, "")
|
||||
alertID := fmt.Sprintf("docker-container-health-%s", resourceID)
|
||||
if !testHasActiveAlert(t, m, alertID) {
|
||||
t.Fatal("expected health alert to be raised")
|
||||
@@ -13427,7 +13684,7 @@ func TestDockerContainerOOMKillAlert(t *testing.T) {
|
||||
|
||||
m.CheckDockerHost(host)
|
||||
|
||||
resourceID := DockerResourceID(host.ID, host.Containers[0].ID)
|
||||
resourceID := DockerContainerResourceID(host.ID, host.Containers[0].ID, "")
|
||||
alertID := fmt.Sprintf("docker-container-oom-%s", resourceID)
|
||||
if testHasActiveAlert(t, m, alertID) {
|
||||
t.Fatal("expected no OOM alert for running container")
|
||||
@@ -13454,7 +13711,7 @@ func TestDockerContainerOOMKillAlert(t *testing.T) {
|
||||
|
||||
m.CheckDockerHost(host)
|
||||
|
||||
resourceID := DockerResourceID(host.ID, host.Containers[0].ID)
|
||||
resourceID := DockerContainerResourceID(host.ID, host.Containers[0].ID, "")
|
||||
alertID := fmt.Sprintf("docker-container-oom-%s", resourceID)
|
||||
if testHasActiveAlert(t, m, alertID) {
|
||||
t.Fatal("expected no OOM alert for container with exit code 1")
|
||||
@@ -13478,7 +13735,7 @@ func TestDockerContainerOOMKillAlert(t *testing.T) {
|
||||
|
||||
m.CheckDockerHost(host)
|
||||
|
||||
resourceID := DockerResourceID(host.ID, host.Containers[0].ID)
|
||||
resourceID := DockerContainerResourceID(host.ID, host.Containers[0].ID, "")
|
||||
if testHasActiveAlert(t, m, fmt.Sprintf("docker-container-oom-%s", resourceID)) {
|
||||
t.Fatal("exit code 137 alone must not create an OOM alert")
|
||||
}
|
||||
@@ -13508,7 +13765,7 @@ func TestDockerContainerOOMKillAlert(t *testing.T) {
|
||||
|
||||
m.CheckDockerHost(host)
|
||||
|
||||
resourceID := DockerResourceID(host.ID, host.Containers[0].ID)
|
||||
resourceID := DockerContainerResourceID(host.ID, host.Containers[0].ID, "")
|
||||
alertID := fmt.Sprintf("docker-container-oom-%s", resourceID)
|
||||
alert, exists := testLookupActiveAlert(t, m, alertID)
|
||||
if !exists {
|
||||
@@ -13552,7 +13809,7 @@ func TestDockerContainerOOMKillAlert(t *testing.T) {
|
||||
|
||||
m.CheckDockerHost(host)
|
||||
|
||||
resourceID := DockerResourceID(host.ID, host.Containers[0].ID)
|
||||
resourceID := DockerContainerResourceID(host.ID, host.Containers[0].ID, "")
|
||||
alertID := fmt.Sprintf("docker-container-oom-%s", resourceID)
|
||||
alert, exists := testLookupActiveAlert(t, m, alertID)
|
||||
if !exists {
|
||||
@@ -13588,7 +13845,7 @@ func TestDockerContainerOOMKillAlert(t *testing.T) {
|
||||
// First check - should create alert
|
||||
m.CheckDockerHost(host)
|
||||
|
||||
resourceID := DockerResourceID(hostID, containerID)
|
||||
resourceID := DockerContainerResourceID(hostID, containerID, "")
|
||||
alertID := fmt.Sprintf("docker-container-oom-%s", resourceID)
|
||||
alert1, exists := testLookupActiveAlert(t, m, alertID)
|
||||
if !exists {
|
||||
@@ -13633,7 +13890,7 @@ func TestDockerContainerOOMKillAlert(t *testing.T) {
|
||||
|
||||
m.CheckDockerHost(hostOOM)
|
||||
|
||||
resourceID := DockerResourceID(hostID, containerID)
|
||||
resourceID := DockerContainerResourceID(hostID, containerID, "")
|
||||
alertID := fmt.Sprintf("docker-container-oom-%s", resourceID)
|
||||
if !testHasActiveAlert(t, m, alertID) {
|
||||
t.Fatal("expected OOM alert to be raised")
|
||||
@@ -13687,7 +13944,7 @@ func TestDockerContainerOOMKillAlert(t *testing.T) {
|
||||
|
||||
m.CheckDockerHost(hostOOM)
|
||||
|
||||
resourceID := DockerResourceID(hostID, containerID)
|
||||
resourceID := DockerContainerResourceID(hostID, containerID, "")
|
||||
alertID := fmt.Sprintf("docker-container-oom-%s", resourceID)
|
||||
if !testHasActiveAlert(t, m, alertID) {
|
||||
t.Fatal("expected OOM alert to be raised")
|
||||
@@ -13738,7 +13995,7 @@ func TestDockerContainerRestartLoopAlert(t *testing.T) {
|
||||
|
||||
m.CheckDockerHost(host)
|
||||
|
||||
resourceID := DockerResourceID(host.ID, host.Containers[0].ID)
|
||||
resourceID := DockerContainerResourceID(host.ID, host.Containers[0].ID, "")
|
||||
alertID := fmt.Sprintf("docker-container-restart-loop-%s", resourceID)
|
||||
if testHasActiveAlert(t, m, alertID) {
|
||||
t.Fatal("expected no restart loop alert on first check (just initializes tracking)")
|
||||
@@ -13786,7 +14043,7 @@ func TestDockerContainerRestartLoopAlert(t *testing.T) {
|
||||
// Third check - still same restart count
|
||||
m.CheckDockerHost(host)
|
||||
|
||||
resourceID := DockerResourceID(hostID, containerID)
|
||||
resourceID := DockerContainerResourceID(hostID, containerID, "")
|
||||
alertID := fmt.Sprintf("docker-container-restart-loop-%s", resourceID)
|
||||
if testHasActiveAlert(t, m, alertID) {
|
||||
t.Fatal("expected no restart loop alert for stable container")
|
||||
@@ -13827,7 +14084,7 @@ func TestDockerContainerRestartLoopAlert(t *testing.T) {
|
||||
host.Containers[0].RestartCount = 3
|
||||
m.CheckDockerHost(host)
|
||||
|
||||
resourceID := DockerResourceID(hostID, containerID)
|
||||
resourceID := DockerContainerResourceID(hostID, containerID, "")
|
||||
alertID := fmt.Sprintf("docker-container-restart-loop-%s", resourceID)
|
||||
if testHasActiveAlert(t, m, alertID) {
|
||||
t.Fatal("expected no restart loop alert when restarts <= threshold")
|
||||
@@ -13873,7 +14130,7 @@ func TestDockerContainerRestartLoopAlert(t *testing.T) {
|
||||
host.Containers[0].RestartCount = 4
|
||||
m.CheckDockerHost(host)
|
||||
|
||||
resourceID := DockerResourceID(hostID, containerID)
|
||||
resourceID := DockerContainerResourceID(hostID, containerID, "")
|
||||
alertID := fmt.Sprintf("docker-container-restart-loop-%s", resourceID)
|
||||
alert, exists := testLookupActiveAlert(t, m, alertID)
|
||||
if !exists {
|
||||
@@ -13909,7 +14166,7 @@ func TestDockerContainerRestartLoopAlert(t *testing.T) {
|
||||
|
||||
hostID := "host-restart-5"
|
||||
containerID := "container-5"
|
||||
resourceID := DockerResourceID(hostID, containerID)
|
||||
resourceID := DockerContainerResourceID(hostID, containerID, "")
|
||||
|
||||
// Manually set up a restart loop state
|
||||
m.mu.Lock()
|
||||
@@ -13989,7 +14246,7 @@ func TestDockerContainerRestartLoopAlert(t *testing.T) {
|
||||
// First check - initializes
|
||||
m.CheckDockerHost(host)
|
||||
|
||||
resourceID := DockerResourceID(hostID, containerID)
|
||||
resourceID := DockerContainerResourceID(hostID, containerID, "")
|
||||
alertID := fmt.Sprintf("docker-container-restart-loop-%s", resourceID)
|
||||
|
||||
// Restart 1
|
||||
@@ -14042,7 +14299,7 @@ func TestDockerContainerRestartLoopAlert(t *testing.T) {
|
||||
host.Containers[0].RestartCount = 5
|
||||
m.CheckDockerHost(host)
|
||||
|
||||
resourceID := DockerResourceID(hostID, containerID)
|
||||
resourceID := DockerContainerResourceID(hostID, containerID, "")
|
||||
alertID := fmt.Sprintf("docker-container-restart-loop-%s", resourceID)
|
||||
alert1, exists := testLookupActiveAlert(t, m, alertID)
|
||||
if !exists {
|
||||
@@ -20293,7 +20550,7 @@ func TestDockerContainerOverrideSurvivesContainerRecreate(t *testing.T) {
|
||||
}
|
||||
}
|
||||
stateAlertID := func(hostID, containerID string) string {
|
||||
return fmt.Sprintf("docker-container-state-%s", DockerResourceID(hostID, containerID))
|
||||
return fmt.Sprintf("docker-container-state-%s", DockerContainerResourceID(hostID, containerID, ""))
|
||||
}
|
||||
|
||||
// Subtests keep the two managers in separate lifetimes: newTestManager
|
||||
@@ -20345,13 +20602,13 @@ func TestDockerContainerOverrideLegacyIDKeyStillHonoured(t *testing.T) {
|
||||
}
|
||||
|
||||
m.mu.Lock()
|
||||
m.config.Overrides[DockerResourceID(host.ID, container.ID)] = ThresholdConfig{Disabled: true}
|
||||
m.config.Overrides[DockerContainerResourceID(host.ID, container.ID, "")] = ThresholdConfig{Disabled: true}
|
||||
m.mu.Unlock()
|
||||
|
||||
m.CheckDockerHost(host)
|
||||
m.CheckDockerHost(host)
|
||||
|
||||
alertID := fmt.Sprintf("docker-container-state-%s", DockerResourceID(host.ID, container.ID))
|
||||
alertID := fmt.Sprintf("docker-container-state-%s", DockerContainerResourceID(host.ID, container.ID, ""))
|
||||
if testHasActiveAlert(t, m, alertID) {
|
||||
t.Fatalf("expected legacy ID-keyed override to still disable container alerts")
|
||||
}
|
||||
|
||||
@@ -16,6 +16,27 @@ func buildCanonicalStateID(resourceID, specID string) string {
|
||||
return resourceID + canonicalStateSeparator + specID
|
||||
}
|
||||
|
||||
// splitCanonicalStateID splits a canonical state ID into its resource and
|
||||
// spec parts. A resource ID can itself contain the separator (a Docker host
|
||||
// disambiguated as "<base>::<suffix>"), so an ID that ends with the alert's
|
||||
// spec ID splits before it, and one that starts with the alert's resource ID
|
||||
// splits after it. Only without either does it split at the first separator.
|
||||
// The spec comes first: an alert restored from before this rule can carry a
|
||||
// resource ID that was cut short at the first separator.
|
||||
func splitCanonicalStateID(id, resourceID, specID string) (resource, spec string, ok bool) {
|
||||
if specID = strings.TrimSpace(specID); specID != "" {
|
||||
if prefix, found := strings.CutSuffix(id, canonicalStateSeparator+specID); found && prefix != "" {
|
||||
return prefix, specID, true
|
||||
}
|
||||
}
|
||||
if resourceID = strings.TrimSpace(resourceID); resourceID != "" {
|
||||
if rest, found := strings.CutPrefix(id, resourceID+canonicalStateSeparator); found && rest != "" {
|
||||
return resourceID, rest, true
|
||||
}
|
||||
}
|
||||
return strings.Cut(id, canonicalStateSeparator)
|
||||
}
|
||||
|
||||
func canonicalMetricSpecID(resourceID, metric string) string {
|
||||
if resourceID == "" || metric == "" {
|
||||
return ""
|
||||
@@ -149,17 +170,24 @@ func deriveCanonicalIdentity(alert *Alert) canonicalIdentity {
|
||||
}
|
||||
}
|
||||
|
||||
// canonicalSpecIDHint is the spec ID an alert already records, on the alert
|
||||
// or in its metadata.
|
||||
func canonicalSpecIDHint(alert *Alert) string {
|
||||
if alert.CanonicalSpecID != "" {
|
||||
return alert.CanonicalSpecID
|
||||
}
|
||||
specID, _ := alert.Metadata["canonicalSpecID"].(string)
|
||||
return specID
|
||||
}
|
||||
|
||||
func inferCanonicalResourceIDFromLegacyAlert(alert *Alert) string {
|
||||
if alert == nil {
|
||||
return ""
|
||||
}
|
||||
|
||||
id := alert.ID
|
||||
if strings.Contains(id, canonicalStateSeparator) {
|
||||
parts := strings.SplitN(id, canonicalStateSeparator, 2)
|
||||
if len(parts) == 2 {
|
||||
return parts[0]
|
||||
}
|
||||
if resource, _, ok := splitCanonicalStateID(id, alert.ResourceID, canonicalSpecIDHint(alert)); ok {
|
||||
return resource
|
||||
}
|
||||
switch {
|
||||
case strings.HasPrefix(id, "host-offline-"):
|
||||
@@ -230,11 +258,8 @@ func inferCanonicalSpecIDFromLegacyAlert(alert *Alert) string {
|
||||
return ""
|
||||
}
|
||||
id := alert.ID
|
||||
if strings.Contains(id, canonicalStateSeparator) {
|
||||
parts := strings.SplitN(id, canonicalStateSeparator, 2)
|
||||
if len(parts) == 2 {
|
||||
return parts[1]
|
||||
}
|
||||
if _, spec, ok := splitCanonicalStateID(id, alert.ResourceID, canonicalSpecIDHint(alert)); ok {
|
||||
return spec
|
||||
}
|
||||
resourceID := inferCanonicalResourceIDFromLegacyAlert(alert)
|
||||
switch {
|
||||
|
||||
@@ -48,7 +48,7 @@ func TestCleanupContinuingDockerUpdateKeepsOccurrence(t *testing.T) {
|
||||
LastChecked: time.Now().Add(-6 * time.Hour),
|
||||
}
|
||||
container := models.DockerContainer{ID: "retention-container", Name: "web", Image: "mongo:7", UpdateStatus: pending}
|
||||
resourceID := DockerResourceID(host.ID, container.ID)
|
||||
resourceID := DockerContainerResourceID(host.ID, container.ID, "")
|
||||
trackingKey := dockerUpdateTrackingKey(host, container)
|
||||
first := time.Now().Add(-25 * time.Hour)
|
||||
m.mu.Lock()
|
||||
|
||||
@@ -284,41 +284,39 @@ func (m *Manager) applyGlobalOfflineSettingsLocked() {
|
||||
}
|
||||
}
|
||||
|
||||
// dockerAlertResourcePath returns the portion of the alert's resource ID after
|
||||
// the "docker:" scheme, matching the IDs built by DockerResourceID and
|
||||
// DockerServiceResourceID. Canonical alerts are stored under
|
||||
// dockerAlertResourcePath splits the alert's resource ID after the "docker:"
|
||||
// scheme into the host and the child path, matching the IDs built by
|
||||
// DockerHostResourceID ("<host>"), DockerContainerResourceID
|
||||
// ("<host>/<container ID>" or "<host>/name:<name>") and
|
||||
// DockerServiceResourceID ("<host>/service/<service ID>" or
|
||||
// "<host>/service/name:<name>"). child is empty for a host's own alert. A
|
||||
// name can contain anything, so callers classify by the first segment of
|
||||
// child, never by searching the whole path. Canonical alerts are stored under
|
||||
// "<resourceID>::<specID>" state IDs, so the legacy "docker-container-" /
|
||||
// "docker-service-" alert-ID prefixes never match them.
|
||||
func dockerAlertResourcePath(alert *Alert) (string, bool) {
|
||||
func dockerAlertResourcePath(alert *Alert) (host, child string, ok bool) {
|
||||
if alert == nil {
|
||||
return "", false
|
||||
return "", "", false
|
||||
}
|
||||
resourceID := strings.TrimSpace(alert.ResourceID)
|
||||
if !strings.HasPrefix(resourceID, "docker:") {
|
||||
return "", false
|
||||
path, found := strings.CutPrefix(strings.TrimSpace(alert.ResourceID), "docker:")
|
||||
if !found {
|
||||
return "", "", false
|
||||
}
|
||||
return strings.TrimPrefix(resourceID, "docker:"), true
|
||||
host, child, _ = strings.Cut(path, "/")
|
||||
return host, child, true
|
||||
}
|
||||
|
||||
func isDockerContainerAlert(alert *Alert) bool {
|
||||
path, ok := dockerAlertResourcePath(alert)
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
// "hostID/containerID" has a path separator; host-level IDs ("hostID") do
|
||||
// not, and services use "hostID/service/serviceID".
|
||||
return strings.Contains(path, "/") && !strings.Contains(path, "/service/")
|
||||
_, child, ok := dockerAlertResourcePath(alert)
|
||||
return ok && child != "" && !strings.HasPrefix(child, "service/")
|
||||
}
|
||||
|
||||
func isDockerServiceAlert(alert *Alert) bool {
|
||||
if alert != nil && strings.HasPrefix(strings.TrimSpace(alert.ResourceID), "docker-service:") {
|
||||
return true
|
||||
}
|
||||
path, ok := dockerAlertResourcePath(alert)
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
return strings.Contains(path, "/service/")
|
||||
_, child, ok := dockerAlertResourcePath(alert)
|
||||
return ok && strings.HasPrefix(child, "service/")
|
||||
}
|
||||
|
||||
func alertPrimaryResourceType(alert *Alert) string {
|
||||
|
||||
+79
-22
@@ -80,22 +80,43 @@ func (m *Manager) lookupDockerContainerOverrideNoLock(hostID, containerName, leg
|
||||
return ThresholdConfig{}, false
|
||||
}
|
||||
|
||||
// DockerResourceID builds a stable identifier for Docker container alerts.
|
||||
// Patrol scope resolution registers these forms as known aliases so an
|
||||
// alert's resource ID resolves to the collected container or host.
|
||||
func DockerResourceID(hostID, containerID string) string {
|
||||
// dockerNameReferencePrefix marks the name of a Docker container or Swarm
|
||||
// service that has no ID in the slot of an alert reference that otherwise
|
||||
// holds the ID. Docker and Swarm IDs never contain a colon, so a name
|
||||
// reference can never equal an ID reference or the host's own reference.
|
||||
const dockerNameReferencePrefix = "name:"
|
||||
|
||||
// DockerHostResourceID builds the reference of a Docker host's own alerts
|
||||
// ("docker:<host ID>"). Only the host's connectivity alert uses it. Returns ""
|
||||
// without a host ID.
|
||||
func DockerHostResourceID(hostID string) string {
|
||||
hostID = strings.TrimSpace(hostID)
|
||||
containerID = strings.TrimSpace(containerID)
|
||||
if containerID == "" {
|
||||
if hostID == "" {
|
||||
return "docker:unknown"
|
||||
if hostID == "" {
|
||||
return ""
|
||||
}
|
||||
return "docker:" + hostID
|
||||
}
|
||||
|
||||
// DockerContainerResourceID builds the reference of a Docker container's
|
||||
// alerts: "docker:<host ID>/<container ID>". A container reported without an
|
||||
// ID is referenced by its name, "docker:<host ID>/name:<name>", so it never
|
||||
// shares its host's reference or another container's. Returns "" for a
|
||||
// container with neither, which has no identity to alert under. Patrol scope
|
||||
// resolution registers the same forms as aliases of the collected container.
|
||||
func DockerContainerResourceID(hostID, containerID, containerName string) string {
|
||||
hostID = strings.TrimSpace(hostID)
|
||||
ref := strings.TrimSpace(containerID)
|
||||
if ref == "" {
|
||||
name := strings.TrimLeft(strings.TrimSpace(containerName), "/")
|
||||
if hostID == "" || name == "" {
|
||||
return ""
|
||||
}
|
||||
return fmt.Sprintf("docker:%s", hostID)
|
||||
ref = dockerNameReferencePrefix + name
|
||||
}
|
||||
if hostID == "" {
|
||||
return fmt.Sprintf("docker:container/%s", containerID)
|
||||
return fmt.Sprintf("docker:container/%s", ref)
|
||||
}
|
||||
return fmt.Sprintf("docker:%s/%s", hostID, containerID)
|
||||
return fmt.Sprintf("docker:%s/%s", hostID, ref)
|
||||
}
|
||||
|
||||
func normalizeDockerUpdateTrackingPart(part string) string {
|
||||
@@ -164,17 +185,18 @@ func dockerServiceDisplayName(service models.DockerService) string {
|
||||
return serviceID
|
||||
}
|
||||
|
||||
// DockerServiceResourceID builds a stable identifier for Docker Swarm service
|
||||
// alerts. Patrol scope resolution registers this form as a known alias on the
|
||||
// owning Docker host record so a service alert's resource ID resolves there.
|
||||
// DockerServiceResourceID builds the reference of a Docker Swarm service's
|
||||
// alerts: "docker:<host ID>/service/<service ID>". A service reported without
|
||||
// an ID is referenced by its normalized name, "docker:<host ID>/service/name:<name>",
|
||||
// so it never shares another service's ID reference. Returns "" for a service
|
||||
// with neither, which has no identity to alert under. Patrol scope resolution
|
||||
// registers the same forms as aliases on the owning Docker host record so a
|
||||
// service alert's resource ID resolves there.
|
||||
func DockerServiceResourceID(hostID, serviceID, serviceName string) string {
|
||||
hostID = strings.TrimSpace(hostID)
|
||||
normalizedServiceID := strings.TrimSpace(serviceID)
|
||||
if normalizedServiceID == "" {
|
||||
name := strings.TrimSpace(serviceName)
|
||||
if name == "" {
|
||||
name = "service"
|
||||
}
|
||||
builder := strings.Builder{}
|
||||
for _, r := range strings.ToLower(name) {
|
||||
switch {
|
||||
@@ -190,11 +212,12 @@ func DockerServiceResourceID(hostID, serviceID, serviceName string) string {
|
||||
}
|
||||
normalizedServiceID = strings.Trim(builder.String(), "-_")
|
||||
if normalizedServiceID == "" {
|
||||
normalizedServiceID = "service"
|
||||
return ""
|
||||
}
|
||||
if len(normalizedServiceID) > 32 {
|
||||
normalizedServiceID = normalizedServiceID[:32]
|
||||
}
|
||||
normalizedServiceID = dockerNameReferencePrefix + normalizedServiceID
|
||||
}
|
||||
if hostID == "" {
|
||||
return fmt.Sprintf("docker-service:%s", normalizedServiceID)
|
||||
@@ -272,8 +295,12 @@ func (m *Manager) CheckDockerHost(host models.DockerHost) {
|
||||
seen := make(map[string]struct{}, len(host.Containers)+len(host.Services))
|
||||
seenUpdateTracking := make(map[string]struct{}, len(host.Containers))
|
||||
for _, container := range host.Containers {
|
||||
resourceID := DockerContainerResourceID(host.ID, container.ID, container.Name)
|
||||
if resourceID == "" {
|
||||
// Neither an ID nor a name: nothing identifies its alerts.
|
||||
continue
|
||||
}
|
||||
containerName := dockerContainerDisplayName(container)
|
||||
resourceID := DockerResourceID(host.ID, container.ID)
|
||||
updateTrackingKey := dockerUpdateTrackingKey(host, container)
|
||||
|
||||
if matchesDockerIgnoredPrefix(containerName, container.ID, ignoredPrefixes) {
|
||||
@@ -301,6 +328,9 @@ func (m *Manager) CheckDockerHost(host models.DockerHost) {
|
||||
|
||||
for _, service := range host.Services {
|
||||
resourceID := DockerServiceResourceID(host.ID, service.ID, service.Name)
|
||||
if resourceID == "" {
|
||||
continue
|
||||
}
|
||||
seen[resourceID] = struct{}{}
|
||||
m.evaluateDockerService(host, service, resourceID)
|
||||
}
|
||||
@@ -656,7 +686,7 @@ func (m *Manager) HandleDockerHostOnline(host models.DockerHost) {
|
||||
return
|
||||
}
|
||||
|
||||
resourceID := fmt.Sprintf("docker:%s", strings.TrimSpace(host.ID))
|
||||
resourceID := DockerHostResourceID(host.ID)
|
||||
alertID := canonicalConnectivityStateID(resourceID)
|
||||
|
||||
m.mu.Lock()
|
||||
@@ -697,7 +727,7 @@ func (m *Manager) HandleDockerHostOffline(host models.DockerHost) {
|
||||
_, disableDockerHostsOffline := m.alertPolicyTypeSwitchesNoLock("docker-host")
|
||||
m.mu.RUnlock()
|
||||
|
||||
resourceID := fmt.Sprintf("docker:%s", strings.TrimSpace(host.ID))
|
||||
resourceID := DockerHostResourceID(host.ID)
|
||||
alertID := canonicalConnectivityStateID(resourceID)
|
||||
instanceName := dockerInstanceName(host)
|
||||
nodeName := strings.TrimSpace(host.Hostname)
|
||||
@@ -1400,6 +1430,16 @@ func (m *Manager) checkDockerContainerImageUpdate(host models.DockerHost, contai
|
||||
active.Type == "docker-container-update" && active.ResourceID == resourceID &&
|
||||
!active.StartTime.IsZero() {
|
||||
firstSeen, exists = active.StartTime, true
|
||||
} else if strings.TrimSpace(container.ID) == "" {
|
||||
// A container without an ID used to record its pending update under
|
||||
// its host's reference. Keep that age; the host's report cleanup then
|
||||
// clears the old occurrence.
|
||||
legacyRef := DockerHostResourceID(host.ID)
|
||||
if legacy, ok := m.getActiveAlertNoLock(buildCanonicalStateID(legacyRef, legacyRef+"-image-update")); ok &&
|
||||
isLegacyDockerHostContainerAlert(legacy, host) && legacy.Type == "docker-container-update" &&
|
||||
alertMetadataString(legacy, "containerName") == containerName && !legacy.StartTime.IsZero() {
|
||||
firstSeen, exists = legacy.StartTime, true
|
||||
}
|
||||
}
|
||||
if !exists {
|
||||
firstSeen = time.Now()
|
||||
@@ -1480,6 +1520,19 @@ func (m *Manager) cleanupDockerContainerAlerts(host models.DockerHost, seen map[
|
||||
m.cleanupDockerContainerAlertsWithTracking(host, seen, nil)
|
||||
}
|
||||
|
||||
// isLegacyDockerHostContainerAlert reports an alert that a container without
|
||||
// an ID on this host raised under the host's own reference, before such
|
||||
// containers had a reference of their own. Only the host's connectivity alert
|
||||
// belongs there, so no producer refreshes these. The recorded host and empty
|
||||
// container ID must match too: an alert restored with a resource ID cut short
|
||||
// at a "::" in another host's ID ("<base>::<suffix>") can carry this host's
|
||||
// reference.
|
||||
func isLegacyDockerHostContainerAlert(alert *Alert, host models.DockerHost) bool {
|
||||
hostRef := DockerHostResourceID(host.ID)
|
||||
return alert != nil && hostRef != "" && alert.ResourceID == hostRef && alert.Type != "docker-host-offline" &&
|
||||
alertMetadataString(alert, "hostId") == strings.TrimSpace(host.ID) && alertMetadataString(alert, "containerId") == ""
|
||||
}
|
||||
|
||||
func (m *Manager) cleanupDockerContainerAlertsWithTracking(host models.DockerHost, seen map[string]struct{}, seenUpdateTracking map[string]struct{}) {
|
||||
prefix := fmt.Sprintf("docker:%s/", strings.TrimSpace(host.ID))
|
||||
updateTrackingPrefix := dockerUpdateTrackingHostPrefix(host)
|
||||
@@ -1488,6 +1541,10 @@ func (m *Manager) cleanupDockerContainerAlertsWithTracking(host models.DockerHos
|
||||
toClear := make([]string, 0)
|
||||
for storageKey, alert := range m.activeAlerts {
|
||||
alertID := effectiveAlertID(alert, storageKey)
|
||||
if isLegacyDockerHostContainerAlert(alert, host) {
|
||||
toClear = append(toClear, alertID)
|
||||
continue
|
||||
}
|
||||
if !strings.HasPrefix(alert.ResourceID, prefix) {
|
||||
continue
|
||||
}
|
||||
@@ -1540,7 +1597,7 @@ func (m *Manager) clearDockerHostContainerAlerts(host models.DockerHost) {
|
||||
toClear := make([]string, 0)
|
||||
for storageKey, alert := range m.activeAlerts {
|
||||
alertID := effectiveAlertID(alert, storageKey)
|
||||
if strings.HasPrefix(alert.ResourceID, prefix) {
|
||||
if strings.HasPrefix(alert.ResourceID, prefix) || isLegacyDockerHostContainerAlert(alert, host) {
|
||||
toClear = append(toClear, alertID)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -75,7 +75,7 @@ func MigrateDockerContainerOverrideKeys(config *AlertConfig, resources []unified
|
||||
|
||||
legacyKey := ""
|
||||
if containerID := strings.TrimSpace(resource.Docker.ContainerID); containerID != "" {
|
||||
legacyKey = DockerResourceID(hostID, containerID)
|
||||
legacyKey = DockerContainerResourceID(hostID, containerID, "")
|
||||
live[legacyKey] = struct{}{}
|
||||
}
|
||||
if legacyKey != "" && legacyKey != stableKey {
|
||||
|
||||
@@ -1287,7 +1287,7 @@ func TestCheckDockerContainerStateAnnotatesCanonicalSpecMetadata(t *testing.T) {
|
||||
m.CheckDockerHost(host)
|
||||
m.CheckDockerHost(host)
|
||||
|
||||
resourceID := DockerResourceID(host.ID, "container-1")
|
||||
resourceID := DockerContainerResourceID(host.ID, "container-1", "")
|
||||
alert := activeAlert(t, m, "docker-container-state-"+resourceID)
|
||||
if got := alert.Metadata["canonicalAlertKind"]; got != "discrete-state" {
|
||||
t.Fatalf("canonicalAlertKind = %v, want discrete-state", got)
|
||||
|
||||
@@ -469,7 +469,7 @@ func TestCheckDockerContainerImageUpdatePreservesDelayAcrossHostIDChange(t *test
|
||||
newHost.ID = "docker-host-new"
|
||||
newHost.DisplayName = "New Docker Host"
|
||||
|
||||
oldResourceID := DockerResourceID(oldHost.ID, container.ID)
|
||||
oldResourceID := DockerContainerResourceID(oldHost.ID, container.ID, "")
|
||||
m.checkDockerContainerImageUpdate(oldHost, container, oldResourceID, "web", "docker-instance", "docker.local")
|
||||
|
||||
firstSeen := time.Now().Add(-25 * time.Hour)
|
||||
@@ -479,7 +479,7 @@ func TestCheckDockerContainerImageUpdatePreservesDelayAcrossHostIDChange(t *test
|
||||
m.dockerUpdateFirstSeenByIdentity[trackingKey] = firstSeen
|
||||
m.mu.Unlock()
|
||||
|
||||
newResourceID := DockerResourceID(newHost.ID, container.ID)
|
||||
newResourceID := DockerContainerResourceID(newHost.ID, container.ID, "")
|
||||
m.checkDockerContainerImageUpdate(newHost, container, newResourceID, "web", "docker-instance", "docker.local")
|
||||
|
||||
alertID := "docker-container-update-" + newResourceID
|
||||
@@ -639,7 +639,7 @@ func TestUpdateConfigClearsDockerContainerUpdateAlertsWhenDisabled(t *testing.T)
|
||||
Name: "/frontend",
|
||||
Image: "nginx:latest",
|
||||
}
|
||||
resourceID := DockerResourceID(host.ID, container.ID)
|
||||
resourceID := DockerContainerResourceID(host.ID, container.ID, "")
|
||||
firstSeen := time.Now().Add(-48 * time.Hour)
|
||||
_, canonicalAlertID, trackingKey := seedDockerUpdateAlert(t, m, host, container, resourceID, firstSeen)
|
||||
|
||||
@@ -677,7 +677,7 @@ func TestUpdateConfigKeepsDockerContainerUpdateAlertsWhenStillEnabled(t *testing
|
||||
Name: "/api",
|
||||
Image: "ghcr.io/example/api:latest",
|
||||
}
|
||||
resourceID := DockerResourceID(host.ID, container.ID)
|
||||
resourceID := DockerContainerResourceID(host.ID, container.ID, "")
|
||||
firstSeen := time.Now().Add(-30 * time.Hour)
|
||||
_, canonicalAlertID, trackingKey := seedDockerUpdateAlert(t, m, host, container, resourceID, firstSeen)
|
||||
|
||||
@@ -723,7 +723,7 @@ func TestEvaluateDockerContainerClearsUpdateAlertWhenOverrideDisabled(t *testing
|
||||
Name: "/worker",
|
||||
Image: "ghcr.io/example/worker:latest",
|
||||
}
|
||||
resourceID := DockerResourceID(host.ID, container.ID)
|
||||
resourceID := DockerContainerResourceID(host.ID, container.ID, "")
|
||||
firstSeen := time.Now().Add(-36 * time.Hour)
|
||||
alertID, canonicalAlertID, trackingKey := seedDockerUpdateAlert(t, m, host, container, resourceID, firstSeen)
|
||||
|
||||
@@ -884,7 +884,7 @@ func TestDockerUpdateRestartRestoresActivePendingAge(t *testing.T) {
|
||||
}
|
||||
container := models.DockerContainer{ID: "container", Name: "web", Image: "mongo:7", State: "running", UpdateStatus: pending}
|
||||
host := models.DockerHost{ID: "host", AgentID: "agent", Hostname: "host", Containers: []models.DockerContainer{container}}
|
||||
resourceID := DockerResourceID(host.ID, container.ID)
|
||||
resourceID := DockerContainerResourceID(host.ID, container.ID, "")
|
||||
canonicalID := buildCanonicalStateID(resourceID, resourceID+"-image-update")
|
||||
trackingKey := dockerUpdateTrackingKey(host, container)
|
||||
first := time.Now().Add(-time.Duration(delay+2) * time.Hour)
|
||||
|
||||
@@ -153,7 +153,10 @@ func TestProxmoxAlertTimelineUsesCanonicalHistoryIdentity(t *testing.T) {
|
||||
// Docker host, Swarm service and sub-resource alerts (a ZFS pool and device,
|
||||
// a host filesystem) carry references the registry has no resource for. Their
|
||||
// lifecycle must reach the owner's canonical history, so the producers' ref
|
||||
// shapes are pinned here against the real alert manager.
|
||||
// shapes are pinned here against the real alert manager. A container or
|
||||
// service reported without an ID alerts under its own name reference, so it
|
||||
// joins neither its host's history nor that of the service whose ID it is
|
||||
// named like.
|
||||
func TestOwnerAlertTimelinesUseCanonicalHistoryIdentity(t *testing.T) {
|
||||
store, err := unifiedresources.NewSQLiteResourceStore(t.TempDir(), "default")
|
||||
require.NoError(t, err)
|
||||
@@ -169,8 +172,10 @@ func TestOwnerAlertTimelinesUseCanonicalHistoryIdentity(t *testing.T) {
|
||||
manager.UpdateConfig(config)
|
||||
now := time.Now()
|
||||
dockerHost := models.DockerHost{ID: "dh-7f3a", Hostname: "tower", DisplayName: "tower", Status: "online", LastSeen: now,
|
||||
Swarm: &models.DockerSwarmInfo{NodeID: "node-a", NodeRole: "manager", LocalState: "active", ControlAvailable: true, ClusterID: "swarm-1"},
|
||||
Services: []models.DockerService{{ID: "x7k2m9q4w1e8r5t3y6u0i2o4p", Name: "web", DesiredTasks: 2, RunningTasks: 0}}}
|
||||
Swarm: &models.DockerSwarmInfo{NodeID: "node-a", NodeRole: "manager", LocalState: "active", ControlAvailable: true, ClusterID: "swarm-1"},
|
||||
Containers: []models.DockerContainer{{Name: "sidecar", State: "running", Health: "unhealthy"}},
|
||||
Services: []models.DockerService{{ID: "x7k2m9q4w1e8r5t3y6u0i2o4p", Name: "web", DesiredTasks: 2, RunningTasks: 0},
|
||||
{Name: "X7K2M9Q4W1E8R5T3Y6U0I2O4P", DesiredTasks: 2, RunningTasks: 0}}}
|
||||
storage := models.Storage{ID: "lab-pve1-local-zfs", Name: "local-zfs", Node: "pve1", Instance: "lab", Type: "zfspool", Status: "available", Total: 100, Used: 10, Usage: 10, Enabled: true, Active: true, LastSeen: now,
|
||||
ZFSPool: &models.ZFSPool{Name: "rpool", State: "DEGRADED", Status: "Degraded", Devices: []models.ZFSDevice{{Name: "sda2", Type: "disk", State: "FAULTED"}}}}
|
||||
agentHost := models.Host{ID: "host-nas", Hostname: "nas", DisplayName: "nas", MachineID: "fedcba9876543210", Status: "online", LastSeen: now,
|
||||
@@ -209,6 +214,23 @@ func TestOwnerAlertTimelinesUseCanonicalHistoryIdentity(t *testing.T) {
|
||||
}
|
||||
require.True(t, found, "%s alert missing from %s history: %+v", alertType, canonicalID, changes)
|
||||
}
|
||||
alertTypes := func(ref string) []any {
|
||||
changes, err := store.GetRecentChangesFiltered(ref, time.Time{}, 10, filters)
|
||||
require.NoError(t, err)
|
||||
types := make([]any, 0, len(changes))
|
||||
for _, change := range changes {
|
||||
types = append(types, change.Metadata["alert_type"])
|
||||
}
|
||||
return types
|
||||
}
|
||||
require.ElementsMatch(t, []any{"docker-host-offline"}, alertTypes(owner["agent/tower"]))
|
||||
require.ElementsMatch(t, []any{"docker-service-health"}, alertTypes(owner["docker-service/web"]))
|
||||
require.ElementsMatch(t, []any{"docker-container-health"}, alertTypes("docker:dh-7f3a/name:sidecar"))
|
||||
require.ElementsMatch(t, []any{"docker-service-health"}, alertTypes("docker:dh-7f3a/service/name:x7k2m9q4w1e8r5t3y6u0i2o4p"))
|
||||
for _, name := range []string{"app-container/sidecar", "docker-service/X7K2M9Q4W1E8R5T3Y6U0I2O4P"} {
|
||||
require.NotEmpty(t, owner[name], name)
|
||||
require.Empty(t, alertTypes(owner[name]), "a name binds nothing: %s", name)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMonitor_HandleAlertFired_Extra(t *testing.T) {
|
||||
|
||||
@@ -26,7 +26,7 @@ func legacyDockerHistoryIdentity(ref string) (sourceID, canonicalID string, ok b
|
||||
return "", "", false
|
||||
}
|
||||
sourceID = host + "/container/" + container
|
||||
if host == "container" { // DockerResourceID's explicit hostless form.
|
||||
if host == "container" { // DockerContainerResourceID's explicit hostless form.
|
||||
sourceID = container
|
||||
}
|
||||
return sourceID, SourceSpecificID(ResourceTypeAppContainer, SourceDocker, sourceID), true
|
||||
@@ -43,19 +43,23 @@ func isDockerHistoryReference(ref string) bool {
|
||||
}
|
||||
|
||||
// isDockerNameHistoryReference marks Docker alert references that are not host
|
||||
// or service references: container references, which only a complete
|
||||
// container ID binds, and hostless service names. A name or shortened ID must
|
||||
// never join history, not even through a retained binding.
|
||||
// or service ID references: container references, which only a complete
|
||||
// container ID binds, the names of containers and services reported without an
|
||||
// ID, and hostless service names. A name or shortened ID must never join
|
||||
// history, not even through a retained binding.
|
||||
func isDockerNameHistoryReference(ref string) bool {
|
||||
_, _, ok := dockerHostHistoryReference(ref)
|
||||
return isDockerHistoryReference(ref) && !ok
|
||||
}
|
||||
|
||||
// dockerHostHistoryReference parses the Docker host alert reference
|
||||
// ("docker:<host ID>", alerts.DockerResourceID without a container) and the
|
||||
// Swarm service alert reference ("docker:<host ID>/service/<service ID>",
|
||||
// ("docker:<host ID>", alerts.DockerHostResourceID) and the Swarm service
|
||||
// alert reference ("docker:<host ID>/service/<service ID>",
|
||||
// alerts.DockerServiceResourceID). serviceID is empty for a host reference.
|
||||
// "docker:unknown" is the shared fallback for alerts without a host.
|
||||
// A service reported without an ID alerts under "name:<name>" in the ID
|
||||
// position, and Swarm IDs never contain a colon, so that is a name, not a
|
||||
// service reference. "docker:unknown" was the old builder's fallback for
|
||||
// alerts without a host.
|
||||
func dockerHostHistoryReference(ref string) (hostID, serviceID string, ok bool) {
|
||||
rest, found := strings.CutPrefix(strings.TrimSpace(ref), "docker:")
|
||||
if !found {
|
||||
@@ -65,7 +69,7 @@ func dockerHostHistoryReference(ref string) (hostID, serviceID string, ok bool)
|
||||
if hostID == "" || hostID == "unknown" || strings.TrimSpace(hostID) != hostID || strings.Contains(hostID, "/") {
|
||||
return "", "", false
|
||||
}
|
||||
if isService && (serviceID == "" || strings.TrimSpace(serviceID) != serviceID || strings.Contains(serviceID, "/")) {
|
||||
if isService && (serviceID == "" || strings.TrimSpace(serviceID) != serviceID || strings.ContainsAny(serviceID, "/:")) {
|
||||
return "", "", false
|
||||
}
|
||||
return hostID, serviceID, true
|
||||
@@ -74,43 +78,37 @@ func dockerHostHistoryReference(ref string) (hostID, serviceID string, ok bool)
|
||||
// dockerHistoryOwnerLocked resolves a Docker host or Swarm service alert
|
||||
// reference through the registry's own Docker source identities. The host ID
|
||||
// is the Docker host's source ID. A service reference names the service with
|
||||
// that ID in the host's Swarm cluster, which every manager of the cluster
|
||||
// reports under its own host ID. A service without an ID is referenced by its
|
||||
// normalized name in the same place, so while the cluster has such a service
|
||||
// every service reference there is a conflict.
|
||||
func (rr *ResourceRegistry) dockerHistoryOwnerLocked(ref string) (resourceID string, conflict bool) {
|
||||
// that exact ID in the host's Swarm cluster, which every manager of the
|
||||
// cluster reports under its own host ID.
|
||||
func (rr *ResourceRegistry) dockerHistoryOwnerLocked(ref string) string {
|
||||
hostID, serviceID, ok := dockerHostHistoryReference(ref)
|
||||
if !ok {
|
||||
return "", false
|
||||
return ""
|
||||
}
|
||||
hostResourceID := rr.bySource[SourceDocker][hostID]
|
||||
host := rr.resources[hostResourceID]
|
||||
if host == nil || CanonicalResourceType(host.Type) != ResourceTypeAgent {
|
||||
return "", false
|
||||
return ""
|
||||
}
|
||||
if serviceID == "" {
|
||||
return hostResourceID, false
|
||||
return hostResourceID
|
||||
}
|
||||
if host.Docker == nil {
|
||||
return "", false
|
||||
return ""
|
||||
}
|
||||
cluster := dockerSwarmClusterKeyFromMeta(host.Docker.Swarm)
|
||||
if cluster == "" {
|
||||
return "", false
|
||||
}
|
||||
for _, other := range rr.resources {
|
||||
if CanonicalResourceType(other.Type) == ResourceTypeDockerService && other.Docker != nil &&
|
||||
strings.TrimSpace(other.Docker.ServiceID) == "" && dockerSwarmClusterKeyFromMeta(other.Docker.Swarm) == cluster {
|
||||
return "", true
|
||||
}
|
||||
return ""
|
||||
}
|
||||
// The registry keys a service without an ID by its raw name, so the entry
|
||||
// must carry exactly this ID.
|
||||
serviceResourceID := rr.bySource[SourceDocker][normalizeSourceID(cluster+":service:"+serviceID)]
|
||||
service := rr.resources[serviceResourceID]
|
||||
if service == nil || CanonicalResourceType(service.Type) != ResourceTypeDockerService ||
|
||||
service.Docker == nil || strings.TrimSpace(service.Docker.ServiceID) != serviceID {
|
||||
return "", false
|
||||
return ""
|
||||
}
|
||||
return serviceResourceID, false
|
||||
return serviceResourceID
|
||||
}
|
||||
|
||||
// historySubResourceOwner names the owner of a sub-resource alert reference,
|
||||
@@ -169,8 +167,8 @@ func (rr *ResourceRegistry) resolveHistoryReference(ref string) (resourceID stri
|
||||
rr.mu.RLock()
|
||||
defer rr.mu.RUnlock()
|
||||
if isDockerHistoryReference(ref) {
|
||||
resourceID, conflict := rr.dockerHistoryOwnerLocked(ref)
|
||||
return resourceID, resourceID != "" || conflict
|
||||
resourceID = rr.dockerHistoryOwnerLocked(ref)
|
||||
return resourceID, resourceID != ""
|
||||
}
|
||||
matches := rr.durableHistoryMatchesLocked(ref)
|
||||
if ownerRef, ownerType, ok := historySubResourceOwner(ref); ok && len(matches) == 0 {
|
||||
|
||||
@@ -391,6 +391,8 @@ func TestHistoryIdentityMonitorAdapterResolvesDockerHostAndServiceReferences(t *
|
||||
hostA.AgentID, hostA.MachineID = "host-tower", "0123456789abcdef"
|
||||
hostA.Containers = []models.DockerContainer{{ID: container, Name: "worker", State: "running"}}
|
||||
hostB := manager("dh-91c0", "rack", "swarm-1", web)
|
||||
// A container without an ID, named like its host's ID.
|
||||
hostB.Containers = []models.DockerContainer{{Name: "dh-91c0", State: "running"}}
|
||||
// In swarm-2 a service without an ID is named like another service's ID.
|
||||
hostC := manager("dh-c4d2", "edge", "swarm-2", models.DockerService{ID: otherServiceID, Name: "api"}, models.DockerService{Name: strings.ToUpper(otherServiceID)})
|
||||
// Rows journaled before this process started are bound by a later generation.
|
||||
@@ -407,6 +409,7 @@ func TestHistoryIdentityMonitorAdapterResolvesDockerHostAndServiceReferences(t *
|
||||
hostAID := registry.sourceResourceID(SourceDocker, "dh-7f3a")
|
||||
hostBID := registry.sourceResourceID(SourceDocker, "dh-91c0")
|
||||
serviceResourceID := historyIdentityResourceID(t, registry, ResourceTypeDockerService, "web")
|
||||
apiID := historyIdentityResourceID(t, registry, ResourceTypeDockerService, "api")
|
||||
containerID := registry.sourceResourceID(SourceDocker, "dh-7f3a/container/"+container)
|
||||
require.Equal(t, registry.sourceResourceID(SourceAgent, "host-tower"), hostAID, "the Docker host merges into its agent's machine")
|
||||
require.NotEqual(t, hostAID, hostBID)
|
||||
@@ -417,17 +420,22 @@ func TestHistoryIdentityMonitorAdapterResolvesDockerHostAndServiceReferences(t *
|
||||
"docker:dh-91c0/service/" + serviceID: serviceResourceID,
|
||||
"docker:dh-7f3a": hostAID,
|
||||
"docker:dh-7f3a/service/" + serviceID: serviceResourceID,
|
||||
// A service without an ID in the same cluster no longer shares this
|
||||
// reference, so it binds.
|
||||
"docker:dh-c4d2/service/" + otherServiceID: apiID,
|
||||
}
|
||||
unbound := []string{
|
||||
"docker:tower", // hostname
|
||||
"docker:dh-7f3a/service/web", // service name
|
||||
"docker:dh-c4d2/service/" + otherServiceID, // its cluster has a service without an ID
|
||||
"docker-service:web", // hostless service name
|
||||
"docker:dh-7f3a/worker", // container name
|
||||
"docker:dh-7f3a/" + container[:12], // shortened container ID
|
||||
"docker:tower", // hostname
|
||||
"docker:dh-7f3a/service/web", // service name
|
||||
"docker:dh-c4d2/service/name:" + otherServiceID, // service without an ID
|
||||
"docker-service:name:web", // hostless service name
|
||||
"docker:dh-7f3a/worker", // container name
|
||||
"docker:dh-7f3a/" + container[:12], // shortened container ID
|
||||
"docker:dh-91c0/name:dh-91c0", // container without an ID
|
||||
"docker:unknown",
|
||||
}
|
||||
for i, ref := range append([]string{"docker:dh-7f3a", "docker:dh-7f3a/service/" + serviceID}, unbound...) {
|
||||
ids := []string{"docker:dh-7f3a", "docker:dh-7f3a/service/" + serviceID, "docker:dh-c4d2/service/" + otherServiceID}
|
||||
for i, ref := range append(ids, unbound...) {
|
||||
if ref == "docker:dh-7f3a/worker" {
|
||||
continue // journaled above
|
||||
}
|
||||
@@ -436,29 +444,29 @@ func TestHistoryIdentityMonitorAdapterResolvesDockerHostAndServiceReferences(t *
|
||||
requireHistoryReferenceBindings(t, store, bound, unbound)
|
||||
// Names and shortened IDs are never retried against later inventory.
|
||||
require.Contains(t, adapter.legacyHistory.pending, "docker:tower")
|
||||
for _, ref := range []string{"docker:dh-7f3a/worker", "docker:dh-7f3a/" + container[:12], "docker-service:web"} {
|
||||
for _, ref := range []string{"docker:dh-7f3a/worker", "docker:dh-7f3a/" + container[:12], "docker-service:name:web",
|
||||
"docker:dh-91c0/name:dh-91c0", "docker:dh-c4d2/service/name:" + otherServiceID} {
|
||||
require.NotContains(t, adapter.legacyHistory.pending, ref)
|
||||
}
|
||||
got, err := store.GetRecentChangesFiltered(containerID, time.Time{}, 10, ResourceChangeFilters{Kinds: []ChangeKind{ChangeAlertFired}})
|
||||
require.NoError(t, err)
|
||||
require.Empty(t, got, "a container name or short ID never joins the container's history")
|
||||
// While its cluster has a service without an ID, a service reference is a
|
||||
// conflict and must not follow a binding recorded earlier.
|
||||
collision := "docker:dh-c4d2/service/" + otherServiceID
|
||||
apiID := historyIdentityResourceID(t, registry, ResourceTypeDockerService, "api")
|
||||
require.NoError(t, store.RecordChangeWithSourceIdentity(ResourceChange{ID: "earlier-api", ResourceID: apiID, Kind: ChangeAlertFired, SourceType: SourceHeuristic, ObservedAt: now}, collision))
|
||||
require.NoError(t, adapter.RecordChange(ResourceChange{ID: "conflict-api", ResourceID: collision, Kind: ChangeAlertFired, SourceType: SourceHeuristic, ObservedAt: now.Add(2 * time.Minute)}))
|
||||
// The same holds once the service with that ID has left inventory.
|
||||
// The service's ID reference follows its binding once the service has left
|
||||
// inventory, while the service without an ID is still there; the name
|
||||
// reference stays its own.
|
||||
hostC.Services = hostC.Services[1:]
|
||||
adapter.PopulateFromSnapshot(models.StateSnapshot{
|
||||
Hosts: []models.Host{{ID: "host-tower", Hostname: "tower", MachineID: "0123456789abcdef", Status: "online", LastSeen: now}},
|
||||
DockerHosts: []models.DockerHost{hostA, hostB, hostC},
|
||||
})
|
||||
require.NoError(t, adapter.RecordChange(ResourceChange{ID: "conflict-api-removed", ResourceID: collision, Kind: ChangeAlertFired, SourceType: SourceHeuristic, ObservedAt: now.Add(3 * time.Minute)}))
|
||||
for _, id := range []string{"conflict-api", "conflict-api-removed"} {
|
||||
nameRef := "docker:dh-c4d2/service/name:" + otherServiceID
|
||||
for id, ref := range map[string]string{"api-removed": "docker:dh-c4d2/service/" + otherServiceID, "name-removed": nameRef} {
|
||||
require.NoError(t, adapter.RecordChange(ResourceChange{ID: id, ResourceID: ref, Kind: ChangeAlertResolved, SourceType: SourceHeuristic, ObservedAt: now.Add(3 * time.Minute)}))
|
||||
}
|
||||
for id, want := range map[string]string{"api-removed": apiID, "name-removed": nameRef} {
|
||||
var recordedAs string
|
||||
require.NoError(t, store.db.QueryRow(`SELECT canonical_id FROM resource_changes WHERE id = ?`, id).Scan(&recordedAs))
|
||||
require.Equal(t, collision, recordedAs, id)
|
||||
require.Equal(t, want, recordedAs, id)
|
||||
}
|
||||
|
||||
// After the service leaves inventory its alerts follow the retained binding;
|
||||
|
||||
Reference in New Issue
Block a user