mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-09-10 02:25:56 +00:00
Measure node connection test outcomes
Telemetry could see only saved connections, so an install that tried to reach a node and could not was indistinguishable from one that never opened the add-node dialog. Both report zero configured connections and stall at the same activation stage. Fleet data shows that population is real and concentrated three to one in container deployments, and nothing recorded whether those installs attempted a connection at all. Record node connection test attempts and failures in a bounded, day-bucketed tally in the config directory, pruned to a 31-day retention window, and report both over the install-ID rotation window as node_test_attempts_30d and node_test_failures_30d. Recording starts only once a request carries a target and credentials, so an incomplete form is never counted as a node that could not be reached. A host string that turns out to be unusable does count, because the attempt was made and it failed. Only the add-node dialog endpoint is instrumented: instrumenting the unused test-config endpoint as well would double-count a single operator action. The tally holds counts alone. Hosts, credentials, and error text never enter it, which is why it is plain JSON rather than encrypted history.
This commit is contained in:
@@ -120,6 +120,8 @@ Every field is listed below with the reason it exists. Nothing else is included
|
||||
| Update successes 30d | `1` | Count successful update attempts in the current 30-day telemetry window |
|
||||
| Update failures 30d | `1` | Count failed or rolled-back update attempts in the current 30-day telemetry window without sending raw errors, logs, URLs, or command output |
|
||||
| Update last failure category | `download` | Send only a coarse category for the latest update failure, such as `download`, `signature`, `checksum`, `disk_space`, `extract`, `backup`, `apply`, `restart`, `rolled_back`, or `unknown` |
|
||||
| Node test attempts 30d | `3` | Count node connection tests that reached the connection stage in the current 30-day telemetry window, without sending hostnames, addresses, credentials, or error text |
|
||||
| Node test failures 30d | `2` | Count those connection tests that could not reach or authenticate against the target, so an install that tried to add a node and failed is distinguishable from one that never attempted it, without sending hostnames, addresses, credentials, or error text |
|
||||
| Pulse Intelligence loop configured | `true`/`false` | See whether Assistant, Patrol, governed actions, or external-agent access is configured so adoption can be measured without sending configuration details |
|
||||
| Pulse Intelligence loop active 30d | `true`/`false` | See whether Assistant, Patrol, external-agent, or governed-action activity occurred in the current 30-day telemetry window |
|
||||
| Pulse Intelligence complete operations loop 30d | `true`/`false` | See whether Patrol issue activity reached an approved or rejected governed-action decision without sending prompts, findings, resource identifiers, command text, or action output |
|
||||
|
||||
@@ -2286,6 +2286,13 @@ the `/api/recovery/points` or `/api/recovery/rollups` list meta; lifecycle
|
||||
code MUST NOT treat that pagination meta as an agent enrollment, admission, or
|
||||
report-ingestion contract.
|
||||
|
||||
The node connection test handlers under `internal/api/configapi/` record a
|
||||
content-free outcome tally for telemetry. This adds no agent lifecycle surface:
|
||||
it does not touch agent registration, capability negotiation, command dispatch,
|
||||
token rotation, or removal state, and it records nothing about agents. Node
|
||||
connection tests target Proxmox, PBS, and PMG endpoints directly and never
|
||||
traverse an agent.
|
||||
|
||||
## Forbidden Paths
|
||||
|
||||
1. New install or update continuity behavior hidden only inside broad monitoring ownership.
|
||||
|
||||
@@ -3325,6 +3325,16 @@ and MUST NOT reconstruct it from `aggregations.bySource`. Proof:
|
||||
`TestContract_ResourceListReportsPlatformAdmission` and the wire-shape pin in
|
||||
`frontend-modern/src/types/__tests__/resource.test.ts`.
|
||||
|
||||
`POST /api/config/nodes/test-connection` records a node connection test outcome
|
||||
for telemetry. Recording begins only after the request is validated as carrying
|
||||
a target and credentials, so a request rejected for incomplete input is never
|
||||
counted as a node that could not be reached. Recording wraps the response
|
||||
writer and must not alter the response contract: status codes and sanitized
|
||||
error bodies stay exactly as they were. `POST /api/config/nodes/test-config` is
|
||||
deliberately not instrumented, because instrumenting both endpoints would
|
||||
double-count a single operator action and corrupt the failure share the
|
||||
counters exist to measure.
|
||||
|
||||
## Forbidden Paths
|
||||
|
||||
1. Handler-local payload shape drift without a contract test
|
||||
|
||||
@@ -568,6 +568,17 @@ the `white_label` branding entitlement.
|
||||
configuration has no key for this boundary and must not gain one that
|
||||
can re-enable credential reads a local operator disabled.
|
||||
|
||||
Node connection test telemetry reports two counters over the install-ID
|
||||
rotation window: `node_test_attempts_30d` and `node_test_failures_30d`. They
|
||||
count only tests that carried a target and credentials, and they carry counts
|
||||
alone. Hostnames, addresses, ports, credentials, fingerprints, and error text
|
||||
must never enter the tally or the outbound ping. The tally is stored as plain
|
||||
JSON in the config directory precisely because it holds no secret material;
|
||||
adding any field that identifies a target would invalidate that storage choice
|
||||
and require the encrypted history path instead. Both fields must stay disclosed
|
||||
in `docs/PRIVACY.md` and `frontend-modern/public/docs/PRIVACY.md`, which
|
||||
`TestAllTelemetryFieldsAreDisclosed` enforces.
|
||||
|
||||
## Forbidden Paths
|
||||
|
||||
1. Changing telemetry payload semantics without updating the canonical privacy disclosure.
|
||||
|
||||
@@ -1857,6 +1857,13 @@ serving page size that does not move the shared handler bounds re-opens the
|
||||
`totalPages` misreport that silently truncates rollup iteration for clients
|
||||
walking the reported page count.
|
||||
|
||||
The node connection test tally persists as a bounded, day-bucketed JSON file in
|
||||
the config directory, pruned to a 31-day retention window so it cannot grow
|
||||
without limit however long an install runs. It is advisory telemetry state: it
|
||||
is outside backup, restore, and recovery, and a missing or corrupt tally must be
|
||||
discarded and rebuilt rather than surfaced as a recovery failure. Nothing may
|
||||
take a correctness dependency on its contents.
|
||||
|
||||
## Forbidden Paths
|
||||
|
||||
1. Reintroducing storage or recovery product logic as ad hoc dashboard-only summaries without a canonical page-surface owner
|
||||
|
||||
@@ -120,6 +120,8 @@ Every field is listed below with the reason it exists. Nothing else is included
|
||||
| Update successes 30d | `1` | Count successful update attempts in the current 30-day telemetry window |
|
||||
| Update failures 30d | `1` | Count failed or rolled-back update attempts in the current 30-day telemetry window without sending raw errors, logs, URLs, or command output |
|
||||
| Update last failure category | `download` | Send only a coarse category for the latest update failure, such as `download`, `signature`, `checksum`, `disk_space`, `extract`, `backup`, `apply`, `restart`, `rolled_back`, or `unknown` |
|
||||
| Node test attempts 30d | `3` | Count node connection tests that reached the connection stage in the current 30-day telemetry window, without sending hostnames, addresses, credentials, or error text |
|
||||
| Node test failures 30d | `2` | Count those connection tests that could not reach or authenticate against the target, so an install that tried to add a node and failed is distinguishable from one that never attempted it, without sending hostnames, addresses, credentials, or error text |
|
||||
| Pulse Intelligence loop configured | `true`/`false` | See whether Assistant, Patrol, governed actions, or external-agent access is configured so adoption can be measured without sending configuration details |
|
||||
| Pulse Intelligence loop active 30d | `true`/`false` | See whether Assistant, Patrol, external-agent, or governed-action activity occurred in the current 30-day telemetry window |
|
||||
| Pulse Intelligence complete operations loop 30d | `true`/`false` | See whether Patrol issue activity reached an approved or rejected governed-action decision without sending prompts, findings, resource identifiers, command text, or action output |
|
||||
|
||||
@@ -770,6 +770,43 @@ func (h *ConfigHandlers) handleAddNode(w http.ResponseWriter, r *http.Request) {
|
||||
|
||||
// HandleTestConnection tests a node connection without saving
|
||||
|
||||
// nodeTestOutcomeRecorder captures the status a node connection test responded
|
||||
// with, so the outcome can be tallied once without threading a result through
|
||||
// every branch of the handler.
|
||||
type nodeTestOutcomeRecorder struct {
|
||||
http.ResponseWriter
|
||||
status int
|
||||
}
|
||||
|
||||
func newNodeTestOutcomeRecorder(w http.ResponseWriter) *nodeTestOutcomeRecorder {
|
||||
return &nodeTestOutcomeRecorder{ResponseWriter: w, status: http.StatusOK}
|
||||
}
|
||||
|
||||
func (rec *nodeTestOutcomeRecorder) WriteHeader(status int) {
|
||||
rec.status = status
|
||||
rec.ResponseWriter.WriteHeader(status)
|
||||
}
|
||||
|
||||
// failed reports whether the test resolved to anything other than success. The
|
||||
// success paths write a body without an explicit status, which is why the
|
||||
// recorder starts at 200.
|
||||
func (rec *nodeTestOutcomeRecorder) failed() bool {
|
||||
return rec.status < 200 || rec.status >= 300
|
||||
}
|
||||
|
||||
// recordNodeTestOutcome tallies one node connection test for telemetry. The
|
||||
// tally is advisory, so a write failure is logged and never surfaced to the
|
||||
// caller: a broken counter must not break a connection test.
|
||||
func (h *ConfigHandlers) recordNodeTestOutcome(ctx context.Context, failed bool) {
|
||||
persistence := h.getPersistence(ctx)
|
||||
if persistence == nil {
|
||||
return
|
||||
}
|
||||
if err := persistence.RecordNodeTestOutcome(failed, time.Now()); err != nil {
|
||||
log.Warn().Err(err).Msg("Failed to record node connection test outcome")
|
||||
}
|
||||
}
|
||||
|
||||
func (h *ConfigHandlers) handleTestConnection(w http.ResponseWriter, r *http.Request) {
|
||||
// Limit request body to 32KB to prevent memory exhaustion
|
||||
r.Body = http.MaxBytesReader(w, r.Body, 32*1024)
|
||||
@@ -851,6 +888,16 @@ func (h *ConfigHandlers) handleTestConnection(w http.ResponseWriter, r *http.Req
|
||||
return
|
||||
}
|
||||
|
||||
// Past this point the request carries a target and credentials, so it is a
|
||||
// real attempt by someone to reach a node. Requests that never carried them
|
||||
// are rejected above and stay out of the tally, which is what keeps an
|
||||
// incomplete form from counting as a node that could not be reached. A host
|
||||
// string that turns out to be unusable still counts: the attempt was made
|
||||
// and it failed.
|
||||
testRecorder := newNodeTestOutcomeRecorder(w)
|
||||
w = testRecorder
|
||||
defer func() { h.recordNodeTestOutcome(r.Context(), testRecorder.failed()) }()
|
||||
|
||||
// Test connection based on type
|
||||
if req.Type == "pve" {
|
||||
host := normalizedHost
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
package configapi
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/config"
|
||||
)
|
||||
|
||||
func nodeTestTallyCounts(t *testing.T, dataPath string) (attempts int, failures int) {
|
||||
t.Helper()
|
||||
tally, err := config.NewConfigPersistence(dataPath).LoadNodeTestTally()
|
||||
if err != nil {
|
||||
t.Fatalf("LoadNodeTestTally: %v", err)
|
||||
}
|
||||
since := time.Now().UTC().AddDate(0, 0, -30)
|
||||
return tally.AttemptsSince(since), tally.FailuresSince(since)
|
||||
}
|
||||
|
||||
func postNodeConnectionTest(t *testing.T, h *ConfigHandlers, body map[string]string) {
|
||||
t.Helper()
|
||||
encoded, err := json.Marshal(body)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal request: %v", err)
|
||||
}
|
||||
req := httptest.NewRequest("POST", "/api/config/nodes/test-connection", bytes.NewBuffer(encoded))
|
||||
h.HandleTestConnection(httptest.NewRecorder(), req)
|
||||
}
|
||||
|
||||
// A request that never carried a target and credentials was not an attempt to
|
||||
// reach a node, so counting it would inflate the failure share this counter
|
||||
// exists to measure.
|
||||
func TestNodeTestTallyIgnoresRequestsWithoutATarget(t *testing.T) {
|
||||
dataPath := t.TempDir()
|
||||
h := newTestConfigHandlers(t, &config.Config{DataPath: dataPath})
|
||||
|
||||
for _, body := range []map[string]string{
|
||||
{"type": "pve"},
|
||||
{"type": "pve", "host": "10.0.0.1"},
|
||||
{"type": "unknown", "host": "10.0.0.1", "user": "root@pam", "password": "x"},
|
||||
} {
|
||||
postNodeConnectionTest(t, h, body)
|
||||
}
|
||||
|
||||
attempts, failures := nodeTestTallyCounts(t, dataPath)
|
||||
if attempts != 0 || failures != 0 {
|
||||
t.Fatalf("targetless requests tallied: attempts=%d failures=%d, want 0 and 0", attempts, failures)
|
||||
}
|
||||
}
|
||||
|
||||
// The other side of that boundary: someone who typed an unusable host and got
|
||||
// an error did attempt to reach a node, and that attempt failed. Excluding it
|
||||
// would hide exactly the population this counter exists to find.
|
||||
func TestNodeTestTallyCountsUnusableHostAsFailedAttempt(t *testing.T) {
|
||||
dataPath := t.TempDir()
|
||||
h := newTestConfigHandlers(t, &config.Config{DataPath: dataPath})
|
||||
|
||||
postNodeConnectionTest(t, h, map[string]string{
|
||||
"type": "pve",
|
||||
"host": "://invalid-url",
|
||||
"user": "root@pam",
|
||||
"password": "password",
|
||||
})
|
||||
|
||||
attempts, failures := nodeTestTallyCounts(t, dataPath)
|
||||
if attempts != 1 || failures != 1 {
|
||||
t.Fatalf("attempts=%d failures=%d, want 1 and 1", attempts, failures)
|
||||
}
|
||||
}
|
||||
|
||||
// An unreachable target is the case the counter exists for. proxmox.NewClient
|
||||
// authenticates eagerly, so this fails during client construction rather than
|
||||
// on a later call, and it must still be counted.
|
||||
func TestNodeTestTallyCountsUnreachableTargetAsFailure(t *testing.T) {
|
||||
dataPath := t.TempDir()
|
||||
h := newTestConfigHandlers(t, &config.Config{DataPath: dataPath})
|
||||
|
||||
postNodeConnectionTest(t, h, map[string]string{
|
||||
"type": "pve",
|
||||
"host": "127.0.0.1:1",
|
||||
"user": "root@pam",
|
||||
"password": "password",
|
||||
})
|
||||
|
||||
attempts, failures := nodeTestTallyCounts(t, dataPath)
|
||||
if attempts != 1 {
|
||||
t.Fatalf("attempts = %d, want 1", attempts)
|
||||
}
|
||||
if failures != 1 {
|
||||
t.Fatalf("failures = %d, want 1", failures)
|
||||
}
|
||||
}
|
||||
|
||||
// The success paths write a body without setting a status explicitly, so the
|
||||
// recorder must treat an unset status as success rather than failure. Driving a
|
||||
// real successful test would need a live Proxmox endpoint, so the mapping is
|
||||
// asserted directly.
|
||||
func TestNodeTestOutcomeRecorderTreatsImplicitStatusAsSuccess(t *testing.T) {
|
||||
rec := newNodeTestOutcomeRecorder(httptest.NewRecorder())
|
||||
if rec.failed() {
|
||||
t.Fatal("implicit 200 reported as failure")
|
||||
}
|
||||
if _, err := rec.Write([]byte(`{"status":"success"}`)); err != nil {
|
||||
t.Fatalf("write: %v", err)
|
||||
}
|
||||
if rec.failed() {
|
||||
t.Fatal("body write without WriteHeader reported as failure")
|
||||
}
|
||||
|
||||
rec.WriteHeader(http.StatusBadRequest)
|
||||
if !rec.failed() {
|
||||
t.Fatal("400 not reported as failure")
|
||||
}
|
||||
}
|
||||
@@ -24432,3 +24432,43 @@ func TestContract_PatrolBlockedCauseTelemetryExportsEnumOnlyWhileBlocked(t *test
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The node connection test tally is only meaningful if it starts after the
|
||||
// request is known to carry a target and credentials. Moving the recorder above
|
||||
// those guards would silently begin counting incomplete forms as nodes that
|
||||
// could not be reached, which is the exact confusion the counter exists to
|
||||
// resolve, and nothing about the response would change to reveal it.
|
||||
func TestContract_NodeTestTelemetryRecordsOnlyAfterTargetValidation(t *testing.T) {
|
||||
source, err := os.ReadFile(filepath.Clean("configapi/config_node_handlers.go"))
|
||||
if err != nil {
|
||||
t.Fatalf("read node handlers: %v", err)
|
||||
}
|
||||
handlers := string(source)
|
||||
|
||||
start := strings.Index(handlers, "func (h *ConfigHandlers) handleTestConnection(")
|
||||
if start < 0 {
|
||||
t.Fatal("handleTestConnection not found")
|
||||
}
|
||||
body := handlers[start:]
|
||||
if end := strings.Index(body[1:], "\nfunc "); end >= 0 {
|
||||
body = body[:end+1]
|
||||
}
|
||||
|
||||
recorderAt := strings.Index(body, "testRecorder := newNodeTestOutcomeRecorder(w)")
|
||||
if recorderAt < 0 {
|
||||
t.Fatal("handleTestConnection no longer records node connection test outcomes")
|
||||
}
|
||||
for _, guard := range []string{
|
||||
`http.Error(w, "Host is required", http.StatusBadRequest)`,
|
||||
`http.Error(w, "Invalid node type", http.StatusBadRequest)`,
|
||||
`http.Error(w, "Authentication credentials required", http.StatusBadRequest)`,
|
||||
} {
|
||||
guardAt := strings.Index(body, guard)
|
||||
if guardAt < 0 {
|
||||
t.Fatalf("validation guard %q missing from handleTestConnection", guard)
|
||||
}
|
||||
if guardAt > recorderAt {
|
||||
t.Fatalf("validation guard %q must run before the node connection test tally starts", guard)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,181 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"time"
|
||||
|
||||
"github.com/rs/zerolog/log"
|
||||
)
|
||||
|
||||
const (
|
||||
nodeTestTallyFileName = "node_test_tally.json"
|
||||
nodeTestTallyVersion = 1
|
||||
// nodeTestTallyRetentionDays bounds the tally. It exceeds the thirty-day
|
||||
// telemetry window by a day so the day the window opens on is still
|
||||
// present when the tally is read.
|
||||
nodeTestTallyRetentionDays = 31
|
||||
)
|
||||
|
||||
// NodeTestTallyData counts node connection tests per UTC day. Telemetry can
|
||||
// otherwise see only the saved-connection count, so an install that tried to
|
||||
// reach a node and failed is indistinguishable from one that never opened the
|
||||
// add-node dialog. Both report zero connections and stall at the "secured"
|
||||
// activation stage. Two day-bucketed integers separate them and cost one small
|
||||
// map entry per retained day.
|
||||
//
|
||||
// The tally holds counts only. Hosts, credentials, and error text never enter
|
||||
// it, which is why it is stored as plain JSON alongside the encrypted
|
||||
// configuration rather than through the encrypted history helpers.
|
||||
type NodeTestTallyData struct {
|
||||
Version int `json:"version"`
|
||||
LastSaved time.Time `json:"last_saved"`
|
||||
// DailyAttempts counts tests that reached the connection stage, meaning
|
||||
// the request carried a well-formed target and credentials.
|
||||
DailyAttempts map[string]int `json:"daily_attempts,omitempty"`
|
||||
// DailyFailures counts the subset of those attempts that could not reach
|
||||
// or authenticate against the target. Successes are the remainder.
|
||||
DailyFailures map[string]int `json:"daily_failures,omitempty"`
|
||||
}
|
||||
|
||||
// NodeTestTallyDayKey renders the UTC day key used by NodeTestTallyData.
|
||||
func NodeTestTallyDayKey(at time.Time) string {
|
||||
return at.UTC().Format("2006-01-02")
|
||||
}
|
||||
|
||||
func nodeTestTallyCountSince(daily map[string]int, since time.Time) int {
|
||||
if len(daily) == 0 {
|
||||
return 0
|
||||
}
|
||||
sinceDay := NodeTestTallyDayKey(since)
|
||||
total := 0
|
||||
for day, count := range daily {
|
||||
if day >= sinceDay {
|
||||
total += count
|
||||
}
|
||||
}
|
||||
return total
|
||||
}
|
||||
|
||||
// AttemptsSince counts node connection tests observed at or after since. The
|
||||
// tally is day-granular, so the day containing since is counted whole.
|
||||
func (data *NodeTestTallyData) AttemptsSince(since time.Time) int {
|
||||
if data == nil {
|
||||
return 0
|
||||
}
|
||||
return nodeTestTallyCountSince(data.DailyAttempts, since)
|
||||
}
|
||||
|
||||
// FailuresSince counts failed node connection tests observed at or after since.
|
||||
func (data *NodeTestTallyData) FailuresSince(since time.Time) int {
|
||||
if data == nil {
|
||||
return 0
|
||||
}
|
||||
return nodeTestTallyCountSince(data.DailyFailures, since)
|
||||
}
|
||||
|
||||
// pruneNodeTestTally drops days outside the retention window so the tally stays
|
||||
// bounded no matter how long an install runs.
|
||||
func pruneNodeTestTally(data *NodeTestTallyData, now time.Time) {
|
||||
if data == nil {
|
||||
return
|
||||
}
|
||||
cutoff := NodeTestTallyDayKey(now.UTC().AddDate(0, 0, -nodeTestTallyRetentionDays))
|
||||
for day := range data.DailyAttempts {
|
||||
if day < cutoff {
|
||||
delete(data.DailyAttempts, day)
|
||||
}
|
||||
}
|
||||
for day := range data.DailyFailures {
|
||||
if day < cutoff {
|
||||
delete(data.DailyFailures, day)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (c *ConfigPersistence) nodeTestTallyPath() string {
|
||||
return filepath.Join(c.configDir, nodeTestTallyFileName)
|
||||
}
|
||||
|
||||
// loadNodeTestTallyLocked reads the tally with c.mu already held. A missing,
|
||||
// empty, or corrupt file yields an empty tally rather than an error: the
|
||||
// counters are advisory telemetry and must never block a connection test.
|
||||
func (c *ConfigPersistence) loadNodeTestTallyLocked() *NodeTestTallyData {
|
||||
data := &NodeTestTallyData{Version: nodeTestTallyVersion}
|
||||
if raw, err := c.fs.ReadFile(c.nodeTestTallyPath()); err == nil && len(raw) > 0 {
|
||||
decoded := &NodeTestTallyData{}
|
||||
if err := json.Unmarshal(raw, decoded); err == nil {
|
||||
data = decoded
|
||||
} else {
|
||||
log.Warn().Err(err).Msg("Discarding unreadable node test tally")
|
||||
}
|
||||
}
|
||||
if data.DailyAttempts == nil {
|
||||
data.DailyAttempts = make(map[string]int, nodeTestTallyRetentionDays)
|
||||
}
|
||||
if data.DailyFailures == nil {
|
||||
data.DailyFailures = make(map[string]int, nodeTestTallyRetentionDays)
|
||||
}
|
||||
return data
|
||||
}
|
||||
|
||||
// LoadNodeTestTally returns the persisted node connection test tally.
|
||||
func (c *ConfigPersistence) LoadNodeTestTally() (*NodeTestTallyData, error) {
|
||||
if c == nil || c.configDir == "" {
|
||||
return &NodeTestTallyData{}, nil
|
||||
}
|
||||
c.mu.RLock()
|
||||
defer c.mu.RUnlock()
|
||||
return c.loadNodeTestTallyLocked(), nil
|
||||
}
|
||||
|
||||
// RecordNodeTestOutcome folds one node connection test into the daily tally.
|
||||
// Callers report only tests that reached the connection stage, so a failure
|
||||
// always means the target could not be reached or authenticated rather than
|
||||
// that the form was incomplete.
|
||||
func (c *ConfigPersistence) RecordNodeTestOutcome(failed bool, now time.Time) error {
|
||||
if c == nil || c.configDir == "" {
|
||||
return nil
|
||||
}
|
||||
if now.IsZero() {
|
||||
now = time.Now()
|
||||
}
|
||||
now = now.UTC()
|
||||
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
|
||||
data := c.loadNodeTestTallyLocked()
|
||||
day := NodeTestTallyDayKey(now)
|
||||
data.DailyAttempts[day]++
|
||||
if failed {
|
||||
data.DailyFailures[day]++
|
||||
}
|
||||
pruneNodeTestTally(data, now)
|
||||
data.Version = nodeTestTallyVersion
|
||||
data.LastSaved = now
|
||||
|
||||
encoded, err := json.Marshal(data)
|
||||
if err != nil {
|
||||
return fmt.Errorf("encode node test tally: %w", err)
|
||||
}
|
||||
|
||||
path := c.nodeTestTallyPath()
|
||||
if err := c.fs.MkdirAll(filepath.Dir(path), 0o700); err != nil {
|
||||
return fmt.Errorf("create node test tally directory: %w", err)
|
||||
}
|
||||
tmpPath := path + ".tmp"
|
||||
if err := c.fs.WriteFile(tmpPath, encoded, 0o600); err != nil {
|
||||
return fmt.Errorf("write temp node test tally: %w", err)
|
||||
}
|
||||
if err := c.fs.Rename(tmpPath, path); err != nil {
|
||||
if removeErr := c.fs.Remove(tmpPath); removeErr != nil && !errors.Is(removeErr, os.ErrNotExist) {
|
||||
log.Warn().Err(removeErr).Str("tmp_path", tmpPath).Msg("Failed to remove temporary node test tally after failed rename")
|
||||
}
|
||||
return fmt.Errorf("commit node test tally: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,109 @@
|
||||
package config_test
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/config"
|
||||
)
|
||||
|
||||
func TestRecordNodeTestOutcomeCountsAttemptsAndFailures(t *testing.T) {
|
||||
cp := config.NewConfigPersistence(t.TempDir())
|
||||
now := time.Now().UTC()
|
||||
|
||||
for i := 0; i < 3; i++ {
|
||||
if err := cp.RecordNodeTestOutcome(true, now); err != nil {
|
||||
t.Fatalf("RecordNodeTestOutcome(failed) = %v", err)
|
||||
}
|
||||
}
|
||||
if err := cp.RecordNodeTestOutcome(false, now); err != nil {
|
||||
t.Fatalf("RecordNodeTestOutcome(success) = %v", err)
|
||||
}
|
||||
|
||||
tally, err := cp.LoadNodeTestTally()
|
||||
if err != nil {
|
||||
t.Fatalf("LoadNodeTestTally = %v", err)
|
||||
}
|
||||
since := now.AddDate(0, 0, -30)
|
||||
if got := tally.AttemptsSince(since); got != 4 {
|
||||
t.Fatalf("AttemptsSince = %d, want 4", got)
|
||||
}
|
||||
if got := tally.FailuresSince(since); got != 3 {
|
||||
t.Fatalf("FailuresSince = %d, want 3", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A success must never be counted as a failure, otherwise the failure share
|
||||
// this counter exists to measure is meaningless.
|
||||
func TestRecordNodeTestOutcomeSuccessLeavesFailuresZero(t *testing.T) {
|
||||
cp := config.NewConfigPersistence(t.TempDir())
|
||||
now := time.Now().UTC()
|
||||
|
||||
if err := cp.RecordNodeTestOutcome(false, now); err != nil {
|
||||
t.Fatalf("RecordNodeTestOutcome = %v", err)
|
||||
}
|
||||
|
||||
tally, err := cp.LoadNodeTestTally()
|
||||
if err != nil {
|
||||
t.Fatalf("LoadNodeTestTally = %v", err)
|
||||
}
|
||||
if got := tally.FailuresSince(now.AddDate(0, 0, -30)); got != 0 {
|
||||
t.Fatalf("FailuresSince = %d, want 0", got)
|
||||
}
|
||||
if got := tally.AttemptsSince(now.AddDate(0, 0, -30)); got != 1 {
|
||||
t.Fatalf("AttemptsSince = %d, want 1", got)
|
||||
}
|
||||
}
|
||||
|
||||
// Counts outside the reporting window must not leak into it, and days outside
|
||||
// the retention window must be pruned so the file stays bounded.
|
||||
func TestNodeTestTallyPrunesAndWindowsCorrectly(t *testing.T) {
|
||||
cp := config.NewConfigPersistence(t.TempDir())
|
||||
now := time.Now().UTC()
|
||||
|
||||
if err := cp.RecordNodeTestOutcome(true, now.AddDate(0, 0, -40)); err != nil {
|
||||
t.Fatalf("RecordNodeTestOutcome(old) = %v", err)
|
||||
}
|
||||
if err := cp.RecordNodeTestOutcome(true, now.AddDate(0, 0, -10)); err != nil {
|
||||
t.Fatalf("RecordNodeTestOutcome(recent) = %v", err)
|
||||
}
|
||||
if err := cp.RecordNodeTestOutcome(true, now); err != nil {
|
||||
t.Fatalf("RecordNodeTestOutcome(now) = %v", err)
|
||||
}
|
||||
|
||||
tally, err := cp.LoadNodeTestTally()
|
||||
if err != nil {
|
||||
t.Fatalf("LoadNodeTestTally = %v", err)
|
||||
}
|
||||
if got := tally.AttemptsSince(now.AddDate(0, 0, -30)); got != 2 {
|
||||
t.Fatalf("AttemptsSince(30d) = %d, want 2", got)
|
||||
}
|
||||
if _, ok := tally.DailyAttempts[config.NodeTestTallyDayKey(now.AddDate(0, 0, -40))]; ok {
|
||||
t.Fatalf("day outside retention window was not pruned")
|
||||
}
|
||||
}
|
||||
|
||||
// The counters are advisory. A corrupt tally must start over rather than make
|
||||
// a connection test fail.
|
||||
func TestRecordNodeTestOutcomeRecoversFromCorruptTally(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
cp := config.NewConfigPersistence(dir)
|
||||
if err := os.WriteFile(filepath.Join(dir, "node_test_tally.json"), []byte("{not json"), 0o600); err != nil {
|
||||
t.Fatalf("seed corrupt tally: %v", err)
|
||||
}
|
||||
|
||||
now := time.Now().UTC()
|
||||
if err := cp.RecordNodeTestOutcome(true, now); err != nil {
|
||||
t.Fatalf("RecordNodeTestOutcome = %v", err)
|
||||
}
|
||||
|
||||
tally, err := cp.LoadNodeTestTally()
|
||||
if err != nil {
|
||||
t.Fatalf("LoadNodeTestTally = %v", err)
|
||||
}
|
||||
if got := tally.AttemptsSince(now.AddDate(0, 0, -30)); got != 1 {
|
||||
t.Fatalf("AttemptsSince = %d, want 1", got)
|
||||
}
|
||||
}
|
||||
@@ -160,7 +160,12 @@ const (
|
||||
// is distinguishable in the fleet from one that runs and finds nothing.
|
||||
// Field telemetry showed both present identically: high run counts with
|
||||
// zero AI calls and zero findings.
|
||||
TelemetrySchemaVersion = 10
|
||||
// Schema v11 adds node connection test attempt and failure counts. The
|
||||
// fleet shows a population that configures authentication and then never
|
||||
// saves a connection, concentrated three to one in container deployments,
|
||||
// and nothing recorded whether those installs tried to reach a node and
|
||||
// failed or never attempted one at all.
|
||||
TelemetrySchemaVersion = 11
|
||||
)
|
||||
|
||||
type installIDRecord struct {
|
||||
@@ -263,6 +268,16 @@ type Ping struct {
|
||||
// Last coarse update failure category; never raw error text.
|
||||
UpdateLastFailureCategory string `json:"update_last_failure_category,omitempty"`
|
||||
|
||||
// Node connection test outcomes over the install-ID rotation window.
|
||||
// ConfiguredConnections counts only connections that were saved, so an
|
||||
// install that tried to reach a node and could not is indistinguishable
|
||||
// from one that never opened the add-node dialog: both report zero
|
||||
// connections and stall at the "secured" activation stage. These separate
|
||||
// the two. Counts only; hosts, credentials, and error text never leave the
|
||||
// install. Successes are attempts minus failures.
|
||||
NodeTestAttempts30d int `json:"node_test_attempts_30d"`
|
||||
NodeTestFailures30d int `json:"node_test_failures_30d"`
|
||||
|
||||
// Core product outcomes. Alert history is retained locally for 30 days;
|
||||
// notification delivery rows are locally retention-bounded to seven days.
|
||||
AlertsFired30d int `json:"alerts_fired_30d"`
|
||||
@@ -412,6 +427,8 @@ type Snapshot struct {
|
||||
UpdateSuccesses30d int
|
||||
UpdateFailures30d int
|
||||
UpdateLastFailureCategory string
|
||||
NodeTestAttempts30d int
|
||||
NodeTestFailures30d int
|
||||
AuthConfigured bool
|
||||
ConfiguredConnections int
|
||||
AlertsFired30d int
|
||||
@@ -1019,6 +1036,8 @@ func applySnapshot(base Ping, fn SnapshotFunc) Ping {
|
||||
ping.UpdateSuccesses30d = s.UpdateSuccesses30d
|
||||
ping.UpdateFailures30d = s.UpdateFailures30d
|
||||
ping.UpdateLastFailureCategory = s.UpdateLastFailureCategory
|
||||
ping.NodeTestAttempts30d = s.NodeTestAttempts30d
|
||||
ping.NodeTestFailures30d = s.NodeTestFailures30d
|
||||
ping.AuthConfigured = s.AuthConfigured
|
||||
ping.ConfiguredConnections = s.ConfiguredConnections
|
||||
ping.AlertsFired30d = s.AlertsFired30d
|
||||
|
||||
@@ -1255,3 +1255,18 @@ func decodeInstallIDRecord(t *testing.T, data []byte) installIDRecord {
|
||||
}
|
||||
return record
|
||||
}
|
||||
|
||||
// The node connection test counts are only useful if they leave the install, so
|
||||
// assert the snapshot-to-ping mapping rather than trusting field order.
|
||||
func TestBuildPingCarriesNodeTestCounts(t *testing.T) {
|
||||
ping := BuildPingForSnapshot(Snapshot{
|
||||
NodeTestAttempts30d: 9,
|
||||
NodeTestFailures30d: 4,
|
||||
})
|
||||
if ping.NodeTestAttempts30d != 9 {
|
||||
t.Fatalf("node_test_attempts_30d = %d, want 9", ping.NodeTestAttempts30d)
|
||||
}
|
||||
if ping.NodeTestFailures30d != 4 {
|
||||
t.Fatalf("node_test_failures_30d = %d, want 4", ping.NodeTestFailures30d)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -616,6 +616,7 @@ func Run(ctx context.Context, version string) error {
|
||||
router.ApplyUpdateTelemetrySnapshot(&snap, now)
|
||||
router.ApplyLicensedFeatureTelemetrySnapshot(&snap, now)
|
||||
}
|
||||
applyNodeTestTelemetrySnapshot(&snap, telemetryPersistence, now)
|
||||
var actionSnapshot telemetry.PulseIntelligenceActionSnapshot
|
||||
if router != nil {
|
||||
actionSnapshot = router.GetPulseIntelligenceActionTelemetry(now.Add(-telemetry.PulseIntelligenceTelemetryWindow))
|
||||
|
||||
@@ -16,6 +16,7 @@ import (
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/agentexec"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/config"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/securityutil"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/telemetry"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/updates"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/extensions"
|
||||
pkglicensing "github.com/rcourtman/pulse-go-rewrite/pkg/licensing"
|
||||
@@ -394,3 +395,42 @@ func TestServerRun_RejectsWildcardTrustedProxyCIDR(t *testing.T) {
|
||||
t.Fatalf("expected wildcard trusted proxy configuration to be rejected, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Node connection test counts must reach the telemetry snapshot, and counts
|
||||
// older than the reporting window must not, otherwise the add-node stall this
|
||||
// counter exists to measure cannot be read from the fleet.
|
||||
func TestApplyNodeTestTelemetrySnapshotReadsTallyWithinWindow(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
persistence := config.NewConfigPersistence(dir)
|
||||
now := time.Now().UTC()
|
||||
|
||||
if err := persistence.RecordNodeTestOutcome(true, now.AddDate(0, 0, -40)); err != nil {
|
||||
t.Fatalf("record stale outcome: %v", err)
|
||||
}
|
||||
if err := persistence.RecordNodeTestOutcome(true, now); err != nil {
|
||||
t.Fatalf("record failure: %v", err)
|
||||
}
|
||||
if err := persistence.RecordNodeTestOutcome(false, now); err != nil {
|
||||
t.Fatalf("record success: %v", err)
|
||||
}
|
||||
|
||||
var snap telemetry.Snapshot
|
||||
applyNodeTestTelemetrySnapshot(&snap, persistence, now)
|
||||
|
||||
if snap.NodeTestAttempts30d != 2 {
|
||||
t.Fatalf("NodeTestAttempts30d = %d, want 2", snap.NodeTestAttempts30d)
|
||||
}
|
||||
if snap.NodeTestFailures30d != 1 {
|
||||
t.Fatalf("NodeTestFailures30d = %d, want 1", snap.NodeTestFailures30d)
|
||||
}
|
||||
}
|
||||
|
||||
// A missing tally is the normal state on an install that has never opened the
|
||||
// add-node dialog, and must report zero rather than fail the snapshot.
|
||||
func TestApplyNodeTestTelemetrySnapshotToleratesMissingTally(t *testing.T) {
|
||||
var snap telemetry.Snapshot
|
||||
applyNodeTestTelemetrySnapshot(&snap, config.NewConfigPersistence(t.TempDir()), time.Now().UTC())
|
||||
if snap.NodeTestAttempts30d != 0 || snap.NodeTestFailures30d != 0 {
|
||||
t.Fatalf("counts = %d/%d, want 0/0", snap.NodeTestAttempts30d, snap.NodeTestFailures30d)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/config"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/telemetry"
|
||||
)
|
||||
|
||||
// applyNodeTestTelemetrySnapshot adds content-free node connection test counts
|
||||
// from the local tally.
|
||||
//
|
||||
// ConfiguredConnections reports only connections that were saved, so without
|
||||
// these an install that tried to reach a node and could not is indistinguishable
|
||||
// from one that never opened the add-node dialog: both report zero connections
|
||||
// and stall at the "secured" activation stage.
|
||||
//
|
||||
// The window is the shared install-ID rotation window, the same one the
|
||||
// Pulse Intelligence counters use, so no counter outlives the pseudonymous
|
||||
// identifier it is reported against.
|
||||
func applyNodeTestTelemetrySnapshot(snap *telemetry.Snapshot, persistence *config.ConfigPersistence, now time.Time) {
|
||||
if snap == nil || persistence == nil {
|
||||
return
|
||||
}
|
||||
if now.IsZero() {
|
||||
now = time.Now()
|
||||
}
|
||||
tally, err := persistence.LoadNodeTestTally()
|
||||
if err != nil || tally == nil {
|
||||
return
|
||||
}
|
||||
since := now.UTC().Add(-telemetry.PulseIntelligenceTelemetryWindow)
|
||||
snap.NodeTestAttempts30d = tally.AttemptsSince(since)
|
||||
snap.NodeTestFailures30d = tally.FailuresSince(since)
|
||||
}
|
||||
Reference in New Issue
Block a user