mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-10-26 05:29:45 +00:00
Keep the metrics DB path override on the default org (#3036)
`PULSE_METRICS_DB_PATH` sets `Config.MetricsDBPath`. `MultiTenantMonitor.GetMonitor` builds each non-default tenant from `baseConfig.DeepCopy()`, which kept that path, so `monitoring.New` opened the same `metrics.db` for every organization. With the override set, per-org charts, reports and Pulse Assistant summaries all read from one shared store. Changes: - `GetMonitor` clears `MetricsDBPath` on non-default tenant configs after setting their `DataPath`. Each org then opens `metrics.db` in its own data directory, and `DeleteOrganization` removes it along with the rest of the org's data. - The default org still uses the override, which covers the single-install tmpfs/RAM-disk case it exists for. Metrics for non-default orgs stay on the data volume. - New `multi_tenant_monitor_metrics_path_test.go` sets the override, provisions two orgs, and checks that each opens its own `metrics.db` and cannot read series written by the default org or the other org. It fails on current `main`. - `TestMetricsStoreRuntimeOverridesStayOnMonitorBoundary` now also requires the clear to come between the tenant `DataPath` assignment and `New`. - The monitoring contract and `CONFIGURATION.md` (plus its public mirror) now describe the default-org-only behaviour. Verification: `go test ./internal/monitoring/ ./internal/config/` passes on Linux. `canonical_completion_guard.py` passes for this commit.
This commit is contained in:
@@ -55,7 +55,8 @@ Path overrides:
|
||||
- `PULSE_DATA_DIR` sets the base directory for `system.json`, encrypted files, and the bootstrap token.
|
||||
- `PULSE_METRICS_DB_PATH` sets only the metrics SQLite database path. Use this
|
||||
for tmpfs-backed metrics history without moving secrets or config off the
|
||||
persistent data directory.
|
||||
persistent data directory. It applies to the default org only; each
|
||||
non-default org keeps its own `metrics.db` in `/orgs/<org-id>/`.
|
||||
|
||||
Multi-tenant layout:
|
||||
- Default org uses the root data directory for backward compatibility.
|
||||
|
||||
@@ -624,6 +624,27 @@ These synthetic source controls reproduce a family matching #2382's second
|
||||
operator response, not that operator's cause, the original downgrade-resolved
|
||||
regression, native appliance acceptance or release availability.
|
||||
|
||||
### Metrics database override stays on the default organization
|
||||
|
||||
`PULSE_METRICS_DB_PATH` names one SQLite file, so it applies only to the
|
||||
default organization's monitor. `MultiTenantMonitor.GetMonitor` clears
|
||||
`MetricsDBPath` on each non-default tenant's detached config copy after
|
||||
setting its `DataPath`, and `monitoring.New` then opens that organization's
|
||||
store at `<org data dir>/metrics.db`. Before this, the deep copy carried the
|
||||
override into every tenant and all organizations wrote to and read from one
|
||||
metrics database, so per-org charts, reports and Pulse Assistant summaries
|
||||
could return another organization's series. Keeping tenant stores inside the
|
||||
org directory also means `DeleteOrganization` removes that org's metrics with
|
||||
the rest of its data. The trade-off is that tmpfs or RAM-disk placement
|
||||
through the override covers the default org only; non-default org metrics
|
||||
stay on the data volume. The default org's override path, rollup interval and
|
||||
retention handling are unchanged. Regression coverage:
|
||||
`internal/monitoring/multi_tenant_monitor_metrics_path_test.go` sets the
|
||||
override, provisions two orgs, and proves each opens its own `metrics.db`
|
||||
and cannot read series written by the default org or by the other org.
|
||||
`TestMetricsStoreRuntimeOverridesStayOnMonitorBoundary` also requires the
|
||||
clear to sit between the tenant `DataPath` assignment and `New`.
|
||||
|
||||
### Lightweight detached History reads
|
||||
|
||||
Guest, node, disk and storage History readers allocate one exact-sized snapshot
|
||||
@@ -5186,7 +5207,9 @@ aggregation cadence: `internal/monitoring/monitor.go` may pass the resolved
|
||||
data path, `PULSE_METRICS_DB_PATH`, and `PULSE_METRICS_ROLLUP_INTERVAL` through
|
||||
to `pkg/metrics`, but the SQLite path normalization, rollup bounds, and write
|
||||
amplification policy stay owned by the metrics store rather than by a
|
||||
monitoring-local helper.
|
||||
monitoring-local helper. The `PULSE_METRICS_DB_PATH` handoff reaches only the
|
||||
default org's monitor; non-default tenants always resolve `metrics.db` from
|
||||
their own data path.
|
||||
Install-wide telemetry counts are also monitoring-owned now. Any telemetry or
|
||||
reporting surface that claims installation totals must aggregate across the
|
||||
provisioned tenant set through the reloadable multi-tenant monitor boundary,
|
||||
|
||||
@@ -55,7 +55,8 @@ Path overrides:
|
||||
- `PULSE_DATA_DIR` sets the base directory for `system.json`, encrypted files, and the bootstrap token.
|
||||
- `PULSE_METRICS_DB_PATH` sets only the metrics SQLite database path. Use this
|
||||
for tmpfs-backed metrics history without moving secrets or config off the
|
||||
persistent data directory.
|
||||
persistent data directory. It applies to the default org only; each
|
||||
non-default org keeps its own `metrics.db` in `/orgs/<org-id>/`.
|
||||
|
||||
Multi-tenant layout:
|
||||
- Default org uses the root data directory for backward compatibility.
|
||||
|
||||
@@ -681,6 +681,19 @@ func TestMetricsStoreRuntimeOverridesStayOnMonitorBoundary(t *testing.T) {
|
||||
t.Fatalf("monitor.go must contain %q", snippet)
|
||||
}
|
||||
}
|
||||
|
||||
// The override names one file, so non-default tenants must not inherit it.
|
||||
data, err = os.ReadFile("multi_tenant_monitor.go")
|
||||
if err != nil {
|
||||
t.Fatalf("failed to read multi_tenant_monitor.go: %v", err)
|
||||
}
|
||||
tenantSource := string(data)
|
||||
dataPathIdx := strings.Index(tenantSource, "tenantConfig.DataPath = tenantPersistence.GetConfigDir()")
|
||||
clearIdx := strings.Index(tenantSource, "tenantConfig.MetricsDBPath = \"\"")
|
||||
newIdx := strings.Index(tenantSource, "monitor, err = New(tenantConfig)")
|
||||
if dataPathIdx < 0 || clearIdx < 0 || newIdx < 0 || !(dataPathIdx < clearIdx && clearIdx < newIdx) {
|
||||
t.Fatal("multi_tenant_monitor.go must clear MetricsDBPath for non-default tenants after setting DataPath and before New")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEscalationDeliveryDefersToCanonicalAlertSuppression(t *testing.T) {
|
||||
|
||||
@@ -220,6 +220,14 @@ func (mtm *MultiTenantMonitor) GetMonitor(orgID string) (*Monitor, error) {
|
||||
return nil, fmt.Errorf("failed to get persistence for org %s: %w", orgID, err)
|
||||
}
|
||||
tenantConfig.DataPath = tenantPersistence.GetConfigDir()
|
||||
if orgID != "default" {
|
||||
// PULSE_METRICS_DB_PATH names one database file. Inherited by every
|
||||
// tenant it would put all organizations' series in the same store, so
|
||||
// the override applies to the default org only. Other orgs keep
|
||||
// metrics.db under their own data directory, where org deletion also
|
||||
// removes it.
|
||||
tenantConfig.MetricsDBPath = ""
|
||||
}
|
||||
|
||||
// Load tenant-specific nodes from <orgDir>/nodes.enc
|
||||
nodesConfig, err := tenantPersistence.LoadNodesConfig()
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
package monitoring
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/config"
|
||||
"github.com/rcourtman/pulse-go-rewrite/internal/models"
|
||||
"github.com/rcourtman/pulse-go-rewrite/pkg/metrics"
|
||||
)
|
||||
|
||||
// PULSE_METRICS_DB_PATH names a single database file. Only the default org may
|
||||
// use it; every other org must keep its own metrics.db under its data
|
||||
// directory so organizations never read each other's series.
|
||||
func TestMultiTenantMonitorMetricsDBPathOverrideStaysOnDefaultOrg(t *testing.T) {
|
||||
mtp, defaultPersistence := newTestTenantPersistence(t)
|
||||
overridePath := filepath.Join(t.TempDir(), "metrics-tmpfs", "metrics.db")
|
||||
baseCfg := &config.Config{
|
||||
DataPath: defaultPersistence.GetConfigDir(),
|
||||
MetricsDBPath: overridePath,
|
||||
}
|
||||
mtm := NewMultiTenantMonitor(baseCfg, mtp, nil)
|
||||
t.Cleanup(mtm.Stop)
|
||||
|
||||
orgIDs := []string{"org-a", "org-b"}
|
||||
for _, orgID := range orgIDs {
|
||||
if err := mtp.SaveOrganization(&models.Organization{ID: orgID, DisplayName: orgID}); err != nil {
|
||||
t.Fatalf("SaveOrganization(%s) error = %v", orgID, err)
|
||||
}
|
||||
}
|
||||
|
||||
stores := make(map[string]*metrics.Store)
|
||||
for _, orgID := range append([]string{"default"}, orgIDs...) {
|
||||
monitor, err := mtm.GetMonitor(orgID)
|
||||
if err != nil {
|
||||
t.Fatalf("GetMonitor(%s) error = %v", orgID, err)
|
||||
}
|
||||
store := monitor.GetMetricsStore()
|
||||
if store == nil {
|
||||
t.Fatalf("GetMonitor(%s) has no metrics store", orgID)
|
||||
}
|
||||
stores[orgID] = store
|
||||
}
|
||||
|
||||
if baseCfg.MetricsDBPath != overridePath {
|
||||
t.Fatalf("base config MetricsDBPath = %q, want %q", baseCfg.MetricsDBPath, overridePath)
|
||||
}
|
||||
if _, err := os.Stat(overridePath); err != nil {
|
||||
t.Fatalf("default org should open the override path %q: %v", overridePath, err)
|
||||
}
|
||||
for _, orgID := range orgIDs {
|
||||
orgPersistence, err := mtp.GetPersistence(orgID)
|
||||
if err != nil {
|
||||
t.Fatalf("GetPersistence(%s) error = %v", orgID, err)
|
||||
}
|
||||
orgDBPath := filepath.Join(orgPersistence.GetConfigDir(), "metrics.db")
|
||||
if _, err := os.Stat(orgDBPath); err != nil {
|
||||
t.Fatalf("%s should open its own metrics database at %q: %v", orgID, orgDBPath, err)
|
||||
}
|
||||
}
|
||||
if stores["org-a"] == stores["org-b"] || stores["org-a"] == stores["default"] {
|
||||
t.Fatal("expected a distinct metrics store per organization")
|
||||
}
|
||||
|
||||
now := time.Now().Truncate(time.Second)
|
||||
for _, orgID := range []string{"default", "org-a"} {
|
||||
stores[orgID].WriteBatchSync([]metrics.WriteMetric{{
|
||||
ResourceType: "vm",
|
||||
ResourceID: "vm-" + orgID,
|
||||
MetricType: "cpu",
|
||||
Value: 42,
|
||||
Timestamp: now,
|
||||
Tier: metrics.TierRaw,
|
||||
}})
|
||||
}
|
||||
|
||||
for _, tc := range []struct {
|
||||
orgID string
|
||||
resourceID string
|
||||
wantPoints bool
|
||||
}{
|
||||
{orgID: "default", resourceID: "vm-default", wantPoints: true},
|
||||
{orgID: "default", resourceID: "vm-org-a", wantPoints: false},
|
||||
{orgID: "org-a", resourceID: "vm-org-a", wantPoints: true},
|
||||
{orgID: "org-a", resourceID: "vm-default", wantPoints: false},
|
||||
{orgID: "org-b", resourceID: "vm-org-a", wantPoints: false},
|
||||
{orgID: "org-b", resourceID: "vm-default", wantPoints: false},
|
||||
} {
|
||||
points, err := stores[tc.orgID].Query("vm", tc.resourceID, "cpu", now.Add(-time.Minute), now.Add(time.Minute), 0)
|
||||
if err != nil {
|
||||
t.Fatalf("%s Query(%s) error = %v", tc.orgID, tc.resourceID, err)
|
||||
}
|
||||
if got := len(points) > 0; got != tc.wantPoints {
|
||||
t.Fatalf("%s store points for %s = %d, want present=%v", tc.orgID, tc.resourceID, len(points), tc.wantPoints)
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user