mirror of
https://github.com/okieselbach/AutopilotMonitor.git
synced 2026-10-24 20:53:06 +00:00
66fc23b609
PR2 of the safe cascade-delete plan
(.claude/plans/session-deletion-cascade-safe.md). Adds the per-table
delete primitives the cascade worker will use in PR4 plus the §17
SessionInventoryContributions side-row that makes per-session inventory
counter increments at-most-once and exactly reversible. No production
caller invokes the cascade primitives yet; the inventory rewrite IS a
call-site change in VulnerabilityCorrelationService and ships in this PR.
== Cascade delete helpers (TableStorageService.Deletion.cs) ==
DeleteByExactKeysInBatchesAsync(tableName, keys) is the load-bearing
helper: groups by PartitionKey (Azure batch transactions are
partition-scoped), chunks at 100 actions per batch, falls back to
per-row DeleteEntityAsync with 404-ignore on whole-batch rollback.
Returns (Attempted, DeletedNow, AlreadyMissing) for accurate progress
accounting on idempotent re-runs of partial cascades. Plus 9 thin
wrappers per the §3 taxonomy: 7 keyed (EventTypeIndex, CveIndex,
SessionsByTerminal, SessionsByStage, DeadEndsByReason,
ClassifierVerdictsByIdLevel, SignalsByKind), 1 single-row
(DeviceSnapshot), and 2 enumerate-then-delete (Signals,
DecisionTransitions for the PK_BY_SESSION class). All fail-loud per
memory feedback_storage_helpers_fail_soft.
== SoftwareInventory side-row (TableStorageService.Inventory.cs) ==
Plan §17 design: each inventory-correlated session contributes ONE
SessionInventoryContributions row (PK=tenantId, RK=sessionId) with the
normalized software keys it added. Increments to SoftwareInventory
counters are now driven by deltas against this row, so:
- WG sessions that fire software_inventory_analysis 4× no longer
over-count by 4× per software per session
- Cascade decrement is exact: the side-row is the source of truth
for which counters this session contributed to
- Pre-side-row sessions (created before this deploy) have no
contributions row → cascade skips inventory entirely for them
(drift accepted on the historical backlog, ages out via 90d
retention).
Helpers (all fail-loud, all bounded retry per §12-Q10 = 10 attempts /
60s wall-clock cap, all ETag-CAS):
- GetSessionInventoryContributionsAsync: returns prior key list or
null on 404
- UpsertSessionInventoryContributionsAsync: side-row CAS that
computes (added, removed) delta against the new key set
- DeleteSessionInventoryContributionsAsync: 404-ignore tombstone
- IncrementSoftwareInventoryEntryAsync: ETag-CAS +1 with first-time
AddEntity that writes full metadata
- DecrementSoftwareInventoryEntryAsync: ETag-CAS -1, clamp >= 0,
deletes the row when count hits zero
Codec: SoftwareKeysJsonCodec encodes the key list as raw JSON below
30 KB, gzip+Base64 above. Same shape DeletionManifestBuilder (PR1)
already reads at preflight, so the cascade decrement reads back what
the increment wrote with no schema gap. Plus deterministic ComputeDelta
(case-insensitive composite key) so concurrent correlations on the
same session collapse to a single counter update.
== VulnerabilityCorrelationService rewrite ==
The Task.Run fire-and-forget at the old line 246 now calls the new
ApplyInventoryContributionsAsync helper instead of
UpsertSoftwareInventoryAsync. Each Increment / Decrement is wrapped
in its OWN try/catch so one counter failure does NOT abort the rest
of the loop (Plan §17.3 + §17.8): without that, the side-row would
already promise the new key set while subsequent runs see an empty
delta and never retry the under-counted keys. Each individual failure
emits an `inventory_counter_write_failed` audit entry with operation
+ vendor + name + version + exception type/message via the existing
LogAuditEntryAsync. IMaintenanceRepository was added as an OPTIONAL
constructor param (positional default null) so existing 8-arg test
factories keep compiling unchanged.
ParseSoftwareInventoryItems exposed as internal static for testability;
preserves the same dedup + cpe-mapping behaviour as the old
UpsertSoftwareInventoryAsync path. The legacy method is left in place
on TableStorageService.Rules.cs for now (no remaining callers; PR7
cleanup can remove if desired).
== Tests (39 new) ==
- SoftwareKeysJsonCodecTests (12): raw vs gzip+Base64 paths,
determinism, malformed-input rejection, delta cases (added /
removed / identity / case-insensitive), composite-key shape.
- TableStorageDeletionBatchTests (8, SDK-mocked via internal test
ctor): empty input, PK grouping (one batch per PK), chunking at
100, batch 404 + all per-row 404, mixed batch 404 (2 succeed +
3 missing) with accurate accounting, non-404 propagates, no
over-deletion, ETag.All on every action.
- TableStorageInventoryCasTests (9): Increment first-create with
full metadata, CAS +1 on existing row, 412 retry, 409 first-create
race fall back to Update, empty-triple skip; Decrement 404 no-op,
clamp-at-zero deletes the row, count>1 subtract, 0-count clamp,
412 retry.
- VulnerabilityCorrelationServiceContributionsTests (10): Parser
correctness, end-to-end add/remove flow with Mock.Verify counts,
empty-input no-op, side-row exhaustion swallowed, per-counter
failure does not abort + audits the failed key (both increment
and decrement variants).
Backend suite: 1290/1290 green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
224 lines
8.2 KiB
C#
224 lines
8.2 KiB
C#
using System.Collections.Generic;
|
|
using System.Linq;
|
|
using AutopilotMonitor.Shared.Models.Deletion;
|
|
using Xunit;
|
|
|
|
namespace AutopilotMonitor.Functions.Tests;
|
|
|
|
/// <summary>
|
|
/// Pure-logic tests for the SoftwareKeysJson codec used by the inventory contributions side-row
|
|
/// (Plan §17). Encoding determinism + delta computation are the load-bearing properties — the
|
|
/// cascade decrement step reads back the side-row's keys and is byte-identical to what the
|
|
/// increment path wrote.
|
|
/// </summary>
|
|
public class SoftwareKeysJsonCodecTests
|
|
{
|
|
[Fact]
|
|
public void Encode_uses_raw_json_below_compression_threshold()
|
|
{
|
|
var keys = new List<DeletionDecrementKey>
|
|
{
|
|
new DeletionDecrementKey { Vendor = "Contoso", Name = "Widget", Version = "1.0" },
|
|
new DeletionDecrementKey { Vendor = "Fabrikam", Name = "Gadget", Version = "2.0" },
|
|
};
|
|
|
|
var encoded = SoftwareKeysJsonCodec.Encode(keys);
|
|
|
|
Assert.False(encoded.IsCompressed);
|
|
Assert.Equal(2, encoded.KeyCount);
|
|
Assert.Contains("Contoso", encoded.Encoded);
|
|
Assert.Contains("Fabrikam", encoded.Encoded);
|
|
}
|
|
|
|
[Fact]
|
|
public void Encode_uses_gzip_base64_above_compression_threshold()
|
|
{
|
|
// Generate >30 KB of raw JSON: 1000 distinct keys with ~30-char vendor each.
|
|
var keys = new List<DeletionDecrementKey>();
|
|
for (var i = 0; i < 1000; i++)
|
|
{
|
|
keys.Add(new DeletionDecrementKey
|
|
{
|
|
Vendor = $"VendorWithALongerNameForBulk-{i:D6}",
|
|
Name = $"Product-{i:D6}",
|
|
Version = $"1.{i}.0",
|
|
});
|
|
}
|
|
|
|
var encoded = SoftwareKeysJsonCodec.Encode(keys);
|
|
|
|
Assert.True(encoded.IsCompressed);
|
|
Assert.Equal(1000, encoded.KeyCount);
|
|
// Compressed payload should be much smaller than raw (heavy redundancy).
|
|
var rawSize = System.Text.Json.JsonSerializer.SerializeToUtf8Bytes(keys, DeletionManifestJson.SerializerOptions).Length;
|
|
Assert.True(encoded.Encoded.Length * 3 / 4 < rawSize / 3,
|
|
$"Compression should shrink payload by >3x. Got base64 {encoded.Encoded.Length} chars vs raw {rawSize} bytes.");
|
|
}
|
|
|
|
[Fact]
|
|
public void Encode_decode_round_trips_keys_in_deterministic_order()
|
|
{
|
|
var keys = new List<DeletionDecrementKey>
|
|
{
|
|
new DeletionDecrementKey { Vendor = "Tailspin", Name = "Service", Version = "3.0" },
|
|
new DeletionDecrementKey { Vendor = "Contoso", Name = "Widget", Version = "1.0" },
|
|
new DeletionDecrementKey { Vendor = "Fabrikam", Name = "Gadget", Version = "2.0" },
|
|
};
|
|
|
|
var encoded = SoftwareKeysJsonCodec.Encode(keys);
|
|
var decoded = SoftwareKeysJsonCodec.Decode(encoded.Encoded, encoded.IsCompressed);
|
|
|
|
Assert.Equal(3, decoded.Count);
|
|
// Builder sorts by (Vendor, Name, Version) so the side-row content is byte-stable
|
|
// across builds — critical for manifest-determinism (Plan §18.6).
|
|
Assert.Equal("Contoso", decoded[0].Vendor);
|
|
Assert.Equal("Fabrikam", decoded[1].Vendor);
|
|
Assert.Equal("Tailspin", decoded[2].Vendor);
|
|
}
|
|
|
|
[Fact]
|
|
public void Encode_with_compression_round_trips_through_gzip_base64()
|
|
{
|
|
// Force the compressed path with a key set that just barely exceeds the threshold.
|
|
var keys = new List<DeletionDecrementKey>();
|
|
for (var i = 0; i < 500; i++)
|
|
{
|
|
keys.Add(new DeletionDecrementKey
|
|
{
|
|
Vendor = "VendorWithALongerNameForBulk-" + i.ToString("D6"),
|
|
Name = "Product-" + i.ToString("D6"),
|
|
Version = "1." + i + ".0",
|
|
});
|
|
}
|
|
|
|
var encoded = SoftwareKeysJsonCodec.Encode(keys);
|
|
var decoded = SoftwareKeysJsonCodec.Decode(encoded.Encoded, encoded.IsCompressed);
|
|
|
|
Assert.True(encoded.IsCompressed);
|
|
Assert.Equal(500, decoded.Count);
|
|
Assert.Equal(keys[0].Vendor, decoded.First(k => k.Vendor == keys[0].Vendor).Vendor);
|
|
}
|
|
|
|
[Fact]
|
|
public void Decode_empty_string_returns_empty_list()
|
|
{
|
|
Assert.Empty(SoftwareKeysJsonCodec.Decode("", false));
|
|
Assert.Empty(SoftwareKeysJsonCodec.Decode("", true));
|
|
}
|
|
|
|
[Fact]
|
|
public void Decode_invalid_base64_throws_invalid_data()
|
|
{
|
|
var ex = Assert.Throws<System.IO.InvalidDataException>(
|
|
() => SoftwareKeysJsonCodec.Decode("!!! not base64 !!!", isCompressed: true));
|
|
Assert.Contains("Base64", ex.Message);
|
|
}
|
|
|
|
[Fact]
|
|
public void Decode_malformed_json_throws_invalid_data()
|
|
{
|
|
Assert.Throws<System.IO.InvalidDataException>(
|
|
() => SoftwareKeysJsonCodec.Decode("{not-an-array}", isCompressed: false));
|
|
}
|
|
|
|
[Fact]
|
|
public void ComputeDelta_returns_all_added_when_oldKeys_is_empty()
|
|
{
|
|
var newKeys = new[]
|
|
{
|
|
new DeletionDecrementKey { Vendor = "Contoso", Name = "Widget", Version = "1.0" },
|
|
new DeletionDecrementKey { Vendor = "Fabrikam", Name = "Gadget", Version = "2.0" },
|
|
};
|
|
|
|
var (added, removed) = SoftwareKeysJsonCodec.ComputeDelta(System.Array.Empty<DeletionDecrementKey>(), newKeys);
|
|
|
|
Assert.Equal(2, added.Count);
|
|
Assert.Empty(removed);
|
|
}
|
|
|
|
[Fact]
|
|
public void ComputeDelta_returns_one_added_when_newKeys_adds_a_phase2_app()
|
|
{
|
|
var oldKeys = new[]
|
|
{
|
|
new DeletionDecrementKey { Vendor = "Contoso", Name = "Widget", Version = "1.0" },
|
|
};
|
|
var newKeys = new[]
|
|
{
|
|
new DeletionDecrementKey { Vendor = "Contoso", Name = "Widget", Version = "1.0" },
|
|
new DeletionDecrementKey { Vendor = "Fabrikam", Name = "NewApp", Version = "2.0" },
|
|
};
|
|
|
|
var (added, removed) = SoftwareKeysJsonCodec.ComputeDelta(oldKeys, newKeys);
|
|
|
|
Assert.Single(added);
|
|
Assert.Contains(SoftwareKeysJsonCodec.CompositeKey("Fabrikam", "NewApp", "2.0"), added);
|
|
Assert.Empty(removed);
|
|
}
|
|
|
|
[Fact]
|
|
public void ComputeDelta_returns_one_removed_when_uninstall_happens_between_phases()
|
|
{
|
|
var oldKeys = new[]
|
|
{
|
|
new DeletionDecrementKey { Vendor = "Contoso", Name = "Widget", Version = "1.0" },
|
|
new DeletionDecrementKey { Vendor = "Fabrikam", Name = "OldApp", Version = "0.9" },
|
|
};
|
|
var newKeys = new[]
|
|
{
|
|
new DeletionDecrementKey { Vendor = "Contoso", Name = "Widget", Version = "1.0" },
|
|
};
|
|
|
|
var (added, removed) = SoftwareKeysJsonCodec.ComputeDelta(oldKeys, newKeys);
|
|
|
|
Assert.Empty(added);
|
|
Assert.Single(removed);
|
|
Assert.Contains(SoftwareKeysJsonCodec.CompositeKey("Fabrikam", "OldApp", "0.9"), removed);
|
|
}
|
|
|
|
[Fact]
|
|
public void ComputeDelta_is_empty_when_keys_are_identical()
|
|
{
|
|
var keys = new[]
|
|
{
|
|
new DeletionDecrementKey { Vendor = "Contoso", Name = "Widget", Version = "1.0" },
|
|
new DeletionDecrementKey { Vendor = "Fabrikam", Name = "Gadget", Version = "2.0" },
|
|
};
|
|
|
|
var (added, removed) = SoftwareKeysJsonCodec.ComputeDelta(keys, keys);
|
|
|
|
Assert.Empty(added);
|
|
Assert.Empty(removed);
|
|
}
|
|
|
|
[Fact]
|
|
public void ComputeDelta_is_case_insensitive()
|
|
{
|
|
// Upstream normalization is best-effort; "Contoso" and "contoso" must collapse so a case-
|
|
// flip between two correlation events on the same session doesn't trigger spurious deltas.
|
|
var oldKeys = new[]
|
|
{
|
|
new DeletionDecrementKey { Vendor = "Contoso", Name = "Widget", Version = "1.0" },
|
|
};
|
|
var newKeys = new[]
|
|
{
|
|
new DeletionDecrementKey { Vendor = "contoso", Name = "widget", Version = "1.0" },
|
|
};
|
|
|
|
var (added, removed) = SoftwareKeysJsonCodec.ComputeDelta(oldKeys, newKeys);
|
|
|
|
Assert.Empty(added);
|
|
Assert.Empty(removed);
|
|
}
|
|
|
|
[Fact]
|
|
public void CompositeKey_builds_three_part_separated_value()
|
|
{
|
|
var ck = SoftwareKeysJsonCodec.CompositeKey("Contoso", "Widget", "1.0");
|
|
Assert.NotEqual("ContosoWidget1.0", ck); // separator IS present
|
|
// Different orderings of the same string components produce different composites.
|
|
var ck2 = SoftwareKeysJsonCodec.CompositeKey("Widget", "Contoso", "1.0");
|
|
Assert.NotEqual(ck, ck2);
|
|
}
|
|
}
|