Invoke-RackpadPopulation/README.md

# Invoke-RackpadPopulation

Idempotently populates one or more Rackpad labs from live infrastructure.

The live systems are authoritative. The settings document only supplies what no API can know, which is the physical layout consisting of rooms, racks, and rack unit placement, together with the address ranges that serve them. It cannot create devices, VLANs, subnets, or SSIDs.

Every run is safe to repeat. Objects are matched on stable natural keys, only differing properties are transmitted, and a run against an unchanged environment produces no write traffic at all.

---

## Ownership model

| Source | Owns |
| --- | --- |
| Omada | Switches, access points, switch ports, cables and topology, VLANs, wireless controller, SSIDs, radios, client associations |
| UniFi | The same shapes as Omada, and can run alongside it |
| Proxmox | Hypervisor hosts, bridges, guests, guest network interfaces. One entry per cluster and per standalone host |
| Hyper-V | The same shapes as Proxmox, and can run alongside it |
| OPNsense | VLANs, subnets, virtual IP addresses, the routed edge, firewall interfaces |
| Public IP lookup | The WAN address as a `/32` subnet carrying ISP, ASN, and location detail |
| Settings document | Rooms, racks, placement rules, IP ranges |

DHCP is deliberately out of scope. No DHCP scope is ever created, updated, or deleted, and IP assignments are always written using the `static` allocation mode. DHCP stays a manual concern inside Rackpad.

---

## How the layer two and layer three split is modelled

The lab has physical layer two equipment and a virtualized layer three edge. The graph is built so that both read correctly in the Rackpad user interface at the same time.

```mermaid
graph TD
    subgraph Physical["Room and rack, from the settings document"]
        RACK["Rack"]
        SW["Switch, rack placed"]
        AP["Access point, room placed"]
        HOST["Hypervisor host, rack placed"]
        RACK --> SW
        RACK --> HOST
    end

    subgraph L2["Layer two, from Omada"]
        SWPORT["Switch ports, rj45 and sfp_plus"]
        UPLINK["Access point uplink port"]
        CABLE["Cable links"]
        SSID["SSIDs and radios"]
        SW --> SWPORT
        AP --> UPLINK
        SWPORT -->|uplink and downlink topology| CABLE
        UPLINK --> CABLE
        AP --> SSID
    end

    subgraph L3["Layer three, virtualized"]
        VSW["Virtual switches, bridges on the host"]
        FW["Firewall, virtual, parented to its host"]
        VM["VMs and containers, virtual"]
        VNIC["Virtual ports, attached to a virtual switch"]
        HOST --> VSW
        HOST --> FW
        HOST --> VM
        FW --> VNIC
        VM --> VNIC
        VNIC --> VSW
    end

    subgraph IPAM["Addressing"]
        VLAN["VLANs"]
        SUBNET["Subnets"]
        ZONE["IP zones, ranges per room and rack"]
        ASSIGN["IP assignments"]
        WAN["WAN slash 32, ISP and ASN"]
        VLAN --> SUBNET
        SUBNET --> ZONE
        SUBNET --> ASSIGN
        FW --> VLAN
        FW --> WAN
    end
```

The result is a rack elevation that reflects reality, alongside a virtual topology that explains where layer three actually lives: the firewall appears as a virtual device parented to its hypervisor, its interfaces appear as ports attached to the correct bridge, and the VLANs and subnets it serves appear in the Networks workspace.

---

## Execution flow

```mermaid
flowchart TD
    START([Start]) --> READ["Read the settings document, expand variables, load with XPath"]
    READ --> LOOP{"For each enabled instance"}

    LOOP --> AUTH["Sign in to Rackpad and capture the bearer token"]
    AUTH --> LAB["Resolve the lab by name, creating it when permitted"]
    LAB --> INV["Pre-load every collection, about twenty reads"]
    INV --> STATE["Load the managed state document"]

    STATE --> COLLECT{"Collect enabled sources"}
    COLLECT --> OMADA["Omada, controller interface preferred"]
    COLLECT --> UNI["UniFi, network interface preferred"]
    COLLECT --> PVE["Proxmox, one call per endpoint"]
    COLLECT --> HV["Hyper-V, local or over a CIM session"]
    COLLECT --> OPN["OPNsense"]
    COLLECT --> PUB["Public address lookup"]

    OMADA --> MERGE
    UNI --> MERGE
    PVE --> MERGE
    HV --> MERGE
    OPN --> MERGE
    PUB --> MERGE

    MERGE["Build the desired state graph on natural keys"] --> PUBLISH["Publish in 13 dependency phases"]
    PUBLISH --> DIFF{"Per object"}
    DIFF -->|no match| CREATE["Create, then record ownership"]
    DIFF -->|match, differs| UPDATE["Patch only the differing properties"]
    DIFF -->|match, identical| SKIP["No request"]

    CREATE --> PRUNE
    UPDATE --> PRUNE
    SKIP --> PRUNE

    PRUNE["Prune recorded objects no longer wanted"] --> SAVE["Save the managed state document"]
    SAVE --> LOOP
    LOOP -->|done| EXIT([Exit code])
```

### Publication phases

Objects are written in dependency order so that everything referenced already exists.

| # | Phase | Depends on |
| --- | --- | --- |
| 1 | Lab | — |
| 2 | VLANs | Lab |
| 3 | Rooms | Lab |
| 4 | Racks | Rooms |
| 5 | Physical devices | Racks, Rooms |
| 6 | Virtual and attached devices | Physical devices |
| 7 | Virtual switches | Host devices |
| 8 | Ports | Devices, VLANs, virtual switches |
| 9 | Subnets | VLANs |
| 10 | IP zones | Subnets |
| 11 | IP assignments | Subnets, devices |
| 12 | Cable links | Ports |
| 13 | Wireless | Devices, VLANs, SSIDs |

---

## Idempotency and pruning

Every object is identified by a natural key rather than by a Rackpad identifier.

| Entity | Natural key |
| --- | --- |
| Room, Rack | lab + name |
| Device | lab + hostname |
| Port | hostname + port name |
| Cable link | both port keys, sorted so direction does not matter |
| Client association | client hostname |
| VLAN | lab + VLAN id |
| Subnet | lab + CIDR |
| IP zone | subnet CIDR + start + end |
| IP assignment | subnet CIDR + address |
| Virtual switch | host hostname + name |
| SSID | lab + name |
| Radio | AP hostname + slot name |

**Pruning only ever deletes objects that this script created.** Each created object is recorded in `Data/<InstanceName>.state.json`. On the next run, any recorded entry whose natural key is no longer present in the desired state is deleted, in reverse dependency order. Objects created by hand in the Rackpad user interface are never recorded, and therefore never pruned. A `404` during deletion counts as success, which keeps the operation repeatable.

Delete the state document and the script loses the ability to prune, but it will never delete anything unexpected.

---

## Getting started

1. Populate the environment variables the settings document references. User and machine scoped variables are promoted into the process automatically, so a variable created after the shell started still resolves.

   ```
   RACKPAD_BASE_URL, RACKPAD_USERNAME, RACKPAD_PASSWORD
   OMADA_API_BASE_URL, OMADA_API_CLIENT_ID, OMADA_API_CLIENT_SECRET
   OMADA_API_USERNAME, OMADA_API_PASSWORD            (controller login: topology, VLANs, SSIDs)
   UNIFI_API_BASE_URL, UNIFI_API_USERNAME, UNIFI_API_PASSWORD, UNIFI_API_KEY
   PROXMOX_API_BASE_URL, PROXMOX_API_TOKEN_ID, PROXMOX_API_TOKEN_SECRET
   OPNSENSE_API_BASE_URL, OPNSENSE_API_KEY, OPNSENSE_API_SECRET
   HYPERV_USERNAME, HYPERV_PASSWORD, HYPERV_GUEST_USERNAME, HYPERV_GUEST_PASSWORD
   ```

2. Edit `Settings/Invoke-RackpadPopulation.xml`. Set `Instance/@Enabled` to `true`, name the lab, enable the sources you have credentials for, and describe your rooms, racks, and placements.

3. Preview.

   ```powershell
   .\Invoke-RackpadPopulation.ps1 -WhatIf -Verbose
   ```

4. Apply.

   ```powershell
   .\Invoke-RackpadPopulation.ps1 -Verbose
   ```

> `-WhatIf` reports the first few phases accurately, then under-reports. Because nothing is actually created, later phases cannot resolve the identifiers of objects that a real run would have created, so ports, cables, and wireless show as zero on a first preview against an empty lab. Preview against an already populated lab and the numbers are exact.

`-Verbose` reports every individual create, update, and skip, including which properties differed. Without it only the per-entity summary is written.

### Parameters

| Parameter | Purpose |
| --- | --- |
| `-ConfigurationPath` | Overrides the settings document. Defaults to `Settings\<ScriptBaseName>.xml`. |
| `-InstanceName` | Processes only the named instances. |
| `-SkipPrune` | Suppresses pruning for this run. |
| `-LogDirectory` | Overrides the transcript directory. |
| `-ContinueOnError` | Continues to the next instance after a failure. |
| `-WhatIf` | Previews without writing. |

### Exit codes

| Code | Meaning |
| --- | --- |
| 0 | Every enabled instance reconciled successfully |
| 1000 | Completed, but one or more individual objects failed to write |
| 2000 | An instance failed outright |
| 6000 | The toolkit failed to load |

---

## Settings document

The document is named after the script, so `Invoke-RackpadPopulation.ps1` reads `Settings\Invoke-RackpadPopulation.xml`.

Multiple instances live in one file, each gated by `Enabled`. Values are expanded in three passes before parsing:

1. Environment variables in the `%NAME%` form, via `System.Environment`.
2. PowerShell expressions, via `$ExecutionContext.InvokeCommand.ExpandString`, so `$env:NAME` and `$($env:NAME)` resolve.
3. The result is loaded into a `System.Xml.XmlDocument` and navigated with XPath.

> Because pass two evaluates PowerShell, treat this document with the same care as a script, and escape any literal `$` that is not meant as an expression with a backtick.

Full PowerShell is available, which is useful for tolerating variable naming drift:

```xml
<Omada BaseURL="$(if ($env:OMADA_API_BASE_URL) { $env:OMADA_API_BASE_URL } else { $env:OAMDA_API_BASE_URL })">
```

Secrets belong in `CDATA` sections so reserved XML characters survive. Prefer keeping the value in an environment variable and referencing it.

```xml
<Rackpad BaseURL="$env:RACKPAD_BASE_URL" SkipCertificateCheck="false">
    <Username><![CDATA[$env:RACKPAD_USERNAME]]></Username>
    <Password><![CDATA[$env:RACKPAD_PASSWORD]]></Password>
</Rackpad>
```

### Rooms, racks, and placement

Leave `Rooms` and `Racks` empty and every discovered device stays loose in the lab. Declare them and they are created, and `Placement` rules file matching equipment into them.

Placement is rule based rather than per device. `InclusionExpression` and `ExclusionExpression` are regular expressions matched case insensitively against whichever value `MatchBy` selects (`Hostname`, `MAC`, or `Serial`). A rule matches when the inclusion expression matches and the exclusion expression, if present, does not.

**Rules are evaluated in order and the first match wins**, so specific rules go above general ones, exactly as in an access list. A handful of rules covers an entire estate:

```xml
<Placements>
    <!-- Wireless first, so the catch-all never claims an access point. -->
    <Placement InclusionExpression=".*-AP-.*" MatchBy="Hostname" Room="Server Room" Enabled="true" />

    <!-- The one device worth pinning to a specific rack unit. -->
    <Placement InclusionExpression=".*-SW-0*1$" MatchBy="Hostname" Rack="RACK-01" StartU="42" Enabled="true" />

    <!-- Everything else racks, except storage. -->
    <Placement InclusionExpression=".*" ExclusionExpression=".*NAS.*|.*STORAGE.*" MatchBy="Hostname" Rack="RACK-01" Enabled="true" />
</Placements>
```

`StartU` is optional. Rackpad requires a rack unit for anything rack placed, so a device matched by a broad rule is assigned the highest free block that fits. **Assignment is sticky**: a device that already holds a position keeps it, so the elevation never reshuffles between runs. If a rack has no free block left, the device is placed loose in the room and a warning is written rather than the write failing.

The literal `Match` attribute from earlier versions is still accepted and is treated as an anchored exact expression.

### IP ranges

Ranges become IP zones inside a subnet that a source already discovered. `Kind` accepts `static`, `reserved`, or `infrastructure`; a `dhcp` kind is rejected and downgraded to `static`. `Room` and `Rack` are recorded in the zone description so the range stays attributable.

```xml
<IPRange Subnet="10.0.0.0/24" Kind="infrastructure" StartIP="10.0.0.1" EndIP="10.0.0.31" Room="Server Room" Rack="RACK-01" Enabled="true" />
```

---

## Layout

```
Invoke-RackpadPopulation.ps1        Orchestration only
Settings/
  Invoke-RackpadPopulation.xml      Named after the script
Data/
  <Instance>.state.json             What this script created, used for pruning
Toolkit/
  Toolkit.ps1                       Dot-sources Functions, imports Modules, manages the transcript
  Functions/
    Invoke-RestAPIRequest.ps1       Shared transport: retry, rate limits, status translation
    Get-RackpadPopulationConfiguration.ps1
    Connect-RackpadInstance.ps1
    Invoke-RackpadRequest.ps1
    Get-RackpadInventory.ps1
    Get-RackpadManagedState.ps1
    Save-RackpadManagedState.ps1
    Sync-RackpadObject.ps1          The create / update / no-op primitive
    Remove-RackpadOrphanedObject.ps1
    Connect-OmadaController.ps1     Chooses the controller interface or the Open API
    Invoke-OmadaRequest.ps1         Per-request degradation between the two
    Get-OmadaInventory.ps1
    Connect-UnifiController.ps1     Chooses the network interface or the integration API
    Invoke-UnifiRequest.ps1         Per-request degradation between the two
    Get-UnifiInventory.ps1
    Get-ColorPalette.ps1            Shared palette for VLAN and SSID colors
    Get-ProxmoxInventory.ps1
    Get-HyperVInventory.ps1
    Get-OPNsenseInventory.ps1
    Get-PublicIPInformation.ps1
    ConvertTo-RackpadDesiredState.ps1
    Publish-RackpadDesiredState.ps1
  Modules/
    Indented.Net.IP/                Bundled; supplies all subnet arithmetic
```

---

## Verification status

| Area | Status |
| --- | --- |
| Rackpad create / update / no-op | Verified live. 9 devices, 117 ports, rack, room, subnet, wireless all created, then a repeat run reported every entity `Unchanged` with zero write traffic. |
| Pruning | Verified live. 133 objects removed, target lab confirmed empty, neighbouring lab untouched. |
| Omada controller interface | Verified live. 9 devices, 117 ports, 16 VLANs, real SSID configuration, and 5 inter-switch cables drawn with both ends known. |
| Combined layer two and layer three | Verified live. 10 devices, 136 ports, 16 VLANs, 18 subnets with 13 linked to their VLAN, 14 IP assignments, all colored, repeat run entirely unchanged. |
| VLAN colors | Verified live. 16 VLANs, all distinct, stable across runs. |
| Omada Open API fallback | Verified live. Sites, devices, switch port lists, and client sessions. |
| UniFi collection | **Unverified.** Written against the documented network and integration API shapes; no controller was available. The Omada path was regression tested after the shared refactor. |
| Placement rules and rack allocation | Verified live. Regex inclusion and exclusion, first match wins, automatic rack units, and stickiness across runs. |
| OPNsense collection | Verified live. 21 interfaces, 13 VLANs, 18 derived subnets, 3 virtual IPs, 4 gateways. |
| Virtual IP handling | Verified live. Two addresses inside the WAN block became assignments; a third outside it contributed its own subnet. |
| Hyper-V collection | Verified live against localhost. Host detail, virtual switches with correct kinds, guest with adapter, MAC, and bridge. |
| Proxmox collection | **Unverified.** Written against the documented API shape; no credentials were available. |
| Multiple Proxmox endpoints | **Unverified.** Overlapping endpoints collapse duplicates on the natural key. |

---

## Service rules worth knowing

These are constraints the Rackpad API enforces that the collectors have to respect. Each was found by a failed write rather than from documentation.

| Rule | How it is handled |
| --- | --- |
| A rack placed device must carry a start unit | Devices matched by a broad placement rule are allocated the highest free block that fits, and keep it thereafter |
| Rack positions may not overlap | Declared positions are reserved in a prior pass, before anything is allocated |
| A subnet may not overlap another subnet in the same lab | The public address only contributes a `/32` when no reported subnet already covers it |
| A subnet gateway may not be an ordinary endpoint assignment | An assignment whose address equals its subnet gateway is reclassified as infrastructure |
| A management address must be a valid IPv4 address | The routed edge resolves to an internal interface address rather than the hostname it is reached by |
| A client association requires the client to exist as a device | Associations are only emitted when `ManageClientEndpoints` creates those devices |
| Optional fields are defaulted rather than nulled | Rack geometry is omitted entirely for anything not rack placed, so it never reports a phantom change |

---

## Implementation notes

**Transport.** Everything flows through one `System.Net.Http` based function. `Invoke-RestMethod` is not used, because Omada transmits its token as `Authorization: AccessToken=<value>`, which standard clients reject during header validation. The transport adds headers without validation, translates status codes into readable text, honours `Retry-After`, and retries `408`, `429`, and `5xx` with linear backoff.

**Subnet arithmetic** is delegated to the bundled `Indented.Net.IP` module, using `Get-NetworkSummary` and `Test-SubnetMember`.

**Omada and UniFi are handled identically.** Both collectors emit the same normalized model, and the desired state graph is built from a combined site list rather than by branching on vendor, so an estate running both is a single picture. Each controller still contributes its own controller record, and every access point is attributed to the controller that reported it.

**Omada** has two transports and prefers the richer one.

The controller interface, under `/{omadacId}/api/v2`, is what the Omada web interface consumes. It reports switch uplinks and downlinks including the peer port, per port LLDP neighbours, VLAN membership and media type per port, VLAN definitions with gateway subnets, and the real SSID configuration. It needs a controller login and is not a published contract.

The Open API, under `/openapi/v1`, is published and versioned but narrower: no uplinks and no SSID configuration, which forces SSIDs to be inferred from whichever clients happen to be associated.

Supply both and the controller interface is used with the Open API held in reserve. Degradation is per request, so an expired session or an unimplemented route costs only that data. Set `PreferControllerInterface` to `false` to pin collection to the Open API.

Two controller quirks are handled rather than hard-coded: some routes reject paging parameters while others require them, so a rejected first page is retried without paging; and collections arrive either as a bare array or as a `data` array with a total, which must be distinguished carefully because asking a bare array for `.data` returns an empty array rather than null.

**UniFi** mirrors the Omada arrangement. The network interface, at `/api/s/{site}/...`, carries per port media, LLDP neighbour tables, uplinks with the remote port, network definitions with subnets, and wireless configuration; it needs a controller login. The integration API, at `/integration/v1/...`, is published and versioned but narrower, covering sites, devices, and clients only; it needs an API key.

Both deployment styles are detected rather than configured. A UniFi OS host signs in at `/api/auth/login` and proxies the network application beneath `/proxy/network`; an older self hosted controller signs in at `/api/login` and serves it from the root. The collector tries both and keeps whichever returns a session cookie.

**Cables.** Access-to-switch and host-to-switch runs are derived from wired client sessions, where the controller supplies the switch and port and the far end is matched by MAC. Access points get a synthesized `Uplink` port so their run has something to anchor to.

Switch-to-switch cabling comes from the controller interface, which reports each device's `uplink` with its local port and the peer port, and reports the same relationship again from the upstream device as a `downlink`. Both directions are emitted and publication collapses the pair, so every inter-switch cable is drawn with both ends known. On the Open API alone this is not derivable at all, because it never says which port faces upstream and `/sites/{id}/lldp` returns only the enable state rather than a neighbour table; those runs are named in the verbose log for manual completion.

**Colors.** VLANs and SSIDs are assigned a color from a shared palette so that the workspace is readable rather than uniformly grey. Allocation walks a queue drawn from `Get-ColorPalette` and recycles it once exhausted, over a list sorted by VLAN identifier rather than discovery order, which is what makes the assignment identical on every run. An SSID bound to a VLAN inherits that VLAN's color so the two read as one network; an unbound SSID takes the next color in the queue.

**Subnets are linked to their VLAN** wherever the source reports a tag. OPNsense supplies the tag per interface, and Omada VLANs that carry a gateway address contribute both the VLAN and its subnet, linked by construction. Subnets with no VLAN, such as VPN tunnels, point to point links, and the WAN block, are left unlinked rather than guessed at.

**SSIDs** come from the real wireless configuration when the controller interface is in use, including SSIDs with no client currently associated and the security mode and VLAN binding that no client session could reveal. On the Open API they fall back to inference from active sessions.

**Wireless client associations** require the client to already exist as a device — Rackpad rejects the write with *"Selected client device does not exist"* otherwise. They are therefore only emitted when `ManageClientEndpoints` is enabled, which creates an `endpoint` device per wireless client parented to its access point. With it disabled, associations are skipped and logged rather than failing.

**Hyper-V** uses the `Hyper-V` module. A local collection talks to the provider directly, because requiring a CIM session for the local machine would mean requiring WinRM, which is often not enabled on a workstation that nonetheless has the Hyper-V role. A remote collection creates a session and falls back from WSMan to DCOM automatically.

The guest operating system is read from the integration services key value pair exchange, which works remotely without a guest credential and is empty for a powered-off guest. `UseGuestIntrospection` additionally attempts PowerShell Direct for a more precise answer, which only works when the script runs on the Hyper-V host itself and a guest credential is supplied. `OperatingSystemType` always stays the virtual machine generation; the guest OS is a separate field.

**OPNsense virtual IPs.** A WAN interface often carries addresses beyond its primary one. Each virtual IP is tested against every interface network; those that fall inside one become an infrastructure assignment, and those that fall outside contribute their own subnet first so the assignment has a parent. Mixed groups on a single interface are handled per address rather than per interface.

**Public IP lookup** is an outbound request to a third party geolocation service. Only the request leaves the environment; no inventory data is transmitted. It is disabled by default.