mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 09:01:31 +00:00
fix: default the OCI helper to the DEFAULT profile (#1174)
* fix: default the OCI helper to the DEFAULT profile oci_env.py shipped our own internal Oracle compartment profile as the default for every reader, with a second internal name as a fallback. A fresh tenancy's config holds only `[DEFAULT]`, so the shipped default was a name nobody else can have, and the fallback chain could land a deployment in whichever tenancy a different profile happened to point at. The profile is now `DEFAULT`, overridable by `OCI_CLI_PROFILE` or `--profile`, and a named profile the config does not hold is an error rather than a fallback. `DEFAULT` needs its own existence test, because configparser treats it as the fallback for every other section and `has_section` denies it exists even when `oci setup config` wrote nothing else. `eip-move-test.sh` and the config table in the README carried the same name. CI reads its credential from the environment rather than from a profile, so nothing there referenced it. * docs: look the OCI compartment OCID up by name The guide asked the reader to paste `ocid1.compartment.oc1..xxx` three times, and the `oci` CLI it already requires can resolve the name instead. The lookup needs its own check. `oci iam compartment list --query ... | [0]` exits 0 and prints nothing when the name matches nothing, so an unset `$CID` would otherwise reach Terraform and fail with an OCI error naming neither the compartment nor the cause. Two compartments in one tenancy can share a name, which the listing command shows and the lookup cannot resolve, so the paste form stays as the answer for that case. Records why the tenancy root is the wrong value: the step 2.1 policy is scoped to whatever `$CID` holds. * Revert "docs: look the OCI compartment OCID up by name" This reverts commit5443149cce. * Reapply "docs: look the OCI compartment OCID up by name" This reverts commit1535d94c8a. * fix: stop validate-topology.sh failing on macOS bash `tf_principal_var` is empty on every run that does not pass `--instance-principal`, and bash before 4.4 treats `"${arr[@]}"` on an empty array as an unbound variable under `set -u`. macOS ships bash 3.2, so the documented deployment command aborted at the tf_vars assembly with `tf_principal_var[@]: unbound variable` before building anything. Assigning the array was not enough, which is why the earlier `HOSTS=()` fix did not cover this: the error is in the expansion, not the reference. Both sites now expand only behind a length test, which is valid on every version. `capture_journals` had the same shape for `HOSTS`, reachable when a run fails before the apply. Reproduced and fixed against bash 3.2.57 in a container, which emits the identical message, and the script now runs to the terraform call on 3.2 for vm-single, vm-single-principal and `--instance-principal`, including the `--keep` cleanup path with no hosts. * docs: record the bash 3.x requirement in the OCI scripts These three are the scripts the OCI guide tells an operator to run, and macOS ships bash 3.2, so a bash 4 construct in any of them breaks a deployment rather than a test. The constraint has now been rediscovered twice, through declare -A and mapfile, and then through an empty array expanded under set -u. Names the three traps at the top of each file so the next change does not have to find them by failing on someone's Mac. * fix: stream the workbook suite instead of hiding it in a file The nine workbooks ran with stdout redirected to `workbooks.log`, so a run wedged inside one printed nothing until the whole suite finished. A CI job then sat silent for hours with no way to tell which workbook it was in, or whether it was working at all. The output is now teed, so the file the per-workbook table is parsed from is unchanged while the run is visible live. Each of the driver's own RUN/PASS/FAIL lines is stamped with elapsed seconds and the workbook name, which is what identifies the slow one without reading the log. `pipefail` carries the ssh exit status through the pipe, so a failed suite still takes the branch that records the table and captures the journals. * fix: grant the OCI policy the subnet verb AssignPrivateIP requires An OCI node with a pool-mode external interface formed, reported healthy, and then refused every launch wanting a public address with a 404 NotAuthorizedOrNotFound from AssignPrivateIP. AssignPrivateIP calls CreatePrivateIp, which Oracle documents as needing PRIVATE_IP_CREATE, PRIVATE_IP_ASSIGN, VNIC_ASSIGN and SUBNET_ATTACH, and DeletePrivateIp as needing SUBNET_DETACH. The policy granted use vnics, manage private-ips and manage public-ips, and no subnet permission at all, so SUBNET_ATTACH was never held. Only the subnets use verb carries it. Measured on a live node as the instance principal: GetVnic and ListPrivateIps succeed, GetSubnet and CreatePrivateIp both 404. Authentication, the dynamic group's matching rule and the compartment were all correct, so the credential was authentic and unauthorised. The documented operator policy for the API-key route carried the same three statements, so neither credential route could allocate. A node now proves its own credential at pool start by making one read in each permission class, and logs the missing verb if one fails. The topology gate waited for "resolved the external VNIC", which comes from the metadata service and needs no IAM policy, so an unauthorised node passed the gate and failed an hour later inside a workbook as capacity. It now waits for the authorisation line and fails immediately on a refusal. Reads rather than a real assign-and-release round trip: a round trip also fails on a full subnet, and reporting that as an authorisation fault sends an operator to IAM for a capacity problem. * docs: say what a principal run costs in a tenancy with no dynamic group No principal topology can pass there, and the only previous hint was one clause about OCI_INSTANCE_PRINCIPAL needing the group to exist. * test: cover the subnet read and the authorisation verdict Diff coverage was 39.1% against a 60% gate. Three of the four gaps were real. GetSubnet on the SDK-backed client had no test, so the request shape was unverified: addressing the collection rather than one subnet by OCID would answer from a different permission and report a credential as working when it is not. Also pins that a sparse body is not an error and that the unauthorised 404 arrives as ErrNotFound, which is what stops the check retrying a verdict that will never change. The fake's GetSubnet was exercised only from the ocinet tests, and coverage is recorded per package under test, so it read as zero-covered. reportAuthorisation now has a test asserting the exact log text, because validate-topology.sh greps for it and a rename would quietly turn the gate into one that never fires. The six lines still uncovered are inside FromPoolConfig, which needs a live JetStream to reach. Diff coverage 87.0%. * fix: refuse a run whose tfvars file contradicts the compartment it was given Terraform ranks *.auto.tfvars above TF_VAR_*, so a leftover file wins silently over an operator targeting a different tenancy by environment, and the guide tells every operator to write one. Hit for real: a stale terraform.auto.tfvars naming a different tenancy's compartment and a bare-metal shape overrode TF_VAR_compartment_ocid, and the plan was for 11 resources in a compartment nobody had chosen. Only a dry run caught it. The run now stops and names both values rather than building. * fix: collect orphaned OCI public IPs and name the limit that was hit A reserved public IP attached to nothing is on no VNIC, so the reconcile that exists to find leaked addresses could not see the ones that matter. It also ran only at startup, while an address is leaked on release and on teardown. Twenty accumulated in one compartment and exhausted the tenancy's limit of 50, after which every launch needing a public address aborted. Reconcile now lists reserved public IPs in the compartment and collects those that carry our prefix, are attached to nothing, are named by no binding and are older than a grace period. All four are needed: an allocated but unassociated Elastic IP is detached too and is the customer's to keep, and a young detached address is what another node's CreatePublicIP looks like until it attaches. A pool with no binding record is not swept at all, since that is indistinguishable from a store that cannot be read. The pass now repeats on a ticker rather than running once at startup. Both OCI ceilings also resolved to InsufficientAddressCapacity, which tells an operator the region is out of addresses when the fix is to release addresses they already hold. AWS separates them, so a tenancy out of reserved public IPs is now AddressLimitExceeded and a full VNIC is PrivateIpAddressLimitExceeded. * feat: run the OCI principal e2e in a tenancy that can hold its dynamic group vm-single-principal needs the tenancy's dynamic group, and creating one is a tenancy-root write that a shared test tenancy does not grant us. The topology could never pass there however correct the code was, so it now takes its own credential from OCI_PRINCIPAL_*, all six or none. A partial set is refused before the build rather than at the apply, because it would build one tenancy's instances against another's compartment. With none set the topology is recorded SKIPPED instead of spending a provision to fail. The sweep takes the same credential, or a leftover principal deployment is destroyed against a tenancy that never held it and reported clean. Two defects found while proving it: --dry-run reported FAIL on every topology with a tree source, because the dry run skips the build and then refuses the tarball it would have produced. The flag operators are told to run first could not pass. The dry run truncated its plan to the last 30 lines, which is the outputs block. That dropped the resource count and every compartment and shape line, so the one check a dry run exists for could not be made from its output. It now writes the whole plan and prints the summary. Removing a deployment is documented as leaving public IPs behind. Spinifex allocates them on the running node, so they are in no Terraform state and destroy cannot see them; twenty accumulated and exhausted a tenancy's limit, which then refused every launch in every compartment. * fix: canonicalise the Opc-Next-Page header in the pagination test CI's linter rejects a non-canonical header name and the local preflight does not, so this passed here and failed there.
This commit is contained in:
Ben Duncan
@@ -146,6 +146,16 @@ jobs:
|
||||
# the allocator gate then fails rather than passing a node that cannot
|
||||
# authenticate.
|
||||
OCI_CREDENTIAL_HOOK: ${{ inputs.auth == 'api_key' && vars.OCI_CREDENTIAL_HOOK || '' }}
|
||||
# vm-single-principal builds in its own tenancy. The dynamic group it needs
|
||||
# is a tenancy-root resource, and the shared test tenancy grants us only
|
||||
# compartment rights, so that topology can never pass there however correct
|
||||
# the code is. Unset skips it rather than spending a provision to fail.
|
||||
OCI_PRINCIPAL_TENANCY_OCID: ${{ secrets.OCI_PRINCIPAL_TENANCY_OCID }}
|
||||
OCI_PRINCIPAL_USER_OCID: ${{ secrets.OCI_PRINCIPAL_USER_OCID }}
|
||||
OCI_PRINCIPAL_FINGERPRINT: ${{ secrets.OCI_PRINCIPAL_FINGERPRINT }}
|
||||
OCI_PRINCIPAL_PRIVATE_KEY: ${{ secrets.OCI_PRINCIPAL_PRIVATE_KEY }}
|
||||
OCI_PRINCIPAL_COMPARTMENT_OCID: ${{ secrets.OCI_PRINCIPAL_COMPARTMENT_OCID }}
|
||||
OCI_PRINCIPAL_REGION: ${{ vars.OCI_PRINCIPAL_REGION }}
|
||||
OCI_ARTIFACT_DIR: ${{ runner.temp }}/e2e-oci
|
||||
OCI_KEEP_ON_FAIL: ${{ inputs.keep_on_fail && '1' || '0' }}
|
||||
# Through the environment, never interpolated into the script below: a
|
||||
@@ -234,6 +244,15 @@ jobs:
|
||||
OCI_PRIVATE_KEY: ${{ secrets.OCI_PRIVATE_KEY }}
|
||||
OCI_REGION: ${{ vars.OCI_REGION || 'ap-sydney-1' }}
|
||||
TF_VAR_compartment_ocid: ${{ secrets.OCI_COMPARTMENT_OCID }}
|
||||
# The sweep destroys by state file, so it needs whichever tenancy each
|
||||
# topology built in. Without these a leftover principal deployment is
|
||||
# swept against a tenancy that never held it and reported clean.
|
||||
OCI_PRINCIPAL_TENANCY_OCID: ${{ secrets.OCI_PRINCIPAL_TENANCY_OCID }}
|
||||
OCI_PRINCIPAL_USER_OCID: ${{ secrets.OCI_PRINCIPAL_USER_OCID }}
|
||||
OCI_PRINCIPAL_FINGERPRINT: ${{ secrets.OCI_PRINCIPAL_FINGERPRINT }}
|
||||
OCI_PRINCIPAL_PRIVATE_KEY: ${{ secrets.OCI_PRINCIPAL_PRIVATE_KEY }}
|
||||
OCI_PRINCIPAL_COMPARTMENT_OCID: ${{ secrets.OCI_PRINCIPAL_COMPARTMENT_OCID }}
|
||||
OCI_PRINCIPAL_REGION: ${{ vars.OCI_PRINCIPAL_REGION }}
|
||||
OCI_ARTIFACT_DIR: ${{ runner.temp }}/e2e-oci-sweep
|
||||
run: ./e2e-oci.sh --sweep-only
|
||||
|
||||
|
||||
@@ -71,6 +71,12 @@ ubuntu-26.04-x86_64 | ubuntu | 26.04 | x86_64 | uefi
|
||||
spx admin images import --name ubuntu-26.04-x86_64
|
||||
```
|
||||
|
||||
Verify the import and note the AMI ID:
|
||||
|
||||
```bash
|
||||
AMI_ID=$(aws ec2 describe-images --query 'Images[0].ImageId' --output text)
|
||||
```
|
||||
|
||||
### Option B: Import a local image file
|
||||
|
||||
```bash
|
||||
@@ -95,12 +101,12 @@ for img in spinifex-rds-postgres spinifex-rds-mariadb spinifex-ecs-node spinifex
|
||||
done
|
||||
```
|
||||
|
||||
| Image | Serves | Contents |
|
||||
| --- | --- | --- |
|
||||
| `spinifex-rds-postgres` | RDS, `--engine postgres` | Alpine 3.24.1 + PostgreSQL 18 + `rds-init` |
|
||||
| `spinifex-rds-mariadb` | RDS, `--engine mariadb` | Alpine 3.24.1 + MariaDB 11.8 + `rds-init` |
|
||||
| `spinifex-ecs-node` | ECS container instances | Alpine 3.21.7 + containerd + ecs-agent |
|
||||
| `spinifex-eks-node` | EKS control plane and nodegroups | Alpine 3.21.7 + K3s v1.32.5 + eks-token-webhook |
|
||||
| Image | Serves | Contents |
|
||||
| ----------------------- | -------------------------------- | ----------------------------------------------- |
|
||||
| `spinifex-rds-postgres` | RDS, `--engine postgres` | Alpine 3.24.1 + PostgreSQL 18 + `rds-init` |
|
||||
| `spinifex-rds-mariadb` | RDS, `--engine mariadb` | Alpine 3.24.1 + MariaDB 11.8 + `rds-init` |
|
||||
| `spinifex-ecs-node` | ECS container instances | Alpine 3.21.7 + containerd + ecs-agent |
|
||||
| `spinifex-eks-node` | EKS control plane and nodegroups | Alpine 3.21.7 + K3s v1.32.5 + eks-token-webhook |
|
||||
|
||||
All four boot **BIOS**, unlike the distro images, which boot UEFI. That is a property of how they are built and nothing you need to pass.
|
||||
|
||||
@@ -168,7 +174,7 @@ aws ec2 attach-internet-gateway \
|
||||
|
||||
### Create a Subnet
|
||||
|
||||
Create the subnet your instances will launch into. The routing and public-IP steps below are what make it a *public* subnet.
|
||||
Create the subnet your instances will launch into. The routing and public-IP steps below are what make it a _public_ subnet.
|
||||
|
||||
```bash
|
||||
SUBNET_ID=$(aws ec2 create-subnet \
|
||||
@@ -349,17 +355,17 @@ https://YOUR_NODE_IP:3000
|
||||
|
||||
On first load, your browser will show a TLS warning — Spinifex generates a self-signed certificate at install time. This is expected.
|
||||
|
||||
1. Accept the warning to reach the login page (exact wording varies by browser — e.g. Chrome: *Advanced → Proceed to ...*, Firefox: *Advanced → Accept the Risk and Continue*).
|
||||
1. Accept the warning to reach the login page (exact wording varies by browser — e.g. Chrome: _Advanced → Proceed to ..._, Firefox: _Advanced → Accept the Risk and Continue_).
|
||||
2. On the login page, click **Download Certificate** and save `spinifex-ca.pem` to your machine.
|
||||
3. Install the certificate as a **trusted root** on your workstation, following the steps for your platform below.
|
||||
4. Restart your browser and reload `https://YOUR_NODE_IP:3000`. The padlock should now show a valid certificate.
|
||||
|
||||
Step 3, by platform:
|
||||
|
||||
- **macOS:** open `spinifex-ca.pem` in Keychain Access → *System* keychain → set *Trust* to **Always Trust**.
|
||||
- **macOS:** open `spinifex-ca.pem` in Keychain Access → _System_ keychain → set _Trust_ to **Always Trust**.
|
||||
- **Linux:** `sudo cp spinifex-ca.pem /usr/local/share/ca-certificates/spinifex-ca.crt && sudo update-ca-certificates`
|
||||
- **Windows:** double-click the file → *Install Certificate* → *Local Machine* → *Trusted Root Certification Authorities*.
|
||||
- **Browser-only (Firefox):** *Settings → Privacy & Security → Certificates → View Certificates → Authorities → Import* and tick *Trust this CA to identify websites*.
|
||||
- **Windows:** double-click the file → _Install Certificate_ → _Local Machine_ → _Trusted Root Certification Authorities_.
|
||||
- **Browser-only (Firefox):** _Settings → Privacy & Security → Certificates → View Certificates → Authorities → Import_ and tick _Trust this CA to identify websites_.
|
||||
|
||||
> **Why this is required:** the UI logs in by reading your AWS credentials through a trusted TLS channel. Browsers refuse to send credentials over an untrusted connection, so the certificate must be installed as trusted — temporary "Proceed anyway" exceptions won't work for login.
|
||||
|
||||
|
||||
+137
-36
@@ -74,12 +74,12 @@ What sits underneath is in [Architecture and Operations](../oci-architecture/REA
|
||||
|
||||
## Prerequisites
|
||||
|
||||
| | |
|
||||
| ------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **An OCI compartment OCID** | This deploys into a compartment that already exists. Creating one needs tenancy-root rights the API user must not have |
|
||||
| **OCI credentials in `~/.oci/config`** | For Terraform, to build the infrastructure. `oci setup config` writes one, or write it by hand |
|
||||
| | |
|
||||
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| **An OCI compartment OCID** | This deploys into a compartment that already exists. Creating one needs tenancy-root rights the API user must not have |
|
||||
| **OCI credentials in `~/.oci/config`** | For Terraform, to build the infrastructure. `oci setup config` writes one, or write it by hand |
|
||||
| **A runtime credential for the nodes** | Separate from the above, and the subject of step 2: either an instance principal or a scoped API key. Without one a node comes up healthy and still cannot give any guest a public address |
|
||||
| **Terraform or OpenTofu, `git`, Python 3** | On your workstation. The Python helper is standard library only, so there is nothing to `pip install` |
|
||||
| **Terraform or OpenTofu, `git`, Python 3** | On your workstation. The Python helper is standard library only, so there is nothing to `pip install` |
|
||||
|
||||
Check two quotas before you start, because both refuse at apply time rather than at plan time. **Reserved public IPs** are capped at 50 per region across the whole tenancy, and the **compute limit for your shape** starts at zero for bare metal on many new tenancies. [Quotas](../oci-architecture/README.md#quotas) has the commands.
|
||||
|
||||
@@ -93,14 +93,46 @@ Once installed configure your OCI credentials (~/.oci/config) which will be used
|
||||
|
||||
`oci setup config`
|
||||
|
||||
Once setup validate your OCI credentials work as expected. `$CID` is the compartment you are deploying into, which is a `ocid1.compartment.oc1..` OCID, or your tenancy OCID if that compartment is the root. The same value is used throughout this guide and again as `compartment_ocid` in step 4.
|
||||
That writes a `[DEFAULT]` profile, which every command below reads with no flag. **If your config holds named profiles instead, name the one to use once:**
|
||||
|
||||
```sh
|
||||
export OCI_CLI_PROFILE=my-profile
|
||||
```
|
||||
|
||||
Spinifex's Terraform helper reads the same variable, so this one export covers the CLI commands here and the deployment in step 5.
|
||||
|
||||
### OCI Compartment
|
||||
|
||||
Now set `$CID` to the compartment you are deploying into. It is used throughout this guide and again as `compartment_ocid` in step 4, so set it once and keep the same shell.
|
||||
|
||||
Which OCID to use depends on how your tenancy is laid out.
|
||||
|
||||
**A tenancy with compartments in it.** List compartments in OCI and pick one where to deploy Spinifex. The OCID is in the `id` column:
|
||||
|
||||
```sh
|
||||
oci iam compartment list --compartment-id-in-subtree true --access-level ACCESSIBLE --all \
|
||||
--query 'data[?"lifecycle-state"==`ACTIVE`].{name:name,id:id}' --output table
|
||||
```
|
||||
|
||||
**A tenancy with nothing but the root.** The command above will return nothing, which is correct rather than broken: it lists the children of your tenancy and never the tenancy itself. To deploy into the root by using the tenancy OCID, which is the `tenancy=` line of your `~/.oci/config`, or ask the CLI for it:
|
||||
|
||||
```sh
|
||||
oci iam availability-domain list --query 'data[0]."compartment-id"' --raw-output
|
||||
```
|
||||
|
||||
Next, specify your tenancy as `CID`:
|
||||
|
||||
```sh
|
||||
export CID="ocid1.compartment.oc1..xxx"
|
||||
```
|
||||
|
||||
Confirm the credential and the compartment together:
|
||||
|
||||
```sh
|
||||
oci compute shape list --compartment-id $CID
|
||||
```
|
||||
|
||||
Expected output to confirm API key works as expected:
|
||||
Expected output to confirm API key and compartment works as expected:
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -121,6 +153,8 @@ The next prerequisite is to install Terraform or OpenTofu to deploy the Spinifex
|
||||
- Terraform - [https://developer.hashicorp.com/terraform/install](https://developer.hashicorp.com/terraform/install)
|
||||
- OpenTofu - [https://opentofu.org/docs/intro/install/](https://opentofu.org/docs/intro/install/)
|
||||
|
||||
The deployment scripts of Spinifex will automatically detect to use Terraform or OpenTofu depending what is setup in your environment.
|
||||
|
||||
## Instructions
|
||||
|
||||
## 1. Clone the Spinifex repository
|
||||
@@ -151,20 +185,26 @@ export OCI_CLI_PROFILE=customprofile
|
||||
|
||||
## 2. OCI Credentials for Spinifex
|
||||
|
||||
Spinifex calls the OCI API at runtime to register each guest's public address, so every node needs a credential that can manage private and public IPs in your compartment. Without one a node forms and passes every health check, then refuses every launch that asks for a public address.
|
||||
Spinifex calls the OCI API at runtime to register each guest's public address, so every node needs a credential that can manage private and public IPs in your compartment.
|
||||
|
||||
There are two ways to provide it, and they differ in who has to authorise them rather than in what Spinifex is allowed to do. Both grant [the same three permissions](../oci-architecture/README.md#the-iam-policy).
|
||||
There are two ways to provide OCI API access to Spinifex, and they differ in who has to authorise them rather than in what Spinifex is allowed to do. Both grant [the same four permissions](../oci-architecture/README.md#the-iam-policy).
|
||||
|
||||
| | [2.1 Instance principal](#21-instance-principal-preferred) | [2.2 OCI IAM user](#22-oci-iam-user-with-scoped-permissions) |
|
||||
| ------------------- | ----------------------------------------------------------- | ------------------------------------------------------------ |
|
||||
| **Key material** | None. Each node authenticates as itself | A private key on every node, at `/etc/spinifex/oci/` |
|
||||
| **Who can set up** | A tenancy admin, once per tenancy | Anyone with identity rights in the tenancy |
|
||||
| **Rotation** | Nothing to rotate | Yours to rotate and redistribute |
|
||||
| **What you set** | `export SPX_PRINCIPAL=adopt`, read by step 4 | Nothing extra; step 5 installs the key for you |
|
||||
| | [2.1 Instance principal](#21-instance-principal-preferred) | [2.2 OCI IAM user](#22-oci-iam-user-with-scoped-permissions) |
|
||||
| ------------------ | ---------------------------------------------------------- | ------------------------------------------------------------ |
|
||||
| **Key material** | None. Each node authenticates as itself | A private key on every node, at `/etc/spinifex/oci/` |
|
||||
| **Who can set up** | A tenancy admin, once per tenancy | Anyone with identity rights in the tenancy |
|
||||
| **Rotation** | Nothing to rotate | Yours to rotate and redistribute |
|
||||
| **What you set** | `export SPX_PRINCIPAL=adopt`, read by step 4 | Nothing extra; step 5 installs the key for you |
|
||||
|
||||
**Use 2.1 if you are a tenancy admin.** It is the better of the two because there is no key to leak, rotate or forget about, and a replaced node needs no handoff. Use 2.2 when your compartment was allocated to you inside someone else's tenancy, which is the common case and the reason 2.2 exists at all.
|
||||
|
||||
Do one of the two, not both, then continue to step 3.
|
||||
Both routes below need `$CID` from the prerequisites step, so confirm it is still set in this shell:
|
||||
|
||||
```bash
|
||||
echo "${CID:?set it as shown under OCI CLI tool}"
|
||||
```
|
||||
|
||||
Do one of the two authentication methods below, not both, then continue to step 3.
|
||||
|
||||
### 2.1 Instance principal (preferred)
|
||||
|
||||
@@ -191,6 +231,8 @@ To remove the group and policy again, `./setup-identity.sh --destroy`. Every nod
|
||||
|
||||
This route creates a dedicated OCI user holding only the three permissions Spinifex needs, and installs its key on each node. It needs identity rights in the tenancy but no tenancy-root rights, so it works in an allocated compartment.
|
||||
|
||||
Skip this section if you have enabled the instance principal authentication path.
|
||||
|
||||
#### Generate the key
|
||||
|
||||
```bash
|
||||
@@ -244,19 +286,18 @@ An email will be sent to the designated `$ADMIN_EMAIL` defined, you are required
|
||||
> [!NOTE]
|
||||
> **If these fail with `NotAuthorizedOrNotFound`, you do not have identity rights in the tenancy**, which is normal when someone allocated you a compartment inside theirs. Ask whoever owns the tenancy to grant you the required permissions.
|
||||
|
||||
#### Grant the group its three permissions
|
||||
#### Grant the group its four permissions
|
||||
|
||||
Spinifex calls [ten operations](../oci-architecture/README.md#the-iam-policy), and these three statements are what authorise them. Attach the policy to the compartment you are deploying into using the command below. Replace `$CID` with that compartment's OCID.
|
||||
Spinifex calls [ten operations](../oci-architecture/README.md#the-iam-policy), and these four statements are what authorise them. This attaches the policy to `$CID`, the compartment you are deploying into.
|
||||
|
||||
```bash
|
||||
export CID="ocid1.compartment.oc1..xxx"
|
||||
|
||||
oci iam policy create --compartment-id "$CID" --name SpinifexOperators \
|
||||
--description "Spinifex external-address allocation" \
|
||||
--statements '[
|
||||
"Allow group SpinifexOperators to use vnics in compartment id '"$CID"'",
|
||||
"Allow group SpinifexOperators to manage private-ips in compartment id '"$CID"'",
|
||||
"Allow group SpinifexOperators to manage public-ips in compartment id '"$CID"'"
|
||||
"Allow group SpinifexOperators to manage public-ips in compartment id '"$CID"'",
|
||||
"Allow group SpinifexOperators to use subnets in compartment id '"$CID"'"
|
||||
]'
|
||||
```
|
||||
|
||||
@@ -350,14 +391,14 @@ Choose how to deploy Spinifex on OCI, either as a single node, or at minimum a t
|
||||
|
||||
## 4. Write your Terraform inputs
|
||||
|
||||
One file, in the directory you are already in. Only `compartment_ocid` is required, because every other variable has a default that builds a working single node:
|
||||
The next step is to define a `terraform.auto.tfvars` configuration file with your desired deployment method for Spinifex on OCI.
|
||||
|
||||
Recommended compute shapes for a VM deployment:
|
||||
|
||||
- `VM.Standard.E6.Flex` - Higher performance, 5th Gen AMD EPYC processor
|
||||
- `VM.Standard.E5.Flex` - Base-line performance, 4th Gen AMD EPYC processor
|
||||
|
||||
Recommended bare metal shapes, preferred for large-scale production:
|
||||
Recommended bare metal shapes, preferred for large-scale production workloads:
|
||||
|
||||
- `BM.Standard.E5.192` - 192 OCPU / 2304 GB RAM. Higher performance, 4th Gen AMD EPYC processor
|
||||
- `BM.Standard3.64` - 64 OCPU / 1024 GB RAM. Baseline, Intel Xeon Platinum 8358
|
||||
@@ -406,9 +447,19 @@ grep compartment_ocid terraform.auto.tfvars
|
||||
|
||||
## 5. Deploy
|
||||
|
||||
Choose the deployment method below:
|
||||
|
||||
- `vm-single` - Single node instance (best for testing/development)
|
||||
- `vm-multi` - A recommended 3-node cluster (production)
|
||||
- `bm` - Bare metal for larger scale production use.
|
||||
|
||||
```bash
|
||||
export TOPOLOGY=vm-single
|
||||
```
|
||||
|
||||
Next deploy Spinifex on OCI:
|
||||
|
||||
```bash
|
||||
./validate-topology.sh \
|
||||
--topology "$TOPOLOGY" \
|
||||
--credential-hook ./spx-oci-config.sh \
|
||||
@@ -420,20 +471,20 @@ The same command covers both credential routes. It reads `instance_principal` fr
|
||||
|
||||
One command builds the infrastructure, installs Spinifex, forms the cluster, configures OCI public addressing and verifies that the address allocator came up.
|
||||
|
||||
| Argument | What to pass |
|
||||
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `--topology` | `vm-single` for one VM, `vm-multi` for three, `bm` for one bare-metal host. Required, with no default, so a command cannot be aimed at the wrong one by omission. It supplies a shape and a node count only where your tfvars is silent |
|
||||
| `--channel` | Nothing, normally. It defaults to `latest`, the published release. Pass `dev` to install the newest pre-release instead, or `--version <tag>` to pin an exact build |
|
||||
| `--credential-hook` | `./spx-oci-config.sh`, which installs the step 2.2 credential on every node. Not used when your tfvars set `instance_principal`, because there is then no key to install |
|
||||
| `--skip-workload` | Stop once the cluster is verified. Drop it and the driver also runs the published Terraform workbooks on the node, which launches real guests on public addresses and tears each one down again |
|
||||
| `--keep` | **This is what makes it a deployment rather than a test.** Without it the driver destroys everything at the end, which is right for CI and wrong here |
|
||||
| Argument | What to pass |
|
||||
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `--topology` | `vm-single` for one VM, `vm-multi` for three, `bm` for one bare-metal host. Required, with no default, so a command cannot be aimed at the wrong one by omission. It supplies a shape and a node count only where your tfvars is silent |
|
||||
| `--channel` | Nothing, normally. It defaults to `latest`, the published release. Pass `dev` to install the newest pre-release instead, or `--version <tag>` to pin an exact build |
|
||||
| `--credential-hook` | `./spx-oci-config.sh`, which installs the step 2.2 credential on every node. Not used when your tfvars set `instance_principal`, because there is then no key to install |
|
||||
| `--skip-workload` | Stop once the cluster is verified. Drop it and the driver also runs the published Terraform workbooks on the node, which launches real guests on public addresses and tears each one down again |
|
||||
| `--keep` | **This is what makes it a deployment rather than a test.** Without it the driver destroys everything at the end, which is right for CI and testing a deployment |
|
||||
|
||||
**Your `terraform.auto.tfvars` outranks the topology**, so `--topology bm` with `compute_shape = "BM.Standard.E5.192"` in that file deploys on that shape rather than the one CI runs. The run logs the shape and node count it actually built, straight after the host addresses.
|
||||
**Your `terraform.auto.tfvars` outranks the topology**, so `--topology bm` with `compute_shape = "BM.Standard.E5.192"` in that file deploys on that shape.
|
||||
|
||||
Add `--dry-run` to print the plan and change nothing.
|
||||
|
||||
> [!NOTE]
|
||||
> **If `--channel dev` fails to download, pass a tag instead.** `dev` is the one channel that resolves through an unauthenticated GitHub API call, so it can be rate-limited and then 404s. Take the newest tag from the [releases page](https://github.com/mulgadc/spinifex/releases) and swap `--channel dev` for `--version <tag>`, which resolves by redirect and cannot trip the same limit. A tag is also what to use for a repeatable deployment, where you want the build pinned rather than current.
|
||||
> **If `--channel dev` fails to download, pass a tag instead.** `dev` is the one channel that resolves through an unauthenticated GitHub API call, so it can be rate-limited by Github. Take the newest tag from the [releases page](https://github.com/mulgadc/spinifex/releases) and swap `--channel dev` for `--version <tag>`, which resolves by redirect and cannot trip the same limit. A tag is also what to use for a repeatable deployment, where you want the build pinned rather than current.
|
||||
|
||||
## 6. Check it worked
|
||||
|
||||
@@ -453,7 +504,7 @@ The stages print as they finish, and all of these have to appear:
|
||||
[validate-vm-single] cluster membership
|
||||
[validate-vm-single] --skip-workload: stopping before the workbook
|
||||
[validate-vm-single] --keep: leaving the infrastructure up, no verdict recorded
|
||||
[validate-vm-single] ssh in with: ssh -i ~/.ssh/oci-spx ubuntu@150.230.13.131
|
||||
[validate-vm-single] ssh in with: ssh -i ~/.ssh/oci-spx ubuntu@public.ip
|
||||
[validate-vm-single] destroy it with: ./validate-topology.sh --topology vm-single --destroy-only
|
||||
```
|
||||
|
||||
@@ -494,6 +545,25 @@ aws ec2 describe-instance-types
|
||||
|
||||
If this returns a list of available instance types, your installation is working.
|
||||
|
||||
### Console access
|
||||
|
||||
Next login to the Spinifex console at:
|
||||
|
||||
```
|
||||
https://YOURIP:3000/
|
||||
```
|
||||
|
||||
Fetch your generated AWS credentials via SSH and login to the console using your new Spinifex account.
|
||||
|
||||
```bash
|
||||
ubuntu@spinifexnode01:~$ cat ~/.aws/credentials
|
||||
[spinifex]
|
||||
aws_access_key_id = XXXX
|
||||
aws_secret_access_key = XXXX
|
||||
```
|
||||
|
||||
|
||||
|
||||
**Congratulations! Spinifex is installed.**
|
||||
|
||||
## 7. Set up your cluster
|
||||
@@ -502,9 +572,6 @@ Spinifex is running, but it holds nothing yet — no machine images, no networks
|
||||
|
||||
Continue to [Setting Up Your Cluster](/docs/setting-up-your-cluster) to import an AMI, create an SSH key pair, create a VPC with a public subnet, and launch your first instance.
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **Probe a guest's public address from your workstation, not from the node.** A node is a poor vantage point for its own network's public addresses, so a timeout measured there says nothing about the guest. Private addresses are a separate question, and those do work from inside.
|
||||
|
||||
## Harden it before production
|
||||
|
||||
The deployment comes up usable, not locked down. Three layers filter traffic here and the first job is telling them apart, which [Firewalls, in all three layers](../oci-architecture/README.md#firewalls-in-all-three-layers) does in full. The short version:
|
||||
@@ -579,6 +646,40 @@ The four are `oracle-10.1-x86_64`, `oracle-10.1-arm64`, `oracle-9.8-x86_64` and
|
||||
|
||||
It destroys whatever that topology's Terraform state holds, so it can only remove what this configuration created.
|
||||
|
||||
### Check for leftover public IPs afterwards
|
||||
|
||||
**Terraform does not own the addresses Spinifex allocates, so a destroy can leave them behind.**
|
||||
A reserved public IP and the private IP under it are created by the running node when a guest asks for an address, not by this configuration.
|
||||
They are in no Terraform state, so `--destroy-only` cannot see them and does not try.
|
||||
|
||||
Each one bills, and each holds one of the tenancy's 50 reserved-public-IP slots.
|
||||
A tenancy at that ceiling refuses every later launch with `AddressLimitExceeded`, on every deployment in it and not only the one that leaked.
|
||||
|
||||
A running node collects its own: the allocator reconciles every ten minutes and deletes detached addresses carrying the `spinifex-` prefix that no binding claims.
|
||||
Destroying the node is what defeats that, because the sweep goes away with the host that ran it.
|
||||
So list them after a teardown:
|
||||
|
||||
```bash
|
||||
oci network public-ip list --compartment-id "$COMPARTMENT_OCID" --scope REGION --lifetime RESERVED --all \
|
||||
--query 'data[?starts_with("display-name", `spinifex-`)].{ip:"ip-address",name:"display-name",attached:"private-ip-id",created:"time-created"}' \
|
||||
--output table
|
||||
```
|
||||
|
||||
**A row with a non-empty `attached` is not garbage.**
|
||||
It is bound to a VNIC, which means either a guest is receiving traffic on it or another deployment in the same compartment holds it.
|
||||
Only rows with an empty `attached` are candidates, and then one at a time:
|
||||
|
||||
```bash
|
||||
oci network public-ip delete --public-ip-id <ocid> --force
|
||||
```
|
||||
|
||||
Use the limits API rather than a count to judge the headroom, because the two disagree and only one of them is what the service enforces:
|
||||
|
||||
```bash
|
||||
oci limits resource-availability get --service-name vcn \
|
||||
--limit-name reserved-public-ip-count --compartment-id "$TENANCY_OCID"
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
The deploy stops at the first failure and names the log it wrote, all under `.validate-<topology>/`. Add `--keep-on-fail` to leave a failed deployment up so you can log in and look. **It keeps billing** until you run `--destroy-only`.
|
||||
|
||||
@@ -174,12 +174,15 @@ The matching policy, scoped to the one compartment Spinifex allocates in:
|
||||
Allow group SpinifexOperators to use vnics in compartment <compartment-name>
|
||||
Allow group SpinifexOperators to manage private-ips in compartment <compartment-name>
|
||||
Allow group SpinifexOperators to manage public-ips in compartment <compartment-name>
|
||||
Allow group SpinifexOperators to use subnets in compartment <compartment-name>
|
||||
```
|
||||
|
||||
`instance-principal.tf` grants the dynamic group these same three statements, so an instance principal can allocate a public address exactly as an IAM user can. The two credential routes differ in who is authorised, never in what.
|
||||
`instance-principal.tf` grants the dynamic group these same four statements, so an instance principal can allocate a public address exactly as an IAM user can. The two credential routes differ in who is authorised, never in what.
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **`use vnics` is required, and `read vnics` is not enough.** Registering a secondary private IP is an operation *on a VNIC*, so OCI checks `VNIC_ASSIGN` as well as `PRIVATE_IP_CREATE`, and `VNIC_ASSIGN` lives in the `use` verb. Grant only `read` and every allocation fails with `NotAuthorizedOrNotFound` on a node that looks entirely healthy, which reads as a wrong compartment OCID rather than a missing verb.
|
||||
> **Allocating one address needs permissions on three resource types, and a 404 names none of them.** `CreatePrivateIp` is checked against `PRIVATE_IP_CREATE`, `PRIVATE_IP_ASSIGN`, `VNIC_ASSIGN` and `SUBNET_ATTACH`, and `DeletePrivateIp` against `SUBNET_DETACH`. `VNIC_ASSIGN` lives in the vnics `use` verb, so `read vnics` is not enough; `SUBNET_ATTACH` and `SUBNET_DETACH` live in the subnets `use` verb, so a policy naming only vnics and IPs authorises nothing. Either omission fails every allocation with `NotAuthorizedOrNotFound` on a node that looks entirely healthy, and OCI uses that one error for "you may not" and "it is not there" alike, so it reads as a wrong compartment OCID rather than a missing verb.
|
||||
|
||||
To tell the two apart, make one call per permission class and see which is the first to fail. `GetVnic` tests `VNIC_READ`, `ListPrivateIps` tests `PRIVATE_IP_READ`, and `GetSubnet` tests `SUBNET_READ`. If the first two succeed and the third does not, the credential is authentic and in the right compartment, and the policy is missing its subnet statement.
|
||||
|
||||
**Scope it to a compartment, not the tenancy.** `manage public-ips` at tenancy level lets a compromised node consume the whole regional reserved-public-IP quota.
|
||||
|
||||
@@ -430,10 +433,20 @@ sudo journalctl -u spinifex-vpcd --since -5m | grep -i ocinet
|
||||
```text
|
||||
"msg":"ocinet resolved the external VNIC from its interface","pool":"oci-public","iface":"br-wan",
|
||||
"vnic_id":"ocid1.vnic.oc1.ap-sydney-1.abzxsljrjwlf...kimuq","private_ip":"10.200.1.31","subnet":"10.200.0.0/23"
|
||||
"msg":"ocinet credential authorised to allocate","pool":"oci-public",
|
||||
"vnic_id":"ocid1.vnic.oc1.ap-sydney-1.abzxsljrjwlf...kimuq","compartment":"ocid1.compartment.oc1..aaaa"
|
||||
"msg":"OCI allocator ready","pool":"oci-public","collected":0,"stale_bindings":0,"skipped":1
|
||||
```
|
||||
|
||||
**`resolved the external VNIC` is the line that matters.** It proves the credentials work, the compartment is right, and `br-wan`'s MAC matched a real VNIC. A node missing it will accept `allocate-address` and then fail it.
|
||||
**`credential authorised to allocate` is the line that matters, and `resolved the external VNIC` is not.** The authorisation line is logged after the node has made one real read in each permission class an allocation needs, so it proves the credential is accepted, the compartment is right and every verb in the policy is held. The resolution line above it comes from the instance metadata service and needs no IAM policy at all, so a node with a policy missing a verb logs it and still refuses every allocation.
|
||||
|
||||
The refusal has its own line, which names the verb:
|
||||
|
||||
```text
|
||||
"msg":"ocinet credential cannot allocate an external address","pool":"oci-public",
|
||||
"error":"reading subnet ocid1.subnet... failed, which needs `use subnets` in compartment ocid1...;
|
||||
AssignPrivateIP requires SUBNET_ATTACH from that same verb: ... NotAuthorizedOrNotFound"
|
||||
```
|
||||
|
||||
On a cold cluster start you may instead see `OCI allocator reconcile failed … nats: no responders available for request` on some nodes. That is vpcd racing JetStream's KV at boot, where the startup reconcile is skipped and not retried. It is harmless on a new cluster where nothing has been allocated, and it is tracked; restart `spinifex-vpcd` on that node to run it.
|
||||
|
||||
|
||||
@@ -57,7 +57,7 @@ python3 scripts/oci_env.py --ssh-public-key-path <path_to_public_ssh_key> -- ter
|
||||
| `terraform.tfvars.example` | Example non-secret Terraform inputs. Copy it to untracked `terraform.auto.tfvars` for local overrides. |
|
||||
| `terraform.tfstate` | Terraform state; generated locally unless a remote backend is configured. Do not commit it. |
|
||||
|
||||
The helper defaults to OCI profile `apacanzset03child03`. If it is absent and `apacanzset03child3` is locally configured, the helper explicitly falls back to that profile. Override either setting when needed:
|
||||
The helper reads the `DEFAULT` profile, which is what `oci setup config` writes. Set `OCI_CLI_PROFILE` or pass `--profile` to read a different one; a named profile the config does not hold is an error rather than a fallback, so a deployment cannot land in a tenancy nobody chose.
|
||||
|
||||
```bash
|
||||
python3 scripts/oci_env.py --profile my-profile --region ap-sydney-1 -- terraform plan
|
||||
@@ -228,9 +228,11 @@ Create them once per tenancy, then every deployment and rebuild afterwards uses
|
||||
|
||||
`setup-identity.sh` targets only those two resources and keeps them in `.identity/terraform.tfstate`, separate from every topology's state. That separation is load-bearing: `validate-topology.sh` destroys its own state at the end of each run, so holding tenancy resources there would let a nightly teardown delete the tenancy's policy.
|
||||
|
||||
**`adopt` references the dynamic group by nothing at all.** Its matching rule is `instance.compartment.id`, so it covers every instance in the compartment and names no OCID — which is why adopting needs no read on an identity resource and no tenancy rights. The cost is that a missing policy is invisible at apply time: the node forms, passes every health check, and then refuses every launch wanting a public address. The allocator gate in `validate-topology.sh` is what catches that, by requiring `resolved the external VNIC` in each node's `spinifex-vpcd` journal.
|
||||
**`adopt` references the dynamic group by nothing at all.** Its matching rule is `instance.compartment.id`, so it covers every instance in the compartment and names no OCID — which is why adopting needs no read on an identity resource and no tenancy rights. The cost is that a missing policy is invisible at apply time: the node forms, passes every health check, and then refuses every launch wanting a public address. The allocator gate in `validate-topology.sh` is what catches that, by requiring `ocinet credential authorised to allocate` in each node's `spinifex-daemon` journal.
|
||||
|
||||
The policy grants three verbs in one compartment — `use vnics`, `manage private-ips`, `manage public-ips` — which is exactly what allocating an external address does and nothing more.
|
||||
The policy grants four verbs in one compartment: `use vnics`, `manage private-ips`, `manage public-ips` and `use subnets`.
|
||||
That is exactly what allocating an external address does and nothing more.
|
||||
`use subnets` looks unrelated and is not: `CreatePrivateIp` is checked against `SUBNET_ATTACH` and `CreatePrivateIp` is how an address is registered, so a policy without it authorises nothing and every allocation returns a 404.
|
||||
|
||||
**An API key is the fallback**, and the default because it needs nothing from a tenancy admin. Grant that user only the operations Spinifex performs on addresses and nothing else; it is not a tenancy admin, and broader rights widen the blast radius of a node compromise for no benefit.
|
||||
|
||||
@@ -271,6 +273,8 @@ The count that later gates read is the number of addresses the `hosts_file` outp
|
||||
|
||||
**The teardown decides the verdict.** A topology or workbook that cannot be destroyed is half proved, and has been a real defect before, so `destroy` runs from an `EXIT` trap even on failure and a teardown failure fails the run. `--keep` leaves everything up and records no verdict. Other flags: `--skip-workload` (form and verify, launch no guests), `--workbook NAME`, `--ssh-public-key` / `--ssh-private-key`.
|
||||
|
||||
**A clean teardown can still leave public IPs behind, and they count against the tenancy.** The addresses Spinifex allocates for guests are created by the running node, so they are in no Terraform state and `destroy` neither sees nor removes them. The node's own reconcile collects detached ones every ten minutes, which is exactly the sweep a destroy takes away. Twenty accumulated in `spxbm` over four runs and exhausted the tenancy's 50 reserved-public-IP limit, at which point every later launch in every compartment failed. [Check for leftover public IPs afterwards](../../../docs/oci-integration/README.md#check-for-leftover-public-ips-afterwards) has the query and the rule for reading it.
|
||||
|
||||
The workbook runs **on the node** against `127.0.0.1`, because the node certificate carries no SAN for its public address — a workbook driven from outside the VCN is still blocked.
|
||||
|
||||
## Common Commands
|
||||
|
||||
@@ -26,6 +26,14 @@
|
||||
# principal. Default 1, and it needs the tenancy's dynamic
|
||||
# group to exist. The vm-single-principal topology ignores it
|
||||
# and always authenticates that way, which is its purpose.
|
||||
#
|
||||
# A tenancy where setup-identity.sh cannot run has no such
|
||||
# group, and then no principal run can pass: the node forms,
|
||||
# authenticates, and is refused every allocation. That now
|
||||
# fails at the allocator gate within a minute of the daemon
|
||||
# starting rather than an hour later inside a workbook, but
|
||||
# it still costs a full provision, so drop the topology from
|
||||
# OCI_TOPOLOGIES until the group exists.
|
||||
# OCI_NO_EXTERNAL_POOL 1 to form with no allocator at all, for a tenancy whose
|
||||
# dynamic group does not exist yet. The allocator and every
|
||||
# workbook are then recorded SKIPPED, never PASS, so a green
|
||||
@@ -37,6 +45,23 @@
|
||||
# before the pool, as "hook <ssh-key> <host>...". An API-key
|
||||
# deployment installs its credential here; instance principal
|
||||
# needs none, so it is skipped unless the pool is key-based.
|
||||
# OCI_PRINCIPAL_TENANCY_OCID, OCI_PRINCIPAL_USER_OCID,
|
||||
# OCI_PRINCIPAL_FINGERPRINT, OCI_PRINCIPAL_PRIVATE_KEY, OCI_PRINCIPAL_REGION,
|
||||
# OCI_PRINCIPAL_COMPARTMENT_OCID
|
||||
# The credential vm-single-principal builds with, replacing the
|
||||
# run's own for that one topology. The dynamic group is a
|
||||
# tenancy-root resource, so a tenancy that will not let
|
||||
# setup-identity.sh create one refuses every allocation however
|
||||
# correct the code is, and the principal route has to be proved
|
||||
# in a tenancy we administer while the rest of the run stays
|
||||
# where it is.
|
||||
#
|
||||
# All six or none. A partial set builds one tenancy's instances
|
||||
# against another's compartment, which is refused rather than
|
||||
# reported; with none set the topology is recorded SKIPPED
|
||||
# rather than costing a provision to fail. Running it in the
|
||||
# run's own tenancy is still available by pointing all six
|
||||
# there, which says so explicitly instead of by default.
|
||||
# OCI_ARTIFACT_DIR Where logs and the verdict land. Default ./.e2e-oci-<stamp>.
|
||||
# OCI_KEEP_ON_FAIL 1 to leave a failed topology up for inspection. Off by
|
||||
# default: an OCI bare-metal host left overnight is expensive.
|
||||
@@ -118,6 +143,59 @@ mkdir -p "$ARTIFACT_DIR"
|
||||
VERDICT="$ARTIFACT_DIR/verdict.txt"
|
||||
: > "$VERDICT"
|
||||
|
||||
# vm-single-principal is the one topology that cannot run wherever the others do.
|
||||
# It needs the tenancy's dynamic group, and creating that is a tenancy-root write,
|
||||
# so the tenancy a shared test account gives us can never pass it. These let it
|
||||
# build somewhere else without moving the rest of the run.
|
||||
PRINCIPAL_CRED_VARS="TENANCY_OCID USER_OCID FINGERPRINT PRIVATE_KEY REGION COMPARTMENT_OCID"
|
||||
|
||||
# principal_cred_state is none, all or partial. Partial is the one worth a stop:
|
||||
# one tenancy's OCIDs with another's compartment builds in a compartment nobody
|
||||
# chose, and the apply reports success doing it.
|
||||
principal_cred_state() {
|
||||
local name var have=0 miss=0
|
||||
for name in $PRINCIPAL_CRED_VARS; do
|
||||
var="OCI_PRINCIPAL_$name"
|
||||
if [ -n "${!var:-}" ]; then
|
||||
have=$((have + 1))
|
||||
else
|
||||
miss=$((miss + 1))
|
||||
fi
|
||||
done
|
||||
if [ "$have" = 0 ]; then
|
||||
printf 'none\n'
|
||||
elif [ "$miss" = 0 ]; then
|
||||
printf 'all\n'
|
||||
else
|
||||
printf 'partial\n'
|
||||
fi
|
||||
}
|
||||
|
||||
# Into an array rather than a rendered string, because the private key is a
|
||||
# multi-line PEM and any line-based plumbing silently truncates it to its header.
|
||||
PRINCIPAL_ENV=()
|
||||
build_principal_env() {
|
||||
local name var
|
||||
PRINCIPAL_ENV=()
|
||||
for name in $PRINCIPAL_CRED_VARS; do
|
||||
var="OCI_PRINCIPAL_$name"
|
||||
case "$name" in
|
||||
COMPARTMENT_OCID) PRINCIPAL_ENV+=("TF_VAR_compartment_ocid=${!var}") ;;
|
||||
*) PRINCIPAL_ENV+=("OCI_$name=${!var}") ;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
# Here rather than at the topology, because the build runs first and takes minutes,
|
||||
# and a credential that cannot work is knowable before any of it. The loop checks
|
||||
# again, since it is what reads the variables.
|
||||
case " $TOPOLOGIES " in
|
||||
*" vm-single-principal "*)
|
||||
[ "$(principal_cred_state)" != partial ] \
|
||||
|| die "the OCI_PRINCIPAL_* credential is incomplete: set all of $PRINCIPAL_CRED_VARS or none of them"
|
||||
;;
|
||||
esac
|
||||
|
||||
# The run page is the evidence, and its readers are not us: someone assessing
|
||||
# whether Spinifex works on their cloud should get the answer without opening a
|
||||
# log or knowing what this harness is. Markdown on stdout as well as to the step
|
||||
@@ -155,6 +233,12 @@ summary() {
|
||||
else
|
||||
printf 'Allocator authenticated with an API key, except `vm-single-principal` which authenticates as the instance.\n\n'
|
||||
fi
|
||||
# Said in the summary because the row is otherwise read as a result about the
|
||||
# same tenancy as every row above it, which is the one thing it is not.
|
||||
if [ "$(principal_cred_state)" = all ]; then
|
||||
printf '`vm-single-principal` built in its own tenancy in `%s`, from the `OCI_PRINCIPAL_*` credential.\n\n' \
|
||||
"$OCI_PRINCIPAL_REGION"
|
||||
fi
|
||||
|
||||
local topology results workbooks gate status detail name secs
|
||||
for topology in $TOPOLOGIES; do
|
||||
@@ -206,6 +290,7 @@ publish_summary() {
|
||||
# destroy what one of these runs created.
|
||||
sweep() {
|
||||
local found=0 dir topology
|
||||
local sweep_env=(env)
|
||||
for dir in "$STATE_ROOT"/.validate-*; do
|
||||
[ -s "$dir/terraform.tfstate" ] || continue
|
||||
python3 -c '
|
||||
@@ -217,8 +302,16 @@ sys.exit(0 if any(r["instances"] for r in state.get("resources", [])) else 1)
|
||||
# The child reads OCI_STATE_ROOT from the environment, so it looks in the
|
||||
# same place this loop found the state rather than beside its own script.
|
||||
found=1
|
||||
# The same credential the topology built with, or the destroy authenticates
|
||||
# against a tenancy that never held these resources and reports a clean
|
||||
# sweep over a deployment that is still billing.
|
||||
sweep_env=(env)
|
||||
if [ "$topology" = vm-single-principal ] && [ "$(principal_cred_state)" = all ]; then
|
||||
build_principal_env
|
||||
sweep_env=(env "${PRINCIPAL_ENV[@]}")
|
||||
fi
|
||||
log "sweep: $topology still holds resources, destroying"
|
||||
if "$HERE/validate-topology.sh" --topology "$topology" --destroy-only \
|
||||
if "${sweep_env[@]}" "$HERE/validate-topology.sh" --topology "$topology" --destroy-only \
|
||||
>> "$ARTIFACT_DIR/sweep.log" 2>&1; then
|
||||
log "sweep: $topology destroyed"
|
||||
else
|
||||
@@ -277,6 +370,25 @@ fi
|
||||
|
||||
RUN_RC=0
|
||||
for topology in $TOPOLOGIES; do
|
||||
# The credential swap happens here and nowhere else, so everything downstream
|
||||
# of the one validate call is unaware the topology built somewhere different.
|
||||
# Always a bare `env` otherwise, because an empty array is an unbound variable
|
||||
# under set -u on the bash 3.2 operators run this with.
|
||||
run_env=(env)
|
||||
if [ "$topology" = vm-single-principal ]; then
|
||||
case "$(principal_cred_state)" in
|
||||
none)
|
||||
log "$topology: SKIPPED -- no OCI_PRINCIPAL_* credential, and the run's own tenancy cannot hold the dynamic group this topology needs"
|
||||
echo "$topology: SKIPPED (no principal credential)" >> "$VERDICT"
|
||||
continue
|
||||
;;
|
||||
partial)
|
||||
die "the OCI_PRINCIPAL_* credential is incomplete: set all of $PRINCIPAL_CRED_VARS or none of them"
|
||||
;;
|
||||
esac
|
||||
build_principal_env
|
||||
run_env=(env "${PRINCIPAL_ENV[@]}")
|
||||
fi
|
||||
args=(--topology "$topology" --ssh-public-key "$SSH_PUBLIC_KEY" --ssh-private-key "$SSH_PRIVATE_KEY")
|
||||
[ "$DRY_RUN" = 1 ] && args+=(--dry-run)
|
||||
# Two separate choices. --no-external-pool is the one that forms without any
|
||||
@@ -293,7 +405,14 @@ for topology in $TOPOLOGIES; do
|
||||
elif [ "${OCI_INSTANCE_PRINCIPAL:-1}" = 1 ]; then
|
||||
args+=(--instance-principal)
|
||||
fi
|
||||
[ -n "$DISTRO_TARBALL" ] && args+=(--distro "$DISTRO_TARBALL" --setup-sh "$ARTIFACT_DIR/setup.sh")
|
||||
# Not under --dry-run, where the build was skipped and naming the tarball it
|
||||
# would have produced fails the readable check, so every topology reported FAIL
|
||||
# and the flag operators are told to run first could never pass. Still
|
||||
# unconditional in a real run, so a build that produced nothing stays loud
|
||||
# rather than quietly installing a release instead.
|
||||
if [ -n "$DISTRO_TARBALL" ] && [ "$DRY_RUN" != 1 ]; then
|
||||
args+=(--distro "$DISTRO_TARBALL" --setup-sh "$ARTIFACT_DIR/setup.sh")
|
||||
fi
|
||||
if [ "$SOURCE" = release ]; then
|
||||
if [ -n "$INSTALL_VERSION" ]; then
|
||||
args+=(--version "$INSTALL_VERSION")
|
||||
@@ -310,7 +429,7 @@ for topology in $TOPOLOGIES; do
|
||||
tname="TestOCITopology_${topology//-/_}"
|
||||
start=$SECONDS
|
||||
echo "=== RUN $tname"
|
||||
if "$HERE/validate-topology.sh" "${args[@]}" 2>&1 | tee "$ARTIFACT_DIR/$topology.log"; then
|
||||
if "${run_env[@]}" "$HERE/validate-topology.sh" "${args[@]}" 2>&1 | tee "$ARTIFACT_DIR/$topology.log"; then
|
||||
printf -- '--- PASS: %s (%d.00s)\n' "$tname" "$((SECONDS - start))"
|
||||
echo "$topology: PASS" >> "$VERDICT"
|
||||
else
|
||||
|
||||
@@ -17,7 +17,7 @@ set -euo pipefail
|
||||
|
||||
KEY=${KEY:-$HOME/.ssh/oci-poc.pem}
|
||||
OCI_BIN=${OCI_BIN:-$HOME/bin/oci}
|
||||
OCI_PROFILE=${OCI_PROFILE:-apacanzset03child03}
|
||||
OCI_PROFILE=${OCI_PROFILE:-DEFAULT}
|
||||
SUBNET=${SUBNET:-ocid1.subnet.oc1.ap-sydney-1.aaaaaaaaimrhjgsqgrfg7bnka2d3nbx2n5ss7kurcwneon22rkclqf3ypfqa}
|
||||
|
||||
# Public address -> node name. The control plane is reached through node01.
|
||||
|
||||
@@ -23,7 +23,7 @@ resource "oci_identity_dynamic_group" "nodes" {
|
||||
|
||||
# Narrower than it looks. The allocator creates, moves and deletes secondary
|
||||
# private IPs on a node's own VNIC and attaches public IPs to them, which is what
|
||||
# these three verbs cover; nothing here grants compute, storage or identity.
|
||||
# these four verbs cover; nothing here grants compute, storage or identity.
|
||||
resource "oci_identity_policy" "nodes" {
|
||||
provider = oci.home
|
||||
count = var.instance_principal == "create" ? 1 : 0
|
||||
@@ -35,5 +35,10 @@ resource "oci_identity_policy" "nodes" {
|
||||
"Allow dynamic-group ${oci_identity_dynamic_group.nodes[0].name} to use vnics in compartment id ${var.compartment_ocid}",
|
||||
"Allow dynamic-group ${oci_identity_dynamic_group.nodes[0].name} to manage private-ips in compartment id ${var.compartment_ocid}",
|
||||
"Allow dynamic-group ${oci_identity_dynamic_group.nodes[0].name} to manage public-ips in compartment id ${var.compartment_ocid}",
|
||||
|
||||
# CreatePrivateIp needs SUBNET_ATTACH and DeletePrivateIp needs SUBNET_DETACH,
|
||||
# both of which only the subnets `use` verb carries. Without it every allocation
|
||||
# is refused as a 404, which reads as a missing resource rather than a missing grant.
|
||||
"Allow dynamic-group ${oci_identity_dynamic_group.nodes[0].name} to use subnets in compartment id ${var.compartment_ocid}",
|
||||
]
|
||||
}
|
||||
|
||||
@@ -23,11 +23,8 @@ import sys
|
||||
|
||||
|
||||
REPOSITORY = Path(__file__).resolve().parent.parent
|
||||
REQUESTED_PROFILE = "apacanzset03child03"
|
||||
FALLBACK_PROFILE = "apacanzset03child3"
|
||||
# The OCI CLI's own name for "the profile when none is named", so a config written
|
||||
# by `oci setup config` works with no flag. Tried only when nothing was asked for
|
||||
# explicitly, which keeps a named profile a hard requirement rather than a hint.
|
||||
# by `oci setup config` works with no flag.
|
||||
DEFAULT_PROFILE = "DEFAULT"
|
||||
|
||||
|
||||
@@ -35,10 +32,9 @@ def default_profile() -> str:
|
||||
"""The profile to read when the caller named none.
|
||||
|
||||
OCI_CLI_PROFILE is the OCI CLI's own variable, so an operator who has already
|
||||
selected a profile for `oci` does not select it twice. The reference tenancy's
|
||||
name remains the last resort, which is what every run of ours uses.
|
||||
selected a profile for `oci` does not select it twice.
|
||||
"""
|
||||
return os.environ.get("OCI_CLI_PROFILE") or REQUESTED_PROFILE
|
||||
return os.environ.get("OCI_CLI_PROFILE") or DEFAULT_PROFILE
|
||||
|
||||
|
||||
def has_profile(parser: configparser.RawConfigParser, name: str) -> bool:
|
||||
@@ -64,22 +60,15 @@ def load_profile(config_path: Path, requested_profile: str) -> tuple[str, dict[s
|
||||
parser = configparser.RawConfigParser()
|
||||
parser.read(config_path)
|
||||
profile = requested_profile
|
||||
# No fallback to another profile. A missing one is an error because the only
|
||||
# alternative is building a cluster in whichever tenancy a different profile
|
||||
# happens to point at.
|
||||
if not has_profile(parser, profile):
|
||||
# Only an unasked-for profile falls back. A name the caller passed and the
|
||||
# config does not hold is an error, because the alternative is building a
|
||||
# cluster in whichever tenancy DEFAULT happens to point at.
|
||||
implicit = profile == default_profile()
|
||||
for candidate in (FALLBACK_PROFILE, DEFAULT_PROFILE) if implicit else ():
|
||||
if has_profile(parser, candidate):
|
||||
print(f"Profile {profile} not found; using {candidate}.", file=sys.stderr)
|
||||
profile = candidate
|
||||
break
|
||||
else:
|
||||
raise ValueError(
|
||||
f"OCI profile {profile!r} was not found in {config_path}; "
|
||||
f"profiles present: {', '.join(profile_names(parser)) or 'none'}. "
|
||||
"Name one with --profile or OCI_CLI_PROFILE."
|
||||
)
|
||||
raise ValueError(
|
||||
f"OCI profile {profile!r} was not found in {config_path}; "
|
||||
f"profiles present: {', '.join(profile_names(parser)) or 'none'}. "
|
||||
"Name one with --profile or OCI_CLI_PROFILE."
|
||||
)
|
||||
|
||||
values = {key: value.strip() for key, value in parser.items(profile)}
|
||||
required = ("tenancy", "user", "fingerprint", "key_file", "region")
|
||||
@@ -209,7 +198,7 @@ def main() -> int:
|
||||
arg_parser.add_argument(
|
||||
"--profile",
|
||||
default=default_profile(),
|
||||
help="Profile in ~/.oci/config. Defaults to $OCI_CLI_PROFILE, then the reference tenancy, then DEFAULT.",
|
||||
help="Profile in ~/.oci/config. Defaults to $OCI_CLI_PROFILE, else DEFAULT.",
|
||||
)
|
||||
arg_parser.add_argument("--region")
|
||||
arg_parser.add_argument(
|
||||
|
||||
@@ -10,6 +10,10 @@
|
||||
# Its own state file, deliberately. validate-topology.sh destroys its state at the
|
||||
# end of every run, and these two resources outlive every topology -- holding them
|
||||
# in a per-topology state would let a nightly teardown delete the tenancy's policy.
|
||||
#
|
||||
# MUST MAINTAIN BASH 3.X COMPATABILITY. Operators run this on macOS, which ships
|
||||
# bash 3.2: no declare -A, no mapfile, and under set -u an empty "${arr[@]}" is an
|
||||
# unbound variable, so guard every such expansion with ${#arr[@]}.
|
||||
set -uo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
@@ -6,6 +6,10 @@
|
||||
#
|
||||
# The argument shape is validate-topology.sh's credential-hook contract, so this
|
||||
# drops straight into --credential-hook or $OCI_CREDENTIAL_HOOK.
|
||||
#
|
||||
# MUST MAINTAIN BASH 3.X COMPATABILITY. Operators run this on macOS, which ships
|
||||
# bash 3.2: no declare -A, no mapfile, and under set -u an empty "${arr[@]}" is an
|
||||
# unbound variable, so guard every such expansion with ${#arr[@]}.
|
||||
set -euo pipefail
|
||||
|
||||
HERE=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd)
|
||||
|
||||
@@ -6,6 +6,10 @@
|
||||
#
|
||||
# Every run is build-prove-destroy. A topology that cannot be torn down is half
|
||||
# proved, so the teardown decides the verdict and runs even on failure.
|
||||
#
|
||||
# MUST MAINTAIN BASH 3.X COMPATABILITY. Operators run this on macOS, which ships
|
||||
# bash 3.2: no declare -A, no mapfile, and under set -u an empty "${arr[@]}" is an
|
||||
# unbound variable, so guard every such expansion with ${#arr[@]}.
|
||||
set -euo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
@@ -223,10 +227,13 @@ tf() {
|
||||
# terraform output takes -state but not -var, so the two sets are kept apart. They
|
||||
# were one set once, and the -var made output fail into the default state file.
|
||||
tf_state=(-state "$STATE_DIR/terraform.tfstate")
|
||||
tf_vars=(
|
||||
"${tf_principal_var[@]}"
|
||||
"${tf_state[@]}"
|
||||
)
|
||||
tf_vars=("${tf_state[@]}")
|
||||
# Expanded only when it holds something. bash before 4.4, which is what macOS
|
||||
# ships, calls "${arr[@]}" on an empty array an unbound variable under set -u, and
|
||||
# tf_principal_var is empty on every run that did not pass --instance-principal.
|
||||
if [ "${#tf_principal_var[@]}" -gt 0 ]; then
|
||||
tf_vars=("${tf_principal_var[@]}" "${tf_vars[@]}")
|
||||
fi
|
||||
|
||||
ssh_node() {
|
||||
local host="$1"
|
||||
@@ -244,6 +251,9 @@ ssh_node() {
|
||||
# tail for context -- unbounded, a multi-hour suite's journal dwarfs the artifact.
|
||||
capture_journals() {
|
||||
local host
|
||||
# A failure before the apply leaves no hosts, and on bash before 4.4 iterating
|
||||
# the empty array is an unbound-variable error rather than zero passes.
|
||||
[ "${#HOSTS[@]}" -gt 0 ] || return 0
|
||||
for host in "${HOSTS[@]}"; do
|
||||
ssh_node "$host" '
|
||||
echo "=== spinifex, warning and above ==="
|
||||
@@ -378,8 +388,38 @@ for res in state.get("resources", []):
|
||||
[ -n "$stray" ] && log "NOTE: the hand-driven state in $HERE still holds: $stray"
|
||||
fi
|
||||
|
||||
# Terraform ranks a *.auto.tfvars file above TF_VAR_*, so a leftover one wins
|
||||
# silently over an operator targeting a different tenancy by environment. The
|
||||
# guide tells every operator to write one, which is what makes this likely.
|
||||
assert_no_tfvars_override() {
|
||||
local f want got
|
||||
want="${TF_VAR_compartment_ocid:-}"
|
||||
[ -n "$want" ] || return 0
|
||||
for f in "$HERE"/*.auto.tfvars "$HERE"/terraform.tfvars; do
|
||||
[ -f "$f" ] || continue
|
||||
got=$(sed -n 's/^[[:space:]]*compartment_ocid[[:space:]]*=[[:space:]]*"\([^"]*\)".*/\1/p' "$f" | tail -1)
|
||||
[ -n "$got" ] || continue
|
||||
[ "$got" = "$want" ] && continue
|
||||
die "${f##*/} sets compartment_ocid=$got, which Terraform ranks above the TF_VAR_compartment_ocid=$want this run was given, so it would build in a compartment nobody chose. Remove that file or make it agree."
|
||||
done
|
||||
}
|
||||
assert_no_tfvars_override
|
||||
|
||||
if [ "$DRY_RUN" = 1 ]; then
|
||||
tf plan -no-color "${tf_vars[@]}" | tail -30
|
||||
# Whole plan to a file, summary to the terminal. A dry run is here to catch a
|
||||
# plan aimed at the wrong compartment, and `tail` dropped the resource count
|
||||
# and every compartment and shape line along with it, leaving only the outputs
|
||||
# block -- so the one check it exists for could not be made from its output.
|
||||
mkdir -p "$STATE_DIR"
|
||||
tf plan -no-color "${tf_vars[@]}" > "$STATE_DIR/plan.txt" 2>&1 \
|
||||
|| { tail -30 "$STATE_DIR/plan.txt"; die "plan failed; see $STATE_DIR/plan.txt"; }
|
||||
grep -E '^Plan: |^(No changes)' "$STATE_DIR/plan.txt" || true
|
||||
printf 'compartment and shape as planned:\n'
|
||||
# Terraform aligns the = to the widest key in each block, so the same value
|
||||
# appears several times at several indents unless the spacing is normalised.
|
||||
grep -hoE '(compartment_id|shape|availability_domain) += +"[^"]*"' "$STATE_DIR/plan.txt" \
|
||||
| sed -E 's/ +=/ =/' | sort -u | sed 's/^/ /'
|
||||
log "full plan in $STATE_DIR/plan.txt"
|
||||
plan="install Spinifex on $NODES node(s), form the cluster"
|
||||
[ "$SKIP_POOL" = 1 ] && plan="$plan, configure no external pool"
|
||||
if [ "$SKIP_WORKLOAD" = 1 ]; then
|
||||
@@ -623,9 +663,16 @@ allocator_diagnostics() {
|
||||
' 2>&1
|
||||
}
|
||||
|
||||
# resolved the external VNIC is the line that proves the credential works, the
|
||||
# compartment is right and br-wan's MAC matched a real VNIC. A node missing it
|
||||
# accepts allocate-address and then fails it.
|
||||
# credential authorised to allocate is the line that proves the credential is
|
||||
# accepted, in the right compartment, and holds every permission an allocation
|
||||
# needs. A node missing it accepts allocate-address and then fails it.
|
||||
#
|
||||
# "resolved the external VNIC" is not enough and was what this used to wait for.
|
||||
# That line comes from the metadata service and needs no IAM policy at all, so a
|
||||
# node with an unauthorised credential passed here and failed hours later inside
|
||||
# a workbook, reported as capacity.
|
||||
ALLOCATOR_READY='ocinet credential authorised to allocate'
|
||||
ALLOCATOR_DENIED='ocinet credential cannot allocate an external address'
|
||||
if [ "$SKIP_POOL" = 1 ]; then
|
||||
log "--no-external-pool: no allocator to check"
|
||||
record "oci allocator" SKIPPED "no external pool configured"
|
||||
@@ -637,10 +684,16 @@ else
|
||||
# spinifex-daemon, not spinifex-vpcd: the allocator is built in the
|
||||
# daemon, and vpcd only consumes the addresses it hands out. Gating on
|
||||
# vpcd's journal failed a node whose allocator was working.
|
||||
if ssh_node "$host" "sudo journalctl -u spinifex-daemon --since -10min --no-pager | grep -q 'resolved the external VNIC'" 2>/dev/null; then
|
||||
if ssh_node "$host" "sudo journalctl -u spinifex-daemon --since -10min --no-pager | grep -q '$ALLOCATOR_READY'" 2>/dev/null; then
|
||||
found=1
|
||||
break
|
||||
fi
|
||||
# A refusal is final, so waiting out the remaining five minutes only
|
||||
# delays a verdict the node has already reached.
|
||||
if denial=$(ssh_node "$host" "sudo journalctl -u spinifex-daemon --since -10min --no-pager | grep -m1 '$ALLOCATOR_DENIED'" 2>/dev/null) && [ -n "$denial" ]; then
|
||||
allocator_diagnostics "$host" > "$STATE_DIR/allocator-$host.log" 2>&1 || true
|
||||
die "$host refused its own OCI credential: ${denial#*$ALLOCATOR_DENIED}. Diagnostics: $STATE_DIR/allocator-$host.log"
|
||||
fi
|
||||
sleep 10
|
||||
done
|
||||
if [ "$found" != 1 ]; then
|
||||
@@ -649,7 +702,7 @@ else
|
||||
# not load, a bridge MAC matching no VNIC, and a pool vpcd never read --
|
||||
# and they are indistinguishable from the missing log line alone.
|
||||
allocator_diagnostics "$host" > "$STATE_DIR/allocator-$host.log" 2>&1 || true
|
||||
die "$host never logged 'resolved the external VNIC'; the OCI allocator is not up, so no guest can get a public address. Diagnostics: $STATE_DIR/allocator-$host.log"
|
||||
die "$host never logged '$ALLOCATOR_READY'; the OCI allocator is not up, so no guest can get a public address. Diagnostics: $STATE_DIR/allocator-$host.log"
|
||||
fi
|
||||
log "$host allocator ready"
|
||||
done
|
||||
@@ -686,6 +739,24 @@ fi
|
||||
# workbooks, so a workbook that passes on a hypervisor and fails on OCI is a
|
||||
# difference in OCI and not in the test. It owns its own per-workbook assertions
|
||||
# and destroys each one it builds.
|
||||
# The suite used to run with stdout redirected to a file, so a run wedged inside a
|
||||
# workbook printed nothing at all until it finished. Streamed now, and each of the
|
||||
# driver's own RUN/PASS/FAIL lines is stamped with elapsed seconds and the workbook.
|
||||
stream_workbooks() {
|
||||
local start=$SECONDS line current="(starting)"
|
||||
while IFS= read -r line; do
|
||||
case "$line" in
|
||||
'=== RUN'*)
|
||||
current="${line##*RUN }"
|
||||
printf '[validate-%s] %5ds workbook START %s\n' "$TOPOLOGY" "$((SECONDS - start))" "$current" ;;
|
||||
'--- PASS'*|'--- FAIL'*|'--- SKIP'*)
|
||||
printf '[validate-%s] %5ds workbook %s\n' "$TOPOLOGY" "$((SECONDS - start))" "$line" ;;
|
||||
*)
|
||||
printf '%s\n' "$line" ;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
log "running the published workbooks"
|
||||
DRIVER="$REPO_ROOT/tests/e2e/run-tofu-examples-e2e.sh"
|
||||
[ -r "$DRIVER" ] || die "no workbook driver at $DRIVER"
|
||||
@@ -760,7 +831,7 @@ if ssh -i "$SSH_PRIVATE_KEY" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/
|
||||
-o ServerAliveInterval=30 -o ServerAliveCountMax=6 \
|
||||
-o ExitOnForwardFailure=yes -R "$PUBLIC_PROXY_PORT" "ubuntu@${HOSTS[0]}" \
|
||||
"chmod +x ~/run-tofu-examples-e2e.sh; $workbook_env ~/run-tofu-examples-e2e.sh" \
|
||||
> "$STATE_DIR/workbooks.log" 2>&1; then
|
||||
2>&1 | tee "$STATE_DIR/workbooks.log" | stream_workbooks; then
|
||||
record_workbooks "$STATE_DIR/workbooks.log"
|
||||
passed="$(grep -c '^--- PASS' "$STATE_DIR/workbooks.log" || true)"
|
||||
ran="$(grep -c '^=== RUN' "$STATE_DIR/workbooks.log" || true)"
|
||||
|
||||
@@ -217,3 +217,94 @@ func TestGetPublicIPByPrivateIPIDAsksByPrivateIP(t *testing.T) {
|
||||
assert.Equal(t, "q1", got.ID)
|
||||
assert.Equal(t, "p1", (*seen)[0].body["privateIpId"])
|
||||
}
|
||||
|
||||
// The subnet read is the allocator's authorisation probe, so the request has to
|
||||
// address one subnet by OCID. A GET of the collection would answer from a
|
||||
// different permission and report a credential as working when it is not.
|
||||
func TestGetSubnetAddressesOneSubnetByOCID(t *testing.T) {
|
||||
c, seen := testClient(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
_, _ = w.Write([]byte(`{"id":"ocid1.subnet.oc1..s1","cidrBlock":"10.200.0.0/23"}`))
|
||||
})
|
||||
|
||||
got, err := c.GetSubnet(context.Background(), "ocid1.subnet.oc1..s1")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "ocid1.subnet.oc1..s1", got.ID)
|
||||
assert.Equal(t, "10.200.0.0/23", got.CIDRBlock)
|
||||
require.Len(t, *seen, 1)
|
||||
assert.Equal(t, http.MethodGet, (*seen)[0].method)
|
||||
assert.Equal(t, "/20160918/subnets/ocid1.subnet.oc1..s1", (*seen)[0].path)
|
||||
}
|
||||
|
||||
// A subnet with no CIDR is not an error: the probe only needs the call to be
|
||||
// authorised, and treating a sparse response as a failure would report a
|
||||
// working credential as broken.
|
||||
func TestGetSubnetKeepsTheOCIDWhenTheBodyHasNoCIDR(t *testing.T) {
|
||||
c, _ := testClient(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
_, _ = w.Write([]byte(`{"id":"ocid1.subnet.oc1..s1"}`))
|
||||
})
|
||||
|
||||
got, err := c.GetSubnet(context.Background(), "ocid1.subnet.oc1..s1")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "ocid1.subnet.oc1..s1", got.ID)
|
||||
assert.Empty(t, got.CIDRBlock)
|
||||
}
|
||||
|
||||
// The 404 OCI returns for an unauthorised call has to arrive as ErrNotFound, or
|
||||
// the allocator's check cannot tell it from a transport failure and would retry
|
||||
// a verdict that will never change.
|
||||
func TestGetSubnetMapsTheUnauthorisedFourOhFour(t *testing.T) {
|
||||
c, _ := testClient(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
_, _ = w.Write([]byte(`{"code":"NotAuthorizedOrNotFound","message":"Authorization failed or requested resource not found."}`))
|
||||
})
|
||||
|
||||
_, err := c.GetSubnet(context.Background(), "ocid1.subnet.oc1..s1")
|
||||
require.Error(t, err)
|
||||
assert.ErrorIs(t, err, oci.ErrNotFound)
|
||||
assert.Contains(t, err.Error(), "GetSubnet")
|
||||
}
|
||||
|
||||
// The only call that can see an address no VNIC references. Both filters are
|
||||
// load bearing: without REGION scope it answers for one availability domain,
|
||||
// and without RESERVED it returns ephemeral addresses nothing may delete.
|
||||
func TestListReservedPublicIPsFiltersToReservedRegionalAndPages(t *testing.T) {
|
||||
page := 0
|
||||
c, seen := testClient(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
page++
|
||||
if page == 1 {
|
||||
w.Header().Set("Opc-Next-Page", "p2")
|
||||
_, _ = w.Write([]byte(`[{"id":"q1","ipAddress":"203.0.113.1","lifecycleState":"AVAILABLE","lifetime":"RESERVED","timeCreated":"2026-10-01T00:00:00.000Z"}]`))
|
||||
return
|
||||
}
|
||||
_, _ = w.Write([]byte(`[{"id":"q2","ipAddress":"203.0.113.2","lifecycleState":"ASSIGNED","lifetime":"RESERVED"}]`))
|
||||
})
|
||||
|
||||
got, err := c.ListReservedPublicIPs(context.Background(), "ocid1.compartment.oc1..c1")
|
||||
require.NoError(t, err)
|
||||
|
||||
require.Len(t, got, 2)
|
||||
assert.Equal(t, "q1", got[0].ID)
|
||||
assert.Equal(t, "q2", got[1].ID)
|
||||
assert.Equal(t, 2026, got[0].TimeCreated.Year(), "the creation time is what the orphan grace period is measured from")
|
||||
assert.True(t, got[1].TimeCreated.IsZero(), "a response with no creation time must not read as the zero epoch")
|
||||
|
||||
require.Len(t, *seen, 2)
|
||||
q := (*seen)[0].query
|
||||
assert.Equal(t, "REGION", q.Get("scope"))
|
||||
assert.Equal(t, "RESERVED", q.Get("lifetime"))
|
||||
assert.Equal(t, "ocid1.compartment.oc1..c1", q.Get("compartmentId"))
|
||||
assert.Equal(t, "p2", (*seen)[1].query.Get("page"))
|
||||
}
|
||||
|
||||
// A credential without `read public-ips` gets the same 404 as a missing
|
||||
// compartment, and reconcile distinguishes them by not treating either as fatal.
|
||||
func TestListReservedPublicIPsMapsTheUnauthorisedFourOhFour(t *testing.T) {
|
||||
c, _ := testClient(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
_, _ = w.Write([]byte(`{"code":"NotAuthorizedOrNotFound","message":"nope"}`))
|
||||
})
|
||||
|
||||
_, err := c.ListReservedPublicIPs(context.Background(), "ocid1.compartment.oc1..c1")
|
||||
require.Error(t, err)
|
||||
assert.ErrorIs(t, err, oci.ErrNotFound)
|
||||
}
|
||||
|
||||
@@ -33,6 +33,11 @@ type Client interface {
|
||||
GetPrivateIP(ctx context.Context, privateIPID string) (PrivateIP, error)
|
||||
ListPrivateIPs(ctx context.Context, vnicID string) ([]PrivateIP, error)
|
||||
|
||||
// GetSubnet reads one subnet. The allocator calls it at startup to prove
|
||||
// its credential holds a subnet permission, since AssignPrivateIP needs
|
||||
// SUBNET_ATTACH from the same verb and is otherwise refused as a 404.
|
||||
GetSubnet(ctx context.Context, subnetID string) (Subnet, error)
|
||||
|
||||
// CreatePublicIP creates a RESERVED public IP in compartmentID and, when
|
||||
// privateIPID is non-empty, attaches it in the same call.
|
||||
CreatePublicIP(ctx context.Context, compartmentID, privateIPID, displayName string) (PublicIP, error)
|
||||
@@ -45,6 +50,10 @@ type Client interface {
|
||||
// GetPublicIPByPrivateIPID returns the public IP attached to a private IP,
|
||||
// or ErrNotFound when it has none.
|
||||
GetPublicIPByPrivateIPID(ctx context.Context, privateIPID string) (PublicIP, error)
|
||||
// ListReservedPublicIPs lists every reserved regional public IP in the
|
||||
// compartment. A detached one is on no VNIC, so this is the only call that
|
||||
// can see an address nothing references but the tenancy quota still counts.
|
||||
ListReservedPublicIPs(ctx context.Context, compartmentID string) ([]PublicIP, error)
|
||||
}
|
||||
|
||||
// apiClient is the real implementation over the OCI Go SDK.
|
||||
@@ -188,6 +197,45 @@ func (c *apiClient) ListPrivateIPs(ctx context.Context, vnicID string) ([]Privat
|
||||
}
|
||||
}
|
||||
|
||||
func (c *apiClient) ListReservedPublicIPs(ctx context.Context, compartmentID string) ([]PublicIP, error) {
|
||||
var out []PublicIP
|
||||
var page *string
|
||||
for {
|
||||
resp, err := c.net.ListPublicIps(ctx, core.ListPublicIpsRequest{
|
||||
CompartmentId: &compartmentID,
|
||||
Scope: core.ListPublicIpsScopeRegion,
|
||||
Lifetime: core.ListPublicIpsLifetimeReserved,
|
||||
Page: page,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, wrap("ListReservedPublicIPs", err)
|
||||
}
|
||||
for _, item := range resp.Items {
|
||||
ip, err := toPublicIP(item)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, ip)
|
||||
}
|
||||
if resp.OpcNextPage == nil {
|
||||
return out, nil
|
||||
}
|
||||
page = resp.OpcNextPage
|
||||
}
|
||||
}
|
||||
|
||||
func (c *apiClient) GetSubnet(ctx context.Context, subnetID string) (Subnet, error) {
|
||||
resp, err := c.net.GetSubnet(ctx, core.GetSubnetRequest{SubnetId: &subnetID})
|
||||
if err != nil {
|
||||
return Subnet{}, wrap("GetSubnet", err)
|
||||
}
|
||||
out := Subnet{ID: subnetID}
|
||||
if resp.CidrBlock != nil {
|
||||
out.CIDRBlock = *resp.CidrBlock
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (c *apiClient) CreatePublicIP(ctx context.Context, compartmentID, privateIPID, displayName string) (PublicIP, error) {
|
||||
details := core.CreatePublicIpDetails{
|
||||
CompartmentId: &compartmentID,
|
||||
@@ -281,6 +329,9 @@ func toPublicIP(p core.PublicIp) (PublicIP, error) {
|
||||
Lifetime: string(p.Lifetime),
|
||||
LifecycleState: string(p.LifecycleState),
|
||||
}
|
||||
if p.TimeCreated != nil {
|
||||
out.TimeCreated = p.TimeCreated.Time
|
||||
}
|
||||
if s := deref(p.IpAddress); s != "" {
|
||||
addr, err := netip.ParseAddr(s)
|
||||
if err != nil {
|
||||
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"net/netip"
|
||||
"slices"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Fake is an in-memory Client for tests. It is in the production package
|
||||
@@ -83,6 +84,23 @@ func (f *Fake) PublicIPs() []PublicIP {
|
||||
return slices.Collect(maps.Values(f.publicIPs))
|
||||
}
|
||||
|
||||
// Compartments are not modelled, so every reserved public IP reads back
|
||||
// whatever compartment is asked for. Only the lifetime filter is real.
|
||||
func (f *Fake) ListReservedPublicIPs(_ context.Context, _ string) ([]PublicIP, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if err := f.take("ListReservedPublicIPs"); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var out []PublicIP
|
||||
for _, p := range f.publicIPs {
|
||||
if p.Lifetime == LifetimeReserved {
|
||||
out = append(out, p)
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// FailOp stages err for the next call to op. Locked, so a test may stage one
|
||||
// against a Fake that a loop under test is already calling.
|
||||
func (f *Fake) FailOp(op string, err error) {
|
||||
@@ -214,6 +232,17 @@ func (f *Fake) ListPrivateIPs(_ context.Context, vnicID string) ([]PrivateIP, er
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// Subnets are not modelled, so any OCID reads back. FailWith["GetSubnet"] is
|
||||
// how a test reproduces the credential that holds no subnet permission.
|
||||
func (f *Fake) GetSubnet(_ context.Context, subnetID string) (Subnet, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if err := f.take("GetSubnet"); err != nil {
|
||||
return Subnet{}, err
|
||||
}
|
||||
return Subnet{ID: subnetID, CIDRBlock: "10.200.0.0/23"}, nil
|
||||
}
|
||||
|
||||
func (f *Fake) CreatePublicIP(_ context.Context, compartmentID, privateIPID, displayName string) (PublicIP, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
@@ -230,6 +259,7 @@ func (f *Fake) CreatePublicIP(_ context.Context, compartmentID, privateIPID, dis
|
||||
DisplayName: displayName,
|
||||
Lifetime: LifetimeReserved,
|
||||
LifecycleState: LifecycleStateAssigned,
|
||||
TimeCreated: time.Now(),
|
||||
}
|
||||
if privateIPID == "" {
|
||||
p.LifecycleState = LifecycleStateAvailable
|
||||
|
||||
@@ -205,3 +205,48 @@ func TestFakeSeedHelpersExposeStateReconcileTestsNeed(t *testing.T) {
|
||||
assert.Len(t, f.PublicIPs(), 1)
|
||||
assert.True(t, f.PrivateIPs()[0].IsPrimary)
|
||||
}
|
||||
|
||||
// Subnets are not modelled, so the fake reads any OCID back. What matters is
|
||||
// that FailWith reaches it, because a credential holding no subnet permission is
|
||||
// the only way the allocator's authorisation check can fail usefully.
|
||||
func TestFakeGetSubnetReadsBackAndHonoursFailWith(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
f := oci.NewFake()
|
||||
|
||||
got, err := f.GetSubnet(ctx, "ocid1.subnet.oc1..s1")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "ocid1.subnet.oc1..s1", got.ID)
|
||||
assert.NotEmpty(t, got.CIDRBlock)
|
||||
|
||||
denied := errors.New("NotAuthorizedOrNotFound")
|
||||
f.FailWith["GetSubnet"] = denied
|
||||
_, err = f.GetSubnet(ctx, "ocid1.subnet.oc1..s1")
|
||||
assert.ErrorIs(t, err, denied)
|
||||
}
|
||||
|
||||
// The fake's list has to honour the lifetime filter, or a test asserting that
|
||||
// reconcile leaves ephemeral addresses alone would pass for the wrong reason.
|
||||
func TestFakeListReservedPublicIPsReturnsOnlyReservedOnes(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
f := oci.NewFake()
|
||||
f.SeedPublicIP(oci.PublicIP{ID: "q-reserved", Lifetime: oci.LifetimeReserved})
|
||||
f.SeedPublicIP(oci.PublicIP{ID: "q-ephemeral", Lifetime: "EPHEMERAL"})
|
||||
|
||||
got, err := f.ListReservedPublicIPs(ctx, "any-compartment")
|
||||
require.NoError(t, err)
|
||||
require.Len(t, got, 1)
|
||||
assert.Equal(t, "q-reserved", got[0].ID)
|
||||
|
||||
f.FailWith["ListReservedPublicIPs"] = oci.ErrNotFound
|
||||
_, err = f.ListReservedPublicIPs(ctx, "any-compartment")
|
||||
assert.ErrorIs(t, err, oci.ErrNotFound)
|
||||
}
|
||||
|
||||
// CreatePublicIP has to date what it makes, or every address it creates is
|
||||
// immediately eligible for the orphan sweep.
|
||||
func TestFakeCreatePublicIPDatesTheAddress(t *testing.T) {
|
||||
f := oci.NewFake()
|
||||
p, err := f.CreatePublicIP(context.Background(), "c1", "", "spinifex-eni-1")
|
||||
require.NoError(t, err)
|
||||
assert.False(t, p.TimeCreated.IsZero())
|
||||
}
|
||||
|
||||
@@ -22,6 +22,7 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/netip"
|
||||
"time"
|
||||
)
|
||||
|
||||
// PrivateIP is a VCN private IP object on a VNIC. The OCID, not the address,
|
||||
@@ -35,6 +36,14 @@ type PrivateIP struct {
|
||||
IsPrimary bool
|
||||
}
|
||||
|
||||
// Subnet is the VCN subnet a VNIC's private IPs live in. Reading one is the
|
||||
// cheapest call needing a subnet permission, which is why the allocator uses it
|
||||
// to prove its credential: CreatePrivateIp is refused without SUBNET_ATTACH.
|
||||
type Subnet struct {
|
||||
ID string
|
||||
CIDRBlock string
|
||||
}
|
||||
|
||||
// PublicIP is an OCI public IP object. PrivateIPID is empty when the address
|
||||
// is reserved but not currently attached to anything — the state an allocated
|
||||
// but unassociated AWS EIP maps onto.
|
||||
@@ -45,6 +54,9 @@ type PublicIP struct {
|
||||
DisplayName string
|
||||
Lifetime string
|
||||
LifecycleState string
|
||||
// TimeCreated is how a detached address is told from one still being
|
||||
// attached, since both report no private IP and only age separates them.
|
||||
TimeCreated time.Time
|
||||
}
|
||||
|
||||
// IsAssigned reports whether the public IP has finished provisioning and is
|
||||
|
||||
@@ -1501,11 +1501,13 @@ func (d *Daemon) installOCIAllocators(ipam *handlers_ec2_vpc.ExternalIPAM, js je
|
||||
continue
|
||||
}
|
||||
slog.Info("OCI allocator ready", "pool", p.Name,
|
||||
"collected", len(res.Collected), "stale_bindings", len(res.Stale), "skipped", res.Skipped)
|
||||
"collected", len(res.Collected), "stale_bindings", len(res.Stale),
|
||||
"orphaned_public_ips", len(res.Orphaned), "skipped", res.Skipped)
|
||||
d.ociAllocators = append(d.ociAllocators, ociPool{name: p.Name, alloc: alloc})
|
||||
}
|
||||
if len(d.ociAllocators) > 0 {
|
||||
go d.runOCIAffinityLoop()
|
||||
go d.runOCIReconcileLoop()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -1550,6 +1552,11 @@ type ociPool struct {
|
||||
// guest has finished coming up.
|
||||
const ociAffinityInterval = 15 * time.Second
|
||||
|
||||
// ociReconcileInterval is how often leaked OCI objects are collected. Slower
|
||||
// than the affinity pass by three orders of magnitude: a leak costs money and
|
||||
// quota, not reachability, and the pass lists a whole compartment.
|
||||
const ociReconcileInterval = 10 * time.Minute
|
||||
|
||||
// runOCIAffinityLoop keeps OCI's idea of where an address lives in step with
|
||||
// where its guest actually runs. Every node runs its own, like the host EIP
|
||||
// loop and for the same reason: the question is about this host's guests, and a
|
||||
@@ -1568,6 +1575,36 @@ func (d *Daemon) runOCIAffinityLoop() {
|
||||
}
|
||||
}
|
||||
|
||||
// runOCIReconcileLoop repeats the startup reconcile for as long as the node
|
||||
// runs. An address is leaked by a release that half-failed or a guest that went
|
||||
// away, both of which happen long after start, and the startup pass alone left
|
||||
// those to accumulate against the tenancy's reserved-address quota until a
|
||||
// launch failed.
|
||||
func (d *Daemon) runOCIReconcileLoop() {
|
||||
ticker := time.NewTicker(ociReconcileInterval)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-d.ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
}
|
||||
for _, p := range d.ociAllocators {
|
||||
res, err := p.alloc.Reconcile(d.ctx)
|
||||
if err != nil {
|
||||
slog.Error("OCI allocator reconcile failed; leaked addresses may be billing",
|
||||
"pool", p.name, "err", err)
|
||||
continue
|
||||
}
|
||||
if len(res.Collected) > 0 || len(res.Stale) > 0 || len(res.Orphaned) > 0 {
|
||||
slog.Info("OCI allocator reconcile collected leaks", "pool", p.name,
|
||||
"collected", len(res.Collected), "stale_bindings", len(res.Stale),
|
||||
"orphaned_public_ips", len(res.Orphaned))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ClaimOCIAddresses runs one affinity pass over every OCI pool and records which
|
||||
// guests OCI is not delivering an address to.
|
||||
//
|
||||
|
||||
+30
-9
@@ -48,6 +48,12 @@ const DisplayNamePrefix = "spinifex-"
|
||||
// an Elastic IP from an auto-assigned address without importing handlers.
|
||||
const purposeEIP = "eip"
|
||||
|
||||
// The two OCI objects Allocate creates, each with its own tenancy limit.
|
||||
const (
|
||||
objectPrivateIP = "private-ip"
|
||||
objectPublicIP = "public-ip"
|
||||
)
|
||||
|
||||
// assignSchedule is the poll ladder for an address to reach ASSIGNED. OCI
|
||||
// assignment is asynchronous: CreatePublicIp returns ASSIGNING and the address
|
||||
// is not reachable until it settles. Shaped like the DHCP manager's DORA ladder
|
||||
@@ -121,6 +127,9 @@ type Config struct {
|
||||
Budget time.Duration
|
||||
// Sleep is the delay function, so tests do not wait.
|
||||
Sleep func(context.Context, time.Duration) error
|
||||
// Now is the clock the orphan grace period is measured against, so a test
|
||||
// can age an address without waiting for it.
|
||||
Now func() time.Time
|
||||
}
|
||||
|
||||
// Store persists bindings. Implemented over JetStream KV in store.go; the
|
||||
@@ -147,6 +156,7 @@ type PoolAllocator struct {
|
||||
schedule []time.Duration
|
||||
budget time.Duration
|
||||
sleep func(context.Context, time.Duration) error
|
||||
now func() time.Time
|
||||
localPorts func(context.Context) (map[string]struct{}, error)
|
||||
localGateway func(context.Context, string) (bool, error)
|
||||
}
|
||||
@@ -177,6 +187,7 @@ func New(client oci.Client, store Store, cfg Config) (*PoolAllocator, error) {
|
||||
schedule: cfg.Schedule,
|
||||
budget: cfg.Budget,
|
||||
sleep: cfg.Sleep,
|
||||
now: cfg.Now,
|
||||
localPorts: cfg.LocalPorts,
|
||||
localGateway: cfg.LocalGateway,
|
||||
}
|
||||
@@ -189,6 +200,9 @@ func New(client oci.Client, store Store, cfg Config) (*PoolAllocator, error) {
|
||||
if a.sleep == nil {
|
||||
a.sleep = sleepCtx
|
||||
}
|
||||
if a.now == nil {
|
||||
a.now = time.Now
|
||||
}
|
||||
return a, nil
|
||||
}
|
||||
|
||||
@@ -211,7 +225,7 @@ func (a *PoolAllocator) Allocate(ctx context.Context, req external.AllocateReque
|
||||
name := displayName(req)
|
||||
priv, err := a.client.AssignPrivateIP(ctx, a.cfg.VNICID, netip.Addr{}, name)
|
||||
if err != nil {
|
||||
return netip.Addr{}, capacityErr(fmt.Errorf("ocinet: assign private ip: %w", err), err)
|
||||
return netip.Addr{}, capacityErr(objectPrivateIP, fmt.Errorf("ocinet: assign private ip: %w", err), err)
|
||||
}
|
||||
|
||||
pub, err := a.client.CreatePublicIP(ctx, a.cfg.CompartmentID, priv.ID, name)
|
||||
@@ -219,7 +233,7 @@ func (a *PoolAllocator) Allocate(ctx context.Context, req external.AllocateReque
|
||||
// Give the private IP back rather than leaving it for Reconcile: we are
|
||||
// still here and we know exactly what we just made.
|
||||
a.undoPrivate(ctx, priv.ID)
|
||||
return netip.Addr{}, capacityErr(fmt.Errorf("ocinet: create public ip: %w", err), err)
|
||||
return netip.Addr{}, capacityErr(objectPublicIP, fmt.Errorf("ocinet: create public ip: %w", err), err)
|
||||
}
|
||||
|
||||
pub, err = a.waitAssigned(ctx, pub)
|
||||
@@ -448,14 +462,21 @@ func (a *PoolAllocator) undoPublic(ctx context.Context, publicIPID string) {
|
||||
}
|
||||
}
|
||||
|
||||
// capacityErr maps an OCI quota refusal onto the AWS error that means the same
|
||||
// thing, so a customer who has hit the 64-per-VNIC or 50-per-region ceiling is
|
||||
// told they are out of addresses rather than shown an Oracle error code.
|
||||
func capacityErr(wrapped, raw error) error {
|
||||
if errors.Is(raw, oci.ErrLimitExceeded) {
|
||||
return fmt.Errorf("%w: %w", errors.New(awserrors.ErrorInsufficientAddressCapacity), wrapped)
|
||||
// capacityErr maps an OCI quota refusal onto the AWS error for the same limit.
|
||||
// The two ceilings are different failures and a single capacity code hides
|
||||
// which one was hit, which is the difference between waiting and releasing.
|
||||
func capacityErr(object string, wrapped, raw error) error {
|
||||
if !errors.Is(raw, oci.ErrLimitExceeded) {
|
||||
return wrapped
|
||||
}
|
||||
return wrapped
|
||||
code := awserrors.ErrorInsufficientAddressCapacity
|
||||
switch object {
|
||||
case objectPublicIP:
|
||||
code = awserrors.ErrorAddressLimitExceeded
|
||||
case objectPrivateIP:
|
||||
code = awserrors.ErrorPrivateIpAddressLimitExceeded
|
||||
}
|
||||
return fmt.Errorf("%w: %w", errors.New(code), wrapped)
|
||||
}
|
||||
|
||||
// displayName marks the object as ours and carries the AWS identity, so the OCI
|
||||
|
||||
+35
-2
@@ -3,6 +3,7 @@ package ocinet_test
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/netip"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -79,7 +80,8 @@ func TestAllocateWaitsOutAsyncAssignment(t *testing.T) {
|
||||
"Allocate must not return an address that is still provisioning")
|
||||
}
|
||||
|
||||
func TestAllocateSurfacesQuotaExhaustionAsInsufficientAddressCapacity(t *testing.T) {
|
||||
// A full VNIC is the AWS per-interface limit, not a shortage of addresses.
|
||||
func TestAllocateSurfacesAFullVNICAsPrivateIpAddressLimitExceeded(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
fake.PrivateLimit = 1
|
||||
@@ -90,10 +92,41 @@ func TestAllocateSurfacesQuotaExhaustionAsInsufficientAddressCapacity(t *testing
|
||||
|
||||
_, err = a.Allocate(ctx, external.AllocateRequest{PoolName: "oci-wan", AllocationID: "eipalloc-2"})
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), awserrors.ErrorInsufficientAddressCapacity,
|
||||
assert.Contains(t, err.Error(), awserrors.ErrorPrivateIpAddressLimitExceeded,
|
||||
"an OCI quota refusal must reach the customer as the AWS error that means the same thing")
|
||||
}
|
||||
|
||||
// A tenancy out of reserved public IPs is the AWS Elastic IP limit. The two
|
||||
// ceilings resolved to one code before, which told an operator to wait for
|
||||
// capacity when the fix was to release an address they already hold.
|
||||
func TestAllocateSurfacesAnExhaustedPublicIPQuotaAsAddressLimitExceeded(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
fake.FailWith["CreatePublicIP"] = fmt.Errorf("public ip limit reached: %w", oci.ErrLimitExceeded)
|
||||
a, _ := newTestAllocator(t, fake)
|
||||
|
||||
_, err := a.Allocate(ctx, external.AllocateRequest{PoolName: "oci-wan", AllocationID: "eipalloc-1"})
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), awserrors.ErrorAddressLimitExceeded)
|
||||
assert.NotContains(t, err.Error(), awserrors.ErrorInsufficientAddressCapacity,
|
||||
"a quota the customer can free by releasing an address must not read as the region being out of them")
|
||||
}
|
||||
|
||||
// Anything that is not a quota keeps its own error rather than being dressed up
|
||||
// as a limit, or an outage reads as an exhausted account.
|
||||
func TestAllocateLeavesANonQuotaFailureUnmapped(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
fake.FailWith["CreatePublicIP"] = errors.New("boom")
|
||||
a, _ := newTestAllocator(t, fake)
|
||||
|
||||
_, err := a.Allocate(ctx, external.AllocateRequest{PoolName: "oci-wan", AllocationID: "eipalloc-1"})
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "boom")
|
||||
assert.NotContains(t, err.Error(), awserrors.ErrorAddressLimitExceeded)
|
||||
assert.NotContains(t, err.Error(), awserrors.ErrorInsufficientAddressCapacity)
|
||||
}
|
||||
|
||||
// A failure after the private IP exists must not leave it behind: we are still
|
||||
// running and know exactly what we just created, so rolling back beats leaving
|
||||
// a billable object for the next reconcile pass.
|
||||
|
||||
+57
@@ -0,0 +1,57 @@
|
||||
package ocinet
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/network/external"
|
||||
)
|
||||
|
||||
// CheckAuthorisation makes the cheapest read in each permission class an
|
||||
// allocation needs, so a credential that cannot allocate is known at startup
|
||||
// rather than at the first launch that wants an address.
|
||||
//
|
||||
// Reads rather than a real assign-and-release round trip, deliberately: a full
|
||||
// round trip would also fail when the subnet is simply out of addresses, and
|
||||
// reporting that as an authorisation problem sends an operator to IAM for a
|
||||
// capacity fault.
|
||||
func (a *PoolAllocator) CheckAuthorisation(ctx context.Context) error {
|
||||
ips, err := a.client.ListPrivateIPs(ctx, a.cfg.VNICID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("listing the private IPs on vnic %s failed, which needs `use vnics` and `manage private-ips` in compartment %s: %w",
|
||||
a.cfg.VNICID, a.cfg.CompartmentID, err)
|
||||
}
|
||||
|
||||
subnetID := ""
|
||||
for _, ip := range ips {
|
||||
if ip.SubnetID != "" {
|
||||
subnetID = ip.SubnetID
|
||||
break
|
||||
}
|
||||
}
|
||||
if subnetID == "" {
|
||||
// The VNIC always has a primary private IP, so an empty list means the
|
||||
// credential read someone else's VNIC or the OCID is stale.
|
||||
return fmt.Errorf("vnic %s reports no private IP with a subnet, so the subnet permission cannot be checked", a.cfg.VNICID)
|
||||
}
|
||||
|
||||
if _, err := a.client.GetSubnet(ctx, subnetID); err != nil {
|
||||
return fmt.Errorf("reading subnet %s failed, which needs `use subnets` in compartment %s; AssignPrivateIP requires SUBNET_ATTACH from that same verb: %w",
|
||||
subnetID, a.cfg.CompartmentID, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// reportAuthorisation logs the verdict and does not fail construction. The node
|
||||
// still serves every other pool and NAT mode, and a credential fault should not
|
||||
// be the reason a whole node refuses to start.
|
||||
func reportAuthorisation(ctx context.Context, a *PoolAllocator, pool external.ExternalPoolConfig) {
|
||||
if err := a.CheckAuthorisation(ctx); err != nil {
|
||||
slog.ErrorContext(ctx, "ocinet credential cannot allocate an external address",
|
||||
"pool", pool.Name, "vnic_id", a.cfg.VNICID, "compartment", a.cfg.CompartmentID, "error", err)
|
||||
return
|
||||
}
|
||||
slog.InfoContext(ctx, "ocinet credential authorised to allocate",
|
||||
"pool", pool.Name, "vnic_id", a.cfg.VNICID, "compartment", a.cfg.CompartmentID)
|
||||
}
|
||||
+113
@@ -0,0 +1,113 @@
|
||||
package ocinet_test
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"net/netip"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/cloud/oci"
|
||||
"github.com/mulgadc/spinifex/spinifex/network/external/ocinet"
|
||||
)
|
||||
|
||||
// The primary private IP is what carries the subnet OCID, so the check needs one
|
||||
// seeded to have anything to ask about.
|
||||
func seedPrimary(fake *oci.Fake) {
|
||||
fake.SeedPrivateIP(oci.PrivateIP{
|
||||
ID: "ocid1.privateip.oc1..primary",
|
||||
Address: netip.MustParseAddr("10.200.0.2"),
|
||||
VNICID: "ocid1.vnic.oc1..vnic1",
|
||||
SubnetID: "ocid1.subnet.oc1..subnet1",
|
||||
IsPrimary: true,
|
||||
})
|
||||
}
|
||||
|
||||
func TestCheckAuthorisationPassesWhenEveryPermissionIsHeld(t *testing.T) {
|
||||
fake := oci.NewFake()
|
||||
seedPrimary(fake)
|
||||
a, _ := newTestAllocator(t, fake)
|
||||
|
||||
require.NoError(t, a.CheckAuthorisation(context.Background()))
|
||||
}
|
||||
|
||||
// The defect this guards: a credential granted vnics and private-ips but no
|
||||
// subnet verb reads both of those fine and is refused every AssignPrivateIP.
|
||||
func TestCheckAuthorisationFailsWhenTheSubnetVerbIsMissing(t *testing.T) {
|
||||
fake := oci.NewFake()
|
||||
seedPrimary(fake)
|
||||
fake.FailWith["GetSubnet"] = errors.New("NotAuthorizedOrNotFound")
|
||||
a, _ := newTestAllocator(t, fake)
|
||||
|
||||
err := a.CheckAuthorisation(context.Background())
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "use subnets")
|
||||
assert.Contains(t, err.Error(), "SUBNET_ATTACH")
|
||||
}
|
||||
|
||||
func TestCheckAuthorisationFailsWhenTheVNICCannotBeRead(t *testing.T) {
|
||||
fake := oci.NewFake()
|
||||
seedPrimary(fake)
|
||||
fake.FailWith["ListPrivateIPs"] = errors.New("NotAuthorizedOrNotFound")
|
||||
a, _ := newTestAllocator(t, fake)
|
||||
|
||||
err := a.CheckAuthorisation(context.Background())
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "use vnics")
|
||||
}
|
||||
|
||||
// An empty list means the OCID is not a VNIC this credential can see, which is a
|
||||
// different fault from a missing verb and has to read differently.
|
||||
func TestCheckAuthorisationFailsWhenTheVNICHasNoSubnet(t *testing.T) {
|
||||
a, _ := newTestAllocator(t, oci.NewFake())
|
||||
|
||||
err := a.CheckAuthorisation(context.Background())
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "no private IP with a subnet")
|
||||
}
|
||||
|
||||
// captureLogs makes slog's default logger write JSON into a buffer for the
|
||||
// duration of one test, because the gate reads these lines out of a journal.
|
||||
func captureLogs(t *testing.T) *bytes.Buffer {
|
||||
t.Helper()
|
||||
var buf bytes.Buffer
|
||||
saved := slog.Default()
|
||||
slog.SetDefault(slog.New(slog.NewJSONHandler(&buf, &slog.HandlerOptions{Level: slog.LevelDebug})))
|
||||
t.Cleanup(func() { slog.SetDefault(saved) })
|
||||
return &buf
|
||||
}
|
||||
|
||||
// The exact string matters more than usual: validate-topology.sh greps for it,
|
||||
// so renaming it silently turns the gate into one that never fires.
|
||||
func TestReportAuthorisationLogsTheLineTheGateWaitsFor(t *testing.T) {
|
||||
fake := oci.NewFake()
|
||||
seedPrimary(fake)
|
||||
a, _ := newTestAllocator(t, fake)
|
||||
buf := captureLogs(t)
|
||||
|
||||
ocinet.ReportAuthorisation(context.Background(), a, ocinet.PoolWithAuth("instance_principal", "", ""))
|
||||
|
||||
out := buf.String()
|
||||
assert.Contains(t, out, "ocinet credential authorised to allocate")
|
||||
assert.NotContains(t, out, "cannot allocate")
|
||||
assert.Contains(t, out, "ocid1.vnic.oc1..vnic1")
|
||||
}
|
||||
|
||||
func TestReportAuthorisationLogsTheRefusalAndNamesTheVerb(t *testing.T) {
|
||||
fake := oci.NewFake()
|
||||
seedPrimary(fake)
|
||||
fake.FailWith["GetSubnet"] = errors.New("NotAuthorizedOrNotFound")
|
||||
a, _ := newTestAllocator(t, fake)
|
||||
buf := captureLogs(t)
|
||||
|
||||
ocinet.ReportAuthorisation(context.Background(), a, ocinet.PoolWithAuth("instance_principal", "", ""))
|
||||
|
||||
out := buf.String()
|
||||
assert.Contains(t, out, "ocinet credential cannot allocate an external address")
|
||||
assert.Contains(t, out, "use subnets")
|
||||
assert.Contains(t, out, `"level":"ERROR"`)
|
||||
}
|
||||
@@ -34,3 +34,7 @@ func PoolWithAuth(auth, configFile, profile string) external.ExternalPoolConfig
|
||||
OCIConfigProfile: profile,
|
||||
}
|
||||
}
|
||||
|
||||
// ReportAuthorisation logs the line validate-topology.sh gates the whole OCI
|
||||
// suite on, so a test pins the exact text and not just the behaviour.
|
||||
var ReportAuthorisation = reportAuthorisation
|
||||
|
||||
+79
@@ -6,6 +6,7 @@ import (
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/mulgadc/spinifex/spinifex/cloud/oci"
|
||||
"github.com/mulgadc/spinifex/spinifex/kvstore"
|
||||
@@ -20,8 +21,16 @@ type ReconcileResult struct {
|
||||
Stale []string
|
||||
// Skipped is the private IPs left alone because they are not ours.
|
||||
Skipped int
|
||||
// Orphaned is the OCIDs of detached reserved public IPs deleted because
|
||||
// nothing references them and no VNIC could have reported them.
|
||||
Orphaned []string
|
||||
}
|
||||
|
||||
// orphanGrace is how long a detached reserved public IP must have existed
|
||||
// before a reconcile will collect it. CreatePublicIP returns before the address
|
||||
// is attached, so anything younger may be another node's allocation in flight.
|
||||
const orphanGrace = 30 * time.Minute
|
||||
|
||||
// Reconcile squares our record against OCI, in both directions. It is required
|
||||
// on startup rather than optional: Allocate creates OCI objects before writing
|
||||
// the record, so a crash in between leaves a reserved public IP that costs
|
||||
@@ -35,6 +44,9 @@ type ReconcileResult struct {
|
||||
// - A binding naming objects OCI no longer has is a stale record from an
|
||||
// interrupted Release, or from someone deleting the address in the console.
|
||||
// Drop it, so the address is not reported to a customer as theirs.
|
||||
// - A reserved public IP attached to nothing is on no VNIC, so neither
|
||||
// direction above can see it, and it holds a quota slot until deleted.
|
||||
// Collect it once it is old enough to not be an allocation in flight.
|
||||
//
|
||||
// **It never touches an object without DisplayNamePrefix.** The reference host
|
||||
// carries operator-created secondary addresses on the same VNIC, and a
|
||||
@@ -53,6 +65,7 @@ func (a *PoolAllocator) Reconcile(ctx context.Context) (ReconcileResult, error)
|
||||
// the leak this pass exists to find is precisely an OCI object created
|
||||
// before its binding was written.
|
||||
rec, err := a.store.Get(ctx, a.cfg.Pool.Name)
|
||||
recorded := err == nil
|
||||
if err != nil && !errors.Is(err, kvstore.ErrNotFound) {
|
||||
return res, fmt.Errorf("ocinet reconcile: read bindings: %w", err)
|
||||
}
|
||||
@@ -149,5 +162,71 @@ func (a *PoolAllocator) Reconcile(ctx context.Context) (ReconcileResult, error)
|
||||
res.Stale = stale
|
||||
}
|
||||
|
||||
if err := a.collectOrphanedPublicIPs(ctx, rec, recorded, &res); err != nil {
|
||||
return res, err
|
||||
}
|
||||
return res, nil
|
||||
}
|
||||
|
||||
// collectOrphanedPublicIPs deletes reserved public IPs that nothing can reach.
|
||||
//
|
||||
// Four conditions have to hold together, and each one rules out a class of
|
||||
// address that must survive:
|
||||
//
|
||||
// - Our display-name prefix, so an operator's own reserved address is never
|
||||
// a candidate.
|
||||
// - No private IP, so no guest anywhere is receiving traffic on it.
|
||||
// - Named by no binding in the pool record, because an allocated but
|
||||
// unassociated Elastic IP is detached too and is the customer's to keep.
|
||||
// - Older than orphanGrace, because a detached address is also what a
|
||||
// CreatePublicIP on another node looks like until it finishes attaching.
|
||||
//
|
||||
// The binding test is only as good as the record, so a pool with no record at
|
||||
// all is not swept. That is a pool which has never allocated and has nothing to
|
||||
// collect, or a store that cannot be read — and those are indistinguishable
|
||||
// from here, while one of them means every held address looks like an orphan.
|
||||
func (a *PoolAllocator) collectOrphanedPublicIPs(ctx context.Context, rec Record, recorded bool, res *ReconcileResult) error {
|
||||
if !recorded {
|
||||
slog.DebugContext(ctx, "ocinet reconcile skipped the orphan sweep: no binding record to judge against",
|
||||
"pool", a.cfg.Pool.Name)
|
||||
return nil
|
||||
}
|
||||
|
||||
reserved, err := a.client.ListReservedPublicIPs(ctx, a.cfg.CompartmentID)
|
||||
if err != nil {
|
||||
// A credential without `read public-ips` in the compartment can still
|
||||
// allocate, so refusing the whole pass here would take a working node
|
||||
// down over a cleanup it cannot perform.
|
||||
slog.WarnContext(ctx, "ocinet reconcile could not list reserved public ips; orphans will accumulate",
|
||||
"pool", a.cfg.Pool.Name, "compartment", a.cfg.CompartmentID, "error", err)
|
||||
return nil
|
||||
}
|
||||
|
||||
bound := make(map[string]struct{}, len(rec.Bindings))
|
||||
for _, b := range rec.Bindings {
|
||||
bound[b.PublicIPID] = struct{}{}
|
||||
}
|
||||
|
||||
cutoff := a.now().Add(-orphanGrace)
|
||||
for _, p := range reserved {
|
||||
if !strings.HasPrefix(p.DisplayName, DisplayNamePrefix) || p.PrivateIPID != "" {
|
||||
continue
|
||||
}
|
||||
if _, ok := bound[p.ID]; ok {
|
||||
continue
|
||||
}
|
||||
// A zero TimeCreated means the age is unknown, and an unknown age is
|
||||
// not evidence of an orphan.
|
||||
if p.TimeCreated.IsZero() || p.TimeCreated.After(cutoff) {
|
||||
continue
|
||||
}
|
||||
if err := a.client.DeletePublicIP(ctx, p.ID); err != nil && !errors.Is(err, oci.ErrNotFound) {
|
||||
return fmt.Errorf("ocinet reconcile: collect orphaned public ip %s: %w", p.ID, err)
|
||||
}
|
||||
slog.WarnContext(ctx, "ocinet reconcile collected an orphaned reserved public ip",
|
||||
"pool", a.cfg.Pool.Name, "public_ip_id", p.ID, "public_ip", p.Address.String(),
|
||||
"display_name", p.DisplayName, "created", p.TimeCreated)
|
||||
res.Orphaned = append(res.Orphaned, p.ID)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
+157
@@ -227,3 +227,160 @@ func TestReconcileStillDropsItsOwnStaleBinding(t *testing.T) {
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, []string{ip.String()}, res.Stale)
|
||||
}
|
||||
|
||||
// orphanAllocator runs with a clock advanced past the grace period, so a
|
||||
// seeded address created "now" reads as old without the test waiting.
|
||||
func orphanAllocator(t *testing.T, fake *oci.Fake) (*ocinet.PoolAllocator, *ocinet.MemStore) {
|
||||
t.Helper()
|
||||
store := ocinet.NewMemStore()
|
||||
a, err := ocinet.New(fake, store, ocinet.Config{
|
||||
Pool: external.ExternalPoolConfig{Name: "oci-wan", Source: external.SourceOCI},
|
||||
VNICID: "ocid1.vnic.oc1..vnic1",
|
||||
CompartmentID: "ocid1.compartment.oc1..comp1",
|
||||
Schedule: []time.Duration{time.Millisecond},
|
||||
Budget: time.Second,
|
||||
Sleep: func(context.Context, time.Duration) error { return nil },
|
||||
Now: func() time.Time { return time.Now().Add(2 * time.Hour) },
|
||||
})
|
||||
require.NoError(t, err)
|
||||
// The sweep needs a readable record to judge bindings against, and a pool
|
||||
// that has ever allocated has one. Creating it empty is that pool.
|
||||
require.NoError(t, store.Mutate(context.Background(), "oci-wan",
|
||||
func(*ocinet.Record) (bool, error) { return true, nil }))
|
||||
return a, store
|
||||
}
|
||||
|
||||
// seedOrphan is a reserved public IP attached to nothing, which is what a
|
||||
// destroyed node or a failed rollback leaves behind.
|
||||
func seedOrphan(fake *oci.Fake, id, name string, created time.Time) {
|
||||
fake.SeedPublicIP(oci.PublicIP{
|
||||
ID: id,
|
||||
Address: netip.MustParseAddr("203.0.113.200"),
|
||||
DisplayName: name,
|
||||
Lifetime: oci.LifetimeReserved,
|
||||
LifecycleState: oci.LifecycleStateAvailable,
|
||||
TimeCreated: created,
|
||||
})
|
||||
}
|
||||
|
||||
// The leak that exhausted the tenancy. A detached reserved public IP is on no
|
||||
// VNIC, so walking the VNIC cannot see it however many passes run.
|
||||
func TestReconcileCollectsADetachedReservedPublicIPNoVNICCanReport(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
seedOrphan(fake, "ocid1.publicip.oc1..orphan", ocinet.DisplayNamePrefix+"eipalloc-gone", time.Now())
|
||||
a, _ := orphanAllocator(t, fake)
|
||||
|
||||
res, err := a.Reconcile(ctx)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Equal(t, []string{"ocid1.publicip.oc1..orphan"}, res.Orphaned)
|
||||
assert.Empty(t, fake.PublicIPs(), "the orphan still holds a tenancy quota slot")
|
||||
}
|
||||
|
||||
func TestReconcileLeavesAReservedPublicIPThatIsNotOursAlone(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
seedOrphan(fake, "ocid1.publicip.oc1..theirs", "IP for NAT gateway", time.Now())
|
||||
a, _ := orphanAllocator(t, fake)
|
||||
|
||||
res, err := a.Reconcile(ctx)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Empty(t, res.Orphaned)
|
||||
assert.Len(t, fake.PublicIPs(), 1, "an address without our prefix is never a candidate")
|
||||
}
|
||||
|
||||
// An allocated but unassociated Elastic IP is detached too, and is the
|
||||
// customer's to keep. The binding is what tells the two apart.
|
||||
func TestReconcileLeavesADetachedButBoundElasticIPAlone(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
seedOrphan(fake, "ocid1.publicip.oc1..heldeip", ocinet.DisplayNamePrefix+"eipalloc-held", time.Now())
|
||||
a, store := orphanAllocator(t, fake)
|
||||
require.NoError(t, store.Mutate(ctx, "oci-wan", func(r *ocinet.Record) (bool, error) {
|
||||
r.Bindings["203.0.113.200"] = ocinet.Binding{
|
||||
PublicIPID: "ocid1.publicip.oc1..heldeip",
|
||||
Purpose: "eip",
|
||||
AllocationID: "eipalloc-held",
|
||||
}
|
||||
return true, nil
|
||||
}))
|
||||
|
||||
res, err := a.Reconcile(ctx)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Empty(t, res.Orphaned)
|
||||
assert.Len(t, fake.PublicIPs(), 1, "a held EIP must survive a reconcile on any node")
|
||||
}
|
||||
|
||||
// CreatePublicIP returns before the address attaches, so a young detached one
|
||||
// may be another node's allocation mid-flight.
|
||||
func TestReconcileLeavesAYoungDetachedAddressAlone(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
seedOrphan(fake, "ocid1.publicip.oc1..inflight", ocinet.DisplayNamePrefix+"eni-new", time.Now())
|
||||
a, store := newTestAllocator(t, fake)
|
||||
require.NoError(t, store.Mutate(ctx, "oci-wan",
|
||||
func(*ocinet.Record) (bool, error) { return true, nil }))
|
||||
|
||||
res, err := a.Reconcile(ctx)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Empty(t, res.Orphaned)
|
||||
assert.Len(t, fake.PublicIPs(), 1, "an in-flight allocation must not be collected")
|
||||
}
|
||||
|
||||
// An unknown age is not evidence of an orphan.
|
||||
func TestReconcileLeavesAnAddressWithNoCreationTimeAlone(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
seedOrphan(fake, "ocid1.publicip.oc1..undated", ocinet.DisplayNamePrefix+"eni-undated", time.Time{})
|
||||
a, _ := orphanAllocator(t, fake)
|
||||
|
||||
res, err := a.Reconcile(ctx)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Empty(t, res.Orphaned)
|
||||
assert.Len(t, fake.PublicIPs(), 1)
|
||||
}
|
||||
|
||||
// A credential that cannot list the compartment can still allocate, so the
|
||||
// pass warns and carries on rather than failing a working node.
|
||||
func TestReconcileSurvivesACredentialThatCannotListReservedPublicIPs(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
seedOrphan(fake, "ocid1.publicip.oc1..orphan", ocinet.DisplayNamePrefix+"eni-x", time.Now())
|
||||
fake.FailWith["ListReservedPublicIPs"] = oci.ErrNotFound
|
||||
a, _ := orphanAllocator(t, fake)
|
||||
|
||||
res, err := a.Reconcile(ctx)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Empty(t, res.Orphaned)
|
||||
}
|
||||
|
||||
// A store that cannot be read looks exactly like a pool that never allocated,
|
||||
// and one of those means every address a customer holds appears unbound.
|
||||
func TestReconcileDoesNotSweepWhenThereIsNoBindingRecord(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
fake := oci.NewFake()
|
||||
seedOrphan(fake, "ocid1.publicip.oc1..orphan", ocinet.DisplayNamePrefix+"eipalloc-x", time.Now())
|
||||
// Deliberately no record: orphanAllocator's empty one is what makes the
|
||||
// sweep safe, so its absence must disable it.
|
||||
store := ocinet.NewMemStore()
|
||||
a, err := ocinet.New(fake, store, ocinet.Config{
|
||||
Pool: external.ExternalPoolConfig{Name: "oci-wan", Source: external.SourceOCI},
|
||||
VNICID: "ocid1.vnic.oc1..vnic1",
|
||||
CompartmentID: "ocid1.compartment.oc1..comp1",
|
||||
Sleep: func(context.Context, time.Duration) error { return nil },
|
||||
Now: func() time.Time { return time.Now().Add(2 * time.Hour) },
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
||||
res, err := a.Reconcile(ctx)
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Empty(t, res.Orphaned)
|
||||
assert.Len(t, fake.PublicIPs(), 1)
|
||||
}
|
||||
|
||||
+10
-1
@@ -48,7 +48,7 @@ func FromPoolConfig(ctx context.Context, js jetstream.JetStream, pool external.E
|
||||
compartmentID = inst.CompartmentID
|
||||
}
|
||||
|
||||
return New(client, NewKVStore(js), Config{
|
||||
alloc, err := New(client, NewKVStore(js), Config{
|
||||
Pool: pool,
|
||||
VNICID: vnicID,
|
||||
CompartmentID: compartmentID,
|
||||
@@ -65,6 +65,15 @@ func FromPoolConfig(ctx context.Context, js jetstream.JetStream, pool external.E
|
||||
GatewayPortLocal(ctx, topology.GatewayChassisRedirectPort(vpcID))
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Before this, resolving the VNIC was the last thing logged, and that comes
|
||||
// from the metadata service and needs no IAM policy at all. So an
|
||||
// unauthorised node looked ready and refused the first launch instead.
|
||||
reportAuthorisation(ctx, alloc, pool)
|
||||
return alloc, nil
|
||||
}
|
||||
|
||||
// newClient authenticates the way the pool asks. Instance principal takes the
|
||||
|
||||
Reference in New Issue
Block a user