mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-16 01:22:00 +00:00
last doc tweaks
This commit is contained in:
Julian Sommer
@@ -29,14 +29,6 @@ Spinifex supports multi-tenant account isolation. Each account gets its own IAM
|
||||
|
||||
## Instructions
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Ensure the AWS profile is set:
|
||||
|
||||
```bash
|
||||
export AWS_PROFILE=spinifex
|
||||
```
|
||||
|
||||
## Create Account
|
||||
|
||||
```bash
|
||||
@@ -44,12 +36,6 @@ spx admin account create --name myteam
|
||||
export AWS_PROFILE=spinifex-myteam
|
||||
```
|
||||
|
||||
## Verify
|
||||
|
||||
```bash
|
||||
aws sts get-caller-identity
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
## Credentials Not Working
|
||||
|
||||
@@ -69,14 +69,3 @@ aws s3 sync s3://local-bucket/ s3://cloud-bucket/ \
|
||||
```
|
||||
|
||||
Only changed or missing files will be transferred on subsequent runs.
|
||||
|
||||
## Permission Errors During Sync
|
||||
|
||||
Verify your AWS credentials are configured for both the source and destination:
|
||||
|
||||
```bash
|
||||
aws sts get-caller-identity --profile spinifex
|
||||
aws sts get-caller-identity --profile default
|
||||
```
|
||||
|
||||
Ensure the IAM user has `s3:GetObject`, `s3:PutObject`, and `s3:ListBucket` permissions on both buckets.
|
||||
|
||||
@@ -53,7 +53,6 @@ When you create an account with `spx admin account create`, Spinifex bootstraps
|
||||
|
||||
```bash
|
||||
export AWS_PROFILE=spinifex-myteam
|
||||
aws sts get-caller-identity
|
||||
```
|
||||
|
||||
## Users
|
||||
@@ -68,13 +67,13 @@ The response includes the user's ARN, unique ID, and creation date:
|
||||
|
||||
```json
|
||||
{
|
||||
"User": {
|
||||
"UserName": "alice",
|
||||
"UserId": "AIDA1A2B3C4D5E6F7890",
|
||||
"Arn": "arn:aws:iam::000000000001:user/alice",
|
||||
"Path": "/",
|
||||
"CreateDate": "2026-03-24T10:00:00Z"
|
||||
}
|
||||
"User": {
|
||||
"UserName": "alice",
|
||||
"UserId": "AIDA1A2B3C4D5E6F7890",
|
||||
"Arn": "arn:aws:iam::000000000001:user/alice",
|
||||
"Path": "/",
|
||||
"CreateDate": "2026-03-24T10:00:00Z"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
@@ -120,7 +119,6 @@ aws iam detach-user-policy --user-name alice \
|
||||
aws iam delete-user --user-name alice
|
||||
```
|
||||
|
||||
|
||||
## Access Keys
|
||||
|
||||
Access keys are how users authenticate with the AWS CLI. Each user can have up to **2 access keys** at a time, allowing key rotation without downtime.
|
||||
@@ -133,13 +131,13 @@ aws iam create-access-key --user-name alice
|
||||
|
||||
```json
|
||||
{
|
||||
"AccessKey": {
|
||||
"UserName": "alice",
|
||||
"AccessKeyId": "AKIA1A2B3C4D5E6F7890ABCD",
|
||||
"Status": "Active",
|
||||
"SecretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
|
||||
"CreateDate": "2026-03-24T10:05:00Z"
|
||||
}
|
||||
"AccessKey": {
|
||||
"UserName": "alice",
|
||||
"AccessKeyId": "AKIA1A2B3C4D5E6F7890ABCD",
|
||||
"Status": "Active",
|
||||
"SecretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
|
||||
"CreateDate": "2026-03-24T10:05:00Z"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
@@ -153,7 +151,6 @@ Add the credentials to `~/.aws/credentials` under a new profile and set `AWS_PRO
|
||||
aws iam list-access-keys --user-name alice
|
||||
```
|
||||
|
||||
|
||||
### Rotate Access Keys
|
||||
|
||||
Create a second key, update your configuration, then delete the old one:
|
||||
@@ -189,7 +186,6 @@ aws iam update-access-key --user-name alice \
|
||||
--status Active
|
||||
```
|
||||
|
||||
|
||||
## Policies
|
||||
|
||||
Policies are JSON documents that define what actions a user can perform on which resources.
|
||||
@@ -198,14 +194,14 @@ Policies are JSON documents that define what actions a user can perform on which
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": ["ec2:RunInstances", "ec2:DescribeInstances"],
|
||||
"Resource": "*"
|
||||
}
|
||||
]
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": ["ec2:RunInstances", "ec2:DescribeInstances"],
|
||||
"Resource": "*"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
@@ -249,32 +245,31 @@ aws iam create-policy \
|
||||
|
||||
```json
|
||||
{
|
||||
"Policy": {
|
||||
"PolicyName": "EC2ReadOnly",
|
||||
"PolicyId": "ANPA1A2B3C4D5E6F7890",
|
||||
"Arn": "arn:aws:iam::000000000001:policy/EC2ReadOnly",
|
||||
"Path": "/",
|
||||
"DefaultVersionId": "v1",
|
||||
"CreateDate": "2026-03-24T10:10:00Z"
|
||||
}
|
||||
"Policy": {
|
||||
"PolicyName": "EC2ReadOnly",
|
||||
"PolicyId": "ANPA1A2B3C4D5E6F7890",
|
||||
"Arn": "arn:aws:iam::000000000001:policy/EC2ReadOnly",
|
||||
"Path": "/",
|
||||
"DefaultVersionId": "v1",
|
||||
"CreateDate": "2026-03-24T10:10:00Z"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
### Common Policy Examples
|
||||
|
||||
**Full administrator access:**
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": "*",
|
||||
"Resource": "*"
|
||||
}
|
||||
]
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": "*",
|
||||
"Resource": "*"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
@@ -282,19 +277,19 @@ aws iam create-policy \
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"s3:GetObject",
|
||||
"s3:PutObject",
|
||||
"s3:ListBucket",
|
||||
"s3:DeleteObject"
|
||||
],
|
||||
"Resource": "*"
|
||||
}
|
||||
]
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"s3:GetObject",
|
||||
"s3:PutObject",
|
||||
"s3:ListBucket",
|
||||
"s3:DeleteObject"
|
||||
],
|
||||
"Resource": "*"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
@@ -302,21 +297,21 @@ aws iam create-policy \
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"ec2:RunInstances",
|
||||
"ec2:StartInstances",
|
||||
"ec2:StopInstances",
|
||||
"ec2:TerminateInstances",
|
||||
"ec2:DescribeInstances",
|
||||
"ec2:DescribeImages"
|
||||
],
|
||||
"Resource": "*"
|
||||
}
|
||||
]
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"ec2:RunInstances",
|
||||
"ec2:StartInstances",
|
||||
"ec2:StopInstances",
|
||||
"ec2:TerminateInstances",
|
||||
"ec2:DescribeInstances",
|
||||
"ec2:DescribeImages"
|
||||
],
|
||||
"Resource": "*"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
@@ -324,14 +319,14 @@ aws iam create-policy \
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Deny",
|
||||
"Action": "ec2:TerminateInstances",
|
||||
"Resource": "*"
|
||||
}
|
||||
]
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Deny",
|
||||
"Action": "ec2:TerminateInstances",
|
||||
"Resource": "*"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
@@ -352,7 +347,6 @@ aws iam get-policy-version \
|
||||
--version-id v1
|
||||
```
|
||||
|
||||
|
||||
### List Policies
|
||||
|
||||
```bash
|
||||
@@ -368,7 +362,6 @@ aws iam delete-policy \
|
||||
--policy-arn arn:aws:iam::000000000001:policy/EC2ReadOnly
|
||||
```
|
||||
|
||||
|
||||
## Attaching Policies
|
||||
|
||||
Policies have no effect until attached to a user.
|
||||
@@ -380,7 +373,6 @@ aws iam attach-user-policy --user-name alice \
|
||||
--policy-arn arn:aws:iam::000000000001:policy/EC2ReadOnly
|
||||
```
|
||||
|
||||
|
||||
### List a User's Policies
|
||||
|
||||
```bash
|
||||
@@ -453,24 +445,24 @@ AWS_PROFILE=spinifex-carol aws ec2 terminate-instances --instance-ids i-123
|
||||
|
||||
## Command Reference
|
||||
|
||||
| Command | Key Parameters | Notes |
|
||||
|---------|---------------|-------|
|
||||
| `create-user` | `--user-name`, `--path` | Path default: `/` |
|
||||
| `get-user` | `--user-name` | |
|
||||
| `list-users` | `--path-prefix` | |
|
||||
| `delete-user` | `--user-name` | Must have no keys or policies |
|
||||
| `create-access-key` | `--user-name` | Max 2 per user; secret shown once |
|
||||
| `list-access-keys` | `--user-name` | |
|
||||
| `delete-access-key` | `--user-name`, `--access-key-id` | |
|
||||
| `update-access-key` | `--user-name`, `--access-key-id`, `--status` | `Active` or `Inactive` |
|
||||
| `create-policy` | `--policy-name`, `--policy-document` | Max 6144 bytes |
|
||||
| `get-policy` | `--policy-arn` | Metadata only |
|
||||
| `get-policy-version` | `--policy-arn`, `--version-id` | Use `v1`; includes document |
|
||||
| `list-policies` | | |
|
||||
| `delete-policy` | `--policy-arn` | Must not be attached |
|
||||
| `attach-user-policy` | `--user-name`, `--policy-arn` | Idempotent |
|
||||
| `detach-user-policy` | `--user-name`, `--policy-arn` | |
|
||||
| `list-attached-user-policies` | `--user-name` | |
|
||||
| Command | Key Parameters | Notes |
|
||||
| ----------------------------- | -------------------------------------------- | --------------------------------- |
|
||||
| `create-user` | `--user-name`, `--path` | Path default: `/` |
|
||||
| `get-user` | `--user-name` | |
|
||||
| `list-users` | `--path-prefix` | |
|
||||
| `delete-user` | `--user-name` | Must have no keys or policies |
|
||||
| `create-access-key` | `--user-name` | Max 2 per user; secret shown once |
|
||||
| `list-access-keys` | `--user-name` | |
|
||||
| `delete-access-key` | `--user-name`, `--access-key-id` | |
|
||||
| `update-access-key` | `--user-name`, `--access-key-id`, `--status` | `Active` or `Inactive` |
|
||||
| `create-policy` | `--policy-name`, `--policy-document` | Max 6144 bytes |
|
||||
| `get-policy` | `--policy-arn` | Metadata only |
|
||||
| `get-policy-version` | `--policy-arn`, `--version-id` | Use `v1`; includes document |
|
||||
| `list-policies` | | |
|
||||
| `delete-policy` | `--policy-arn` | Must not be attached |
|
||||
| `attach-user-policy` | `--user-name`, `--policy-arn` | Idempotent |
|
||||
| `detach-user-policy` | `--user-name`, `--policy-arn` | |
|
||||
| `list-attached-user-policies` | `--user-name` | |
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
|
||||
@@ -37,14 +37,6 @@ Spinifex provides drop-in compatibility with AWS APIs, making it possible to mig
|
||||
|
||||
## Instructions
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Ensure the AWS profile is set:
|
||||
|
||||
```bash
|
||||
export AWS_PROFILE=spinifex
|
||||
```
|
||||
|
||||
## Configure AWS CLI
|
||||
|
||||
```bash
|
||||
|
||||
@@ -33,14 +33,6 @@ All services in the Spinifex platform are managed through this single binary.
|
||||
|
||||
## Instructions
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Ensure the AWS profile is set:
|
||||
|
||||
```bash
|
||||
export AWS_PROFILE=spinifex
|
||||
```
|
||||
|
||||
## Account Management
|
||||
|
||||
```bash
|
||||
|
||||
@@ -1,8 +1,7 @@
|
||||
---
|
||||
title: "Terraform: Bastion with Private Subnet"
|
||||
title: "Bastion with Private Subnet"
|
||||
description: "Deploy a bastion host to access an isolated private compute instance with no internet connectivity."
|
||||
category: "Terraform Workbooks"
|
||||
badge: "Workbook"
|
||||
tags:
|
||||
- terraform
|
||||
- bastion
|
||||
|
||||
@@ -1,8 +1,7 @@
|
||||
---
|
||||
title: "Terraform: Nginx Web Server"
|
||||
title: "Nginx Web Server"
|
||||
description: "Deploy a VPC with a public subnet and an EC2 instance running Nginx using Terraform on Spinifex."
|
||||
category: "Terraform Workbooks"
|
||||
badge: "Workbook"
|
||||
tags:
|
||||
- terraform
|
||||
- nginx
|
||||
|
||||
@@ -1,8 +1,7 @@
|
||||
---
|
||||
title: "Terraform: S3-Backed Web App"
|
||||
title: "S3-Backed Web App"
|
||||
description: "Deploy a Flask file-sharing web application backed by S3 (Predastore) using Terraform on Spinifex."
|
||||
category: "Terraform Workbooks"
|
||||
badge: "Workbook"
|
||||
tags:
|
||||
- terraform
|
||||
- s3
|
||||
|
||||
@@ -23,26 +23,18 @@ Instances can operate in two modes: **private** (overlay-only, no WAN access) or
|
||||
|
||||
## Instructions
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Ensure the AWS profile is set:
|
||||
|
||||
```bash
|
||||
export AWS_PROFILE=spinifex
|
||||
```
|
||||
|
||||
## How It Works
|
||||
|
||||
Spinifex maps AWS VPC concepts directly to OVN constructs:
|
||||
|
||||
| AWS Concept | OVN Construct | What It Does |
|
||||
|-------------|---------------|--------------|
|
||||
| VPC | Logical Router | Isolates tenant networks, routes between subnets |
|
||||
| Subnet | Logical Switch + DHCP | L2 broadcast domain with automatic IP assignment |
|
||||
| ENI | Logical Switch Port | Per-instance network interface with MAC/IP binding |
|
||||
| Internet Gateway | External Switch + NAT | Connects VPC router to physical WAN |
|
||||
| Security Group | Port Group + ACLs | Stateful firewall rules enforced in OVS datapath |
|
||||
| Elastic IP | `dnat_and_snat` NAT rule | Static 1:1 NAT between public and private IP |
|
||||
| AWS Concept | OVN Construct | What It Does |
|
||||
| ---------------- | ------------------------ | -------------------------------------------------- |
|
||||
| VPC | Logical Router | Isolates tenant networks, routes between subnets |
|
||||
| Subnet | Logical Switch + DHCP | L2 broadcast domain with automatic IP assignment |
|
||||
| ENI | Logical Switch Port | Per-instance network interface with MAC/IP binding |
|
||||
| Internet Gateway | External Switch + NAT | Connects VPC router to physical WAN |
|
||||
| Security Group | Port Group + ACLs | Stateful firewall rules enforced in OVS datapath |
|
||||
| Elastic IP | `dnat_and_snat` NAT rule | Static 1:1 NAT between public and private IP |
|
||||
|
||||
## Network Path
|
||||
|
||||
@@ -120,14 +112,14 @@ Inbound: 203.0.113.10 ──→ DNAT ──→ arrives at 10.0.1.5
|
||||
|
||||
## Comparison
|
||||
|
||||
| | Private Subnet | Public Subnet |
|
||||
|---|---|---|
|
||||
| Private IP | Yes | Yes |
|
||||
| Public IP | No | Auto-assigned from pool |
|
||||
| Outbound internet | Only if VPC has IGW (shared SNAT) | Yes (own public IP via SNAT) |
|
||||
| Inbound from WAN | No | Yes (via 1:1 NAT to public IP) |
|
||||
| Instance sees public IP? | N/A | No — only sees private IP |
|
||||
| Elastic IP support | Only if explicitly associated | Yes |
|
||||
| | Private Subnet | Public Subnet |
|
||||
| ------------------------ | --------------------------------- | ------------------------------ |
|
||||
| Private IP | Yes | Yes |
|
||||
| Public IP | No | Auto-assigned from pool |
|
||||
| Outbound internet | Only if VPC has IGW (shared SNAT) | Yes (own public IP via SNAT) |
|
||||
| Inbound from WAN | No | Yes (via 1:1 NAT to public IP) |
|
||||
| Instance sees public IP? | N/A | No — only sees private IP |
|
||||
| Elastic IP support | Only if explicitly associated | Yes |
|
||||
|
||||
## External Connectivity Modes
|
||||
|
||||
@@ -221,14 +213,14 @@ is identical:
|
||||
|
||||
### Choosing Static vs DHCP
|
||||
|
||||
| | Static Range | DHCP Source |
|
||||
|---|---|---|
|
||||
| **Public IPs from** | Admin-defined `range_start`..`range_end` | Router's DHCP server |
|
||||
| **IP predictability** | You know the exact range | Router assigns whatever is available |
|
||||
| **Setup effort** | Must reserve range, shrink router DHCP scope | Just set `source = "dhcp"` |
|
||||
| **Dependency** | None | Requires `dhclient` on host, working router DHCP |
|
||||
| **Best for** | Datacenters, ISP blocks, production | Homelabs, dev environments, shared networks |
|
||||
| **Capacity** | Exact: `range_end - range_start` IPs | Limited by router's DHCP pool size |
|
||||
| | Static Range | DHCP Source |
|
||||
| --------------------- | -------------------------------------------- | ------------------------------------------------ |
|
||||
| **Public IPs from** | Admin-defined `range_start`..`range_end` | Router's DHCP server |
|
||||
| **IP predictability** | You know the exact range | Router assigns whatever is available |
|
||||
| **Setup effort** | Must reserve range, shrink router DHCP scope | Just set `source = "dhcp"` |
|
||||
| **Dependency** | None | Requires `dhclient` on host, working router DHCP |
|
||||
| **Best for** | Datacenters, ISP blocks, production | Homelabs, dev environments, shared networks |
|
||||
| **Capacity** | Exact: `range_end - range_start` IPs | Limited by router's DHCP pool size |
|
||||
|
||||
Both support the same AWS features: public subnets, Elastic IPs, security groups,
|
||||
DescribeInstances showing public IPs.
|
||||
@@ -265,16 +257,16 @@ communicate within their VPC.
|
||||
|
||||
## Mode Comparison
|
||||
|
||||
| Capability | `pool` (static) | `pool` (dhcp) | `nat` | Disabled |
|
||||
|------------|-----------------|---------------|-------|----------|
|
||||
| Outbound internet | Yes | Yes | Yes | No |
|
||||
| Inbound from WAN | Yes (1:1 NAT) | Yes (1:1 NAT) | No | No |
|
||||
| Public subnets | Yes | Yes | No | No |
|
||||
| Auto-assign public IPs | Yes | Yes | No | No |
|
||||
| Elastic IPs | Yes | Yes | No | No |
|
||||
| DescribeInstances shows public IP | Yes | Yes | No | No |
|
||||
| Admin must reserve IP range | Yes | No | No | No |
|
||||
| Needs router DHCP | No | Yes | Optional | No |
|
||||
| Capability | `pool` (static) | `pool` (dhcp) | `nat` | Disabled |
|
||||
| --------------------------------- | --------------- | ------------- | -------- | -------- |
|
||||
| Outbound internet | Yes | Yes | Yes | No |
|
||||
| Inbound from WAN | Yes (1:1 NAT) | Yes (1:1 NAT) | No | No |
|
||||
| Public subnets | Yes | Yes | No | No |
|
||||
| Auto-assign public IPs | Yes | Yes | No | No |
|
||||
| Elastic IPs | Yes | Yes | No | No |
|
||||
| DescribeInstances shows public IP | Yes | Yes | No | No |
|
||||
| Admin must reserve IP range | Yes | No | No | No |
|
||||
| Needs router DHCP | No | Yes | Optional | No |
|
||||
|
||||
If you start with `nat` and later need public subnets, switch to `pool` and
|
||||
define a range (or use `source = "dhcp"`) — no data migration needed.
|
||||
@@ -294,10 +286,10 @@ the best approach:
|
||||
|
||||
Every Spinifex node has two OVS bridges:
|
||||
|
||||
| Bridge | Purpose | Ports |
|
||||
|--------|---------|-------|
|
||||
| `br-int` | VM overlay traffic (Geneve tunnels) | VM TAP devices, tunnel ports |
|
||||
| `br-wan` | WAN uplink for public subnet traffic | macvlan on WAN NIC |
|
||||
| Bridge | Purpose | Ports |
|
||||
| -------- | ------------------------------------ | ---------------------------- |
|
||||
| `br-int` | VM overlay traffic (Geneve tunnels) | VM TAP devices, tunnel ports |
|
||||
| `br-wan` | WAN uplink for public subnet traffic | macvlan on WAN NIC |
|
||||
|
||||
`br-int` is always created by `setup-ovn.sh`. `br-wan` is created only when
|
||||
a WAN bridge is configured (auto-detected or via `--wan-bridge`, required for public subnets / external connectivity).
|
||||
@@ -330,6 +322,7 @@ used for SSH or management. Datacenter servers with separate management +
|
||||
public NICs. Homelab hosts with 2+ NICs.
|
||||
|
||||
**Benefits over macvlan:**
|
||||
|
||||
- **Distributed NAT**: DNAT processed on the VM's own chassis (no hairpin)
|
||||
- **No MAC workarounds**: OVS sees all frames, any MAC
|
||||
- **Host can reach VMs**: No macvlan parent↔child isolation
|
||||
@@ -362,6 +355,7 @@ WAN NIC (host IP — unchanged)
|
||||
edge deployments. Adding the NIC directly to OVS would break host connectivity.
|
||||
|
||||
**Limitations:**
|
||||
|
||||
- Centralized NAT only (macvlan filters unicast not matching its MAC)
|
||||
- Host cannot reach VMs at their public IPs (macvlan isolation)
|
||||
- vpcd must align macvlan MAC with OVN router MAC on startup
|
||||
@@ -373,23 +367,23 @@ sudo setup-ovn.sh --macvlan --wan-iface=eth0
|
||||
|
||||
### Mode Comparison
|
||||
|
||||
| Capability | Direct bridge | Macvlan |
|
||||
|------------|--------------|---------|
|
||||
| NAT mode | Distributed (local DNAT) | Centralized (gateway hairpin) |
|
||||
| Multi-node external perf | Optimal | Hairpin through gateway chassis |
|
||||
| Host → VM public IP | Works | Blocked (kernel isolation) |
|
||||
| MAC alignment needed | No | Yes (fragile startup step) |
|
||||
| Gratuitous ARPs needed | No | Yes (`nat-addresses=router`) |
|
||||
| Works on single-NIC | No (breaks SSH) | Yes |
|
||||
| DHCP lease stability | Clean | Fragile (MAC changes post-lease) |
|
||||
| Capability | Direct bridge | Macvlan |
|
||||
| ------------------------ | ------------------------ | -------------------------------- |
|
||||
| NAT mode | Distributed (local DNAT) | Centralized (gateway hairpin) |
|
||||
| Multi-node external perf | Optimal | Hairpin through gateway chassis |
|
||||
| Host → VM public IP | Works | Blocked (kernel isolation) |
|
||||
| MAC alignment needed | No | Yes (fragile startup step) |
|
||||
| Gratuitous ARPs needed | No | Yes (`nat-addresses=router`) |
|
||||
| Works on single-NIC | No (breaks SSH) | Yes |
|
||||
| DHCP lease stability | Clean | Fragile (MAC changes post-lease) |
|
||||
|
||||
### How setup-ovn.sh Decides
|
||||
|
||||
| Flags | Result |
|
||||
|-------|--------|
|
||||
| `--wan-bridge=br-wan --wan-iface=eth1` | Direct bridge: NIC added to br-wan |
|
||||
| `--macvlan --wan-iface=eth0` | Macvlan: sub-interface created, added to br-wan |
|
||||
| (no WAN bridge) | Only br-int created, no WAN connectivity |
|
||||
| Flags | Result |
|
||||
| -------------------------------------- | ----------------------------------------------- |
|
||||
| `--wan-bridge=br-wan --wan-iface=eth1` | Direct bridge: NIC added to br-wan |
|
||||
| `--macvlan --wan-iface=eth0` | Macvlan: sub-interface created, added to br-wan |
|
||||
| (no WAN bridge) | Only br-int created, no WAN connectivity |
|
||||
|
||||
### Setup (Macvlan Mode)
|
||||
|
||||
@@ -410,12 +404,12 @@ macvlan in `bridge` mode creates a virtual interface that shares the parent NIC'
|
||||
physical wire but has its own MAC address. The Linux kernel blocks direct L2
|
||||
frames between a parent interface and its macvlan children.
|
||||
|
||||
| Path | Works? | Why |
|
||||
|------|--------|-----|
|
||||
| VM → internet | Yes | SNAT through OVN router → macvlan → WAN NIC → WAN |
|
||||
| LAN device → VM public IP | Yes | LAN → WAN NIC → macvlan → br-wan → OVN |
|
||||
| Host → VM public IP | No | macvlan isolation (kernel blocks parent↔child) |
|
||||
| Host → VM private IP | Yes | Overlay via br-int (unrelated to br-wan) |
|
||||
| Path | Works? | Why |
|
||||
| ------------------------- | ------ | ------------------------------------------------- |
|
||||
| VM → internet | Yes | SNAT through OVN router → macvlan → WAN NIC → WAN |
|
||||
| LAN device → VM public IP | Yes | LAN → WAN NIC → macvlan → br-wan → OVN |
|
||||
| Host → VM public IP | No | macvlan isolation (kernel blocks parent↔child) |
|
||||
| Host → VM private IP | Yes | Overlay via br-int (unrelated to br-wan) |
|
||||
|
||||
## Per-Node Configuration
|
||||
|
||||
@@ -504,11 +498,11 @@ spinifex.toml
|
||||
external_mode = "pool" # "pool", "nat", or "" (disabled)
|
||||
```
|
||||
|
||||
| Value | Behavior |
|
||||
|-------|----------|
|
||||
| `"pool"` | Full public networking — public subnets, auto-assign, Elastic IPs |
|
||||
| `"nat"` | Outbound-only SNAT — all VMs share one external IP |
|
||||
| `""` / omitted | Overlay-only — no external connectivity |
|
||||
| Value | Behavior |
|
||||
| -------------- | ----------------------------------------------------------------- |
|
||||
| `"pool"` | Full public networking — public subnets, auto-assign, Elastic IPs |
|
||||
| `"nat"` | Outbound-only SNAT — all VMs share one external IP |
|
||||
| `""` / omitted | Overlay-only — no external connectivity |
|
||||
|
||||
## IP Pools: network.external_pools
|
||||
|
||||
@@ -531,18 +525,18 @@ dns_servers = ["8.8.8.8"] # DNS for VMs (optional)
|
||||
|
||||
### Field Details
|
||||
|
||||
| Field | Required | Description |
|
||||
|-------|----------|-------------|
|
||||
| `name` | Yes | Unique pool name. Used as NATS KV key and in `AllocateAddress`. |
|
||||
| `source` | No | IP source: `"static"` (default) uses `range_start`/`range_end`. `"dhcp"` obtains IPs from the router's DHCP server on each VM launch. |
|
||||
| `range_start` | Static only | First IP in the range. First IP is reserved for OVN gateway SNAT (unless `gateway_ip` overrides). |
|
||||
| `range_end` | Static only | Last IP in the range. |
|
||||
| `gateway` | Yes | Physical router/switch — the WAN default gateway. OVN sets `0.0.0.0/0 → gateway`. |
|
||||
| `gateway_ip` | NAT mode | Static IP for OVN router SNAT. In pool mode, defaults to `range_start` (static) or first DHCP lease (dhcp). In NAT mode, this is the single external IP all VMs share. |
|
||||
| `prefix_len` | Yes | Subnet mask for the external network (e.g., 24 = /24). |
|
||||
| `region` | No | Scopes pool to a region. Instances in this region prefer this pool. |
|
||||
| `az` | No | Scopes pool to an AZ. More specific than region. |
|
||||
| `dns_servers` | No | DNS servers propagated to VMs via OVN DHCP. |
|
||||
| Field | Required | Description |
|
||||
| ------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `name` | Yes | Unique pool name. Used as NATS KV key and in `AllocateAddress`. |
|
||||
| `source` | No | IP source: `"static"` (default) uses `range_start`/`range_end`. `"dhcp"` obtains IPs from the router's DHCP server on each VM launch. |
|
||||
| `range_start` | Static only | First IP in the range. First IP is reserved for OVN gateway SNAT (unless `gateway_ip` overrides). |
|
||||
| `range_end` | Static only | Last IP in the range. |
|
||||
| `gateway` | Yes | Physical router/switch — the WAN default gateway. OVN sets `0.0.0.0/0 → gateway`. |
|
||||
| `gateway_ip` | NAT mode | Static IP for OVN router SNAT. In pool mode, defaults to `range_start` (static) or first DHCP lease (dhcp). In NAT mode, this is the single external IP all VMs share. |
|
||||
| `prefix_len` | Yes | Subnet mask for the external network (e.g., 24 = /24). |
|
||||
| `region` | No | Scopes pool to a region. Instances in this region prefer this pool. |
|
||||
| `az` | No | Scopes pool to an AZ. More specific than region. |
|
||||
| `dns_servers` | No | DNS servers propagated to VMs via OVN DHCP. |
|
||||
|
||||
### Why range_start/range_end Instead of CIDR?
|
||||
|
||||
@@ -569,8 +563,8 @@ ovn_sb_addr = "tcp:10.1.3.181:6642" # OVN Southbound DB
|
||||
external_interface = "eth1" # WAN NIC name
|
||||
```
|
||||
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| Field | Description |
|
||||
| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `external_interface` | Physical NIC for WAN traffic. A macvlan sub-interface is created on this NIC for br-wan. Different servers may have different names (eth1, eno2, enp3s0, bond0). |
|
||||
|
||||
## Pool Selection Logic
|
||||
@@ -596,8 +590,12 @@ name. Allocation uses CAS (Compare-And-Set) for lock-free concurrent access:
|
||||
"range_start": "192.168.1.150",
|
||||
"range_end": "192.168.1.250",
|
||||
"allocated": {
|
||||
"192.168.1.150": {"type": "gateway"},
|
||||
"192.168.1.151": {"type": "auto_assign", "eni_id": "eni-abc", "instance_id": "i-123"}
|
||||
"192.168.1.150": { "type": "gateway" },
|
||||
"192.168.1.151": {
|
||||
"type": "auto_assign",
|
||||
"eni_id": "eni-abc",
|
||||
"instance_id": "i-123"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
@@ -782,20 +780,21 @@ on the port group control traffic:
|
||||
## Default Security Group
|
||||
|
||||
Every VPC gets a default security group that:
|
||||
|
||||
- Allows all inbound from instances in the same security group
|
||||
- Allows all outbound
|
||||
- Denies all other inbound
|
||||
|
||||
## AWS Rule → OVN ACL Translation
|
||||
|
||||
| AWS Security Group Rule | OVN ACL Match |
|
||||
|------------------------|---------------|
|
||||
| Ingress TCP/22 from 0.0.0.0/0 | `outport == @sg && ip4 && tcp.dst == 22` |
|
||||
| AWS Security Group Rule | OVN ACL Match |
|
||||
| ------------------------------- | ------------------------------------------------------------------ |
|
||||
| Ingress TCP/22 from 0.0.0.0/0 | `outport == @sg && ip4 && tcp.dst == 22` |
|
||||
| Ingress TCP/443 from 10.0.0.0/8 | `outport == @sg && ip4 && tcp.dst == 443 && ip4.src == 10.0.0.0/8` |
|
||||
| Ingress ALL from sg-other | `outport == @sg && ip4 && ip4.src == $sg_other_ip4` |
|
||||
| Ingress ICMP from anywhere | `outport == @sg && ip4 && icmp4` |
|
||||
| Egress ALL to 0.0.0.0/0 | `inport == @sg && ip4` |
|
||||
| Default deny inbound | `outport == @sg && ip4` (priority 900, action=drop) |
|
||||
| Ingress ALL from sg-other | `outport == @sg && ip4 && ip4.src == $sg_other_ip4` |
|
||||
| Ingress ICMP from anywhere | `outport == @sg && ip4 && icmp4` |
|
||||
| Egress ALL to 0.0.0.0/0 | `inport == @sg && ip4` |
|
||||
| Default deny inbound | `outport == @sg && ip4` (priority 900, action=drop) |
|
||||
|
||||
## Example: Allow SSH + HTTP
|
||||
|
||||
@@ -1132,18 +1131,21 @@ ip route show
|
||||
### Instance Has No Public IP
|
||||
|
||||
1. Check subnet has `MapPublicIpOnLaunch`:
|
||||
|
||||
```bash
|
||||
aws ec2 describe-subnets --subnet-ids $SUBNET \
|
||||
--query 'Subnets[0].MapPublicIpOnLaunch'
|
||||
```
|
||||
|
||||
2. Check IGW is attached:
|
||||
|
||||
```bash
|
||||
aws ec2 describe-internet-gateways \
|
||||
--filters Name=attachment.vpc-id,Values=$VPC
|
||||
```
|
||||
|
||||
3. Check external IP pool:
|
||||
|
||||
```bash
|
||||
nats kv get spinifex-external-ipam wan
|
||||
```
|
||||
@@ -1280,6 +1282,7 @@ sudo systemctl start ovn-central ovn-controller
|
||||
`ovn-sbctl show`. Gateway traffic (SNAT/DNAT) doesn't work.
|
||||
|
||||
**Cause:** Three names must align:
|
||||
|
||||
1. OVS `external_ids:system-id` → what ovn-controller registers as
|
||||
2. OVN SB `Chassis.name` → what actually exists in the SB DB
|
||||
3. vpcd's chassis name → `chassis-{nodeName}` from config
|
||||
|
||||
Reference in New Issue
Block a user