# Linux-VM-Bootstrapper

Idempotent, URL-executable bootstrap scripts for Docker VMs. Two node roles:

| Script | Role | What it deploys |
|---|---|---|
| `ParentNodeBootstrapper.sh` | **Parent (edge) node** | Nginx UI reverse proxy (public 80/443, default HTTPS routes to the UIs) + Dockhand (fleet manager, backups, auto-updates) + Webmin |
| `ChildNodeBootstrapper.sh` | **Child (branch) node** | Hawser agent (managed remotely by the parent node's Dockhand) + Webmin (agent) |

Both roles also get fail2ban (public hosts only) and a weekly patch/reboot
maintenance window.

Both scripts are safe to re-run at any time — packages update, existing `.env`
files are preserved, and running stacks are only recreated when their compose
definition actually changed. Every run is logged to `/custom/logs` (only the
3 most recent run logs are kept per script).

**Supported:** Ubuntu, Debian, RHEL (plus RHEL-likes: Rocky, Alma, CentOS Stream, Oracle Linux).

---

## Quick start

Replace `<raw-url>` with the raw-file base URL of wherever you host these
scripts. **Pin production machines to a release tag**, not a branch — with
Gitea-style raw paths that means `/raw/tag/v1.0.0/ParentNodeBootstrapper.sh`
instead of `/raw/branch/main/ParentNodeBootstrapper.sh`, so a fleet always
bootstraps from a known revision.

### Parent node

```bash
# Tailscale (default), register with an auth key, auto firewall:
curl -fsSL https://<raw-url>/ParentNodeBootstrapper.sh \
  | sudo -E bash -s -- --auth-key=tskey-auth-XXXXXXXX

# No auth key = VPN is installed but registration is skipped:
curl -fsSL https://<raw-url>/ParentNodeBootstrapper.sh | sudo -E bash

# NetBird instead of Tailscale:
curl -fsSL https://<raw-url>/ParentNodeBootstrapper.sh \
  | sudo -E bash -s -- --vpn=netbird --auth-key=<setup-key>
```

### Child node

```bash
# Standard mode (agent listens; Dockhand connects to it over the VPN):
curl -fsSL https://<raw-url>/ChildNodeBootstrapper.sh \
  | sudo -E bash -s -- --auth-key=tskey-auth-XXXXXXXX

# Edge mode (agent dials out to Dockhand; nothing listens):
curl -fsSL https://<raw-url>/ChildNodeBootstrapper.sh \
  | sudo -E bash -s -- \
  --auth-key=tskey-auth-XXXXXXXX \
  --dockhand-url=https://dockhand.example.com \
  --token=<agent token from Dockhand>
```

Every flag also has an environment-variable form (`VPN_PROVIDER`, `AUTH_KEY`,
`FIREWALL`, `SSH_PORT`, `TZ`, `HAWSER_MODE`, `HAWSER_TOKEN`,
`DOCKHAND_SERVER_URL`, …) — use `sudo -E` so exported variables survive. Flags
win over environment variables.

---

## What gets installed

Both scripts, silently and idempotently:

- Full package update/upgrade (`--skip-upgrade` to skip the upgrade pass)
- **Base tooling**: `openssl`, `ping` (iputils), `traceroute`, `tcpdump`,
  `dig` (dnsutils/bind-utils), `ifconfig` (net-tools), `jq`, `tar`, cron
- **PowerShell** (`pwsh`) — Microsoft repo, with a GitHub-tarball fallback
- **Docker Engine + Compose v2** from Docker's official repositories
- **Stable docker service account** — a `docker` user and `docker` group with
  fixed ids (default `2000:2000`, override with `--docker-uid=`/`--docker-gid=`
  or `DOCKER_UID`/`DOCKER_GID`) so bind-mount ownership is identical on every
  node. The group is created *before* Docker installs so the package adopts
  it; an existing group is renumbered (with a docker restart) on first run.
- **`/custom` permissions** — recursive POSIX ACLs grant the docker account,
  uid/gid `1000`, and root full access to everything under `/custom`, with
  default ACLs so newly created stack data inherits the same grants.
  Ownership of container-managed files is never changed (`chown -R` would
  break databases and other uid-sensitive data — ACLs add access instead).
- **Tailscale** (default) or **NetBird** (`--vpn=netbird`)
  - Registration happens **only when an auth/setup key is given**; otherwise
    installation completes and registration is skipped
  - Tailscale registration uses `--accept-routes --accept-dns=true
    --accept-risk=all` and does **not** advertise or use an exit node
- **Webmin** (`https://<host>:10000`) — installed on both roles; the firewall
  keeps 10000 off public interfaces. On the parent it is also proxied at
  root on TLS port `10445`.
- **fail2ban** — installed and enabled only when the firewall engages (public
  IP present): sshd jail, systemd backend, 5 tries / 10 min window / 1 h ban.
- **Weekly maintenance window** (`/etc/cron.d/vmboot-maintenance`) — schedule
  randomized once with `shuf` and preserved on re-runs:
  - OS patching: Saturday between 20:00–21:59
    (`/custom/scripts/cron/Invoke-AutomaticPatchInstallation.sh`)
  - Reboot: Sunday between 22:00–23:59
    (`/custom/scripts/cron/Invoke-AutomaticReboot.sh`)
  - Both log to `/custom/logs` (3 most recent kept per script)
- `/custom` skeleton (see layout below)

## Directory layout

```
/custom
├── scripts/                       # your operational scripts
│   └── cron/                      # generated maintenance payloads
│       ├── Invoke-AutomaticPatchInstallation.sh
│       └── Invoke-AutomaticReboot.sh
├── cron/                          # your cron payloads
├── logs/                          # bootstrapper + maintenance logs (last 3 kept per kind)
├── backups/
│   └── dockhand/                  # Dockhand local backup destination (parent)
└── docker/
    └── stacks/                    # STACK_BINDMOUNTROOT
        ├── stk-edge-proxy-00001/  # parent: Nginx UI (compose + .env + data)
        │   └── Proxy/{etc/nginx,etc/nginx-ui,etc/ssl,var/log/nginx,var/www}
        ├── stk-dockhand-00001/    # parent: Dockhand (compose + .env + App/Data)
        └── stk-hawser-00001/      # child: Hawser agent (compose + .env)
```

Stack conventions follow this repo's compose/env stack specs: stack names
`stk-<name>-00001`, static container names `{STACKNAME}-{TIER}-00001` (no
`hostname:`), network names never contain the stack name, `.env` + compose
defaults, `logging: local`, `dockhand.update` labels (apps on, high-risk off).

## Network architecture (parent node)

Two **host-wide** docker networks are pre-created by the script (fixed names,
no stack prefixes):

```
docker network create --driver bridge --attachable            EDGE-PROXY-EXTERNAL
docker network create --driver bridge --attachable --internal EDGE-PROXY-INTERNAL
```

```
                    Internet                    VPN (Tailscale/NetBird)
                       │                                  │
              0.0.0.0:80 / 0.0.0.0:443          <tunnel-ip>:8000 / :13000
                       │  (host-mode binds)               │
                ┌──────▼──────────────────────────────────▼─────┐
EDGE-PROXY-     │            EDGE-PROXY-PROXY-00001              │
EXTERNAL ──────▶│                (uozi/nginx-ui)                 │
                └──────┬─────────────────────────────────────────┘
                       │ EDGE-PROXY-INTERNAL (internal: true backplane)
        ┌──────────────┼──────────────────┐
        ▼              ▼                  ▼
 DOCKHAND-APP-00001  MYAPP-APP-00001   (your stacks…)
```

- **EDGE-PROXY-EXTERNAL** is attached **only** to the reverse proxy.
- **EDGE-PROXY-INTERNAL** is the private backplane (`internal: true` — no
  direct internet egress). App stacks attach to it to become routable.
- The reverse proxy publishes 80/443 with **long-form, host-mode port
  bindings** — a deliberate drift from the stack spec for this one service. It
  delivers the true client IP to Nginx while upstream routing still uses
  Docker's internal DNS, so the container does not need full host networking.

### Attaching an app stack to the proxy

```yaml
networks:
  INTERNAL:
    name: EDGE-PROXY-INTERNAL
    external: true

services:
  App:
    container_name: MYAPP-APP-00001
    networks:
      INTERNAL:
    expose:
      - '8080'        # INTERNAL-only services use expose, not host ports
```

Then in Nginx UI, route by **container name** (cross-stack DNS on a shared
network resolves container names):

```nginx
proxy_pass http://MYAPP-APP-00001:8080;
```

### Real client IP

Host-mode published ports mean direct public traffic already arrives with the
real source IP. Additionally, `ParentNodeBootstrapper.sh` seeds
`Proxy/etc/nginx/conf.d/real-ip.conf` so requests forwarded by a trusted local
hop (host gateway, private ranges, VPN/CGNAT) are un-masked via
`X-Forwarded-For`:

```nginx
set_real_ip_from  127.0.0.1;
set_real_ip_from  10.0.0.0/8;
set_real_ip_from  172.16.0.0/12;
set_real_ip_from  192.168.0.0/16;
set_real_ip_from  100.64.0.0/10;
real_ip_header    X-Forwarded-For;
real_ip_recursive on;
```

## Deployed stacks

### `stk-edge-proxy-00001` — Nginx UI (parent)

| Binding | Address | Purpose |
|---|---|---|
| `:80 → 80` | `HOST_PORT_BIND_ADDRESS` (default `0.0.0.0`) | Public HTTP + ACME HTTP-01 challenges |
| `:443 → 443` | `HOST_PORT_BIND_ADDRESS` | Public HTTPS |
| `:8000 → 9000` | `PRIVATE_TUNNEL_BIND_ADDRESS` (tunnel > private LAN) | Nginx UI dashboard (backend port 9000) |
| `:10443–10445` | `HOST_PORT_BIND_ADDRESS` | Management TLS ports at root (Nginx UI / Dockhand / Webmin); public exposure gated by the firewall |

First visit to `http://<tunnel-ip>:8000/` creates the admin account. The
image's default site proxies the dashboard on port 80 as well — replace it with
your real vhosts once setup is done. `/etc/ssl` is bind-mounted for cert
persistence; the script pre-seeds it from the image so the CA bundle isn't
masked.

**Let's Encrypt / ACME HTTP-01 flow:** challenges always arrive on **public
port 80**. Because nginx itself owns port 80 inside the container, the Nginx UI
backend answers challenges on a separate internal listener, the
**HTTPChallengePort (default 9180)** — nginx proxies
`/.well-known/acme-challenge` to it in-container. Port 9180 is
**container-internal only**: it is never published in the compose file and
never opened in the firewall. The only ports that must stay reachable from the
internet for issuance/renewal are 80 (HTTP-01) and 443 (TLS-ALPN).

**Management TLS ports (no sub-paths):** the script generates a self-signed
certificate (`Proxy/etc/ssl/selfsigned/selfsigned.{crt,key}`) and seeds
`sites-available/automatic.rules.conf` (symlinked into `sites-enabled/`).
Every UI is served at its **root path** on a dedicated TLS port — no sub-path
rewriting, no app-side base-path configuration, nothing to break:

| Port | Upstream |
|---|---|
| `https://<addr>:10443/` | Nginx UI backend (in-container `127.0.0.1:9000`) |
| `https://<addr>:10444/` | `DOCKHAND-APP-00001:3000` over `EDGE-PROXY-INTERNAL` |
| `https://<addr>:10445/` | host Webmin `:10000` via `host.docker.internal` (host-gateway) |

A minimal `default_server` on 443 returns 404 for unmatched TLS traffic, and
443 itself stays free for real vhosts. Expect the browser's self-signed-cert
warning until you replace the certificate. The Dockhand upstream resolves
through Docker's embedded DNS with deferred resolution, so nginx keeps
serving while containers restart. The site file is created once and then left
alone — manage it from Nginx UI afterwards. (Sub-paths were deliberately
rejected: prefix-stripping cannot fix SPAs that emit absolute asset URLs, and
per-app base-path configuration was ruled out. Ports and hostname vhosts are
the two trap-free options; vhosts need DNS, so ports are the default and
Nginx UI vhosts are the upgrade path.)

**Access priority** — how you reach the UIs, in order:

1. **VPN tunnel IP** (Tailscale/NetBird) — dashboards bind to it directly.
2. **Private LAN IP** — used automatically when no tunnel exists yet; upgraded
   to the tunnel IP on re-run after VPN registration.
3. **Public address via the proxy TLS ports** — the firewall opens
   10443–10445 on public interfaces **only when the host has neither a tunnel
   nor a private LAN address** (re-detected every boot; once a VPN comes up, a
   reboot or `systemctl restart vmboot-firewall` closes them again).

### `stk-dockhand-00001` — Dockhand (parent)

- Dashboard: `http://<tunnel-ip>:13000/` (VPN bind only, never public).
- Manages local Docker via `/var/run/docker.sock`; `/custom/docker/stacks` is
  mounted at the **same path** inside the container so managed compose files
  resolve identically.
- **Backups**: `/custom/backups/dockhand` is mounted at `/backups`. Dockhand
  configures backups in the UI (no env switch exists): **Settings → Backups →
  destination `Local` → `/backups` → set a schedule → enable**. This is the
  one post-install click the script cannot do for you.
- Child nodes are added under **Settings → Environments** as *Hawser
  Standard* or *Hawser Edge*.
- **Adopting the bootstrap stacks**: the same-path mount makes every compose
  file under `/custom/docker/stacks` visible inside the Dockhand container at
  the identical path. Use **Stacks → Import** and browse to
  `/custom/docker/stacks/stk-*/docker-compose.yml` — imported stacks become
  fully managed while their compose/.env files stay in place.

### `stk-hawser-00001` — Hawser agent (child)

| Mode | Direction | Needs |
|---|---|---|
| **standard** (default) | Dockhand → agent, `tcp://<tunnel-ip>:12376` | token (auto-generated into `.env` if omitted) |
| **edge** (`--dockhand-url=…`) | agent → Dockhand (outbound WebSocket) | `--token` from Dockhand's *Hawser Edge* dialog |

`--dockhand-url` accepts a plain base URL — `https://dockhand.example.com` is
normalized to `wss://dockhand.example.com/api/hawser/connect` automatically.
`STACKS_DIR` points at `/custom/docker/stacks` (same-path mount), so stacks
deployed from Dockhand land in the standard location.

## Auto-updates (Dockhand labels)

Container updates are handled by Dockhand's built-in updater — no Watchtower.
Every deployed service carries the `dockhand.update` label wired to its
`.env` flag:

```yaml
labels:
  dockhand.update: '${MYAPP_ENABLEAUTOMATICUPDATES:-true}'
  # dockhand.url: 'https://<host>/myapp/'
  # dockhand.changelog.url: 'https://.../releases'
```

- **General/app services default `true`**; set the update **schedule** in
  Dockhand under **Settings → Updates** (per-container or environment-wide,
  with optional vulnerability-gating).
- **High-risk services are always `false`**: databases, Valkey/Redis-style
  caches, message brokers, search/index stores, object stores — anything
  SHA256-pinned.
- URL labels (`dockhand.url`, `dockhand.port.<hostPort>.url`,
  `dockhand.changelog.url`) are included **commented out** — uncomment
  per-host if you want clickable links in the Dockhand UI. Other supported
  labels: `dockhand.hidden`, `dockhand.notify`, `dockhand.order`,
  `dockhand.adopt`.

## Weekly maintenance window

`/etc/cron.d/vmboot-maintenance` is generated on first run with
`shuf`-randomized times, then **preserved on re-runs** (delete the file to
re-randomize):

- **Patch**: Saturday at a random minute between 20:00 and 21:59 —
  `/custom/scripts/cron/Invoke-AutomaticPatchInstallation.sh` runs a silent
  full upgrade (`dist-upgrade`/`dnf upgrade` + autoremove).
- **Reboot**: Sunday at a random minute between 22:00 and 23:59 —
  `/custom/scripts/cron/Invoke-AutomaticReboot.sh`.
- Both log to `/custom/logs/<ScriptName>-*.log` (3 most recent kept each).

The exact times for a host are printed in the bootstrap summary and in the
cron file's header comment.

## Firewall (iptables, Docker-safe)

When any interface carries a **public IP directly** (`--firewall=auto`, the
default), the script locks the box down with **raw iptables — never ufw or
firewalld** (both are disabled if found active, since they'd fight the rules):

- **Host traffic** (`EDGE-PROXY-INPUT`, jumped from `INPUT`): on public interfaces
  only — allow established/related, ICMP, SSH (lockout guard, see below),
  parent: TCP 80/443 (80 covers ACME HTTP-01, relayed in-container to the
  Nginx UI HTTPChallengePort 9180; 443 covers TLS-ALPN), VPN direct-connection
  UDP (41641 Tailscale / 51820 NetBird), drop the rest. On the parent, the
  management TLS ports 10443–10445 are additionally allowed **only when the
  host has no tunnel and no private LAN address** (last-resort access;
  re-evaluated on every boot/run).
- **Container traffic** (`EDGE-PROXY-DOCKER-INPUT`, jumped from `DOCKER-USER`):
  matches the **pre-DNAT** port via `conntrack --ctorigdstport`, so on public
  interfaces only 80/443 reach containers (parent) or nothing does (child).
  VPN-side and LAN-side traffic is untouched.
- **Docker chains are never flushed or reordered** — only the
  `EDGE-PROXY-INPUT` / `EDGE-PROXY-DOCKER-INPUT` chains are managed, so
  ordering relative to Docker (before/after, restarts, reboots) can't break
  Docker networking. There is deliberately no OUTPUT chain — egress is not
  filtered. Chains from pre-rename releases (`VMBOOT-*`) are removed
  automatically.
- Persistence: `/usr/local/sbin/vmboot-firewall.sh` +
  `vmboot-firewall.service` re-detect the public interface and re-apply on
  every boot. Hosts with only private/CGNAT addresses get **no** rules.
- IPv6 is mirrored (ICMPv6/NDP and DHCPv6-client stay open).

**SSH stays allowed by default** on `SSH_PORT` (22) as a lockout guard —
locking a public VPS to literally 80/443 before the VPN is registered would
cut off access. Once your VPN access is verified, re-run with
`--no-ssh-allow` to close it.

**fail2ban** rides along whenever the firewall engages: an sshd jail
(systemd backend, `maxretry 5`, `findtime 10m`, `bantime 1h`) via
`/etc/fail2ban/jail.d/vmboot-sshd.local`. The firewall unit orders itself
`Before=fail2ban.service` and the bootstrap restarts fail2ban after applying
rules, so ban chains always sit above `EDGE-PROXY-INPUT` in the `INPUT` chain.

## Logging

Each run writes a full transcript to
`/custom/logs/<ScriptName>-YYYYMMDD-HHMMSS.log` (console output is unchanged —
the log is a `tee`). Retention is automatic: only the **3 most recent** logs
per script are kept; older ones are deleted at the start of each run.

## Idempotency & re-runs

- `docker-compose.yml` files are **rewritten** each run (the script is their
  source of truth); `.env` files are **preserved** once created.
- The bind address follows the priority **tunnel IP > private LAN IP >
  loopback (parent) / 0.0.0.0 (child standard agent)**. After registering the
  VPN, **re-run the script** — auto-managed values (placeholders or a
  previously detected LAN address) are upgraded to the tunnel IP; custom
  values are left alone.
- Firewall chains are flushed and rebuilt atomically per run; disabling is
  `--firewall=off` (existing `EDGE-PROXY-*` firewall chains are then left
  as-is; run `iptables -F EDGE-PROXY-INPUT` etc. to clear manually).

## Notes & caveats

- **Tunnel-IP port binds after reboot**: Docker may start before the VPN
  interface has its IP; the containers simply restart until the bind succeeds
  (`restart: unless-stopped`).
- **RHEL**: firewalld is disabled when the firewall engages — that is
  intentional per the raw-iptables mandate.
- **Dockhand backups** must be toggled once in the UI (path `/backups`); the
  destination directory, mount, and instructions are provisioned for you.
- `curl | bash` runs third-party installers (Docker repo setup, Tailscale
  `install.sh`, NetBird `install.sh`, Microsoft packages) — vet them if your
  environment requires it.
- Scripts must keep **LF line endings** (enforced via `.gitattributes`) or
  `curl | bash` will fail with `\r` errors.