Linux-VM-Bootstrapper/README.md

Linux-VM-Bootstrapper

Idempotent, URL-executable bootstrap scripts for Docker VMs. Two node roles:

Script	Role	What it deploys
ParentNodeBootstrapper.sh	Parent (edge) node	Nginx UI reverse proxy (public 80/443, default HTTPS routes to the UIs) + Dockhand (fleet manager, backups, auto-updates) + Webmin
ChildNodeBootstrapper.sh	Child (branch) node	Hawser agent (managed remotely by the parent node's Dockhand) + Webmin (agent)

Both roles also get fail2ban (public hosts only) and a weekly patch/reboot maintenance window.

Both scripts are safe to re-run at any time — packages update, existing .env files are preserved, and running stacks are only recreated when their compose definition actually changed. Every run is logged to /custom/logs (only the 3 most recent run logs are kept per script).

Supported: Ubuntu, Debian, RHEL (plus RHEL-likes: Rocky, Alma, CentOS Stream, Oracle Linux).

---

Quick start

Replace <raw-url> with the raw-file base URL of wherever you host these scripts. Pin production machines to a release tag, not a branch — with Gitea-style raw paths that means /raw/tag/v1.0.0/ParentNodeBootstrapper.sh instead of /raw/branch/main/ParentNodeBootstrapper.sh, so a fleet always bootstraps from a known revision.

Parent node

# Tailscale (default), register with an auth key, auto firewall:
curl -fsSL https://<raw-url>/ParentNodeBootstrapper.sh \
  | sudo -E bash -s -- --auth-key=tskey-auth-XXXXXXXX

# No auth key = VPN is installed but registration is skipped:
curl -fsSL https://<raw-url>/ParentNodeBootstrapper.sh | sudo -E bash

# NetBird instead of Tailscale:
curl -fsSL https://<raw-url>/ParentNodeBootstrapper.sh \
  | sudo -E bash -s -- --vpn=netbird --auth-key=<setup-key>

Child node

# Standard mode (agent listens; Dockhand connects to it over the VPN):
curl -fsSL https://<raw-url>/ChildNodeBootstrapper.sh \
  | sudo -E bash -s -- --auth-key=tskey-auth-XXXXXXXX

# Edge mode (agent dials out to Dockhand; nothing listens):
curl -fsSL https://<raw-url>/ChildNodeBootstrapper.sh \
  | sudo -E bash -s -- \
  --auth-key=tskey-auth-XXXXXXXX \
  --dockhand-url=https://dockhand.example.com \
  --token=<agent token from Dockhand>

Every flag also has an environment-variable form (VPN_PROVIDER, AUTH_KEY, FIREWALL, SSH_PORT, TZ, HAWSER_MODE, HAWSER_TOKEN, DOCKHAND_SERVER_URL, …) — use sudo -E so exported variables survive. Flags win over environment variables.

---

What gets installed

Both scripts, silently and idempotently:

• Full package update/upgrade (--skip-upgrade to skip the upgrade pass)
• Base tooling: openssl, ping (iputils), traceroute, tcpdump, dig (dnsutils/bind-utils), ifconfig (net-tools), jq, tar, cron
• PowerShell (pwsh) — Microsoft repo, with a GitHub-tarball fallback
• Docker Engine + Compose v2 from Docker's official repositories
• Stable docker service account — a docker user and docker group with fixed ids (default 2000:2000, override with --docker-uid=/--docker-gid= or DOCKER_UID/DOCKER_GID) so bind-mount ownership is identical on every node. The group is created before Docker installs so the package adopts it; an existing group is renumbered (with a docker restart) on first run.
• /custom permissions — recursive POSIX ACLs grant the docker account, uid/gid 1000, and root full access to everything under /custom, with default ACLs so newly created stack data inherits the same grants. Ownership of container-managed files is never changed (chown -R would break databases and other uid-sensitive data — ACLs add access instead).
• Tailscale (default) or NetBird (--vpn=netbird)
  • Registration happens only when an auth/setup key is given; otherwise installation completes and registration is skipped
  • Tailscale registration uses --accept-routes --accept-dns=true --accept-risk=all and does not advertise or use an exit node
• Webmin (https://<host>:10000) — installed on both roles; the firewall keeps 10000 off public interfaces. On the parent it is also proxied at root on TLS port 10445.
• fail2ban — installed and enabled only when the firewall engages (public IP present): sshd jail, systemd backend, 5 tries / 10 min window / 1 h ban.
• Weekly maintenance window (/etc/cron.d/vmboot-maintenance) — schedule randomized once with shuf and preserved on re-runs:
  • OS patching: Saturday between 20:00–21:59 (/custom/scripts/cron/Invoke-AutomaticPatchInstallation.sh)
  • Reboot: Sunday between 22:00–23:59 (/custom/scripts/cron/Invoke-AutomaticReboot.sh)
  • Both log to /custom/logs (3 most recent kept per script)
• /custom skeleton (see layout below)

Directory layout

/custom
├── scripts/                       # your operational scripts
│   └── cron/                      # generated maintenance payloads
│       ├── Invoke-AutomaticPatchInstallation.sh
│       └── Invoke-AutomaticReboot.sh
├── cron/                          # your cron payloads
├── logs/                          # bootstrapper + maintenance logs (last 3 kept per kind)
├── backups/
│   └── dockhand/                  # Dockhand local backup destination (parent)
└── docker/
    └── stacks/                    # STACK_BINDMOUNTROOT
        ├── stk-edge-proxy-00001/  # parent: Nginx UI (compose + .env + data)
        │   └── Proxy/{etc/nginx,etc/nginx-ui,etc/ssl,var/log/nginx,var/www}
        ├── stk-dockhand-00001/    # parent: Dockhand (compose + .env + App/Data)
        └── stk-hawser-00001/      # child: Hawser agent (compose + .env)

Stack conventions follow this repo's compose/env stack specs: stack names stk-<name>-00001, static container names {STACKNAME}-{TIER}-00001 (no hostname:), network names never contain the stack name, .env + compose defaults, logging: local, dockhand.update labels (apps on, high-risk off).

Network architecture (parent node)

Two host-wide docker networks are pre-created by the script (fixed names, no stack prefixes):

docker network create --driver bridge --attachable            EDGE-PROXY-EXTERNAL
docker network create --driver bridge --attachable --internal EDGE-PROXY-INTERNAL

                    Internet                    VPN (Tailscale/NetBird)
                       │                                  │
              0.0.0.0:80 / 0.0.0.0:443          <tunnel-ip>:8000 / :13000
                       │  (host-mode binds)               │
                ┌──────▼──────────────────────────────────▼─────┐
EDGE-PROXY-     │            EDGE-PROXY-PROXY-00001              │
EXTERNAL ──────▶│                (uozi/nginx-ui)                 │
                └──────┬─────────────────────────────────────────┘
                       │ EDGE-PROXY-INTERNAL (internal: true backplane)
        ┌──────────────┼──────────────────┐
        ▼              ▼                  ▼
 DOCKHAND-APP-00001  MYAPP-APP-00001   (your stacks…)

• EDGE-PROXY-EXTERNAL is attached only to the reverse proxy.
• EDGE-PROXY-INTERNAL is the private backplane (internal: true — no direct internet egress). App stacks attach to it to become routable.
• The reverse proxy publishes 80/443 with long-form, host-mode port bindings — a deliberate drift from the stack spec for this one service. It delivers the true client IP to Nginx while upstream routing still uses Docker's internal DNS, so the container does not need full host networking.

Attaching an app stack to the proxy

networks:
  INTERNAL:
    name: EDGE-PROXY-INTERNAL
    external: true

services:
  App:
    container_name: MYAPP-APP-00001
    networks:
      INTERNAL:
    expose:
      - '8080'        # INTERNAL-only services use expose, not host ports

Then in Nginx UI, route by container name (cross-stack DNS on a shared network resolves container names):

proxy_pass http://MYAPP-APP-00001:8080;

Real client IP

Host-mode published ports mean direct public traffic already arrives with the real source IP. Additionally, ParentNodeBootstrapper.sh seeds Proxy/etc/nginx/conf.d/real-ip.conf so requests forwarded by a trusted local hop (host gateway, private ranges, VPN/CGNAT) are un-masked via X-Forwarded-For:

set_real_ip_from  127.0.0.1;
set_real_ip_from  10.0.0.0/8;
set_real_ip_from  172.16.0.0/12;
set_real_ip_from  192.168.0.0/16;
set_real_ip_from  100.64.0.0/10;
real_ip_header    X-Forwarded-For;
real_ip_recursive on;

Deployed stacks

stk-edge-proxy-00001 — Nginx UI (parent)

Binding	Address	Purpose
:80 → 80	HOST_PORT_BIND_ADDRESS (default 0.0.0.0)	Public HTTP + ACME HTTP-01 challenges
:443 → 443	HOST_PORT_BIND_ADDRESS	Public HTTPS
:8000 → 9000	PRIVATE_TUNNEL_BIND_ADDRESS (tunnel > private LAN)	Nginx UI dashboard (backend port 9000)
:10443–10445	HOST_PORT_BIND_ADDRESS	Management TLS ports at root (Nginx UI / Dockhand / Webmin); public exposure gated by the firewall

First visit to http://<tunnel-ip>:8000/ creates the admin account. The image's default site proxies the dashboard on port 80 as well — replace it with your real vhosts once setup is done. /etc/ssl is bind-mounted for cert persistence; the script pre-seeds it from the image so the CA bundle isn't masked.

Let's Encrypt / ACME HTTP-01 flow: challenges always arrive on public port 80. Because nginx itself owns port 80 inside the container, the Nginx UI backend answers challenges on a separate internal listener, the HTTPChallengePort (default 9180) — nginx proxies /.well-known/acme-challenge to it in-container. Port 9180 is container-internal only: it is never published in the compose file and never opened in the firewall. The only ports that must stay reachable from the internet for issuance/renewal are 80 (HTTP-01) and 443 (TLS-ALPN).

Management TLS ports (no sub-paths): the script generates a self-signed certificate (Proxy/etc/ssl/selfsigned/selfsigned.{crt,key}) and seeds sites-available/automatic.rules.conf (symlinked into sites-enabled/). Every UI is served at its root path on a dedicated TLS port — no sub-path rewriting, no app-side base-path configuration, nothing to break:

Port	Upstream
https://<addr>:10443/	Nginx UI backend (in-container 127.0.0.1:9000)
https://<addr>:10444/	DOCKHAND-APP-00001:3000 over EDGE-PROXY-INTERNAL
https://<addr>:10445/	host Webmin :10000 via host.docker.internal (host-gateway)

A minimal default_server on 443 returns 404 for unmatched TLS traffic, and 443 itself stays free for real vhosts. Expect the browser's self-signed-cert warning until you replace the certificate. The Dockhand upstream resolves through Docker's embedded DNS with deferred resolution, so nginx keeps serving while containers restart. The site file is created once and then left alone — manage it from Nginx UI afterwards. (Sub-paths were deliberately rejected: prefix-stripping cannot fix SPAs that emit absolute asset URLs, and per-app base-path configuration was ruled out. Ports and hostname vhosts are the two trap-free options; vhosts need DNS, so ports are the default and Nginx UI vhosts are the upgrade path.)

Access priority — how you reach the UIs, in order:

1. VPN tunnel IP (Tailscale/NetBird) — dashboards bind to it directly.
2. Private LAN IP — used automatically when no tunnel exists yet; upgraded to the tunnel IP on re-run after VPN registration.
3. Public address via the proxy TLS ports — the firewall opens 10443–10445 on public interfaces only when the host has neither a tunnel nor a private LAN address (re-detected every boot; once a VPN comes up, a reboot or systemctl restart vmboot-firewall closes them again).

stk-dockhand-00001 — Dockhand (parent)

• Dashboard: http://<tunnel-ip>:13000/ (VPN bind only, never public).
• Manages local Docker via /var/run/docker.sock; /custom/docker/stacks is mounted at the same path inside the container so managed compose files resolve identically.
• Backups: /custom/backups/dockhand is mounted at /backups. Dockhand configures backups in the UI (no env switch exists): Settings → Backups → destination Local → /backups → set a schedule → enable. This is the one post-install click the script cannot do for you.
• Child nodes are added under Settings → Environments as Hawser Standard or Hawser Edge.
• Adopting the bootstrap stacks: the same-path mount makes every compose file under /custom/docker/stacks visible inside the Dockhand container at the identical path. Use Stacks → Import and browse to /custom/docker/stacks/stk-*/docker-compose.yml — imported stacks become fully managed while their compose/.env files stay in place.

stk-hawser-00001 — Hawser agent (child)

Mode	Direction	Needs
standard (default)	Dockhand → agent, tcp://<tunnel-ip>:12376	token (auto-generated into .env if omitted)
edge (--dockhand-url=…)	agent → Dockhand (outbound WebSocket)	--token from Dockhand's Hawser Edge dialog

--dockhand-url accepts a plain base URL — https://dockhand.example.com is normalized to wss://dockhand.example.com/api/hawser/connect automatically. STACKS_DIR points at /custom/docker/stacks (same-path mount), so stacks deployed from Dockhand land in the standard location.

Auto-updates (Dockhand labels)

Container updates are handled by Dockhand's built-in updater — no Watchtower. Every deployed service carries the dockhand.update label wired to its .env flag:

labels:
  dockhand.update: '${MYAPP_ENABLEAUTOMATICUPDATES:-true}'
  # dockhand.url: 'https://<host>/myapp/'
  # dockhand.changelog.url: 'https://.../releases'

• General/app services default true; set the update schedule in Dockhand under Settings → Updates (per-container or environment-wide, with optional vulnerability-gating).
• High-risk services are always false: databases, Valkey/Redis-style caches, message brokers, search/index stores, object stores — anything SHA256-pinned.
• URL labels (dockhand.url, dockhand.port.<hostPort>.url, dockhand.changelog.url) are included commented out — uncomment per-host if you want clickable links in the Dockhand UI. Other supported labels: dockhand.hidden, dockhand.notify, dockhand.order, dockhand.adopt.

Weekly maintenance window

/etc/cron.d/vmboot-maintenance is generated on first run with shuf-randomized times, then preserved on re-runs (delete the file to re-randomize):

• Patch: Saturday at a random minute between 20:00 and 21:59 — /custom/scripts/cron/Invoke-AutomaticPatchInstallation.sh runs a silent full upgrade (dist-upgrade/dnf upgrade + autoremove).
• Reboot: Sunday at a random minute between 22:00 and 23:59 — /custom/scripts/cron/Invoke-AutomaticReboot.sh.
• Both log to /custom/logs/<ScriptName>-*.log (3 most recent kept each).

The exact times for a host are printed in the bootstrap summary and in the cron file's header comment.

Firewall (iptables, Docker-safe)

When any interface carries a public IP directly (--firewall=auto, the default), the script locks the box down with raw iptables — never ufw or firewalld (both are disabled if found active, since they'd fight the rules):

• Host traffic (EDGE-PROXY-INPUT, jumped from INPUT): on public interfaces only — allow established/related, ICMP, SSH (lockout guard, see below), parent: TCP 80/443 (80 covers ACME HTTP-01, relayed in-container to the Nginx UI HTTPChallengePort 9180; 443 covers TLS-ALPN), VPN direct-connection UDP (41641 Tailscale / 51820 NetBird), drop the rest. On the parent, the management TLS ports 10443–10445 are additionally allowed only when the host has no tunnel and no private LAN address (last-resort access; re-evaluated on every boot/run).
• Container traffic (EDGE-PROXY-DOCKER-INPUT, jumped from DOCKER-USER): matches the pre-DNAT port via conntrack --ctorigdstport, so on public interfaces only 80/443 reach containers (parent) or nothing does (child). VPN-side and LAN-side traffic is untouched.
• Docker chains are never flushed or reordered — only the EDGE-PROXY-INPUT / EDGE-PROXY-DOCKER-INPUT chains are managed, so ordering relative to Docker (before/after, restarts, reboots) can't break Docker networking. There is deliberately no OUTPUT chain — egress is not filtered. Chains from pre-rename releases (VMBOOT-*) are removed automatically.
• Persistence: /usr/local/sbin/vmboot-firewall.sh + vmboot-firewall.service re-detect the public interface and re-apply on every boot. Hosts with only private/CGNAT addresses get no rules.
• IPv6 is mirrored (ICMPv6/NDP and DHCPv6-client stay open).

SSH stays allowed by default on SSH_PORT (22) as a lockout guard — locking a public VPS to literally 80/443 before the VPN is registered would cut off access. Once your VPN access is verified, re-run with --no-ssh-allow to close it.

fail2ban rides along whenever the firewall engages: an sshd jail (systemd backend, maxretry 5, findtime 10m, bantime 1h) via /etc/fail2ban/jail.d/vmboot-sshd.local. The firewall unit orders itself Before=fail2ban.service and the bootstrap restarts fail2ban after applying rules, so ban chains always sit above EDGE-PROXY-INPUT in the INPUT chain.

Logging

Each run writes a full transcript to /custom/logs/<ScriptName>-YYYYMMDD-HHMMSS.log (console output is unchanged — the log is a tee). Retention is automatic: only the 3 most recent logs per script are kept; older ones are deleted at the start of each run.

Idempotency & re-runs

• docker-compose.yml files are rewritten each run (the script is their source of truth); .env files are preserved once created.
• The bind address follows the priority tunnel IP > private LAN IP > loopback (parent) / 0.0.0.0 (child standard agent). After registering the VPN, re-run the script — auto-managed values (placeholders or a previously detected LAN address) are upgraded to the tunnel IP; custom values are left alone.
• Firewall chains are flushed and rebuilt atomically per run; disabling is --firewall=off (existing EDGE-PROXY-* firewall chains are then left as-is; run iptables -F EDGE-PROXY-INPUT etc. to clear manually).

Notes & caveats

• Tunnel-IP port binds after reboot: Docker may start before the VPN interface has its IP; the containers simply restart until the bind succeeds (restart: unless-stopped).
• RHEL: firewalld is disabled when the firewall engages — that is intentional per the raw-iptables mandate.
• Dockhand backups must be toggled once in the UI (path /backups); the destination directory, mount, and instructions are provisioned for you.
• curl | bash runs third-party installers (Docker repo setup, Tailscale install.sh, NetBird install.sh, Microsoft packages) — vet them if your environment requires it.
• Scripts must keep LF line endings (enforced via .gitattributes) or curl | bash will fail with \r errors.