docs: rewrite for the product this actually became

The README was written when Aegis was a monitoring and gateway tool, and it has
been describing that product for three months while the thing itself grew into a
platform. The headline still read "Self-hosted Envoy gateway · AI threat
analysis · TLS automation · Real-time dashboard" — all true, all now one
subsystem of several, and none of it the reason someone would choose this.

README now leads with what it is: one control plane from the bare metal up,
air-gapped by default, with the customer's model weights staying on the
customer's storage. The gateway keeps its place; it just stops being the whole
story. A note records the scope change rather than pretending the old framing
never existed.

ROADMAP is rewritten, not amended. It listed a container manager, a Docker
registry, an embedded DNS server and AI-driven deployment orchestration as
future phases — all four shipped. A roadmap that describes delivered features as
upcoming undersells the product to exactly the reader who bothered to open it.
It is now built / next / then, matching the private status doc so the two cannot
drift.

CHANGELOG gains 1.3, covering three months: bare-metal provisioning, the
Kubernetes lifecycle with air-gapped installs validated live, Depot with
pluggable storage backends and reference-only artifacts, GPU-aware workloads,
Owl's risk-classified tool surface, and the auth middleware rewrite that
replaced a method-blind public-route match.

FIXED A BROKEN QUICKSTART, which is the part that mattered most. The published
docker-compose.yml exposes only 8765, while agents dial in on 8766 and the
overlay needs 3478/udp and 51820/udp. Anyone following the new one-line
enrolment instructions would have watched the installer succeed and the node
never appear. Both ports are now published with comments explaining when they
are needed, and AGENT_HOST is present with a note that it must be set to
something the nodes can actually resolve before enrolling any.

Every internal link in README and ROADMAP verified to resolve; compose file
validated.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B9dzsPuPpFbdgCuUDgsG8L
This commit is contained in:
Yang Yang Xie
2026-08-20 15:02:01 -07:00
parent ce683d4fd1
commit a92bbcd782
7 changed files with 315 additions and 122 deletions
Vendored
BIN
View File
Binary file not shown.
+84
View File
@@ -2,6 +2,90 @@
All notable changes to Aegis are documented here.
## [1.3] — 2026-08-20
The release where Aegis stopped being a gateway with extras and became a
platform. Three months of work; the highlights only.
### Fleet & provisioning
- **Bare metal, end to end** — Redfish/BMC power control, UEFI HTTP Boot with no
L2 adjacency required, serial console, and libvirt/KVM provisioning with
cloud-init. Power on an unracked server and finish with it in a cluster.
- **BMC discovery that discriminates** — a sweep no longer lists every host with
443 open. Candidates are labelled with the fleet member already at that
address, and proven negatives fold out of the way.
- **Node join tells the truth** — enrolment is recorded as an event, rejected
agents are reported with the reason and the fix instead of failing silently
every 30 seconds, and re-running the installer archives the old identity
rather than fighting it.
- **Idempotent repair** — `curl -fsSL https://<hub>/reinstall.sh | sudo bash` is
safe to run anywhere: it compares digests and does nothing if the node is
already current. Agent upgrades verify the download and arm a systemd
auto-rollback before replacing the binary.
- **Fleet activity log** with per-row prune, and location badges that classify
every node as LAN, cloud, mesh, remote or pending.
### Kubernetes
- **Full lifecycle via a reconciler** — create, scale, upgrade, back up, restore
and tear down, on k3s or kubeadm.
- **Air-gapped installs validated live** — a 5-node HA cluster brought up with no
egress at all.
- In-browser kubectl console and one-click Expose.
### Models & data (Depot)
- **Catalogue** for models and datasets with versions, digests, lineage and
per-owner quotas; HuggingFace and URL pull-through ingest with progress and
gated-model secrets.
- **LAN staging** — the hub egresses once and nodes pull from the hub, so a node
with no internet access can still receive a 57.9 GB model.
- **Pluggable storage backends** — keep artifact bytes on the hub's own disk, a
mounted NAS share, or your own cloud bucket. Switched at runtime from
Settings → Storage, no redeploy, and previously used backends stay readable so
a switch never orphans what you already had.
- **Reference-only artifacts** — catalogue what you already hold without copying
it. A 57 GB repository registers in seconds having downloaded nothing, and
costs nothing against quota.
### Workloads
- GPU-aware scheduling with eviction and sweeps, managed volumes with a file
browser and write API, notebooks, serving, and promote-a-result into the
catalogue.
### Owl / AI operations
- Owl acts through **risk-classified tools** with indirect prompt-injection
defences, and has deployed and configured Prometheus end to end unattended.
- Owl affordances hide themselves when no model key is configured, rather than
offering actions that cannot work.
### Security
- **Auth middleware rewritten.** The old public-route list matched by path
prefix and ignored the HTTP method, which silently exempted every sub-path and
every method beneath a public entry. Public routes are now exact
(method, prefix) pairs. Regression tests lock this in and run on every push.
- Per-node client certificates issued by the hub's own CA, with the agent
pinning the CA fingerprint; encrypted secret store for tokens and credentials.
### Project
- CI on every push: build, vet and the full Go test suite, plus a frontend
type-check and a check that fails the build if a React component nothing
imports would ship unreachable.
- Public documentation site at **[aegis.jerxie.com](https://aegis.jerxie.com)**,
with an assistant that answers from this documentation — including the parts
about what Aegis cannot do yet.
### Known gaps
Stated plainly rather than left to be discovered: single operator account (no
multi-user or RBAC), no control-plane HA, training not yet modelled as a managed
workload, and scale validated at nine nodes.
## [1.1.0] — 2026-05-23
### Gateway — Protection
+104 -120
View File
@@ -5,7 +5,7 @@
<h1 align="center">Aegis</h1>
<p align="center">
<strong>Self-hosted Envoy gateway · AI threat analysis · TLS automation · Real-time dashboard</strong>
<strong>Run AI on hardware you control.</strong>
</p>
<p align="center">
@@ -15,7 +15,13 @@
</p>
<p align="center">
Aegis sits between the internet and your services — one container that controls Envoy Proxy in real time, blocks threats automatically, manages TLS certificates, and lets you chat with your gateway through an AI assistant.
One control plane from the bare metal up — provisioning, networking, Kubernetes, models and serving.<br />
Fully air-gapped when you need it. Your model weights stay on your storage.<br />
With an AI operator that does the routine work instead of describing it.
</p>
<p align="center">
<a href="https://aegis.jerxie.com"><strong>aegis.jerxie.com</strong></a> — ask the assistant anything about the platform
</p>
---
@@ -24,6 +30,24 @@
---
## What Aegis is
Every organisation that cannot put its data in someone else's cloud currently
assembles an on-prem AI stack out of six tools — a mesh VPN, an ingress gateway,
a Kubernetes distribution, a bare-metal provisioner, an artifact registry, a
monitoring stack — plus a team to keep them in step. That team is the product
they are actually buying, and it does not scale.
Aegis replaces the assembly. One hub container runs the control plane; a single
agent enrols each machine. From there you provision bare metal, build clusters,
catalogue models, and serve them — without any of it leaving your network.
> **Note on scope.** Earlier versions of this README described Aegis as an Envoy
> gateway with AI threat analysis. That is now one subsystem of several. The
> gateway still does everything it did; the product around it grew.
---
## Quick start
```bash
@@ -36,151 +60,110 @@ curl -o envoy/envoy.yaml https://raw.githubusercontent.com/axieyangb/aegis/main/
docker compose up -d
```
Open **`http://localhost:8765`** — default login: `admin` / `changeme`.
Open **`http://localhost:8765`** — default login `admin` / `changeme`.
> Edit `docker-compose.yml` and set `ADMIN_PASSWORD` before exposing to the network.
> Set `ADMIN_PASSWORD` in `docker-compose.yml` before exposing this to anything.
On first boot Aegis seeds a working gateway baseline — HTTP (port 10080) and HTTPS (port 10443) listeners ready to accept filter chains.
Then enrol a machine — one line on the target, no SSH keys to distribute and no
inbound ports to open:
```bash
curl -fsSL "http://<your-hub>:8765/install.sh?token=<join-token>" | sudo bash
```
The node dials out to the hub, receives an identity signed by your hub's own CA,
and appears in the fleet. Bare metal with a BMC can instead be powered on and
installed from the UI.
**Requirements:** the hub needs any Linux host with Docker (2 vCPU / 2 GB is
plenty). Nodes need Linux with systemd, x86_64 or arm64. GPUs are detected and
scheduled on automatically. Internet access is optional after the first pull.
---
## Features
## What it does
<table>
<tr>
<td align="center" width="50%">
| | |
|---|---|
| **Fleet** | One-line enrolment, per-node mTLS identity, digest-verified agent upgrades with systemd auto-rollback, activity log |
| **Networking** | WireGuard overlay mesh, STUN NAT traversal, automatic LAN-direct routing with mesh fallback, embedded fleet DNS for hosts *and* containers |
| **Gateway** | Envoy with a full xDS control plane, SDS certificate management, automated TLS issuance, staged pending→apply config, rate limiting, one-click Expose |
| **Bare metal** | Redfish/BMC power control, UEFI HTTP Boot with no L2 adjacency required, serial console, libvirt/KVM provisioning with cloud-init |
| **Kubernetes** | Create, scale, upgrade, back up, restore and tear down clusters via a reconciler — k3s and kubeadm, with air-gapped installs validated live |
| **Models & data** | Catalogue models and datasets with versions, digests and lineage; pull from HuggingFace once and stage to nodes over the LAN; quotas per owner |
| **Your storage** | Keep artifact bytes on your own NAS mount or cloud bucket, switched at runtime from the UI. Reference-only artifacts catalogue what you already have without copying it |
| **Workloads** | GPU-aware scheduling, managed volumes with a file browser, notebooks, serving, promote-a-result-into-the-catalogue |
| **AI operations** | Owl, an assistant that acts through risk-classified tools with prompt-injection defences — it has deployed and configured Prometheus end to end, unattended |
| **Protection** | AI threat analysis over gateway access logs, automatic blocking, patrol sweeps, notifications |
**🦉 Owl AI Assistant**
---
![Owl Chat](docs/screenshots/02-owl-chat.png)
## What makes it different
Chat with your gateway in plain English. Owl analyses traffic, surfaces threats, and can configure your gateway end-to-end — clusters, certs, filter chains — from a single prompt.
Most of the above has a good commercial equivalent. These do not:
</td>
<td align="center" width="50%">
- **Air-gap is the default, not an enterprise tier.** A 5-node HA Kubernetes
cluster installed with no internet access at all, validated live. It falls out
of the architecture: the hub reaches the internet once, every node pulls from
the hub.
- **An AI operator that operates.** Owl changes infrastructure through scoped,
risk-classified tools — not a chat box that summarises a dashboard.
- **Bare metal to running model in one control plane.** The seams between MAAS,
Rancher and Harbor are where an ops team currently lives.
- **Your weights never touch our disk.** We catalogue; your storage holds. A
57 GB repository can be registered in seconds having downloaded nothing.
- **One transfer, then LAN.** A node with no internet access still gets a
57.9 GB model.
**🛡 Envoy Gateway Control**
---
![Gateway](docs/screenshots/04-gateway.png)
## What it does not do yet
Visual editor for listeners, filter chains, and clusters. Changes are validated and pushed live to Envoy via xDS — no restarts, no YAML files.
Published deliberately, because finding this out later is worse:
</td>
</tr>
<tr>
<td align="center" width="50%">
**🔍 IP Intelligence**
![IP Intelligence](docs/screenshots/03-ip-intelligence.png)
Every IP auto-profiled: geolocation, ASN, VPN/Tor detection, AbuseIPDB score, and full request history. AI patrol sweeps run in the background and auto-block threats.
</td>
<td align="center" width="50%">
**🔒 TLS Automation**
![Certs](docs/screenshots/05-certs.png)
ACME auto-renewal (Let's Encrypt, ZeroSSL), HTTP-01 & DNS-01 challenges, and a built-in Local CA for internal services — all pushed directly to Envoy SDS.
</td>
</tr>
<tr>
<td align="center" width="50%">
**🔔 AI Patrol & Alerts**
![Patrol](docs/screenshots/06-patrol.png)
Scheduled AI sweeps classify traffic around the clock. Blocks and anomalies are pushed to Telegram, Discord, Slack, or webhook.
</td>
<td align="center" width="50%">
**📱 Mobile-ready**
![Mobile](docs/screenshots/07-mobile.png)
Full dashboard and Owl chat from any device. Ask Owl what happened in the last two hours — it triages threats, blocks IPs, and confirms — all from your phone.
</td>
</tr>
<tr>
<td align="center" width="50%">
**🚧 Protection & Maintenance**
Styled 403/429 block pages with a customisable contact message — replace bare status codes with a branded denial page. Maintenance mode puts any listener (or a single SNI domain) into a 503 page while leaving all other services online. Preview pages inline before pushing to Envoy.
</td>
<td align="center" width="50%">
**⚡ Per-Domain Maintenance via Owl**
Ask Owl to put a single domain into maintenance while the rest of your services stay live: Owl calls `gateway_list_listeners` to discover the SNI, then targets only that filter chain — no manual JSON editing required.
</td>
</tr>
</table>
| Gap | Where it stands |
|---|---|
| **Multi-user / RBAC** | Single operator account. Ownership already threads through the data model; the identity layer on top is the next major workstream. |
| **Control-plane HA** | One hub, one database. Workloads survive a hub outage; management does not. |
| **Managed training runs** | The GPU scheduler, artifact capture and volumes all exist, but training is not yet modelled as a workload — a run is still started by hand. |
| **Scale** | Validated at nine nodes. We quote that number rather than implying a larger one. |
---
## Architecture
```
Internet ──▶ Envoy Proxy ──▶ Your services
gRPC xDS (port 18000)
┌────▼─────┐
│ Aegis │ port 8765
│ │
│ xDS CP │ controls Envoy live
│ Analytics│ reads Envoy ALS logs
│ AI Engine│ classifies IPs
│ Cert Mgr │ ACME + Local CA → Envoy SDS
│ Dashboard│ web UI + REST API
└──────────┘
```
Aegis runs as two containers — the hub (control plane, API, UI) and Envoy
(data plane, driven entirely over xDS). Agents on each node dial **out** to the
hub over a persistent connection, so no node needs an inbound port and nothing
needs an SSH key distributed to it.
`linux/amd64` and `linux/arm64` — runs on x86 servers, Raspberry Pi, Synology NAS, and Apple Silicon.
See [deployment architectures](docs/deployment-architectures.md) for direct
exposure, VPS relay and Cloudflare Tunnel topologies, and
[sequence diagrams](docs/sequence-diagrams.md) for the request paths.
---
## Configuration
| Variable | Default | Description |
|---|---|---|
| `PORT` | `8765` | Dashboard + API port |
| `XDS_PORT` | `18000` | Envoy gRPC xDS port |
| `DATA_DIR` | `/data` | Persistent data directory |
| `ADMIN_USERNAME` | `admin` | Admin username |
| `ADMIN_PASSWORD` | `aegis` | Admin password — **change this** |
| `AUTH_ENABLED` | `true` | Require login |
| `BLOCK_ENABLED` | `true` | Enable auto IP blocking |
| `NODE_ID` | `home` | Envoy node ID (must match envoy.yaml) |
Data is persisted at `/data/aegis.db` (SQLite). Mount a volume to keep data across container updates.
Everything is environment variables in `docker-compose.yml`. See
[configs/](configs/) for annotated examples and
[envoy-config.md](docs/envoy-config.md) for the gateway bootstrap.
---
## Docs & Tutorials
## Docs & tutorials
- [Getting started](docs/getting-started.md)
- [Envoy configuration](docs/envoy-config.md)
- [AI setup — Owl chat + threat analysis](docs/ai-setup.md)
- [Notifications — Telegram, Discord, webhooks](docs/notifications.md)
- [AI setup](docs/ai-setup.md) — connecting Owl to a model provider
- [Notifications](docs/notifications.md)
- [Deployment architectures](docs/deployment-architectures.md)
### Tutorial series: Exposing a service with Aegis
### Tutorial series: exposing a service with Aegis
| # | Tutorial | Description |
|---|---|---|
| 01 | [Local HTTPS with a whoami service](docs/tutorials/01-whoami-local-https.md) | Configure the gateway manually through the UI |
| 02 | [Configure the Gateway with Owl AI](docs/tutorials/02-whoami-ai-setup.md) | Same setup — let Owl AI do the configuration from a single prompt |
| 03 | [Understanding the Dashboard](docs/tutorials/03-understanding-the-dashboard.md) | Read live traffic data and analyse request patterns with Owl |
| 04 | [AI-Driven Protection](docs/tutorials/04-ai-driven-protection.md) | Use Owl to disable a service under attack and bring it back |
1. [Local HTTPS with whoami](docs/tutorials/01-whoami-local-https.md)
2. [AI setup](docs/tutorials/02-whoami-ai-setup.md)
3. [Understanding the dashboard](docs/tutorials/03-understanding-the-dashboard.md)
4. [AI-driven protection](docs/tutorials/04-ai-driven-protection.md)
### Videos
@@ -211,13 +194,14 @@ One prompt to Owl AI sets up the cluster, issues a certificate, and wires the fi
Distributed as a compiled binary. Source code is proprietary. See [LICENSE](LICENSE).
Community tier is **free forever**. Pro unlocks unlimited notification channels, longer log retention, and unlimited AI patrol sweeps.
Community tier is **free forever**. Pro unlocks unlimited notification channels,
longer log retention, and unlimited AI patrol sweeps.
---
## About
Built by **Jerry Xie** — formerly network security at Palo Alto Networks, now Senior Software Engineer specialising in identity, distributed cloud, Kubernetes, and AI. Aegis started as a home lab project and grew into a product.
**Issues & feature requests:** [GitHub Issues](https://github.com/axieyangb/aegis/issues)
**Enterprise / custom integrations:** [yyangxie@gmail.com](mailto:yyangxie@gmail.com)
Built and run on a real fleet — nine nodes spanning LAN, cloud and overlay-only,
managed through this platform daily. Questions are welcome at
[aegis.jerxie.com](https://aegis.jerxie.com), where an assistant answers from
this documentation, including the parts about what Aegis cannot do yet.
+110
View File
@@ -0,0 +1,110 @@
# Aegis Roadmap
**Updated:** 2026-08-20
Three sections: what exists, what makes it sellable, what makes it defensible.
Dates are deliberately absent below the first section — a roadmap with dates on
unstarted work is a wish list.
> The previous version of this file listed a container manager, a Docker
> registry, an embedded DNS server and AI-driven deployment orchestration as
> future phases. All four shipped. It is rewritten rather than amended, because
> a roadmap that describes shipped features as upcoming is worse than no
> roadmap.
---
## Built
Available today and running on a live nine-node fleet.
**Fleet** — one-line enrolment, per-node mTLS identity from the hub's own CA,
digest-verified agent upgrades with systemd auto-rollback, idempotent repair,
activity log.
**Networking** — WireGuard overlay mesh, STUN NAT traversal, automatic
LAN-direct routing with mesh fallback, embedded fleet DNS covering hosts and
bridge containers.
**Gateway** — Envoy with a full xDS control plane, SDS certificate management,
automated TLS issuance, staged pending→apply configuration, rate limiting,
maintenance pages, one-click Expose.
**Bare metal** — Redfish/BMC power control, UEFI HTTP Boot with no L2 adjacency
required, serial console, libvirt/KVM provisioning with cloud-init.
**Kubernetes** — create, scale, upgrade, back up, restore and tear down via a
reconciler, on k3s or kubeadm. Air-gapped installs validated live on a 5-node HA
cluster.
**Models and data** — catalogue with versions, digests, lineage and per-owner
quotas; HuggingFace and URL ingest; LAN staging so the hub egresses once;
pluggable storage backends (local, mounted share, cloud bucket) switchable at
runtime; reference-only artifacts that catalogue without copying.
**Workloads** — GPU-aware scheduling, managed volumes, notebooks, serving,
promote-to-catalogue.
**AI operations** — Owl acting through risk-classified tools with
prompt-injection defences; AI threat analysis over gateway access logs.
---
## Next — what makes it sellable
The gap between "impressive" and "a customer can run this without us".
**Multi-user, roles and service accounts.** Today there is one operator account
and `authenticated` means `admin` everywhere. Ownership already threads through
the data model — artifacts, workloads and projects all carry an owner, and
quotas are per-owner — so this is adding an identity layer on top of existing
scoping rather than retrofitting ownership. This is the single largest gap.
**Training as a managed workload.** The GPU scheduler, artifact capture, managed
volumes and result promotion all exist; training is simply not modelled as a
workload yet, so a run is still started by hand and observed with `tail -f`. The
defining workload of an AI platform should not be the one the platform cannot
see.
**Control-plane backup and documented recovery.** Cluster backup and restore
exist. The hub's own state does not have an equivalent a customer could follow.
**Audit trail.** Node lifecycle is recorded; a general who-did-what across
Depot, clusters, gateway and secrets is not.
**A quickstart a stranger can complete.** If we have to install it, it is a
service rather than a product.
---
## Then — what makes it defensible
**Control-plane high availability.** One hub and one database today. Workloads
survive a hub outage; management does not.
**Serving as a product surface** — autoscaling, canary rollouts, token metering.
Serving works; it is not yet something you sell by the seat.
**Experiment tracking and lineage** across runs, so results are comparable
rather than merely stored.
**Supply chain** — signed artifacts, SBOM, dependency CVE scanning.
**Scale validation** well beyond a single site. Nine nodes is what we have
tested, and it is the number we quote.
---
## Deliberately not doing yet
- **Multi-region / federation.** One site per hub until one site is excellent.
- **A custom scheduler beyond GPU awareness.** Kubernetes exists; we orchestrate
it rather than replace it.
- **Billing and metering.** Not until someone is billed.
- **A Windows agent.** Linux edge is the market.
---
Questions about any of this are welcome at
[aegis.jerxie.com](https://aegis.jerxie.com), where an assistant answers from
this documentation — including the parts about what is missing.
+16 -1
View File
@@ -4,7 +4,17 @@ services:
container_name: aegis
restart: unless-stopped
ports:
- "8765:8765"
- "8765:8765" # UI + API
# Agents dial IN to this port over mTLS. Without it published, a node can
# be issued a join token and will never be able to use it — the one-line
# installer completes and the node never appears. Only needed if you are
# enrolling machines; a single-host gateway install can drop it.
- "8766:8766" # agent control channel (mTLS)
# Overlay mesh. Needed for nodes that are not on this LAN; harmless if
# unused. UDP, and both are required — STUN discovers the path, WireGuard
# carries the traffic.
- "3478:3478/udp" # STUN — NAT traversal
- "51820:51820/udp" # WireGuard overlay
volumes:
- aegis_data:/data
environment:
@@ -13,6 +23,11 @@ services:
- AUTH_ENABLED=true
- BLOCK_ENABLED=true
- NODE_ID=home
# The address agents should dial back on. Leave blank for a gateway-only
# install; set it to a hostname or IP the nodes can actually reach before
# enrolling anything, or the token will carry an address that does not
# resolve from the node.
- AGENT_HOST=
envoy:
image: envoyproxy/envoy:v1.35-latest
BIN
View File
Binary file not shown.
+1 -1
View File
@@ -1,6 +1,6 @@
# Deployment Architectures (Exposing Envoy)
To make Aegis and Envoy accept real public traffic, you need to expose Envoy's ports (Host `80`/`443`) to the internet. Depending on your network setup (home lab, static IP, CGNAT, VPS), you should choose one of the two main architectures below.
To make Aegis and Envoy accept real public traffic, you need to expose Envoy's ports (Host `80`/`443`) to the internet. Depending on your network setup (on-prem LAN, static IP, CGNAT, VPS), you should choose one of the two main architectures below.
---