gsadmin/k7
1
0
Fork 0
mirror of https://github.com/Katakate/k7.git synced 2026-09-21 09:33:19 +00:00
Code Issues Packages Projects Releases Wiki Activity

Release 0.4.0

Browse Source 
Per-key node pins, k7d-fc pause/resume/exec, and HA-soak fixes. Playbook
pins k7d 0.7.0. GitHub .deb, Launchpad PPA, and PyPI k7-sdk are 0.4.0.
This commit is contained in:
G
2026-09-19 23:22:14 +02:00
parent 9ff951506c
commit 13fafe0997
52 changed files with 2452 additions and 511 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHALLENGES.md
+157 -1
View File
@@ -524,7 +524,25 @@ timed out after 5s`. Create→Ready and exec *before* pause had worked
docker-perf failure. Lifecycle was aborted; do not quote k7d-fc
resume→exec from that run.
**Reference:** CHALLENGES #15; k7d #232.
**Fixed (lifecycle resume), spec 42a:** not #15. Two bugs, one per
layer. (1) Firecracker v1.16.0/v1.16.1's vsock device armed its
`TRANSPORT_RESET` RX gate on every `PATCH /vm Resumed` with no reset
event for the guest to ack, so after a bare pause → resume no
host→guest packet — no `CONNECT` reply — was ever delivered; upstream
fixed it as #6100 in v1.16.2, and k7d's `guest/fc/pins.env` plus the
vendored `src/k7/deploy/k7d-fc/pins.env` now pin v1.16.2. k7d's
`resume_vm` also dials the resumed FC guest and fails loud if it does
not answer. (2) The shim's exec bridge never opened containerd's
stdout/stderr FIFOs on its failure path, so a probe exec that could
not reach the paused guest left kubelet's prober parked in `ExecSync`
for its 2-minute gRPC deadline — the pod stayed Ready through a whole
pause on `k7d` *and* `k7d-fc`. `tests/integration/test_pause_resume.py::TestPauseResumeExecAnswers`
covers pause → resume → `exec_command("echo hi")` on both; the
`PERFORMANCE.md` k7d vs k7d-fc lifecycle table has the resume→exec
number. Root cause and timings: k7d CHALLENGES #240 (vsock gate) and
#241 (FIFOs).
**Reference:** CHALLENGES #15; k7d #232, #240, #241; Firecracker #6100.
**Time lost:** ~15 min on the OutOfcpu wait; lifecycle resume hung until
the pytest process was killed (~8 min).
@@ -589,3 +607,141 @@ hostPath); `K7D_DOCKER_PAYLOAD_DOCKERD` in `src/k7/core/docker.py`.
**Time lost:** ~30 min diagnosing why the live cluster had dockerd but the
API refused `--docker`.
---
## 20. Laptop `k7 api status` crashed; docs `k7.yaml` 128Mi never went Ready
**Symptom:** After `apt install k7` (PPA 0.3.1) on a laptop with
`k7 config set api.url` / `api.ca` / `api.key`, `k7 list` and
`k7 nodes storage` worked, but the Quickstart's next commands
`k7 api status` and `k7 api endpoint` raised `FileNotFoundError: kubectl`.
The same Quickstart's `examples/k7.yaml` (`cpu: 100m`, `memory: 128Mi`,
`before_script: apk add curl`, default backend kfd) timed out with
"Timed out waiting for sandbox container to start".
**Root cause:** `k7 api status` / `endpoint` always shelled out to
`kubectl` (or `k3s kubectl`) without checking the binary exists. A laptop
that only has the .deb has no kubeconfig and no kubectl. Separately, the
docs `k7.yaml` set `memory: 128Mi`, which Kata stamps as
`io.katacontainers.config.hypervisor.default_memory: 128`. The Firecracker
shim refuses anything below **256Mi**, so the pod stays `ContainerCreating`
(`FailedCreatePodSandBox`) until create times out. `before_script` never
runs.
**Fix:** missing kubectl falls back to `GET /health` on the configured API
URL (status) / prints that URL (endpoint). Create rejects Kata memory
below 256Mi immediately. Docs + `examples/k7.yaml` use `backend: k7d`
and `cpu: "1"` / `memory: "1Gi"`. PPA is 0.3.1, not 0.2.2.
**Reference:** `src/k7/cli/k7.py` (`_kubectl_run`, `_api_status_via_https`);
`examples/k7.yaml`.
**Time lost:** ~20 min reproducing on a 3-node HA soak after a public 0.3.1
release; the install itself was one command and succeeded.
---
## 21. `--expose-port` NodePort is dead until Ready; `k7 exec -- sh -c` self-nests
**Symptom:** Docs `k7 create --expose-port 8000 --before-script 'nohup python3 -m http.server 8000 &'` printed a NodePort, but curling it from a laptop timed out. Cilium showed the Service in **maintenance**. `k7 exec NAME -- sh -c 'echo hi > /tmp/x'` failed with Nuitka/PyInstaller-style "tried to call itself with '-c'". `k7 restore --latest` and `k7 delete-all -y` do not exist.
**Root cause:** `externalTrafficPolicy: Local` plus Cilium keeps a NodePort in maintenance while endpoints are `notReadyAddresses`. k7d/kfd pid 1 is `sleep 365d`; a bare `&` in `before_script` is killed when the script exits, so the Ready probe never sees the http server (and a missing `touch` of the done file has the same effect). `k7 exec` already wraps the joined argv in `sh -c`, so a nested `sh -c` is the CLI binary eating `-c`. Restore takes two positionals; `delete-all` confirms interactively with no `-y`.
**Fix:** Docs: trailing `sleep 1` after nohup so before_script can finish and Ready can fire; curl the **pod's** node, not an arbitrary master. Exec: one quoted string, no extra `sh -c`. Restore/delete-all examples match the CLI. Create success text points at `k7 list`, not `k7 list --name`.
**Reference:** `src/k7/cli/k7.py` exec/create; docs `k7/guides/cli.mdx`.
**Time lost:** ~40 min on the public 0.3.1 HA soak (expose looked like a CNI bug until Ready flipped).
---
## 22. k7d docker graph image is a teardown race, not a leak (spec 39a)
**Symptom:** `TestDockerK7d` / `TestDockerK7dFc` failed after `k7 delete` + 3s with `leaked k7d docker volume images: {scratch-vm-…-docker.img}`. Hours later, same k7d PID, the files were gone.
**Root cause:** `k7 delete` returns when Kubernetes objects are gone. The VM's `ScratchDisk::drop` unlinks the graph image asynchronously when the containerd shim `Delete` lands. On a busy 3-node HA box that is more than 3s.
**Fix:** bounded poll (~60s) in the test. `delete_sandbox` does **not** wait for the k7d VM — coupling API latency to shim teardown would stall every delete.
**Reference:** `tests/integration/test_docker.py` `_wait_k7d_docker_disks_gone`; `K7Core.delete_sandbox`.
**Time lost:** the soak already established this; the 3s sleep was the only defect.
---
## 23. Firecracker jailer test missed `firecracker-v1.` (spec 39a)
**Symptom:** `test_jailer_active` asserted "No firecracker processes found on the host" while a jailed kfd VM was running.
**Root cause:** Linux truncates `comm` to 15 characters. The pinned binary is `firecracker-v1.16.1`, so `comm` is `firecracker-v1.`. The helper compared `== "firecracker"`. The chroot binary is also versioned (`firecracker-v1.16.1`, not `/firecracker`). The jail itself is correct: `/etc/passwd`, `/etc/shadow`, `/usr`, `/boot` absent; `vmlinux` + `rootfs` present. `readlink /proc/<pid>/root` is `/` because the jailer pivot-roots in a private mount ns.
**Fix:** match `comm` by prefix `firecracker-`, require `fcConfig.json` / `--config-file` on the cmdline, skip `(deleted)` orphans, require `vmlinux`+`rootfs`, glob `firecracker*` in the chroot. Host-FS-unreachable asserts stay.
**Reference:** `tests/integration/test_firecracker.py` `_get_live_firecracker_pids`.
**Time lost:** ~20 min of `/proc` on the soak node.
---
## 24. Expose tests timed out because the pod was on another node (spec 39a)
**Symptom:** `TestSandboxExpose` curled `http://<other-node>:<nodeport>` from the first master and timed out. Off-cluster, the pod's node answered HTTP 200 and the other nodes refused connect.
**Root cause:** `externalTrafficPolicy: Local` is required (without it, `cidr:` rules see SNAT). Cilium socket-LB intercepts in-cluster-node origin to a NodePort on a different node. The tests did not pin `node_name`.
**Fix:** pin expose sandboxes to `os.uname().nodename`. Do not switch the Service to `Cluster`.
**Reference:** `tests/integration/test_sandbox_ingress.py` `TestSandboxExpose._exposed`.
**Time lost:** ~15 min confirming Local vs Cilium vs a wrong-node curl.
---
## 25. kql live `--docker` fork never went Ready: overlay2 was crash-inconsistent (spec 39a)
**Symptom:** `TestDockerKQL.test_fork_clones_both_pvcs` — both VolumeSnapshots ready, both child PVCs Bound, child never Ready in 240s.
**Root cause (live, spec 39a):** the child stuck in `Init:0/2` with `FailedAttachVolume: volume is not ready for workloads`. Kubernetes Bound is not Longhorn-ready-to-attach; HA r=3 clone hydration takes longer than 240s. The qemu fork test already waits 600s for this. `sync` is also not enough for a busy overlay2 once the guest *does* start.
**Fix:** wait 600s for the child (same bound as `test_qemu.test_fork_clones_data`). Plus `fsfreeze` the graph: alpine `docker:27.5.1-dind` has no `fsfreeze`, so stage it from the ubuntu sandbox via the shared `/tmp` emptyDir, copy into the vehicle rootfs, freeze around VolumeSnapshot create, thaw in `finally`.
**Reference:** `K7Core._create_kata_snapshots_quiesced`.
**Time lost:** soak diagnosis; freeze is the product answer rather than refusing live forks.
---
## 26. Partner-facing 0.3.1 traps: kfd fork 404, SDK snippet missing CA, `k7 logs` empty
**Symptom:** Following docs.katakate.org / the k7 README on a 3-node HA PPA 0.3.1 cluster:
- `k7 fork` of a kfd sandbox printed `Source root PVC <name>-root-lh not found; cannot fork storage` instead of "kfd cannot fork".
- `k7 api status` printed `Client(endpoint=..., api_key=...)` with no `verify_ssl`; that fails against the playbook-minted cluster CA.
- `k7 logs demo --tail 20` printed nothing (exit 0) after only `k7 exec`.
- `python3 -m venv` on the node failed (`ensurepip is not available`).
- `k7 resume` of a kql sandbox returned immediately while the pod was still `Pending`.
**Root cause:** `fork_sandbox` treated missing kfd PVCs as a generic storage 404. The status command's help snippet never grew `verify_ssl` when HTTPS-by-default landed. `k7 logs` is a CRI container snapshot; exec goes through the agent. Ubuntu 24.04 cloud images omit `python3-venv`. HA Longhorn attach is slower than `resume()` returning.
**Fix:** reject every kfd fork up front (`KFD_FORK_REJECT`). Print `verify_ssl='./k7-ca.crt'` in `k7 api status`. Docs: wait-until-Ready after kql resume, empty logs are success, SDK is a client install (`apt install python3-venv` on the node).
**Reference:** none.
**Time lost:** ~30 min walking the quickstart on the HA cluster.
---
## 27. `k7 --core` leaked kubernetes_asyncio/aiohttp sessions (`Event loop is closed`)
**Symptom:** `TestSnapshotCrud.test_create_list_inspect_delete_round_trip` failed with `assert snap in cp.stdout` and `cp.stdout == '\n'`. Interpreter also logged `Unclosed client session` / `Event loop is closed` after `k7 --core snapshot create`.
**Root cause:** Each `CoreV1Api()` / `AppsV1Api()` / `CustomObjectsApi()` constructed its own `ApiClient` (aiohttp session) bound to the `asyncio.run` loop. Typer handlers never called `close()`, so loop teardown raced the session destructor. Separately, `_create_kata_snapshots_quiesced` dropped the successful `OperationResult.message` from `_create_volume_snapshot`, so the CLI echoed a blank line even when the snapshot existed.
**Fix:** One shared `ApiClient` per `K7Core`, `async def aclose()`, CLI `_core_run` / API `get_k7_core` / snapshot-gc always close. Quiesced snapshot success now keeps the create message (`Snapshot <name> created for PVC …`).
**Reference:** kubernetes_asyncio `ApiClient.close` → `rest_client.close()` (aiohttp).
**Time lost:** caught on the HA integration run after the partner walkthrough.
CHANGELOG.md
+61
View File
@@ -7,6 +7,67 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [0.4.0] — 2026-09-19
Per-key node pins, k7d-fc pause/resume/exec, and HA-soak fixes. Playbook
pins k7d **0.7.0**. GitHub `.deb`, Launchpad PPA, and PyPI `k7-sdk` are
**0.4.0**.
### Added
- **Optional per-key node placement** for tenant isolation on the
shared per-node k7d daemon. `k7 generate-api-key --node <node>`
(repeatable) stores `"nodes"` on the key, analogous to `-n` for
namespaces. Placement uses the Node's `kubernetes.io/hostname`
label, not `spec.nodeName`. `k7 nodes dedicate NODE --tenant ID`
writes `k7.katakate.org/tenant` as a label and a NoSchedule taint so
other sandboxes cannot share that daemon. `k7 nodes list` shows the
K3s Node names (Linux hostname / `kubernetes.io/hostname`) to pass
to `--node`; inventory.ini does not write that label — `k7_backends`
only stamps backend labels at install. A single-node key auto-pins
creates/restores; a multi-node key requires an explicit `node_name`
/ `k7 create --node`. Forks must already sit on an allowed node.
`GET /api/v1/nodes/storage` is 403 for a node-scoped key. Unscoped
keys keep unrestricted placement. Root on `[k7_servers]` is still
cluster-admin; pins do not contain a host compromise — tenant
sandboxes belong on `[k7_agents]`. `k7-agent` has no kube SA token
and no `remote-node` ingress, so a dedicated worker breakout is that
tenant, not cluster-admin.
- **k7d-fc resume → exec**: pin Firecracker v1.16.2; pause/resume/exec
integration row and filled `PERFORMANCE.md` lifecycle table.
### Fixed
- **`k7 api status` / `k7 api endpoint` from a laptop** no longer crash with
`FileNotFoundError: kubectl`. With `k7 config set api.url` they print the
configured URL and probe `GET /health`. `k7 api enable`/`disable` still
need kubectl on a cluster node and now say so instead of tracebacking.
- **Kata memory below 256Mi** is rejected at create time instead of waiting
300s for `FailedCreatePodSandBox` (docs `k7.yaml` used `128Mi`).
- **Create success text** after `--expose-port` points at `k7 list`, not a
nonexistent `k7 list --name`.
- **Kata kql `--docker` live fork / snapshot** now `fsfreeze`s
`/var/lib/docker` in the docker-vehicle around VolumeSnapshot create
(thaw in `finally`). alpine dind has no `fsfreeze`; it is staged from
the sandbox image via the shared `/tmp`. `sync` alone left overlay2
crash-inconsistent on Longhorn r=3.
- **`k7 delete` does not wait for k7d VM teardown.** The docker-graph
leftover assertion polls up to 60s instead of a 3s sleep.
- **Loud kfd fork reject**, cluster CA snippet in the SDK/docs path, and
`K7Core` aiohttp session close on teardown.
### Changed
- **`k7 install --backend` is required.** There is no default runtime
list (kfd needs a spare disk; installing all backends grows attack
surface). Inventory hosts must set `k7_backends` (group vars are
fine). `none` is the only empty set — scheduling-only masters use
`[k7_servers:vars] k7_backends=none`. Empty/omitted is an error, not
kfd.
- Playbook default `k7d_version` is **0.7.0** (was 0.6.0). README,
Hetzner tutorial, and inventory examples target k7 **0.4.0** on
GitHub, PPA, and PyPI.
## [0.3.1] — 2026-09-14
HA install and API-path `--docker` fixes for the 0.3.0 line. README
PERFORMANCE.md
+31 -3
View File
@@ -324,9 +324,37 @@ fork is the k8s adopt path (not the ~5 ms VMM CoW floor); alpine
create→Ready / fork→exec on the same node this run were **2.15 s /
2.54 s** (k7d) and **2.14 s / 2.45 s** (k7d-fc), n=1.
k7-fc CRI exec after pause/resume hung (readiness probe timeout,
`guest_cid=0` retained) — see CHALLENGES #17. Do not quote a
resume→exec number for k7d-fc from this cut.
### k7d vs k7d-fc lifecycle — 2026-09-14
Same 3-node HA cluster (k7-node-01 as the bench host: Hetzner AX41,
Ryzen 5 3600, 64 GiB, NVMe, Ubuntu 24.04, kernel 6.8.0-138, k3s
v1.36.4), k7d 0.6.0 built from the Firecracker jail-integrity branch
(not the public GitHub tarball of the same version string), Firecracker v1.16.2,
`alpine:3.20`, 3 reps interleaved, median. `resume → exec` is
`resume_sandbox()` until an `echo` exec answers; `pause` is the API
call until the VM is frozen.
```bash
K7_BENCH_BACKENDS=k7d,k7d-fc K7_BENCH_REPS=3 \
uv run pytest -m bench tests/integration/bench_backend_lifecycle.py -v -s
```
| Operation | k7d | k7d-fc |
|-----------|-----|--------|
| create → Ready | 2.39 s | 2.38 s |
| exec round-trip (median of 10) | 0.37 s | 0.37 s |
| fork call | 2.73 s | 2.66 s |
| fork → Ready + exec answers | 3.23 s | 3.15 s |
| pause effective | 0.35 s | 0.35 s |
| **resume → exec answers** | **0.67 s** (0.62–0.76) | **0.70 s** (0.69–0.72) |
| delete | 0.70 s | 0.66 s |
The earlier caveat on this section — k7-fc CRI exec after pause/resume
hung, do not quote a resume→exec number — is gone: the hang was
Firecracker v1.16.0/v1.16.1 gating vsock RX after a bare resume
(upstream #6100, fixed in v1.16.2), plus a shim exec bridge that left
kubelet's probe parked (k7d CHALLENGES #240 / #241; CHALLENGES #17
here).
## k7d `--docker` (first-class guest service)
README.md
+28 -18
View File
@@ -76,7 +76,7 @@ The Tech Stack
Sandbox backends
</h3>
`k7 install --backend <kfd|kql|k7d|k7d-fc>` provisions one or more backends per node; `k7 create --backend …` picks one per sandbox. See [docs/BACKENDS.md](docs/BACKENDS.md) for the architecture and [PERFORMANCE.md](PERFORMANCE.md) for the full measurements (Hetzner AX41 node, medians).
`k7 install --backend <kfd|kql|k7d|k7d-fc>` provisions those backends (required; there is no default). `none` installs the cluster with no sandbox runtime — typical for scheduling-only masters (`k7_backends=none` in inventory). Each sandbox picks one runtime (`k7 create --backend …`). See [docs/BACKENDS.md](docs/BACKENDS.md) for the architecture and [PERFORMANCE.md](PERFORMANCE.md) for the full measurements (Hetzner AX41 node, medians).
| | `kfd` (kata-firecracker-devmapper) | `kql` (kata-qemu-longhorn) | `k7d` | `k7d-fc` |
|---|---|---|---|---|
@@ -107,7 +107,7 @@ Also available today
- 🌐 Multi-node clusters (Ansible + Longhorn)
- 🔍 Cilium CNI with FQDN egress policies (optional Hubble flow observability via <code>k7 install --hubble</code>; off by default, observability only)
- 📸 Pause / resume / fork / restore and <code>k7 snapshot</code> lifecycle
- 🐍 Python SDK: <code>pip install k7-sdk</code> (<code>katakate</code> package deprecated)
- 🐍 Python SDK: <code>pip install k7-sdk==0.4.0</code> (<code>katakate</code> package deprecated)
📋 **See [ROADMAP.md](ROADMAP.md) for upcoming work (GPU passthrough, …).**
@@ -127,7 +127,7 @@ We provide a:
- **CLI**: to use on the node(s) directly --> `apt install k7`
- **API**: deployed automatically by `k7 install` (toggle with `k7 api enable` / `k7 api disable`)
- **Python SDK**: HTTP client sync/async --> `pip install k7-sdk`
- **Python SDK**: HTTP client sync/async --> `pip install k7-sdk==0.4.0`
## Current requirements
@@ -147,7 +147,7 @@ We provide a:
- Azure: Dv3, Ev3, Dv4, Ev4, Dv5, Ev5 (Intel/AMD x86) or Dpdsv5, Dpldsv5, Epsv5 (ARM64).
- DigitalOcean: Premium Intel and AMD droplets with nested virtualization enabled.
- Others: in general, hardware virtualization is not exposed on cloud VPS, so you'll likely want a dedicated / bare metal.
- One raw disk (unformatted, unpartitioned) for the thin-pool that k7 will provision for efficient disk usage of sandboxes.
- **kfd only:** one raw disk (unformatted, unpartitioned) for the thin-pool. Other backends (`kql`, `k7d`) install without a spare drive. `k7 install` does not imply kfd.
- Use `./utils/wipe-disk.sh /your/disk` to wipe a disk clean before provisioning. DANGER: destructive - it will remove data/partitions/formatting/SWRAID.
- Ansible (for installer):
```bash
@@ -174,7 +174,7 @@ The **`.deb` / PPA package is Linux-only** (amd64/arm64). On a MacBook:
- **CLI from source:** `./src/k7/cli/dev.sh` (same commands as `k7`; uses `uv` + `PYTHONPATH=src`)
- **API client from laptop:** set `K7_API_URL` and `K7_API_KEY`, then `dev.sh create` / `dev.sh list` (no `--core`)
- **`k7 install`** targets Linux servers with KVM — run on the node or via SSH, not on macOS locally
- **`pip install k7-sdk`** for Python scripts only
- **`pip install k7-sdk==0.4.0`** for Python scripts only
Do not install the Ubuntu `.deb` on macOS.
@@ -183,21 +183,26 @@ Do not install the Ubuntu `.deb` on macOS.
### Get your node(s) ready
The Launchpad PPA currently publishes **0.2.2**. For **0.3.1** (HTTPS API,
`--docker`, k7d 0.6.0, HA `k7d-fc` copy) install the GitHub release `.deb`,
then clone the matching source — `k7 install` builds `k7-api:local` from
the current working directory:
The Launchpad PPA publishes **0.4.0**. Install the CLI, then clone the matching
source — `k7 install` builds `k7-api:local` from the current working directory:
```shell
curl -fsSL -O https://github.com/Katakate/k7/releases/download/v0.3.1/k7_0.3.1_amd64.deb
sudo apt install ./k7_0.3.1_amd64.deb
git clone --branch v0.3.1 https://github.com/Katakate/k7.git
sudo add-apt-repository ppa:katakate.org/k7
sudo apt update
sudo apt install k7
k7 -V # 0.4.0
# GitHub .deb is the same package if you prefer not to add the PPA:
# curl -fsSL -O https://github.com/Katakate/k7/releases/download/v0.4.0/k7_0.4.0_amd64.deb
# sudo apt install ./k7_0.4.0_amd64.deb
git clone --branch v0.4.0 https://github.com/Katakate/k7.git
cd k7
sudo apt install -y ansible
curl -fsSL https://get.docker.com | sh
```
Then let `k7` get your node ready:
Then let `k7` get your node ready. **`--backend` is required** (no silent
default). kfd needs a spare raw disk; kql uses Longhorn on the OS disk; k7d
is the warm-fork daemon. `none` is control-plane only (no sandbox runtime):
```console
$ k7 install --backend kfd,kql,k7d
@@ -215,11 +220,13 @@ Optionally pass `-v` for a verbose output.
> [tutorials/k7_hetzner_node_setup.md](tutorials/k7_hetzner_node_setup.md)
> (this file is also in public [Katakate/k7](https://github.com/Katakate/k7)).
>
> The playbook pins k7d **0.6.0**. `--docker` needs the guest docker
> The playbook pins k7d **0.7.0**. `--docker` needs the guest docker
> service (payload on the node); an older k7d fails loudly with
> `this k7d has no docker service; upgrade`. Multi-node inventory shapes
> (2-node server+agent, 3-node `--ha`) are in
> `src/k7/deploy/inventory.ini.example`.
> `src/k7/deploy/inventory.ini.example`. Every inventory host must set
> `k7_backends` (or inherit group vars). Use `k7_backends=none` on
> `[k7_servers]` when agents run the sandboxes — empty/omitted is an error.
`k7 install` serves `k7-api` on NodePort `31007` over HTTPS. The default
is a playbook-minted cluster CA (Let's Encrypt cannot issue for a bare
@@ -355,8 +362,11 @@ separate "start" step.
k7 api status
k7 api endpoint
# Generate API key
# Generate API key (optionally pin to a namespace and/or node)
k7 generate-api-key my-key1
k7 nodes list # NAME is the Linux hostname K3s registered; not an inventory label
k7 generate-api-key tenant-a -n tenant-a --node k7-node-01
k7 nodes dedicate k7-node-01 --tenant acme
# Temporarily disable / re-enable
k7 api disable
@@ -373,12 +383,12 @@ After your k7 API is up, usage is very simple.
Install the Python SDK via:
```shell
pip install k7-sdk
pip install k7-sdk==0.4.0
```
Or if you want async support:
```shell
pip install "k7-sdk[async]"
pip install "k7-sdk[async]==0.4.0"
```
The legacy `katakate` PyPI name remains as a one-release shim that re-exports `k7_sdk` with a deprecation warning.
ROADMAP.md
+3 -4
View File
@@ -7,8 +7,7 @@ Where **K7** is headed — for contributors and operators.
## Current focus
Release engineering: keep the apt/PPA, GHCR `k7-api`, and PyPI `k7-sdk`
pipelines current with `main` (the PPA is still 0.2.2; HTTPS, `--docker`,
and `k7d-fc` are unreleased).
pipelines current with `main` (PPA, GitHub `.deb`, and PyPI `k7-sdk` are **0.4.0**).
---
@@ -24,7 +23,7 @@ and `k7d-fc` are unreleased).
- [x] Firecracker jailer
- [x] `k7d-fc` backend (k7d driving stock Firecracker + jailer)
- [x] Python SDK as **`k7-sdk`** / `k7_sdk` (`katakate` deprecated)
- [x] `k7d` backend install path (public `Katakate/k7d` GitHub Releases; playbook pin 0.6.0)
- [x] `k7d` backend install path (public `Katakate/k7d` GitHub Releases; playbook pin 0.7.0)
- [x] HTTPS-by-default for `k7-api` (Caddy sidecar, cluster CA)
- [x] Network security hardening: cluster-wide sandbox→platform
isolation, opt-in sandbox ingress / `--expose-port`, Hubble, and a
@@ -35,7 +34,7 @@ and `k7d-fc` are unreleased).
## Next goals
- [ ] PPA / GHCR / PyPI cut of the unreleased work (HTTPS, `--docker`, `k7d-fc`)
- [x] PPA / GHCR / PyPI cut of the unreleased work (HTTPS, `--docker`, `k7d-fc`)
- [ ] Optional macOS CLI artifacts (tarball / Homebrew) — after the above
---
SECURITY.md
+48 -5
View File
@@ -51,11 +51,26 @@ Do **not** open a public issue for security-sensitive reports.
- The control plane API uses API keys with hashed storage and expiry
(file-backed by default at `/etc/k7/api_keys.json` — rotate and protect
that file). Keys may optionally be scoped to one or more namespaces
(`k7 generate-api-key -n <ns>`); absent/empty scope keeps the historical
unrestricted behaviour (backward compatible). Scoped keys are enforced
on every namespace-bearing and cluster-scoped control-plane route —
they cannot list across all namespaces, touch namespaces outside their
list, or read cluster-wide topology (e.g. ``GET /api/v1/nodes/storage``).
(`k7 generate-api-key -n <ns>`); they may also be scoped to one or more
Kubernetes nodes (`k7 generate-api-key --node <node>`). Absent/empty
scope keeps the historical unrestricted behaviour (backward compatible).
Scoped keys are enforced on every namespace-bearing and cluster-scoped
control-plane route — they cannot list across all namespaces, touch
namespaces outside their list, or read cluster-wide topology (e.g.
``GET /api/v1/nodes/storage``). A **node-scoped** key may only
create/restore/fork sandboxes onto its listed nodes (a single-node
scope auto-pins; several nodes require an explicit ``node_name``).
Namespace scope is the control-plane tenancy boundary; node scope is
the placement boundary (API-key `--node` is a Kubernetes node name
from `k7 nodes list` / `kubectl get nodes` — the Linux hostname K3s
registered — joined via the kubelet-stamped `kubernetes.io/hostname`
label; inventory.ini does not write that label). Together they
keep a tenant off another tenant's k7d daemon **only if that node is
also dedicated** (`k7 nodes dedicate NODE --tenant ID` writes
`k7.katakate.org/tenant` as a label and a NoSchedule taint). Without
the taint, an unscoped key can still land on the same node. Unscoped
keys still have full cross-namespace control-plane access and
unrestricted placement.
- **The control-plane API is on a public NodePort (`31007`) on a public
node, over HTTPS by default**. A Caddy sidecar terminates TLS; the API
container and its probes stay HTTP on `:8000`. The default cert is a
@@ -117,6 +132,21 @@ Do **not** open a public issue for security-sensitive reports.
the node and the API server.
- **Multi-node** clusters are supported (Ansible inventory; Longhorn for
the QEMU/`kql` path). Cilium FQDN egress applies cluster-wide.
- **Root on a K3s server is full cluster control.** Nodes in
`[k7_servers]` (including the 3-node `--ha` example, where every
host is a server) have `/etc/rancher/k3s/k3s.yaml` as **cluster-admin**
(playbook `--write-kubeconfig-mode 644`), the join token, and etcd.
The first master also has `k7-api` and `/etc/k7/api_keys.json`. Node
pins and `k7 nodes dedicate` do **not** contain that: they stop two
tenants sharing a k7d daemon, they do not survive a VM escape onto
a master. Put tenant sandboxes on `[k7_agents]`; do not dedicate a
tenant onto a server running `k7-api`. Root on an **agent** is not
the admin kubeconfig. `k7-agent` has no ServiceAccount token and
ingress from `remote-node` is denied, so that breakout owns **that
node's** k7d (that tenant, if dedicated) — not other agents and not
the Kubernetes API. Per-node isolation is the blast radius for a
worker compromise; it is not useless. The shared agent token remains
on every node as defence-in-depth after the CNP.
See also the docs: security model, networking, and backends comparison.
@@ -131,11 +161,24 @@ See also the docs: security model, networking, and backends comparison.
- API key storage is local file-backed; treat the API host as trusted.
Namespace scoping is an opt-in tenancy boundary on top of that model —
unscoped keys still have full cross-namespace control-plane access.
Node scoping is the matching opt-in **placement** boundary: without
`--node` on the key, two namespaces can still share a node's k7d
daemon. Pair `-n` and `--node`, and run `k7 nodes dedicate` on that
node, for tenant isolation on k7d. The dedicate step is a label +
taint (`k7.katakate.org/tenant`); a hostname pin alone does not keep
other sandboxes off the node. Neither contains a host compromise:
root on `[k7_servers]` is cluster-admin.
- Young project; no independent security audit yet.
- The `k7d` backend has a different isolation trade-off for CoW sibling
forks — see k7d's `SECURITY.md`.
- Prefer a dedicated RBAC-restricted kubeconfig for the API rather than
cluster-admin credentials in production.
- **Host compromise is cluster-wide on servers.** `--node` / `dedicate`
isolate k7d placement, not a root shell on `[k7_servers]`. The HA
inventory example makes every node a server. Tenant boxes belong in
`[k7_agents]`. `k7-agent` has no ServiceAccount token and no
`remote-node` ingress, so root on an agent is that node (that tenant
if dedicated), not k7-api RBAC on the rest of the cluster.
## Responsible Disclosure
debian/changelog
Vendored
+6
View File
@@ -1,3 +1,9 @@
k7 (0.4.0) noble; urgency=medium
* Release 0.4.0.
-- Katakate <hi@katakate.org> Sat, 19 Sep 2026 22:58:56 +0200
k7 (0.3.1) noble; urgency=medium
* Release 0.3.1.
docs/BACKENDS.md
+32 -14
View File
@@ -5,6 +5,10 @@ hardware-isolated microVM. *How* that VM is built, stored, snapshotted, and
forked is the backend's job. Four backends exist today; a node can install
any combination (`k7 install --backend kfd,kql,k7d,k7d-fc`) and each sandbox picks
one (`k7 create --backend …`, annotation `k7.katakate.org/backend`).
`--backend` / inventory `k7_backends` is **required** — there is no default
list. `none` means no sandbox runtime on that node (scheduling-only
master: `[k7_servers:vars] k7_backends=none`). Empty/omitted is an error,
not kfd.
| | `kfd` — kata-firecracker-devmapper | `kql` — kata-qemu-longhorn | `k7d` | `k7d-fc` |
|---|---|---|---|---|
@@ -58,7 +62,7 @@ CLI.
You lose virtiofs/hostPath and time-warp. See k7d
`SECURITY.md` "Firecracker profile" and `docs/backends.md`. Kata's
Firecracker (`kfd`) is a different binary (`/opt/kata/bin`, older pin);
k7d-fc installs upstream v1.16.1 at `/usr/local/bin`.
k7d-fc installs upstream v1.16.2 at `/usr/local/bin`.
## The k7d backend
@@ -76,7 +80,7 @@ The Ansible playbook:
clone writable volume images with `FICLONE` reflinks);
3. downloads the k7d release tarball (`k7d_artifact_url`, default the
public `Katakate/k7d` GitHub release for `k7d_version`, currently
**0.6.0**) and runs the bundled `install.sh`, which installs `k7d` +
**0.7.0**) and runs the bundled `install.sh`, which installs `k7d` +
`containerd-shim-k7-v1` into `/usr/local/bin`, guest kernel/initramfs
into `/usr/local/share/k7d`, and starts `k7d.service` (control socket
`/run/k7d/k7d.sock`). Override with `k7 install --k7d-version <ver>`
@@ -171,19 +175,33 @@ node**:
- A VM op on a sandbox co-located with the k7-api pod runs directly (the
deployment mounts both sockets and ships `crictl`).
- A VM op on a sandbox on any OTHER node is forwarded to the k7-agent pod
on that node (`POST /agent/v1/vm/{pause,resume,fork,lookup}` on the pod
IP). Forwarding authenticates with the shared token the install playbook
on that node (`POST /agent/v1/vm/{pause,resume,lookup}` on the pod
IP). The agent does **not** create Deployments or patch nodes
(no ServiceAccount token). Fork Kubernetes writes stay on `k7-api`.
Forwarding authenticates with the token the install playbook
writes to `/etc/k7/agent_token` (root, 0600) on every node; a
CiliumNetworkPolicy limits pod-originated agent ingress to the k7-api
pod. No Ready agent on the node / missing token → loud error, never a
silent no-op.
- **CLI on a node** also works for any sandbox: local sandboxes talk to the
local daemon, remote ones are forwarded the same way (root can read the
token).
- A fork still **lands on the source's node** — the agent proxies to the
node-local daemon; the cross-node fork *data path* is future k7d work
(daemon side, not built yet). When it lands, only the forwarding target
changes.
CiliumNetworkPolicy allows agent ingress from the k7-api pod and
local `host` only (`remote-node` denied). No Ready agent / missing
token → loud error, never a silent no-op.
- **CLI on a node** talks to the local daemon for local sandboxes.
Remote VM ops go through `k7-api` (default CLI path). `k7 --core`
cannot host-forward to another node's agent.
- A fork still **lands on the source's node** (`spec.nodeName` on the
fork Deployment). k7-api creates that Deployment after an agent
lookup; CoW is node-local on that k7d daemon. Cross-node fork *data
path* is future k7d work (daemon side, not built yet).
k7d is one daemon per node. Namespace scoping on API keys does not stop
two tenants sharing that daemon. Pin a tenant with
`k7 generate-api-key tenant-a -n tenant-a --node <node>` so creates
cannot *choose* another node. `<node>` is the Kubernetes Node name
(`k7 nodes list` / `kubectl get nodes`) — the Linux hostname K3s
registered. The pin is a `nodeSelector` on `kubernetes.io/hostname`
(kubelet stamps that label; `inventory.ini` does not). Inventory
`k7_backends` only stamps `k7.katakate.org/backend-*` at install.
To keep **other** sandboxes off that node, also run
`k7 nodes dedicate <node> --tenant <id>` (label + NoSchedule taint
`k7.katakate.org/tenant`). See `SECURITY.md`.
kql pause/resume/fork have none of these constraints (they are pure
Kubernetes/Longhorn operations) and work through the API for any node.
examples/k7.yaml
+6 -3
View File
@@ -1,10 +1,13 @@
name: demo
image: alpine:3.20
namespace: default
# node_name: k7-node-01 # copy NAME from `k7 nodes list` (Linux hostname); dedicate to lock others out
backend: k7d
limits:
cpu: "100m"
memory: "128Mi"
cpu: "1"
memory: "1Gi"
before_script: |
# Installing curl. Egress open during before_script, then restricted (empty whitelist) afterwards
# Installing curl. Egress is open during before_script, then restricted
# (empty whitelist) afterwards. 128Mi is too small for a VM sandbox.
apk add curl
egress_whitelist: []
pyproject.toml
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "k7"
version = "0.3.1"
version = "0.4.0"
description = "Self-hosted VM sandboxes for untrusted and AI code (CLI, API, SDK)"
readme = "README.md"
requires-python = ">=3.10.11"
setup.py
+1 -1
View File
@@ -4,7 +4,7 @@ from setuptools import find_packages, setup
setup(
name="k7-sdk",
version="0.3.1",
version="0.4.0",
description="K7 sandbox management Python SDK (HTTP client for the k7 API)",
long_description=Path(__file__).with_name("README.md").read_text(),
long_description_content_type="text/markdown",
src/k7/__init__.py
+1 -1
View File
@@ -1,3 +1,3 @@
"""K7 Sandbox Management System"""
__version__ = "0.3.1"
__version__ = "0.4.0"
src/k7/api/agent.py
+58 -39
View File
@@ -2,21 +2,19 @@
Runs as a DaemonSet (``k7-agent``, kube-system) on every node, reusing the
``k7-api:local`` image with an overridden command
(``uvicorn k7.api.agent:app``). It exposes ONLY the node-local operations
that the centralized k7-api pod cannot perform for sandboxes on other
nodes:
(``uvicorn k7.api.agent:app``). It exposes ONLY node-local operations
that need the k7d socket / crictl / ``lvs`` — never Kubernetes writes:
- ``POST /agent/v1/vm/{pause,resume,fork,lookup}`` — thin wrappers over the
``K7Core`` k7d helpers (the k7d daemon socket, containerd socket, and
crictl are all node-local).
- ``GET /agent/v1/storage`` — node storage-pool utilization (kfd LVM
thin-pool via ``lvs``, k7d disks pool via ``df``).
- ``POST /agent/v1/vm/{pause,resume,lookup}`` — crictl + k7d daemon.
- ``GET /agent/v1/storage`` — kfd thin-pool / k7d disks utilization.
Auth: every request must carry the shared agent token (header
``X-K7-Agent-Token``) generated by the install playbook and distributed to
``/etc/k7/agent_token`` on every node. A CiliumNetworkPolicy additionally
restricts pod-originated ingress to the k7-api pod. Requests without a
valid token are rejected — never served.
Fork, pause annotations, and NetworkPolicy stay on ``k7-api`` (ClusterRole).
The agent ServiceAccount has no token (``automountServiceAccountToken:
false``), so root on an agent cannot steal cluster-admin-equivalent RBAC.
Auth: ``X-K7-Agent-Token`` from ``/etc/k7/agent_token``. A CiliumNetworkPolicy
allows ingress from the k7-api pod and the local ``host`` (kubelet probes).
``remote-node`` is denied so a compromised agent cannot call other agents.
"""
import json
@@ -29,6 +27,7 @@ from fastapi.responses import JSONResponse
from .. import __version__
from ..core.core import K7Core
from ..core.models import OperationResult
app = FastAPI(title="K7 Node Agent", version=__version__)
@@ -38,7 +37,7 @@ K7D_DISKS_DIR = os.getenv("K7D_DISKS_DIR", "/var/lib/k7d/disks")
def _load_agent_token() -> str:
"""Read the shared agent token — unreadable/empty is a deployment bug
"""Read the agent token — unreadable/empty is a deployment bug
and must fail loudly (a silent 401 would be misdiagnosed as a bad
caller token)."""
try:
@@ -64,7 +63,6 @@ async def verify_agent_token(x_k7_agent_token: str | None = Header(None)):
@app.exception_handler(Exception)
async def unhandled_exception_handler(request: Request, exc: Exception): # type: ignore[override]
# Fail loud WITH the message — the k7-api forwarder surfaces it verbatim.
return JSONResponse(
content={"error": {"code": "InternalServerError", "message": str(exc)}},
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
@@ -84,44 +82,65 @@ def _required_name(body: dict | None) -> tuple[str, str]:
return name, body.get("namespace", "default")
def _required_pod_name(body: dict | None) -> str:
pod_name = (body or {}).get("pod_name")
if not pod_name or not isinstance(pod_name, str):
raise HTTPException(status_code=400, detail="pod_name is required")
return pod_name
async def _lookup_local_vm(pod_name: str, namespace: str) -> dict:
"""crictl + k7d only — no Kubernetes client."""
core = K7Core()
cri_sandbox_id = await core._k7d_cri_sandbox_id(pod_name, namespace)
resp = await core._k7d_request({"op": "lookup_sandbox", "sandbox_id": cri_sandbox_id})
if resp.get("status") != "sandbox_found":
raise RuntimeError(f"unexpected k7d lookup_sandbox response: {resp}")
resp["sandbox_id"] = cri_sandbox_id
return resp
@app.post("/agent/v1/vm/pause", dependencies=[Depends(verify_agent_token)])
async def vm_pause(body: dict | None = None):
name, namespace = _required_name(body)
result = await K7Core().pause_sandbox(name=name, namespace=namespace, snapshot_name=(body or {}).get("snapshot"))
return result.to_dict()
_name, namespace = _required_name(body)
pod_name = _required_pod_name(body)
vm = await _lookup_local_vm(pod_name, namespace)
await K7Core()._k7d_request({"op": "pause_vm", "vm_id": vm["vm_id"]})
return OperationResult(
success=True,
message=f"k7d VM {vm['vm_id']} frozen in place; memory retained, vCPUs stopped.",
).to_dict()
@app.post("/agent/v1/vm/resume", dependencies=[Depends(verify_agent_token)])
async def vm_resume(body: dict | None = None):
name, namespace = _required_name(body)
result = await K7Core().resume_sandbox(name=name, namespace=namespace)
return result.to_dict()
_name, namespace = _required_name(body)
pod_name = _required_pod_name(body)
vm = await _lookup_local_vm(pod_name, namespace)
await K7Core()._k7d_request({"op": "resume_vm", "vm_id": vm["vm_id"]})
return OperationResult(
success=True,
message=f"k7d VM {vm['vm_id']} vCPUs restarted.",
).to_dict()
@app.post("/agent/v1/vm/fork", dependencies=[Depends(verify_agent_token)])
async def vm_fork(body: dict | None = None):
name, namespace = _required_name(body)
new_name = (body or {}).get("new_name")
if not new_name or not isinstance(new_name, str):
raise HTTPException(status_code=400, detail="new_name is required")
result = await K7Core().fork_sandbox(
source_name=name,
new_name=new_name,
namespace=namespace,
snapshot_name=(body or {}).get("snapshot"),
async def vm_fork(_body: dict | None = None):
raise HTTPException(
status_code=400,
detail=(
"k7d fork is a control-plane operation (k7-api ClusterRole creates the "
"Deployment). The agent only runs lookup/pause/resume/storage so a "
"compromised worker cannot create sandboxes on other nodes."
),
)
return result.to_dict()
@app.post("/agent/v1/vm/lookup", dependencies=[Depends(verify_agent_token)])
async def vm_lookup(body: dict | None = None):
name, namespace = _required_name(body)
return await K7Core().lookup_k7d_vm(name, namespace)
# ---------------------------------------------------------------------------
# Node storage-pool utilization.
# ---------------------------------------------------------------------------
_name, namespace = _required_name(body)
pod_name = _required_pod_name(body)
return await _lookup_local_vm(pod_name, namespace)
def _run(cmd: list[str]) -> str:
src/k7/api/main.py
+142 -39
View File
@@ -3,6 +3,7 @@ import json
import os
import secrets
import time
from collections.abc import AsyncIterator
from datetime import timedelta
from pathlib import Path
from typing import Any
@@ -11,7 +12,7 @@ from fastapi import Depends, FastAPI, Header, HTTPException, Request, status
from fastapi.responses import JSONResponse
from .. import __version__
from ..core.core import K7Core
from ..core.core import K7Core, aclose_k7_core
from ..core.models import SandboxConfig
app = FastAPI(title="K7 Sandbox API", version=__version__)
@@ -142,6 +143,34 @@ def authorize_namespace(
)
def apply_node_scope(key_data: dict, node_name: str | None) -> str | None:
"""Enforce optional per-key node placement and return the pin.
Absent/empty ``nodes`` on the key ⇒ unrestricted (backward compatible).
A caller-supplied ``node_name`` is kept for operator/test pinning.
Scoped keys may only place sandboxes on listed nodes; a single-node
scope auto-pins when ``node_name`` is omitted; a multi-node scope
requires an explicit allowed node (fail loud, never silently pick).
"""
allowed = [n for n in (key_data.get("nodes") or []) if n]
requested = node_name.strip() if isinstance(node_name, str) and node_name.strip() else None
if not allowed:
return requested
if requested is None:
if len(allowed) == 1:
return allowed[0]
raise HTTPException(
status_code=403,
detail="API key is node-scoped; pass an explicit allowed node",
)
if requested not in allowed:
raise HTTPException(
status_code=403,
detail=f"API key is not authorized for node '{requested}'",
)
return requested
def success_response(
data: Any, status_code: int = status.HTTP_200_OK, headers: dict[str, str] | None = None
) -> JSONResponse:
@@ -152,6 +181,15 @@ def error_response(code: str, message: str, status_code: int) -> JSONResponse:
return JSONResponse(content={"error": {"code": code, "message": message}}, status_code=status_code)
async def get_k7_core() -> AsyncIterator[K7Core]:
"""Per-request K7Core; close the kube aiohttp session when the handler ends."""
core = K7Core()
try:
yield core
finally:
await aclose_k7_core(core)
@app.exception_handler(HTTPException)
async def http_exception_handler(request: Request, exc: HTTPException): # type: ignore[override]
# Map common status codes to generic error codes
@@ -188,12 +226,12 @@ async def health():
@app.post("/api/v1/sandboxes")
async def create_sandbox(config: dict, key_data: dict = Depends(verify_api_key)):
async def create_sandbox(config: dict, key_data: dict = Depends(verify_api_key), core: K7Core = Depends(get_k7_core)):
"""Create a new sandbox."""
try:
sandbox_config = SandboxConfig.from_dict(config)
authorize_namespace(key_data, sandbox_config.namespace)
core = K7Core()
sandbox_config.node_name = apply_node_scope(key_data, sandbox_config.node_name)
result = await core.create_sandbox(sandbox_config)
if result.success:
@@ -216,19 +254,26 @@ async def create_sandbox(config: dict, key_data: dict = Depends(verify_api_key))
@app.get("/api/v1/sandboxes")
async def list_sandboxes(namespace: str | None = None, key_data: dict = Depends(verify_api_key)):
async def list_sandboxes(
namespace: str | None = None,
key_data: dict = Depends(verify_api_key),
core: K7Core = Depends(get_k7_core),
):
"""List all sandboxes."""
authorize_namespace(key_data, namespace)
core = K7Core()
sandboxes = await core.list_sandboxes(namespace)
return success_response([sandbox.to_dict() for sandbox in sandboxes])
@app.get("/api/v1/sandboxes/{name}")
async def get_sandbox(name: str, namespace: str = "default", key_data: dict = Depends(verify_api_key)):
async def get_sandbox(
name: str,
namespace: str = "default",
key_data: dict = Depends(verify_api_key),
core: K7Core = Depends(get_k7_core),
):
"""Get a single sandbox by name."""
authorize_namespace(key_data, namespace)
core = K7Core()
items = await core.list_sandboxes(namespace)
for s in items:
if s.name == name:
@@ -237,10 +282,14 @@ async def get_sandbox(name: str, namespace: str = "default", key_data: dict = De
@app.delete("/api/v1/sandboxes/{name}")
async def delete_sandbox(name: str, namespace: str = "default", key_data: dict = Depends(verify_api_key)):
async def delete_sandbox(
name: str,
namespace: str = "default",
key_data: dict = Depends(verify_api_key),
core: K7Core = Depends(get_k7_core),
):
"""Delete a sandbox."""
authorize_namespace(key_data, namespace)
core = K7Core()
result = await core.delete_sandbox(name, namespace)
if result.success:
@@ -250,10 +299,13 @@ async def delete_sandbox(name: str, namespace: str = "default", key_data: dict =
@app.delete("/api/v1/sandboxes")
async def delete_all_sandboxes(namespace: str = "default", key_data: dict = Depends(verify_api_key)):
async def delete_all_sandboxes(
namespace: str = "default",
key_data: dict = Depends(verify_api_key),
core: K7Core = Depends(get_k7_core),
):
"""Delete all sandboxes in a namespace."""
authorize_namespace(key_data, namespace)
core = K7Core()
result = await core.delete_all_sandboxes(namespace)
if result.success:
@@ -263,7 +315,12 @@ async def delete_all_sandboxes(namespace: str = "default", key_data: dict = Depe
@app.post("/api/v1/sandboxes/{name}/pause")
async def pause_sandbox(name: str, body: dict | None = None, key_data: dict = Depends(verify_api_key)):
async def pause_sandbox(
name: str,
body: dict | None = None,
key_data: dict = Depends(verify_api_key),
core: K7Core = Depends(get_k7_core),
):
"""Pause a sandbox (scale to 0) and optionally take a Longhorn VolumeSnapshot.
Body keys (all optional):
@@ -273,7 +330,6 @@ async def pause_sandbox(name: str, body: dict | None = None, key_data: dict = De
body = body or {}
namespace = body.get("namespace", "default")
authorize_namespace(key_data, namespace)
core = K7Core()
result = await core.pause_sandbox(
name=name,
namespace=namespace,
@@ -285,12 +341,16 @@ async def pause_sandbox(name: str, body: dict | None = None, key_data: dict = De
@app.post("/api/v1/sandboxes/{name}/resume")
async def resume_sandbox(name: str, body: dict | None = None, key_data: dict = Depends(verify_api_key)):
async def resume_sandbox(
name: str,
body: dict | None = None,
key_data: dict = Depends(verify_api_key),
core: K7Core = Depends(get_k7_core),
):
"""Resume a paused sandbox (scale back to 1)."""
body = body or {}
namespace = body.get("namespace", "default")
authorize_namespace(key_data, namespace)
core = K7Core()
result = await core.resume_sandbox(name=name, namespace=namespace)
if result.success:
return success_response({"message": result.message})
@@ -298,7 +358,12 @@ async def resume_sandbox(name: str, body: dict | None = None, key_data: dict = D
@app.post("/api/v1/sandboxes/{name}/fork")
async def fork_sandbox(name: str, body: dict, key_data: dict = Depends(verify_api_key)):
async def fork_sandbox(
name: str,
body: dict,
key_data: dict = Depends(verify_api_key),
core: K7Core = Depends(get_k7_core),
):
"""Fork a kata-qemu-longhorn sandbox into a new name with a cloned root disk.
Required body key: new_name. Optional: namespace, snapshot.
@@ -310,7 +375,17 @@ async def fork_sandbox(name: str, body: dict, key_data: dict = Depends(verify_ap
namespace = body.get("namespace", "default")
authorize_namespace(key_data, namespace)
snapshot = body.get("snapshot")
core = K7Core()
if key_data.get("nodes"):
items = await core.list_sandboxes(namespace)
source = next((s for s in items if s.name == name), None)
if source is None:
raise HTTPException(status_code=404, detail=f"Sandbox {name} not found in namespace {namespace}")
if not source.node:
raise HTTPException(
status_code=409,
detail=f"Sandbox {name} is not scheduled to a node yet",
)
apply_node_scope(key_data, source.node)
result = await core.fork_sandbox(
source_name=name,
new_name=new_name,
@@ -342,10 +417,10 @@ async def get_sandbox_logs(
tail: int = 200,
since: int = 0,
key_data: dict = Depends(verify_api_key),
core: K7Core = Depends(get_k7_core),
):
"""Read pod logs (snapshot; no streaming yet)."""
authorize_namespace(key_data, namespace)
core = K7Core()
result = await core.get_logs(
sandbox_name=name,
namespace=namespace,
@@ -367,6 +442,7 @@ async def exec_command(
command_data: dict,
namespace: str = "default",
key_data: dict = Depends(verify_api_key),
core: K7Core = Depends(get_k7_core),
):
"""Execute a command in a sandbox."""
authorize_namespace(key_data, namespace)
@@ -374,15 +450,13 @@ async def exec_command(
if not command:
raise HTTPException(status_code=400, detail="Command is required")
core = K7Core()
result = await core.exec_command(name, command, namespace)
return success_response(result.to_dict())
@app.post("/api/v1/install", dependencies=[Depends(verify_api_key)])
async def install_node(install_data: dict):
async def install_node(install_data: dict, core: K7Core = Depends(get_k7_core)):
"""Install K7 on target hosts."""
core = K7Core()
result = core.install_node(
install_data.get("playbook"),
install_data.get("inventory"),
@@ -396,24 +470,31 @@ async def install_node(install_data: dict):
@app.get("/api/v1/nodes/storage")
async def get_nodes_storage(key_data: dict = Depends(verify_api_key)):
async def get_nodes_storage(key_data: dict = Depends(verify_api_key), core: K7Core = Depends(get_k7_core)):
"""Per-node storage-pool utilization (kfd thin-pool + k7d disks pool),
aggregated from the k7-agent DaemonSet. A node whose agent
is unreachable gets an ``{"error": ...}`` entry — never omitted.
Cluster-scoped (all-namespaces): a namespace-scoped key is 403.
Fail loud — do not silently narrow the per-node map.
Cluster-scoped (all-namespaces): a namespace-scoped or node-scoped
key is 403. Fail loud — do not silently narrow the per-node map.
"""
authorize_namespace(key_data, None, all_namespaces=True)
core = K7Core()
if key_data.get("nodes"):
raise HTTPException(
status_code=403,
detail="API key is not authorized for cluster-wide node operations",
)
return success_response(await core.nodes_storage())
@app.get("/api/v1/sandboxes/metrics")
async def get_sandbox_metrics(namespace: str | None = None, key_data: dict = Depends(verify_api_key)):
async def get_sandbox_metrics(
namespace: str | None = None,
key_data: dict = Depends(verify_api_key),
core: K7Core = Depends(get_k7_core),
):
"""Get resource usage metrics for sandboxes."""
authorize_namespace(key_data, namespace)
core = K7Core()
metrics = await core.get_sandbox_metrics(namespace)
return success_response(metrics)
@@ -443,10 +524,10 @@ async def list_snapshots(
sandbox: str | None = None,
kind: str | None = None,
key_data: dict = Depends(verify_api_key),
core: K7Core = Depends(get_k7_core),
):
"""List VolumeSnapshots, optionally filtered by namespace / sandbox / kind."""
authorize_namespace(key_data, None if all_namespaces else namespace, all_namespaces=all_namespaces)
core = K7Core()
snaps = await core.list_snapshots(
namespace=namespace,
all_namespaces=all_namespaces,
@@ -457,10 +538,14 @@ async def list_snapshots(
@app.get("/api/v1/snapshots/{name}")
async def get_snapshot(name: str, namespace: str = "default", key_data: dict = Depends(verify_api_key)):
async def get_snapshot(
name: str,
namespace: str = "default",
key_data: dict = Depends(verify_api_key),
core: K7Core = Depends(get_k7_core),
):
"""Inspect a single VolumeSnapshot by name."""
authorize_namespace(key_data, namespace)
core = K7Core()
snap = await core.get_snapshot(name, namespace=namespace)
if snap is None:
raise HTTPException(status_code=404, detail=f"Snapshot {name} not found in namespace {namespace}")
@@ -468,7 +553,12 @@ async def get_snapshot(name: str, namespace: str = "default", key_data: dict = D
@app.post("/api/v1/sandboxes/{name}/snapshot")
async def create_snapshot(name: str, body: dict, key_data: dict = Depends(verify_api_key)):
async def create_snapshot(
name: str,
body: dict,
key_data: dict = Depends(verify_api_key),
core: K7Core = Depends(get_k7_core),
):
"""Snapshot a running sandbox's root PVC without pausing it (kind=named).
Body keys: ``snapshot_name`` (required), ``namespace`` (default ``"default"``).
@@ -478,7 +568,6 @@ async def create_snapshot(name: str, body: dict, key_data: dict = Depends(verify
raise HTTPException(status_code=400, detail="snapshot_name is required")
namespace = body.get("namespace", "default")
authorize_namespace(key_data, namespace)
core = K7Core()
result = await core.create_snapshot(sandbox_name=name, snapshot_name=snapshot_name, namespace=namespace)
if result.success:
resource = {"name": snapshot_name, "namespace": namespace, "source_sandbox": name}
@@ -491,10 +580,14 @@ async def create_snapshot(name: str, body: dict, key_data: dict = Depends(verify
@app.delete("/api/v1/snapshots/{name}")
async def delete_snapshot(name: str, namespace: str = "default", key_data: dict = Depends(verify_api_key)):
async def delete_snapshot(
name: str,
namespace: str = "default",
key_data: dict = Depends(verify_api_key),
core: K7Core = Depends(get_k7_core),
):
"""Delete a VolumeSnapshot by name."""
authorize_namespace(key_data, namespace)
core = K7Core()
result = await core.delete_snapshot(name, namespace=namespace)
if result.success:
return success_response({"message": result.message})
@@ -504,7 +597,12 @@ async def delete_snapshot(name: str, namespace: str = "default", key_data: dict
@app.post("/api/v1/snapshots/{name}/restore")
async def restore_snapshot(name: str, body: dict, key_data: dict = Depends(verify_api_key)):
async def restore_snapshot(
name: str,
body: dict,
key_data: dict = Depends(verify_api_key),
core: K7Core = Depends(get_k7_core),
):
"""Boot a brand-new sandbox from a standalone VolumeSnapshot.
Body keys:
@@ -512,7 +610,8 @@ async def restore_snapshot(name: str, body: dict, key_data: dict = Depends(verif
``namespace`` (default ``"default"``),
``overrides`` (optional dict: image, backend, root_disk_size, sidecar,
limits, entrypoint, cmd, before_script),
``keep_snapshot`` (default ``true``).
``keep_snapshot`` (default ``true``),
``node_name`` (optional pin; a node-scoped key stamps or rejects this).
"""
body = body or {}
new_name = body.get("new_sandbox_name")
@@ -520,6 +619,7 @@ async def restore_snapshot(name: str, body: dict, key_data: dict = Depends(verif
raise HTTPException(status_code=400, detail="new_sandbox_name is required")
namespace = body.get("namespace", "default")
authorize_namespace(key_data, namespace)
pinned_node = apply_node_scope(key_data, body.get("node_name"))
keep_snapshot = bool(body.get("keep_snapshot", True))
overrides_dict = body.get("overrides") or {}
@@ -532,13 +632,13 @@ async def restore_snapshot(name: str, body: dict, key_data: dict = Depends(verif
overrides = SandboxConfigOverrides(**filtered) if filtered else None
core = K7Core()
result = await core.restore_sandbox(
snapshot_name=name,
new_sandbox_name=new_name,
namespace=namespace,
overrides=overrides,
keep_snapshot=keep_snapshot,
node_name=pinned_node,
)
if result.success:
resource = {
@@ -559,7 +659,11 @@ async def restore_snapshot(name: str, body: dict, key_data: dict = Depends(verif
@app.post("/api/v1/snapshots/gc")
async def gc_snapshots(body: dict | None = None, key_data: dict = Depends(verify_api_key)):
async def gc_snapshots(
body: dict | None = None,
key_data: dict = Depends(verify_api_key),
core: K7Core = Depends(get_k7_core),
):
"""Sweep stale ``kind=fork`` snapshots older than ``keep_fork_for``.
Body (all optional):
@@ -573,7 +677,6 @@ async def gc_snapshots(body: dict | None = None, key_data: dict = Depends(verify
namespace = body.get("namespace", "default")
authorize_namespace(key_data, None if all_namespaces else namespace, all_namespaces=all_namespaces)
keep_for = _parse_keep_fork_for(body.get("keep_fork_for"))
core = K7Core()
result = await core.gc_snapshots(
namespace=namespace,
all_namespaces=all_namespaces,
src/k7/api/snapshot_gc.py
+16 -13
View File
@@ -52,19 +52,22 @@ async def _main() -> int:
keep_minutes = _env_int("K7_GC_KEEP_FORK_FOR_MINUTES", 10)
dry_run = _env_bool("K7_GC_DRY_RUN", False)
core = K7Core()
result = await core.gc_snapshots(
all_namespaces=True,
keep_fork_for=timedelta(minutes=keep_minutes),
dry_run=dry_run,
)
if not result.success:
print(f"❌ snapshot-gc failed: {result.error}", file=sys.stderr)
return 1
print(result.message)
for record in result.data or []:
marker = "would-delete" if dry_run else ("deleted" if record.get("deleted") else "failed")
print(f" [{marker}] {record['namespace']}/{record['name']} (age={record['age']})")
return 0
try:
result = await core.gc_snapshots(
all_namespaces=True,
keep_fork_for=timedelta(minutes=keep_minutes),
dry_run=dry_run,
)
if not result.success:
print(f"❌ snapshot-gc failed: {result.error}", file=sys.stderr)
return 1
print(result.message)
for record in result.data or []:
marker = "would-delete" if dry_run else ("deleted" if record.get("deleted") else "failed")
print(f" [{marker}] {record['namespace']}/{record['name']} (age={record['age']})")
return 0
finally:
await core.aclose()
if __name__ == "__main__":
src/k7/cli/k7.py
+278 -90
View File
@@ -15,6 +15,7 @@ import time
from datetime import datetime, timedelta
from pathlib import Path
import requests
import typer
from click.core import ParameterSource
from rich.console import Console, Group
@@ -31,13 +32,31 @@ from rich.table import Table
from rich.text import Text
from k7 import __version__ as K7_VERSION
from k7.cli._client import CliContext, handle_api_call
from k7.cli._client import CliContext, _resolve_api_ca, _resolve_api_url, handle_api_call, verify_ssl_for_url
from k7.cli._config import config_app
from k7.core.core import K7Core
from k7.core.core import K7Core, aclose_k7_core
from k7.core.docker import parse_docker_disk
from k7.core.models import SandboxConfig, SandboxConfigOverrides
from k7.core.sidecar import SIDECAR_REGISTRY
def _core_run(core: K7Core, coro):
"""Run one K7Core coroutine and close its kube ApiClient.
Typer is sync and uses ``asyncio.run``, which tears down the loop when
the coroutine returns. kubernetes_asyncio binds aiohttp to that loop;
without ``aclose`` the interpreter logs ``Event loop is closed``.
"""
async def _run():
try:
return await coro
finally:
await aclose_k7_core(core)
return asyncio.run(_run())
app = typer.Typer(context_settings={"help_option_names": ["-h", "--help"]})
app.add_typer(config_app, name="config")
@@ -139,6 +158,19 @@ BACKEND_ALIASES = {
"k7": "k7d",
"k7-fc": "k7d-fc",
}
# Install-time empty set. Not a RuntimeClass — omit/blank is an error, not none.
BACKEND_NONE = "none"
INSTALL_BACKEND_REQUIRED = (
"Specify --backend (required; there is no default).\n"
" kfd Firecracker + jailer; needs a spare raw disk\n"
" kql Kata QEMU + Longhorn on the OS disk\n"
" k7d warm-fork microVM daemon (shared per node)\n"
" k7d-fc same daemon, Firecracker under the jailer\n"
" none no sandbox runtime on this node (control-plane only)\n"
"Example: k7 install --backend kfd,kql,k7d\n"
"Inventory (-i): every host needs k7_backends=… or k7_backends=none "
"(group vars are fine). Empty/omitted is not none."
)
# Deprecated short + pre-rename full names. Still accepted; emit a warning.
BACKEND_DEPRECATED_ALIASES = {
"fd": "kata-firecracker-devmapper",
@@ -177,12 +209,21 @@ def _parse_backends(value: str) -> list[str]:
Used by `k7 install` where multiple backends can be installed on a node.
Preserves first-seen order; raises typer.BadParameter on unknown entries.
``none`` is the empty set (no sandbox runtime); it cannot mix with others.
"""
if value is None:
raise typer.BadParameter("Backend value cannot be empty.")
raw = [p for p in (s.strip() for s in value.split(",")) if p]
if not raw:
raise typer.BadParameter("Backend value cannot be empty.")
lowered = [p.lower() for p in raw]
if BACKEND_NONE in lowered:
if lowered != [BACKEND_NONE]:
raise typer.BadParameter(
f"'{BACKEND_NONE}' cannot be combined with other backends. "
"Use only none, or a list of runtimes (kfd, kql, k7d, k7d-fc)."
)
return []
seen: builtins.list[str] = []
for entry in raw:
normalized = _normalize_backend(entry)
@@ -196,6 +237,44 @@ def _kubectl_cmd() -> list[str]:
return ["k3s", "kubectl"] if shutil.which("k3s") else ["kubectl"]
def _kubectl_run(kubectl: list[str], args: list[str]) -> subprocess.CompletedProcess[str] | None:
"""Run kubectl. ``None`` when the binary is missing (laptop / API-only host)."""
try:
return subprocess.run(kubectl + args, capture_output=True, text=True)
except FileNotFoundError:
return None
def _api_status_via_https() -> None:
"""Laptop path: probe ``GET /health`` on the configured API URL.
``k7 api status`` historically shelled out to kubectl, which crashes with
``FileNotFoundError`` on a machine that only has the .deb CLI. Fail loud
if nothing is configured; otherwise print reachability + the URL.
"""
url = _resolve_api_url(None)
if not url:
typer.echo("❌ kubectl is not installed, and no API URL is configured.")
typer.echo("On a cluster node, `k7 api status` uses kubectl.")
typer.echo("From a laptop, after `k7 config set api.url` / `api.ca` / `api.key`:")
typer.echo(" curl --cacert ./k7-ca.crt https://<node-ip>:31007/health")
raise typer.Exit(1)
ca = _resolve_api_ca(None)
verify = verify_ssl_for_url(url, ca)
try:
resp = requests.get(f"{url.rstrip('/')}/health", timeout=10, verify=verify)
except requests.RequestException as e:
typer.echo(f"❌ Could not reach K7 API at {url}: {e}", err=True)
raise typer.Exit(1) from e
if resp.ok:
typer.echo("✅ K7 API is reachable")
typer.echo(f"🌐 API endpoint: {url}")
typer.echo("\n📝 Replica counts and pod placement need kubectl on a cluster node.")
return
typer.echo(f"❌ K7 API at {url} returned HTTP {resp.status_code}", err=True)
raise typer.Exit(1)
def _build_default_inventory(
hosts: list[str] | None,
role: str,
@@ -206,13 +285,12 @@ def _build_default_inventory(
"""Build a minimal multi-group inventory when the user did not provide one.
Single-host shortcut: no `hosts` argument → localhost server with the
user-supplied backends. `backends` may contain one or both of
`kata-firecracker-devmapper` and `kata-qemu-longhorn`; the playbook will install
each backend's prerequisites only when listed.
user-supplied backends (``--backend`` is required; ``none`` →
``k7_backends=none``).
"""
server_lines: builtins.list[str] = []
agent_lines: builtins.list[str] = []
backends_csv = ",".join(backends)
backends_csv = ",".join(backends) if backends else BACKEND_NONE
has_devmapper = "kata-firecracker-devmapper" in backends
has_longhorn = "kata-qemu-longhorn" in backends
@@ -294,20 +372,23 @@ def _read_api_endpoint(kubectl: list[str]) -> str | None:
return written
except OSError:
pass
port_result = subprocess.run(
kubectl
+ [
"get",
"svc",
"k7-api",
"-n",
"kube-system",
"-o",
"jsonpath={.spec.ports[0].nodePort},{.spec.ports[0].targetPort}",
],
capture_output=True,
text=True,
)
try:
port_result = subprocess.run(
kubectl
+ [
"get",
"svc",
"k7-api",
"-n",
"kube-system",
"-o",
"jsonpath={.spec.ports[0].nodePort},{.spec.ports[0].targetPort}",
],
capture_output=True,
text=True,
)
except FileNotFoundError:
return _resolve_api_url(None)
if port_result.returncode != 0 or not port_result.stdout.strip():
return None
node_port, _, target_port = port_result.stdout.strip().partition(",")
@@ -365,22 +446,21 @@ def install(
"--disk",
help="Block device to use for LVM thin-pool (e.g., /dev/nvme2n1)",
),
backend: str = typer.Option(
"kata-firecracker-devmapper,kata-qemu-longhorn",
backend: str | None = typer.Option(
None,
"--backend",
"-b",
help=(
"Comma-separated sandbox backends to install on this node. "
"Choices: kata-firecracker-devmapper (kfd), kata-qemu-longhorn (kql), k7d, k7d-fc (alias k7-fc). "
"Default installs the two Kata backends; add k7d / k7d-fc explicitly for the "
"warm-fork microVM runtime (k7d-fc is Firecracker under the jailer)."
"Comma-separated sandbox backends to install. Required when not using "
"-i inventory (every inventory host must set k7_backends). "
"Choices: kfd, kql, k7d, k7d-fc, none. No default — omit is an error. "
"none means no sandbox runtime (scheduling-only node)."
),
show_default=True,
),
k7d_version: str | None = typer.Option(
None,
"--k7d-version",
help="k7d release version to install (k7d backend only; default from playbook: 0.6.0)",
help="k7d release version to install (k7d backend only; default from playbook: 0.7.0)",
),
k7d_artifact: str | None = typer.Option(
None,
@@ -481,15 +561,17 @@ def install(
):
"""Install K7 on target hosts using Ansible.
Single-node (default): `k7 install` provisions localhost with **both**
Kata backends (kata-firecracker-devmapper + kata-qemu-longhorn). Pass
`--backend kfd,kql,k7d` to also install the k7d warm-fork runtime
(downloads `Katakate/k7d` v0.6.0 unless `--k7d-version` / `--k7d-artifact`
override it). `--backend k7d-fc` (alias `k7-fc`) adds RuntimeClass
`k7-fc` and the pinned Firecracker+jailer next to that daemon.
Single-node: `k7 install --backend kfd,kql,k7d` (``--backend`` is required;
there is no default). ``none`` installs K3s with no sandbox runtime.
Pass ``k7d`` / ``k7d-fc`` for the warm-fork daemon (downloads
`Katakate/k7d` v0.7.0 unless `--k7d-version` / `--k7d-artifact` override
it). `--backend k7d-fc` (alias `k7-fc`) adds RuntimeClass `k7-fc` and
the pinned Firecracker+jailer next to that daemon.
Multi-node: `k7 install -i inventory.ini` reads roles, backends, and disks
from the Ansible inventory. See `inventory.ini.example` for the layout.
from the Ansible inventory. Every host must set ``k7_backends`` (or inherit
it from group vars); use ``k7_backends=none`` for scheduling-only masters.
Empty/omitted is not none. See `inventory.ini.example`.
Dual-NVMe boxes need the OS on one disk and a raw spare for kfd — see
`tutorials/k7_hetzner_node_setup.md` (`TWO_DISK=1`).
"""
@@ -565,11 +647,16 @@ def install(
with open(playbook) as f:
playbook_content = f.read()
backends_list = _parse_backends(backend)
inventory_file = inventory if inventory and os.path.exists(inventory) else None
backend_explicit = ctx.get_parameter_source("backend") == ParameterSource.COMMANDLINE
if not backend_explicit and not inventory_file:
typer.echo(INSTALL_BACKEND_REQUIRED, err=True)
raise typer.Exit(1)
backends_list = _parse_backends(backend) if backend_explicit and backend is not None else []
inventory_content = None
if inventory and os.path.exists(inventory):
with open(inventory) as f:
if inventory_file:
with open(inventory_file) as f:
inventory_content = f.read()
else:
inventory_content = _build_default_inventory(
@@ -641,7 +728,7 @@ def install(
# to min(3, node count). When the user supplied an inventory we can't
# know the node count up-front, so we let the playbook compute its own
# default (min(3, len(k7_cluster))) and only forward an explicit value.
inventory_user_supplied = bool(inventory)
inventory_user_supplied = bool(inventory_file)
node_count = len(hosts) if hosts else 1
if replicas is not None:
effective_replicas: int | None = replicas
@@ -666,11 +753,9 @@ def install(
# `k7_disk` is the legacy name; the playbook prefers `k7_devmapper_disk`
# but still falls back to `k7_disk` for backward compat.
"k7_disk": disk,
# `k7_backends` (plural, comma-separated) is authoritative; the
# playbook accepts a single `k7_backend` only as a back-compat
# fallback when neither inventory `k7_backends` nor extra-var
# `k7_backends` is set.
"k7_backends": ",".join(backends_list),
# `k7_backends` is required on every host. Empty extra-var would
# look like "omitted" in Ansible — send the `none` sentinel.
"k7_backends": ",".join(backends_list) if backends_list else BACKEND_NONE,
"longhorn_replicas": effective_replicas,
"longhorn_data_path": longhorn_data_path,
"longhorn_extra_disk": longhorn_extra_disk,
@@ -860,6 +945,11 @@ def create(
"--docker-disk",
help="Docker graph disk size for --docker (e.g. 20Gi, 40Gi). Default 20Gi on all backends.",
),
node: str | None = typer.Option(
None,
"--node",
help="Pin to this Kubernetes node name (`kubectl get nodes` / `k7 nodes list`). Joins via kubernetes.io/hostname (kubelet-stamped). YAML field: node_name.",
),
):
"""Create a new sandbox from YAML config or CLI arguments."""
# Three explicit egress modes. --egress-open → open
@@ -1009,6 +1099,9 @@ def create(
hint = others if others else "no non-docker sidecar types (--sidecar docker is --docker)"
raise typer.BadParameter(f"Unknown sidecar type '{sandbox_config.sidecar}'. Available: {hint}")
if node:
sandbox_config.node_name = node
core = K7Core()
progress = Progress(
SpinnerColumn(),
@@ -1417,12 +1510,12 @@ def create(
if cli_ctx.use_core:
group = RichGroup(status_text, details_text, progress)
with RichLive(group, refresh_per_second=8, transient=False) as live:
result = asyncio.run(core.create_sandbox(sandbox_config, progress_callback=on_progress))
result = _core_run(core, core.create_sandbox(sandbox_config, progress_callback=on_progress))
if not result.success:
typer.echo(f"❌ Failed to create sandbox: {result.error}", err=True)
raise typer.Exit(1)
endpoints = (result.data or {}).get("endpoints") or []
sandboxes_for_ready = asyncio.run(core.list_sandboxes(sandbox_config.namespace))
sandboxes_for_ready = _core_run(core, core.list_sandboxes(sandbox_config.namespace))
target_ready = any(s.name == sandbox_config.name and s.ready == "True" for s in sandboxes_for_ready)
else:
# The API doesn't stream progress yet (a separate spec). Print a
@@ -1436,7 +1529,7 @@ def create(
if sandbox_config.expose_ports:
# The SDK's create() returns a proxy, so the API path cannot show a
# resolved URL here. Point at the NodePorts instead of guessing one.
typer.echo(f"🌐 Exposed ports: see the NodePorts in k7 list --name {sandbox_config.name}")
typer.echo(f"🌐 Exposed ports: see the NodePorts column in `k7 list` (sandbox {sandbox_config.name})")
for endpoint in endpoints:
typer.echo(f"🌐 Port {endpoint['port']} exposed at {endpoint['url']}")
if target_ready:
@@ -1475,7 +1568,7 @@ def list(
cli_ctx: CliContext = ctx.obj
if cli_ctx.use_core:
core = K7Core()
sandboxes = [s.to_dict() for s in asyncio.run(core.list_sandboxes(namespace))]
sandboxes = [s.to_dict() for s in _core_run(core, core.list_sandboxes(namespace))]
else:
sandboxes = handle_api_call(lambda: cli_ctx.client().list(namespace=namespace))
@@ -1555,7 +1648,7 @@ def delete(
cli_ctx: CliContext = ctx.obj
if cli_ctx.use_core:
core = K7Core()
result = asyncio.run(core.delete_sandbox(name, namespace))
result = _core_run(core, core.delete_sandbox(name, namespace))
if not result.success:
typer.echo(f"❌ Failed to delete sandbox: {result.error}", err=True)
raise typer.Exit(1)
@@ -1581,7 +1674,7 @@ def delete_all(
# goes through the same routing as `k7 list` so off-cluster runs work.
if cli_ctx.use_core:
core = K7Core()
sandboxes = [s.to_dict() for s in asyncio.run(core.list_sandboxes(namespace))]
sandboxes = [s.to_dict() for s in _core_run(core, core.list_sandboxes(namespace))]
else:
sandboxes = handle_api_call(lambda: cli_ctx.client().list(namespace=namespace))
@@ -1600,7 +1693,7 @@ def delete_all(
if cli_ctx.use_core:
core = K7Core()
result = asyncio.run(core.delete_all_sandboxes(namespace))
result = _core_run(core, core.delete_all_sandboxes(namespace))
if result.success:
typer.echo(f"✅ {result.message}")
return
@@ -1711,7 +1804,7 @@ def exec(
cli_ctx: CliContext = ctx.obj
if cli_ctx.use_core:
core = K7Core()
result = asyncio.run(core.exec_command(name, joined, namespace=namespace))
result = _core_run(core, core.exec_command(name, joined, namespace=namespace))
data = result.to_dict()
else:
data = handle_api_call(lambda: cli_ctx.client().exec(name, joined, namespace=namespace))
@@ -1742,7 +1835,7 @@ def top(
table.add_column("CPU Usage (cores)")
table.add_column("Memory Usage (MiB)")
metrics_list = asyncio.run(core.get_sandbox_metrics(namespace))
metrics_list = _core_run(core, core.get_sandbox_metrics(namespace))
for metric in metrics_list:
sb_name = metric["name"]
@@ -1809,6 +1902,17 @@ def top(
time.sleep(refresh_interval)
def _dedupe_keep_order(values: builtins.list[str] | None) -> builtins.list[str]:
"""Preserve first-seen order; drop empties and duplicates."""
seen: set[str] = set()
out: builtins.list[str] = []
for v in values or []:
if v and v not in seen:
seen.add(v)
out.append(v)
return out
@app.command()
def generate_api_key(
name: str,
@@ -1819,6 +1923,11 @@ def generate_api_key(
"-n",
help="Restrict key to this namespace (repeatable). Omit for unrestricted access.",
),
node: builtins.list[str] | None = typer.Option(
None,
"--node",
help="Restrict key to this Kubernetes node name (repeatable; copy from `k7 nodes list`). Placement uses kubernetes.io/hostname (kubelet-stamped, not inventory). Omit for unrestricted placement.",
),
):
"""Generate a new API key."""
api_key = secrets.token_urlsafe(32)
@@ -1836,16 +1945,12 @@ def generate_api_key(
"expires": expiry_timestamp,
"last_used": None,
}
if namespace:
# Preserve order, drop empties/duplicates.
seen: set[str] = set()
scoped: builtins.list[str] = []
for ns in namespace:
if ns and ns not in seen:
seen.add(ns)
scoped.append(ns)
if scoped:
entry["namespaces"] = scoped
scoped_ns = _dedupe_keep_order(namespace)
if scoped_ns:
entry["namespaces"] = scoped_ns
scoped_nodes = _dedupe_keep_order(node)
if scoped_nodes:
entry["nodes"] = scoped_nodes
api_keys[key_hash] = entry
_write_api_keys(api_keys)
@@ -1857,6 +1962,10 @@ def generate_api_key(
typer.echo(f"Namespaces: {', '.join(entry['namespaces'])}")
else:
typer.echo("Namespaces: * (unrestricted)")
if entry.get("nodes"):
typer.echo(f"Nodes: {', '.join(entry['nodes'])}")
else:
typer.echo("Nodes: * (unrestricted)")
typer.echo("Keep this key secure - it won't be shown again!")
@@ -1877,6 +1986,7 @@ def list_api_keys():
table.add_column("Expires", style="yellow")
table.add_column("Last Used", style="green")
table.add_column("Namespaces", style="magenta")
table.add_column("Nodes", style="magenta")
for _key_hash, key_data in api_keys.items():
created = datetime.fromtimestamp(key_data["created"]).strftime("%Y-%m-%d %H:%M")
@@ -1886,8 +1996,10 @@ def list_api_keys():
last_used = datetime.fromtimestamp(key_data["last_used"]).strftime("%Y-%m-%d %H:%M")
namespaces = key_data.get("namespaces") or []
ns_col = "*" if not namespaces else ", ".join(namespaces)
nodes = key_data.get("nodes") or []
node_col = "*" if not nodes else ", ".join(nodes)
table.add_row(key_data["name"], created, expires, last_used, ns_col)
table.add_row(key_data["name"], created, expires, last_used, ns_col, node_col)
console.print(table)
@@ -1967,7 +2079,7 @@ def pause(
cli_ctx: CliContext = ctx.obj
if cli_ctx.use_core:
core = K7Core()
result = asyncio.run(core.pause_sandbox(name=name, namespace=namespace, snapshot_name=snapshot_name))
result = _core_run(core, core.pause_sandbox(name=name, namespace=namespace, snapshot_name=snapshot_name))
if not result.success:
typer.echo(f"❌ {result.error}", err=True)
raise typer.Exit(1)
@@ -1987,7 +2099,7 @@ def resume(
cli_ctx: CliContext = ctx.obj
if cli_ctx.use_core:
core = K7Core()
result = asyncio.run(core.resume_sandbox(name=name, namespace=namespace))
result = _core_run(core, core.resume_sandbox(name=name, namespace=namespace))
if not result.success:
typer.echo(f"❌ {result.error}", err=True)
raise typer.Exit(1)
@@ -2061,14 +2173,15 @@ def restore(
cli_ctx: CliContext = ctx.obj
if cli_ctx.use_core:
core = K7Core()
result = asyncio.run(
result = _core_run(
core,
core.restore_sandbox(
snapshot_name=snapshot_name,
new_sandbox_name=new_sandbox_name,
namespace=namespace,
overrides=overrides,
keep_snapshot=keep_snapshot,
)
),
)
if not result.success:
typer.echo(f"❌ {result.error}", err=True)
@@ -2106,13 +2219,14 @@ def fork(
typer.echo(f"Forked {source} -> {new_name}")
return
core = K7Core()
result = asyncio.run(
result = _core_run(
core,
core.fork_sandbox(
source_name=source,
new_name=new_name,
namespace=namespace,
snapshot_name=snapshot,
)
),
)
if result.success:
typer.echo(result.message)
@@ -2225,11 +2339,10 @@ def _api_scale_deployment(replicas: int) -> None:
failure mode this is meant to eliminate.
"""
kubectl = _kubectl_cmd()
result = subprocess.run(
kubectl + ["scale", "deployment", "k7-api", "-n", "kube-system", f"--replicas={replicas}"],
capture_output=True,
text=True,
)
result = _kubectl_run(kubectl, ["scale", "deployment", "k7-api", "-n", "kube-system", f"--replicas={replicas}"])
if result is None:
typer.echo("❌ `k7 api enable`/`disable` need kubectl on a cluster node.", err=True)
raise typer.Exit(1)
if result.returncode != 0:
typer.echo(f"❌ Failed to scale k7-api to {replicas}: {result.stderr.strip()}", err=True)
raise typer.Exit(1)
@@ -2239,9 +2352,9 @@ def _api_scale_deployment(replicas: int) -> None:
def api_status_cmd():
"""Show API server readiness, endpoint, and key-management hints."""
kubectl = _kubectl_cmd()
result = subprocess.run(
kubectl
+ [
result = _kubectl_run(
kubectl,
[
"get",
"deployment",
"k7-api",
@@ -2250,9 +2363,10 @@ def api_status_cmd():
"-o",
"jsonpath={.status.readyReplicas}/{.spec.replicas}",
],
capture_output=True,
text=True,
)
if result is None:
_api_status_via_https()
return
if result.returncode != 0:
typer.echo("❌ K7 API deployment not found in K3s")
typer.echo("Run `k7 install` to deploy it, or `k7 install --no-api` if that's deliberate.")
@@ -2284,7 +2398,10 @@ def api_status_cmd():
typer.echo("\n📝 SDK usage example:")
typer.echo(" from k7_sdk import Client")
typer.echo(" k7 = Client(endpoint='<endpoint>', api_key='<key>')")
typer.echo(" k7 = Client(")
typer.echo(" endpoint='<endpoint>', api_key='<key>',")
typer.echo(" verify_ssl='./k7-ca.crt', # cluster CA; on the node use /etc/k7/tls/ca.crt")
typer.echo(" )")
typer.echo(" sb = k7.create({'name': 'test', 'image': 'alpine:3.21'})")
typer.echo("\n🔐 Manage API keys:")
typer.echo(" k7 generate-api-key <name>")
@@ -2437,12 +2554,48 @@ def _deprecated_get_api_endpoint():
nodes_app = typer.Typer(
help="Query cluster nodes (storage pools, …).",
help="Query and dedicate cluster nodes (storage pools, tenant isolation).",
context_settings={"help_option_names": ["-h", "--help"]},
)
app.add_typer(nodes_app, name="nodes")
def _print_nodes_list_table(rows: builtins.list[dict]) -> None:
"""Render ``list_cluster_nodes()`` as NAME / HOSTNAME / BACKENDS / TENANT."""
if not rows:
typer.echo("No nodes found.")
return
table = Table(title="K7 Nodes")
table.add_column("NAME", style="cyan")
table.add_column("HOSTNAME")
table.add_column("BACKENDS")
table.add_column("TENANT")
for n in rows:
backends = ",".join(n.get("backends") or []) or "-"
table.add_row(n.get("name") or "-", n.get("hostname") or "-", backends, n.get("tenant") or "-")
Console().print(table)
@nodes_app.command("list")
def nodes_list(
as_json: bool = typer.Option(False, "--json", help="Print raw JSON instead of a table"),
):
"""Show cluster node names, hostname labels, backends, and tenant dedication.
NAME is what ``--node`` and ``k7 nodes dedicate`` take. It is the K3s
Node name (machine hostname), not something you label by hand.
HOSTNAME is ``kubernetes.io/hostname`` (kubelet). BACKENDS come from
``k7 install --backend`` / inventory ``k7_backends`` (``none`` means
no sandbox runtime). TENANT comes from ``k7 nodes dedicate``.
"""
core = K7Core()
rows = _core_run(core, core.list_cluster_nodes())
if as_json:
typer.echo(json.dumps(rows, indent=2, default=str))
return
_print_nodes_list_table(rows)
def _print_nodes_storage_table(data: dict) -> None:
"""Render ``nodes_storage()`` output as a per-node Rich table."""
if not data:
@@ -2483,7 +2636,8 @@ def nodes_storage(
"""Show per-node kfd thin-pool and k7d disks-pool utilization."""
cli_ctx: CliContext = ctx.obj
if cli_ctx.use_core:
data = asyncio.run(K7Core().nodes_storage())
core = K7Core()
data = _core_run(core, core.nodes_storage())
else:
data = handle_api_call(lambda: cli_ctx.client().nodes_storage())
if as_json:
@@ -2492,6 +2646,38 @@ def nodes_storage(
_print_nodes_storage_table(data)
@nodes_app.command("dedicate")
def nodes_dedicate(
node: str = typer.Argument(..., help="Kubernetes node name (metadata.name / hostname)"),
tenant: str = typer.Option(..., "--tenant", help="Tenant id written as k7.katakate.org/tenant"),
):
"""Label and taint a node so only that tenant's sandboxes can land there.
The label is the join key; the NoSchedule taint is the exclusive lock.
Must run with cluster access (on a node, or kubeconfig). Does not go
through k7-api.
"""
core = K7Core()
result = _core_run(core, core.dedicate_node(node, tenant))
if not result.success:
typer.echo(f"❌ {result.error}", err=True)
raise typer.Exit(1)
typer.echo(f"✅ {result.message}")
@nodes_app.command("undedicate")
def nodes_undedicate(
node: str = typer.Argument(..., help="Kubernetes node name (metadata.name / hostname)"),
):
"""Remove the k7 tenant label and taint from a node."""
core = K7Core()
result = _core_run(core, core.undedicate_node(node))
if not result.success:
typer.echo(f"❌ {result.error}", err=True)
raise typer.Exit(1)
typer.echo(f"✅ {result.message}")
# ``k7 snapshot …`` sub-app for VolumeSnapshot CRUD + GC.
# ---------------------------------------------------------------------------
@@ -2566,13 +2752,14 @@ def snapshot_list(
cli_ctx: CliContext = ctx.obj
if cli_ctx.use_core:
core = K7Core()
snap_objs = asyncio.run(
snap_objs = _core_run(
core,
core.list_snapshots(
namespace=namespace,
all_namespaces=all_namespaces,
sandbox=sandbox,
kind=kind,
)
),
)
snaps = [s.to_dict() for s in snap_objs]
else:
@@ -2597,7 +2784,7 @@ def snapshot_inspect(
cli_ctx: CliContext = ctx.obj
if cli_ctx.use_core:
core = K7Core()
snap = asyncio.run(core.get_snapshot(name, namespace=namespace))
snap = _core_run(core, core.get_snapshot(name, namespace=namespace))
if snap is None:
typer.echo(f"❌ Snapshot {name} not found in namespace {namespace}", err=True)
raise typer.Exit(1)
@@ -2621,7 +2808,7 @@ def snapshot_create(
cli_ctx: CliContext = ctx.obj
if cli_ctx.use_core:
core = K7Core()
result = asyncio.run(core.create_snapshot(sandbox_name, snapshot_name, namespace=namespace))
result = _core_run(core, core.create_snapshot(sandbox_name, snapshot_name, namespace=namespace))
if not result.success:
typer.echo(f"❌ {result.error}", err=True)
raise typer.Exit(1)
@@ -2650,7 +2837,7 @@ def snapshot_delete(
cli_ctx: CliContext = ctx.obj
if cli_ctx.use_core:
core = K7Core()
result = asyncio.run(core.delete_snapshot(name, namespace=namespace))
result = _core_run(core, core.delete_snapshot(name, namespace=namespace))
if not result.success:
typer.echo(f"❌ {result.error}", err=True)
raise typer.Exit(1)
@@ -2688,13 +2875,14 @@ def snapshot_gc(
else:
td = timedelta(seconds=int(keep_fork_for))
core = K7Core()
result = asyncio.run(
result = _core_run(
core,
core.gc_snapshots(
namespace=namespace,
all_namespaces=all_namespaces,
keep_fork_for=td,
dry_run=dry_run,
)
),
)
if not result.success:
typer.echo(f"❌ {result.error}", err=True)
src/k7/core/core.py
+439 -147
View File
@@ -1,5 +1,6 @@
import asyncio
import copy
import inspect
import ipaddress
import json
import math
@@ -34,10 +35,14 @@ from .docker import (
DIND_IMAGE,
DOCKER_HOST_URL,
DOCKER_UNSUPPORTED_K7D,
FSFREEZE_STAGED,
FSFREEZE_VEHICLE,
GRAPH_DEVICE_PATH,
KATA_FORK_GRAPH_REJECT,
KATA_GRAPH_MOUNT,
KATA_SOCKET_DIR,
KFD_DOCKER_STORAGE_CLASS,
KFD_FORK_REJECT,
STAGE_FSFREEZE_CMD,
VEHICLE_CONTAINER_NAME,
k7d_supports_docker,
parse_docker_disk,
@@ -63,6 +68,9 @@ K7D_ANN_FORK_SOURCE_VM = "k7d.katakate.org/fork-source-vm"
# Deployment-level marker k7 stamps so `k7 list` / resume can tell a
# VM-frozen (paused) k7d sandbox from a running one.
K7D_ANN_PAUSED = "k7.katakate.org/k7d-paused"
# Tenant isolation: label is the join key (nodeSelector), matching taint is
# the lock (NoSchedule). Same family as the backend labels.
K7_TENANT_LABEL = "k7.katakate.org/tenant"
# Message pointing users at k7d's native VM snapshot trees for the verbs
# that intentionally do NOT map onto the k7d backend.
K7D_SNAPSHOT_UNSUPPORTED = (
@@ -84,6 +92,8 @@ _ALLOWED_BACKENDS = (
"k7d-fc",
)
_K7D_FAMILY = frozenset({"k7d", "k7d-fc"})
# Kata refuses hypervisor memory below this (shim: "minimum required 256").
KATA_MIN_HYPERVISOR_MEMORY_MIB = 256
# CRDs (VolumeSnapshotContent) and PVC finalizer lists ignore strategic-merge
# patches. kubectl --type=merge works; the Python client must say so too.
_K8S_MERGE_PATCH = "application/merge-patch+json"
@@ -94,6 +104,16 @@ def _is_k7d_family(backend: str | None) -> bool:
return backend in _K7D_FAMILY
async def aclose_k7_core(core: object) -> None:
"""Close a K7Core kube session; no-op for test doubles that are not awaitable."""
closer = getattr(core, "aclose", None)
if closer is None:
return
result = closer()
if inspect.isawaitable(result):
await result
def _volume_is_hostpath_or_pvc(vol: dict[str, Any]) -> bool:
return any(k in vol for k in ("hostPath", "host_path", "persistentVolumeClaim", "persistent_volume_claim"))
@@ -176,6 +196,8 @@ class K7Core:
self._networking_v1_client = None
self._metrics_client = None
self._custom_objects_client = None
self._batch_v1_client = None
self._apiextensions_v1_client = None
self._config_loaded = False
self._api_client = None
@@ -212,41 +234,74 @@ class K7Core:
self._config_loaded = True
async def _get_api_client(self):
"""One kubernetes_asyncio ApiClient (aiohttp session) per K7Core."""
if self._api_client is None:
await self._load_k3s_config()
self._api_client = client.ApiClient()
return self._api_client
async def aclose(self) -> None:
"""Close the kube ApiClient before the event loop dies.
``asyncio.run`` shuts the loop when the coroutine returns. An open
kubernetes_asyncio session then logs ``Unclosed client session`` /
``Event loop is closed`` (and can wipe CLI stdout).
"""
api_client = self._api_client
self._api_client = None
self._apps_v1_client = None
self._core_v1_client = None
self._networking_v1_client = None
self._metrics_client = None
self._custom_objects_client = None
self._batch_v1_client = None
self._apiextensions_v1_client = None
if api_client is not None:
await api_client.close()
async def _get_apps_v1_client(self):
"""Get or create AppsV1Api client instance."""
if self._apps_v1_client is None:
await self._load_k3s_config()
self._apps_v1_client = client.AppsV1Api()
self._apps_v1_client = client.AppsV1Api(await self._get_api_client())
return self._apps_v1_client
async def _get_core_v1_client(self):
"""Get or create CoreV1Api client instance."""
if self._core_v1_client is None:
await self._load_k3s_config()
self._core_v1_client = client.CoreV1Api()
self._core_v1_client = client.CoreV1Api(await self._get_api_client())
return self._core_v1_client
async def _get_networking_v1_client(self):
"""Get or create NetworkingV1Api client instance."""
if self._networking_v1_client is None:
await self._load_k3s_config()
self._networking_v1_client = client.NetworkingV1Api()
self._networking_v1_client = client.NetworkingV1Api(await self._get_api_client())
return self._networking_v1_client
async def _get_metrics_client(self):
"""Get or create CustomObjectsApi client for metrics."""
if self._metrics_client is None:
await self._load_k3s_config()
self._metrics_client = client.CustomObjectsApi()
self._metrics_client = client.CustomObjectsApi(await self._get_api_client())
return self._metrics_client
async def _get_custom_objects_client(self):
"""Get or create CustomObjectsApi client for VolumeSnapshots."""
if self._custom_objects_client is None:
await self._load_k3s_config()
self._custom_objects_client = client.CustomObjectsApi()
self._custom_objects_client = client.CustomObjectsApi(await self._get_api_client())
return self._custom_objects_client
async def _get_batch_v1_client(self):
"""Get or create BatchV1Api client instance."""
if self._batch_v1_client is None:
self._batch_v1_client = client.BatchV1Api(await self._get_api_client())
return self._batch_v1_client
async def _get_apiextensions_v1_client(self):
"""Get or create ApiextensionsV1Api client instance."""
if self._apiextensions_v1_client is None:
self._apiextensions_v1_client = client.ApiextensionsV1Api(await self._get_api_client())
return self._apiextensions_v1_client
def _load_persist_bind_script(self) -> str:
"""Load the bind-mount wrapper script from package assets."""
try:
@@ -498,24 +553,133 @@ class K7Core:
cmd = list(override_cmd) if override_cmd is not None else list(image_cmd)
return ep + cmd
async def _list_backends_per_node(self) -> dict[str, list[str]]:
"""Return {node_name: [supported backends]} from k7 backend labels."""
def _backends_from_node_labels(self, labels: dict) -> list[str]:
return sorted(
{
self._canonicalize_backend(k.removeprefix("k7.katakate.org/backend-")) or ""
for k, v in labels.items()
if k.startswith("k7.katakate.org/backend-") and v == "true"
}
- {""}
)
async def list_cluster_nodes(self) -> list[dict]:
"""Return name, hostname label, backends, tenant for each Node.
``name`` is what ``--node`` / ``k7 nodes dedicate`` take (K3s
``metadata.name``, usually the Linux hostname). ``hostname`` is
``kubernetes.io/hostname`` — kubelet stamps it; inventory.ini does not.
Backend labels come from ``k7 install``. Tenant is ``k7 nodes dedicate``.
"""
v1 = await self._get_core_v1_client()
nodes = await v1.list_node()
out: dict[str, list[str]] = {}
out: list[dict] = []
for n in nodes.items:
labels = n.metadata.labels or {}
backends = sorted(
taints = list((n.spec.taints if n.spec else None) or [])
tenant = labels.get(K7_TENANT_LABEL) or ""
if not tenant:
for t in taints:
if getattr(t, "key", None) == K7_TENANT_LABEL:
tenant = getattr(t, "value", None) or ""
break
out.append(
{
self._canonicalize_backend(k.removeprefix("k7.katakate.org/backend-")) or ""
for k, v in labels.items()
if k.startswith("k7.katakate.org/backend-") and v == "true"
"name": n.metadata.name,
"hostname": labels.get("kubernetes.io/hostname") or "",
"backends": self._backends_from_node_labels(labels),
"tenant": tenant,
}
- {""}
)
out[n.metadata.name] = backends
return out
async def _list_backends_per_node(self) -> dict[str, list[str]]:
"""Return {node_name: [supported backends]} from k7 backend labels."""
return {n["name"]: n["backends"] for n in await self.list_cluster_nodes()}
async def _tenant_tolerations_for_node(self, node_name: str) -> list:
"""Return a NoSchedule toleration if ``node_name`` is tenant-tainted.
Fail loud when the node does not exist — a pin to a missing name is
a config bug, not a Pending pod.
"""
v1 = await self._get_core_v1_client()
try:
node = await v1.read_node(node_name)
except ApiException as e:
if e.status == 404:
raise RuntimeError(f"node {node_name} not found") from e
raise
taints = (node.spec.taints if node.spec else None) or []
out: list = []
for t in taints:
if getattr(t, "key", None) == K7_TENANT_LABEL and getattr(t, "effect", None) == "NoSchedule":
out.append(
client.V1Toleration(
key=K7_TENANT_LABEL,
operator="Equal",
value=getattr(t, "value", None),
effect="NoSchedule",
)
)
return out
async def dedicate_node(self, node_name: str, tenant: str) -> OperationResult:
"""Label + taint a node for one tenant. Other sandboxes cannot land there."""
if not node_name or not tenant:
return OperationResult(success=False, error="node name and --tenant are required")
v1 = await self._get_core_v1_client()
try:
node = await v1.read_node(node_name)
except ApiException as e:
if e.status == 404:
return OperationResult(success=False, error=f"node {node_name} not found")
return OperationResult(success=False, error=f"read node {node_name}: {e}")
taints = [
t for t in ((node.spec.taints if node.spec else None) or []) if getattr(t, "key", None) != K7_TENANT_LABEL
]
taints.append(client.V1Taint(key=K7_TENANT_LABEL, value=tenant, effect="NoSchedule"))
body = {
"metadata": {"labels": {K7_TENANT_LABEL: tenant}},
"spec": {
"taints": [{"key": t.key, "value": getattr(t, "value", None), "effect": t.effect} for t in taints]
},
}
try:
await v1.patch_node(node_name, body)
except ApiException as e:
return OperationResult(success=False, error=f"patch node {node_name}: {e}")
return OperationResult(
success=True,
message=f"node {node_name} dedicated to tenant {tenant} (label+taint {K7_TENANT_LABEL})",
)
async def undedicate_node(self, node_name: str) -> OperationResult:
"""Drop the k7 tenant label and taint from a node."""
if not node_name:
return OperationResult(success=False, error="node name is required")
v1 = await self._get_core_v1_client()
try:
node = await v1.read_node(node_name)
except ApiException as e:
if e.status == 404:
return OperationResult(success=False, error=f"node {node_name} not found")
return OperationResult(success=False, error=f"read node {node_name}: {e}")
taints = [
t for t in ((node.spec.taints if node.spec else None) or []) if getattr(t, "key", None) != K7_TENANT_LABEL
]
body = {
"metadata": {"labels": {K7_TENANT_LABEL: None}},
"spec": {
"taints": [{"key": t.key, "value": getattr(t, "value", None), "effect": t.effect} for t in taints]
},
}
try:
await v1.patch_node(node_name, body)
except ApiException as e:
return OperationResult(success=False, error=f"patch node {node_name}: {e}")
return OperationResult(success=True, message=f"node {node_name} undedicated")
async def _check_scheduling(
self,
sandbox_name: str,
@@ -621,7 +785,11 @@ class K7Core:
return legacy.get(backend, backend)
async def _detect_backend(self, sandbox_name: str | None = None, namespace: str = "default") -> str:
"""Detect backend with priority: deployment annotation > /etc/k7/backend > default."""
"""Detect backend with priority: deployment annotation > /etc/k7/backend > kfd.
``none`` in ``/etc/k7/backend`` is not a runtime — return empty so
create fails loud instead of falling through to kfd.
"""
allowed = _ALLOWED_BACKENDS
if sandbox_name:
try:
@@ -636,9 +804,12 @@ class K7Core:
try:
if os.path.exists("/etc/k7/backend"):
with open("/etc/k7/backend") as f:
backend = self._canonicalize_backend(f.read().strip())
if backend in allowed:
return backend
raw = f.read().strip()
if raw.lower() == "none":
return ""
backend = self._canonicalize_backend(raw)
if backend in allowed:
return backend
except Exception:
pass
@@ -790,8 +961,8 @@ class K7Core:
raise RuntimeError(f"k7 agent token {path} is empty — re-run `k7 install`")
return token
async def _k7d_sandbox_node(self, sandbox_name: str, namespace: str) -> str:
"""Node hosting the sandbox's Running pod (loud when there is none)."""
async def _k7d_sandbox_placement(self, sandbox_name: str, namespace: str) -> tuple[str, str]:
"""Return ``(node_name, pod_name)`` for the Running sandbox pod."""
v1 = await self._get_core_v1_client()
pods = await v1.list_namespaced_pod(namespace=namespace, label_selector=f"app={sandbox_name}")
running = [
@@ -799,7 +970,16 @@ class K7Core:
]
if not running:
raise RuntimeError(f"no Running pod found for sandbox {sandbox_name} in {namespace}")
return running[0].spec.node_name
pod = running[0]
node = pod.spec.node_name if pod.spec else None
if not node:
raise RuntimeError(f"sandbox {sandbox_name} in {namespace} has no spec.nodeName yet")
return node, pod.metadata.name
async def _k7d_sandbox_node(self, sandbox_name: str, namespace: str) -> str:
"""Node hosting the sandbox's Running pod (loud when there is none)."""
node, _pod = await self._k7d_sandbox_placement(sandbox_name, namespace)
return node
async def _k7d_agent_base_url(self, node_name: str) -> str:
"""Resolve the Ready k7-agent pod on ``node_name`` to its pod IP."""
@@ -842,14 +1022,14 @@ class K7Core:
async def lookup_k7d_vm(self, sandbox_name: str, namespace: str = "default") -> dict:
"""Resolve a k7d sandbox to its daemon VM record, forwarding to the
per-node agent when the sandbox lives on another node."""
node = await self._k7d_sandbox_node(sandbox_name, namespace)
if node and node != self._k7d_local_node():
node, pod_name = await self._k7d_sandbox_placement(sandbox_name, namespace)
if node != self._k7d_local_node():
base = await self._k7d_agent_base_url(node)
token = self._k7d_agent_token()
async with httpx.AsyncClient(timeout=60) as http:
resp = await http.post(
f"{base}/agent/v1/vm/lookup",
json={"name": sandbox_name, "namespace": namespace},
json={"name": sandbox_name, "namespace": namespace, "pod_name": pod_name},
headers={"X-K7-Agent-Token": token},
)
if resp.status_code != 200:
@@ -954,6 +1134,18 @@ class K7Core:
return int(math.ceil(mib))
def _kata_hypervisor_memory_mib(self, value: str) -> int:
"""Kata ``default_memory`` MiB. Loud if below the shim's 256Mi floor."""
memory_mib = self._memory_limit_to_mib(value)
if memory_mib < KATA_MIN_HYPERVISOR_MEMORY_MIB:
raise ValueError(
f"memory limit {value} is below Kata's minimum "
f"{KATA_MIN_HYPERVISOR_MEMORY_MIB}Mi "
"(io.katacontainers.config.hypervisor.default_memory). "
"Use at least 256Mi, or omit limits to use the hypervisor default."
)
return memory_mib
def _count_playbook_tasks(self, playbook_content: str) -> int:
"""Count tasks in Ansible playbook."""
try:
@@ -1278,13 +1470,122 @@ class K7Core:
def _docker_snapshot_name(self, snapshot_name: str) -> str:
return f"{snapshot_name}-docker"
async def _freeze_docker_graph(self, sandbox_name: str, namespace: str) -> OperationResult:
# docker:27.5.1-dind is alpine and has no fsfreeze. The sandbox image
# (ubuntu:24.04 in the product tests) ships util-linux; /tmp is the
# shared emptyDir. Copy into the vehicle rootfs before exec in case
# emptyDir is noexec.
stage = await self.exec_command(sandbox_name, STAGE_FSFREEZE_CMD, namespace=namespace)
if stage.exit_code != 0:
return OperationResult(
success=False,
error=(
f"could not stage fsfreeze from sandbox: {stage.stderr}. "
"Pause the sandbox (`k7 pause --snapshot`) and retry, or use the k7d backend."
),
)
freeze = await self.exec_command(
sandbox_name,
f"cp {FSFREEZE_STAGED} {FSFREEZE_VEHICLE} && chmod 755 {FSFREEZE_VEHICLE} && "
f"{FSFREEZE_VEHICLE} -f {KATA_GRAPH_MOUNT}",
namespace=namespace,
container=VEHICLE_CONTAINER_NAME,
)
if freeze.exit_code != 0:
return OperationResult(
success=False,
error=(
f"fsfreeze -f {KATA_GRAPH_MOUNT} in docker-vehicle failed: {freeze.stderr}. "
"Pause the sandbox (`k7 pause --snapshot`) and retry, or use the k7d backend."
),
)
return OperationResult(success=True)
async def _thaw_docker_graph(self, sandbox_name: str, namespace: str) -> OperationResult:
thaw = await self.exec_command(
sandbox_name,
f"{FSFREEZE_VEHICLE} -u {KATA_GRAPH_MOUNT}",
namespace=namespace,
container=VEHICLE_CONTAINER_NAME,
)
if thaw.exit_code != 0:
return OperationResult(
success=False,
error=f"fsfreeze -u {KATA_GRAPH_MOUNT} in docker-vehicle failed: {thaw.stderr}",
)
return OperationResult(success=True)
async def _create_kata_snapshots_quiesced(
self,
sandbox_name: str,
namespace: str,
*,
root_pvc: str,
snapshot_name: str,
kind: str,
running: bool,
docker_pvc: str | None,
fail_prefix: str,
) -> OperationResult:
"""Create root (+ docker) VolumeSnapshots.
``sync`` the sandbox root, then ``fsfreeze`` the docker graph for
the create calls only. Thaw in ``finally`` so a failed snapshot
cannot leave the source frozen. Wait-for-ready stays outside.
"""
want_docker = docker_pvc is not None
graph_frozen = False
result = OperationResult(success=True)
try:
if running:
sync_res = await self.exec_command(sandbox_name, "sync", namespace=namespace)
if sync_res.exit_code != 0:
result = OperationResult(
success=False,
error=f"{fail_prefix} sync in source sandbox failed: {sync_res.stderr}",
)
elif want_docker:
freeze = await self._freeze_docker_graph(sandbox_name, namespace)
if not freeze.success:
result = freeze
else:
graph_frozen = True
if result.success:
result = await self._create_volume_snapshot(
pvc_name=root_pvc,
snapshot_name=snapshot_name,
snapshot_class="longhorn",
namespace=namespace,
kind=kind,
source_sandbox=sandbox_name,
)
if result.success and docker_pvc is not None:
docker_snap = await self._create_volume_snapshot(
pvc_name=docker_pvc,
snapshot_name=self._docker_snapshot_name(snapshot_name),
snapshot_class="longhorn",
namespace=namespace,
kind=kind,
source_sandbox=sandbox_name,
)
if not docker_snap.success:
result = docker_snap
finally:
if graph_frozen:
thaw = await self._thaw_docker_graph(sandbox_name, namespace)
if not thaw.success:
if result.success:
result = thaw
else:
result = OperationResult(success=False, error=f"{result.error}; {thaw.error}")
return result
async def _wait_for_job(self, job_name: str, namespace: str = "default", timeout: int = 600) -> OperationResult:
"""Wait for a Kubernetes Job to complete."""
await self._load_k3s_config()
batch = await self._get_batch_v1_client()
start = time.time()
while time.time() - start < timeout:
try:
batch = client.BatchV1Api()
job = await batch.read_namespaced_job(name=job_name, namespace=namespace)
status = job.status
if status.succeeded and status.succeeded >= 1:
@@ -1624,8 +1925,7 @@ class K7Core:
async def _cilium_available(self) -> bool:
"""Return True when the CiliumNetworkPolicy CRD is registered on the cluster."""
await self._load_k3s_config()
api_ext = client.ApiextensionsV1Api()
api_ext = await self._get_apiextensions_v1_client()
try:
await api_ext.read_custom_resource_definition(name="ciliumnetworkpolicies.cilium.io")
return True
@@ -2573,7 +2873,7 @@ class K7Core:
# cpu/memory limits.
if config.limits and "memory" in config.limits and not _is_k7d_family(backend):
try:
memory_mib = self._memory_limit_to_mib(config.limits["memory"])
memory_mib = self._kata_hypervisor_memory_mib(config.limits["memory"])
except ValueError as e:
return OperationResult(success=False, error=str(e))
base_annotations["io.katacontainers.config.hypervisor.default_memory"] = str(memory_mib)
@@ -2662,13 +2962,26 @@ class K7Core:
fs_group=65532,
)
node_selector = {f"k7.katakate.org/backend-{backend}": "true"}
hostname = getattr(config, "node_name", None) or None
tolerations = None
if hostname:
# Link is the Node's kubernetes.io/hostname label (already on
# every node), not spec.nodeName (that bypasses the scheduler).
node_selector["kubernetes.io/hostname"] = hostname
try:
tols = await self._tenant_tolerations_for_node(hostname)
except RuntimeError as e:
return OperationResult(success=False, error=str(e))
if tols:
tolerations = tols
pod_spec = client.V1PodSpec(
containers=[container],
runtime_class_name=runtime_class,
restart_policy="Always",
security_context=pod_sec_ctx,
node_selector={f"k7.katakate.org/backend-{backend}": "true"},
node_name=getattr(config, "node_name", None) or None,
node_selector=node_selector,
tolerations=tolerations,
)
if root_pvc_name and wrapper_cm_name:
@@ -3152,11 +3465,17 @@ class K7Core:
success=False,
error=f"--snapshot is not supported when pausing a k7d sandbox: {K7D_SNAPSHOT_UNSUPPORTED}",
)
node = await self._k7d_sandbox_node(name, namespace)
if node and node != self._k7d_local_node():
return await self._k7d_forward_vm_op(
node, "pause", {"name": name, "namespace": namespace}, timeout=120
node, pod_name = await self._k7d_sandbox_placement(name, namespace)
if node != self._k7d_local_node():
result = await self._k7d_forward_vm_op(
node,
"pause",
{"name": name, "namespace": namespace, "pod_name": pod_name},
timeout=120,
)
if result.success:
await self._update_deployment_annotation(name, namespace, K7D_ANN_PAUSED, "true")
return result
vm = await self._k7d_vm_for_sandbox(name, namespace)
await self._k7d_request({"op": "pause_vm", "vm_id": vm["vm_id"]})
await self._update_deployment_annotation(name, namespace, K7D_ANN_PAUSED, "true")
@@ -3197,11 +3516,17 @@ class K7Core:
try:
backend = await self._detect_backend(name, namespace)
if _is_k7d_family(backend):
node = await self._k7d_sandbox_node(name, namespace)
if node and node != self._k7d_local_node():
return await self._k7d_forward_vm_op(
node, "resume", {"name": name, "namespace": namespace}, timeout=120
node, pod_name = await self._k7d_sandbox_placement(name, namespace)
if node != self._k7d_local_node():
result = await self._k7d_forward_vm_op(
node,
"resume",
{"name": name, "namespace": namespace, "pod_name": pod_name},
timeout=120,
)
if result.success:
await self._update_deployment_annotation(name, namespace, K7D_ANN_PAUSED, "false")
return result
vm = await self._k7d_vm_for_sandbox(name, namespace)
await self._k7d_request({"op": "resume_vm", "vm_id": vm["vm_id"]})
await self._update_deployment_annotation(name, namespace, K7D_ANN_PAUSED, "false")
@@ -3225,8 +3550,15 @@ class K7Core:
namespace: str = "default",
) -> OperationResult:
"""Shell into a sandbox pod via kubectl exec. Stays sync (interactive CLI-only)."""
v1 = asyncio.run(self._get_core_v1_client())
pods = asyncio.run(v1.list_namespaced_pod(namespace=namespace, label_selector=f"app={sandbox_name}"))
async def _list_pods():
try:
v1 = await self._get_core_v1_client()
return await v1.list_namespaced_pod(namespace=namespace, label_selector=f"app={sandbox_name}")
finally:
await self.aclose()
pods = asyncio.run(_list_pods())
if not pods.items:
return OperationResult(success=False, error="Pod not found")
@@ -3271,13 +3603,13 @@ class K7Core:
snapshot_name=snapshot_name,
fork_start=fork_start,
)
if backend == "kata-firecracker-devmapper":
return OperationResult(success=False, error=KFD_FORK_REJECT)
apps_v1 = await self._get_apps_v1_client()
v1 = await self._get_core_v1_client()
src = await apps_v1.read_namespaced_deployment(name=source_name, namespace=namespace)
src_ann = (src.metadata.annotations or {}) if src.metadata else {}
if backend == "kata-firecracker-devmapper" and src_ann.get(ANN_K7_DOCKER) == "true":
return OperationResult(success=False, error=KATA_FORK_GRAPH_REJECT)
source_pvc_name = self._root_pvc_name(source_name)
target_pvc_name = self._root_pvc_name(new_name)
@@ -3305,34 +3637,37 @@ class K7Core:
snap_name = f"{source_name}-fork-{int(time.time())}"
snap_kind = SNAPSHOT_KIND_FORK
auto_temp_snapshot = True
# A Longhorn VolumeSnapshot is block-level and only
# crash-consistent: guest writes still sitting in the VM's page
# cache would be missing from the fork's clone. Flush them first
# when the source is running (fail loudly if the flush fails);
# a scaled-to-0 (paused) source has no writers and needs no flush.
if (src.status.ready_replicas or 0) > 0:
sync_res = await self.exec_command(source_name, "sync", namespace=namespace)
if sync_res.exit_code != 0:
return OperationResult(
success=False,
error=f"pre-fork sync in source sandbox failed: {sync_res.stderr}",
source_docker_pvc_name = src_ann.get(ANN_DOCKER_PVC) or self._docker_pvc_name(source_name)
target_docker_pvc_name = self._docker_pvc_name(new_name)
docker_snap_name = None
source_docker_pvc = None
if src_ann.get(ANN_K7_DOCKER) == "true":
try:
source_docker_pvc = await v1.read_namespaced_persistent_volume_claim(
name=source_docker_pvc_name, namespace=namespace
)
if src_ann.get(ANN_K7_DOCKER) == "true":
vehicle_sync = await self.exec_command(
source_name, "sync", namespace=namespace, container=VEHICLE_CONTAINER_NAME
)
if vehicle_sync.exit_code != 0:
except ApiException as e:
if e.status == 404:
return OperationResult(
success=False,
error=f"pre-fork sync in docker-vehicle failed: {vehicle_sync.stderr}",
error=f"Source docker PVC {source_docker_pvc_name} not found; cannot fork docker graph",
)
snap_result = await self._create_volume_snapshot(
pvc_name=source_pvc_name,
return OperationResult(success=False, error=f"Docker PVC lookup error: {e}")
docker_snap_name = self._docker_snapshot_name(snap_name)
# Longhorn VolumeSnapshots are block-level. sync the sandbox
# root, and fsfreeze the docker graph for the create calls so a
# live overlay2 is filesystem-consistent. Thaw before waiting
# for Ready so dockerd is not frozen across CSI provisioning.
snap_result = await self._create_kata_snapshots_quiesced(
source_name,
namespace,
root_pvc=source_pvc_name,
snapshot_name=snap_name,
snapshot_class="longhorn",
namespace=namespace,
kind=snap_kind,
source_sandbox=source_name,
running=(src.status.ready_replicas or 0) > 0,
docker_pvc=source_docker_pvc_name if src_ann.get(ANN_K7_DOCKER) == "true" else None,
fail_prefix="pre-fork",
)
if not snap_result.success:
return snap_result
@@ -3357,32 +3692,12 @@ class K7Core:
if not pvc_clone.success:
return pvc_clone
source_docker_pvc_name = src_ann.get(ANN_DOCKER_PVC) or self._docker_pvc_name(source_name)
target_docker_pvc_name = self._docker_pvc_name(new_name)
docker_snap_name = None
if src_ann.get(ANN_K7_DOCKER) == "true":
try:
source_docker_pvc = await v1.read_namespaced_persistent_volume_claim(
name=source_docker_pvc_name, namespace=namespace
if not docker_snap_name or source_docker_pvc is None:
return OperationResult(
success=False,
error=f"docker snapshot missing after quiesced create for {source_name}",
)
except ApiException as e:
if e.status == 404:
return OperationResult(
success=False,
error=f"Source docker PVC {source_docker_pvc_name} not found; cannot fork docker graph",
)
return OperationResult(success=False, error=f"Docker PVC lookup error: {e}")
docker_snap_name = self._docker_snapshot_name(snap_name)
docker_snap = await self._create_volume_snapshot(
pvc_name=source_docker_pvc_name,
snapshot_name=docker_snap_name,
snapshot_class="longhorn",
namespace=namespace,
kind=snap_kind,
source_sandbox=source_name,
)
if not docker_snap.success:
return docker_snap
docker_ready = await self._wait_for_snapshot_ready(docker_snap_name, namespace=namespace)
if not docker_ready.success:
return docker_ready
@@ -3564,21 +3879,11 @@ class K7Core:
),
)
# The fork must run on the source VM's node (the daemon
# socket is node-local; the fork pod is pinned there too). Forward
# the WHOLE fork to that node's k7-agent when the source is remote.
node = await self._k7d_sandbox_node(source_name, namespace)
if node and node != self._k7d_local_node():
return await self._k7d_forward_vm_op(
node,
"fork",
{"name": source_name, "new_name": new_name, "namespace": namespace, "snapshot": snapshot_name},
timeout=600,
)
# Resolve the live source VM up-front so a dead/unknown source fails
# here with a clear message instead of leaving a CrashLooping fork pod.
vm = await self._k7d_vm_for_sandbox(source_name, namespace)
# Kubernetes writes stay on the caller (k7-api ClusterRole). The
# agent has no ServiceAccount token — it only looks up the live VM
# so these fork-source annotations are correct.
node, _pod = await self._k7d_sandbox_placement(source_name, namespace)
vm = await self.lookup_k7d_vm(source_name, namespace)
fork_annotations = {
K7D_ANN_FORK_SOURCE_CLUSTER: vm.get("cluster_id") or vm["sandbox_id"],
K7D_ANN_FORK_SOURCE_VM: vm["sandbox_id"],
@@ -3615,8 +3920,7 @@ class K7Core:
new_dep.spec.replicas = 1
# The fork must land on the source VM's node — the k7d daemon socket
# is node-local (cross-node fork is future k7d work, not built yet).
source_pod = await self._k7d_running_pod(source_name, namespace)
new_dep.spec.template.spec.node_name = source_pod.spec.node_name
new_dep.spec.template.spec.node_name = node
try:
await apps_v1.create_namespaced_deployment(namespace=namespace, body=new_dep)
@@ -3862,49 +4166,25 @@ class K7Core:
return OperationResult(success=False, error=f"Deployment lookup error: {e}")
anns = (dep.metadata.annotations or {}) if dep.metadata else {}
want_docker = anns.get(ANN_K7_DOCKER) == "true"
docker_pvc = (anns.get(ANN_DOCKER_PVC) or self._docker_pvc_name(sandbox_name)) if want_docker else None
# Crash-consistency: guest page cache is invisible to Longhorn.
# Same flush as fork (sandbox + docker-vehicle). A paused
# (scaled-to-0) source has no writers and needs no flush.
if (dep.status.ready_replicas or 0) > 0:
sync_res = await self.exec_command(sandbox_name, "sync", namespace=namespace)
if sync_res.exit_code != 0:
return OperationResult(
success=False,
error=f"pre-snapshot sync in source sandbox failed: {sync_res.stderr}",
)
if want_docker:
vehicle_sync = await self.exec_command(
sandbox_name, "sync", namespace=namespace, container=VEHICLE_CONTAINER_NAME
)
if vehicle_sync.exit_code != 0:
return OperationResult(
success=False,
error=f"pre-snapshot sync in docker-vehicle failed: {vehicle_sync.stderr}",
)
root = await self._create_volume_snapshot(
pvc_name=self._root_pvc_name(sandbox_name),
# Same quiesce as fork (sandbox sync + docker-vehicle fsfreeze).
# A paused (scaled-to-0) source has no writers and needs no freeze.
root = await self._create_kata_snapshots_quiesced(
sandbox_name,
namespace,
root_pvc=self._root_pvc_name(sandbox_name),
snapshot_name=snapshot_name,
snapshot_class="longhorn",
namespace=namespace,
kind=SNAPSHOT_KIND_NAMED,
source_sandbox=sandbox_name,
running=(dep.status.ready_replicas or 0) > 0,
docker_pvc=docker_pvc,
fail_prefix="pre-snapshot",
)
if not root.success:
return root
if not want_docker:
return root
docker_pvc = anns.get(ANN_DOCKER_PVC) or self._docker_pvc_name(sandbox_name)
docker_snap_name = self._docker_snapshot_name(snapshot_name)
docker_snap = await self._create_volume_snapshot(
pvc_name=docker_pvc,
snapshot_name=docker_snap_name,
snapshot_class="longhorn",
namespace=namespace,
kind=SNAPSHOT_KIND_NAMED,
source_sandbox=sandbox_name,
)
if not docker_snap.success:
return docker_snap
root_ready = await self._wait_for_snapshot_ready(snapshot_name, namespace=namespace)
if not root_ready.success:
return root_ready
@@ -4195,6 +4475,7 @@ class K7Core:
namespace: str = "default",
overrides: SandboxConfigOverrides | None = None,
keep_snapshot: bool = True,
node_name: str | None = None,
) -> OperationResult:
"""Boot a brand-new sandbox from a standalone ``VolumeSnapshot``.
@@ -4204,6 +4485,8 @@ class K7Core:
field. Restore is **kata-qemu-longhorn only** — there is no PVC to clone
from in the kata-firecracker-devmapper backend.
``node_name`` pins the restored pod (API-key node scope stamps this).
The cloned PVC is named ``_root_pvc_name(new_sandbox_name)`` and is
pre-created from the snapshot before ``create_sandbox`` runs; the
idempotent ``_ensure_root_pvc`` in ``create_sandbox`` then detects
@@ -4249,6 +4532,8 @@ class K7Core:
if not config_result.success:
return config_result
config: SandboxConfig = config_result.data
if node_name:
config.node_name = node_name
# Restore is kata-qemu-longhorn-only: kata-firecracker-devmapper has no PVC to
# clone, so it has no snapshots either.
@@ -4331,7 +4616,14 @@ class K7Core:
)
async def delete_sandbox(self, name: str, namespace: str = "default") -> OperationResult:
"""Delete a sandbox."""
"""Delete a sandbox's Kubernetes objects.
Returns when Deployments, PVCs, Services, and related objects are
gone. k7d VM teardown (including the scratch docker-graph unlink)
is asynchronous via the containerd shim ``Delete`` and is **not**
waited on here — coupling API latency to shim teardown would stall
every delete. Callers that need the graph file gone must poll.
"""
return await self._delete_sandbox_resources(name, namespace)
async def delete_all_sandboxes(self, namespace: str = "default") -> OperationResult:
src/k7/core/docker.py
+15
View File
@@ -36,6 +36,16 @@ KATA_SOCKET_DIR = "/run/k7/docker"
DOCKER_HOST_URL = "unix:///run/k7/docker/docker.sock"
KATA_DAEMON_ARGS = "--host=unix:///run/k7/docker/docker.sock --tls=false --storage-driver=overlay2"
GRAPH_DEVICE_PATH = "/dev/k7docker"
KATA_GRAPH_MOUNT = "/var/lib/docker"
# alpine dind has no fsfreeze; ubuntu sandbox does. Stage via the shared /tmp
# emptyDir, then exec from the vehicle's own rootfs (emptyDir may be noexec).
FSFREEZE_STAGED = "/tmp/k7-fsfreeze"
FSFREEZE_VEHICLE = "/usr/local/bin/k7-fsfreeze"
STAGE_FSFREEZE_CMD = (
f"if [ -x /usr/sbin/fsfreeze ]; then cp /usr/sbin/fsfreeze {FSFREEZE_STAGED}; "
f"elif [ -x /sbin/fsfreeze ]; then cp /sbin/fsfreeze {FSFREEZE_STAGED}; "
"else echo 'fsfreeze not found in sandbox image (need util-linux)' >&2; exit 1; fi"
)
VEHICLE_CONTAINER_NAME = "docker-vehicle"
KFD_DOCKER_STORAGE_CLASS = "k7-docker-lvm"
# linux/amd64 digest of docker:27.5.1-dind (Docker Hub, 2026-09).
@@ -52,6 +62,11 @@ ANN_DOCKER_PVC = "k7.katakate.org/docker-pvc-name"
KATA_FORK_GRAPH_REJECT = (
"k7 fork of a kfd --docker sandbox is not supported: the docker graph cannot be cloned (ephemeral LV)"
)
KFD_FORK_REJECT = (
"k7 fork is not supported on the kfd (kata-firecracker-devmapper) backend: "
"there is no persistent PVC to clone (and a --docker graph cannot be cloned — ephemeral LV). "
"Use kql for disk-level fork, or k7d / k7d-fc for warm VM fork."
)
# Playbook records the installed k7d version here. Guest dockerd
# and k7-fc shipped in public k7d 0.6.0; a node whose recorded version
src/k7/core/models.py
+3 -2
View File
@@ -34,8 +34,9 @@ class SandboxConfig:
container_non_root: bool = False
cap_drop: list[str] | None = None # default behavior handled in core: drop ALL
cap_add: list[str] | None = None
# Optional explicit node placement (sets pod's node_name). Used by tests
# that need to inspect host-side state for a sandbox they just created.
# Optional node pin. Applied as nodeSelector kubernetes.io/hostname
# (the label every Node already has) — not spec.nodeName. A tenant
# taint on that node is the exclusive lock; see k7 nodes dedicate.
node_name: str | None = None
# Ingress is denied by default and opt-in per sandbox.
# ``ingress_ports`` are TCP ports to open; ``None``/``[]`` denies everything.
src/k7/deploy/inventory.ini.example
+56 -15
View File
@@ -1,14 +1,27 @@
; Example multi-node inventory for `k7 install -i inventory.ini`.
;
; Each host declares which backend(s) it supports via `k7_backends` (comma-separated).
; A node may support one or both of:
; - kata-qemu-longhorn (uses Kata QEMU + overlayfs + Longhorn for sandbox storage)
; - kata-firecracker-devmapper (uses Kata Firecracker + LVM thin-pool snapshotter)
; The left-hand name (k7-node-01) MUST match the machine hostname
; (`hostnamectl` / installimage HOSTNAME). K3s names the Node after that
; hostname and stamps kubernetes.io/hostname=<name> automatically.
; This file does not write that label. `k7_backends` tells the playbook
; which k7.katakate.org/backend-* labels to stamp after the node joins.
; Tenant isolation is later: `k7 nodes dedicate NAME --tenant ID`.
; Copy NAME from `k7 nodes list` / `kubectl get nodes` into `--node`.
;
; `k7_backends` is required on every host (or inherited from group vars).
; Empty/omitted is an error, not "no runtimes". Use `none` for a node that
; must not run sandboxes (typical [k7_servers] in a cluster that has agents).
; Allowed: kfd, kql, k7d, k7d-fc, none (comma-separated except none alone).
;
; A node may list one or more of:
; - kata-qemu-longhorn / kql (Kata QEMU + overlayfs + Longhorn)
; - kata-firecracker-devmapper / kfd (Kata Firecracker + LVM thin-pool; spare disk)
; - k7d / k7d-fc
;
; Per-host vars:
; ansible_host : SSH target IP
; ansible_user : SSH user (defaults to root)
; k7_backends : comma-separated backend list
; k7_backends : comma-separated backend list, or none
; k7_devmapper_disk : block device for FD thin-pool (e.g. /dev/nvme1n1)
; longhorn_extra_disk : extra mount path to register with Longhorn
; kata_thinpool_pv_size : LVM PV size for the kfd thin-pool (default 100G)
@@ -41,6 +54,10 @@ longhorn_replicas=2
; tolerance — for a 2-node cluster, put the second node in [k7_agents].
; The first host listed in [k7_servers] is the cluster-init server.
;
; Every [k7_servers] host is a K3s master: root there is cluster-admin
; (k3s.yaml + join token + etcd). Put tenant sandboxes on [k7_agents]
; and pin/dedicate those agent names. Do not dedicate the first master.
;
; Do NOT pass --backend alongside -i: per-host `k7_backends` in the
; inventory is authoritative (an explicit --backend overrides it).
;
@@ -56,16 +73,16 @@ longhorn_replicas=2
; 2-node, ALL backends, from the PPA CLI (no --ha — that needs 3+ servers).
; After TWO_DISK=1 so each box has a spare raw NVMe:
;
; # PPA is still 0.2.2; prefer the GitHub 0.3.0 .deb, then:
; git clone --branch v0.3.0 https://github.com/Katakate/k7.git && cd k7
; # PPA and GitHub .deb are 0.4.0:
; git clone --branch v0.4.0 https://github.com/Katakate/k7.git && cd k7
; # run from that checkout so the playbook can build k7-api:local
; k7 install -i inventory.ini --k7d-version 0.6.0
; k7 install -i inventory.ini --k7d-version 0.7.0
;
; [k7_servers]
; k7-node-01 ansible_host=203.0.113.10 k7_backends=kfd,kql,k7d
; k7-node-01 ansible_host=203.0.113.10 k7_backends=kfd,kql,k7d,k7d-fc
;
; [k7_agents]
; k7-node-02 ansible_host=203.0.113.11 k7_backends=kfd,kql,k7d
; k7-node-02 ansible_host=203.0.113.11 k7_backends=kfd,kql,k7d,k7d-fc
;
; [k7_cluster:children]
; k7_servers
@@ -77,14 +94,38 @@ longhorn_replicas=2
; ansible_ssh_common_args=-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
; longhorn_replicas=2
;
; 3-node HA with ALL backends on every node. ONE command:
; 3 small masters (no sandboxes) + agents. `none` is required on servers —
; omitting k7_backends is an error, not an implicit empty set.
;
; k7 install -i inventory.ini --ha --k7d-version 0.6.0
; k7 install -i inventory.ini --ha --k7d-version 0.7.0
;
; [k7_servers]
; k7-node-01 ansible_host=203.0.113.10 k7_backends=kfd,kql,k7d
; k7-node-02 ansible_host=203.0.113.11 k7_backends=kfd,kql,k7d
; k7-node-03 ansible_host=203.0.113.12 k7_backends=kfd,kql,k7d
; k7-node-01 ansible_host=203.0.113.10
; k7-node-02 ansible_host=203.0.113.11
; k7-node-03 ansible_host=203.0.113.12
;
; [k7_servers:vars]
; k7_backends=none
;
; [k7_agents]
; k7-node-04 ansible_host=203.0.113.13
; k7-node-05 ansible_host=203.0.113.14
;
; [k7_agents:vars]
; k7_backends=kql,k7d
;
; [k7_cluster:children]
; k7_servers
; k7_agents
;
; 3-node HA with ALL backends on every node (soak / demo). ONE command:
;
; k7 install -i inventory.ini --ha --k7d-version 0.7.0
;
; [k7_servers]
; k7-node-01 ansible_host=203.0.113.10 k7_backends=kfd,kql,k7d,k7d-fc
; k7-node-02 ansible_host=203.0.113.11 k7_backends=kfd,kql,k7d,k7d-fc
; k7-node-03 ansible_host=203.0.113.12 k7_backends=kfd,kql,k7d,k7d-fc
;
; [k7_cluster:children]
; k7_servers
src/k7/deploy/inventory.local.ini
+3 -2
View File
@@ -1,6 +1,7 @@
; Single-node localhost inventory used by `k7 install` when no inventory is given.
; Unused by `k7 install` (the CLI builds inventory from --backend).
; If you pass -i this file, k7_backends is required — empty is not none.
[k7_servers]
localhost ansible_connection=local ansible_user=root
localhost ansible_connection=local ansible_user=root k7_backends=kql
[k7_agents]
src/k7/deploy/k7-install-node.yaml
+37 -19
View File
@@ -3,7 +3,8 @@
become: true
vars:
target_user: "{{ ansible_user | default('root') }}"
k7_backend_default: "kata-firecracker-devmapper"
# No implicit backend default — every host must set k7_backends (or inherit
# group vars). Use k7_backends=none for a node with no sandbox runtime.
snapshotter_version: "v8.4.0"
kata_version: "3.24.0"
cni_plugins_version: "v1.9.0"
@@ -13,7 +14,7 @@
# k7d backend: release artifact source. Override
# `k7d_artifact_local_path` (path on the Ansible controller) to install
# from a locally built tarball instead of the GitHub release.
k7d_version: "0.6.0"
k7d_version: "0.7.0"
k7d_artifact_url: "https://github.com/katakate/k7d/releases/download/v{{ k7d_version }}/k7d-v{{ k7d_version }}-x86_64-linux.tar.gz"
k7d_artifact_local_path: ""
# Sparse XFS (reflink=1) loopback image backing /var/lib/k7d/disks —
@@ -71,28 +72,45 @@
# ──────────────────────────────────────────────────────────────────────
# Compute per-host backend set + cluster role
# ──────────────────────────────────────────────────────────────────────
- name: Require k7_backends on every host
ansible.builtin.fail:
msg: >-
k7_backends is not set on {{ inventory_hostname }} (empty/omitted is not
'none'). Set it on the host or a group vars section ([k7_servers:vars] /
[k7_agents:vars]). Allowed: kfd, kql, k7d, k7d-fc, or none (no sandbox
runtime on this node).
when: k7_backends is not defined or (k7_backends | string | trim | length) == 0
tags: ['always']
- name: Compute per-host backend list (raw)
ansible.builtin.set_fact:
k7_backends_raw: >-
{{
(k7_backends.split(',') | map('trim') | reject('equalto','') | list)
if (k7_backends is defined and k7_backends | length > 0)
else ([k7_backend] if (k7_backend is defined and k7_backend | length > 0)
else [k7_backend_default])
}}
k7_backends_raw: "{{ (k7_backends | string).split(',') | map('trim') | reject('equalto', '') | list }}"
tags: ['always']
- name: Reject none mixed with other backends
ansible.builtin.fail:
msg: >-
k7_backends={{ k7_backends }} on {{ inventory_hostname }} mixes 'none'
with other backends. Use only none, or a list of runtimes.
when:
- (k7_backends_raw | map('lower') | list | select('equalto', 'none') | list | length) > 0
- (k7_backends_raw | length) > 1
tags: ['always']
- name: Normalize deprecated backend aliases
ansible.builtin.set_fact:
k7_backends_list: >-
{{
k7_backends_raw
| map('regex_replace', '^(fd|kfd|firecracker-devmapper)$', 'kata-firecracker-devmapper')
| map('regex_replace', '^(ql|kql|qemu-longhorn)$', 'kata-qemu-longhorn')
| map('regex_replace', '^k7$', 'k7d')
| map('regex_replace', '^k7-fc$', 'k7d-fc')
| unique
| list
[] if (k7_backends_raw | map('lower') | list == ['none'])
else (
k7_backends_raw
| map('regex_replace', '^(fd|kfd|firecracker-devmapper)$', 'kata-firecracker-devmapper')
| map('regex_replace', '^(ql|kql|qemu-longhorn)$', 'kata-qemu-longhorn')
| map('regex_replace', '^k7$', 'k7d')
| map('regex_replace', '^k7-fc$', 'k7d-fc')
| unique
| list
)
}}
tags: ['always']
@@ -102,12 +120,12 @@
Deprecated backend name(s) in inventory were remapped:
{{ k7_backends_raw }} → {{ k7_backends_list }}.
Prefer kata-firecracker-devmapper (kfd) / kata-qemu-longhorn (kql).
when: k7_backends_raw != k7_backends_list
when: k7_backends_list | length > 0 and k7_backends_raw != k7_backends_list
tags: ['always']
- name: Validate backends list
ansible.builtin.fail:
msg: "Unknown backend(s) in k7_backends={{ k7_backends_list }}; allowed: kata-firecracker-devmapper, kata-qemu-longhorn, k7d, k7d-fc"
msg: "Unknown backend(s) in k7_backends={{ k7_backends_list }}; allowed: kata-firecracker-devmapper, kata-qemu-longhorn, k7d, k7d-fc, or none"
when: k7_backends_list | difference(['kata-firecracker-devmapper', 'kata-qemu-longhorn', 'k7d', 'k7d-fc']) | length > 0
tags: ['always']
@@ -2139,7 +2157,7 @@
- name: Store K7 backend configuration (primary backend = first in list)
ansible.builtin.copy:
dest: /etc/k7/backend
content: "{{ k7_backends_list[0] }}"
content: "{{ k7_backends_list[0] if k7_backends_list | length > 0 else 'none' }}"
mode: '0644'
tags: ['k7', 'config']
src/k7/deploy/k7d-fc/pins.env
+7 -5
View File
@@ -1,7 +1,9 @@
# Vendored from k7d guest/fc/pins.env.
# Bump together with k7d's pin; sha mismatch is fatal.
FIRECRACKER_VERSION=v1.16.1
FIRECRACKER_RELEASE_URL=https://github.com/firecracker-microvm/firecracker/releases/download/v1.16.1/firecracker-v1.16.1-x86_64.tgz
FIRECRACKER_TGZ_SHA256=382a02a869e4d6d5cb14c40577f9545e8458021ea8b0b2d3fc10ec14d9c242e6
FIRECRACKER_SHA256=2fd0171309af7e24cf8dafc8a6f921c1434c49b5f9349bb996b7ed0a4deb8aa7
JAILER_SHA256=1f3a0c1fe86212d0001819bfe0819071c01208b3ccc9398c3b3bc1b84cf21edd
# v1.16.2 over v1.16.1: upstream #6100 — earlier 1.16.x gated vsock RX
# after a bare pause → resume, so no exec answered (CHALLENGES #17).
FIRECRACKER_VERSION=v1.16.2
FIRECRACKER_RELEASE_URL=https://github.com/firecracker-microvm/firecracker/releases/download/v1.16.2/firecracker-v1.16.2-x86_64.tgz
FIRECRACKER_TGZ_SHA256=32e3cdcd4081f91fe2b024a266f57dcb3b4e5fec5033e0cb22467ad7f7820bda
FIRECRACKER_SHA256=8227875ceda44177a4d501052dae4a2f9d7837362f5399f02cf0748e68418377
JAILER_SHA256=99832d2270741af67ffacd46da716b4b016338541af5a742907ab965b2e9ac9e
src/k7/deploy/manifests/k7-api/agent-daemonset.yaml
+4 -4
View File
@@ -14,10 +14,10 @@ spec:
labels:
app: k7-agent
spec:
# Reuses the k7-api ServiceAccount/RBAC: the agent runs the same
# K7Core code (pod lookup, deployment annotate/create for
# pause/resume/fork).
serviceAccountName: k7-api
# Dedicated SA with no token and no ClusterRole — VM ops are crictl
# + k7d.sock only. Kubernetes writes stay on k7-api.
serviceAccountName: k7-agent
automountServiceAccountToken: false
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
src/k7/deploy/manifests/k7-api/cilium/agent-networkpolicy.yaml
+8 -6
View File
@@ -1,9 +1,12 @@
# Restrict pod-originated ingress to the k7-agent DaemonSet to
# the k7-api pod only. `host` / `remote-node` entities stay allowed so
# kubelet health probes and root CLI usage on cluster nodes keep working —
# those callers still need the shared token (/etc/k7/agent_token, root
# 0600), which is the actual authentication. Sandbox pods and any other
# workload pods are denied at the network layer.
# the k7-api pod only. `host` stays allowed so kubelet health probes
# (and root CLI on THAT node) keep working — those callers still need
# the agent token (/etc/k7/agent_token, root 0600).
#
# `remote-node` is intentionally omitted: a compromised agent must not
# reach k7-agent pods on other workers. k7-api talks to agents on the
# pod network (`fromEndpoints: app=k7-api`). `k7 --core` VM ops for a
# sandbox on another node go through k7-api, not host-to-agent.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
@@ -19,4 +22,3 @@ spec:
app: k7-api
- fromEntities:
- host
- remote-node
src/k7/deploy/manifests/k7-api/serviceaccount.yaml
+11
View File
@@ -5,3 +5,14 @@ metadata:
namespace: kube-system
labels:
app: k7-api
---
# No ClusterRoleBinding. automount is false on the DaemonSet too — root on
# an agent must not steal k7-api RBAC.
apiVersion: v1
kind: ServiceAccount
metadata:
name: k7-agent
namespace: kube-system
labels:
app: k7-agent
automountServiceAccountToken: false
src/k7_sdk/__init__.py
+1 -1
View File
@@ -8,4 +8,4 @@ __all__ = [
"SandboxProxy",
]
__version__ = "0.3.1"
__version__ = "0.4.0"
src/k7_sdk/client.py
+1
View File
@@ -82,6 +82,7 @@ class Client:
"ingress_ports": [8000],
"ingress_from": ["sandbox:client-a"], # default: same-namespace sandboxes
# "expose_ports": [8000], # NodePort outside the cluster; needs the same ingress_ports
# "node_name": "k7-node-01", # pin; a node-scoped API key also stamps/rejects this
})
"""
response = self.session.post(f"{self.base_url}/api/v1/sandboxes", json=sandbox_config)
tests/integration/test_api.py
+73 -1
View File
@@ -7,6 +7,7 @@ ServiceAccount RBAC, and in-cluster config all work end-to-end.
import hashlib
import json
import os
import subprocess
import time
from pathlib import Path
@@ -93,7 +94,12 @@ def _api_pod_ready() -> bool:
return result.returncode == 0 and result.stdout.strip() not in ("", "0")
def _generate_test_api_key(*, name: str = "integration-test", namespaces: list[str] | None = None) -> str:
def _generate_test_api_key(
*,
name: str = "integration-test",
namespaces: list[str] | None = None,
nodes: list[str] | None = None,
) -> str:
"""Write a test API key directly into the keys file, return the raw token."""
import os
import secrets as _secrets
@@ -113,6 +119,8 @@ def _generate_test_api_key(*, name: str = "integration-test", namespaces: list[s
}
if namespaces:
entry["namespaces"] = list(namespaces)
if nodes:
entry["nodes"] = list(nodes)
keys[key_hash] = entry
K7_API_KEYS_FILE.parent.mkdir(parents=True, exist_ok=True)
K7_API_KEYS_FILE.write_text(json.dumps(keys, indent=2))
@@ -853,3 +861,67 @@ class TestApiNamespaceAuthz:
assert isinstance(data, dict) and data, f"expected per-node map, got {data}"
finally:
_revoke_test_api_key(scoped_name)
class TestApiNodeAuthz:
"""Node-scoped API keys may only place sandboxes on listed nodes."""
def test_single_node_key_auto_pins_and_rejects_other(
self,
api_base_url: str,
test_namespace: str,
cleanup_sandbox,
):
local = os.uname().nodename
scoped_name = "integ-scoped-node"
scoped = _generate_test_api_key(name=scoped_name, nodes=[local])
scoped_headers = {"X-API-Key": scoped}
name = "node-authz-sb"
cleanup_sandbox(name, test_namespace)
try:
r = httpx.post(
f"{api_base_url}/api/v1/sandboxes",
json={"name": name, "image": "alpine:3.21", "namespace": test_namespace},
headers=scoped_headers,
timeout=180,
)
assert r.status_code == 201, r.text
_wait_for_sandbox_ready(api_base_url, scoped_headers, name, test_namespace)
got = httpx.get(
f"{api_base_url}/api/v1/sandboxes/{name}",
params={"namespace": test_namespace},
headers=scoped_headers,
timeout=10,
)
assert got.status_code == 200, got.text
assert got.json()["data"].get("node") == local
denied = httpx.post(
f"{api_base_url}/api/v1/sandboxes",
json={
"name": f"{name}-x",
"image": "alpine:3.21",
"namespace": test_namespace,
"node_name": "not-a-k7-node",
},
headers=scoped_headers,
timeout=15,
)
assert denied.status_code == 403, denied.text
assert "not-a-k7-node" in denied.json()["error"]["message"]
finally:
_revoke_test_api_key(scoped_name)
def test_node_scoped_key_nodes_storage_forbidden(self, api_base_url: str):
scoped_name = "integ-scoped-nodes-only"
scoped = _generate_test_api_key(name=scoped_name, nodes=[os.uname().nodename])
try:
scoped_r = httpx.get(
f"{api_base_url}/api/v1/nodes/storage",
headers={"X-API-Key": scoped},
timeout=30,
)
assert scoped_r.status_code == 403, scoped_r.text
assert "cluster-wide node" in scoped_r.json()["error"]["message"]
finally:
_revoke_test_api_key(scoped_name)
tests/integration/test_docker.py
+21 -4
View File
@@ -145,6 +145,22 @@ def _k7d_docker_disk_names() -> set[str]:
return {p.name for p in _K7D_DISKS.iterdir() if "docker" in p.name}
async def _wait_k7d_docker_disks_gone(before: set[str], timeout: float = 60.0) -> None:
"""k7 delete returns when K8s objects are gone; k7d unlinks the graph later.
``delete_sandbox`` does not wait for VM teardown (API latency must not
couple to shim Delete). Poll here; a leftover after ``timeout`` is a leak.
"""
started = time.time()
leftover: set[str] = set()
while time.time() - started < timeout:
leftover = _k7d_docker_disk_names() - before
if not leftover:
return
await asyncio.sleep(1)
raise AssertionError(f"leaked k7d docker volume images after {time.time() - started:.1f}s: {leftover}")
def _pod_annotations(name: str, namespace: str) -> dict[str, str]:
result = subprocess.run(
[
@@ -237,9 +253,7 @@ class TestDockerK7d:
assert capbnd != "000001ffffffffff", f"sandbox CapBnd is full: {capbnd}"
finally:
await k7_core.delete_sandbox(name, namespace=test_namespace)
await asyncio.sleep(3)
leftover = _k7d_docker_disk_names() - before_disks
assert not leftover, f"leaked k7d docker volume images: {leftover}"
await _wait_k7d_docker_disks_gone(before_disks)
async def test_docker_build_and_compose(self, k7_core: K7Core, test_namespace: str):
"""docker build + compose of a one-service fixture."""
@@ -582,7 +596,10 @@ class TestDockerKQL(_DockerKataMixin):
await _sh(k7_core, source, test_namespace, "docker pull alpine:3.21")
fork = await k7_core.fork_sandbox(source, fork_name, namespace=test_namespace)
assert fork.success, fork.error
_wait_all_containers_ready(fork_name, test_namespace, timeout=240)
# Same 600s bound as test_qemu.test_fork_clones_data: on HA
# longhorn_replicas=3 the cloned volumes hydrate before first
# attach ("not ready for workloads"). 240s is a false timeout.
_wait_all_containers_ready(fork_name, test_namespace, timeout=600)
await _wait_docker_ready(k7_core, fork_name, test_namespace)
has = await _sh(
k7_core, fork_name, test_namespace, "docker image inspect alpine:3.21 >/dev/null && echo yes"
tests/integration/test_firecracker.py
+16 -10
View File
@@ -66,11 +66,13 @@ def _wait_for_pod_running(name: str, namespace: str, timeout: int = 120) -> None
def _get_live_firecracker_pids() -> list[str]:
"""Return PIDs of firecracker processes whose root fs is still live.
"""Return PIDs of live jailed Firecracker VMs.
Scans /proc directly instead of relying on pgrep. Skips stale
orphan processes whose chroot has been cleaned up (readlink shows
``(deleted)``).
Kata's binary is ``firecracker-v1.16.1``; Linux truncates ``comm`` to
15 characters (``firecracker-v1.``). Match by prefix, not equality.
The jailer pivot-roots in a private mount ns, so ``/proc/<pid>/root``
reads as ``/`` — skip ``(deleted)`` orphans and require ``vmlinux`` +
``rootfs`` in the jail.
"""
pids = []
for entry in os.listdir("/proc"):
@@ -78,10 +80,13 @@ def _get_live_firecracker_pids() -> list[str]:
continue
try:
with open(f"/proc/{entry}/comm") as f:
if f.read().strip() != "firecracker":
continue
root_link = os.readlink(f"/proc/{entry}/root")
if "(deleted)" in root_link:
comm = f.read().strip()
if comm != "firecracker" and not comm.startswith("firecracker-"):
continue
if "(deleted)" in os.readlink(f"/proc/{entry}/root"):
continue
fc_root = f"/proc/{entry}/root"
if not os.path.isfile(f"{fc_root}/vmlinux") or not os.path.isfile(f"{fc_root}/rootfs"):
continue
pids.append(entry)
except (OSError, PermissionError):
@@ -215,8 +220,9 @@ class TestFirecrackerBackend:
f"Firecracker PID {pid} can see /boot — host root filesystem is exposed. Jailer not active."
)
assert os.path.isfile(f"{fc_root}/firecracker"), (
f"Firecracker PID {pid} chroot missing firecracker binary"
fc_bins = [e for e in os.listdir(fc_root) if e.startswith("firecracker")]
assert fc_bins, (
f"Firecracker PID {pid} chroot missing firecracker* binary; entries={sorted(os.listdir(fc_root))}"
)
assert os.path.isfile(f"{fc_root}/vmlinux"), f"Firecracker PID {pid} chroot missing vmlinux kernel"
assert os.path.isfile(f"{fc_root}/rootfs"), f"Firecracker PID {pid} chroot missing rootfs image"
tests/integration/test_pause_resume.py
+51 -1
View File
@@ -10,6 +10,7 @@ We invoke ``k7`` via ``src/k7/cli/dev.sh`` so we don't depend on a packaged
binary being installed on the node.
"""
import asyncio
import subprocess
import time
from pathlib import Path
@@ -106,7 +107,7 @@ def k7_core() -> K7Core:
@pytest.fixture()
def ql_sandbox(k7_core: K7Core, test_namespace: str):
"""Provision a fresh kata-qemu-longhorn sandbox; yield its name; tear down."""
name = f"pause-spec10c-{uuid4().hex[:6]}"
name = f"pause-qemu-{uuid4().hex[:6]}"
import asyncio
cfg = SandboxConfig(
@@ -225,6 +226,55 @@ class TestPauseSnapshotBugRegression:
assert sandbox_snaps == [], f"unexpected snapshot(s) created for {ql_sandbox}: {sandbox_snaps}"
def _runtimeclass_present(name: str) -> bool:
out = subprocess.run([_K3S, "kubectl", "get", "runtimeclass", name], capture_output=True, text=True, check=False)
return out.returncode == 0
class TestPauseResumeExecAnswers:
"""``pause`` → ``resume`` → ``exec_command("echo hi")`` answers, on the
daemon-socket backends.
CHALLENGES #17: on ``k7d-fc`` the exec after resume hung — kubelet's
probe timed out and ``_wait_exec`` never returned. Root cause was
in Firecracker's vsock device (k7d CHALLENGES, jail-integrity fix); the k7d
tarball this pins carries the fix. ``k7d`` is the control.
"""
_EXEC_BUDGET_S = 15.0
@pytest.mark.parametrize(("backend", "runtime_class"), [("k7d", "k7"), ("k7d-fc", "k7-fc")])
async def test_pause_resume_exec_answers(
self, k7_core: K7Core, test_namespace: str, backend: str, runtime_class: str
):
if not _runtimeclass_present(runtime_class):
pytest.skip(f"RuntimeClass {runtime_class} not registered")
name = f"pause-exec-{backend}-{uuid4().hex[:6]}"
cfg = SandboxConfig(name=name, image="alpine:3.20", namespace=test_namespace, backend=backend)
result = await k7_core.create_sandbox(cfg)
assert result.success, f"{backend}: create failed: {result.error}"
try:
_wait_pod_ready(name, test_namespace)
before = await k7_core.exec_command(name, "echo hi", namespace=test_namespace)
assert "hi" in (before.stdout or ""), f"{backend}: exec before pause: {before!r}"
paused = await k7_core.pause_sandbox(name, namespace=test_namespace)
assert paused.success, f"{backend}: pause failed: {paused.error}"
await asyncio.sleep(3)
resumed = await k7_core.resume_sandbox(name, namespace=test_namespace)
assert resumed.success, f"{backend}: resume failed: {resumed.error}"
t0 = time.monotonic()
# ``exec_command`` has no deadline of its own; the #17 symptom
# was a hang, so a regression must fail here, not park pytest.
after = await asyncio.wait_for(k7_core.exec_command(name, "echo hi", namespace=test_namespace), timeout=60)
elapsed = time.monotonic() - t0
assert "hi" in (after.stdout or ""), f"{backend}: exec after resume: {after!r}"
assert elapsed < self._EXEC_BUDGET_S, f"{backend}: resume → exec took {elapsed:.2f}s"
finally:
await k7_core.delete_sandbox(name, namespace=test_namespace)
class TestRemovedPauseFlags:
"""``--pvc`` and ``--snapshot-class`` were removed from ``k7 pause``."""
tests/integration/test_sandbox_ingress.py
+7
View File
@@ -7,6 +7,7 @@ ingress no matter how locked down its source was.
import asyncio
import ipaddress
import os
import subprocess
import time
@@ -19,6 +20,7 @@ from k7.core.models import OperationResult, SandboxConfig
pytestmark = pytest.mark.integration
_K3S = "/usr/local/bin/k3s"
_LOCAL_NODE = os.uname().nodename
def _cilium_crd_present() -> bool:
@@ -449,6 +451,10 @@ class TestSandboxExpose:
ingress_ports=[8000],
ingress_from=sources,
expose_ports=[8000],
# Pin to the suite node. ``externalTrafficPolicy: Local`` only
# answers on the pod's node; in-cluster curl of another node's
# NodePort is intercepted by Cilium socket-LB and times out.
node_name=_LOCAL_NODE,
)
result = await k7_core.create_sandbox(cfg)
assert result.success, f"create {name} failed: {result.error}"
@@ -458,6 +464,7 @@ class TestSandboxExpose:
return endpoints[0]["url"]
async def test_expose_is_reachable_and_cleaned_up(self, k7_core: K7Core, test_namespace: str):
"""NodePort answers on the suite node; Local + Cilium is not Cluster."""
name = "integ-exp-open"
try:
url = await self._exposed(k7_core, test_namespace, name, ["cidr:0.0.0.0/0"])
tests/unit/test_agent.py
+61
View File
@@ -0,0 +1,61 @@
"""k7-agent is node-local only: no Kubernetes writes, fork refused."""
from pathlib import Path
from unittest.mock import AsyncMock, patch
import httpx
from k7.api import agent as agent_mod
from k7.api.agent import app
TOKEN = "agent-secret"
def _client(tmp_path: Path, monkeypatch):
token_file = tmp_path / "agent_token"
token_file.write_text(TOKEN)
monkeypatch.setattr(agent_mod, "AGENT_TOKEN_FILE", str(token_file))
return httpx.ASGITransport(app=app)
async def test_fork_refused(tmp_path: Path, monkeypatch):
transport = _client(tmp_path, monkeypatch)
async with httpx.AsyncClient(transport=transport, base_url="http://agent") as client:
r = await client.post(
"/agent/v1/vm/fork",
json={"name": "src", "new_name": "dst"},
headers={"X-K7-Agent-Token": TOKEN},
)
assert r.status_code == 400
assert "control-plane" in r.json()["detail"]
async def test_pause_is_crictl_and_k7d_only(tmp_path: Path, monkeypatch):
transport = _client(tmp_path, monkeypatch)
vm = {"vm_id": "vm-1", "status": "sandbox_found", "sandbox_id": "cri"}
with (
patch("k7.api.agent._lookup_local_vm", new=AsyncMock(return_value=vm)),
patch("k7.api.agent.K7Core") as core_cls,
):
core_cls.return_value._k7d_request = AsyncMock(return_value={"status": "ok"})
async with httpx.AsyncClient(transport=transport, base_url="http://agent") as client:
r = await client.post(
"/agent/v1/vm/pause",
json={"name": "sb", "namespace": "ns", "pod_name": "sb-pod"},
headers={"X-K7-Agent-Token": TOKEN},
)
assert r.status_code == 200, r.text
assert r.json()["success"] is True
core_cls.return_value._k7d_request.assert_awaited_once_with({"op": "pause_vm", "vm_id": "vm-1"})
async def test_pause_requires_pod_name(tmp_path: Path, monkeypatch):
transport = _client(tmp_path, monkeypatch)
async with httpx.AsyncClient(transport=transport, base_url="http://agent") as client:
r = await client.post(
"/agent/v1/vm/pause",
json={"name": "sb"},
headers={"X-K7-Agent-Token": TOKEN},
)
assert r.status_code == 400
assert "pod_name" in r.json()["detail"]
tests/unit/test_agent_manifests.py
+29
View File
@@ -0,0 +1,29 @@
"""Agent must not inherit k7-api ClusterRole or accept remote-node ingress."""
from pathlib import Path
MANIFESTS = Path(__file__).resolve().parents[2] / "src/k7/deploy/manifests/k7-api"
def test_agent_daemonset_has_no_api_rbac():
ds = (MANIFESTS / "agent-daemonset.yaml").read_text()
assert "serviceAccountName: k7-agent" in ds
assert "automountServiceAccountToken: false" in ds
assert "serviceAccountName: k7-api" not in ds
def test_agent_sa_has_no_clusterrolebinding():
binding = (MANIFESTS / "clusterrolebinding.yaml").read_text()
assert "name: k7-api" in binding
assert "k7-agent" not in binding
sa = (MANIFESTS / "serviceaccount.yaml").read_text()
assert "name: k7-agent" in sa
assert "automountServiceAccountToken: false" in sa
def test_agent_cnp_denies_remote_node():
cnp = (MANIFESTS / "cilium/agent-networkpolicy.yaml").read_text()
spec = cnp.split("spec:", 1)[1]
assert "app: k7-api" in spec
assert "- host" in spec
assert "remote-node" not in spec
tests/unit/test_api_auth.py
+122 -1
View File
@@ -4,13 +4,15 @@ import hashlib
import json
import time
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import AsyncMock, patch
import httpx
import pytest
from fastapi import HTTPException
from k7.api.main import app, authorize_namespace, load_api_keys
from k7.api.main import app, apply_node_scope, authorize_namespace, load_api_keys
from k7.core.models import OperationResult
TEST_KEY = "k7-test-secret-key-abc123"
TEST_KEY_HASH = hashlib.sha256(TEST_KEY.encode()).hexdigest()
@@ -21,6 +23,7 @@ def _make_keys_data(
expires: int | None = None,
last_used: int | None = None,
namespaces: list[str] | None = None,
nodes: list[str] | None = None,
) -> dict:
entry: dict = {"name": "test-key"}
if expires is not None:
@@ -29,6 +32,8 @@ def _make_keys_data(
entry["last_used"] = last_used
if namespaces is not None:
entry["namespaces"] = namespaces
if nodes is not None:
entry["nodes"] = nodes
return {TEST_KEY_HASH: entry}
@@ -217,3 +222,119 @@ class TestAuthorizeNamespace:
assert resp.status_code == 200
assert resp.json()["data"] == payload
core_cls.return_value.nodes_storage.assert_awaited_once()
# --- apply_node_scope ---
class TestApplyNodeScope:
def test_unrestricted_key_keeps_caller_pin(self):
assert apply_node_scope({"name": "u"}, None) is None
assert apply_node_scope({"name": "u", "nodes": []}, "k7-node-01") == "k7-node-01"
def test_single_node_key_auto_pins(self):
assert apply_node_scope({"nodes": ["k7-node-01"]}, None) == "k7-node-01"
def test_single_node_key_allows_listed_node(self):
assert apply_node_scope({"nodes": ["k7-node-01"]}, "k7-node-01") == "k7-node-01"
def test_scoped_key_denied_other_node(self):
with pytest.raises(HTTPException) as exc:
apply_node_scope({"nodes": ["k7-node-01"]}, "k7-node-02")
assert exc.value.status_code == 403
assert "k7-node-02" in str(exc.value.detail)
def test_multi_node_key_requires_explicit_node(self):
with pytest.raises(HTTPException) as exc:
apply_node_scope({"nodes": ["k7-node-01", "k7-node-02"]}, None)
assert exc.value.status_code == 403
assert "explicit allowed node" in str(exc.value.detail)
def test_multi_node_key_allows_listed_node(self):
assert apply_node_scope({"nodes": ["k7-node-01", "k7-node-02"]}, "k7-node-02") == "k7-node-02"
async def test_single_node_key_create_stamps_node_name(self, _patch_keys_file, keys_file: Path):
future_ts = int(time.time()) + 86400
data = _make_keys_data(expires=future_ts, nodes=["k7-node-01"])
keys_file.write_text(json.dumps(data))
with patch("k7.api.main.K7Core") as core_cls:
core_cls.return_value.create_sandbox = AsyncMock(return_value=OperationResult(success=True))
transport = httpx.ASGITransport(app=app)
async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
resp = await client.post(
"/api/v1/sandboxes",
headers={"X-API-Key": TEST_KEY},
json={"name": "t", "image": "alpine"},
)
assert resp.status_code == 201, resp.text
cfg = core_cls.return_value.create_sandbox.await_args.args[0]
assert cfg.node_name == "k7-node-01"
async def test_scoped_key_create_other_node_returns_403(self, _patch_keys_file, keys_file: Path):
future_ts = int(time.time()) + 86400
data = _make_keys_data(expires=future_ts, nodes=["k7-node-01"])
keys_file.write_text(json.dumps(data))
with patch("k7.api.main.K7Core") as core_cls:
core_cls.return_value.create_sandbox = AsyncMock(return_value=OperationResult(success=True))
transport = httpx.ASGITransport(app=app)
async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
resp = await client.post(
"/api/v1/sandboxes",
headers={"X-API-Key": TEST_KEY},
json={"name": "t", "image": "alpine", "node_name": "k7-node-02"},
)
assert resp.status_code == 403
core_cls.return_value.create_sandbox.assert_not_called()
async def test_node_scoped_key_nodes_storage_returns_403(self, _patch_keys_file, keys_file: Path):
future_ts = int(time.time()) + 86400
data = _make_keys_data(expires=future_ts, nodes=["k7-node-01"])
keys_file.write_text(json.dumps(data))
transport = httpx.ASGITransport(app=app)
async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
resp = await client.get(
"/api/v1/nodes/storage",
headers={"X-API-Key": TEST_KEY},
)
assert resp.status_code == 403
assert "cluster-wide node" in resp.json()["error"]["message"]
async def test_restore_stamps_node_name(self, _patch_keys_file, keys_file: Path):
future_ts = int(time.time()) + 86400
data = _make_keys_data(expires=future_ts, nodes=["k7-node-01"])
keys_file.write_text(json.dumps(data))
with patch("k7.api.main.K7Core") as core_cls:
core_cls.return_value.restore_sandbox = AsyncMock(return_value=OperationResult(success=True))
transport = httpx.ASGITransport(app=app)
async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
resp = await client.post(
"/api/v1/snapshots/snap1/restore",
headers={"X-API-Key": TEST_KEY},
json={"new_sandbox_name": "restored"},
)
assert resp.status_code == 201, resp.text
kwargs = core_cls.return_value.restore_sandbox.await_args.kwargs
assert kwargs["node_name"] == "k7-node-01"
async def test_fork_denied_when_source_on_other_node(self, _patch_keys_file, keys_file: Path):
future_ts = int(time.time()) + 86400
data = _make_keys_data(expires=future_ts, nodes=["k7-node-01"])
keys_file.write_text(json.dumps(data))
source = SimpleNamespace(name="src", node="k7-node-02")
with patch("k7.api.main.K7Core") as core_cls:
core_cls.return_value.list_sandboxes = AsyncMock(return_value=[source])
core_cls.return_value.fork_sandbox = AsyncMock(return_value=OperationResult(success=True))
transport = httpx.ASGITransport(app=app)
async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
resp = await client.post(
"/api/v1/sandboxes/src/fork",
headers={"X-API-Key": TEST_KEY},
json={"new_name": "forked"},
)
assert resp.status_code == 403
core_cls.return_value.fork_sandbox.assert_not_called()
tests/unit/test_api_lifecycle_endpoints.py
+1 -1
View File
@@ -17,7 +17,7 @@ import pytest
from k7.api.main import app
from k7.core.models import OperationResult
TEST_KEY = "k7-test-secret-key-spec10a"
TEST_KEY = "k7-test-secret-key-lifecycle"
TEST_KEY_HASH = hashlib.sha256(TEST_KEY.encode()).hexdigest()
tests/unit/test_cli_api_keys.py
+57 -1
View File
@@ -2,7 +2,7 @@
import json
from pathlib import Path
from unittest.mock import patch
from unittest.mock import AsyncMock, patch
from typer.testing import CliRunner
@@ -34,6 +34,20 @@ class TestGenerateApiKeyNamespaces:
data = json.loads(keys_file.read_text())
entry = next(iter(data.values()))
assert "namespaces" not in entry
assert "nodes" not in entry
def test_scoped_nodes_persisted(self, tmp_path: Path):
keys_file = tmp_path / "api_keys.json"
with patch("k7.cli.k7.API_KEYS_FILE", keys_file):
result = runner.invoke(
app,
["generate-api-key", "pinned", "--node", "k7-node-01", "--node", "k7-node-02"],
)
assert result.exit_code == 0, result.output
data = json.loads(keys_file.read_text())
entry = next(iter(data.values()))
assert entry["nodes"] == ["k7-node-01", "k7-node-02"]
assert "Nodes: k7-node-01, k7-node-02" in result.output
def test_list_shows_namespaces_column(self, tmp_path: Path):
keys_file = tmp_path / "api_keys.json"
@@ -53,6 +67,13 @@ class TestGenerateApiKeyNamespaces:
"expires": 2,
"last_used": None,
},
"h3": {
"name": "pinned",
"created": 1,
"expires": 2,
"last_used": None,
"nodes": ["k7-node-01"],
},
}
)
)
@@ -60,5 +81,40 @@ class TestGenerateApiKeyNamespaces:
result = runner.invoke(app, ["list-api-keys"])
assert result.exit_code == 0, result.output
assert "Namespaces" in result.output
assert "Nodes" in result.output
assert "alpha" in result.output
assert "k7-node-01" in result.output
assert "*" in result.output
class TestNodesDedicateCli:
def test_dedicate_requires_tenant(self):
result = runner.invoke(app, ["nodes", "dedicate", "k7-node-01"])
assert result.exit_code != 0
def test_dedicate_calls_core(self):
from k7.core.models import OperationResult
with patch("k7.cli.k7.K7Core") as core_cls:
core_cls.return_value.dedicate_node = AsyncMock(return_value=OperationResult(success=True, message="ok"))
result = runner.invoke(app, ["nodes", "dedicate", "k7-node-01", "--tenant", "acme"])
assert result.exit_code == 0, result.output
core_cls.return_value.dedicate_node.assert_awaited_once_with("k7-node-01", "acme")
class TestNodesListCli:
def test_json_prints_k3s_names(self):
rows = [
{
"name": "k7-node-01",
"hostname": "k7-node-01",
"backends": ["k7d"],
"tenant": "",
}
]
with patch("k7.cli.k7.K7Core") as core_cls:
core_cls.return_value.list_cluster_nodes = AsyncMock(return_value=rows)
result = runner.invoke(app, ["nodes", "list", "--json"])
assert result.exit_code == 0, result.output
assert "k7-node-01" in result.output
assert "k7d" in result.output
tests/unit/test_cli_api_lifecycle.py
+44 -4
View File
@@ -70,6 +70,12 @@ class TestApiEnableDisable:
assert result.exit_code != 0
assert "failed" in result.output.lower()
def test_enable_without_kubectl_exits(self):
with patch("k7.cli.k7._kubectl_run", return_value=None):
result = runner.invoke(app, ["api", "enable"])
assert result.exit_code != 0
assert "kubectl" in result.output.lower()
# ---------------------------------------------------------------------------
# ``k7 api status``.
@@ -114,6 +120,30 @@ class TestApiStatus:
assert "not found" in result.output.lower()
assert "k7 install" in result.output
def test_without_kubectl_probes_configured_health(self):
fake = SimpleNamespace(ok=True, status_code=200)
with (
patch("k7.cli.k7._kubectl_run", return_value=None),
patch("k7.cli.k7._resolve_api_url", return_value="https://10.0.0.1:31007"),
patch("k7.cli.k7._resolve_api_ca", return_value="/tmp/ca.crt"),
patch("k7.cli.k7.requests.get", return_value=fake) as get,
):
result = runner.invoke(app, ["api", "status"])
assert result.exit_code == 0, result.output
assert "reachable" in result.output.lower()
assert "https://10.0.0.1:31007" in result.output
get.assert_called_once()
assert get.call_args.args[0] == "https://10.0.0.1:31007/health"
def test_without_kubectl_or_url_exits(self):
with (
patch("k7.cli.k7._kubectl_run", return_value=None),
patch("k7.cli.k7._resolve_api_url", return_value=None),
):
result = runner.invoke(app, ["api", "status"])
assert result.exit_code != 0
assert "kubectl is not installed" in result.output
# ---------------------------------------------------------------------------
# ``k7 api endpoint``.
@@ -132,6 +162,16 @@ class TestApiEndpoint:
result = runner.invoke(app, ["api", "endpoint"])
assert result.exit_code != 0
def test_without_kubectl_prints_configured_url(self, tmp_path):
with (
patch("k7.cli.k7._ETC_K7_API_ENDPOINT", tmp_path / "missing"),
patch("k7.cli.k7.subprocess.run", side_effect=FileNotFoundError("kubectl")),
patch("k7.cli.k7._resolve_api_url", return_value="https://10.0.0.1:31007"),
):
result = runner.invoke(app, ["api", "endpoint"])
assert result.exit_code == 0, result.output
assert result.output.strip() == "https://10.0.0.1:31007"
# ---------------------------------------------------------------------------
# Deprecated command aliases.
@@ -201,7 +241,7 @@ class TestInstallNoApi:
patch("k7.cli.k7._build_default_inventory", return_value="[k7_servers]\nhost ansible_host=1.2.3.4"),
):
core_cls.return_value.install_node = _capture
result = runner.invoke(app, ["install", "--no-api", "host"])
result = runner.invoke(app, ["install", "--backend", "kql", "--no-api", "host"])
assert result.exit_code == 0, result.output
extra_vars = seen["kwargs"]["extra_vars"]
assert extra_vars.get("k7_api_enabled") == "false"
@@ -218,7 +258,7 @@ class TestInstallNoApi:
patch("k7.cli.k7._build_default_inventory", return_value="[k7_servers]\nhost ansible_host=1.2.3.4"),
):
core_cls.return_value.install_node = _capture
result = runner.invoke(app, ["install", "host"])
result = runner.invoke(app, ["install", "--backend", "kql", "host"])
assert result.exit_code == 0, result.output
extra_vars = seen["kwargs"]["extra_vars"]
assert extra_vars.get("k7_api_enabled") == "true"
@@ -237,7 +277,7 @@ class TestInstallHubble:
patch("k7.cli.k7._build_default_inventory", return_value="[k7_servers]\nhost ansible_host=1.2.3.4"),
):
core_cls.return_value.install_node = _capture
result = runner.invoke(app, ["install", "--hubble", "host"])
result = runner.invoke(app, ["install", "--backend", "kql", "--hubble", "host"])
assert result.exit_code == 0, result.output
extra_vars = seen["kwargs"]["extra_vars"]
assert extra_vars.get("k7_hubble_enabled") == "true"
@@ -254,7 +294,7 @@ class TestInstallHubble:
patch("k7.cli.k7._build_default_inventory", return_value="[k7_servers]\nhost ansible_host=1.2.3.4"),
):
core_cls.return_value.install_node = _capture
result = runner.invoke(app, ["install", "host"])
result = runner.invoke(app, ["install", "--backend", "kql", "host"])
assert result.exit_code == 0, result.output
extra_vars = seen["kwargs"]["extra_vars"]
assert extra_vars.get("k7_hubble_enabled") == "false"
tests/unit/test_cli_install_api_allowlist.py
+2
View File
@@ -16,6 +16,8 @@ runner = CliRunner()
def _install(args: list[str]):
if "--backend" not in args and "-b" not in args:
args = ["--backend", "kql", *args]
with patch("k7.cli.k7.K7Core") as core_cls:
install = MagicMock(return_value=OperationResult(success=True, message="ok"))
core_cls.return_value.install_node = install
tests/unit/test_cli_install_api_tls.py
+2
View File
@@ -22,6 +22,8 @@ _CLUSTERROLE = (Path(__file__).resolve().parents[2] / "src/k7/deploy/manifests/k
def _install(args: list[str]):
if "--backend" not in args and "-b" not in args:
args = ["--backend", "kql", *args]
with patch("k7.cli.k7.K7Core") as core_cls:
install = MagicMock(return_value=OperationResult(success=True, message="ok"))
core_cls.return_value.install_node = install
tests/unit/test_cli_install_backends.py
+53
View File
@@ -0,0 +1,53 @@
"""``k7 install`` requires --backend; none is the empty set."""
from pathlib import Path
from unittest.mock import MagicMock, patch
from typer.testing import CliRunner
from k7.cli.k7 import INSTALL_BACKEND_REQUIRED, app
from k7.core.models import OperationResult
runner = CliRunner()
def _install(args: list[str]):
with patch("k7.cli.k7.K7Core") as core_cls:
install = MagicMock(return_value=OperationResult(success=True, message="ok"))
core_cls.return_value.install_node = install
result = runner.invoke(app, ["--core", "install", *args])
return result, install
def test_install_without_backend_fails():
result, install = _install([])
assert result.exit_code != 0
assert "Specify --backend" in result.output
assert "none" in result.output
install.assert_not_called()
assert "kfd" in INSTALL_BACKEND_REQUIRED
def test_install_backend_none_forwards_sentinel():
result, install = _install(["--backend", "none"])
assert result.exit_code == 0, result.output
extra = install.call_args.kwargs["extra_vars"]
assert extra["k7_backends"] == "none"
inv = install.call_args.args[1]
assert "k7_backends=none" in inv
def test_install_inventory_without_backend_does_not_send_extra_var(tmp_path: Path):
inv = tmp_path / "inventory.ini"
inv.write_text("[k7_servers]\nhost ansible_host=1.2.3.4 k7_backends=kql\n")
result, install = _install(["-i", str(inv)])
assert result.exit_code == 0, result.output
extra = install.call_args.kwargs["extra_vars"]
assert "k7_backends" not in extra
def test_install_none_mixed_fails_before_ansible():
result, install = _install(["--backend", "none,kql"])
assert result.exit_code != 0
assert "cannot be combined" in result.output
install.assert_not_called()
tests/unit/test_cli_utils.py
+55 -2
View File
@@ -9,6 +9,7 @@ import typer
from k7.cli.k7 import (
_build_default_inventory,
_core_run,
_kubectl_cmd,
_normalize_backend,
_parse_backends,
@@ -159,6 +160,13 @@ class TestReadApiEndpoint:
result = _read_api_endpoint(["kubectl"])
assert result == "https://10.0.0.1:31007"
def test_falls_back_to_configured_url_when_kubectl_missing(self):
with (
patch("k7.cli.k7.subprocess.run", side_effect=FileNotFoundError("kubectl")),
patch("k7.cli.k7._resolve_api_url", return_value="https://10.0.0.1:31007"),
):
assert _read_api_endpoint(["kubectl"]) == "https://10.0.0.1:31007"
# --- _build_default_inventory ---
@@ -178,8 +186,17 @@ class TestBuildDefaultInventory:
assert "[k7_agents]" in inv
assert "[k7_cluster:children]" in inv
def test_localhost_both_backends_default(self):
# `k7 install` with no flags installs both backends by default.
def test_localhost_none_backend(self):
inv = _build_default_inventory(
hosts=None,
role="server",
backends=[],
disk=None,
longhorn_extra_disk=None,
)
assert "k7_backends=none" in inv
def test_localhost_both_kata_backends(self):
inv = _build_default_inventory(
hosts=None,
role="server",
@@ -290,3 +307,39 @@ class TestParseBackends:
_parse_backends("")
with pytest.raises(typer.BadParameter, match="cannot be empty"):
_parse_backends(" , , ")
def test_none_is_empty_set(self):
assert _parse_backends("none") == []
assert _parse_backends("None") == []
def test_none_cannot_mix(self):
with pytest.raises(typer.BadParameter, match="cannot be combined"):
_parse_backends("none,kql")
with pytest.raises(typer.BadParameter, match="cannot be combined"):
_parse_backends("kfd,none")
class TestCoreRun:
def test_closes_real_k7core(self):
from k7.core.core import K7Core
core = K7Core()
closed: list[bool] = []
async def aclose():
closed.append(True)
async def work():
return "ok"
core.aclose = aclose # type: ignore[method-assign]
assert _core_run(core, work()) == "ok"
assert closed == [True]
def test_skips_aclose_on_mocks(self):
fake = MagicMock()
async def work():
return 7
assert _core_run(fake, work()) == 7
tests/unit/test_core_k7d.py
+130 -4
View File
@@ -5,11 +5,13 @@ are never touched. The real end-to-end behaviour is covered by
``tests/integration/test_k7d.py`` on a k7d-capable node.
"""
from contextlib import contextmanager
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from k7.core.core import (
K7_TENANT_LABEL,
K7D_ANN_FORK_SOURCE_CLUSTER,
K7D_ANN_FORK_SOURCE_VM,
K7D_ANN_PAUSED,
@@ -20,10 +22,15 @@ from k7.core.models import OperationResult, SandboxConfig
from tests.unit.conftest import mock_deployment, mock_pod
def _local_node(core: K7Core):
"""Patch the node resolution so the sandbox appears LOCAL —
these tests exercise the direct daemon-socket path, not forwarding."""
return patch.object(core, "_k7d_sandbox_node", new=AsyncMock(return_value=core._k7d_local_node()))
@contextmanager
def _local_node(core: K7Core, node: str = "k7-node-01", pod: str = "src-sb-abc"):
"""Sandbox appears LOCAL — direct daemon-socket path, not agent forward."""
with (
patch.object(core, "_k7d_local_node", return_value=node),
patch.object(core, "_k7d_sandbox_node", new=AsyncMock(return_value=node)),
patch.object(core, "_k7d_sandbox_placement", new=AsyncMock(return_value=(node, pod))),
):
yield
def _k7d_deployment(name: str = "src-sb", sidecar: str | None = None):
@@ -101,6 +108,76 @@ class TestK7dCreateSandbox:
# Deployment carries the backend annotation for later detection.
assert deployment.metadata.annotations["k7.katakate.org/backend"] == "k7d"
async def test_node_name_uses_hostname_selector_not_spec_node_name(self, core: K7Core):
mock_apps = AsyncMock()
mock_v1 = AsyncMock()
mock_net = AsyncMock()
node = MagicMock()
node.spec.taints = []
mock_v1.read_node.return_value = node
core._apps_v1_client = mock_apps
core._core_v1_client = mock_v1
core._networking_v1_client = mock_net
sched_ok = MagicMock()
sched_ok.success = True
with patch.object(core, "_check_scheduling", new=AsyncMock(return_value=sched_ok)):
cfg = SandboxConfig(name="k7d-pin", image="alpine:3.20", backend="k7d", node_name="k7-node-01")
result = await core.create_sandbox(cfg)
assert result.success, result.error
pod_spec = mock_apps.create_namespaced_deployment.call_args.kwargs["body"].spec.template.spec
assert pod_spec.node_name in (None, "")
assert pod_spec.node_selector == {
"k7.katakate.org/backend-k7d": "true",
"kubernetes.io/hostname": "k7-node-01",
}
assert not pod_spec.tolerations
mock_v1.read_node.assert_awaited_once_with("k7-node-01")
async def test_dedicated_node_adds_tenant_toleration(self, core: K7Core):
mock_apps = AsyncMock()
mock_v1 = AsyncMock()
mock_net = AsyncMock()
taint = MagicMock()
taint.key = "k7.katakate.org/tenant"
taint.value = "acme"
taint.effect = "NoSchedule"
node = MagicMock()
node.spec.taints = [taint]
mock_v1.read_node.return_value = node
core._apps_v1_client = mock_apps
core._core_v1_client = mock_v1
core._networking_v1_client = mock_net
sched_ok = MagicMock()
sched_ok.success = True
with patch.object(core, "_check_scheduling", new=AsyncMock(return_value=sched_ok)):
cfg = SandboxConfig(name="k7d-ten", image="alpine:3.20", backend="k7d", node_name="k7-node-01")
result = await core.create_sandbox(cfg)
assert result.success, result.error
pod_spec = mock_apps.create_namespaced_deployment.call_args.kwargs["body"].spec.template.spec
assert pod_spec.tolerations
tol = pod_spec.tolerations[0]
assert tol.key == "k7.katakate.org/tenant"
assert tol.value == "acme"
assert tol.effect == "NoSchedule"
async def test_missing_pin_node_fails_loud(self, core: K7Core):
from kubernetes_asyncio.client.exceptions import ApiException
mock_v1 = AsyncMock()
mock_v1.read_node.side_effect = ApiException(status=404)
core._apps_v1_client = AsyncMock()
core._core_v1_client = mock_v1
core._networking_v1_client = AsyncMock()
cfg = SandboxConfig(name="k7d-gone", image="alpine:3.20", backend="k7d", node_name="no-such-node")
result = await core.create_sandbox(cfg)
assert not result.success
assert "no-such-node" in (result.error or "")
core._apps_v1_client.create_namespaced_deployment.assert_not_called()
async def test_create_k7d_fc_uses_runtime_class_k7_fc(self, core: K7Core):
mock_apps = AsyncMock()
mock_v1 = AsyncMock()
@@ -481,3 +558,52 @@ class TestK7dRequest:
pytest.raises(RuntimeError, match="no such VM"),
):
await core._k7d_request({"op": "pause_vm", "vm_id": "vm-9"})
class TestDedicateNode:
async def test_dedicate_writes_label_and_taint(self, core: K7Core):
mock_v1 = AsyncMock()
node = MagicMock()
node.spec.taints = []
mock_v1.read_node.return_value = node
core._core_v1_client = mock_v1
result = await core.dedicate_node("k7-node-01", "acme")
assert result.success, result.error
body = mock_v1.patch_node.await_args.args[1]
assert body["metadata"]["labels"][K7_TENANT_LABEL] == "acme"
assert body["spec"]["taints"] == [{"key": K7_TENANT_LABEL, "value": "acme", "effect": "NoSchedule"}]
async def test_dedicate_missing_node_fails_loud(self, core: K7Core):
from kubernetes_asyncio.client.exceptions import ApiException
mock_v1 = AsyncMock()
mock_v1.read_node.side_effect = ApiException(status=404)
core._core_v1_client = mock_v1
result = await core.dedicate_node("missing", "acme")
assert not result.success
assert "not found" in (result.error or "")
mock_v1.patch_node.assert_not_called()
async def test_undedicate_drops_label_and_taint(self, core: K7Core):
mock_v1 = AsyncMock()
other = MagicMock()
other.key = "node.kubernetes.io/unreachable"
other.value = None
other.effect = "NoExecute"
ours = MagicMock()
ours.key = K7_TENANT_LABEL
ours.value = "acme"
ours.effect = "NoSchedule"
node = MagicMock()
node.spec.taints = [other, ours]
mock_v1.read_node.return_value = node
core._core_v1_client = mock_v1
result = await core.undedicate_node("k7-node-01")
assert result.success, result.error
body = mock_v1.patch_node.await_args.args[1]
assert body["metadata"]["labels"][K7_TENANT_LABEL] is None
assert body["spec"]["taints"] == [
{"key": "node.kubernetes.io/unreachable", "value": None, "effect": "NoExecute"}
]
tests/unit/test_core_mocked.py
+159 -38
View File
@@ -5,8 +5,8 @@ from unittest.mock import AsyncMock, MagicMock, patch
from kubernetes_asyncio.client.exceptions import ApiException
from k7.core.core import K7Core
from k7.core.models import OperationResult, SandboxConfig
from k7.core.core import K7_TENANT_LABEL, K7Core
from k7.core.models import ExecResult, OperationResult, SandboxConfig
from tests.unit.conftest import mock_deployment, mock_pod
# ---------------------------------------------------------------------------
@@ -14,7 +14,7 @@ from tests.unit.conftest import mock_deployment, mock_pod
# ---------------------------------------------------------------------------
def _setup_clients(core, apps=None, v1=None, net=None, metrics=None, custom=None):
def _setup_clients(core, apps=None, v1=None, net=None, metrics=None, custom=None, batch=None, apiext=None):
"""Wire MagicMock clients into a K7Core instance."""
core._config_loaded = True
if apps is not None:
@@ -27,6 +27,10 @@ def _setup_clients(core, apps=None, v1=None, net=None, metrics=None, custom=None
core._metrics_client = metrics
if custom is not None:
core._custom_objects_client = custom
if batch is not None:
core._batch_v1_client = batch
if apiext is not None:
core._apiextensions_v1_client = apiext
# --- _detect_backend ---
@@ -72,6 +76,12 @@ class TestDetectBackend:
with patch("os.path.exists", return_value=False):
assert await core._detect_backend("missing", "default") == "kata-firecracker-devmapper"
async def test_none_file_is_not_kfd(self, core: K7Core, tmp_path):
backend_file = tmp_path / "backend"
backend_file.write_text("none\n")
with patch("os.path.exists", return_value=True), patch("builtins.open", return_value=open(backend_file)):
assert await core._detect_backend(None) == ""
# --- delete_sandbox ---
@@ -542,6 +552,7 @@ class TestForkSandbox:
mock_custom = AsyncMock()
mock_custom.get_namespaced_custom_object.side_effect = ApiException(status=404)
_setup_clients(core, apps=mock_apps, v1=mock_v1, net=mock_net, custom=mock_custom)
core._detect_backend = AsyncMock(return_value="kata-qemu-longhorn")
src_dep = MagicMock()
src_dep.metadata.name = "src"
@@ -569,6 +580,7 @@ class TestForkSandbox:
mock_apps = AsyncMock()
mock_apps.read_namespaced_deployment.side_effect = ApiException(status=404)
_setup_clients(core, apps=mock_apps, v1=AsyncMock())
core._detect_backend = AsyncMock(return_value="kata-qemu-longhorn")
result = await core.fork_sandbox("src", "dst")
assert result.success is False
@@ -578,6 +590,7 @@ class TestForkSandbox:
mock_v1 = AsyncMock()
mock_v1.read_namespaced_persistent_volume_claim.side_effect = ApiException(status=404)
_setup_clients(core, apps=mock_apps, v1=mock_v1)
core._detect_backend = AsyncMock(return_value="kata-qemu-longhorn")
result = await core.fork_sandbox("src", "dst")
assert result.success is False
@@ -978,8 +991,8 @@ class TestReadSandboxEgressWhitelist:
class TestK7dAgentForwarding:
def _remote_pod(self, node: str = "node-remote"):
pod = mock_pod(phase="Running")
def _remote_pod(self, node: str = "node-remote", name: str = "sb1-pod"):
pod = mock_pod(name=name, phase="Running")
pod.spec.node_name = node
return pod
@@ -1007,29 +1020,38 @@ class TestK7dAgentForwarding:
fwd.assert_called_once()
assert fwd.call_args.args[0] == "node-remote"
assert fwd.call_args.args[1] == "pause"
assert fwd.call_args.args[2]["pod_name"] == "sb1-pod"
mock_apps.patch_namespaced_deployment.assert_awaited()
async def test_fork_forwards_to_remote_agent(self, core: K7Core):
async def test_fork_looks_up_on_remote_creates_locally(self, core: K7Core):
"""k7d fork Kubernetes writes stay on k7-api; agent only looks up the VM."""
dep = mock_deployment(annotations={"k7.katakate.org/backend": "k7d"})
dep.spec.template.metadata.labels = {"app": "sb1"}
dep.metadata.labels = {"app": "sb1"}
dep.spec.selector.match_labels = {"app": "sb1"}
mock_apps = AsyncMock()
mock_apps.read_namespaced_deployment.return_value = dep
mock_v1 = AsyncMock()
mock_v1.list_namespaced_pod.return_value = MagicMock(items=[self._remote_pod()])
_setup_clients(core, apps=mock_apps, v1=mock_v1)
vm = {"vm_id": "vm-1-0", "sandbox_id": "cri-src", "cluster_id": "cri-src"}
with (
patch.dict("os.environ", {"K7_NODE_NAME": "node-local"}),
patch.object(
core,
"_k7d_forward_vm_op",
new_callable=AsyncMock,
return_value=OperationResult(success=True, message="forwarded"),
) as fwd,
patch.object(core, "lookup_k7d_vm", new=AsyncMock(return_value=vm)),
patch.object(core, "_wait_for_pod_container_started", new=AsyncMock(return_value="fork-pod")),
patch.object(core, "_read_sandbox_egress_whitelist", new=AsyncMock(return_value=[])),
patch.object(core, "_read_sandbox_ingress_rules", new=AsyncMock(return_value=([], []))),
patch.object(core, "_apply_sandbox_network_policies", new=AsyncMock(return_value=OperationResult(True))),
patch.object(core, "_k7d_forward_vm_op", new_callable=AsyncMock) as fwd,
):
result = await core.fork_sandbox("sb1", "sb1-fork")
assert result.success is True
assert fwd.call_args.args[1] == "fork"
assert fwd.call_args.args[2]["new_name"] == "sb1-fork"
assert result.success, result.error
fwd.assert_not_called()
mock_apps.create_namespaced_deployment.assert_awaited()
new_dep = mock_apps.create_namespaced_deployment.call_args.kwargs["body"]
assert new_dep.spec.template.spec.node_name == "node-remote"
async def test_agent_refuses_to_reforward(self, core: K7Core):
"""An agent that resolves a sandbox to yet another node must fail
@@ -1283,10 +1305,9 @@ class TestWaitForJob:
job.status.succeeded = 1
job.status.failed = 0
mock_batch.read_namespaced_job.return_value = job
_setup_clients(core)
_setup_clients(core, batch=mock_batch)
with patch("k7.core.core.client.BatchV1Api", return_value=mock_batch):
result = await core._wait_for_job("j1", timeout=5)
result = await core._wait_for_job("j1", timeout=5)
assert result.success is True
@@ -1296,10 +1317,9 @@ class TestWaitForJob:
job.status.succeeded = 0
job.status.failed = 1
mock_batch.read_namespaced_job.return_value = job
_setup_clients(core)
_setup_clients(core, batch=mock_batch)
with patch("k7.core.core.client.BatchV1Api", return_value=mock_batch):
result = await core._wait_for_job("j1", timeout=5)
result = await core._wait_for_job("j1", timeout=5)
assert result.success is False
assert "failed" in result.error
@@ -1310,12 +1330,9 @@ class TestWaitForJob:
job_ok.status.succeeded = 1
job_ok.status.failed = 0
mock_batch.read_namespaced_job.side_effect = [ApiException(status=404), job_ok]
_setup_clients(core)
_setup_clients(core, batch=mock_batch)
with (
patch("k7.core.core.client.BatchV1Api", return_value=mock_batch),
patch("k7.core.core.asyncio.sleep", new_callable=AsyncMock),
):
with patch("k7.core.core.asyncio.sleep", new_callable=AsyncMock):
result = await core._wait_for_job("j1", timeout=10)
assert result.success is True
@@ -1323,10 +1340,9 @@ class TestWaitForJob:
async def test_non_404_error(self, core: K7Core):
mock_batch = AsyncMock()
mock_batch.read_namespaced_job.side_effect = ApiException(status=500)
_setup_clients(core)
_setup_clients(core, batch=mock_batch)
with patch("k7.core.core.client.BatchV1Api", return_value=mock_batch):
result = await core._wait_for_job("j1", timeout=5)
result = await core._wait_for_job("j1", timeout=5)
assert result.success is False
assert "wait error" in result.error
@@ -1337,7 +1353,7 @@ class TestWaitForJob:
job.status.succeeded = 0
job.status.failed = 0
mock_batch.read_namespaced_job.return_value = job
_setup_clients(core)
_setup_clients(core, batch=mock_batch)
elapsed = 0.0
@@ -1347,7 +1363,6 @@ class TestWaitForJob:
return elapsed
with (
patch("k7.core.core.client.BatchV1Api", return_value=mock_batch),
patch("k7.core.core.time.time", side_effect=advancing_time),
patch("k7.core.core.asyncio.sleep", new_callable=AsyncMock),
):
@@ -1493,10 +1508,11 @@ class TestWaitForPodContainerStarted:
# --- _list_backends_per_node ---
def _node(name: str, labels: dict[str, str]):
def _node(name: str, labels: dict[str, str], taints: list | None = None):
n = MagicMock()
n.metadata.name = name
n.metadata.labels = labels
n.spec.taints = taints or []
return n
@@ -1534,6 +1550,35 @@ class TestListBackendsPerNode:
assert result["n"] == []
class TestListClusterNodes:
async def test_hostname_backends_and_tenant(self, core: K7Core):
taint = SimpleNamespace(key=K7_TENANT_LABEL, value="acme", effect="NoSchedule")
mock_v1 = AsyncMock()
mock_v1.list_node.return_value = SimpleNamespace(
items=[
_node(
"k7-node-01",
{
"kubernetes.io/hostname": "k7-node-01",
"k7.katakate.org/backend-k7d": "true",
K7_TENANT_LABEL: "acme",
},
taints=[taint],
),
_node("k7-node-02", {"kubernetes.io/hostname": "k7-node-02"}),
]
)
_setup_clients(core, v1=mock_v1)
rows = await core.list_cluster_nodes()
by_name = {r["name"]: r for r in rows}
assert by_name["k7-node-01"]["hostname"] == "k7-node-01"
assert by_name["k7-node-01"]["backends"] == ["k7d"]
assert by_name["k7-node-01"]["tenant"] == "acme"
assert by_name["k7-node-02"]["hostname"] == "k7-node-02"
assert by_name["k7-node-02"]["backends"] == []
assert by_name["k7-node-02"]["tenant"] == ""
# --- _check_scheduling ---
@@ -1633,18 +1678,16 @@ class TestCheckScheduling:
class TestCiliumAvailable:
async def test_returns_true_when_crd_exists(self, core: K7Core):
core._config_loaded = True
fake_api_ext = AsyncMock()
fake_api_ext.read_custom_resource_definition.return_value = MagicMock()
with patch("k7.core.core.client.ApiextensionsV1Api", return_value=fake_api_ext):
assert await core._cilium_available() is True
_setup_clients(core, apiext=fake_api_ext)
assert await core._cilium_available() is True
async def test_returns_false_when_crd_missing(self, core: K7Core):
core._config_loaded = True
fake_api_ext = AsyncMock()
fake_api_ext.read_custom_resource_definition.side_effect = ApiException(status=404)
with patch("k7.core.core.client.ApiextensionsV1Api", return_value=fake_api_ext):
assert await core._cilium_available() is False
_setup_clients(core, apiext=fake_api_ext)
assert await core._cilium_available() is False
class TestApplyCiliumEgressPolicy:
@@ -1711,3 +1754,81 @@ class TestApplyCiliumEgressPolicy:
result = await core._apply_cilium_egress_policy("sb1", "default", [], ["api.openai.com"])
assert result.success is False
assert "CiliumNetworkPolicy" in result.error
# --- kubernetes_asyncio ApiClient lifecycle ---
class TestApiClientLifecycle:
async def test_typed_clients_share_one_api_client(self, core: K7Core):
api = MagicMock()
api.close = AsyncMock()
with (
patch("k7.core.core.client.ApiClient", return_value=api) as api_cls,
patch("k7.core.core.client.CoreV1Api") as core_cls,
patch("k7.core.core.client.AppsV1Api") as apps_cls,
):
await core._get_core_v1_client()
await core._get_apps_v1_client()
api_cls.assert_called_once()
core_cls.assert_called_once_with(api)
apps_cls.assert_called_once_with(api)
await core.aclose()
api.close.assert_awaited_once()
assert core._api_client is None
assert core._core_v1_client is None
assert core._apps_v1_client is None
async def test_aclose_is_idempotent(self, core: K7Core):
await core.aclose()
await core.aclose()
async def test_aclose_allows_a_new_session(self, core: K7Core):
first = MagicMock()
first.close = AsyncMock()
second = MagicMock()
second.close = AsyncMock()
with (
patch("k7.core.core.client.ApiClient", side_effect=[first, second]),
patch("k7.core.core.client.CoreV1Api"),
):
await core._get_core_v1_client()
await core.aclose()
first.close.assert_awaited_once()
await core._get_core_v1_client()
await core.aclose()
second.close.assert_awaited_once()
class TestCreateSnapshotMessage:
async def test_named_snapshot_keeps_create_message(self, core: K7Core):
"""Regression: quiesced success used to drop the VolumeSnapshot name.
``k7 snapshot create`` echoes ``result.message``; empty stdout failed
``TestSnapshotCrud`` with ``assert snap in cp.stdout``.
"""
dep = MagicMock()
dep.metadata.annotations = {}
dep.status.ready_replicas = 1
mock_apps = AsyncMock()
mock_apps.read_namespaced_deployment.return_value = dep
_setup_clients(core, apps=mock_apps)
with (
patch.object(core, "_detect_backend", new=AsyncMock(return_value="kata-qemu-longhorn")),
patch.object(
core,
"_create_volume_snapshot",
new=AsyncMock(
return_value=OperationResult(success=True, message="Snapshot snap1 created for PVC sb-root-lh")
),
),
patch.object(
core,
"exec_command",
new=AsyncMock(return_value=ExecResult(exit_code=0, stdout="", stderr="", duration_ms=0)),
),
):
result = await core.create_snapshot("sb", "snap1")
assert result.success, result.error
assert "snap1" in result.message
tests/unit/test_core_utils.py
+12
View File
@@ -47,6 +47,18 @@ class TestMemoryLimitToMib:
core._memory_limit_to_mib(None) # type: ignore[arg-type]
class TestKataHypervisorMemoryMib:
def test_256mi_ok(self, core: K7Core):
assert core._kata_hypervisor_memory_mib("256Mi") == 256
def test_1gi_ok(self, core: K7Core):
assert core._kata_hypervisor_memory_mib("1Gi") == 1024
def test_128mi_rejected(self, core: K7Core):
with pytest.raises(ValueError, match="below Kata's minimum 256Mi"):
core._kata_hypervisor_memory_mib("128Mi")
# --- _parse_resource_value ---
tests/unit/test_kata_docker.py
+73 -4
View File
@@ -12,10 +12,14 @@ from k7.core.docker import (
CLI_STAGING_DIR,
DIND_IMAGE,
DOCKER_HOST_URL,
FSFREEZE_STAGED,
FSFREEZE_VEHICLE,
GRAPH_DEVICE_PATH,
KATA_FORK_GRAPH_REJECT,
KATA_GRAPH_MOUNT,
KATA_SOCKET_DIR,
KFD_DOCKER_STORAGE_CLASS,
KFD_FORK_REJECT,
STAGE_FSFREEZE_CMD,
VEHICLE_CONTAINER_NAME,
)
from k7.core.models import ExecResult, SandboxConfig
@@ -137,9 +141,67 @@ class TestKataDockerSnapshot:
assert wait.await_count == 2
names = [c.args[0] for c in wait.await_args_list]
assert names == ["snap1", "snap1-docker"]
assert exec_cmd.await_count == 2
cmds = [c.args[1] for c in exec_cmd.await_args_list]
freeze_cmd = (
f"cp {FSFREEZE_STAGED} {FSFREEZE_VEHICLE} && chmod 755 {FSFREEZE_VEHICLE} && "
f"{FSFREEZE_VEHICLE} -f {KATA_GRAPH_MOUNT}"
)
thaw_cmd = f"{FSFREEZE_VEHICLE} -u {KATA_GRAPH_MOUNT}"
assert cmds == ["sync", STAGE_FSFREEZE_CMD, freeze_cmd, thaw_cmd]
assert exec_cmd.await_args_list[0].kwargs.get("container") in (None, "sandbox")
assert exec_cmd.await_args_list[1].kwargs.get("container") == VEHICLE_CONTAINER_NAME
assert exec_cmd.await_args_list[1].kwargs.get("container") in (None, "sandbox")
assert exec_cmd.await_args_list[2].kwargs.get("container") == VEHICLE_CONTAINER_NAME
assert exec_cmd.await_args_list[3].kwargs.get("container") == VEHICLE_CONTAINER_NAME
async def test_create_snapshot_freeze_failure_is_loud(self, core: K7Core):
dep = MagicMock()
dep.metadata.annotations = {ANN_K7_DOCKER: "true", ANN_DOCKER_PVC: "sb-docker-lh"}
dep.status.ready_replicas = 1
mock_apps = AsyncMock()
mock_apps.read_namespaced_deployment.return_value = dep
_setup(core, apps=mock_apps, v1=AsyncMock())
create = AsyncMock(return_value=_ok())
async def exec_cmd(name, command, **kwargs):
if " -f " in command and "k7-fsfreeze" in command:
return ExecResult(exit_code=1, stdout="", stderr="EBUSY", duration_ms=0)
return _exec_ok()
with (
patch.object(core, "_detect_backend", new=AsyncMock(return_value="kata-qemu-longhorn")),
patch.object(core, "_create_volume_snapshot", new=create),
patch.object(core, "exec_command", new=AsyncMock(side_effect=exec_cmd)),
):
result = await core.create_snapshot("sb", "snap1")
assert not result.success
assert "fsfreeze -f" in result.error
create.assert_not_awaited()
async def test_create_snapshot_thaws_after_snapshot_failure(self, core: K7Core):
dep = MagicMock()
dep.metadata.annotations = {ANN_K7_DOCKER: "true", ANN_DOCKER_PVC: "sb-docker-lh"}
dep.status.ready_replicas = 1
mock_apps = AsyncMock()
mock_apps.read_namespaced_deployment.return_value = dep
_setup(core, apps=mock_apps, v1=AsyncMock())
fail = MagicMock()
fail.success = False
fail.error = "root snap fail"
exec_cmd = AsyncMock(return_value=_exec_ok())
with (
patch.object(core, "_detect_backend", new=AsyncMock(return_value="kata-qemu-longhorn")),
patch.object(core, "_create_volume_snapshot", new=AsyncMock(return_value=fail)),
patch.object(core, "exec_command", new=exec_cmd),
):
result = await core.create_snapshot("sb", "snap1")
assert not result.success
cmds = [c.args[1] for c in exec_cmd.await_args_list]
freeze_cmd = (
f"cp {FSFREEZE_STAGED} {FSFREEZE_VEHICLE} && chmod 755 {FSFREEZE_VEHICLE} && "
f"{FSFREEZE_VEHICLE} -f {KATA_GRAPH_MOUNT}"
)
thaw_cmd = f"{FSFREEZE_VEHICLE} -u {KATA_GRAPH_MOUNT}"
assert cmds == ["sync", STAGE_FSFREEZE_CMD, freeze_cmd, thaw_cmd]
async def test_create_snapshot_skips_sync_when_paused(self, core: K7Core):
dep = MagicMock()
@@ -274,7 +336,14 @@ class TestKataDockerForkReject:
result = await core.fork_sandbox("src", "dst")
assert not result.success
assert "cannot be cloned" in result.error
assert result.error == KATA_FORK_GRAPH_REJECT
assert result.error == KFD_FORK_REJECT
async def test_kfd_plain_fork_rejected(self, core: K7Core):
"""Non-docker kfd has no PVC either — reject before a confusing 404."""
with patch.object(core, "_detect_backend", new=AsyncMock(return_value="kata-firecracker-devmapper")):
result = await core.fork_sandbox("src", "dst")
assert not result.success
assert result.error == KFD_FORK_REJECT
class TestDindImagePin:
tests/unit/test_models.py
+2
View File
@@ -32,6 +32,7 @@ class TestSandboxConfigFromDict:
"container_non_root": True,
"cap_drop": ["ALL"],
"cap_add": ["NET_ADMIN"],
"node_name": "k7-node-01",
}
cfg = SandboxConfig.from_dict(data)
assert cfg.backend == "kata-qemu-longhorn"
@@ -39,6 +40,7 @@ class TestSandboxConfigFromDict:
assert cfg.limits == {"cpu": "2", "memory": "4Gi"}
assert cfg.pod_non_root is True
assert cfg.cap_add == ["NET_ADMIN"]
assert cfg.node_name == "k7-node-01"
def test_ignores_unknown_keys(self):
cfg = SandboxConfig.from_dict({"name": "x", "image": "img", "unknown_field": 42, "another": "val"})
tests/unit/test_playbook_backends.py
+21
View File
@@ -0,0 +1,21 @@
"""Playbook must fail closed on omitted k7_backends; none is the empty set."""
from pathlib import Path
PLAYBOOK = Path(__file__).resolve().parents[2] / "src/k7/deploy/k7-install-node.yaml"
EXAMPLE = Path(__file__).resolve().parents[2] / "src/k7/deploy/inventory.ini.example"
def test_playbook_has_no_backend_default():
text = PLAYBOOK.read_text()
assert "k7_backend_default" not in text
assert "empty/omitted is not" in text
assert "k7_backends=none" in text or "or none" in text
assert "[] if (k7_backends_raw | map('lower') | list == ['none'])" in text
assert "content: \"{{ k7_backends_list[0] if k7_backends_list | length > 0 else 'none' }}\"" in text
def test_inventory_example_documents_none_on_servers():
text = EXAMPLE.read_text()
assert "k7_backends=none" in text
assert "[k7_servers:vars]" in text
tests/unit/test_restore.py
+1 -1
View File
@@ -33,7 +33,7 @@ from k7.core.models import (
)
from k7_sdk.client import Client, SandboxProxy
TEST_KEY = "k7-test-secret-key-spec10f"
TEST_KEY = "k7-test-secret-key-restore"
TEST_KEY_HASH = hashlib.sha256(TEST_KEY.encode()).hexdigest()
tutorials/k7_hetzner_node_setup.md
+5 -5
View File
@@ -78,11 +78,11 @@ lsblk -f # spare should show no FSTYPE / no RAID members
sudo add-apt-repository ppa:katakate.org/k7
sudo apt update
sudo apt install k7
k7 -V # 0.4.0
# dual-NVMe box: let the playbook auto-detect the raw spare disk.
# Launchpad PPA is still 0.2.2; install the GitHub 0.3.1 .deb (or this tree's CLI).
# Run from a checkout of this repo (or Katakate/k7) so k7-api:local can build.
sudo k7 install --backend kfd,kql,k7d
sudo k7 install --backend kfd,kql,k7d,k7d-fc
```
`k7 install` provisions the LVM thin-pool on that disk for the `kfd` backend.
@@ -90,8 +90,8 @@ Other backends (`kql`, `k7d`) do not need this spare disk, but keeping one raw
NVMe lets you compare all backends on the same node.
Two-node (one server + one agent, no `--ha`) is the same idea: write an
inventory with `k7_backends=kfd,kql,k7d` on both hosts, omit
`k7_devmapper_disk`, and run **one** `k7 install -i inventory.ini --k7d-version 0.6.0`
inventory with `k7_backends=kfd,kql,k7d,k7d-fc` on both hosts, omit
`k7_devmapper_disk`, and run **one** `k7 install -i inventory.ini --k7d-version 0.7.0`
from the first master. See `src/k7/deploy/inventory.ini.example`.
> **Do NOT pin the disk on dual-NVMe boxes.** NVMe enumeration
@@ -113,5 +113,5 @@ from the first master. See `src/k7/deploy/inventory.ini.example`.
- [ ] `ls /dev/kvm` exists
- [ ] OS root is on a single disk (`findmnt /` → `/dev/nvme0n1p…`, not `/dev/md…`)
- [ ] Spare disk has no filesystem (`lsblk -f` empty FSTYPE)
- [ ] `k7 install --backend kfd,kql,k7d --k7d-version 0.6.0` succeeds (auto-detects the spare disk — do not pin `--disk` on dual-NVMe)
- [ ] `k7 install --backend kfd,kql,k7d,k7d-fc --k7d-version 0.7.0` succeeds (auto-detects the spare disk — do not pin `--disk` on dual-NVMe)
- [ ] `k7 create --backend kfd …` can start a sandbox
uv.lock
Generated
+1 -1
View File
@@ -1069,7 +1069,7 @@ wheels = [
[[package]]
name = "k7"
version = "0.3.1"
version = "0.4.0"
source = { editable = "." }
dependencies = [
{ name = "fastapi" },