diff --git a/CHALLENGES.md b/CHALLENGES.md index 9b221dd..c9b7962 100644 --- a/CHALLENGES.md +++ b/CHALLENGES.md @@ -524,7 +524,25 @@ timed out after 5s`. Create→Ready and exec *before* pause had worked docker-perf failure. Lifecycle was aborted; do not quote k7d-fc resume→exec from that run. -**Reference:** CHALLENGES #15; k7d #232. +**Fixed (lifecycle resume), spec 42a:** not #15. Two bugs, one per +layer. (1) Firecracker v1.16.0/v1.16.1's vsock device armed its +`TRANSPORT_RESET` RX gate on every `PATCH /vm Resumed` with no reset +event for the guest to ack, so after a bare pause → resume no +host→guest packet — no `CONNECT` reply — was ever delivered; upstream +fixed it as #6100 in v1.16.2, and k7d's `guest/fc/pins.env` plus the +vendored `src/k7/deploy/k7d-fc/pins.env` now pin v1.16.2. k7d's +`resume_vm` also dials the resumed FC guest and fails loud if it does +not answer. (2) The shim's exec bridge never opened containerd's +stdout/stderr FIFOs on its failure path, so a probe exec that could +not reach the paused guest left kubelet's prober parked in `ExecSync` +for its 2-minute gRPC deadline — the pod stayed Ready through a whole +pause on `k7d` *and* `k7d-fc`. `tests/integration/test_pause_resume.py::TestPauseResumeExecAnswers` +covers pause → resume → `exec_command("echo hi")` on both; the +`PERFORMANCE.md` k7d vs k7d-fc lifecycle table has the resume→exec +number. Root cause and timings: k7d CHALLENGES #240 (vsock gate) and +#241 (FIFOs). + +**Reference:** CHALLENGES #15; k7d #232, #240, #241; Firecracker #6100. **Time lost:** ~15 min on the OutOfcpu wait; lifecycle resume hung until the pytest process was killed (~8 min). @@ -589,3 +607,141 @@ hostPath); `K7D_DOCKER_PAYLOAD_DOCKERD` in `src/k7/core/docker.py`. **Time lost:** ~30 min diagnosing why the live cluster had dockerd but the API refused `--docker`. + +--- + +## 20. Laptop `k7 api status` crashed; docs `k7.yaml` 128Mi never went Ready + +**Symptom:** After `apt install k7` (PPA 0.3.1) on a laptop with +`k7 config set api.url` / `api.ca` / `api.key`, `k7 list` and +`k7 nodes storage` worked, but the Quickstart's next commands +`k7 api status` and `k7 api endpoint` raised `FileNotFoundError: kubectl`. +The same Quickstart's `examples/k7.yaml` (`cpu: 100m`, `memory: 128Mi`, +`before_script: apk add curl`, default backend kfd) timed out with +"Timed out waiting for sandbox container to start". + +**Root cause:** `k7 api status` / `endpoint` always shelled out to +`kubectl` (or `k3s kubectl`) without checking the binary exists. A laptop +that only has the .deb has no kubeconfig and no kubectl. Separately, the +docs `k7.yaml` set `memory: 128Mi`, which Kata stamps as +`io.katacontainers.config.hypervisor.default_memory: 128`. The Firecracker +shim refuses anything below **256Mi**, so the pod stays `ContainerCreating` +(`FailedCreatePodSandBox`) until create times out. `before_script` never +runs. + +**Fix:** missing kubectl falls back to `GET /health` on the configured API +URL (status) / prints that URL (endpoint). Create rejects Kata memory +below 256Mi immediately. Docs + `examples/k7.yaml` use `backend: k7d` +and `cpu: "1"` / `memory: "1Gi"`. PPA is 0.3.1, not 0.2.2. + +**Reference:** `src/k7/cli/k7.py` (`_kubectl_run`, `_api_status_via_https`); +`examples/k7.yaml`. + +**Time lost:** ~20 min reproducing on a 3-node HA soak after a public 0.3.1 +release; the install itself was one command and succeeded. + +--- + +## 21. `--expose-port` NodePort is dead until Ready; `k7 exec -- sh -c` self-nests + +**Symptom:** Docs `k7 create --expose-port 8000 --before-script 'nohup python3 -m http.server 8000 &'` printed a NodePort, but curling it from a laptop timed out. Cilium showed the Service in **maintenance**. `k7 exec NAME -- sh -c 'echo hi > /tmp/x'` failed with Nuitka/PyInstaller-style "tried to call itself with '-c'". `k7 restore --latest` and `k7 delete-all -y` do not exist. + +**Root cause:** `externalTrafficPolicy: Local` plus Cilium keeps a NodePort in maintenance while endpoints are `notReadyAddresses`. k7d/kfd pid 1 is `sleep 365d`; a bare `&` in `before_script` is killed when the script exits, so the Ready probe never sees the http server (and a missing `touch` of the done file has the same effect). `k7 exec` already wraps the joined argv in `sh -c`, so a nested `sh -c` is the CLI binary eating `-c`. Restore takes two positionals; `delete-all` confirms interactively with no `-y`. + +**Fix:** Docs: trailing `sleep 1` after nohup so before_script can finish and Ready can fire; curl the **pod's** node, not an arbitrary master. Exec: one quoted string, no extra `sh -c`. Restore/delete-all examples match the CLI. Create success text points at `k7 list`, not `k7 list --name`. + +**Reference:** `src/k7/cli/k7.py` exec/create; docs `k7/guides/cli.mdx`. + +**Time lost:** ~40 min on the public 0.3.1 HA soak (expose looked like a CNI bug until Ready flipped). + +--- + +## 22. k7d docker graph image is a teardown race, not a leak (spec 39a) + +**Symptom:** `TestDockerK7d` / `TestDockerK7dFc` failed after `k7 delete` + 3s with `leaked k7d docker volume images: {scratch-vm-…-docker.img}`. Hours later, same k7d PID, the files were gone. + +**Root cause:** `k7 delete` returns when Kubernetes objects are gone. The VM's `ScratchDisk::drop` unlinks the graph image asynchronously when the containerd shim `Delete` lands. On a busy 3-node HA box that is more than 3s. + +**Fix:** bounded poll (~60s) in the test. `delete_sandbox` does **not** wait for the k7d VM — coupling API latency to shim teardown would stall every delete. + +**Reference:** `tests/integration/test_docker.py` `_wait_k7d_docker_disks_gone`; `K7Core.delete_sandbox`. + +**Time lost:** the soak already established this; the 3s sleep was the only defect. + +--- + +## 23. Firecracker jailer test missed `firecracker-v1.` (spec 39a) + +**Symptom:** `test_jailer_active` asserted "No firecracker processes found on the host" while a jailed kfd VM was running. + +**Root cause:** Linux truncates `comm` to 15 characters. The pinned binary is `firecracker-v1.16.1`, so `comm` is `firecracker-v1.`. The helper compared `== "firecracker"`. The chroot binary is also versioned (`firecracker-v1.16.1`, not `/firecracker`). The jail itself is correct: `/etc/passwd`, `/etc/shadow`, `/usr`, `/boot` absent; `vmlinux` + `rootfs` present. `readlink /proc//root` is `/` because the jailer pivot-roots in a private mount ns. + +**Fix:** match `comm` by prefix `firecracker-`, require `fcConfig.json` / `--config-file` on the cmdline, skip `(deleted)` orphans, require `vmlinux`+`rootfs`, glob `firecracker*` in the chroot. Host-FS-unreachable asserts stay. + +**Reference:** `tests/integration/test_firecracker.py` `_get_live_firecracker_pids`. + +**Time lost:** ~20 min of `/proc` on the soak node. + +--- + +## 24. Expose tests timed out because the pod was on another node (spec 39a) + +**Symptom:** `TestSandboxExpose` curled `http://:` from the first master and timed out. Off-cluster, the pod's node answered HTTP 200 and the other nodes refused connect. + +**Root cause:** `externalTrafficPolicy: Local` is required (without it, `cidr:` rules see SNAT). Cilium socket-LB intercepts in-cluster-node origin to a NodePort on a different node. The tests did not pin `node_name`. + +**Fix:** pin expose sandboxes to `os.uname().nodename`. Do not switch the Service to `Cluster`. + +**Reference:** `tests/integration/test_sandbox_ingress.py` `TestSandboxExpose._exposed`. + +**Time lost:** ~15 min confirming Local vs Cilium vs a wrong-node curl. + +--- + +## 25. kql live `--docker` fork never went Ready: overlay2 was crash-inconsistent (spec 39a) + +**Symptom:** `TestDockerKQL.test_fork_clones_both_pvcs` — both VolumeSnapshots ready, both child PVCs Bound, child never Ready in 240s. + +**Root cause (live, spec 39a):** the child stuck in `Init:0/2` with `FailedAttachVolume: volume is not ready for workloads`. Kubernetes Bound is not Longhorn-ready-to-attach; HA r=3 clone hydration takes longer than 240s. The qemu fork test already waits 600s for this. `sync` is also not enough for a busy overlay2 once the guest *does* start. + +**Fix:** wait 600s for the child (same bound as `test_qemu.test_fork_clones_data`). Plus `fsfreeze` the graph: alpine `docker:27.5.1-dind` has no `fsfreeze`, so stage it from the ubuntu sandbox via the shared `/tmp` emptyDir, copy into the vehicle rootfs, freeze around VolumeSnapshot create, thaw in `finally`. + +**Reference:** `K7Core._create_kata_snapshots_quiesced`. + +**Time lost:** soak diagnosis; freeze is the product answer rather than refusing live forks. + +--- + +## 26. Partner-facing 0.3.1 traps: kfd fork 404, SDK snippet missing CA, `k7 logs` empty + +**Symptom:** Following docs.katakate.org / the k7 README on a 3-node HA PPA 0.3.1 cluster: + +- `k7 fork` of a kfd sandbox printed `Source root PVC -root-lh not found; cannot fork storage` instead of "kfd cannot fork". +- `k7 api status` printed `Client(endpoint=..., api_key=...)` with no `verify_ssl`; that fails against the playbook-minted cluster CA. +- `k7 logs demo --tail 20` printed nothing (exit 0) after only `k7 exec`. +- `python3 -m venv` on the node failed (`ensurepip is not available`). +- `k7 resume` of a kql sandbox returned immediately while the pod was still `Pending`. + +**Root cause:** `fork_sandbox` treated missing kfd PVCs as a generic storage 404. The status command's help snippet never grew `verify_ssl` when HTTPS-by-default landed. `k7 logs` is a CRI container snapshot; exec goes through the agent. Ubuntu 24.04 cloud images omit `python3-venv`. HA Longhorn attach is slower than `resume()` returning. + +**Fix:** reject every kfd fork up front (`KFD_FORK_REJECT`). Print `verify_ssl='./k7-ca.crt'` in `k7 api status`. Docs: wait-until-Ready after kql resume, empty logs are success, SDK is a client install (`apt install python3-venv` on the node). + +**Reference:** none. + +**Time lost:** ~30 min walking the quickstart on the HA cluster. + +--- + +## 27. `k7 --core` leaked kubernetes_asyncio/aiohttp sessions (`Event loop is closed`) + +**Symptom:** `TestSnapshotCrud.test_create_list_inspect_delete_round_trip` failed with `assert snap in cp.stdout` and `cp.stdout == '\n'`. Interpreter also logged `Unclosed client session` / `Event loop is closed` after `k7 --core snapshot create`. + +**Root cause:** Each `CoreV1Api()` / `AppsV1Api()` / `CustomObjectsApi()` constructed its own `ApiClient` (aiohttp session) bound to the `asyncio.run` loop. Typer handlers never called `close()`, so loop teardown raced the session destructor. Separately, `_create_kata_snapshots_quiesced` dropped the successful `OperationResult.message` from `_create_volume_snapshot`, so the CLI echoed a blank line even when the snapshot existed. + +**Fix:** One shared `ApiClient` per `K7Core`, `async def aclose()`, CLI `_core_run` / API `get_k7_core` / snapshot-gc always close. Quiesced snapshot success now keeps the create message (`Snapshot created for PVC …`). + +**Reference:** kubernetes_asyncio `ApiClient.close` → `rest_client.close()` (aiohttp). + +**Time lost:** caught on the HA integration run after the partner walkthrough. + + diff --git a/CHANGELOG.md b/CHANGELOG.md index b0aceff..6dcbb67 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,67 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [0.4.0] — 2026-09-19 + +Per-key node pins, k7d-fc pause/resume/exec, and HA-soak fixes. Playbook +pins k7d **0.7.0**. GitHub `.deb`, Launchpad PPA, and PyPI `k7-sdk` are +**0.4.0**. + +### Added + +- **Optional per-key node placement** for tenant isolation on the + shared per-node k7d daemon. `k7 generate-api-key --node ` + (repeatable) stores `"nodes"` on the key, analogous to `-n` for + namespaces. Placement uses the Node's `kubernetes.io/hostname` + label, not `spec.nodeName`. `k7 nodes dedicate NODE --tenant ID` + writes `k7.katakate.org/tenant` as a label and a NoSchedule taint so + other sandboxes cannot share that daemon. `k7 nodes list` shows the + K3s Node names (Linux hostname / `kubernetes.io/hostname`) to pass + to `--node`; inventory.ini does not write that label — `k7_backends` + only stamps backend labels at install. A single-node key auto-pins + creates/restores; a multi-node key requires an explicit `node_name` + / `k7 create --node`. Forks must already sit on an allowed node. + `GET /api/v1/nodes/storage` is 403 for a node-scoped key. Unscoped + keys keep unrestricted placement. Root on `[k7_servers]` is still + cluster-admin; pins do not contain a host compromise — tenant + sandboxes belong on `[k7_agents]`. `k7-agent` has no kube SA token + and no `remote-node` ingress, so a dedicated worker breakout is that + tenant, not cluster-admin. +- **k7d-fc resume → exec**: pin Firecracker v1.16.2; pause/resume/exec + integration row and filled `PERFORMANCE.md` lifecycle table. + +### Fixed + +- **`k7 api status` / `k7 api endpoint` from a laptop** no longer crash with + `FileNotFoundError: kubectl`. With `k7 config set api.url` they print the + configured URL and probe `GET /health`. `k7 api enable`/`disable` still + need kubectl on a cluster node and now say so instead of tracebacking. +- **Kata memory below 256Mi** is rejected at create time instead of waiting + 300s for `FailedCreatePodSandBox` (docs `k7.yaml` used `128Mi`). +- **Create success text** after `--expose-port` points at `k7 list`, not a + nonexistent `k7 list --name`. +- **Kata kql `--docker` live fork / snapshot** now `fsfreeze`s + `/var/lib/docker` in the docker-vehicle around VolumeSnapshot create + (thaw in `finally`). alpine dind has no `fsfreeze`; it is staged from + the sandbox image via the shared `/tmp`. `sync` alone left overlay2 + crash-inconsistent on Longhorn r=3. +- **`k7 delete` does not wait for k7d VM teardown.** The docker-graph + leftover assertion polls up to 60s instead of a 3s sleep. +- **Loud kfd fork reject**, cluster CA snippet in the SDK/docs path, and + `K7Core` aiohttp session close on teardown. + +### Changed + +- **`k7 install --backend` is required.** There is no default runtime + list (kfd needs a spare disk; installing all backends grows attack + surface). Inventory hosts must set `k7_backends` (group vars are + fine). `none` is the only empty set — scheduling-only masters use + `[k7_servers:vars] k7_backends=none`. Empty/omitted is an error, not + kfd. +- Playbook default `k7d_version` is **0.7.0** (was 0.6.0). README, + Hetzner tutorial, and inventory examples target k7 **0.4.0** on + GitHub, PPA, and PyPI. + ## [0.3.1] — 2026-09-14 HA install and API-path `--docker` fixes for the 0.3.0 line. README diff --git a/PERFORMANCE.md b/PERFORMANCE.md index 08c859f..cf0b1b2 100644 --- a/PERFORMANCE.md +++ b/PERFORMANCE.md @@ -324,9 +324,37 @@ fork is the k8s adopt path (not the ~5 ms VMM CoW floor); alpine create→Ready / fork→exec on the same node this run were **2.15 s / 2.54 s** (k7d) and **2.14 s / 2.45 s** (k7d-fc), n=1. -k7-fc CRI exec after pause/resume hung (readiness probe timeout, -`guest_cid=0` retained) — see CHALLENGES #17. Do not quote a -resume→exec number for k7d-fc from this cut. +### k7d vs k7d-fc lifecycle — 2026-09-14 + +Same 3-node HA cluster (k7-node-01 as the bench host: Hetzner AX41, +Ryzen 5 3600, 64 GiB, NVMe, Ubuntu 24.04, kernel 6.8.0-138, k3s +v1.36.4), k7d 0.6.0 built from the Firecracker jail-integrity branch +(not the public GitHub tarball of the same version string), Firecracker v1.16.2, +`alpine:3.20`, 3 reps interleaved, median. `resume → exec` is +`resume_sandbox()` until an `echo` exec answers; `pause` is the API +call until the VM is frozen. + +```bash +K7_BENCH_BACKENDS=k7d,k7d-fc K7_BENCH_REPS=3 \ + uv run pytest -m bench tests/integration/bench_backend_lifecycle.py -v -s +``` + +| Operation | k7d | k7d-fc | +|-----------|-----|--------| +| create → Ready | 2.39 s | 2.38 s | +| exec round-trip (median of 10) | 0.37 s | 0.37 s | +| fork call | 2.73 s | 2.66 s | +| fork → Ready + exec answers | 3.23 s | 3.15 s | +| pause effective | 0.35 s | 0.35 s | +| **resume → exec answers** | **0.67 s** (0.62–0.76) | **0.70 s** (0.69–0.72) | +| delete | 0.70 s | 0.66 s | + +The earlier caveat on this section — k7-fc CRI exec after pause/resume +hung, do not quote a resume→exec number — is gone: the hang was +Firecracker v1.16.0/v1.16.1 gating vsock RX after a bare resume +(upstream #6100, fixed in v1.16.2), plus a shim exec bridge that left +kubelet's probe parked (k7d CHALLENGES #240 / #241; CHALLENGES #17 +here). ## k7d `--docker` (first-class guest service) diff --git a/README.md b/README.md index 06204cb..ea172a2 100644 --- a/README.md +++ b/README.md @@ -76,7 +76,7 @@ The Tech Stack Sandbox backends -`k7 install --backend ` provisions one or more backends per node; `k7 create --backend …` picks one per sandbox. See [docs/BACKENDS.md](docs/BACKENDS.md) for the architecture and [PERFORMANCE.md](PERFORMANCE.md) for the full measurements (Hetzner AX41 node, medians). +`k7 install --backend ` provisions those backends (required; there is no default). `none` installs the cluster with no sandbox runtime — typical for scheduling-only masters (`k7_backends=none` in inventory). Each sandbox picks one runtime (`k7 create --backend …`). See [docs/BACKENDS.md](docs/BACKENDS.md) for the architecture and [PERFORMANCE.md](PERFORMANCE.md) for the full measurements (Hetzner AX41 node, medians). | | `kfd` (kata-firecracker-devmapper) | `kql` (kata-qemu-longhorn) | `k7d` | `k7d-fc` | |---|---|---|---|---| @@ -107,7 +107,7 @@ Also available today - 🌐 Multi-node clusters (Ansible + Longhorn) - 🔍 Cilium CNI with FQDN egress policies (optional Hubble flow observability via k7 install --hubble; off by default, observability only) - 📸 Pause / resume / fork / restore and k7 snapshot lifecycle -- 🐍 Python SDK: pip install k7-sdk (katakate package deprecated) +- 🐍 Python SDK: pip install k7-sdk==0.4.0 (katakate package deprecated) 📋 **See [ROADMAP.md](ROADMAP.md) for upcoming work (GPU passthrough, …).** @@ -127,7 +127,7 @@ We provide a: - **CLI**: to use on the node(s) directly --> `apt install k7` - **API**: deployed automatically by `k7 install` (toggle with `k7 api enable` / `k7 api disable`) -- **Python SDK**: HTTP client sync/async --> `pip install k7-sdk` +- **Python SDK**: HTTP client sync/async --> `pip install k7-sdk==0.4.0` ## Current requirements @@ -147,7 +147,7 @@ We provide a: - Azure: Dv3, Ev3, Dv4, Ev4, Dv5, Ev5 (Intel/AMD x86) or Dpdsv5, Dpldsv5, Epsv5 (ARM64). - DigitalOcean: Premium Intel and AMD droplets with nested virtualization enabled. - Others: in general, hardware virtualization is not exposed on cloud VPS, so you'll likely want a dedicated / bare metal. -- One raw disk (unformatted, unpartitioned) for the thin-pool that k7 will provision for efficient disk usage of sandboxes. +- **kfd only:** one raw disk (unformatted, unpartitioned) for the thin-pool. Other backends (`kql`, `k7d`) install without a spare drive. `k7 install` does not imply kfd. - Use `./utils/wipe-disk.sh /your/disk` to wipe a disk clean before provisioning. DANGER: destructive - it will remove data/partitions/formatting/SWRAID. - Ansible (for installer): ```bash @@ -174,7 +174,7 @@ The **`.deb` / PPA package is Linux-only** (amd64/arm64). On a MacBook: - **CLI from source:** `./src/k7/cli/dev.sh` (same commands as `k7`; uses `uv` + `PYTHONPATH=src`) - **API client from laptop:** set `K7_API_URL` and `K7_API_KEY`, then `dev.sh create` / `dev.sh list` (no `--core`) - **`k7 install`** targets Linux servers with KVM — run on the node or via SSH, not on macOS locally -- **`pip install k7-sdk`** for Python scripts only +- **`pip install k7-sdk==0.4.0`** for Python scripts only Do not install the Ubuntu `.deb` on macOS. @@ -183,21 +183,26 @@ Do not install the Ubuntu `.deb` on macOS. ### Get your node(s) ready -The Launchpad PPA currently publishes **0.2.2**. For **0.3.1** (HTTPS API, -`--docker`, k7d 0.6.0, HA `k7d-fc` copy) install the GitHub release `.deb`, -then clone the matching source — `k7 install` builds `k7-api:local` from -the current working directory: +The Launchpad PPA publishes **0.4.0**. Install the CLI, then clone the matching +source — `k7 install` builds `k7-api:local` from the current working directory: ```shell -curl -fsSL -O https://github.com/Katakate/k7/releases/download/v0.3.1/k7_0.3.1_amd64.deb -sudo apt install ./k7_0.3.1_amd64.deb -git clone --branch v0.3.1 https://github.com/Katakate/k7.git +sudo add-apt-repository ppa:katakate.org/k7 +sudo apt update +sudo apt install k7 +k7 -V # 0.4.0 +# GitHub .deb is the same package if you prefer not to add the PPA: +# curl -fsSL -O https://github.com/Katakate/k7/releases/download/v0.4.0/k7_0.4.0_amd64.deb +# sudo apt install ./k7_0.4.0_amd64.deb +git clone --branch v0.4.0 https://github.com/Katakate/k7.git cd k7 sudo apt install -y ansible curl -fsSL https://get.docker.com | sh ``` -Then let `k7` get your node ready: +Then let `k7` get your node ready. **`--backend` is required** (no silent +default). kfd needs a spare raw disk; kql uses Longhorn on the OS disk; k7d +is the warm-fork daemon. `none` is control-plane only (no sandbox runtime): ```console $ k7 install --backend kfd,kql,k7d @@ -215,11 +220,13 @@ Optionally pass `-v` for a verbose output. > [tutorials/k7_hetzner_node_setup.md](tutorials/k7_hetzner_node_setup.md) > (this file is also in public [Katakate/k7](https://github.com/Katakate/k7)). > -> The playbook pins k7d **0.6.0**. `--docker` needs the guest docker +> The playbook pins k7d **0.7.0**. `--docker` needs the guest docker > service (payload on the node); an older k7d fails loudly with > `this k7d has no docker service; upgrade`. Multi-node inventory shapes > (2-node server+agent, 3-node `--ha`) are in -> `src/k7/deploy/inventory.ini.example`. +> `src/k7/deploy/inventory.ini.example`. Every inventory host must set +> `k7_backends` (or inherit group vars). Use `k7_backends=none` on +> `[k7_servers]` when agents run the sandboxes — empty/omitted is an error. `k7 install` serves `k7-api` on NodePort `31007` over HTTPS. The default is a playbook-minted cluster CA (Let's Encrypt cannot issue for a bare @@ -355,8 +362,11 @@ separate "start" step. k7 api status k7 api endpoint -# Generate API key +# Generate API key (optionally pin to a namespace and/or node) k7 generate-api-key my-key1 +k7 nodes list # NAME is the Linux hostname K3s registered; not an inventory label +k7 generate-api-key tenant-a -n tenant-a --node k7-node-01 +k7 nodes dedicate k7-node-01 --tenant acme # Temporarily disable / re-enable k7 api disable @@ -373,12 +383,12 @@ After your k7 API is up, usage is very simple. Install the Python SDK via: ```shell -pip install k7-sdk +pip install k7-sdk==0.4.0 ``` Or if you want async support: ```shell -pip install "k7-sdk[async]" +pip install "k7-sdk[async]==0.4.0" ``` The legacy `katakate` PyPI name remains as a one-release shim that re-exports `k7_sdk` with a deprecation warning. diff --git a/ROADMAP.md b/ROADMAP.md index c74f2d8..899a8b8 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -7,8 +7,7 @@ Where **K7** is headed — for contributors and operators. ## Current focus Release engineering: keep the apt/PPA, GHCR `k7-api`, and PyPI `k7-sdk` -pipelines current with `main` (the PPA is still 0.2.2; HTTPS, `--docker`, -and `k7d-fc` are unreleased). +pipelines current with `main` (PPA, GitHub `.deb`, and PyPI `k7-sdk` are **0.4.0**). --- @@ -24,7 +23,7 @@ and `k7d-fc` are unreleased). - [x] Firecracker jailer - [x] `k7d-fc` backend (k7d driving stock Firecracker + jailer) - [x] Python SDK as **`k7-sdk`** / `k7_sdk` (`katakate` deprecated) -- [x] `k7d` backend install path (public `Katakate/k7d` GitHub Releases; playbook pin 0.6.0) +- [x] `k7d` backend install path (public `Katakate/k7d` GitHub Releases; playbook pin 0.7.0) - [x] HTTPS-by-default for `k7-api` (Caddy sidecar, cluster CA) - [x] Network security hardening: cluster-wide sandbox→platform isolation, opt-in sandbox ingress / `--expose-port`, Hubble, and a @@ -35,7 +34,7 @@ and `k7d-fc` are unreleased). ## Next goals -- [ ] PPA / GHCR / PyPI cut of the unreleased work (HTTPS, `--docker`, `k7d-fc`) +- [x] PPA / GHCR / PyPI cut of the unreleased work (HTTPS, `--docker`, `k7d-fc`) - [ ] Optional macOS CLI artifacts (tarball / Homebrew) — after the above --- diff --git a/SECURITY.md b/SECURITY.md index 26645d2..541b10d 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -51,11 +51,26 @@ Do **not** open a public issue for security-sensitive reports. - The control plane API uses API keys with hashed storage and expiry (file-backed by default at `/etc/k7/api_keys.json` — rotate and protect that file). Keys may optionally be scoped to one or more namespaces - (`k7 generate-api-key -n `); absent/empty scope keeps the historical - unrestricted behaviour (backward compatible). Scoped keys are enforced - on every namespace-bearing and cluster-scoped control-plane route — - they cannot list across all namespaces, touch namespaces outside their - list, or read cluster-wide topology (e.g. ``GET /api/v1/nodes/storage``). + (`k7 generate-api-key -n `); they may also be scoped to one or more + Kubernetes nodes (`k7 generate-api-key --node `). Absent/empty + scope keeps the historical unrestricted behaviour (backward compatible). + Scoped keys are enforced on every namespace-bearing and cluster-scoped + control-plane route — they cannot list across all namespaces, touch + namespaces outside their list, or read cluster-wide topology (e.g. + ``GET /api/v1/nodes/storage``). A **node-scoped** key may only + create/restore/fork sandboxes onto its listed nodes (a single-node + scope auto-pins; several nodes require an explicit ``node_name``). + Namespace scope is the control-plane tenancy boundary; node scope is + the placement boundary (API-key `--node` is a Kubernetes node name + from `k7 nodes list` / `kubectl get nodes` — the Linux hostname K3s + registered — joined via the kubelet-stamped `kubernetes.io/hostname` + label; inventory.ini does not write that label). Together they + keep a tenant off another tenant's k7d daemon **only if that node is + also dedicated** (`k7 nodes dedicate NODE --tenant ID` writes + `k7.katakate.org/tenant` as a label and a NoSchedule taint). Without + the taint, an unscoped key can still land on the same node. Unscoped + keys still have full cross-namespace control-plane access and + unrestricted placement. - **The control-plane API is on a public NodePort (`31007`) on a public node, over HTTPS by default**. A Caddy sidecar terminates TLS; the API container and its probes stay HTTP on `:8000`. The default cert is a @@ -117,6 +132,21 @@ Do **not** open a public issue for security-sensitive reports. the node and the API server. - **Multi-node** clusters are supported (Ansible inventory; Longhorn for the QEMU/`kql` path). Cilium FQDN egress applies cluster-wide. +- **Root on a K3s server is full cluster control.** Nodes in + `[k7_servers]` (including the 3-node `--ha` example, where every + host is a server) have `/etc/rancher/k3s/k3s.yaml` as **cluster-admin** + (playbook `--write-kubeconfig-mode 644`), the join token, and etcd. + The first master also has `k7-api` and `/etc/k7/api_keys.json`. Node + pins and `k7 nodes dedicate` do **not** contain that: they stop two + tenants sharing a k7d daemon, they do not survive a VM escape onto + a master. Put tenant sandboxes on `[k7_agents]`; do not dedicate a + tenant onto a server running `k7-api`. Root on an **agent** is not + the admin kubeconfig. `k7-agent` has no ServiceAccount token and + ingress from `remote-node` is denied, so that breakout owns **that + node's** k7d (that tenant, if dedicated) — not other agents and not + the Kubernetes API. Per-node isolation is the blast radius for a + worker compromise; it is not useless. The shared agent token remains + on every node as defence-in-depth after the CNP. See also the docs: security model, networking, and backends comparison. @@ -131,11 +161,24 @@ See also the docs: security model, networking, and backends comparison. - API key storage is local file-backed; treat the API host as trusted. Namespace scoping is an opt-in tenancy boundary on top of that model — unscoped keys still have full cross-namespace control-plane access. + Node scoping is the matching opt-in **placement** boundary: without + `--node` on the key, two namespaces can still share a node's k7d + daemon. Pair `-n` and `--node`, and run `k7 nodes dedicate` on that + node, for tenant isolation on k7d. The dedicate step is a label + + taint (`k7.katakate.org/tenant`); a hostname pin alone does not keep + other sandboxes off the node. Neither contains a host compromise: + root on `[k7_servers]` is cluster-admin. - Young project; no independent security audit yet. - The `k7d` backend has a different isolation trade-off for CoW sibling forks — see k7d's `SECURITY.md`. - Prefer a dedicated RBAC-restricted kubeconfig for the API rather than cluster-admin credentials in production. +- **Host compromise is cluster-wide on servers.** `--node` / `dedicate` + isolate k7d placement, not a root shell on `[k7_servers]`. The HA + inventory example makes every node a server. Tenant boxes belong in + `[k7_agents]`. `k7-agent` has no ServiceAccount token and no + `remote-node` ingress, so root on an agent is that node (that tenant + if dedicated), not k7-api RBAC on the rest of the cluster. ## Responsible Disclosure diff --git a/debian/changelog b/debian/changelog index 6a45e9a..e182483 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,9 @@ +k7 (0.4.0) noble; urgency=medium + + * Release 0.4.0. + + -- Katakate Sat, 19 Sep 2026 22:58:56 +0200 + k7 (0.3.1) noble; urgency=medium * Release 0.3.1. diff --git a/docs/BACKENDS.md b/docs/BACKENDS.md index 3041d0d..53edc5f 100644 --- a/docs/BACKENDS.md +++ b/docs/BACKENDS.md @@ -5,6 +5,10 @@ hardware-isolated microVM. *How* that VM is built, stored, snapshotted, and forked is the backend's job. Four backends exist today; a node can install any combination (`k7 install --backend kfd,kql,k7d,k7d-fc`) and each sandbox picks one (`k7 create --backend …`, annotation `k7.katakate.org/backend`). +`--backend` / inventory `k7_backends` is **required** — there is no default +list. `none` means no sandbox runtime on that node (scheduling-only +master: `[k7_servers:vars] k7_backends=none`). Empty/omitted is an error, +not kfd. | | `kfd` — kata-firecracker-devmapper | `kql` — kata-qemu-longhorn | `k7d` | `k7d-fc` | |---|---|---|---|---| @@ -58,7 +62,7 @@ CLI. You lose virtiofs/hostPath and time-warp. See k7d `SECURITY.md` "Firecracker profile" and `docs/backends.md`. Kata's Firecracker (`kfd`) is a different binary (`/opt/kata/bin`, older pin); - k7d-fc installs upstream v1.16.1 at `/usr/local/bin`. + k7d-fc installs upstream v1.16.2 at `/usr/local/bin`. ## The k7d backend @@ -76,7 +80,7 @@ The Ansible playbook: clone writable volume images with `FICLONE` reflinks); 3. downloads the k7d release tarball (`k7d_artifact_url`, default the public `Katakate/k7d` GitHub release for `k7d_version`, currently - **0.6.0**) and runs the bundled `install.sh`, which installs `k7d` + + **0.7.0**) and runs the bundled `install.sh`, which installs `k7d` + `containerd-shim-k7-v1` into `/usr/local/bin`, guest kernel/initramfs into `/usr/local/share/k7d`, and starts `k7d.service` (control socket `/run/k7d/k7d.sock`). Override with `k7 install --k7d-version ` @@ -171,19 +175,33 @@ node**: - A VM op on a sandbox co-located with the k7-api pod runs directly (the deployment mounts both sockets and ships `crictl`). - A VM op on a sandbox on any OTHER node is forwarded to the k7-agent pod - on that node (`POST /agent/v1/vm/{pause,resume,fork,lookup}` on the pod - IP). Forwarding authenticates with the shared token the install playbook + on that node (`POST /agent/v1/vm/{pause,resume,lookup}` on the pod + IP). The agent does **not** create Deployments or patch nodes + (no ServiceAccount token). Fork Kubernetes writes stay on `k7-api`. + Forwarding authenticates with the token the install playbook writes to `/etc/k7/agent_token` (root, 0600) on every node; a - CiliumNetworkPolicy limits pod-originated agent ingress to the k7-api - pod. No Ready agent on the node / missing token → loud error, never a - silent no-op. -- **CLI on a node** also works for any sandbox: local sandboxes talk to the - local daemon, remote ones are forwarded the same way (root can read the - token). -- A fork still **lands on the source's node** — the agent proxies to the - node-local daemon; the cross-node fork *data path* is future k7d work - (daemon side, not built yet). When it lands, only the forwarding target - changes. + CiliumNetworkPolicy allows agent ingress from the k7-api pod and + local `host` only (`remote-node` denied). No Ready agent / missing + token → loud error, never a silent no-op. +- **CLI on a node** talks to the local daemon for local sandboxes. + Remote VM ops go through `k7-api` (default CLI path). `k7 --core` + cannot host-forward to another node's agent. +- A fork still **lands on the source's node** (`spec.nodeName` on the + fork Deployment). k7-api creates that Deployment after an agent + lookup; CoW is node-local on that k7d daemon. Cross-node fork *data + path* is future k7d work (daemon side, not built yet). + +k7d is one daemon per node. Namespace scoping on API keys does not stop +two tenants sharing that daemon. Pin a tenant with +`k7 generate-api-key tenant-a -n tenant-a --node ` so creates +cannot *choose* another node. `` is the Kubernetes Node name +(`k7 nodes list` / `kubectl get nodes`) — the Linux hostname K3s +registered. The pin is a `nodeSelector` on `kubernetes.io/hostname` +(kubelet stamps that label; `inventory.ini` does not). Inventory +`k7_backends` only stamps `k7.katakate.org/backend-*` at install. +To keep **other** sandboxes off that node, also run +`k7 nodes dedicate --tenant ` (label + NoSchedule taint +`k7.katakate.org/tenant`). See `SECURITY.md`. kql pause/resume/fork have none of these constraints (they are pure Kubernetes/Longhorn operations) and work through the API for any node. diff --git a/examples/k7.yaml b/examples/k7.yaml index 957f5a0..eb72bd1 100644 --- a/examples/k7.yaml +++ b/examples/k7.yaml @@ -1,10 +1,13 @@ name: demo image: alpine:3.20 namespace: default +# node_name: k7-node-01 # copy NAME from `k7 nodes list` (Linux hostname); dedicate to lock others out +backend: k7d limits: - cpu: "100m" - memory: "128Mi" + cpu: "1" + memory: "1Gi" before_script: | - # Installing curl. Egress open during before_script, then restricted (empty whitelist) afterwards + # Installing curl. Egress is open during before_script, then restricted + # (empty whitelist) afterwards. 128Mi is too small for a VM sandbox. apk add curl egress_whitelist: [] diff --git a/pyproject.toml b/pyproject.toml index 5803602..3504965 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "k7" -version = "0.3.1" +version = "0.4.0" description = "Self-hosted VM sandboxes for untrusted and AI code (CLI, API, SDK)" readme = "README.md" requires-python = ">=3.10.11" diff --git a/setup.py b/setup.py index 125f13a..baa1aad 100644 --- a/setup.py +++ b/setup.py @@ -4,7 +4,7 @@ from setuptools import find_packages, setup setup( name="k7-sdk", - version="0.3.1", + version="0.4.0", description="K7 sandbox management Python SDK (HTTP client for the k7 API)", long_description=Path(__file__).with_name("README.md").read_text(), long_description_content_type="text/markdown", diff --git a/src/k7/__init__.py b/src/k7/__init__.py index d4e052e..a6d8f80 100644 --- a/src/k7/__init__.py +++ b/src/k7/__init__.py @@ -1,3 +1,3 @@ """K7 Sandbox Management System""" -__version__ = "0.3.1" +__version__ = "0.4.0" diff --git a/src/k7/api/agent.py b/src/k7/api/agent.py index 0c6fa7e..348bcf6 100644 --- a/src/k7/api/agent.py +++ b/src/k7/api/agent.py @@ -2,21 +2,19 @@ Runs as a DaemonSet (``k7-agent``, kube-system) on every node, reusing the ``k7-api:local`` image with an overridden command -(``uvicorn k7.api.agent:app``). It exposes ONLY the node-local operations -that the centralized k7-api pod cannot perform for sandboxes on other -nodes: +(``uvicorn k7.api.agent:app``). It exposes ONLY node-local operations +that need the k7d socket / crictl / ``lvs`` — never Kubernetes writes: -- ``POST /agent/v1/vm/{pause,resume,fork,lookup}`` — thin wrappers over the - ``K7Core`` k7d helpers (the k7d daemon socket, containerd socket, and - crictl are all node-local). -- ``GET /agent/v1/storage`` — node storage-pool utilization (kfd LVM - thin-pool via ``lvs``, k7d disks pool via ``df``). +- ``POST /agent/v1/vm/{pause,resume,lookup}`` — crictl + k7d daemon. +- ``GET /agent/v1/storage`` — kfd thin-pool / k7d disks utilization. -Auth: every request must carry the shared agent token (header -``X-K7-Agent-Token``) generated by the install playbook and distributed to -``/etc/k7/agent_token`` on every node. A CiliumNetworkPolicy additionally -restricts pod-originated ingress to the k7-api pod. Requests without a -valid token are rejected — never served. +Fork, pause annotations, and NetworkPolicy stay on ``k7-api`` (ClusterRole). +The agent ServiceAccount has no token (``automountServiceAccountToken: +false``), so root on an agent cannot steal cluster-admin-equivalent RBAC. + +Auth: ``X-K7-Agent-Token`` from ``/etc/k7/agent_token``. A CiliumNetworkPolicy +allows ingress from the k7-api pod and the local ``host`` (kubelet probes). +``remote-node`` is denied so a compromised agent cannot call other agents. """ import json @@ -29,6 +27,7 @@ from fastapi.responses import JSONResponse from .. import __version__ from ..core.core import K7Core +from ..core.models import OperationResult app = FastAPI(title="K7 Node Agent", version=__version__) @@ -38,7 +37,7 @@ K7D_DISKS_DIR = os.getenv("K7D_DISKS_DIR", "/var/lib/k7d/disks") def _load_agent_token() -> str: - """Read the shared agent token — unreadable/empty is a deployment bug + """Read the agent token — unreadable/empty is a deployment bug and must fail loudly (a silent 401 would be misdiagnosed as a bad caller token).""" try: @@ -64,7 +63,6 @@ async def verify_agent_token(x_k7_agent_token: str | None = Header(None)): @app.exception_handler(Exception) async def unhandled_exception_handler(request: Request, exc: Exception): # type: ignore[override] - # Fail loud WITH the message — the k7-api forwarder surfaces it verbatim. return JSONResponse( content={"error": {"code": "InternalServerError", "message": str(exc)}}, status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, @@ -84,44 +82,65 @@ def _required_name(body: dict | None) -> tuple[str, str]: return name, body.get("namespace", "default") +def _required_pod_name(body: dict | None) -> str: + pod_name = (body or {}).get("pod_name") + if not pod_name or not isinstance(pod_name, str): + raise HTTPException(status_code=400, detail="pod_name is required") + return pod_name + + +async def _lookup_local_vm(pod_name: str, namespace: str) -> dict: + """crictl + k7d only — no Kubernetes client.""" + core = K7Core() + cri_sandbox_id = await core._k7d_cri_sandbox_id(pod_name, namespace) + resp = await core._k7d_request({"op": "lookup_sandbox", "sandbox_id": cri_sandbox_id}) + if resp.get("status") != "sandbox_found": + raise RuntimeError(f"unexpected k7d lookup_sandbox response: {resp}") + resp["sandbox_id"] = cri_sandbox_id + return resp + + @app.post("/agent/v1/vm/pause", dependencies=[Depends(verify_agent_token)]) async def vm_pause(body: dict | None = None): - name, namespace = _required_name(body) - result = await K7Core().pause_sandbox(name=name, namespace=namespace, snapshot_name=(body or {}).get("snapshot")) - return result.to_dict() + _name, namespace = _required_name(body) + pod_name = _required_pod_name(body) + vm = await _lookup_local_vm(pod_name, namespace) + await K7Core()._k7d_request({"op": "pause_vm", "vm_id": vm["vm_id"]}) + return OperationResult( + success=True, + message=f"k7d VM {vm['vm_id']} frozen in place; memory retained, vCPUs stopped.", + ).to_dict() @app.post("/agent/v1/vm/resume", dependencies=[Depends(verify_agent_token)]) async def vm_resume(body: dict | None = None): - name, namespace = _required_name(body) - result = await K7Core().resume_sandbox(name=name, namespace=namespace) - return result.to_dict() + _name, namespace = _required_name(body) + pod_name = _required_pod_name(body) + vm = await _lookup_local_vm(pod_name, namespace) + await K7Core()._k7d_request({"op": "resume_vm", "vm_id": vm["vm_id"]}) + return OperationResult( + success=True, + message=f"k7d VM {vm['vm_id']} vCPUs restarted.", + ).to_dict() @app.post("/agent/v1/vm/fork", dependencies=[Depends(verify_agent_token)]) -async def vm_fork(body: dict | None = None): - name, namespace = _required_name(body) - new_name = (body or {}).get("new_name") - if not new_name or not isinstance(new_name, str): - raise HTTPException(status_code=400, detail="new_name is required") - result = await K7Core().fork_sandbox( - source_name=name, - new_name=new_name, - namespace=namespace, - snapshot_name=(body or {}).get("snapshot"), +async def vm_fork(_body: dict | None = None): + raise HTTPException( + status_code=400, + detail=( + "k7d fork is a control-plane operation (k7-api ClusterRole creates the " + "Deployment). The agent only runs lookup/pause/resume/storage so a " + "compromised worker cannot create sandboxes on other nodes." + ), ) - return result.to_dict() @app.post("/agent/v1/vm/lookup", dependencies=[Depends(verify_agent_token)]) async def vm_lookup(body: dict | None = None): - name, namespace = _required_name(body) - return await K7Core().lookup_k7d_vm(name, namespace) - - -# --------------------------------------------------------------------------- -# Node storage-pool utilization. -# --------------------------------------------------------------------------- + _name, namespace = _required_name(body) + pod_name = _required_pod_name(body) + return await _lookup_local_vm(pod_name, namespace) def _run(cmd: list[str]) -> str: diff --git a/src/k7/api/main.py b/src/k7/api/main.py index 9091154..16fb6a2 100644 --- a/src/k7/api/main.py +++ b/src/k7/api/main.py @@ -3,6 +3,7 @@ import json import os import secrets import time +from collections.abc import AsyncIterator from datetime import timedelta from pathlib import Path from typing import Any @@ -11,7 +12,7 @@ from fastapi import Depends, FastAPI, Header, HTTPException, Request, status from fastapi.responses import JSONResponse from .. import __version__ -from ..core.core import K7Core +from ..core.core import K7Core, aclose_k7_core from ..core.models import SandboxConfig app = FastAPI(title="K7 Sandbox API", version=__version__) @@ -142,6 +143,34 @@ def authorize_namespace( ) +def apply_node_scope(key_data: dict, node_name: str | None) -> str | None: + """Enforce optional per-key node placement and return the pin. + + Absent/empty ``nodes`` on the key ⇒ unrestricted (backward compatible). + A caller-supplied ``node_name`` is kept for operator/test pinning. + Scoped keys may only place sandboxes on listed nodes; a single-node + scope auto-pins when ``node_name`` is omitted; a multi-node scope + requires an explicit allowed node (fail loud, never silently pick). + """ + allowed = [n for n in (key_data.get("nodes") or []) if n] + requested = node_name.strip() if isinstance(node_name, str) and node_name.strip() else None + if not allowed: + return requested + if requested is None: + if len(allowed) == 1: + return allowed[0] + raise HTTPException( + status_code=403, + detail="API key is node-scoped; pass an explicit allowed node", + ) + if requested not in allowed: + raise HTTPException( + status_code=403, + detail=f"API key is not authorized for node '{requested}'", + ) + return requested + + def success_response( data: Any, status_code: int = status.HTTP_200_OK, headers: dict[str, str] | None = None ) -> JSONResponse: @@ -152,6 +181,15 @@ def error_response(code: str, message: str, status_code: int) -> JSONResponse: return JSONResponse(content={"error": {"code": code, "message": message}}, status_code=status_code) +async def get_k7_core() -> AsyncIterator[K7Core]: + """Per-request K7Core; close the kube aiohttp session when the handler ends.""" + core = K7Core() + try: + yield core + finally: + await aclose_k7_core(core) + + @app.exception_handler(HTTPException) async def http_exception_handler(request: Request, exc: HTTPException): # type: ignore[override] # Map common status codes to generic error codes @@ -188,12 +226,12 @@ async def health(): @app.post("/api/v1/sandboxes") -async def create_sandbox(config: dict, key_data: dict = Depends(verify_api_key)): +async def create_sandbox(config: dict, key_data: dict = Depends(verify_api_key), core: K7Core = Depends(get_k7_core)): """Create a new sandbox.""" try: sandbox_config = SandboxConfig.from_dict(config) authorize_namespace(key_data, sandbox_config.namespace) - core = K7Core() + sandbox_config.node_name = apply_node_scope(key_data, sandbox_config.node_name) result = await core.create_sandbox(sandbox_config) if result.success: @@ -216,19 +254,26 @@ async def create_sandbox(config: dict, key_data: dict = Depends(verify_api_key)) @app.get("/api/v1/sandboxes") -async def list_sandboxes(namespace: str | None = None, key_data: dict = Depends(verify_api_key)): +async def list_sandboxes( + namespace: str | None = None, + key_data: dict = Depends(verify_api_key), + core: K7Core = Depends(get_k7_core), +): """List all sandboxes.""" authorize_namespace(key_data, namespace) - core = K7Core() sandboxes = await core.list_sandboxes(namespace) return success_response([sandbox.to_dict() for sandbox in sandboxes]) @app.get("/api/v1/sandboxes/{name}") -async def get_sandbox(name: str, namespace: str = "default", key_data: dict = Depends(verify_api_key)): +async def get_sandbox( + name: str, + namespace: str = "default", + key_data: dict = Depends(verify_api_key), + core: K7Core = Depends(get_k7_core), +): """Get a single sandbox by name.""" authorize_namespace(key_data, namespace) - core = K7Core() items = await core.list_sandboxes(namespace) for s in items: if s.name == name: @@ -237,10 +282,14 @@ async def get_sandbox(name: str, namespace: str = "default", key_data: dict = De @app.delete("/api/v1/sandboxes/{name}") -async def delete_sandbox(name: str, namespace: str = "default", key_data: dict = Depends(verify_api_key)): +async def delete_sandbox( + name: str, + namespace: str = "default", + key_data: dict = Depends(verify_api_key), + core: K7Core = Depends(get_k7_core), +): """Delete a sandbox.""" authorize_namespace(key_data, namespace) - core = K7Core() result = await core.delete_sandbox(name, namespace) if result.success: @@ -250,10 +299,13 @@ async def delete_sandbox(name: str, namespace: str = "default", key_data: dict = @app.delete("/api/v1/sandboxes") -async def delete_all_sandboxes(namespace: str = "default", key_data: dict = Depends(verify_api_key)): +async def delete_all_sandboxes( + namespace: str = "default", + key_data: dict = Depends(verify_api_key), + core: K7Core = Depends(get_k7_core), +): """Delete all sandboxes in a namespace.""" authorize_namespace(key_data, namespace) - core = K7Core() result = await core.delete_all_sandboxes(namespace) if result.success: @@ -263,7 +315,12 @@ async def delete_all_sandboxes(namespace: str = "default", key_data: dict = Depe @app.post("/api/v1/sandboxes/{name}/pause") -async def pause_sandbox(name: str, body: dict | None = None, key_data: dict = Depends(verify_api_key)): +async def pause_sandbox( + name: str, + body: dict | None = None, + key_data: dict = Depends(verify_api_key), + core: K7Core = Depends(get_k7_core), +): """Pause a sandbox (scale to 0) and optionally take a Longhorn VolumeSnapshot. Body keys (all optional): @@ -273,7 +330,6 @@ async def pause_sandbox(name: str, body: dict | None = None, key_data: dict = De body = body or {} namespace = body.get("namespace", "default") authorize_namespace(key_data, namespace) - core = K7Core() result = await core.pause_sandbox( name=name, namespace=namespace, @@ -285,12 +341,16 @@ async def pause_sandbox(name: str, body: dict | None = None, key_data: dict = De @app.post("/api/v1/sandboxes/{name}/resume") -async def resume_sandbox(name: str, body: dict | None = None, key_data: dict = Depends(verify_api_key)): +async def resume_sandbox( + name: str, + body: dict | None = None, + key_data: dict = Depends(verify_api_key), + core: K7Core = Depends(get_k7_core), +): """Resume a paused sandbox (scale back to 1).""" body = body or {} namespace = body.get("namespace", "default") authorize_namespace(key_data, namespace) - core = K7Core() result = await core.resume_sandbox(name=name, namespace=namespace) if result.success: return success_response({"message": result.message}) @@ -298,7 +358,12 @@ async def resume_sandbox(name: str, body: dict | None = None, key_data: dict = D @app.post("/api/v1/sandboxes/{name}/fork") -async def fork_sandbox(name: str, body: dict, key_data: dict = Depends(verify_api_key)): +async def fork_sandbox( + name: str, + body: dict, + key_data: dict = Depends(verify_api_key), + core: K7Core = Depends(get_k7_core), +): """Fork a kata-qemu-longhorn sandbox into a new name with a cloned root disk. Required body key: new_name. Optional: namespace, snapshot. @@ -310,7 +375,17 @@ async def fork_sandbox(name: str, body: dict, key_data: dict = Depends(verify_ap namespace = body.get("namespace", "default") authorize_namespace(key_data, namespace) snapshot = body.get("snapshot") - core = K7Core() + if key_data.get("nodes"): + items = await core.list_sandboxes(namespace) + source = next((s for s in items if s.name == name), None) + if source is None: + raise HTTPException(status_code=404, detail=f"Sandbox {name} not found in namespace {namespace}") + if not source.node: + raise HTTPException( + status_code=409, + detail=f"Sandbox {name} is not scheduled to a node yet", + ) + apply_node_scope(key_data, source.node) result = await core.fork_sandbox( source_name=name, new_name=new_name, @@ -342,10 +417,10 @@ async def get_sandbox_logs( tail: int = 200, since: int = 0, key_data: dict = Depends(verify_api_key), + core: K7Core = Depends(get_k7_core), ): """Read pod logs (snapshot; no streaming yet).""" authorize_namespace(key_data, namespace) - core = K7Core() result = await core.get_logs( sandbox_name=name, namespace=namespace, @@ -367,6 +442,7 @@ async def exec_command( command_data: dict, namespace: str = "default", key_data: dict = Depends(verify_api_key), + core: K7Core = Depends(get_k7_core), ): """Execute a command in a sandbox.""" authorize_namespace(key_data, namespace) @@ -374,15 +450,13 @@ async def exec_command( if not command: raise HTTPException(status_code=400, detail="Command is required") - core = K7Core() result = await core.exec_command(name, command, namespace) return success_response(result.to_dict()) @app.post("/api/v1/install", dependencies=[Depends(verify_api_key)]) -async def install_node(install_data: dict): +async def install_node(install_data: dict, core: K7Core = Depends(get_k7_core)): """Install K7 on target hosts.""" - core = K7Core() result = core.install_node( install_data.get("playbook"), install_data.get("inventory"), @@ -396,24 +470,31 @@ async def install_node(install_data: dict): @app.get("/api/v1/nodes/storage") -async def get_nodes_storage(key_data: dict = Depends(verify_api_key)): +async def get_nodes_storage(key_data: dict = Depends(verify_api_key), core: K7Core = Depends(get_k7_core)): """Per-node storage-pool utilization (kfd thin-pool + k7d disks pool), aggregated from the k7-agent DaemonSet. A node whose agent is unreachable gets an ``{"error": ...}`` entry — never omitted. - Cluster-scoped (all-namespaces): a namespace-scoped key is 403. - Fail loud — do not silently narrow the per-node map. + Cluster-scoped (all-namespaces): a namespace-scoped or node-scoped + key is 403. Fail loud — do not silently narrow the per-node map. """ authorize_namespace(key_data, None, all_namespaces=True) - core = K7Core() + if key_data.get("nodes"): + raise HTTPException( + status_code=403, + detail="API key is not authorized for cluster-wide node operations", + ) return success_response(await core.nodes_storage()) @app.get("/api/v1/sandboxes/metrics") -async def get_sandbox_metrics(namespace: str | None = None, key_data: dict = Depends(verify_api_key)): +async def get_sandbox_metrics( + namespace: str | None = None, + key_data: dict = Depends(verify_api_key), + core: K7Core = Depends(get_k7_core), +): """Get resource usage metrics for sandboxes.""" authorize_namespace(key_data, namespace) - core = K7Core() metrics = await core.get_sandbox_metrics(namespace) return success_response(metrics) @@ -443,10 +524,10 @@ async def list_snapshots( sandbox: str | None = None, kind: str | None = None, key_data: dict = Depends(verify_api_key), + core: K7Core = Depends(get_k7_core), ): """List VolumeSnapshots, optionally filtered by namespace / sandbox / kind.""" authorize_namespace(key_data, None if all_namespaces else namespace, all_namespaces=all_namespaces) - core = K7Core() snaps = await core.list_snapshots( namespace=namespace, all_namespaces=all_namespaces, @@ -457,10 +538,14 @@ async def list_snapshots( @app.get("/api/v1/snapshots/{name}") -async def get_snapshot(name: str, namespace: str = "default", key_data: dict = Depends(verify_api_key)): +async def get_snapshot( + name: str, + namespace: str = "default", + key_data: dict = Depends(verify_api_key), + core: K7Core = Depends(get_k7_core), +): """Inspect a single VolumeSnapshot by name.""" authorize_namespace(key_data, namespace) - core = K7Core() snap = await core.get_snapshot(name, namespace=namespace) if snap is None: raise HTTPException(status_code=404, detail=f"Snapshot {name} not found in namespace {namespace}") @@ -468,7 +553,12 @@ async def get_snapshot(name: str, namespace: str = "default", key_data: dict = D @app.post("/api/v1/sandboxes/{name}/snapshot") -async def create_snapshot(name: str, body: dict, key_data: dict = Depends(verify_api_key)): +async def create_snapshot( + name: str, + body: dict, + key_data: dict = Depends(verify_api_key), + core: K7Core = Depends(get_k7_core), +): """Snapshot a running sandbox's root PVC without pausing it (kind=named). Body keys: ``snapshot_name`` (required), ``namespace`` (default ``"default"``). @@ -478,7 +568,6 @@ async def create_snapshot(name: str, body: dict, key_data: dict = Depends(verify raise HTTPException(status_code=400, detail="snapshot_name is required") namespace = body.get("namespace", "default") authorize_namespace(key_data, namespace) - core = K7Core() result = await core.create_snapshot(sandbox_name=name, snapshot_name=snapshot_name, namespace=namespace) if result.success: resource = {"name": snapshot_name, "namespace": namespace, "source_sandbox": name} @@ -491,10 +580,14 @@ async def create_snapshot(name: str, body: dict, key_data: dict = Depends(verify @app.delete("/api/v1/snapshots/{name}") -async def delete_snapshot(name: str, namespace: str = "default", key_data: dict = Depends(verify_api_key)): +async def delete_snapshot( + name: str, + namespace: str = "default", + key_data: dict = Depends(verify_api_key), + core: K7Core = Depends(get_k7_core), +): """Delete a VolumeSnapshot by name.""" authorize_namespace(key_data, namespace) - core = K7Core() result = await core.delete_snapshot(name, namespace=namespace) if result.success: return success_response({"message": result.message}) @@ -504,7 +597,12 @@ async def delete_snapshot(name: str, namespace: str = "default", key_data: dict @app.post("/api/v1/snapshots/{name}/restore") -async def restore_snapshot(name: str, body: dict, key_data: dict = Depends(verify_api_key)): +async def restore_snapshot( + name: str, + body: dict, + key_data: dict = Depends(verify_api_key), + core: K7Core = Depends(get_k7_core), +): """Boot a brand-new sandbox from a standalone VolumeSnapshot. Body keys: @@ -512,7 +610,8 @@ async def restore_snapshot(name: str, body: dict, key_data: dict = Depends(verif ``namespace`` (default ``"default"``), ``overrides`` (optional dict: image, backend, root_disk_size, sidecar, limits, entrypoint, cmd, before_script), - ``keep_snapshot`` (default ``true``). + ``keep_snapshot`` (default ``true``), + ``node_name`` (optional pin; a node-scoped key stamps or rejects this). """ body = body or {} new_name = body.get("new_sandbox_name") @@ -520,6 +619,7 @@ async def restore_snapshot(name: str, body: dict, key_data: dict = Depends(verif raise HTTPException(status_code=400, detail="new_sandbox_name is required") namespace = body.get("namespace", "default") authorize_namespace(key_data, namespace) + pinned_node = apply_node_scope(key_data, body.get("node_name")) keep_snapshot = bool(body.get("keep_snapshot", True)) overrides_dict = body.get("overrides") or {} @@ -532,13 +632,13 @@ async def restore_snapshot(name: str, body: dict, key_data: dict = Depends(verif overrides = SandboxConfigOverrides(**filtered) if filtered else None - core = K7Core() result = await core.restore_sandbox( snapshot_name=name, new_sandbox_name=new_name, namespace=namespace, overrides=overrides, keep_snapshot=keep_snapshot, + node_name=pinned_node, ) if result.success: resource = { @@ -559,7 +659,11 @@ async def restore_snapshot(name: str, body: dict, key_data: dict = Depends(verif @app.post("/api/v1/snapshots/gc") -async def gc_snapshots(body: dict | None = None, key_data: dict = Depends(verify_api_key)): +async def gc_snapshots( + body: dict | None = None, + key_data: dict = Depends(verify_api_key), + core: K7Core = Depends(get_k7_core), +): """Sweep stale ``kind=fork`` snapshots older than ``keep_fork_for``. Body (all optional): @@ -573,7 +677,6 @@ async def gc_snapshots(body: dict | None = None, key_data: dict = Depends(verify namespace = body.get("namespace", "default") authorize_namespace(key_data, None if all_namespaces else namespace, all_namespaces=all_namespaces) keep_for = _parse_keep_fork_for(body.get("keep_fork_for")) - core = K7Core() result = await core.gc_snapshots( namespace=namespace, all_namespaces=all_namespaces, diff --git a/src/k7/api/snapshot_gc.py b/src/k7/api/snapshot_gc.py index 29039ad..6677324 100644 --- a/src/k7/api/snapshot_gc.py +++ b/src/k7/api/snapshot_gc.py @@ -52,19 +52,22 @@ async def _main() -> int: keep_minutes = _env_int("K7_GC_KEEP_FORK_FOR_MINUTES", 10) dry_run = _env_bool("K7_GC_DRY_RUN", False) core = K7Core() - result = await core.gc_snapshots( - all_namespaces=True, - keep_fork_for=timedelta(minutes=keep_minutes), - dry_run=dry_run, - ) - if not result.success: - print(f"❌ snapshot-gc failed: {result.error}", file=sys.stderr) - return 1 - print(result.message) - for record in result.data or []: - marker = "would-delete" if dry_run else ("deleted" if record.get("deleted") else "failed") - print(f" [{marker}] {record['namespace']}/{record['name']} (age={record['age']})") - return 0 + try: + result = await core.gc_snapshots( + all_namespaces=True, + keep_fork_for=timedelta(minutes=keep_minutes), + dry_run=dry_run, + ) + if not result.success: + print(f"❌ snapshot-gc failed: {result.error}", file=sys.stderr) + return 1 + print(result.message) + for record in result.data or []: + marker = "would-delete" if dry_run else ("deleted" if record.get("deleted") else "failed") + print(f" [{marker}] {record['namespace']}/{record['name']} (age={record['age']})") + return 0 + finally: + await core.aclose() if __name__ == "__main__": diff --git a/src/k7/cli/k7.py b/src/k7/cli/k7.py index c54012b..1994e34 100644 --- a/src/k7/cli/k7.py +++ b/src/k7/cli/k7.py @@ -15,6 +15,7 @@ import time from datetime import datetime, timedelta from pathlib import Path +import requests import typer from click.core import ParameterSource from rich.console import Console, Group @@ -31,13 +32,31 @@ from rich.table import Table from rich.text import Text from k7 import __version__ as K7_VERSION -from k7.cli._client import CliContext, handle_api_call +from k7.cli._client import CliContext, _resolve_api_ca, _resolve_api_url, handle_api_call, verify_ssl_for_url from k7.cli._config import config_app -from k7.core.core import K7Core +from k7.core.core import K7Core, aclose_k7_core from k7.core.docker import parse_docker_disk from k7.core.models import SandboxConfig, SandboxConfigOverrides from k7.core.sidecar import SIDECAR_REGISTRY + +def _core_run(core: K7Core, coro): + """Run one K7Core coroutine and close its kube ApiClient. + + Typer is sync and uses ``asyncio.run``, which tears down the loop when + the coroutine returns. kubernetes_asyncio binds aiohttp to that loop; + without ``aclose`` the interpreter logs ``Event loop is closed``. + """ + + async def _run(): + try: + return await coro + finally: + await aclose_k7_core(core) + + return asyncio.run(_run()) + + app = typer.Typer(context_settings={"help_option_names": ["-h", "--help"]}) app.add_typer(config_app, name="config") @@ -139,6 +158,19 @@ BACKEND_ALIASES = { "k7": "k7d", "k7-fc": "k7d-fc", } +# Install-time empty set. Not a RuntimeClass — omit/blank is an error, not none. +BACKEND_NONE = "none" +INSTALL_BACKEND_REQUIRED = ( + "Specify --backend (required; there is no default).\n" + " kfd Firecracker + jailer; needs a spare raw disk\n" + " kql Kata QEMU + Longhorn on the OS disk\n" + " k7d warm-fork microVM daemon (shared per node)\n" + " k7d-fc same daemon, Firecracker under the jailer\n" + " none no sandbox runtime on this node (control-plane only)\n" + "Example: k7 install --backend kfd,kql,k7d\n" + "Inventory (-i): every host needs k7_backends=… or k7_backends=none " + "(group vars are fine). Empty/omitted is not none." +) # Deprecated short + pre-rename full names. Still accepted; emit a warning. BACKEND_DEPRECATED_ALIASES = { "fd": "kata-firecracker-devmapper", @@ -177,12 +209,21 @@ def _parse_backends(value: str) -> list[str]: Used by `k7 install` where multiple backends can be installed on a node. Preserves first-seen order; raises typer.BadParameter on unknown entries. + ``none`` is the empty set (no sandbox runtime); it cannot mix with others. """ if value is None: raise typer.BadParameter("Backend value cannot be empty.") raw = [p for p in (s.strip() for s in value.split(",")) if p] if not raw: raise typer.BadParameter("Backend value cannot be empty.") + lowered = [p.lower() for p in raw] + if BACKEND_NONE in lowered: + if lowered != [BACKEND_NONE]: + raise typer.BadParameter( + f"'{BACKEND_NONE}' cannot be combined with other backends. " + "Use only none, or a list of runtimes (kfd, kql, k7d, k7d-fc)." + ) + return [] seen: builtins.list[str] = [] for entry in raw: normalized = _normalize_backend(entry) @@ -196,6 +237,44 @@ def _kubectl_cmd() -> list[str]: return ["k3s", "kubectl"] if shutil.which("k3s") else ["kubectl"] +def _kubectl_run(kubectl: list[str], args: list[str]) -> subprocess.CompletedProcess[str] | None: + """Run kubectl. ``None`` when the binary is missing (laptop / API-only host).""" + try: + return subprocess.run(kubectl + args, capture_output=True, text=True) + except FileNotFoundError: + return None + + +def _api_status_via_https() -> None: + """Laptop path: probe ``GET /health`` on the configured API URL. + + ``k7 api status`` historically shelled out to kubectl, which crashes with + ``FileNotFoundError`` on a machine that only has the .deb CLI. Fail loud + if nothing is configured; otherwise print reachability + the URL. + """ + url = _resolve_api_url(None) + if not url: + typer.echo("❌ kubectl is not installed, and no API URL is configured.") + typer.echo("On a cluster node, `k7 api status` uses kubectl.") + typer.echo("From a laptop, after `k7 config set api.url` / `api.ca` / `api.key`:") + typer.echo(" curl --cacert ./k7-ca.crt https://:31007/health") + raise typer.Exit(1) + ca = _resolve_api_ca(None) + verify = verify_ssl_for_url(url, ca) + try: + resp = requests.get(f"{url.rstrip('/')}/health", timeout=10, verify=verify) + except requests.RequestException as e: + typer.echo(f"❌ Could not reach K7 API at {url}: {e}", err=True) + raise typer.Exit(1) from e + if resp.ok: + typer.echo("✅ K7 API is reachable") + typer.echo(f"🌐 API endpoint: {url}") + typer.echo("\n📝 Replica counts and pod placement need kubectl on a cluster node.") + return + typer.echo(f"❌ K7 API at {url} returned HTTP {resp.status_code}", err=True) + raise typer.Exit(1) + + def _build_default_inventory( hosts: list[str] | None, role: str, @@ -206,13 +285,12 @@ def _build_default_inventory( """Build a minimal multi-group inventory when the user did not provide one. Single-host shortcut: no `hosts` argument → localhost server with the - user-supplied backends. `backends` may contain one or both of - `kata-firecracker-devmapper` and `kata-qemu-longhorn`; the playbook will install - each backend's prerequisites only when listed. + user-supplied backends (``--backend`` is required; ``none`` → + ``k7_backends=none``). """ server_lines: builtins.list[str] = [] agent_lines: builtins.list[str] = [] - backends_csv = ",".join(backends) + backends_csv = ",".join(backends) if backends else BACKEND_NONE has_devmapper = "kata-firecracker-devmapper" in backends has_longhorn = "kata-qemu-longhorn" in backends @@ -294,20 +372,23 @@ def _read_api_endpoint(kubectl: list[str]) -> str | None: return written except OSError: pass - port_result = subprocess.run( - kubectl - + [ - "get", - "svc", - "k7-api", - "-n", - "kube-system", - "-o", - "jsonpath={.spec.ports[0].nodePort},{.spec.ports[0].targetPort}", - ], - capture_output=True, - text=True, - ) + try: + port_result = subprocess.run( + kubectl + + [ + "get", + "svc", + "k7-api", + "-n", + "kube-system", + "-o", + "jsonpath={.spec.ports[0].nodePort},{.spec.ports[0].targetPort}", + ], + capture_output=True, + text=True, + ) + except FileNotFoundError: + return _resolve_api_url(None) if port_result.returncode != 0 or not port_result.stdout.strip(): return None node_port, _, target_port = port_result.stdout.strip().partition(",") @@ -365,22 +446,21 @@ def install( "--disk", help="Block device to use for LVM thin-pool (e.g., /dev/nvme2n1)", ), - backend: str = typer.Option( - "kata-firecracker-devmapper,kata-qemu-longhorn", + backend: str | None = typer.Option( + None, "--backend", "-b", help=( - "Comma-separated sandbox backends to install on this node. " - "Choices: kata-firecracker-devmapper (kfd), kata-qemu-longhorn (kql), k7d, k7d-fc (alias k7-fc). " - "Default installs the two Kata backends; add k7d / k7d-fc explicitly for the " - "warm-fork microVM runtime (k7d-fc is Firecracker under the jailer)." + "Comma-separated sandbox backends to install. Required when not using " + "-i inventory (every inventory host must set k7_backends). " + "Choices: kfd, kql, k7d, k7d-fc, none. No default — omit is an error. " + "none means no sandbox runtime (scheduling-only node)." ), - show_default=True, ), k7d_version: str | None = typer.Option( None, "--k7d-version", - help="k7d release version to install (k7d backend only; default from playbook: 0.6.0)", + help="k7d release version to install (k7d backend only; default from playbook: 0.7.0)", ), k7d_artifact: str | None = typer.Option( None, @@ -481,15 +561,17 @@ def install( ): """Install K7 on target hosts using Ansible. - Single-node (default): `k7 install` provisions localhost with **both** - Kata backends (kata-firecracker-devmapper + kata-qemu-longhorn). Pass - `--backend kfd,kql,k7d` to also install the k7d warm-fork runtime - (downloads `Katakate/k7d` v0.6.0 unless `--k7d-version` / `--k7d-artifact` - override it). `--backend k7d-fc` (alias `k7-fc`) adds RuntimeClass - `k7-fc` and the pinned Firecracker+jailer next to that daemon. + Single-node: `k7 install --backend kfd,kql,k7d` (``--backend`` is required; + there is no default). ``none`` installs K3s with no sandbox runtime. + Pass ``k7d`` / ``k7d-fc`` for the warm-fork daemon (downloads + `Katakate/k7d` v0.7.0 unless `--k7d-version` / `--k7d-artifact` override + it). `--backend k7d-fc` (alias `k7-fc`) adds RuntimeClass `k7-fc` and + the pinned Firecracker+jailer next to that daemon. Multi-node: `k7 install -i inventory.ini` reads roles, backends, and disks - from the Ansible inventory. See `inventory.ini.example` for the layout. + from the Ansible inventory. Every host must set ``k7_backends`` (or inherit + it from group vars); use ``k7_backends=none`` for scheduling-only masters. + Empty/omitted is not none. See `inventory.ini.example`. Dual-NVMe boxes need the OS on one disk and a raw spare for kfd — see `tutorials/k7_hetzner_node_setup.md` (`TWO_DISK=1`). """ @@ -565,11 +647,16 @@ def install( with open(playbook) as f: playbook_content = f.read() - backends_list = _parse_backends(backend) + inventory_file = inventory if inventory and os.path.exists(inventory) else None + backend_explicit = ctx.get_parameter_source("backend") == ParameterSource.COMMANDLINE + if not backend_explicit and not inventory_file: + typer.echo(INSTALL_BACKEND_REQUIRED, err=True) + raise typer.Exit(1) + backends_list = _parse_backends(backend) if backend_explicit and backend is not None else [] inventory_content = None - if inventory and os.path.exists(inventory): - with open(inventory) as f: + if inventory_file: + with open(inventory_file) as f: inventory_content = f.read() else: inventory_content = _build_default_inventory( @@ -641,7 +728,7 @@ def install( # to min(3, node count). When the user supplied an inventory we can't # know the node count up-front, so we let the playbook compute its own # default (min(3, len(k7_cluster))) and only forward an explicit value. - inventory_user_supplied = bool(inventory) + inventory_user_supplied = bool(inventory_file) node_count = len(hosts) if hosts else 1 if replicas is not None: effective_replicas: int | None = replicas @@ -666,11 +753,9 @@ def install( # `k7_disk` is the legacy name; the playbook prefers `k7_devmapper_disk` # but still falls back to `k7_disk` for backward compat. "k7_disk": disk, - # `k7_backends` (plural, comma-separated) is authoritative; the - # playbook accepts a single `k7_backend` only as a back-compat - # fallback when neither inventory `k7_backends` nor extra-var - # `k7_backends` is set. - "k7_backends": ",".join(backends_list), + # `k7_backends` is required on every host. Empty extra-var would + # look like "omitted" in Ansible — send the `none` sentinel. + "k7_backends": ",".join(backends_list) if backends_list else BACKEND_NONE, "longhorn_replicas": effective_replicas, "longhorn_data_path": longhorn_data_path, "longhorn_extra_disk": longhorn_extra_disk, @@ -860,6 +945,11 @@ def create( "--docker-disk", help="Docker graph disk size for --docker (e.g. 20Gi, 40Gi). Default 20Gi on all backends.", ), + node: str | None = typer.Option( + None, + "--node", + help="Pin to this Kubernetes node name (`kubectl get nodes` / `k7 nodes list`). Joins via kubernetes.io/hostname (kubelet-stamped). YAML field: node_name.", + ), ): """Create a new sandbox from YAML config or CLI arguments.""" # Three explicit egress modes. --egress-open → open @@ -1009,6 +1099,9 @@ def create( hint = others if others else "no non-docker sidecar types (--sidecar docker is --docker)" raise typer.BadParameter(f"Unknown sidecar type '{sandbox_config.sidecar}'. Available: {hint}") + if node: + sandbox_config.node_name = node + core = K7Core() progress = Progress( SpinnerColumn(), @@ -1417,12 +1510,12 @@ def create( if cli_ctx.use_core: group = RichGroup(status_text, details_text, progress) with RichLive(group, refresh_per_second=8, transient=False) as live: - result = asyncio.run(core.create_sandbox(sandbox_config, progress_callback=on_progress)) + result = _core_run(core, core.create_sandbox(sandbox_config, progress_callback=on_progress)) if not result.success: typer.echo(f"❌ Failed to create sandbox: {result.error}", err=True) raise typer.Exit(1) endpoints = (result.data or {}).get("endpoints") or [] - sandboxes_for_ready = asyncio.run(core.list_sandboxes(sandbox_config.namespace)) + sandboxes_for_ready = _core_run(core, core.list_sandboxes(sandbox_config.namespace)) target_ready = any(s.name == sandbox_config.name and s.ready == "True" for s in sandboxes_for_ready) else: # The API doesn't stream progress yet (a separate spec). Print a @@ -1436,7 +1529,7 @@ def create( if sandbox_config.expose_ports: # The SDK's create() returns a proxy, so the API path cannot show a # resolved URL here. Point at the NodePorts instead of guessing one. - typer.echo(f"🌐 Exposed ports: see the NodePorts in k7 list --name {sandbox_config.name}") + typer.echo(f"🌐 Exposed ports: see the NodePorts column in `k7 list` (sandbox {sandbox_config.name})") for endpoint in endpoints: typer.echo(f"🌐 Port {endpoint['port']} exposed at {endpoint['url']}") if target_ready: @@ -1475,7 +1568,7 @@ def list( cli_ctx: CliContext = ctx.obj if cli_ctx.use_core: core = K7Core() - sandboxes = [s.to_dict() for s in asyncio.run(core.list_sandboxes(namespace))] + sandboxes = [s.to_dict() for s in _core_run(core, core.list_sandboxes(namespace))] else: sandboxes = handle_api_call(lambda: cli_ctx.client().list(namespace=namespace)) @@ -1555,7 +1648,7 @@ def delete( cli_ctx: CliContext = ctx.obj if cli_ctx.use_core: core = K7Core() - result = asyncio.run(core.delete_sandbox(name, namespace)) + result = _core_run(core, core.delete_sandbox(name, namespace)) if not result.success: typer.echo(f"❌ Failed to delete sandbox: {result.error}", err=True) raise typer.Exit(1) @@ -1581,7 +1674,7 @@ def delete_all( # goes through the same routing as `k7 list` so off-cluster runs work. if cli_ctx.use_core: core = K7Core() - sandboxes = [s.to_dict() for s in asyncio.run(core.list_sandboxes(namespace))] + sandboxes = [s.to_dict() for s in _core_run(core, core.list_sandboxes(namespace))] else: sandboxes = handle_api_call(lambda: cli_ctx.client().list(namespace=namespace)) @@ -1600,7 +1693,7 @@ def delete_all( if cli_ctx.use_core: core = K7Core() - result = asyncio.run(core.delete_all_sandboxes(namespace)) + result = _core_run(core, core.delete_all_sandboxes(namespace)) if result.success: typer.echo(f"✅ {result.message}") return @@ -1711,7 +1804,7 @@ def exec( cli_ctx: CliContext = ctx.obj if cli_ctx.use_core: core = K7Core() - result = asyncio.run(core.exec_command(name, joined, namespace=namespace)) + result = _core_run(core, core.exec_command(name, joined, namespace=namespace)) data = result.to_dict() else: data = handle_api_call(lambda: cli_ctx.client().exec(name, joined, namespace=namespace)) @@ -1742,7 +1835,7 @@ def top( table.add_column("CPU Usage (cores)") table.add_column("Memory Usage (MiB)") - metrics_list = asyncio.run(core.get_sandbox_metrics(namespace)) + metrics_list = _core_run(core, core.get_sandbox_metrics(namespace)) for metric in metrics_list: sb_name = metric["name"] @@ -1809,6 +1902,17 @@ def top( time.sleep(refresh_interval) +def _dedupe_keep_order(values: builtins.list[str] | None) -> builtins.list[str]: + """Preserve first-seen order; drop empties and duplicates.""" + seen: set[str] = set() + out: builtins.list[str] = [] + for v in values or []: + if v and v not in seen: + seen.add(v) + out.append(v) + return out + + @app.command() def generate_api_key( name: str, @@ -1819,6 +1923,11 @@ def generate_api_key( "-n", help="Restrict key to this namespace (repeatable). Omit for unrestricted access.", ), + node: builtins.list[str] | None = typer.Option( + None, + "--node", + help="Restrict key to this Kubernetes node name (repeatable; copy from `k7 nodes list`). Placement uses kubernetes.io/hostname (kubelet-stamped, not inventory). Omit for unrestricted placement.", + ), ): """Generate a new API key.""" api_key = secrets.token_urlsafe(32) @@ -1836,16 +1945,12 @@ def generate_api_key( "expires": expiry_timestamp, "last_used": None, } - if namespace: - # Preserve order, drop empties/duplicates. - seen: set[str] = set() - scoped: builtins.list[str] = [] - for ns in namespace: - if ns and ns not in seen: - seen.add(ns) - scoped.append(ns) - if scoped: - entry["namespaces"] = scoped + scoped_ns = _dedupe_keep_order(namespace) + if scoped_ns: + entry["namespaces"] = scoped_ns + scoped_nodes = _dedupe_keep_order(node) + if scoped_nodes: + entry["nodes"] = scoped_nodes api_keys[key_hash] = entry _write_api_keys(api_keys) @@ -1857,6 +1962,10 @@ def generate_api_key( typer.echo(f"Namespaces: {', '.join(entry['namespaces'])}") else: typer.echo("Namespaces: * (unrestricted)") + if entry.get("nodes"): + typer.echo(f"Nodes: {', '.join(entry['nodes'])}") + else: + typer.echo("Nodes: * (unrestricted)") typer.echo("Keep this key secure - it won't be shown again!") @@ -1877,6 +1986,7 @@ def list_api_keys(): table.add_column("Expires", style="yellow") table.add_column("Last Used", style="green") table.add_column("Namespaces", style="magenta") + table.add_column("Nodes", style="magenta") for _key_hash, key_data in api_keys.items(): created = datetime.fromtimestamp(key_data["created"]).strftime("%Y-%m-%d %H:%M") @@ -1886,8 +1996,10 @@ def list_api_keys(): last_used = datetime.fromtimestamp(key_data["last_used"]).strftime("%Y-%m-%d %H:%M") namespaces = key_data.get("namespaces") or [] ns_col = "*" if not namespaces else ", ".join(namespaces) + nodes = key_data.get("nodes") or [] + node_col = "*" if not nodes else ", ".join(nodes) - table.add_row(key_data["name"], created, expires, last_used, ns_col) + table.add_row(key_data["name"], created, expires, last_used, ns_col, node_col) console.print(table) @@ -1967,7 +2079,7 @@ def pause( cli_ctx: CliContext = ctx.obj if cli_ctx.use_core: core = K7Core() - result = asyncio.run(core.pause_sandbox(name=name, namespace=namespace, snapshot_name=snapshot_name)) + result = _core_run(core, core.pause_sandbox(name=name, namespace=namespace, snapshot_name=snapshot_name)) if not result.success: typer.echo(f"❌ {result.error}", err=True) raise typer.Exit(1) @@ -1987,7 +2099,7 @@ def resume( cli_ctx: CliContext = ctx.obj if cli_ctx.use_core: core = K7Core() - result = asyncio.run(core.resume_sandbox(name=name, namespace=namespace)) + result = _core_run(core, core.resume_sandbox(name=name, namespace=namespace)) if not result.success: typer.echo(f"❌ {result.error}", err=True) raise typer.Exit(1) @@ -2061,14 +2173,15 @@ def restore( cli_ctx: CliContext = ctx.obj if cli_ctx.use_core: core = K7Core() - result = asyncio.run( + result = _core_run( + core, core.restore_sandbox( snapshot_name=snapshot_name, new_sandbox_name=new_sandbox_name, namespace=namespace, overrides=overrides, keep_snapshot=keep_snapshot, - ) + ), ) if not result.success: typer.echo(f"❌ {result.error}", err=True) @@ -2106,13 +2219,14 @@ def fork( typer.echo(f"Forked {source} -> {new_name}") return core = K7Core() - result = asyncio.run( + result = _core_run( + core, core.fork_sandbox( source_name=source, new_name=new_name, namespace=namespace, snapshot_name=snapshot, - ) + ), ) if result.success: typer.echo(result.message) @@ -2225,11 +2339,10 @@ def _api_scale_deployment(replicas: int) -> None: failure mode this is meant to eliminate. """ kubectl = _kubectl_cmd() - result = subprocess.run( - kubectl + ["scale", "deployment", "k7-api", "-n", "kube-system", f"--replicas={replicas}"], - capture_output=True, - text=True, - ) + result = _kubectl_run(kubectl, ["scale", "deployment", "k7-api", "-n", "kube-system", f"--replicas={replicas}"]) + if result is None: + typer.echo("❌ `k7 api enable`/`disable` need kubectl on a cluster node.", err=True) + raise typer.Exit(1) if result.returncode != 0: typer.echo(f"❌ Failed to scale k7-api to {replicas}: {result.stderr.strip()}", err=True) raise typer.Exit(1) @@ -2239,9 +2352,9 @@ def _api_scale_deployment(replicas: int) -> None: def api_status_cmd(): """Show API server readiness, endpoint, and key-management hints.""" kubectl = _kubectl_cmd() - result = subprocess.run( - kubectl - + [ + result = _kubectl_run( + kubectl, + [ "get", "deployment", "k7-api", @@ -2250,9 +2363,10 @@ def api_status_cmd(): "-o", "jsonpath={.status.readyReplicas}/{.spec.replicas}", ], - capture_output=True, - text=True, ) + if result is None: + _api_status_via_https() + return if result.returncode != 0: typer.echo("❌ K7 API deployment not found in K3s") typer.echo("Run `k7 install` to deploy it, or `k7 install --no-api` if that's deliberate.") @@ -2284,7 +2398,10 @@ def api_status_cmd(): typer.echo("\n📝 SDK usage example:") typer.echo(" from k7_sdk import Client") - typer.echo(" k7 = Client(endpoint='', api_key='')") + typer.echo(" k7 = Client(") + typer.echo(" endpoint='', api_key='',") + typer.echo(" verify_ssl='./k7-ca.crt', # cluster CA; on the node use /etc/k7/tls/ca.crt") + typer.echo(" )") typer.echo(" sb = k7.create({'name': 'test', 'image': 'alpine:3.21'})") typer.echo("\n🔐 Manage API keys:") typer.echo(" k7 generate-api-key ") @@ -2437,12 +2554,48 @@ def _deprecated_get_api_endpoint(): nodes_app = typer.Typer( - help="Query cluster nodes (storage pools, …).", + help="Query and dedicate cluster nodes (storage pools, tenant isolation).", context_settings={"help_option_names": ["-h", "--help"]}, ) app.add_typer(nodes_app, name="nodes") +def _print_nodes_list_table(rows: builtins.list[dict]) -> None: + """Render ``list_cluster_nodes()`` as NAME / HOSTNAME / BACKENDS / TENANT.""" + if not rows: + typer.echo("No nodes found.") + return + table = Table(title="K7 Nodes") + table.add_column("NAME", style="cyan") + table.add_column("HOSTNAME") + table.add_column("BACKENDS") + table.add_column("TENANT") + for n in rows: + backends = ",".join(n.get("backends") or []) or "-" + table.add_row(n.get("name") or "-", n.get("hostname") or "-", backends, n.get("tenant") or "-") + Console().print(table) + + +@nodes_app.command("list") +def nodes_list( + as_json: bool = typer.Option(False, "--json", help="Print raw JSON instead of a table"), +): + """Show cluster node names, hostname labels, backends, and tenant dedication. + + NAME is what ``--node`` and ``k7 nodes dedicate`` take. It is the K3s + Node name (machine hostname), not something you label by hand. + HOSTNAME is ``kubernetes.io/hostname`` (kubelet). BACKENDS come from + ``k7 install --backend`` / inventory ``k7_backends`` (``none`` means + no sandbox runtime). TENANT comes from ``k7 nodes dedicate``. + """ + core = K7Core() + rows = _core_run(core, core.list_cluster_nodes()) + if as_json: + typer.echo(json.dumps(rows, indent=2, default=str)) + return + _print_nodes_list_table(rows) + + def _print_nodes_storage_table(data: dict) -> None: """Render ``nodes_storage()`` output as a per-node Rich table.""" if not data: @@ -2483,7 +2636,8 @@ def nodes_storage( """Show per-node kfd thin-pool and k7d disks-pool utilization.""" cli_ctx: CliContext = ctx.obj if cli_ctx.use_core: - data = asyncio.run(K7Core().nodes_storage()) + core = K7Core() + data = _core_run(core, core.nodes_storage()) else: data = handle_api_call(lambda: cli_ctx.client().nodes_storage()) if as_json: @@ -2492,6 +2646,38 @@ def nodes_storage( _print_nodes_storage_table(data) +@nodes_app.command("dedicate") +def nodes_dedicate( + node: str = typer.Argument(..., help="Kubernetes node name (metadata.name / hostname)"), + tenant: str = typer.Option(..., "--tenant", help="Tenant id written as k7.katakate.org/tenant"), +): + """Label and taint a node so only that tenant's sandboxes can land there. + + The label is the join key; the NoSchedule taint is the exclusive lock. + Must run with cluster access (on a node, or kubeconfig). Does not go + through k7-api. + """ + core = K7Core() + result = _core_run(core, core.dedicate_node(node, tenant)) + if not result.success: + typer.echo(f"❌ {result.error}", err=True) + raise typer.Exit(1) + typer.echo(f"✅ {result.message}") + + +@nodes_app.command("undedicate") +def nodes_undedicate( + node: str = typer.Argument(..., help="Kubernetes node name (metadata.name / hostname)"), +): + """Remove the k7 tenant label and taint from a node.""" + core = K7Core() + result = _core_run(core, core.undedicate_node(node)) + if not result.success: + typer.echo(f"❌ {result.error}", err=True) + raise typer.Exit(1) + typer.echo(f"✅ {result.message}") + + # ``k7 snapshot …`` sub-app for VolumeSnapshot CRUD + GC. # --------------------------------------------------------------------------- @@ -2566,13 +2752,14 @@ def snapshot_list( cli_ctx: CliContext = ctx.obj if cli_ctx.use_core: core = K7Core() - snap_objs = asyncio.run( + snap_objs = _core_run( + core, core.list_snapshots( namespace=namespace, all_namespaces=all_namespaces, sandbox=sandbox, kind=kind, - ) + ), ) snaps = [s.to_dict() for s in snap_objs] else: @@ -2597,7 +2784,7 @@ def snapshot_inspect( cli_ctx: CliContext = ctx.obj if cli_ctx.use_core: core = K7Core() - snap = asyncio.run(core.get_snapshot(name, namespace=namespace)) + snap = _core_run(core, core.get_snapshot(name, namespace=namespace)) if snap is None: typer.echo(f"❌ Snapshot {name} not found in namespace {namespace}", err=True) raise typer.Exit(1) @@ -2621,7 +2808,7 @@ def snapshot_create( cli_ctx: CliContext = ctx.obj if cli_ctx.use_core: core = K7Core() - result = asyncio.run(core.create_snapshot(sandbox_name, snapshot_name, namespace=namespace)) + result = _core_run(core, core.create_snapshot(sandbox_name, snapshot_name, namespace=namespace)) if not result.success: typer.echo(f"❌ {result.error}", err=True) raise typer.Exit(1) @@ -2650,7 +2837,7 @@ def snapshot_delete( cli_ctx: CliContext = ctx.obj if cli_ctx.use_core: core = K7Core() - result = asyncio.run(core.delete_snapshot(name, namespace=namespace)) + result = _core_run(core, core.delete_snapshot(name, namespace=namespace)) if not result.success: typer.echo(f"❌ {result.error}", err=True) raise typer.Exit(1) @@ -2688,13 +2875,14 @@ def snapshot_gc( else: td = timedelta(seconds=int(keep_fork_for)) core = K7Core() - result = asyncio.run( + result = _core_run( + core, core.gc_snapshots( namespace=namespace, all_namespaces=all_namespaces, keep_fork_for=td, dry_run=dry_run, - ) + ), ) if not result.success: typer.echo(f"❌ {result.error}", err=True) diff --git a/src/k7/core/core.py b/src/k7/core/core.py index f6cd1b1..0263db5 100644 --- a/src/k7/core/core.py +++ b/src/k7/core/core.py @@ -1,5 +1,6 @@ import asyncio import copy +import inspect import ipaddress import json import math @@ -34,10 +35,14 @@ from .docker import ( DIND_IMAGE, DOCKER_HOST_URL, DOCKER_UNSUPPORTED_K7D, + FSFREEZE_STAGED, + FSFREEZE_VEHICLE, GRAPH_DEVICE_PATH, - KATA_FORK_GRAPH_REJECT, + KATA_GRAPH_MOUNT, KATA_SOCKET_DIR, KFD_DOCKER_STORAGE_CLASS, + KFD_FORK_REJECT, + STAGE_FSFREEZE_CMD, VEHICLE_CONTAINER_NAME, k7d_supports_docker, parse_docker_disk, @@ -63,6 +68,9 @@ K7D_ANN_FORK_SOURCE_VM = "k7d.katakate.org/fork-source-vm" # Deployment-level marker k7 stamps so `k7 list` / resume can tell a # VM-frozen (paused) k7d sandbox from a running one. K7D_ANN_PAUSED = "k7.katakate.org/k7d-paused" +# Tenant isolation: label is the join key (nodeSelector), matching taint is +# the lock (NoSchedule). Same family as the backend labels. +K7_TENANT_LABEL = "k7.katakate.org/tenant" # Message pointing users at k7d's native VM snapshot trees for the verbs # that intentionally do NOT map onto the k7d backend. K7D_SNAPSHOT_UNSUPPORTED = ( @@ -84,6 +92,8 @@ _ALLOWED_BACKENDS = ( "k7d-fc", ) _K7D_FAMILY = frozenset({"k7d", "k7d-fc"}) +# Kata refuses hypervisor memory below this (shim: "minimum required 256"). +KATA_MIN_HYPERVISOR_MEMORY_MIB = 256 # CRDs (VolumeSnapshotContent) and PVC finalizer lists ignore strategic-merge # patches. kubectl --type=merge works; the Python client must say so too. _K8S_MERGE_PATCH = "application/merge-patch+json" @@ -94,6 +104,16 @@ def _is_k7d_family(backend: str | None) -> bool: return backend in _K7D_FAMILY +async def aclose_k7_core(core: object) -> None: + """Close a K7Core kube session; no-op for test doubles that are not awaitable.""" + closer = getattr(core, "aclose", None) + if closer is None: + return + result = closer() + if inspect.isawaitable(result): + await result + + def _volume_is_hostpath_or_pvc(vol: dict[str, Any]) -> bool: return any(k in vol for k in ("hostPath", "host_path", "persistentVolumeClaim", "persistent_volume_claim")) @@ -176,6 +196,8 @@ class K7Core: self._networking_v1_client = None self._metrics_client = None self._custom_objects_client = None + self._batch_v1_client = None + self._apiextensions_v1_client = None self._config_loaded = False self._api_client = None @@ -212,41 +234,74 @@ class K7Core: self._config_loaded = True + async def _get_api_client(self): + """One kubernetes_asyncio ApiClient (aiohttp session) per K7Core.""" + if self._api_client is None: + await self._load_k3s_config() + self._api_client = client.ApiClient() + return self._api_client + + async def aclose(self) -> None: + """Close the kube ApiClient before the event loop dies. + + ``asyncio.run`` shuts the loop when the coroutine returns. An open + kubernetes_asyncio session then logs ``Unclosed client session`` / + ``Event loop is closed`` (and can wipe CLI stdout). + """ + api_client = self._api_client + self._api_client = None + self._apps_v1_client = None + self._core_v1_client = None + self._networking_v1_client = None + self._metrics_client = None + self._custom_objects_client = None + self._batch_v1_client = None + self._apiextensions_v1_client = None + if api_client is not None: + await api_client.close() + async def _get_apps_v1_client(self): """Get or create AppsV1Api client instance.""" if self._apps_v1_client is None: - await self._load_k3s_config() - self._apps_v1_client = client.AppsV1Api() + self._apps_v1_client = client.AppsV1Api(await self._get_api_client()) return self._apps_v1_client async def _get_core_v1_client(self): """Get or create CoreV1Api client instance.""" if self._core_v1_client is None: - await self._load_k3s_config() - self._core_v1_client = client.CoreV1Api() + self._core_v1_client = client.CoreV1Api(await self._get_api_client()) return self._core_v1_client async def _get_networking_v1_client(self): """Get or create NetworkingV1Api client instance.""" if self._networking_v1_client is None: - await self._load_k3s_config() - self._networking_v1_client = client.NetworkingV1Api() + self._networking_v1_client = client.NetworkingV1Api(await self._get_api_client()) return self._networking_v1_client async def _get_metrics_client(self): """Get or create CustomObjectsApi client for metrics.""" if self._metrics_client is None: - await self._load_k3s_config() - self._metrics_client = client.CustomObjectsApi() + self._metrics_client = client.CustomObjectsApi(await self._get_api_client()) return self._metrics_client async def _get_custom_objects_client(self): """Get or create CustomObjectsApi client for VolumeSnapshots.""" if self._custom_objects_client is None: - await self._load_k3s_config() - self._custom_objects_client = client.CustomObjectsApi() + self._custom_objects_client = client.CustomObjectsApi(await self._get_api_client()) return self._custom_objects_client + async def _get_batch_v1_client(self): + """Get or create BatchV1Api client instance.""" + if self._batch_v1_client is None: + self._batch_v1_client = client.BatchV1Api(await self._get_api_client()) + return self._batch_v1_client + + async def _get_apiextensions_v1_client(self): + """Get or create ApiextensionsV1Api client instance.""" + if self._apiextensions_v1_client is None: + self._apiextensions_v1_client = client.ApiextensionsV1Api(await self._get_api_client()) + return self._apiextensions_v1_client + def _load_persist_bind_script(self) -> str: """Load the bind-mount wrapper script from package assets.""" try: @@ -498,24 +553,133 @@ class K7Core: cmd = list(override_cmd) if override_cmd is not None else list(image_cmd) return ep + cmd - async def _list_backends_per_node(self) -> dict[str, list[str]]: - """Return {node_name: [supported backends]} from k7 backend labels.""" + def _backends_from_node_labels(self, labels: dict) -> list[str]: + return sorted( + { + self._canonicalize_backend(k.removeprefix("k7.katakate.org/backend-")) or "" + for k, v in labels.items() + if k.startswith("k7.katakate.org/backend-") and v == "true" + } + - {""} + ) + + async def list_cluster_nodes(self) -> list[dict]: + """Return name, hostname label, backends, tenant for each Node. + + ``name`` is what ``--node`` / ``k7 nodes dedicate`` take (K3s + ``metadata.name``, usually the Linux hostname). ``hostname`` is + ``kubernetes.io/hostname`` — kubelet stamps it; inventory.ini does not. + Backend labels come from ``k7 install``. Tenant is ``k7 nodes dedicate``. + """ v1 = await self._get_core_v1_client() nodes = await v1.list_node() - out: dict[str, list[str]] = {} + out: list[dict] = [] for n in nodes.items: labels = n.metadata.labels or {} - backends = sorted( + taints = list((n.spec.taints if n.spec else None) or []) + tenant = labels.get(K7_TENANT_LABEL) or "" + if not tenant: + for t in taints: + if getattr(t, "key", None) == K7_TENANT_LABEL: + tenant = getattr(t, "value", None) or "" + break + out.append( { - self._canonicalize_backend(k.removeprefix("k7.katakate.org/backend-")) or "" - for k, v in labels.items() - if k.startswith("k7.katakate.org/backend-") and v == "true" + "name": n.metadata.name, + "hostname": labels.get("kubernetes.io/hostname") or "", + "backends": self._backends_from_node_labels(labels), + "tenant": tenant, } - - {""} ) - out[n.metadata.name] = backends return out + async def _list_backends_per_node(self) -> dict[str, list[str]]: + """Return {node_name: [supported backends]} from k7 backend labels.""" + return {n["name"]: n["backends"] for n in await self.list_cluster_nodes()} + + async def _tenant_tolerations_for_node(self, node_name: str) -> list: + """Return a NoSchedule toleration if ``node_name`` is tenant-tainted. + + Fail loud when the node does not exist — a pin to a missing name is + a config bug, not a Pending pod. + """ + v1 = await self._get_core_v1_client() + try: + node = await v1.read_node(node_name) + except ApiException as e: + if e.status == 404: + raise RuntimeError(f"node {node_name} not found") from e + raise + taints = (node.spec.taints if node.spec else None) or [] + out: list = [] + for t in taints: + if getattr(t, "key", None) == K7_TENANT_LABEL and getattr(t, "effect", None) == "NoSchedule": + out.append( + client.V1Toleration( + key=K7_TENANT_LABEL, + operator="Equal", + value=getattr(t, "value", None), + effect="NoSchedule", + ) + ) + return out + + async def dedicate_node(self, node_name: str, tenant: str) -> OperationResult: + """Label + taint a node for one tenant. Other sandboxes cannot land there.""" + if not node_name or not tenant: + return OperationResult(success=False, error="node name and --tenant are required") + v1 = await self._get_core_v1_client() + try: + node = await v1.read_node(node_name) + except ApiException as e: + if e.status == 404: + return OperationResult(success=False, error=f"node {node_name} not found") + return OperationResult(success=False, error=f"read node {node_name}: {e}") + taints = [ + t for t in ((node.spec.taints if node.spec else None) or []) if getattr(t, "key", None) != K7_TENANT_LABEL + ] + taints.append(client.V1Taint(key=K7_TENANT_LABEL, value=tenant, effect="NoSchedule")) + body = { + "metadata": {"labels": {K7_TENANT_LABEL: tenant}}, + "spec": { + "taints": [{"key": t.key, "value": getattr(t, "value", None), "effect": t.effect} for t in taints] + }, + } + try: + await v1.patch_node(node_name, body) + except ApiException as e: + return OperationResult(success=False, error=f"patch node {node_name}: {e}") + return OperationResult( + success=True, + message=f"node {node_name} dedicated to tenant {tenant} (label+taint {K7_TENANT_LABEL})", + ) + + async def undedicate_node(self, node_name: str) -> OperationResult: + """Drop the k7 tenant label and taint from a node.""" + if not node_name: + return OperationResult(success=False, error="node name is required") + v1 = await self._get_core_v1_client() + try: + node = await v1.read_node(node_name) + except ApiException as e: + if e.status == 404: + return OperationResult(success=False, error=f"node {node_name} not found") + return OperationResult(success=False, error=f"read node {node_name}: {e}") + taints = [ + t for t in ((node.spec.taints if node.spec else None) or []) if getattr(t, "key", None) != K7_TENANT_LABEL + ] + body = { + "metadata": {"labels": {K7_TENANT_LABEL: None}}, + "spec": { + "taints": [{"key": t.key, "value": getattr(t, "value", None), "effect": t.effect} for t in taints] + }, + } + try: + await v1.patch_node(node_name, body) + except ApiException as e: + return OperationResult(success=False, error=f"patch node {node_name}: {e}") + return OperationResult(success=True, message=f"node {node_name} undedicated") + async def _check_scheduling( self, sandbox_name: str, @@ -621,7 +785,11 @@ class K7Core: return legacy.get(backend, backend) async def _detect_backend(self, sandbox_name: str | None = None, namespace: str = "default") -> str: - """Detect backend with priority: deployment annotation > /etc/k7/backend > default.""" + """Detect backend with priority: deployment annotation > /etc/k7/backend > kfd. + + ``none`` in ``/etc/k7/backend`` is not a runtime — return empty so + create fails loud instead of falling through to kfd. + """ allowed = _ALLOWED_BACKENDS if sandbox_name: try: @@ -636,9 +804,12 @@ class K7Core: try: if os.path.exists("/etc/k7/backend"): with open("/etc/k7/backend") as f: - backend = self._canonicalize_backend(f.read().strip()) - if backend in allowed: - return backend + raw = f.read().strip() + if raw.lower() == "none": + return "" + backend = self._canonicalize_backend(raw) + if backend in allowed: + return backend except Exception: pass @@ -790,8 +961,8 @@ class K7Core: raise RuntimeError(f"k7 agent token {path} is empty — re-run `k7 install`") return token - async def _k7d_sandbox_node(self, sandbox_name: str, namespace: str) -> str: - """Node hosting the sandbox's Running pod (loud when there is none).""" + async def _k7d_sandbox_placement(self, sandbox_name: str, namespace: str) -> tuple[str, str]: + """Return ``(node_name, pod_name)`` for the Running sandbox pod.""" v1 = await self._get_core_v1_client() pods = await v1.list_namespaced_pod(namespace=namespace, label_selector=f"app={sandbox_name}") running = [ @@ -799,7 +970,16 @@ class K7Core: ] if not running: raise RuntimeError(f"no Running pod found for sandbox {sandbox_name} in {namespace}") - return running[0].spec.node_name + pod = running[0] + node = pod.spec.node_name if pod.spec else None + if not node: + raise RuntimeError(f"sandbox {sandbox_name} in {namespace} has no spec.nodeName yet") + return node, pod.metadata.name + + async def _k7d_sandbox_node(self, sandbox_name: str, namespace: str) -> str: + """Node hosting the sandbox's Running pod (loud when there is none).""" + node, _pod = await self._k7d_sandbox_placement(sandbox_name, namespace) + return node async def _k7d_agent_base_url(self, node_name: str) -> str: """Resolve the Ready k7-agent pod on ``node_name`` to its pod IP.""" @@ -842,14 +1022,14 @@ class K7Core: async def lookup_k7d_vm(self, sandbox_name: str, namespace: str = "default") -> dict: """Resolve a k7d sandbox to its daemon VM record, forwarding to the per-node agent when the sandbox lives on another node.""" - node = await self._k7d_sandbox_node(sandbox_name, namespace) - if node and node != self._k7d_local_node(): + node, pod_name = await self._k7d_sandbox_placement(sandbox_name, namespace) + if node != self._k7d_local_node(): base = await self._k7d_agent_base_url(node) token = self._k7d_agent_token() async with httpx.AsyncClient(timeout=60) as http: resp = await http.post( f"{base}/agent/v1/vm/lookup", - json={"name": sandbox_name, "namespace": namespace}, + json={"name": sandbox_name, "namespace": namespace, "pod_name": pod_name}, headers={"X-K7-Agent-Token": token}, ) if resp.status_code != 200: @@ -954,6 +1134,18 @@ class K7Core: return int(math.ceil(mib)) + def _kata_hypervisor_memory_mib(self, value: str) -> int: + """Kata ``default_memory`` MiB. Loud if below the shim's 256Mi floor.""" + memory_mib = self._memory_limit_to_mib(value) + if memory_mib < KATA_MIN_HYPERVISOR_MEMORY_MIB: + raise ValueError( + f"memory limit {value} is below Kata's minimum " + f"{KATA_MIN_HYPERVISOR_MEMORY_MIB}Mi " + "(io.katacontainers.config.hypervisor.default_memory). " + "Use at least 256Mi, or omit limits to use the hypervisor default." + ) + return memory_mib + def _count_playbook_tasks(self, playbook_content: str) -> int: """Count tasks in Ansible playbook.""" try: @@ -1278,13 +1470,122 @@ class K7Core: def _docker_snapshot_name(self, snapshot_name: str) -> str: return f"{snapshot_name}-docker" + async def _freeze_docker_graph(self, sandbox_name: str, namespace: str) -> OperationResult: + # docker:27.5.1-dind is alpine and has no fsfreeze. The sandbox image + # (ubuntu:24.04 in the product tests) ships util-linux; /tmp is the + # shared emptyDir. Copy into the vehicle rootfs before exec in case + # emptyDir is noexec. + stage = await self.exec_command(sandbox_name, STAGE_FSFREEZE_CMD, namespace=namespace) + if stage.exit_code != 0: + return OperationResult( + success=False, + error=( + f"could not stage fsfreeze from sandbox: {stage.stderr}. " + "Pause the sandbox (`k7 pause --snapshot`) and retry, or use the k7d backend." + ), + ) + freeze = await self.exec_command( + sandbox_name, + f"cp {FSFREEZE_STAGED} {FSFREEZE_VEHICLE} && chmod 755 {FSFREEZE_VEHICLE} && " + f"{FSFREEZE_VEHICLE} -f {KATA_GRAPH_MOUNT}", + namespace=namespace, + container=VEHICLE_CONTAINER_NAME, + ) + if freeze.exit_code != 0: + return OperationResult( + success=False, + error=( + f"fsfreeze -f {KATA_GRAPH_MOUNT} in docker-vehicle failed: {freeze.stderr}. " + "Pause the sandbox (`k7 pause --snapshot`) and retry, or use the k7d backend." + ), + ) + return OperationResult(success=True) + + async def _thaw_docker_graph(self, sandbox_name: str, namespace: str) -> OperationResult: + thaw = await self.exec_command( + sandbox_name, + f"{FSFREEZE_VEHICLE} -u {KATA_GRAPH_MOUNT}", + namespace=namespace, + container=VEHICLE_CONTAINER_NAME, + ) + if thaw.exit_code != 0: + return OperationResult( + success=False, + error=f"fsfreeze -u {KATA_GRAPH_MOUNT} in docker-vehicle failed: {thaw.stderr}", + ) + return OperationResult(success=True) + + async def _create_kata_snapshots_quiesced( + self, + sandbox_name: str, + namespace: str, + *, + root_pvc: str, + snapshot_name: str, + kind: str, + running: bool, + docker_pvc: str | None, + fail_prefix: str, + ) -> OperationResult: + """Create root (+ docker) VolumeSnapshots. + + ``sync`` the sandbox root, then ``fsfreeze`` the docker graph for + the create calls only. Thaw in ``finally`` so a failed snapshot + cannot leave the source frozen. Wait-for-ready stays outside. + """ + want_docker = docker_pvc is not None + graph_frozen = False + result = OperationResult(success=True) + try: + if running: + sync_res = await self.exec_command(sandbox_name, "sync", namespace=namespace) + if sync_res.exit_code != 0: + result = OperationResult( + success=False, + error=f"{fail_prefix} sync in source sandbox failed: {sync_res.stderr}", + ) + elif want_docker: + freeze = await self._freeze_docker_graph(sandbox_name, namespace) + if not freeze.success: + result = freeze + else: + graph_frozen = True + if result.success: + result = await self._create_volume_snapshot( + pvc_name=root_pvc, + snapshot_name=snapshot_name, + snapshot_class="longhorn", + namespace=namespace, + kind=kind, + source_sandbox=sandbox_name, + ) + if result.success and docker_pvc is not None: + docker_snap = await self._create_volume_snapshot( + pvc_name=docker_pvc, + snapshot_name=self._docker_snapshot_name(snapshot_name), + snapshot_class="longhorn", + namespace=namespace, + kind=kind, + source_sandbox=sandbox_name, + ) + if not docker_snap.success: + result = docker_snap + finally: + if graph_frozen: + thaw = await self._thaw_docker_graph(sandbox_name, namespace) + if not thaw.success: + if result.success: + result = thaw + else: + result = OperationResult(success=False, error=f"{result.error}; {thaw.error}") + return result + async def _wait_for_job(self, job_name: str, namespace: str = "default", timeout: int = 600) -> OperationResult: """Wait for a Kubernetes Job to complete.""" - await self._load_k3s_config() + batch = await self._get_batch_v1_client() start = time.time() while time.time() - start < timeout: try: - batch = client.BatchV1Api() job = await batch.read_namespaced_job(name=job_name, namespace=namespace) status = job.status if status.succeeded and status.succeeded >= 1: @@ -1624,8 +1925,7 @@ class K7Core: async def _cilium_available(self) -> bool: """Return True when the CiliumNetworkPolicy CRD is registered on the cluster.""" - await self._load_k3s_config() - api_ext = client.ApiextensionsV1Api() + api_ext = await self._get_apiextensions_v1_client() try: await api_ext.read_custom_resource_definition(name="ciliumnetworkpolicies.cilium.io") return True @@ -2573,7 +2873,7 @@ class K7Core: # cpu/memory limits. if config.limits and "memory" in config.limits and not _is_k7d_family(backend): try: - memory_mib = self._memory_limit_to_mib(config.limits["memory"]) + memory_mib = self._kata_hypervisor_memory_mib(config.limits["memory"]) except ValueError as e: return OperationResult(success=False, error=str(e)) base_annotations["io.katacontainers.config.hypervisor.default_memory"] = str(memory_mib) @@ -2662,13 +2962,26 @@ class K7Core: fs_group=65532, ) + node_selector = {f"k7.katakate.org/backend-{backend}": "true"} + hostname = getattr(config, "node_name", None) or None + tolerations = None + if hostname: + # Link is the Node's kubernetes.io/hostname label (already on + # every node), not spec.nodeName (that bypasses the scheduler). + node_selector["kubernetes.io/hostname"] = hostname + try: + tols = await self._tenant_tolerations_for_node(hostname) + except RuntimeError as e: + return OperationResult(success=False, error=str(e)) + if tols: + tolerations = tols pod_spec = client.V1PodSpec( containers=[container], runtime_class_name=runtime_class, restart_policy="Always", security_context=pod_sec_ctx, - node_selector={f"k7.katakate.org/backend-{backend}": "true"}, - node_name=getattr(config, "node_name", None) or None, + node_selector=node_selector, + tolerations=tolerations, ) if root_pvc_name and wrapper_cm_name: @@ -3152,11 +3465,17 @@ class K7Core: success=False, error=f"--snapshot is not supported when pausing a k7d sandbox: {K7D_SNAPSHOT_UNSUPPORTED}", ) - node = await self._k7d_sandbox_node(name, namespace) - if node and node != self._k7d_local_node(): - return await self._k7d_forward_vm_op( - node, "pause", {"name": name, "namespace": namespace}, timeout=120 + node, pod_name = await self._k7d_sandbox_placement(name, namespace) + if node != self._k7d_local_node(): + result = await self._k7d_forward_vm_op( + node, + "pause", + {"name": name, "namespace": namespace, "pod_name": pod_name}, + timeout=120, ) + if result.success: + await self._update_deployment_annotation(name, namespace, K7D_ANN_PAUSED, "true") + return result vm = await self._k7d_vm_for_sandbox(name, namespace) await self._k7d_request({"op": "pause_vm", "vm_id": vm["vm_id"]}) await self._update_deployment_annotation(name, namespace, K7D_ANN_PAUSED, "true") @@ -3197,11 +3516,17 @@ class K7Core: try: backend = await self._detect_backend(name, namespace) if _is_k7d_family(backend): - node = await self._k7d_sandbox_node(name, namespace) - if node and node != self._k7d_local_node(): - return await self._k7d_forward_vm_op( - node, "resume", {"name": name, "namespace": namespace}, timeout=120 + node, pod_name = await self._k7d_sandbox_placement(name, namespace) + if node != self._k7d_local_node(): + result = await self._k7d_forward_vm_op( + node, + "resume", + {"name": name, "namespace": namespace, "pod_name": pod_name}, + timeout=120, ) + if result.success: + await self._update_deployment_annotation(name, namespace, K7D_ANN_PAUSED, "false") + return result vm = await self._k7d_vm_for_sandbox(name, namespace) await self._k7d_request({"op": "resume_vm", "vm_id": vm["vm_id"]}) await self._update_deployment_annotation(name, namespace, K7D_ANN_PAUSED, "false") @@ -3225,8 +3550,15 @@ class K7Core: namespace: str = "default", ) -> OperationResult: """Shell into a sandbox pod via kubectl exec. Stays sync (interactive CLI-only).""" - v1 = asyncio.run(self._get_core_v1_client()) - pods = asyncio.run(v1.list_namespaced_pod(namespace=namespace, label_selector=f"app={sandbox_name}")) + + async def _list_pods(): + try: + v1 = await self._get_core_v1_client() + return await v1.list_namespaced_pod(namespace=namespace, label_selector=f"app={sandbox_name}") + finally: + await self.aclose() + + pods = asyncio.run(_list_pods()) if not pods.items: return OperationResult(success=False, error="Pod not found") @@ -3271,13 +3603,13 @@ class K7Core: snapshot_name=snapshot_name, fork_start=fork_start, ) + if backend == "kata-firecracker-devmapper": + return OperationResult(success=False, error=KFD_FORK_REJECT) apps_v1 = await self._get_apps_v1_client() v1 = await self._get_core_v1_client() src = await apps_v1.read_namespaced_deployment(name=source_name, namespace=namespace) src_ann = (src.metadata.annotations or {}) if src.metadata else {} - if backend == "kata-firecracker-devmapper" and src_ann.get(ANN_K7_DOCKER) == "true": - return OperationResult(success=False, error=KATA_FORK_GRAPH_REJECT) source_pvc_name = self._root_pvc_name(source_name) target_pvc_name = self._root_pvc_name(new_name) @@ -3305,34 +3637,37 @@ class K7Core: snap_name = f"{source_name}-fork-{int(time.time())}" snap_kind = SNAPSHOT_KIND_FORK auto_temp_snapshot = True - # A Longhorn VolumeSnapshot is block-level and only - # crash-consistent: guest writes still sitting in the VM's page - # cache would be missing from the fork's clone. Flush them first - # when the source is running (fail loudly if the flush fails); - # a scaled-to-0 (paused) source has no writers and needs no flush. - if (src.status.ready_replicas or 0) > 0: - sync_res = await self.exec_command(source_name, "sync", namespace=namespace) - if sync_res.exit_code != 0: - return OperationResult( - success=False, - error=f"pre-fork sync in source sandbox failed: {sync_res.stderr}", + source_docker_pvc_name = src_ann.get(ANN_DOCKER_PVC) or self._docker_pvc_name(source_name) + target_docker_pvc_name = self._docker_pvc_name(new_name) + docker_snap_name = None + source_docker_pvc = None + if src_ann.get(ANN_K7_DOCKER) == "true": + try: + source_docker_pvc = await v1.read_namespaced_persistent_volume_claim( + name=source_docker_pvc_name, namespace=namespace ) - if src_ann.get(ANN_K7_DOCKER) == "true": - vehicle_sync = await self.exec_command( - source_name, "sync", namespace=namespace, container=VEHICLE_CONTAINER_NAME - ) - if vehicle_sync.exit_code != 0: + except ApiException as e: + if e.status == 404: return OperationResult( success=False, - error=f"pre-fork sync in docker-vehicle failed: {vehicle_sync.stderr}", + error=f"Source docker PVC {source_docker_pvc_name} not found; cannot fork docker graph", ) - snap_result = await self._create_volume_snapshot( - pvc_name=source_pvc_name, + return OperationResult(success=False, error=f"Docker PVC lookup error: {e}") + docker_snap_name = self._docker_snapshot_name(snap_name) + + # Longhorn VolumeSnapshots are block-level. sync the sandbox + # root, and fsfreeze the docker graph for the create calls so a + # live overlay2 is filesystem-consistent. Thaw before waiting + # for Ready so dockerd is not frozen across CSI provisioning. + snap_result = await self._create_kata_snapshots_quiesced( + source_name, + namespace, + root_pvc=source_pvc_name, snapshot_name=snap_name, - snapshot_class="longhorn", - namespace=namespace, kind=snap_kind, - source_sandbox=source_name, + running=(src.status.ready_replicas or 0) > 0, + docker_pvc=source_docker_pvc_name if src_ann.get(ANN_K7_DOCKER) == "true" else None, + fail_prefix="pre-fork", ) if not snap_result.success: return snap_result @@ -3357,32 +3692,12 @@ class K7Core: if not pvc_clone.success: return pvc_clone - source_docker_pvc_name = src_ann.get(ANN_DOCKER_PVC) or self._docker_pvc_name(source_name) - target_docker_pvc_name = self._docker_pvc_name(new_name) - docker_snap_name = None if src_ann.get(ANN_K7_DOCKER) == "true": - try: - source_docker_pvc = await v1.read_namespaced_persistent_volume_claim( - name=source_docker_pvc_name, namespace=namespace + if not docker_snap_name or source_docker_pvc is None: + return OperationResult( + success=False, + error=f"docker snapshot missing after quiesced create for {source_name}", ) - except ApiException as e: - if e.status == 404: - return OperationResult( - success=False, - error=f"Source docker PVC {source_docker_pvc_name} not found; cannot fork docker graph", - ) - return OperationResult(success=False, error=f"Docker PVC lookup error: {e}") - docker_snap_name = self._docker_snapshot_name(snap_name) - docker_snap = await self._create_volume_snapshot( - pvc_name=source_docker_pvc_name, - snapshot_name=docker_snap_name, - snapshot_class="longhorn", - namespace=namespace, - kind=snap_kind, - source_sandbox=source_name, - ) - if not docker_snap.success: - return docker_snap docker_ready = await self._wait_for_snapshot_ready(docker_snap_name, namespace=namespace) if not docker_ready.success: return docker_ready @@ -3564,21 +3879,11 @@ class K7Core: ), ) - # The fork must run on the source VM's node (the daemon - # socket is node-local; the fork pod is pinned there too). Forward - # the WHOLE fork to that node's k7-agent when the source is remote. - node = await self._k7d_sandbox_node(source_name, namespace) - if node and node != self._k7d_local_node(): - return await self._k7d_forward_vm_op( - node, - "fork", - {"name": source_name, "new_name": new_name, "namespace": namespace, "snapshot": snapshot_name}, - timeout=600, - ) - - # Resolve the live source VM up-front so a dead/unknown source fails - # here with a clear message instead of leaving a CrashLooping fork pod. - vm = await self._k7d_vm_for_sandbox(source_name, namespace) + # Kubernetes writes stay on the caller (k7-api ClusterRole). The + # agent has no ServiceAccount token — it only looks up the live VM + # so these fork-source annotations are correct. + node, _pod = await self._k7d_sandbox_placement(source_name, namespace) + vm = await self.lookup_k7d_vm(source_name, namespace) fork_annotations = { K7D_ANN_FORK_SOURCE_CLUSTER: vm.get("cluster_id") or vm["sandbox_id"], K7D_ANN_FORK_SOURCE_VM: vm["sandbox_id"], @@ -3615,8 +3920,7 @@ class K7Core: new_dep.spec.replicas = 1 # The fork must land on the source VM's node — the k7d daemon socket # is node-local (cross-node fork is future k7d work, not built yet). - source_pod = await self._k7d_running_pod(source_name, namespace) - new_dep.spec.template.spec.node_name = source_pod.spec.node_name + new_dep.spec.template.spec.node_name = node try: await apps_v1.create_namespaced_deployment(namespace=namespace, body=new_dep) @@ -3862,49 +4166,25 @@ class K7Core: return OperationResult(success=False, error=f"Deployment lookup error: {e}") anns = (dep.metadata.annotations or {}) if dep.metadata else {} want_docker = anns.get(ANN_K7_DOCKER) == "true" + docker_pvc = (anns.get(ANN_DOCKER_PVC) or self._docker_pvc_name(sandbox_name)) if want_docker else None # Crash-consistency: guest page cache is invisible to Longhorn. - # Same flush as fork (sandbox + docker-vehicle). A paused - # (scaled-to-0) source has no writers and needs no flush. - if (dep.status.ready_replicas or 0) > 0: - sync_res = await self.exec_command(sandbox_name, "sync", namespace=namespace) - if sync_res.exit_code != 0: - return OperationResult( - success=False, - error=f"pre-snapshot sync in source sandbox failed: {sync_res.stderr}", - ) - if want_docker: - vehicle_sync = await self.exec_command( - sandbox_name, "sync", namespace=namespace, container=VEHICLE_CONTAINER_NAME - ) - if vehicle_sync.exit_code != 0: - return OperationResult( - success=False, - error=f"pre-snapshot sync in docker-vehicle failed: {vehicle_sync.stderr}", - ) - root = await self._create_volume_snapshot( - pvc_name=self._root_pvc_name(sandbox_name), + # Same quiesce as fork (sandbox sync + docker-vehicle fsfreeze). + # A paused (scaled-to-0) source has no writers and needs no freeze. + root = await self._create_kata_snapshots_quiesced( + sandbox_name, + namespace, + root_pvc=self._root_pvc_name(sandbox_name), snapshot_name=snapshot_name, - snapshot_class="longhorn", - namespace=namespace, kind=SNAPSHOT_KIND_NAMED, - source_sandbox=sandbox_name, + running=(dep.status.ready_replicas or 0) > 0, + docker_pvc=docker_pvc, + fail_prefix="pre-snapshot", ) if not root.success: return root if not want_docker: return root - docker_pvc = anns.get(ANN_DOCKER_PVC) or self._docker_pvc_name(sandbox_name) docker_snap_name = self._docker_snapshot_name(snapshot_name) - docker_snap = await self._create_volume_snapshot( - pvc_name=docker_pvc, - snapshot_name=docker_snap_name, - snapshot_class="longhorn", - namespace=namespace, - kind=SNAPSHOT_KIND_NAMED, - source_sandbox=sandbox_name, - ) - if not docker_snap.success: - return docker_snap root_ready = await self._wait_for_snapshot_ready(snapshot_name, namespace=namespace) if not root_ready.success: return root_ready @@ -4195,6 +4475,7 @@ class K7Core: namespace: str = "default", overrides: SandboxConfigOverrides | None = None, keep_snapshot: bool = True, + node_name: str | None = None, ) -> OperationResult: """Boot a brand-new sandbox from a standalone ``VolumeSnapshot``. @@ -4204,6 +4485,8 @@ class K7Core: field. Restore is **kata-qemu-longhorn only** — there is no PVC to clone from in the kata-firecracker-devmapper backend. + ``node_name`` pins the restored pod (API-key node scope stamps this). + The cloned PVC is named ``_root_pvc_name(new_sandbox_name)`` and is pre-created from the snapshot before ``create_sandbox`` runs; the idempotent ``_ensure_root_pvc`` in ``create_sandbox`` then detects @@ -4249,6 +4532,8 @@ class K7Core: if not config_result.success: return config_result config: SandboxConfig = config_result.data + if node_name: + config.node_name = node_name # Restore is kata-qemu-longhorn-only: kata-firecracker-devmapper has no PVC to # clone, so it has no snapshots either. @@ -4331,7 +4616,14 @@ class K7Core: ) async def delete_sandbox(self, name: str, namespace: str = "default") -> OperationResult: - """Delete a sandbox.""" + """Delete a sandbox's Kubernetes objects. + + Returns when Deployments, PVCs, Services, and related objects are + gone. k7d VM teardown (including the scratch docker-graph unlink) + is asynchronous via the containerd shim ``Delete`` and is **not** + waited on here — coupling API latency to shim teardown would stall + every delete. Callers that need the graph file gone must poll. + """ return await self._delete_sandbox_resources(name, namespace) async def delete_all_sandboxes(self, namespace: str = "default") -> OperationResult: diff --git a/src/k7/core/docker.py b/src/k7/core/docker.py index 2f4b81d..81e3b45 100644 --- a/src/k7/core/docker.py +++ b/src/k7/core/docker.py @@ -36,6 +36,16 @@ KATA_SOCKET_DIR = "/run/k7/docker" DOCKER_HOST_URL = "unix:///run/k7/docker/docker.sock" KATA_DAEMON_ARGS = "--host=unix:///run/k7/docker/docker.sock --tls=false --storage-driver=overlay2" GRAPH_DEVICE_PATH = "/dev/k7docker" +KATA_GRAPH_MOUNT = "/var/lib/docker" +# alpine dind has no fsfreeze; ubuntu sandbox does. Stage via the shared /tmp +# emptyDir, then exec from the vehicle's own rootfs (emptyDir may be noexec). +FSFREEZE_STAGED = "/tmp/k7-fsfreeze" +FSFREEZE_VEHICLE = "/usr/local/bin/k7-fsfreeze" +STAGE_FSFREEZE_CMD = ( + f"if [ -x /usr/sbin/fsfreeze ]; then cp /usr/sbin/fsfreeze {FSFREEZE_STAGED}; " + f"elif [ -x /sbin/fsfreeze ]; then cp /sbin/fsfreeze {FSFREEZE_STAGED}; " + "else echo 'fsfreeze not found in sandbox image (need util-linux)' >&2; exit 1; fi" +) VEHICLE_CONTAINER_NAME = "docker-vehicle" KFD_DOCKER_STORAGE_CLASS = "k7-docker-lvm" # linux/amd64 digest of docker:27.5.1-dind (Docker Hub, 2026-09). @@ -52,6 +62,11 @@ ANN_DOCKER_PVC = "k7.katakate.org/docker-pvc-name" KATA_FORK_GRAPH_REJECT = ( "k7 fork of a kfd --docker sandbox is not supported: the docker graph cannot be cloned (ephemeral LV)" ) +KFD_FORK_REJECT = ( + "k7 fork is not supported on the kfd (kata-firecracker-devmapper) backend: " + "there is no persistent PVC to clone (and a --docker graph cannot be cloned — ephemeral LV). " + "Use kql for disk-level fork, or k7d / k7d-fc for warm VM fork." +) # Playbook records the installed k7d version here. Guest dockerd # and k7-fc shipped in public k7d 0.6.0; a node whose recorded version diff --git a/src/k7/core/models.py b/src/k7/core/models.py index faed98b..b75f658 100644 --- a/src/k7/core/models.py +++ b/src/k7/core/models.py @@ -34,8 +34,9 @@ class SandboxConfig: container_non_root: bool = False cap_drop: list[str] | None = None # default behavior handled in core: drop ALL cap_add: list[str] | None = None - # Optional explicit node placement (sets pod's node_name). Used by tests - # that need to inspect host-side state for a sandbox they just created. + # Optional node pin. Applied as nodeSelector kubernetes.io/hostname + # (the label every Node already has) — not spec.nodeName. A tenant + # taint on that node is the exclusive lock; see k7 nodes dedicate. node_name: str | None = None # Ingress is denied by default and opt-in per sandbox. # ``ingress_ports`` are TCP ports to open; ``None``/``[]`` denies everything. diff --git a/src/k7/deploy/inventory.ini.example b/src/k7/deploy/inventory.ini.example index f5ad3da..e54d44f 100644 --- a/src/k7/deploy/inventory.ini.example +++ b/src/k7/deploy/inventory.ini.example @@ -1,14 +1,27 @@ ; Example multi-node inventory for `k7 install -i inventory.ini`. ; -; Each host declares which backend(s) it supports via `k7_backends` (comma-separated). -; A node may support one or both of: -; - kata-qemu-longhorn (uses Kata QEMU + overlayfs + Longhorn for sandbox storage) -; - kata-firecracker-devmapper (uses Kata Firecracker + LVM thin-pool snapshotter) +; The left-hand name (k7-node-01) MUST match the machine hostname +; (`hostnamectl` / installimage HOSTNAME). K3s names the Node after that +; hostname and stamps kubernetes.io/hostname= automatically. +; This file does not write that label. `k7_backends` tells the playbook +; which k7.katakate.org/backend-* labels to stamp after the node joins. +; Tenant isolation is later: `k7 nodes dedicate NAME --tenant ID`. +; Copy NAME from `k7 nodes list` / `kubectl get nodes` into `--node`. +; +; `k7_backends` is required on every host (or inherited from group vars). +; Empty/omitted is an error, not "no runtimes". Use `none` for a node that +; must not run sandboxes (typical [k7_servers] in a cluster that has agents). +; Allowed: kfd, kql, k7d, k7d-fc, none (comma-separated except none alone). +; +; A node may list one or more of: +; - kata-qemu-longhorn / kql (Kata QEMU + overlayfs + Longhorn) +; - kata-firecracker-devmapper / kfd (Kata Firecracker + LVM thin-pool; spare disk) +; - k7d / k7d-fc ; ; Per-host vars: ; ansible_host : SSH target IP ; ansible_user : SSH user (defaults to root) -; k7_backends : comma-separated backend list +; k7_backends : comma-separated backend list, or none ; k7_devmapper_disk : block device for FD thin-pool (e.g. /dev/nvme1n1) ; longhorn_extra_disk : extra mount path to register with Longhorn ; kata_thinpool_pv_size : LVM PV size for the kfd thin-pool (default 100G) @@ -41,6 +54,10 @@ longhorn_replicas=2 ; tolerance — for a 2-node cluster, put the second node in [k7_agents]. ; The first host listed in [k7_servers] is the cluster-init server. ; +; Every [k7_servers] host is a K3s master: root there is cluster-admin +; (k3s.yaml + join token + etcd). Put tenant sandboxes on [k7_agents] +; and pin/dedicate those agent names. Do not dedicate the first master. +; ; Do NOT pass --backend alongside -i: per-host `k7_backends` in the ; inventory is authoritative (an explicit --backend overrides it). ; @@ -56,16 +73,16 @@ longhorn_replicas=2 ; 2-node, ALL backends, from the PPA CLI (no --ha — that needs 3+ servers). ; After TWO_DISK=1 so each box has a spare raw NVMe: ; -; # PPA is still 0.2.2; prefer the GitHub 0.3.0 .deb, then: -; git clone --branch v0.3.0 https://github.com/Katakate/k7.git && cd k7 +; # PPA and GitHub .deb are 0.4.0: +; git clone --branch v0.4.0 https://github.com/Katakate/k7.git && cd k7 ; # run from that checkout so the playbook can build k7-api:local -; k7 install -i inventory.ini --k7d-version 0.6.0 +; k7 install -i inventory.ini --k7d-version 0.7.0 ; ; [k7_servers] -; k7-node-01 ansible_host=203.0.113.10 k7_backends=kfd,kql,k7d +; k7-node-01 ansible_host=203.0.113.10 k7_backends=kfd,kql,k7d,k7d-fc ; ; [k7_agents] -; k7-node-02 ansible_host=203.0.113.11 k7_backends=kfd,kql,k7d +; k7-node-02 ansible_host=203.0.113.11 k7_backends=kfd,kql,k7d,k7d-fc ; ; [k7_cluster:children] ; k7_servers @@ -77,14 +94,38 @@ longhorn_replicas=2 ; ansible_ssh_common_args=-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ; longhorn_replicas=2 ; -; 3-node HA with ALL backends on every node. ONE command: +; 3 small masters (no sandboxes) + agents. `none` is required on servers — +; omitting k7_backends is an error, not an implicit empty set. ; -; k7 install -i inventory.ini --ha --k7d-version 0.6.0 +; k7 install -i inventory.ini --ha --k7d-version 0.7.0 ; ; [k7_servers] -; k7-node-01 ansible_host=203.0.113.10 k7_backends=kfd,kql,k7d -; k7-node-02 ansible_host=203.0.113.11 k7_backends=kfd,kql,k7d -; k7-node-03 ansible_host=203.0.113.12 k7_backends=kfd,kql,k7d +; k7-node-01 ansible_host=203.0.113.10 +; k7-node-02 ansible_host=203.0.113.11 +; k7-node-03 ansible_host=203.0.113.12 +; +; [k7_servers:vars] +; k7_backends=none +; +; [k7_agents] +; k7-node-04 ansible_host=203.0.113.13 +; k7-node-05 ansible_host=203.0.113.14 +; +; [k7_agents:vars] +; k7_backends=kql,k7d +; +; [k7_cluster:children] +; k7_servers +; k7_agents +; +; 3-node HA with ALL backends on every node (soak / demo). ONE command: +; +; k7 install -i inventory.ini --ha --k7d-version 0.7.0 +; +; [k7_servers] +; k7-node-01 ansible_host=203.0.113.10 k7_backends=kfd,kql,k7d,k7d-fc +; k7-node-02 ansible_host=203.0.113.11 k7_backends=kfd,kql,k7d,k7d-fc +; k7-node-03 ansible_host=203.0.113.12 k7_backends=kfd,kql,k7d,k7d-fc ; ; [k7_cluster:children] ; k7_servers diff --git a/src/k7/deploy/inventory.local.ini b/src/k7/deploy/inventory.local.ini index b239fd0..6604c5a 100644 --- a/src/k7/deploy/inventory.local.ini +++ b/src/k7/deploy/inventory.local.ini @@ -1,6 +1,7 @@ -; Single-node localhost inventory used by `k7 install` when no inventory is given. +; Unused by `k7 install` (the CLI builds inventory from --backend). +; If you pass -i this file, k7_backends is required — empty is not none. [k7_servers] -localhost ansible_connection=local ansible_user=root +localhost ansible_connection=local ansible_user=root k7_backends=kql [k7_agents] diff --git a/src/k7/deploy/k7-install-node.yaml b/src/k7/deploy/k7-install-node.yaml index 1fd7ce7..054b7e1 100644 --- a/src/k7/deploy/k7-install-node.yaml +++ b/src/k7/deploy/k7-install-node.yaml @@ -3,7 +3,8 @@ become: true vars: target_user: "{{ ansible_user | default('root') }}" - k7_backend_default: "kata-firecracker-devmapper" + # No implicit backend default — every host must set k7_backends (or inherit + # group vars). Use k7_backends=none for a node with no sandbox runtime. snapshotter_version: "v8.4.0" kata_version: "3.24.0" cni_plugins_version: "v1.9.0" @@ -13,7 +14,7 @@ # k7d backend: release artifact source. Override # `k7d_artifact_local_path` (path on the Ansible controller) to install # from a locally built tarball instead of the GitHub release. - k7d_version: "0.6.0" + k7d_version: "0.7.0" k7d_artifact_url: "https://github.com/katakate/k7d/releases/download/v{{ k7d_version }}/k7d-v{{ k7d_version }}-x86_64-linux.tar.gz" k7d_artifact_local_path: "" # Sparse XFS (reflink=1) loopback image backing /var/lib/k7d/disks — @@ -71,28 +72,45 @@ # ────────────────────────────────────────────────────────────────────── # Compute per-host backend set + cluster role # ────────────────────────────────────────────────────────────────────── + - name: Require k7_backends on every host + ansible.builtin.fail: + msg: >- + k7_backends is not set on {{ inventory_hostname }} (empty/omitted is not + 'none'). Set it on the host or a group vars section ([k7_servers:vars] / + [k7_agents:vars]). Allowed: kfd, kql, k7d, k7d-fc, or none (no sandbox + runtime on this node). + when: k7_backends is not defined or (k7_backends | string | trim | length) == 0 + tags: ['always'] + - name: Compute per-host backend list (raw) ansible.builtin.set_fact: - k7_backends_raw: >- - {{ - (k7_backends.split(',') | map('trim') | reject('equalto','') | list) - if (k7_backends is defined and k7_backends | length > 0) - else ([k7_backend] if (k7_backend is defined and k7_backend | length > 0) - else [k7_backend_default]) - }} + k7_backends_raw: "{{ (k7_backends | string).split(',') | map('trim') | reject('equalto', '') | list }}" + tags: ['always'] + + - name: Reject none mixed with other backends + ansible.builtin.fail: + msg: >- + k7_backends={{ k7_backends }} on {{ inventory_hostname }} mixes 'none' + with other backends. Use only none, or a list of runtimes. + when: + - (k7_backends_raw | map('lower') | list | select('equalto', 'none') | list | length) > 0 + - (k7_backends_raw | length) > 1 tags: ['always'] - name: Normalize deprecated backend aliases ansible.builtin.set_fact: k7_backends_list: >- {{ - k7_backends_raw - | map('regex_replace', '^(fd|kfd|firecracker-devmapper)$', 'kata-firecracker-devmapper') - | map('regex_replace', '^(ql|kql|qemu-longhorn)$', 'kata-qemu-longhorn') - | map('regex_replace', '^k7$', 'k7d') - | map('regex_replace', '^k7-fc$', 'k7d-fc') - | unique - | list + [] if (k7_backends_raw | map('lower') | list == ['none']) + else ( + k7_backends_raw + | map('regex_replace', '^(fd|kfd|firecracker-devmapper)$', 'kata-firecracker-devmapper') + | map('regex_replace', '^(ql|kql|qemu-longhorn)$', 'kata-qemu-longhorn') + | map('regex_replace', '^k7$', 'k7d') + | map('regex_replace', '^k7-fc$', 'k7d-fc') + | unique + | list + ) }} tags: ['always'] @@ -102,12 +120,12 @@ Deprecated backend name(s) in inventory were remapped: {{ k7_backends_raw }} → {{ k7_backends_list }}. Prefer kata-firecracker-devmapper (kfd) / kata-qemu-longhorn (kql). - when: k7_backends_raw != k7_backends_list + when: k7_backends_list | length > 0 and k7_backends_raw != k7_backends_list tags: ['always'] - name: Validate backends list ansible.builtin.fail: - msg: "Unknown backend(s) in k7_backends={{ k7_backends_list }}; allowed: kata-firecracker-devmapper, kata-qemu-longhorn, k7d, k7d-fc" + msg: "Unknown backend(s) in k7_backends={{ k7_backends_list }}; allowed: kata-firecracker-devmapper, kata-qemu-longhorn, k7d, k7d-fc, or none" when: k7_backends_list | difference(['kata-firecracker-devmapper', 'kata-qemu-longhorn', 'k7d', 'k7d-fc']) | length > 0 tags: ['always'] @@ -2139,7 +2157,7 @@ - name: Store K7 backend configuration (primary backend = first in list) ansible.builtin.copy: dest: /etc/k7/backend - content: "{{ k7_backends_list[0] }}" + content: "{{ k7_backends_list[0] if k7_backends_list | length > 0 else 'none' }}" mode: '0644' tags: ['k7', 'config'] diff --git a/src/k7/deploy/k7d-fc/pins.env b/src/k7/deploy/k7d-fc/pins.env index e57df07..409a293 100644 --- a/src/k7/deploy/k7d-fc/pins.env +++ b/src/k7/deploy/k7d-fc/pins.env @@ -1,7 +1,9 @@ # Vendored from k7d guest/fc/pins.env. # Bump together with k7d's pin; sha mismatch is fatal. -FIRECRACKER_VERSION=v1.16.1 -FIRECRACKER_RELEASE_URL=https://github.com/firecracker-microvm/firecracker/releases/download/v1.16.1/firecracker-v1.16.1-x86_64.tgz -FIRECRACKER_TGZ_SHA256=382a02a869e4d6d5cb14c40577f9545e8458021ea8b0b2d3fc10ec14d9c242e6 -FIRECRACKER_SHA256=2fd0171309af7e24cf8dafc8a6f921c1434c49b5f9349bb996b7ed0a4deb8aa7 -JAILER_SHA256=1f3a0c1fe86212d0001819bfe0819071c01208b3ccc9398c3b3bc1b84cf21edd +# v1.16.2 over v1.16.1: upstream #6100 — earlier 1.16.x gated vsock RX +# after a bare pause → resume, so no exec answered (CHALLENGES #17). +FIRECRACKER_VERSION=v1.16.2 +FIRECRACKER_RELEASE_URL=https://github.com/firecracker-microvm/firecracker/releases/download/v1.16.2/firecracker-v1.16.2-x86_64.tgz +FIRECRACKER_TGZ_SHA256=32e3cdcd4081f91fe2b024a266f57dcb3b4e5fec5033e0cb22467ad7f7820bda +FIRECRACKER_SHA256=8227875ceda44177a4d501052dae4a2f9d7837362f5399f02cf0748e68418377 +JAILER_SHA256=99832d2270741af67ffacd46da716b4b016338541af5a742907ab965b2e9ac9e diff --git a/src/k7/deploy/manifests/k7-api/agent-daemonset.yaml b/src/k7/deploy/manifests/k7-api/agent-daemonset.yaml index cf38398..75717de 100644 --- a/src/k7/deploy/manifests/k7-api/agent-daemonset.yaml +++ b/src/k7/deploy/manifests/k7-api/agent-daemonset.yaml @@ -14,10 +14,10 @@ spec: labels: app: k7-agent spec: - # Reuses the k7-api ServiceAccount/RBAC: the agent runs the same - # K7Core code (pod lookup, deployment annotate/create for - # pause/resume/fork). - serviceAccountName: k7-api + # Dedicated SA with no token and no ClusterRole — VM ops are crictl + # + k7d.sock only. Kubernetes writes stay on k7-api. + serviceAccountName: k7-agent + automountServiceAccountToken: false tolerations: - key: node-role.kubernetes.io/control-plane operator: Exists diff --git a/src/k7/deploy/manifests/k7-api/cilium/agent-networkpolicy.yaml b/src/k7/deploy/manifests/k7-api/cilium/agent-networkpolicy.yaml index 70484a5..eb184c4 100644 --- a/src/k7/deploy/manifests/k7-api/cilium/agent-networkpolicy.yaml +++ b/src/k7/deploy/manifests/k7-api/cilium/agent-networkpolicy.yaml @@ -1,9 +1,12 @@ # Restrict pod-originated ingress to the k7-agent DaemonSet to -# the k7-api pod only. `host` / `remote-node` entities stay allowed so -# kubelet health probes and root CLI usage on cluster nodes keep working — -# those callers still need the shared token (/etc/k7/agent_token, root -# 0600), which is the actual authentication. Sandbox pods and any other -# workload pods are denied at the network layer. +# the k7-api pod only. `host` stays allowed so kubelet health probes +# (and root CLI on THAT node) keep working — those callers still need +# the agent token (/etc/k7/agent_token, root 0600). +# +# `remote-node` is intentionally omitted: a compromised agent must not +# reach k7-agent pods on other workers. k7-api talks to agents on the +# pod network (`fromEndpoints: app=k7-api`). `k7 --core` VM ops for a +# sandbox on another node go through k7-api, not host-to-agent. apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: @@ -19,4 +22,3 @@ spec: app: k7-api - fromEntities: - host - - remote-node diff --git a/src/k7/deploy/manifests/k7-api/serviceaccount.yaml b/src/k7/deploy/manifests/k7-api/serviceaccount.yaml index c7fcf70..9cd75b5 100644 --- a/src/k7/deploy/manifests/k7-api/serviceaccount.yaml +++ b/src/k7/deploy/manifests/k7-api/serviceaccount.yaml @@ -5,3 +5,14 @@ metadata: namespace: kube-system labels: app: k7-api +--- +# No ClusterRoleBinding. automount is false on the DaemonSet too — root on +# an agent must not steal k7-api RBAC. +apiVersion: v1 +kind: ServiceAccount +metadata: + name: k7-agent + namespace: kube-system + labels: + app: k7-agent +automountServiceAccountToken: false diff --git a/src/k7_sdk/__init__.py b/src/k7_sdk/__init__.py index ac0e758..c576367 100644 --- a/src/k7_sdk/__init__.py +++ b/src/k7_sdk/__init__.py @@ -8,4 +8,4 @@ __all__ = [ "SandboxProxy", ] -__version__ = "0.3.1" +__version__ = "0.4.0" diff --git a/src/k7_sdk/client.py b/src/k7_sdk/client.py index c06b802..25329f5 100644 --- a/src/k7_sdk/client.py +++ b/src/k7_sdk/client.py @@ -82,6 +82,7 @@ class Client: "ingress_ports": [8000], "ingress_from": ["sandbox:client-a"], # default: same-namespace sandboxes # "expose_ports": [8000], # NodePort outside the cluster; needs the same ingress_ports + # "node_name": "k7-node-01", # pin; a node-scoped API key also stamps/rejects this }) """ response = self.session.post(f"{self.base_url}/api/v1/sandboxes", json=sandbox_config) diff --git a/tests/integration/test_api.py b/tests/integration/test_api.py index 3064121..3393e05 100644 --- a/tests/integration/test_api.py +++ b/tests/integration/test_api.py @@ -7,6 +7,7 @@ ServiceAccount RBAC, and in-cluster config all work end-to-end. import hashlib import json +import os import subprocess import time from pathlib import Path @@ -93,7 +94,12 @@ def _api_pod_ready() -> bool: return result.returncode == 0 and result.stdout.strip() not in ("", "0") -def _generate_test_api_key(*, name: str = "integration-test", namespaces: list[str] | None = None) -> str: +def _generate_test_api_key( + *, + name: str = "integration-test", + namespaces: list[str] | None = None, + nodes: list[str] | None = None, +) -> str: """Write a test API key directly into the keys file, return the raw token.""" import os import secrets as _secrets @@ -113,6 +119,8 @@ def _generate_test_api_key(*, name: str = "integration-test", namespaces: list[s } if namespaces: entry["namespaces"] = list(namespaces) + if nodes: + entry["nodes"] = list(nodes) keys[key_hash] = entry K7_API_KEYS_FILE.parent.mkdir(parents=True, exist_ok=True) K7_API_KEYS_FILE.write_text(json.dumps(keys, indent=2)) @@ -853,3 +861,67 @@ class TestApiNamespaceAuthz: assert isinstance(data, dict) and data, f"expected per-node map, got {data}" finally: _revoke_test_api_key(scoped_name) + + +class TestApiNodeAuthz: + """Node-scoped API keys may only place sandboxes on listed nodes.""" + + def test_single_node_key_auto_pins_and_rejects_other( + self, + api_base_url: str, + test_namespace: str, + cleanup_sandbox, + ): + local = os.uname().nodename + scoped_name = "integ-scoped-node" + scoped = _generate_test_api_key(name=scoped_name, nodes=[local]) + scoped_headers = {"X-API-Key": scoped} + name = "node-authz-sb" + cleanup_sandbox(name, test_namespace) + try: + r = httpx.post( + f"{api_base_url}/api/v1/sandboxes", + json={"name": name, "image": "alpine:3.21", "namespace": test_namespace}, + headers=scoped_headers, + timeout=180, + ) + assert r.status_code == 201, r.text + _wait_for_sandbox_ready(api_base_url, scoped_headers, name, test_namespace) + got = httpx.get( + f"{api_base_url}/api/v1/sandboxes/{name}", + params={"namespace": test_namespace}, + headers=scoped_headers, + timeout=10, + ) + assert got.status_code == 200, got.text + assert got.json()["data"].get("node") == local + + denied = httpx.post( + f"{api_base_url}/api/v1/sandboxes", + json={ + "name": f"{name}-x", + "image": "alpine:3.21", + "namespace": test_namespace, + "node_name": "not-a-k7-node", + }, + headers=scoped_headers, + timeout=15, + ) + assert denied.status_code == 403, denied.text + assert "not-a-k7-node" in denied.json()["error"]["message"] + finally: + _revoke_test_api_key(scoped_name) + + def test_node_scoped_key_nodes_storage_forbidden(self, api_base_url: str): + scoped_name = "integ-scoped-nodes-only" + scoped = _generate_test_api_key(name=scoped_name, nodes=[os.uname().nodename]) + try: + scoped_r = httpx.get( + f"{api_base_url}/api/v1/nodes/storage", + headers={"X-API-Key": scoped}, + timeout=30, + ) + assert scoped_r.status_code == 403, scoped_r.text + assert "cluster-wide node" in scoped_r.json()["error"]["message"] + finally: + _revoke_test_api_key(scoped_name) diff --git a/tests/integration/test_docker.py b/tests/integration/test_docker.py index 654e705..fbc5f5c 100644 --- a/tests/integration/test_docker.py +++ b/tests/integration/test_docker.py @@ -145,6 +145,22 @@ def _k7d_docker_disk_names() -> set[str]: return {p.name for p in _K7D_DISKS.iterdir() if "docker" in p.name} +async def _wait_k7d_docker_disks_gone(before: set[str], timeout: float = 60.0) -> None: + """k7 delete returns when K8s objects are gone; k7d unlinks the graph later. + + ``delete_sandbox`` does not wait for VM teardown (API latency must not + couple to shim Delete). Poll here; a leftover after ``timeout`` is a leak. + """ + started = time.time() + leftover: set[str] = set() + while time.time() - started < timeout: + leftover = _k7d_docker_disk_names() - before + if not leftover: + return + await asyncio.sleep(1) + raise AssertionError(f"leaked k7d docker volume images after {time.time() - started:.1f}s: {leftover}") + + def _pod_annotations(name: str, namespace: str) -> dict[str, str]: result = subprocess.run( [ @@ -237,9 +253,7 @@ class TestDockerK7d: assert capbnd != "000001ffffffffff", f"sandbox CapBnd is full: {capbnd}" finally: await k7_core.delete_sandbox(name, namespace=test_namespace) - await asyncio.sleep(3) - leftover = _k7d_docker_disk_names() - before_disks - assert not leftover, f"leaked k7d docker volume images: {leftover}" + await _wait_k7d_docker_disks_gone(before_disks) async def test_docker_build_and_compose(self, k7_core: K7Core, test_namespace: str): """docker build + compose of a one-service fixture.""" @@ -582,7 +596,10 @@ class TestDockerKQL(_DockerKataMixin): await _sh(k7_core, source, test_namespace, "docker pull alpine:3.21") fork = await k7_core.fork_sandbox(source, fork_name, namespace=test_namespace) assert fork.success, fork.error - _wait_all_containers_ready(fork_name, test_namespace, timeout=240) + # Same 600s bound as test_qemu.test_fork_clones_data: on HA + # longhorn_replicas=3 the cloned volumes hydrate before first + # attach ("not ready for workloads"). 240s is a false timeout. + _wait_all_containers_ready(fork_name, test_namespace, timeout=600) await _wait_docker_ready(k7_core, fork_name, test_namespace) has = await _sh( k7_core, fork_name, test_namespace, "docker image inspect alpine:3.21 >/dev/null && echo yes" diff --git a/tests/integration/test_firecracker.py b/tests/integration/test_firecracker.py index 5806226..fbb59b8 100644 --- a/tests/integration/test_firecracker.py +++ b/tests/integration/test_firecracker.py @@ -66,11 +66,13 @@ def _wait_for_pod_running(name: str, namespace: str, timeout: int = 120) -> None def _get_live_firecracker_pids() -> list[str]: - """Return PIDs of firecracker processes whose root fs is still live. + """Return PIDs of live jailed Firecracker VMs. - Scans /proc directly instead of relying on pgrep. Skips stale - orphan processes whose chroot has been cleaned up (readlink shows - ``(deleted)``). + Kata's binary is ``firecracker-v1.16.1``; Linux truncates ``comm`` to + 15 characters (``firecracker-v1.``). Match by prefix, not equality. + The jailer pivot-roots in a private mount ns, so ``/proc//root`` + reads as ``/`` — skip ``(deleted)`` orphans and require ``vmlinux`` + + ``rootfs`` in the jail. """ pids = [] for entry in os.listdir("/proc"): @@ -78,10 +80,13 @@ def _get_live_firecracker_pids() -> list[str]: continue try: with open(f"/proc/{entry}/comm") as f: - if f.read().strip() != "firecracker": - continue - root_link = os.readlink(f"/proc/{entry}/root") - if "(deleted)" in root_link: + comm = f.read().strip() + if comm != "firecracker" and not comm.startswith("firecracker-"): + continue + if "(deleted)" in os.readlink(f"/proc/{entry}/root"): + continue + fc_root = f"/proc/{entry}/root" + if not os.path.isfile(f"{fc_root}/vmlinux") or not os.path.isfile(f"{fc_root}/rootfs"): continue pids.append(entry) except (OSError, PermissionError): @@ -215,8 +220,9 @@ class TestFirecrackerBackend: f"Firecracker PID {pid} can see /boot — host root filesystem is exposed. Jailer not active." ) - assert os.path.isfile(f"{fc_root}/firecracker"), ( - f"Firecracker PID {pid} chroot missing firecracker binary" + fc_bins = [e for e in os.listdir(fc_root) if e.startswith("firecracker")] + assert fc_bins, ( + f"Firecracker PID {pid} chroot missing firecracker* binary; entries={sorted(os.listdir(fc_root))}" ) assert os.path.isfile(f"{fc_root}/vmlinux"), f"Firecracker PID {pid} chroot missing vmlinux kernel" assert os.path.isfile(f"{fc_root}/rootfs"), f"Firecracker PID {pid} chroot missing rootfs image" diff --git a/tests/integration/test_pause_resume.py b/tests/integration/test_pause_resume.py index 9656c19..4fe38db 100644 --- a/tests/integration/test_pause_resume.py +++ b/tests/integration/test_pause_resume.py @@ -10,6 +10,7 @@ We invoke ``k7`` via ``src/k7/cli/dev.sh`` so we don't depend on a packaged binary being installed on the node. """ +import asyncio import subprocess import time from pathlib import Path @@ -106,7 +107,7 @@ def k7_core() -> K7Core: @pytest.fixture() def ql_sandbox(k7_core: K7Core, test_namespace: str): """Provision a fresh kata-qemu-longhorn sandbox; yield its name; tear down.""" - name = f"pause-spec10c-{uuid4().hex[:6]}" + name = f"pause-qemu-{uuid4().hex[:6]}" import asyncio cfg = SandboxConfig( @@ -225,6 +226,55 @@ class TestPauseSnapshotBugRegression: assert sandbox_snaps == [], f"unexpected snapshot(s) created for {ql_sandbox}: {sandbox_snaps}" +def _runtimeclass_present(name: str) -> bool: + out = subprocess.run([_K3S, "kubectl", "get", "runtimeclass", name], capture_output=True, text=True, check=False) + return out.returncode == 0 + + +class TestPauseResumeExecAnswers: + """``pause`` → ``resume`` → ``exec_command("echo hi")`` answers, on the + daemon-socket backends. + + CHALLENGES #17: on ``k7d-fc`` the exec after resume hung — kubelet's + probe timed out and ``_wait_exec`` never returned. Root cause was + in Firecracker's vsock device (k7d CHALLENGES, jail-integrity fix); the k7d + tarball this pins carries the fix. ``k7d`` is the control. + """ + + _EXEC_BUDGET_S = 15.0 + + @pytest.mark.parametrize(("backend", "runtime_class"), [("k7d", "k7"), ("k7d-fc", "k7-fc")]) + async def test_pause_resume_exec_answers( + self, k7_core: K7Core, test_namespace: str, backend: str, runtime_class: str + ): + if not _runtimeclass_present(runtime_class): + pytest.skip(f"RuntimeClass {runtime_class} not registered") + name = f"pause-exec-{backend}-{uuid4().hex[:6]}" + cfg = SandboxConfig(name=name, image="alpine:3.20", namespace=test_namespace, backend=backend) + result = await k7_core.create_sandbox(cfg) + assert result.success, f"{backend}: create failed: {result.error}" + try: + _wait_pod_ready(name, test_namespace) + before = await k7_core.exec_command(name, "echo hi", namespace=test_namespace) + assert "hi" in (before.stdout or ""), f"{backend}: exec before pause: {before!r}" + + paused = await k7_core.pause_sandbox(name, namespace=test_namespace) + assert paused.success, f"{backend}: pause failed: {paused.error}" + await asyncio.sleep(3) + + resumed = await k7_core.resume_sandbox(name, namespace=test_namespace) + assert resumed.success, f"{backend}: resume failed: {resumed.error}" + t0 = time.monotonic() + # ``exec_command`` has no deadline of its own; the #17 symptom + # was a hang, so a regression must fail here, not park pytest. + after = await asyncio.wait_for(k7_core.exec_command(name, "echo hi", namespace=test_namespace), timeout=60) + elapsed = time.monotonic() - t0 + assert "hi" in (after.stdout or ""), f"{backend}: exec after resume: {after!r}" + assert elapsed < self._EXEC_BUDGET_S, f"{backend}: resume → exec took {elapsed:.2f}s" + finally: + await k7_core.delete_sandbox(name, namespace=test_namespace) + + class TestRemovedPauseFlags: """``--pvc`` and ``--snapshot-class`` were removed from ``k7 pause``.""" diff --git a/tests/integration/test_sandbox_ingress.py b/tests/integration/test_sandbox_ingress.py index 7d6b573..dcded3c 100644 --- a/tests/integration/test_sandbox_ingress.py +++ b/tests/integration/test_sandbox_ingress.py @@ -7,6 +7,7 @@ ingress no matter how locked down its source was. import asyncio import ipaddress +import os import subprocess import time @@ -19,6 +20,7 @@ from k7.core.models import OperationResult, SandboxConfig pytestmark = pytest.mark.integration _K3S = "/usr/local/bin/k3s" +_LOCAL_NODE = os.uname().nodename def _cilium_crd_present() -> bool: @@ -449,6 +451,10 @@ class TestSandboxExpose: ingress_ports=[8000], ingress_from=sources, expose_ports=[8000], + # Pin to the suite node. ``externalTrafficPolicy: Local`` only + # answers on the pod's node; in-cluster curl of another node's + # NodePort is intercepted by Cilium socket-LB and times out. + node_name=_LOCAL_NODE, ) result = await k7_core.create_sandbox(cfg) assert result.success, f"create {name} failed: {result.error}" @@ -458,6 +464,7 @@ class TestSandboxExpose: return endpoints[0]["url"] async def test_expose_is_reachable_and_cleaned_up(self, k7_core: K7Core, test_namespace: str): + """NodePort answers on the suite node; Local + Cilium is not Cluster.""" name = "integ-exp-open" try: url = await self._exposed(k7_core, test_namespace, name, ["cidr:0.0.0.0/0"]) diff --git a/tests/unit/test_agent.py b/tests/unit/test_agent.py new file mode 100644 index 0000000..b42af36 --- /dev/null +++ b/tests/unit/test_agent.py @@ -0,0 +1,61 @@ +"""k7-agent is node-local only: no Kubernetes writes, fork refused.""" + +from pathlib import Path +from unittest.mock import AsyncMock, patch + +import httpx + +from k7.api import agent as agent_mod +from k7.api.agent import app + +TOKEN = "agent-secret" + + +def _client(tmp_path: Path, monkeypatch): + token_file = tmp_path / "agent_token" + token_file.write_text(TOKEN) + monkeypatch.setattr(agent_mod, "AGENT_TOKEN_FILE", str(token_file)) + return httpx.ASGITransport(app=app) + + +async def test_fork_refused(tmp_path: Path, monkeypatch): + transport = _client(tmp_path, monkeypatch) + async with httpx.AsyncClient(transport=transport, base_url="http://agent") as client: + r = await client.post( + "/agent/v1/vm/fork", + json={"name": "src", "new_name": "dst"}, + headers={"X-K7-Agent-Token": TOKEN}, + ) + assert r.status_code == 400 + assert "control-plane" in r.json()["detail"] + + +async def test_pause_is_crictl_and_k7d_only(tmp_path: Path, monkeypatch): + transport = _client(tmp_path, monkeypatch) + vm = {"vm_id": "vm-1", "status": "sandbox_found", "sandbox_id": "cri"} + with ( + patch("k7.api.agent._lookup_local_vm", new=AsyncMock(return_value=vm)), + patch("k7.api.agent.K7Core") as core_cls, + ): + core_cls.return_value._k7d_request = AsyncMock(return_value={"status": "ok"}) + async with httpx.AsyncClient(transport=transport, base_url="http://agent") as client: + r = await client.post( + "/agent/v1/vm/pause", + json={"name": "sb", "namespace": "ns", "pod_name": "sb-pod"}, + headers={"X-K7-Agent-Token": TOKEN}, + ) + assert r.status_code == 200, r.text + assert r.json()["success"] is True + core_cls.return_value._k7d_request.assert_awaited_once_with({"op": "pause_vm", "vm_id": "vm-1"}) + + +async def test_pause_requires_pod_name(tmp_path: Path, monkeypatch): + transport = _client(tmp_path, monkeypatch) + async with httpx.AsyncClient(transport=transport, base_url="http://agent") as client: + r = await client.post( + "/agent/v1/vm/pause", + json={"name": "sb"}, + headers={"X-K7-Agent-Token": TOKEN}, + ) + assert r.status_code == 400 + assert "pod_name" in r.json()["detail"] diff --git a/tests/unit/test_agent_manifests.py b/tests/unit/test_agent_manifests.py new file mode 100644 index 0000000..9cb74f8 --- /dev/null +++ b/tests/unit/test_agent_manifests.py @@ -0,0 +1,29 @@ +"""Agent must not inherit k7-api ClusterRole or accept remote-node ingress.""" + +from pathlib import Path + +MANIFESTS = Path(__file__).resolve().parents[2] / "src/k7/deploy/manifests/k7-api" + + +def test_agent_daemonset_has_no_api_rbac(): + ds = (MANIFESTS / "agent-daemonset.yaml").read_text() + assert "serviceAccountName: k7-agent" in ds + assert "automountServiceAccountToken: false" in ds + assert "serviceAccountName: k7-api" not in ds + + +def test_agent_sa_has_no_clusterrolebinding(): + binding = (MANIFESTS / "clusterrolebinding.yaml").read_text() + assert "name: k7-api" in binding + assert "k7-agent" not in binding + sa = (MANIFESTS / "serviceaccount.yaml").read_text() + assert "name: k7-agent" in sa + assert "automountServiceAccountToken: false" in sa + + +def test_agent_cnp_denies_remote_node(): + cnp = (MANIFESTS / "cilium/agent-networkpolicy.yaml").read_text() + spec = cnp.split("spec:", 1)[1] + assert "app: k7-api" in spec + assert "- host" in spec + assert "remote-node" not in spec diff --git a/tests/unit/test_api_auth.py b/tests/unit/test_api_auth.py index f8b05ce..71a3221 100644 --- a/tests/unit/test_api_auth.py +++ b/tests/unit/test_api_auth.py @@ -4,13 +4,15 @@ import hashlib import json import time from pathlib import Path +from types import SimpleNamespace from unittest.mock import AsyncMock, patch import httpx import pytest from fastapi import HTTPException -from k7.api.main import app, authorize_namespace, load_api_keys +from k7.api.main import app, apply_node_scope, authorize_namespace, load_api_keys +from k7.core.models import OperationResult TEST_KEY = "k7-test-secret-key-abc123" TEST_KEY_HASH = hashlib.sha256(TEST_KEY.encode()).hexdigest() @@ -21,6 +23,7 @@ def _make_keys_data( expires: int | None = None, last_used: int | None = None, namespaces: list[str] | None = None, + nodes: list[str] | None = None, ) -> dict: entry: dict = {"name": "test-key"} if expires is not None: @@ -29,6 +32,8 @@ def _make_keys_data( entry["last_used"] = last_used if namespaces is not None: entry["namespaces"] = namespaces + if nodes is not None: + entry["nodes"] = nodes return {TEST_KEY_HASH: entry} @@ -217,3 +222,119 @@ class TestAuthorizeNamespace: assert resp.status_code == 200 assert resp.json()["data"] == payload core_cls.return_value.nodes_storage.assert_awaited_once() + + +# --- apply_node_scope --- + + +class TestApplyNodeScope: + def test_unrestricted_key_keeps_caller_pin(self): + assert apply_node_scope({"name": "u"}, None) is None + assert apply_node_scope({"name": "u", "nodes": []}, "k7-node-01") == "k7-node-01" + + def test_single_node_key_auto_pins(self): + assert apply_node_scope({"nodes": ["k7-node-01"]}, None) == "k7-node-01" + + def test_single_node_key_allows_listed_node(self): + assert apply_node_scope({"nodes": ["k7-node-01"]}, "k7-node-01") == "k7-node-01" + + def test_scoped_key_denied_other_node(self): + with pytest.raises(HTTPException) as exc: + apply_node_scope({"nodes": ["k7-node-01"]}, "k7-node-02") + assert exc.value.status_code == 403 + assert "k7-node-02" in str(exc.value.detail) + + def test_multi_node_key_requires_explicit_node(self): + with pytest.raises(HTTPException) as exc: + apply_node_scope({"nodes": ["k7-node-01", "k7-node-02"]}, None) + assert exc.value.status_code == 403 + assert "explicit allowed node" in str(exc.value.detail) + + def test_multi_node_key_allows_listed_node(self): + assert apply_node_scope({"nodes": ["k7-node-01", "k7-node-02"]}, "k7-node-02") == "k7-node-02" + + async def test_single_node_key_create_stamps_node_name(self, _patch_keys_file, keys_file: Path): + future_ts = int(time.time()) + 86400 + data = _make_keys_data(expires=future_ts, nodes=["k7-node-01"]) + keys_file.write_text(json.dumps(data)) + + with patch("k7.api.main.K7Core") as core_cls: + core_cls.return_value.create_sandbox = AsyncMock(return_value=OperationResult(success=True)) + transport = httpx.ASGITransport(app=app) + async with httpx.AsyncClient(transport=transport, base_url="http://test") as client: + resp = await client.post( + "/api/v1/sandboxes", + headers={"X-API-Key": TEST_KEY}, + json={"name": "t", "image": "alpine"}, + ) + assert resp.status_code == 201, resp.text + cfg = core_cls.return_value.create_sandbox.await_args.args[0] + assert cfg.node_name == "k7-node-01" + + async def test_scoped_key_create_other_node_returns_403(self, _patch_keys_file, keys_file: Path): + future_ts = int(time.time()) + 86400 + data = _make_keys_data(expires=future_ts, nodes=["k7-node-01"]) + keys_file.write_text(json.dumps(data)) + + with patch("k7.api.main.K7Core") as core_cls: + core_cls.return_value.create_sandbox = AsyncMock(return_value=OperationResult(success=True)) + transport = httpx.ASGITransport(app=app) + async with httpx.AsyncClient(transport=transport, base_url="http://test") as client: + resp = await client.post( + "/api/v1/sandboxes", + headers={"X-API-Key": TEST_KEY}, + json={"name": "t", "image": "alpine", "node_name": "k7-node-02"}, + ) + assert resp.status_code == 403 + core_cls.return_value.create_sandbox.assert_not_called() + + async def test_node_scoped_key_nodes_storage_returns_403(self, _patch_keys_file, keys_file: Path): + future_ts = int(time.time()) + 86400 + data = _make_keys_data(expires=future_ts, nodes=["k7-node-01"]) + keys_file.write_text(json.dumps(data)) + + transport = httpx.ASGITransport(app=app) + async with httpx.AsyncClient(transport=transport, base_url="http://test") as client: + resp = await client.get( + "/api/v1/nodes/storage", + headers={"X-API-Key": TEST_KEY}, + ) + assert resp.status_code == 403 + assert "cluster-wide node" in resp.json()["error"]["message"] + + async def test_restore_stamps_node_name(self, _patch_keys_file, keys_file: Path): + future_ts = int(time.time()) + 86400 + data = _make_keys_data(expires=future_ts, nodes=["k7-node-01"]) + keys_file.write_text(json.dumps(data)) + + with patch("k7.api.main.K7Core") as core_cls: + core_cls.return_value.restore_sandbox = AsyncMock(return_value=OperationResult(success=True)) + transport = httpx.ASGITransport(app=app) + async with httpx.AsyncClient(transport=transport, base_url="http://test") as client: + resp = await client.post( + "/api/v1/snapshots/snap1/restore", + headers={"X-API-Key": TEST_KEY}, + json={"new_sandbox_name": "restored"}, + ) + assert resp.status_code == 201, resp.text + kwargs = core_cls.return_value.restore_sandbox.await_args.kwargs + assert kwargs["node_name"] == "k7-node-01" + + async def test_fork_denied_when_source_on_other_node(self, _patch_keys_file, keys_file: Path): + future_ts = int(time.time()) + 86400 + data = _make_keys_data(expires=future_ts, nodes=["k7-node-01"]) + keys_file.write_text(json.dumps(data)) + source = SimpleNamespace(name="src", node="k7-node-02") + + with patch("k7.api.main.K7Core") as core_cls: + core_cls.return_value.list_sandboxes = AsyncMock(return_value=[source]) + core_cls.return_value.fork_sandbox = AsyncMock(return_value=OperationResult(success=True)) + transport = httpx.ASGITransport(app=app) + async with httpx.AsyncClient(transport=transport, base_url="http://test") as client: + resp = await client.post( + "/api/v1/sandboxes/src/fork", + headers={"X-API-Key": TEST_KEY}, + json={"new_name": "forked"}, + ) + assert resp.status_code == 403 + core_cls.return_value.fork_sandbox.assert_not_called() diff --git a/tests/unit/test_api_lifecycle_endpoints.py b/tests/unit/test_api_lifecycle_endpoints.py index 95d2e9b..ee09018 100644 --- a/tests/unit/test_api_lifecycle_endpoints.py +++ b/tests/unit/test_api_lifecycle_endpoints.py @@ -17,7 +17,7 @@ import pytest from k7.api.main import app from k7.core.models import OperationResult -TEST_KEY = "k7-test-secret-key-spec10a" +TEST_KEY = "k7-test-secret-key-lifecycle" TEST_KEY_HASH = hashlib.sha256(TEST_KEY.encode()).hexdigest() diff --git a/tests/unit/test_cli_api_keys.py b/tests/unit/test_cli_api_keys.py index a8dc979..5b1ffc2 100644 --- a/tests/unit/test_cli_api_keys.py +++ b/tests/unit/test_cli_api_keys.py @@ -2,7 +2,7 @@ import json from pathlib import Path -from unittest.mock import patch +from unittest.mock import AsyncMock, patch from typer.testing import CliRunner @@ -34,6 +34,20 @@ class TestGenerateApiKeyNamespaces: data = json.loads(keys_file.read_text()) entry = next(iter(data.values())) assert "namespaces" not in entry + assert "nodes" not in entry + + def test_scoped_nodes_persisted(self, tmp_path: Path): + keys_file = tmp_path / "api_keys.json" + with patch("k7.cli.k7.API_KEYS_FILE", keys_file): + result = runner.invoke( + app, + ["generate-api-key", "pinned", "--node", "k7-node-01", "--node", "k7-node-02"], + ) + assert result.exit_code == 0, result.output + data = json.loads(keys_file.read_text()) + entry = next(iter(data.values())) + assert entry["nodes"] == ["k7-node-01", "k7-node-02"] + assert "Nodes: k7-node-01, k7-node-02" in result.output def test_list_shows_namespaces_column(self, tmp_path: Path): keys_file = tmp_path / "api_keys.json" @@ -53,6 +67,13 @@ class TestGenerateApiKeyNamespaces: "expires": 2, "last_used": None, }, + "h3": { + "name": "pinned", + "created": 1, + "expires": 2, + "last_used": None, + "nodes": ["k7-node-01"], + }, } ) ) @@ -60,5 +81,40 @@ class TestGenerateApiKeyNamespaces: result = runner.invoke(app, ["list-api-keys"]) assert result.exit_code == 0, result.output assert "Namespaces" in result.output + assert "Nodes" in result.output assert "alpha" in result.output + assert "k7-node-01" in result.output assert "*" in result.output + + +class TestNodesDedicateCli: + def test_dedicate_requires_tenant(self): + result = runner.invoke(app, ["nodes", "dedicate", "k7-node-01"]) + assert result.exit_code != 0 + + def test_dedicate_calls_core(self): + from k7.core.models import OperationResult + + with patch("k7.cli.k7.K7Core") as core_cls: + core_cls.return_value.dedicate_node = AsyncMock(return_value=OperationResult(success=True, message="ok")) + result = runner.invoke(app, ["nodes", "dedicate", "k7-node-01", "--tenant", "acme"]) + assert result.exit_code == 0, result.output + core_cls.return_value.dedicate_node.assert_awaited_once_with("k7-node-01", "acme") + + +class TestNodesListCli: + def test_json_prints_k3s_names(self): + rows = [ + { + "name": "k7-node-01", + "hostname": "k7-node-01", + "backends": ["k7d"], + "tenant": "", + } + ] + with patch("k7.cli.k7.K7Core") as core_cls: + core_cls.return_value.list_cluster_nodes = AsyncMock(return_value=rows) + result = runner.invoke(app, ["nodes", "list", "--json"]) + assert result.exit_code == 0, result.output + assert "k7-node-01" in result.output + assert "k7d" in result.output diff --git a/tests/unit/test_cli_api_lifecycle.py b/tests/unit/test_cli_api_lifecycle.py index 6cee469..5e62f80 100644 --- a/tests/unit/test_cli_api_lifecycle.py +++ b/tests/unit/test_cli_api_lifecycle.py @@ -70,6 +70,12 @@ class TestApiEnableDisable: assert result.exit_code != 0 assert "failed" in result.output.lower() + def test_enable_without_kubectl_exits(self): + with patch("k7.cli.k7._kubectl_run", return_value=None): + result = runner.invoke(app, ["api", "enable"]) + assert result.exit_code != 0 + assert "kubectl" in result.output.lower() + # --------------------------------------------------------------------------- # ``k7 api status``. @@ -114,6 +120,30 @@ class TestApiStatus: assert "not found" in result.output.lower() assert "k7 install" in result.output + def test_without_kubectl_probes_configured_health(self): + fake = SimpleNamespace(ok=True, status_code=200) + with ( + patch("k7.cli.k7._kubectl_run", return_value=None), + patch("k7.cli.k7._resolve_api_url", return_value="https://10.0.0.1:31007"), + patch("k7.cli.k7._resolve_api_ca", return_value="/tmp/ca.crt"), + patch("k7.cli.k7.requests.get", return_value=fake) as get, + ): + result = runner.invoke(app, ["api", "status"]) + assert result.exit_code == 0, result.output + assert "reachable" in result.output.lower() + assert "https://10.0.0.1:31007" in result.output + get.assert_called_once() + assert get.call_args.args[0] == "https://10.0.0.1:31007/health" + + def test_without_kubectl_or_url_exits(self): + with ( + patch("k7.cli.k7._kubectl_run", return_value=None), + patch("k7.cli.k7._resolve_api_url", return_value=None), + ): + result = runner.invoke(app, ["api", "status"]) + assert result.exit_code != 0 + assert "kubectl is not installed" in result.output + # --------------------------------------------------------------------------- # ``k7 api endpoint``. @@ -132,6 +162,16 @@ class TestApiEndpoint: result = runner.invoke(app, ["api", "endpoint"]) assert result.exit_code != 0 + def test_without_kubectl_prints_configured_url(self, tmp_path): + with ( + patch("k7.cli.k7._ETC_K7_API_ENDPOINT", tmp_path / "missing"), + patch("k7.cli.k7.subprocess.run", side_effect=FileNotFoundError("kubectl")), + patch("k7.cli.k7._resolve_api_url", return_value="https://10.0.0.1:31007"), + ): + result = runner.invoke(app, ["api", "endpoint"]) + assert result.exit_code == 0, result.output + assert result.output.strip() == "https://10.0.0.1:31007" + # --------------------------------------------------------------------------- # Deprecated command aliases. @@ -201,7 +241,7 @@ class TestInstallNoApi: patch("k7.cli.k7._build_default_inventory", return_value="[k7_servers]\nhost ansible_host=1.2.3.4"), ): core_cls.return_value.install_node = _capture - result = runner.invoke(app, ["install", "--no-api", "host"]) + result = runner.invoke(app, ["install", "--backend", "kql", "--no-api", "host"]) assert result.exit_code == 0, result.output extra_vars = seen["kwargs"]["extra_vars"] assert extra_vars.get("k7_api_enabled") == "false" @@ -218,7 +258,7 @@ class TestInstallNoApi: patch("k7.cli.k7._build_default_inventory", return_value="[k7_servers]\nhost ansible_host=1.2.3.4"), ): core_cls.return_value.install_node = _capture - result = runner.invoke(app, ["install", "host"]) + result = runner.invoke(app, ["install", "--backend", "kql", "host"]) assert result.exit_code == 0, result.output extra_vars = seen["kwargs"]["extra_vars"] assert extra_vars.get("k7_api_enabled") == "true" @@ -237,7 +277,7 @@ class TestInstallHubble: patch("k7.cli.k7._build_default_inventory", return_value="[k7_servers]\nhost ansible_host=1.2.3.4"), ): core_cls.return_value.install_node = _capture - result = runner.invoke(app, ["install", "--hubble", "host"]) + result = runner.invoke(app, ["install", "--backend", "kql", "--hubble", "host"]) assert result.exit_code == 0, result.output extra_vars = seen["kwargs"]["extra_vars"] assert extra_vars.get("k7_hubble_enabled") == "true" @@ -254,7 +294,7 @@ class TestInstallHubble: patch("k7.cli.k7._build_default_inventory", return_value="[k7_servers]\nhost ansible_host=1.2.3.4"), ): core_cls.return_value.install_node = _capture - result = runner.invoke(app, ["install", "host"]) + result = runner.invoke(app, ["install", "--backend", "kql", "host"]) assert result.exit_code == 0, result.output extra_vars = seen["kwargs"]["extra_vars"] assert extra_vars.get("k7_hubble_enabled") == "false" diff --git a/tests/unit/test_cli_install_api_allowlist.py b/tests/unit/test_cli_install_api_allowlist.py index c1f3dc5..a849981 100644 --- a/tests/unit/test_cli_install_api_allowlist.py +++ b/tests/unit/test_cli_install_api_allowlist.py @@ -16,6 +16,8 @@ runner = CliRunner() def _install(args: list[str]): + if "--backend" not in args and "-b" not in args: + args = ["--backend", "kql", *args] with patch("k7.cli.k7.K7Core") as core_cls: install = MagicMock(return_value=OperationResult(success=True, message="ok")) core_cls.return_value.install_node = install diff --git a/tests/unit/test_cli_install_api_tls.py b/tests/unit/test_cli_install_api_tls.py index 709ab7b..86827ab 100644 --- a/tests/unit/test_cli_install_api_tls.py +++ b/tests/unit/test_cli_install_api_tls.py @@ -22,6 +22,8 @@ _CLUSTERROLE = (Path(__file__).resolve().parents[2] / "src/k7/deploy/manifests/k def _install(args: list[str]): + if "--backend" not in args and "-b" not in args: + args = ["--backend", "kql", *args] with patch("k7.cli.k7.K7Core") as core_cls: install = MagicMock(return_value=OperationResult(success=True, message="ok")) core_cls.return_value.install_node = install diff --git a/tests/unit/test_cli_install_backends.py b/tests/unit/test_cli_install_backends.py new file mode 100644 index 0000000..e4b7f36 --- /dev/null +++ b/tests/unit/test_cli_install_backends.py @@ -0,0 +1,53 @@ +"""``k7 install`` requires --backend; none is the empty set.""" + +from pathlib import Path +from unittest.mock import MagicMock, patch + +from typer.testing import CliRunner + +from k7.cli.k7 import INSTALL_BACKEND_REQUIRED, app +from k7.core.models import OperationResult + +runner = CliRunner() + + +def _install(args: list[str]): + with patch("k7.cli.k7.K7Core") as core_cls: + install = MagicMock(return_value=OperationResult(success=True, message="ok")) + core_cls.return_value.install_node = install + result = runner.invoke(app, ["--core", "install", *args]) + return result, install + + +def test_install_without_backend_fails(): + result, install = _install([]) + assert result.exit_code != 0 + assert "Specify --backend" in result.output + assert "none" in result.output + install.assert_not_called() + assert "kfd" in INSTALL_BACKEND_REQUIRED + + +def test_install_backend_none_forwards_sentinel(): + result, install = _install(["--backend", "none"]) + assert result.exit_code == 0, result.output + extra = install.call_args.kwargs["extra_vars"] + assert extra["k7_backends"] == "none" + inv = install.call_args.args[1] + assert "k7_backends=none" in inv + + +def test_install_inventory_without_backend_does_not_send_extra_var(tmp_path: Path): + inv = tmp_path / "inventory.ini" + inv.write_text("[k7_servers]\nhost ansible_host=1.2.3.4 k7_backends=kql\n") + result, install = _install(["-i", str(inv)]) + assert result.exit_code == 0, result.output + extra = install.call_args.kwargs["extra_vars"] + assert "k7_backends" not in extra + + +def test_install_none_mixed_fails_before_ansible(): + result, install = _install(["--backend", "none,kql"]) + assert result.exit_code != 0 + assert "cannot be combined" in result.output + install.assert_not_called() diff --git a/tests/unit/test_cli_utils.py b/tests/unit/test_cli_utils.py index 03ad613..d7daee8 100644 --- a/tests/unit/test_cli_utils.py +++ b/tests/unit/test_cli_utils.py @@ -9,6 +9,7 @@ import typer from k7.cli.k7 import ( _build_default_inventory, + _core_run, _kubectl_cmd, _normalize_backend, _parse_backends, @@ -159,6 +160,13 @@ class TestReadApiEndpoint: result = _read_api_endpoint(["kubectl"]) assert result == "https://10.0.0.1:31007" + def test_falls_back_to_configured_url_when_kubectl_missing(self): + with ( + patch("k7.cli.k7.subprocess.run", side_effect=FileNotFoundError("kubectl")), + patch("k7.cli.k7._resolve_api_url", return_value="https://10.0.0.1:31007"), + ): + assert _read_api_endpoint(["kubectl"]) == "https://10.0.0.1:31007" + # --- _build_default_inventory --- @@ -178,8 +186,17 @@ class TestBuildDefaultInventory: assert "[k7_agents]" in inv assert "[k7_cluster:children]" in inv - def test_localhost_both_backends_default(self): - # `k7 install` with no flags installs both backends by default. + def test_localhost_none_backend(self): + inv = _build_default_inventory( + hosts=None, + role="server", + backends=[], + disk=None, + longhorn_extra_disk=None, + ) + assert "k7_backends=none" in inv + + def test_localhost_both_kata_backends(self): inv = _build_default_inventory( hosts=None, role="server", @@ -290,3 +307,39 @@ class TestParseBackends: _parse_backends("") with pytest.raises(typer.BadParameter, match="cannot be empty"): _parse_backends(" , , ") + + def test_none_is_empty_set(self): + assert _parse_backends("none") == [] + assert _parse_backends("None") == [] + + def test_none_cannot_mix(self): + with pytest.raises(typer.BadParameter, match="cannot be combined"): + _parse_backends("none,kql") + with pytest.raises(typer.BadParameter, match="cannot be combined"): + _parse_backends("kfd,none") + + +class TestCoreRun: + def test_closes_real_k7core(self): + from k7.core.core import K7Core + + core = K7Core() + closed: list[bool] = [] + + async def aclose(): + closed.append(True) + + async def work(): + return "ok" + + core.aclose = aclose # type: ignore[method-assign] + assert _core_run(core, work()) == "ok" + assert closed == [True] + + def test_skips_aclose_on_mocks(self): + fake = MagicMock() + + async def work(): + return 7 + + assert _core_run(fake, work()) == 7 diff --git a/tests/unit/test_core_k7d.py b/tests/unit/test_core_k7d.py index bf1f542..4ad9155 100644 --- a/tests/unit/test_core_k7d.py +++ b/tests/unit/test_core_k7d.py @@ -5,11 +5,13 @@ are never touched. The real end-to-end behaviour is covered by ``tests/integration/test_k7d.py`` on a k7d-capable node. """ +from contextlib import contextmanager from unittest.mock import AsyncMock, MagicMock, patch import pytest from k7.core.core import ( + K7_TENANT_LABEL, K7D_ANN_FORK_SOURCE_CLUSTER, K7D_ANN_FORK_SOURCE_VM, K7D_ANN_PAUSED, @@ -20,10 +22,15 @@ from k7.core.models import OperationResult, SandboxConfig from tests.unit.conftest import mock_deployment, mock_pod -def _local_node(core: K7Core): - """Patch the node resolution so the sandbox appears LOCAL — - these tests exercise the direct daemon-socket path, not forwarding.""" - return patch.object(core, "_k7d_sandbox_node", new=AsyncMock(return_value=core._k7d_local_node())) +@contextmanager +def _local_node(core: K7Core, node: str = "k7-node-01", pod: str = "src-sb-abc"): + """Sandbox appears LOCAL — direct daemon-socket path, not agent forward.""" + with ( + patch.object(core, "_k7d_local_node", return_value=node), + patch.object(core, "_k7d_sandbox_node", new=AsyncMock(return_value=node)), + patch.object(core, "_k7d_sandbox_placement", new=AsyncMock(return_value=(node, pod))), + ): + yield def _k7d_deployment(name: str = "src-sb", sidecar: str | None = None): @@ -101,6 +108,76 @@ class TestK7dCreateSandbox: # Deployment carries the backend annotation for later detection. assert deployment.metadata.annotations["k7.katakate.org/backend"] == "k7d" + async def test_node_name_uses_hostname_selector_not_spec_node_name(self, core: K7Core): + mock_apps = AsyncMock() + mock_v1 = AsyncMock() + mock_net = AsyncMock() + node = MagicMock() + node.spec.taints = [] + mock_v1.read_node.return_value = node + core._apps_v1_client = mock_apps + core._core_v1_client = mock_v1 + core._networking_v1_client = mock_net + + sched_ok = MagicMock() + sched_ok.success = True + with patch.object(core, "_check_scheduling", new=AsyncMock(return_value=sched_ok)): + cfg = SandboxConfig(name="k7d-pin", image="alpine:3.20", backend="k7d", node_name="k7-node-01") + result = await core.create_sandbox(cfg) + + assert result.success, result.error + pod_spec = mock_apps.create_namespaced_deployment.call_args.kwargs["body"].spec.template.spec + assert pod_spec.node_name in (None, "") + assert pod_spec.node_selector == { + "k7.katakate.org/backend-k7d": "true", + "kubernetes.io/hostname": "k7-node-01", + } + assert not pod_spec.tolerations + mock_v1.read_node.assert_awaited_once_with("k7-node-01") + + async def test_dedicated_node_adds_tenant_toleration(self, core: K7Core): + mock_apps = AsyncMock() + mock_v1 = AsyncMock() + mock_net = AsyncMock() + taint = MagicMock() + taint.key = "k7.katakate.org/tenant" + taint.value = "acme" + taint.effect = "NoSchedule" + node = MagicMock() + node.spec.taints = [taint] + mock_v1.read_node.return_value = node + core._apps_v1_client = mock_apps + core._core_v1_client = mock_v1 + core._networking_v1_client = mock_net + + sched_ok = MagicMock() + sched_ok.success = True + with patch.object(core, "_check_scheduling", new=AsyncMock(return_value=sched_ok)): + cfg = SandboxConfig(name="k7d-ten", image="alpine:3.20", backend="k7d", node_name="k7-node-01") + result = await core.create_sandbox(cfg) + + assert result.success, result.error + pod_spec = mock_apps.create_namespaced_deployment.call_args.kwargs["body"].spec.template.spec + assert pod_spec.tolerations + tol = pod_spec.tolerations[0] + assert tol.key == "k7.katakate.org/tenant" + assert tol.value == "acme" + assert tol.effect == "NoSchedule" + + async def test_missing_pin_node_fails_loud(self, core: K7Core): + from kubernetes_asyncio.client.exceptions import ApiException + + mock_v1 = AsyncMock() + mock_v1.read_node.side_effect = ApiException(status=404) + core._apps_v1_client = AsyncMock() + core._core_v1_client = mock_v1 + core._networking_v1_client = AsyncMock() + cfg = SandboxConfig(name="k7d-gone", image="alpine:3.20", backend="k7d", node_name="no-such-node") + result = await core.create_sandbox(cfg) + assert not result.success + assert "no-such-node" in (result.error or "") + core._apps_v1_client.create_namespaced_deployment.assert_not_called() + async def test_create_k7d_fc_uses_runtime_class_k7_fc(self, core: K7Core): mock_apps = AsyncMock() mock_v1 = AsyncMock() @@ -481,3 +558,52 @@ class TestK7dRequest: pytest.raises(RuntimeError, match="no such VM"), ): await core._k7d_request({"op": "pause_vm", "vm_id": "vm-9"}) + + +class TestDedicateNode: + async def test_dedicate_writes_label_and_taint(self, core: K7Core): + mock_v1 = AsyncMock() + node = MagicMock() + node.spec.taints = [] + mock_v1.read_node.return_value = node + core._core_v1_client = mock_v1 + + result = await core.dedicate_node("k7-node-01", "acme") + assert result.success, result.error + body = mock_v1.patch_node.await_args.args[1] + assert body["metadata"]["labels"][K7_TENANT_LABEL] == "acme" + assert body["spec"]["taints"] == [{"key": K7_TENANT_LABEL, "value": "acme", "effect": "NoSchedule"}] + + async def test_dedicate_missing_node_fails_loud(self, core: K7Core): + from kubernetes_asyncio.client.exceptions import ApiException + + mock_v1 = AsyncMock() + mock_v1.read_node.side_effect = ApiException(status=404) + core._core_v1_client = mock_v1 + result = await core.dedicate_node("missing", "acme") + assert not result.success + assert "not found" in (result.error or "") + mock_v1.patch_node.assert_not_called() + + async def test_undedicate_drops_label_and_taint(self, core: K7Core): + mock_v1 = AsyncMock() + other = MagicMock() + other.key = "node.kubernetes.io/unreachable" + other.value = None + other.effect = "NoExecute" + ours = MagicMock() + ours.key = K7_TENANT_LABEL + ours.value = "acme" + ours.effect = "NoSchedule" + node = MagicMock() + node.spec.taints = [other, ours] + mock_v1.read_node.return_value = node + core._core_v1_client = mock_v1 + + result = await core.undedicate_node("k7-node-01") + assert result.success, result.error + body = mock_v1.patch_node.await_args.args[1] + assert body["metadata"]["labels"][K7_TENANT_LABEL] is None + assert body["spec"]["taints"] == [ + {"key": "node.kubernetes.io/unreachable", "value": None, "effect": "NoExecute"} + ] diff --git a/tests/unit/test_core_mocked.py b/tests/unit/test_core_mocked.py index 511237e..538bf6f 100644 --- a/tests/unit/test_core_mocked.py +++ b/tests/unit/test_core_mocked.py @@ -5,8 +5,8 @@ from unittest.mock import AsyncMock, MagicMock, patch from kubernetes_asyncio.client.exceptions import ApiException -from k7.core.core import K7Core -from k7.core.models import OperationResult, SandboxConfig +from k7.core.core import K7_TENANT_LABEL, K7Core +from k7.core.models import ExecResult, OperationResult, SandboxConfig from tests.unit.conftest import mock_deployment, mock_pod # --------------------------------------------------------------------------- @@ -14,7 +14,7 @@ from tests.unit.conftest import mock_deployment, mock_pod # --------------------------------------------------------------------------- -def _setup_clients(core, apps=None, v1=None, net=None, metrics=None, custom=None): +def _setup_clients(core, apps=None, v1=None, net=None, metrics=None, custom=None, batch=None, apiext=None): """Wire MagicMock clients into a K7Core instance.""" core._config_loaded = True if apps is not None: @@ -27,6 +27,10 @@ def _setup_clients(core, apps=None, v1=None, net=None, metrics=None, custom=None core._metrics_client = metrics if custom is not None: core._custom_objects_client = custom + if batch is not None: + core._batch_v1_client = batch + if apiext is not None: + core._apiextensions_v1_client = apiext # --- _detect_backend --- @@ -72,6 +76,12 @@ class TestDetectBackend: with patch("os.path.exists", return_value=False): assert await core._detect_backend("missing", "default") == "kata-firecracker-devmapper" + async def test_none_file_is_not_kfd(self, core: K7Core, tmp_path): + backend_file = tmp_path / "backend" + backend_file.write_text("none\n") + with patch("os.path.exists", return_value=True), patch("builtins.open", return_value=open(backend_file)): + assert await core._detect_backend(None) == "" + # --- delete_sandbox --- @@ -542,6 +552,7 @@ class TestForkSandbox: mock_custom = AsyncMock() mock_custom.get_namespaced_custom_object.side_effect = ApiException(status=404) _setup_clients(core, apps=mock_apps, v1=mock_v1, net=mock_net, custom=mock_custom) + core._detect_backend = AsyncMock(return_value="kata-qemu-longhorn") src_dep = MagicMock() src_dep.metadata.name = "src" @@ -569,6 +580,7 @@ class TestForkSandbox: mock_apps = AsyncMock() mock_apps.read_namespaced_deployment.side_effect = ApiException(status=404) _setup_clients(core, apps=mock_apps, v1=AsyncMock()) + core._detect_backend = AsyncMock(return_value="kata-qemu-longhorn") result = await core.fork_sandbox("src", "dst") assert result.success is False @@ -578,6 +590,7 @@ class TestForkSandbox: mock_v1 = AsyncMock() mock_v1.read_namespaced_persistent_volume_claim.side_effect = ApiException(status=404) _setup_clients(core, apps=mock_apps, v1=mock_v1) + core._detect_backend = AsyncMock(return_value="kata-qemu-longhorn") result = await core.fork_sandbox("src", "dst") assert result.success is False @@ -978,8 +991,8 @@ class TestReadSandboxEgressWhitelist: class TestK7dAgentForwarding: - def _remote_pod(self, node: str = "node-remote"): - pod = mock_pod(phase="Running") + def _remote_pod(self, node: str = "node-remote", name: str = "sb1-pod"): + pod = mock_pod(name=name, phase="Running") pod.spec.node_name = node return pod @@ -1007,29 +1020,38 @@ class TestK7dAgentForwarding: fwd.assert_called_once() assert fwd.call_args.args[0] == "node-remote" assert fwd.call_args.args[1] == "pause" + assert fwd.call_args.args[2]["pod_name"] == "sb1-pod" + mock_apps.patch_namespaced_deployment.assert_awaited() - async def test_fork_forwards_to_remote_agent(self, core: K7Core): + async def test_fork_looks_up_on_remote_creates_locally(self, core: K7Core): + """k7d fork Kubernetes writes stay on k7-api; agent only looks up the VM.""" dep = mock_deployment(annotations={"k7.katakate.org/backend": "k7d"}) + dep.spec.template.metadata.labels = {"app": "sb1"} + dep.metadata.labels = {"app": "sb1"} + dep.spec.selector.match_labels = {"app": "sb1"} mock_apps = AsyncMock() mock_apps.read_namespaced_deployment.return_value = dep mock_v1 = AsyncMock() mock_v1.list_namespaced_pod.return_value = MagicMock(items=[self._remote_pod()]) _setup_clients(core, apps=mock_apps, v1=mock_v1) + vm = {"vm_id": "vm-1-0", "sandbox_id": "cri-src", "cluster_id": "cri-src"} with ( patch.dict("os.environ", {"K7_NODE_NAME": "node-local"}), - patch.object( - core, - "_k7d_forward_vm_op", - new_callable=AsyncMock, - return_value=OperationResult(success=True, message="forwarded"), - ) as fwd, + patch.object(core, "lookup_k7d_vm", new=AsyncMock(return_value=vm)), + patch.object(core, "_wait_for_pod_container_started", new=AsyncMock(return_value="fork-pod")), + patch.object(core, "_read_sandbox_egress_whitelist", new=AsyncMock(return_value=[])), + patch.object(core, "_read_sandbox_ingress_rules", new=AsyncMock(return_value=([], []))), + patch.object(core, "_apply_sandbox_network_policies", new=AsyncMock(return_value=OperationResult(True))), + patch.object(core, "_k7d_forward_vm_op", new_callable=AsyncMock) as fwd, ): result = await core.fork_sandbox("sb1", "sb1-fork") - assert result.success is True - assert fwd.call_args.args[1] == "fork" - assert fwd.call_args.args[2]["new_name"] == "sb1-fork" + assert result.success, result.error + fwd.assert_not_called() + mock_apps.create_namespaced_deployment.assert_awaited() + new_dep = mock_apps.create_namespaced_deployment.call_args.kwargs["body"] + assert new_dep.spec.template.spec.node_name == "node-remote" async def test_agent_refuses_to_reforward(self, core: K7Core): """An agent that resolves a sandbox to yet another node must fail @@ -1283,10 +1305,9 @@ class TestWaitForJob: job.status.succeeded = 1 job.status.failed = 0 mock_batch.read_namespaced_job.return_value = job - _setup_clients(core) + _setup_clients(core, batch=mock_batch) - with patch("k7.core.core.client.BatchV1Api", return_value=mock_batch): - result = await core._wait_for_job("j1", timeout=5) + result = await core._wait_for_job("j1", timeout=5) assert result.success is True @@ -1296,10 +1317,9 @@ class TestWaitForJob: job.status.succeeded = 0 job.status.failed = 1 mock_batch.read_namespaced_job.return_value = job - _setup_clients(core) + _setup_clients(core, batch=mock_batch) - with patch("k7.core.core.client.BatchV1Api", return_value=mock_batch): - result = await core._wait_for_job("j1", timeout=5) + result = await core._wait_for_job("j1", timeout=5) assert result.success is False assert "failed" in result.error @@ -1310,12 +1330,9 @@ class TestWaitForJob: job_ok.status.succeeded = 1 job_ok.status.failed = 0 mock_batch.read_namespaced_job.side_effect = [ApiException(status=404), job_ok] - _setup_clients(core) + _setup_clients(core, batch=mock_batch) - with ( - patch("k7.core.core.client.BatchV1Api", return_value=mock_batch), - patch("k7.core.core.asyncio.sleep", new_callable=AsyncMock), - ): + with patch("k7.core.core.asyncio.sleep", new_callable=AsyncMock): result = await core._wait_for_job("j1", timeout=10) assert result.success is True @@ -1323,10 +1340,9 @@ class TestWaitForJob: async def test_non_404_error(self, core: K7Core): mock_batch = AsyncMock() mock_batch.read_namespaced_job.side_effect = ApiException(status=500) - _setup_clients(core) + _setup_clients(core, batch=mock_batch) - with patch("k7.core.core.client.BatchV1Api", return_value=mock_batch): - result = await core._wait_for_job("j1", timeout=5) + result = await core._wait_for_job("j1", timeout=5) assert result.success is False assert "wait error" in result.error @@ -1337,7 +1353,7 @@ class TestWaitForJob: job.status.succeeded = 0 job.status.failed = 0 mock_batch.read_namespaced_job.return_value = job - _setup_clients(core) + _setup_clients(core, batch=mock_batch) elapsed = 0.0 @@ -1347,7 +1363,6 @@ class TestWaitForJob: return elapsed with ( - patch("k7.core.core.client.BatchV1Api", return_value=mock_batch), patch("k7.core.core.time.time", side_effect=advancing_time), patch("k7.core.core.asyncio.sleep", new_callable=AsyncMock), ): @@ -1493,10 +1508,11 @@ class TestWaitForPodContainerStarted: # --- _list_backends_per_node --- -def _node(name: str, labels: dict[str, str]): +def _node(name: str, labels: dict[str, str], taints: list | None = None): n = MagicMock() n.metadata.name = name n.metadata.labels = labels + n.spec.taints = taints or [] return n @@ -1534,6 +1550,35 @@ class TestListBackendsPerNode: assert result["n"] == [] +class TestListClusterNodes: + async def test_hostname_backends_and_tenant(self, core: K7Core): + taint = SimpleNamespace(key=K7_TENANT_LABEL, value="acme", effect="NoSchedule") + mock_v1 = AsyncMock() + mock_v1.list_node.return_value = SimpleNamespace( + items=[ + _node( + "k7-node-01", + { + "kubernetes.io/hostname": "k7-node-01", + "k7.katakate.org/backend-k7d": "true", + K7_TENANT_LABEL: "acme", + }, + taints=[taint], + ), + _node("k7-node-02", {"kubernetes.io/hostname": "k7-node-02"}), + ] + ) + _setup_clients(core, v1=mock_v1) + rows = await core.list_cluster_nodes() + by_name = {r["name"]: r for r in rows} + assert by_name["k7-node-01"]["hostname"] == "k7-node-01" + assert by_name["k7-node-01"]["backends"] == ["k7d"] + assert by_name["k7-node-01"]["tenant"] == "acme" + assert by_name["k7-node-02"]["hostname"] == "k7-node-02" + assert by_name["k7-node-02"]["backends"] == [] + assert by_name["k7-node-02"]["tenant"] == "" + + # --- _check_scheduling --- @@ -1633,18 +1678,16 @@ class TestCheckScheduling: class TestCiliumAvailable: async def test_returns_true_when_crd_exists(self, core: K7Core): - core._config_loaded = True fake_api_ext = AsyncMock() fake_api_ext.read_custom_resource_definition.return_value = MagicMock() - with patch("k7.core.core.client.ApiextensionsV1Api", return_value=fake_api_ext): - assert await core._cilium_available() is True + _setup_clients(core, apiext=fake_api_ext) + assert await core._cilium_available() is True async def test_returns_false_when_crd_missing(self, core: K7Core): - core._config_loaded = True fake_api_ext = AsyncMock() fake_api_ext.read_custom_resource_definition.side_effect = ApiException(status=404) - with patch("k7.core.core.client.ApiextensionsV1Api", return_value=fake_api_ext): - assert await core._cilium_available() is False + _setup_clients(core, apiext=fake_api_ext) + assert await core._cilium_available() is False class TestApplyCiliumEgressPolicy: @@ -1711,3 +1754,81 @@ class TestApplyCiliumEgressPolicy: result = await core._apply_cilium_egress_policy("sb1", "default", [], ["api.openai.com"]) assert result.success is False assert "CiliumNetworkPolicy" in result.error + + +# --- kubernetes_asyncio ApiClient lifecycle --- + + +class TestApiClientLifecycle: + async def test_typed_clients_share_one_api_client(self, core: K7Core): + api = MagicMock() + api.close = AsyncMock() + with ( + patch("k7.core.core.client.ApiClient", return_value=api) as api_cls, + patch("k7.core.core.client.CoreV1Api") as core_cls, + patch("k7.core.core.client.AppsV1Api") as apps_cls, + ): + await core._get_core_v1_client() + await core._get_apps_v1_client() + api_cls.assert_called_once() + core_cls.assert_called_once_with(api) + apps_cls.assert_called_once_with(api) + await core.aclose() + api.close.assert_awaited_once() + assert core._api_client is None + assert core._core_v1_client is None + assert core._apps_v1_client is None + + async def test_aclose_is_idempotent(self, core: K7Core): + await core.aclose() + await core.aclose() + + async def test_aclose_allows_a_new_session(self, core: K7Core): + first = MagicMock() + first.close = AsyncMock() + second = MagicMock() + second.close = AsyncMock() + with ( + patch("k7.core.core.client.ApiClient", side_effect=[first, second]), + patch("k7.core.core.client.CoreV1Api"), + ): + await core._get_core_v1_client() + await core.aclose() + first.close.assert_awaited_once() + await core._get_core_v1_client() + await core.aclose() + second.close.assert_awaited_once() + + +class TestCreateSnapshotMessage: + async def test_named_snapshot_keeps_create_message(self, core: K7Core): + """Regression: quiesced success used to drop the VolumeSnapshot name. + + ``k7 snapshot create`` echoes ``result.message``; empty stdout failed + ``TestSnapshotCrud`` with ``assert snap in cp.stdout``. + """ + dep = MagicMock() + dep.metadata.annotations = {} + dep.status.ready_replicas = 1 + mock_apps = AsyncMock() + mock_apps.read_namespaced_deployment.return_value = dep + _setup_clients(core, apps=mock_apps) + + with ( + patch.object(core, "_detect_backend", new=AsyncMock(return_value="kata-qemu-longhorn")), + patch.object( + core, + "_create_volume_snapshot", + new=AsyncMock( + return_value=OperationResult(success=True, message="Snapshot snap1 created for PVC sb-root-lh") + ), + ), + patch.object( + core, + "exec_command", + new=AsyncMock(return_value=ExecResult(exit_code=0, stdout="", stderr="", duration_ms=0)), + ), + ): + result = await core.create_snapshot("sb", "snap1") + assert result.success, result.error + assert "snap1" in result.message diff --git a/tests/unit/test_core_utils.py b/tests/unit/test_core_utils.py index cdd479a..0c65880 100644 --- a/tests/unit/test_core_utils.py +++ b/tests/unit/test_core_utils.py @@ -47,6 +47,18 @@ class TestMemoryLimitToMib: core._memory_limit_to_mib(None) # type: ignore[arg-type] +class TestKataHypervisorMemoryMib: + def test_256mi_ok(self, core: K7Core): + assert core._kata_hypervisor_memory_mib("256Mi") == 256 + + def test_1gi_ok(self, core: K7Core): + assert core._kata_hypervisor_memory_mib("1Gi") == 1024 + + def test_128mi_rejected(self, core: K7Core): + with pytest.raises(ValueError, match="below Kata's minimum 256Mi"): + core._kata_hypervisor_memory_mib("128Mi") + + # --- _parse_resource_value --- diff --git a/tests/unit/test_kata_docker.py b/tests/unit/test_kata_docker.py index b4d1c20..c8535e7 100644 --- a/tests/unit/test_kata_docker.py +++ b/tests/unit/test_kata_docker.py @@ -12,10 +12,14 @@ from k7.core.docker import ( CLI_STAGING_DIR, DIND_IMAGE, DOCKER_HOST_URL, + FSFREEZE_STAGED, + FSFREEZE_VEHICLE, GRAPH_DEVICE_PATH, - KATA_FORK_GRAPH_REJECT, + KATA_GRAPH_MOUNT, KATA_SOCKET_DIR, KFD_DOCKER_STORAGE_CLASS, + KFD_FORK_REJECT, + STAGE_FSFREEZE_CMD, VEHICLE_CONTAINER_NAME, ) from k7.core.models import ExecResult, SandboxConfig @@ -137,9 +141,67 @@ class TestKataDockerSnapshot: assert wait.await_count == 2 names = [c.args[0] for c in wait.await_args_list] assert names == ["snap1", "snap1-docker"] - assert exec_cmd.await_count == 2 + cmds = [c.args[1] for c in exec_cmd.await_args_list] + freeze_cmd = ( + f"cp {FSFREEZE_STAGED} {FSFREEZE_VEHICLE} && chmod 755 {FSFREEZE_VEHICLE} && " + f"{FSFREEZE_VEHICLE} -f {KATA_GRAPH_MOUNT}" + ) + thaw_cmd = f"{FSFREEZE_VEHICLE} -u {KATA_GRAPH_MOUNT}" + assert cmds == ["sync", STAGE_FSFREEZE_CMD, freeze_cmd, thaw_cmd] assert exec_cmd.await_args_list[0].kwargs.get("container") in (None, "sandbox") - assert exec_cmd.await_args_list[1].kwargs.get("container") == VEHICLE_CONTAINER_NAME + assert exec_cmd.await_args_list[1].kwargs.get("container") in (None, "sandbox") + assert exec_cmd.await_args_list[2].kwargs.get("container") == VEHICLE_CONTAINER_NAME + assert exec_cmd.await_args_list[3].kwargs.get("container") == VEHICLE_CONTAINER_NAME + + async def test_create_snapshot_freeze_failure_is_loud(self, core: K7Core): + dep = MagicMock() + dep.metadata.annotations = {ANN_K7_DOCKER: "true", ANN_DOCKER_PVC: "sb-docker-lh"} + dep.status.ready_replicas = 1 + mock_apps = AsyncMock() + mock_apps.read_namespaced_deployment.return_value = dep + _setup(core, apps=mock_apps, v1=AsyncMock()) + create = AsyncMock(return_value=_ok()) + + async def exec_cmd(name, command, **kwargs): + if " -f " in command and "k7-fsfreeze" in command: + return ExecResult(exit_code=1, stdout="", stderr="EBUSY", duration_ms=0) + return _exec_ok() + + with ( + patch.object(core, "_detect_backend", new=AsyncMock(return_value="kata-qemu-longhorn")), + patch.object(core, "_create_volume_snapshot", new=create), + patch.object(core, "exec_command", new=AsyncMock(side_effect=exec_cmd)), + ): + result = await core.create_snapshot("sb", "snap1") + assert not result.success + assert "fsfreeze -f" in result.error + create.assert_not_awaited() + + async def test_create_snapshot_thaws_after_snapshot_failure(self, core: K7Core): + dep = MagicMock() + dep.metadata.annotations = {ANN_K7_DOCKER: "true", ANN_DOCKER_PVC: "sb-docker-lh"} + dep.status.ready_replicas = 1 + mock_apps = AsyncMock() + mock_apps.read_namespaced_deployment.return_value = dep + _setup(core, apps=mock_apps, v1=AsyncMock()) + fail = MagicMock() + fail.success = False + fail.error = "root snap fail" + exec_cmd = AsyncMock(return_value=_exec_ok()) + with ( + patch.object(core, "_detect_backend", new=AsyncMock(return_value="kata-qemu-longhorn")), + patch.object(core, "_create_volume_snapshot", new=AsyncMock(return_value=fail)), + patch.object(core, "exec_command", new=exec_cmd), + ): + result = await core.create_snapshot("sb", "snap1") + assert not result.success + cmds = [c.args[1] for c in exec_cmd.await_args_list] + freeze_cmd = ( + f"cp {FSFREEZE_STAGED} {FSFREEZE_VEHICLE} && chmod 755 {FSFREEZE_VEHICLE} && " + f"{FSFREEZE_VEHICLE} -f {KATA_GRAPH_MOUNT}" + ) + thaw_cmd = f"{FSFREEZE_VEHICLE} -u {KATA_GRAPH_MOUNT}" + assert cmds == ["sync", STAGE_FSFREEZE_CMD, freeze_cmd, thaw_cmd] async def test_create_snapshot_skips_sync_when_paused(self, core: K7Core): dep = MagicMock() @@ -274,7 +336,14 @@ class TestKataDockerForkReject: result = await core.fork_sandbox("src", "dst") assert not result.success assert "cannot be cloned" in result.error - assert result.error == KATA_FORK_GRAPH_REJECT + assert result.error == KFD_FORK_REJECT + + async def test_kfd_plain_fork_rejected(self, core: K7Core): + """Non-docker kfd has no PVC either — reject before a confusing 404.""" + with patch.object(core, "_detect_backend", new=AsyncMock(return_value="kata-firecracker-devmapper")): + result = await core.fork_sandbox("src", "dst") + assert not result.success + assert result.error == KFD_FORK_REJECT class TestDindImagePin: diff --git a/tests/unit/test_models.py b/tests/unit/test_models.py index a538db7..a2837d2 100644 --- a/tests/unit/test_models.py +++ b/tests/unit/test_models.py @@ -32,6 +32,7 @@ class TestSandboxConfigFromDict: "container_non_root": True, "cap_drop": ["ALL"], "cap_add": ["NET_ADMIN"], + "node_name": "k7-node-01", } cfg = SandboxConfig.from_dict(data) assert cfg.backend == "kata-qemu-longhorn" @@ -39,6 +40,7 @@ class TestSandboxConfigFromDict: assert cfg.limits == {"cpu": "2", "memory": "4Gi"} assert cfg.pod_non_root is True assert cfg.cap_add == ["NET_ADMIN"] + assert cfg.node_name == "k7-node-01" def test_ignores_unknown_keys(self): cfg = SandboxConfig.from_dict({"name": "x", "image": "img", "unknown_field": 42, "another": "val"}) diff --git a/tests/unit/test_playbook_backends.py b/tests/unit/test_playbook_backends.py new file mode 100644 index 0000000..158c609 --- /dev/null +++ b/tests/unit/test_playbook_backends.py @@ -0,0 +1,21 @@ +"""Playbook must fail closed on omitted k7_backends; none is the empty set.""" + +from pathlib import Path + +PLAYBOOK = Path(__file__).resolve().parents[2] / "src/k7/deploy/k7-install-node.yaml" +EXAMPLE = Path(__file__).resolve().parents[2] / "src/k7/deploy/inventory.ini.example" + + +def test_playbook_has_no_backend_default(): + text = PLAYBOOK.read_text() + assert "k7_backend_default" not in text + assert "empty/omitted is not" in text + assert "k7_backends=none" in text or "or none" in text + assert "[] if (k7_backends_raw | map('lower') | list == ['none'])" in text + assert "content: \"{{ k7_backends_list[0] if k7_backends_list | length > 0 else 'none' }}\"" in text + + +def test_inventory_example_documents_none_on_servers(): + text = EXAMPLE.read_text() + assert "k7_backends=none" in text + assert "[k7_servers:vars]" in text diff --git a/tests/unit/test_restore.py b/tests/unit/test_restore.py index 9e29637..77b0fa0 100644 --- a/tests/unit/test_restore.py +++ b/tests/unit/test_restore.py @@ -33,7 +33,7 @@ from k7.core.models import ( ) from k7_sdk.client import Client, SandboxProxy -TEST_KEY = "k7-test-secret-key-spec10f" +TEST_KEY = "k7-test-secret-key-restore" TEST_KEY_HASH = hashlib.sha256(TEST_KEY.encode()).hexdigest() diff --git a/tutorials/k7_hetzner_node_setup.md b/tutorials/k7_hetzner_node_setup.md index 2726b55..844d942 100644 --- a/tutorials/k7_hetzner_node_setup.md +++ b/tutorials/k7_hetzner_node_setup.md @@ -78,11 +78,11 @@ lsblk -f # spare should show no FSTYPE / no RAID members sudo add-apt-repository ppa:katakate.org/k7 sudo apt update sudo apt install k7 +k7 -V # 0.4.0 # dual-NVMe box: let the playbook auto-detect the raw spare disk. -# Launchpad PPA is still 0.2.2; install the GitHub 0.3.1 .deb (or this tree's CLI). # Run from a checkout of this repo (or Katakate/k7) so k7-api:local can build. -sudo k7 install --backend kfd,kql,k7d +sudo k7 install --backend kfd,kql,k7d,k7d-fc ``` `k7 install` provisions the LVM thin-pool on that disk for the `kfd` backend. @@ -90,8 +90,8 @@ Other backends (`kql`, `k7d`) do not need this spare disk, but keeping one raw NVMe lets you compare all backends on the same node. Two-node (one server + one agent, no `--ha`) is the same idea: write an -inventory with `k7_backends=kfd,kql,k7d` on both hosts, omit -`k7_devmapper_disk`, and run **one** `k7 install -i inventory.ini --k7d-version 0.6.0` +inventory with `k7_backends=kfd,kql,k7d,k7d-fc` on both hosts, omit +`k7_devmapper_disk`, and run **one** `k7 install -i inventory.ini --k7d-version 0.7.0` from the first master. See `src/k7/deploy/inventory.ini.example`. > **Do NOT pin the disk on dual-NVMe boxes.** NVMe enumeration @@ -113,5 +113,5 @@ from the first master. See `src/k7/deploy/inventory.ini.example`. - [ ] `ls /dev/kvm` exists - [ ] OS root is on a single disk (`findmnt /` → `/dev/nvme0n1p…`, not `/dev/md…`) - [ ] Spare disk has no filesystem (`lsblk -f` empty FSTYPE) -- [ ] `k7 install --backend kfd,kql,k7d --k7d-version 0.6.0` succeeds (auto-detects the spare disk — do not pin `--disk` on dual-NVMe) +- [ ] `k7 install --backend kfd,kql,k7d,k7d-fc --k7d-version 0.7.0` succeeds (auto-detects the spare disk — do not pin `--disk` on dual-NVMe) - [ ] `k7 create --backend kfd …` can start a sandbox diff --git a/uv.lock b/uv.lock index dd67209..7e2e019 100644 --- a/uv.lock +++ b/uv.lock @@ -1069,7 +1069,7 @@ wheels = [ [[package]] name = "k7" -version = "0.3.1" +version = "0.4.0" source = { editable = "." } dependencies = [ { name = "fastapi" },