mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 17:11:30 +00:00
Josh Siviour
eff11b611b
* fix(dhcp): release stranded leases and label them with the owning node Two paths dropped a lease from KV without returning it to the upstream server, so a source="dhcp" pool leaked addresses permanently. Manager.Start deleted an expired entry outright. Dropping the entry alone abandons the binding rather than returning it, and a server that ages leases by liveness holds the address for as long as anything answers ARP for it. Adoption now sends a bounded, best-effort RELEASE first, skipping leases with no stored offer/ack because nclient4 cannot rebuild those. Release-by-IP answered SUCCESS when the IP was absent from KV, telling the caller the address was freed when nothing went on the wire. It now returns an error. Every ReleaseIP call site logs and continues, so this surfaces the strand without wedging teardown. Leases were also unattributable upstream: chaddr is a derived HashMAC and option 61 carries that same MAC, so no field identified the host that took the lease. Option 12 is now "<node>-<client-id>" and option 60 is "mulga-spinifex/<node>", giving the lease table a per-lease identity and a per-host column to group on. HashMAC is untouched — encoding the node there would cut per-node entropy and reverse a documented decision. Observed live: 20 office-LAN addresses held by one node's orphaned OVN gateway LRPs, with no lease records and no renewals. * fix: rebind the datapath when a DHCP lease moves to a new address A lease re-issued on a different IP was persisted to KV with no notification of any kind. doAcquire replaced the lease after an expiry and doRenew accepted whatever a rebind answered with, while the gateway LRP's address is written once at attach and never revisited. Each such event burned one address permanently: the old one squatted by OVN's ARP responder with no renewer and no DHCPRELEASE, the new one renewed forever attached to nothing. Route both paths through applyLease, which releases the superseded address and invokes an IPChangeHook so the datapath can follow. vpcd registers a hook that moves the gateway LRP's networks and gateway_ip stamp, re-runs the routed leg so the VPC SNAT and host ingress route follow, and waits on the flows barrier. EIP, ENI-public and NATGW addresses are recorded outside vpcd in DescribeAddresses and DescribeInstances, so the hook reports those rather than repointing a datapath the control plane no longer agrees with. The release half still applies to every purpose, so the leak itself is closed either way. A failed rebind does not fail the lease loop: the new address is held regardless, and dropping it would strand both. * fix: reconcile API-visible records when their DHCP lease moves An EIP or auto-assigned public IP whose lease was re-issued elsewhere left DescribeAddresses and DescribeNetworkInterfaces advertising an address vpcd had already released. The datapath kept the old dnat_and_snat too, so the address was dead on both sides while the upstream server was free to hand it to someone else. vpcd cannot fix this alone: it holds the lease, but the records naming those addresses live daemon-side. Add vpc.dhcp.lease-changed, which vpcd requests and the daemon serves, dispatching to the service owning the moved address. RebindPublicIP and RebindENIPublicIP commit the new external IP through vpc.add-nat first — AddEIP scrubs the predecessor row keyed on the logical IP, so one commit is enough — and only then move the record, so a NAT failure never leaves a record advertising an address that does not route. Both refuse to move a record naming neither address, so a stale request cannot overwrite an allocation the lease does not own, and both are idempotent under duplicate delivery. A failed or unanswered reconcile propagates back to the lease loop rather than reading as success. vpc.dhcp.lease-expired, listed in the service manifest but never implemented, is replaced by the subject that now exists. * fix: dedupe retried AllocateAddress so one call takes one address EC2 AllocateAddress carries no ClientToken, and vpcd's DORA ladder can outlast the AWS client read timeout, so every SDK-level retry minted a fresh allocation ID and drew another upstream DHCP lease. The caller kept one EIP and the rest stayed leased with no record naming them, so nothing ever released them. Retries are now identified by the SDK's amz-sdk-invocation-id, which is stable across attempts of one call. The gateway lifts it off the request, NATSRequest forwards it as a header, and the daemon puts it back on the context, so the EIP service sees the same token every attempt. singleflight joins a retry to the call still in flight — the common case — and a TTL'd KV record answers retries that arrive after the first call completed. * fix: reclaim DHCP leases whose owner is gone and stop dropping expired ones Three leaks that all end the same way: an address stays leased upstream with nothing renewing it on purpose and nothing that will ever release it. Manager.Start deleted any lease that expired while vpcd was down, after releasing it. The resource it was taken for still existed, so the datapath kept forwarding on an address the upstream server now considers free — a duplicate-IP conflict waiting for the server to reissue it — and nothing ever re-acquired. Expired leases now get a renewal loop like any other, which re-DORAs on its first pass; the existing IP-change hook rebinds the datapath when the new address differs. The reaffirm RENEW is skipped for these, since a lease the server has already aged out will only fail it. Nothing reconciled leases against the resources that justify them, so a resource deleted out from under its lease left the address renewed forever. ReapOrphans sweeps them: vpcd asks the daemon over vpc.dhcp.owner-check whether the EIP, ENI or VPC still exists, and releases the lease when it does not. A gateway lease also has a router port to remove, so DetachIGW runs before the address goes back — releasing it while OVN still answers ARP for it invites the same conflict. Anything the daemon cannot resolve is reported unknown and kept: an unreachable daemon must never read as a fleet-wide deletion. The acquire ladder ran to ~62s, just past the 60s read timeout botocore defaults to, which is what turned one AllocateAddress into two leases. It now totals 40s with a 45s budget, and the DORA RPC 50s, so the caller hears the outcome instead of giving up and retrying. This is a mitigation, not a fix, for callers that send no retry token: botocore emits no amz-sdk-invocation-id, so the dedupe added earlier cannot see their retries at all. Fixes mulga-j19gj, mulga-9exe6, mulga-7qyzj. * feat: add spx get dhcp-leases so held external addresses are visible The upstream leases behind a source="dhcp" external pool lived in a per-AZ KV bucket with no read path, so working out which addresses were held, by what, and whether the owner still existed meant writing a NATS client or reading journals. That was the main obstacle to diagnosing every leak in this series. The command lists each lease with its purpose, VPC, address, MAC, holding node and remaining time, and resolves the owner over vpc.dhcp.owner-check so an orphan is visible as such. --orphans narrows the output to leaks; --skip-owner-check drops the lookup when only the addresses matter. Fixes mulga-655iu. * fix(vpcd): reconcile gateway ports onto their lease address at startup A gateway LRP configured from a static pool keeps its address when the pool is switched to source="dhcp", so the VPC forwards on an address nothing leases while the lease it holds goes unused. No lease event ever fires to correct it. vpcd now sweeps every gw-lrp lease on startup and rebinds the port to the address the lease actually holds. RebindGatewayIP is idempotent, so a port already in agreement costs one OVSDB read. * test: cover dhcp lease owner-check handlers and the lease CLI The owner-check path decides whether the reaper releases an address, so every inconclusive branch is asserted to report unknown rather than gone. * fix(dhcp): make the outcome of a stale lease release observable A stale RELEASE was logged only when the transport failed. A lease with no raw offer/ack was skipped in silence, and a RELEASE the server ignored looked identical to one it honoured, so an address left bound upstream could not be traced to either cause. The skip now warns, and a successful release logs the chaddr it used, which is what a server matches a RELEASE against. * fix(dhcp): take the RELEASE chaddr from the lease being released A server matches a RELEASE to a binding on chaddr, and the release path re-resolved it live. For a UseIfaceMAC lease whose interface MAC had changed since, that sends an address the binding never had: the server drops the RELEASE and says nothing, so the address stays held with no renewer. The chaddr now comes from the lease record, which carries what took the lease out, falling back to live resolution only when the record has none.
379 lines
12 KiB
YAML
379 lines
12 KiB
YAML
# Service interface manifest.
|
|
#
|
|
# Source of truth for "which services produce/consume which NATS subjects"
|
|
# and "which e2e suites cover which services". Consumed by
|
|
# tests/e2e/select-suites (Bead 4) to derive targeted suite selection from
|
|
# a PR diff.
|
|
#
|
|
# See docs/development/improvements/e2e-targeted-suite-selection.md.
|
|
#
|
|
# Editing rules
|
|
# -------------
|
|
# - subscribes / publishes: subjects this service receives / sends. Both
|
|
# lists use the literal subject string, or a "<prefix>.*" wildcard for
|
|
# pattern subscriptions (e.g. "ebs.snapshot.*").
|
|
# - depends_on: subjects this service requires from peers. Used by the
|
|
# selector to compute "if X changes, which suites might break".
|
|
# - Subjects must match what is actually referenced in source. The lint
|
|
# in Bead 5 greps `"(ec2|vpc|ebs|s3|iam|elbv2|spinifex)\.[a-zA-Z._-]+"`
|
|
# and fails if any subject is absent from this manifest.
|
|
|
|
version: 1
|
|
|
|
services:
|
|
|
|
awsgw:
|
|
path: spinifex/services/awsgw
|
|
# TLS frontend, SigV4 auth. Pure router: HTTP → NATS request/response.
|
|
# No subscribes; only publishes (via NATSRequest).
|
|
subscribes: []
|
|
publishes:
|
|
- "ec2.*"
|
|
- "elbv2.*"
|
|
- "acm.*"
|
|
- "iam.*"
|
|
depends_on:
|
|
- "ec2.*"
|
|
- "elbv2.*"
|
|
- "acm.*"
|
|
- "iam.*"
|
|
|
|
spinifex_daemon:
|
|
path: spinifex/services/spinifex
|
|
# Orchestrator. Owns most ec2.* + all elbv2.* + iam event publication.
|
|
# Translates VPC / EBS / S3 operations into vpc.* / ebs.* / s3.* and
|
|
# forwards to vpcd / viperblockd / predastore.
|
|
#
|
|
# Source layout: spinifex/services/spinifex (entry), spinifex/daemon
|
|
# (subscribe registration), spinifex/handlers/{ec2,elbv2,iam} (per-
|
|
# subject handler bodies). Selector treats all four paths as belonging
|
|
# to this service via the `additional_paths` field.
|
|
additional_paths:
|
|
- spinifex/daemon
|
|
- spinifex/handlers/ec2
|
|
- spinifex/handlers/elbv2
|
|
- spinifex/handlers/acm
|
|
- spinifex/handlers/iam
|
|
subscribes:
|
|
# EC2 instance lifecycle.
|
|
- "ec2.RunInstances"
|
|
- "ec2.DescribeInstances"
|
|
- "ec2.DescribeStoppedInstances"
|
|
- "ec2.DescribeTerminatedInstances"
|
|
- "ec2.DescribeInstanceAttribute"
|
|
- "ec2.DescribeInstanceStatus"
|
|
- "ec2.DescribeInstanceTypes"
|
|
- "ec2.ModifyInstanceAttribute"
|
|
- "ec2.ModifyVolume"
|
|
- "ec2.start"
|
|
- "ec2.terminate"
|
|
- "ec2.launch"
|
|
- "ec2.cmd.*"
|
|
# EC2 VPC / network ops handled by daemon before delegation to vpcd.
|
|
- "ec2.CreateVpc"
|
|
- "ec2.DescribeVpcs"
|
|
- "ec2.CreateSubnet"
|
|
- "ec2.CreateInternetGateway"
|
|
- "ec2.CreateKeyPair"
|
|
- "ec2.CreateImage"
|
|
- "ec2.DescribePlacementGroups"
|
|
# Cluster coordination subjects.
|
|
- "ec2.ReserveClusterNode"
|
|
- "ec2.FinalizeClusterInstances"
|
|
- "spinifex.node.status"
|
|
- "spinifex.node.vms"
|
|
- "spinifex.nodes.discover"
|
|
- "spinifex.cluster.shutdown.progress"
|
|
# Coordinated cluster-shutdown phases (fan-out, no queue group).
|
|
- "spinifex.cluster.shutdown.gate"
|
|
- "spinifex.cluster.shutdown.drain"
|
|
- "spinifex.cluster.shutdown.storage"
|
|
- "spinifex.cluster.shutdown.persist"
|
|
- "spinifex.cluster.shutdown.infra"
|
|
- "spinifex.storage.config"
|
|
# ELBv2 control plane (handlers/elbv2/service_nats.go topic map).
|
|
- "elbv2.CreateLoadBalancer"
|
|
- "elbv2.DeleteLoadBalancer"
|
|
- "elbv2.DescribeLoadBalancers"
|
|
- "elbv2.CreateTargetGroup"
|
|
- "elbv2.DeleteTargetGroup"
|
|
- "elbv2.DescribeTargetGroups"
|
|
- "elbv2.RegisterTargets"
|
|
- "elbv2.DeregisterTargets"
|
|
- "elbv2.DescribeTargetHealth"
|
|
- "elbv2.CreateListener"
|
|
- "elbv2.DeleteListener"
|
|
- "elbv2.DescribeListeners"
|
|
# ACM-alike certificate store (handlers/acm/service_nats.go topic map).
|
|
- "acm.ImportCertificate"
|
|
- "acm.DescribeCertificate"
|
|
- "acm.ListCertificates"
|
|
- "acm.DeleteCertificate"
|
|
- "elbv2.AddListenerCertificates"
|
|
- "elbv2.RemoveListenerCertificates"
|
|
- "elbv2.DescribeListenerCertificates"
|
|
- "elbv2.DescribeSSLPolicies"
|
|
publishes:
|
|
# → vpcd
|
|
- "vpc.create"
|
|
- "vpc.delete"
|
|
- "vpc.create-subnet"
|
|
- "vpc.delete-subnet"
|
|
- "vpc.create-sg"
|
|
- "vpc.delete-sg"
|
|
- "vpc.update-sg"
|
|
- "vpc.create-port"
|
|
- "vpc.delete-port"
|
|
- "vpc.update-port-sgs"
|
|
- "vpc.igw-attach"
|
|
- "vpc.igw-detach"
|
|
- "vpc.add-nat-gateway"
|
|
- "vpc.delete-nat-gateway"
|
|
- "vpc.add-nat"
|
|
- "vpc.delete-nat"
|
|
- "vpc.dhcp.acquire"
|
|
- "vpc.dhcp.release"
|
|
# → viperblockd
|
|
- "ebs.mount"
|
|
- "ebs.unmount"
|
|
- "ebs.delete"
|
|
- "ebs.sync"
|
|
- "ebs.snapshot.*"
|
|
- "ebs.config"
|
|
- "ebs.config.*"
|
|
# → predastore (S3 / AMI / keypair store)
|
|
- "s3.*"
|
|
# IAM lifecycle event
|
|
- "iam.account.created"
|
|
depends_on:
|
|
- "vpc.*"
|
|
- "ebs.*"
|
|
- "s3.*"
|
|
|
|
vpcd:
|
|
path: spinifex/vpcd
|
|
# OVN-backed VPC / subnet / SG / NAT / IGW data plane.
|
|
subscribes:
|
|
- "vpc.create"
|
|
- "vpc.delete"
|
|
- "vpc.create-subnet"
|
|
- "vpc.delete-subnet"
|
|
- "vpc.create-sg"
|
|
- "vpc.delete-sg"
|
|
- "vpc.update-sg"
|
|
- "vpc.create-port"
|
|
- "vpc.delete-port"
|
|
- "vpc.update-port-sgs"
|
|
- "vpc.igw-attach"
|
|
- "vpc.igw-detach"
|
|
- "vpc.add-nat-gateway"
|
|
- "vpc.delete-nat-gateway"
|
|
- "vpc.add-nat"
|
|
- "vpc.delete-nat"
|
|
- "vpc.dhcp.acquire"
|
|
- "vpc.dhcp.release"
|
|
publishes:
|
|
- "vpc.port-status"
|
|
- "vpc.dhcp.lease-changed"
|
|
- "vpc.dhcp.owner-check"
|
|
depends_on: []
|
|
|
|
viperblockd:
|
|
path: spinifex/services/viperblockd
|
|
# EBS data plane. Backed by the viperblock storage engine in the
|
|
# mulga/viperblock submodule. Submodule pointer bump → selector forces
|
|
# "run all" (contract changes can be invisible at the spinifex source
|
|
# level).
|
|
subscribes:
|
|
- "ebs.mount"
|
|
- "ebs.unmount"
|
|
- "ebs.delete"
|
|
- "ebs.sync"
|
|
- "ebs.snapshot.*"
|
|
- "ebs.config"
|
|
- "ebs.config.*"
|
|
publishes:
|
|
- "ebs.mount.response"
|
|
- "ebs.unmount.response"
|
|
depends_on: []
|
|
|
|
predastore:
|
|
path: spinifex/services/predastore
|
|
# S3 / AMI / keypair storage. External submodule under mulga/predastore;
|
|
# this dir contains only the spinifex-side adapter. Submodule pointer
|
|
# bump → selector forces "run all".
|
|
subscribes:
|
|
- "s3.*"
|
|
publishes: []
|
|
depends_on: []
|
|
|
|
nats:
|
|
path: spinifex/services/nats
|
|
# NATS broker bootstrap / cluster config. Not a NATS pub/sub
|
|
# participant. Any change here is infra-path; selector forces
|
|
# "run all".
|
|
subscribes: []
|
|
publishes: []
|
|
depends_on: []
|
|
|
|
spinifexui:
|
|
path: spinifex/services/spinifexui
|
|
# Web UI. Talks to awsgw over HTTP, never NATS. Diffs limited to this
|
|
# dir do not need any e2e suite to run.
|
|
subscribes: []
|
|
publishes: []
|
|
depends_on: []
|
|
covers_no_suite: true
|
|
|
|
|
|
# Fixture → service mapping. Each Ensure* helper in
|
|
# tests/e2e/harness/fixtures.go exercises this set of services; the
|
|
# Bead 4 selector AST-scans test files for harness.Ensure* call sites
|
|
# and resolves coverage via this block.
|
|
fixtures:
|
|
|
|
EnsureKeyPair:
|
|
services: [spinifex_daemon, predastore]
|
|
|
|
EnsureAMI:
|
|
services: [spinifex_daemon, predastore, viperblockd]
|
|
|
|
EnsureDefaultVPC:
|
|
services: [vpcd, spinifex_daemon]
|
|
|
|
EnsureSubnet:
|
|
services: [vpcd, spinifex_daemon]
|
|
|
|
EnsureSG:
|
|
services: [vpcd, spinifex_daemon]
|
|
|
|
EnsureInstance:
|
|
services: [spinifex_daemon, viperblockd, vpcd]
|
|
|
|
EnsureVolume:
|
|
services: [spinifex_daemon, viperblockd]
|
|
|
|
EnsureSnapshot:
|
|
services: [spinifex_daemon, viperblockd, predastore]
|
|
|
|
EnsureNATGateway:
|
|
services: [vpcd, spinifex_daemon]
|
|
|
|
EnsureLoadBalancer:
|
|
services: [spinifex_daemon, awsgw]
|
|
|
|
|
|
suites:
|
|
|
|
e2e-cert:
|
|
path: tests/e2e/cert
|
|
covers: [awsgw, spinifex_daemon, vpcd, predastore]
|
|
|
|
e2e-lb:
|
|
path: tests/e2e/lb
|
|
covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
|
|
# Exercises full ELBv2 control plane (handlers/elbv2) + data plane
|
|
# (LB agent on the system VM). ELBv2 is part of spinifex_daemon in
|
|
# this manifest; revisit if it splits into a standalone service.
|
|
|
|
e2e-single:
|
|
path: tests/e2e/single
|
|
covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
|
|
# Full single-node lifecycle: VPC, AMI, keypair, instance, volume,
|
|
# snapshot, IAM. Currently runs all phases sequentially under one
|
|
# `TestSingleNode`. Bead 3 (single-fixture-migration) splits this
|
|
# into per-test fixture-driven coverage so the selector can pick
|
|
# subtests, not just whole suites.
|
|
|
|
e2e-iam:
|
|
path: tests/e2e/iam
|
|
covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
|
|
# "Everything IAM": the IAM/STS control plane plus the instance-identity
|
|
# tests (instance-profile association + IMDS), which boot guest VMs — OVN
|
|
# via vpcd, AMI via predastore, root volume via viperblockd, lifecycle via
|
|
# spinifex_daemon. Single-mode, VM-booting leg.
|
|
|
|
e2e-ecr:
|
|
path: tests/e2e/ecr
|
|
covers: [awsgw, spinifex_daemon, predastore]
|
|
# ECR control plane (handlers/ecr, gateway/ecr*) + OCI data plane: blobs and
|
|
# manifests in predastore, control + auth bridge in awsgw, KV via
|
|
# spinifex_daemon. docker/crane/skopeo data-plane subtests skip when the
|
|
# client binary or registry-host resolution is absent.
|
|
|
|
e2e-bedrock:
|
|
path: tests/e2e/bedrock
|
|
covers: [awsgw]
|
|
# Ochre (AWS-Bedrock) control plane: static catalog list/get served
|
|
# straight from awsgw (gateway/bedrock.go, gateway/bedrock/) — no NATS
|
|
# hop to spinifex_daemon. Converse (Anthropic-direct / self-host vLLM)
|
|
# subtests are gated on backend availability and skip by default.
|
|
|
|
e2e-ecs:
|
|
path: tests/e2e/ecs
|
|
covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
|
|
# ECS data plane end-to-end: cluster + a real container instance launched
|
|
# from the spinifex-ecs-node AMI (boots, registers over the gateway, runs
|
|
# tasks through containerd), standalone tasks in awsvpc/bridge/host modes,
|
|
# and an awsvpc service fronted by an ELBv2 target group. Boots a guest VM,
|
|
# so vpcd (OVN + awsvpc ENI), viperblockd (root volume), predastore (AMI)
|
|
# are all in the path.
|
|
|
|
e2e-rds:
|
|
path: tests/e2e/rds
|
|
covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
|
|
# RDS create/describe/connect: CreateDBInstance boots a dual-NIC DB VM from
|
|
# the engine AMI (predastore) with a Viperblock data volume (viperblockd) and
|
|
# a customer ENI (vpcd), and the reconciler flips it to available on the
|
|
# in-guest agent's first healthy heartbeat over awsgw. Opt-in beyond
|
|
# SPINIFEX_E2E: DeleteDBInstance is not implemented, so a run leaves the
|
|
# instance behind.
|
|
|
|
e2e-ddil:
|
|
path: tests/e2e/ddil
|
|
covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
|
|
# DDIL = degraded / disconnected / intermittent / low-bandwidth.
|
|
# Exercises the full daemon stack under network failure injection;
|
|
# any service touching the wire path is in scope.
|
|
|
|
# Suites below are bash-only today. Entries activate as the Go ports
|
|
# land (Bead 2 fixtures unblock new-shape ports for multinode etc.).
|
|
#
|
|
# e2e-multinode:
|
|
# path: spinifex/tests/e2e/multinode
|
|
# covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
|
|
# e2e-reboot:
|
|
# path: spinifex/tests/e2e/reboot
|
|
# covers: [spinifex_daemon, viperblockd]
|
|
# e2e-tofu:
|
|
# path: spinifex/tests/e2e/tofu
|
|
# covers: [awsgw, spinifex_daemon, vpcd]
|
|
# e2e-bm:
|
|
# path: spinifex/tests/e2e/baremetal
|
|
# covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
|
|
#
|
|
# e2e-natuplink activates once CI can provision an external_mode=nat
|
|
# node (setup-ovn.sh --nat-uplink + spx admin init --external-mode=nat).
|
|
# e2e-natuplink:
|
|
# path: spinifex/tests/e2e/natuplink
|
|
# covers: [awsgw, spinifex_daemon, vpcd, viperblockd, predastore]
|
|
|
|
|
|
# Special-case rules consumed by the selector (Bead 4). Hard-coded in
|
|
# the tool, listed here for visibility.
|
|
#
|
|
# infra_paths_force_all:
|
|
# - spinifex/services/nats/**
|
|
# - spinifex/tests/e2e/harness/**
|
|
# - spinifex/tests/e2e/lib/**
|
|
# - spinifex/.github/workflows/e2e-go.yml
|
|
# - spinifex/.github/scripts/**
|
|
# - spinifex/Makefile
|
|
# - mulga/scripts/tofu-cluster/**
|
|
#
|
|
# submodule_force_all:
|
|
# - mulga/predastore (submodule pointer)
|
|
# - mulga/viperblock (submodule pointer)
|
|
#
|
|
# many_services_force_all_threshold: 3 # ≥ N services touched → run all
|