Salts restart per operation, so tag, untag and update cases only reached a resource when the same-numbered create case had made one. Create an IAM, ECS and ECR fixture per run and point every operation but a create or delete at it, and give ECR's policy and manifest documents values AWS accepts.
Clear the warnings the conformance suite caused itself, so what remains is a real divergence from AWS:
- Seed conditionally required members into the base request of RDS CreateDBInstance and RestoreDBInstanceFromDBSnapshot, ECS CreateCapacityProvider and CreateService, and EC2 CreateCapacityReservation.
- Give loosely modelled members values AWS accepts: ELBv2 ProtocolVersion and Matcher.HttpCode, EKS policyArn, addonName and access-entry type, EC2 SourceRegion and STS AccessKeyId.
- Import a CA-signed certificate, key and chain in ACM ImportCertificate.
- Rewrite ACM's lookahead domain pattern as RE2 so RequestCertificate is generated, and give the integration ACM service a tenant CA permitting example.com.
- Run the service daemon-lite behind the cross-tenant boundary test.
- Accept ELBv2's common parameter validation errors in responses.
Tests added since main used in-package test packages, failing the
test-package check. Convert them to <pkg>_test with export_test.go
hooks; the ecs-agent main package keeps an in-package marker.
The internal report listed every unimplemented operation together, so the 87
EC2 operations declared not applicable sat in the same list as the work, where
anything reading it for candidates would pick them up.
They are now listed apart, grouped under the reason they were declared with,
and the counts line reports them as not applicable rather than as gaps.
Multi-AZ clusters, read replicas and point-in-time restore were declared not
applicable, which says the platform will never serve them. All three are
planned, so the pages claimed something the roadmap contradicts.
Only a declaration now publishes that claim. A handler that refuses an
operation today establishes nothing about tomorrow, so it reads as the gap it
is rather than a permanent one.
Splitting them into a second table repeated a reason on every row that shared
it and separated an operation from the rest of the surface it belongs to. One
table carries every operation the platform answers, served or refused, and the
reasons return to numbered footnotes below it.
S3 was the one served surface with no coverage page. The gateway has no
operation-name dispatch table for it, so there was nothing to compare against
the model.
Predastore now declares the operations each of its S3 routes answers, so the
generator compares that inventory the same way it compares the other nine
services. The page credits Predastore rather than Spinifex, and its intro
separates what is routed from what conforms, pointing at the ceph/s3-tests
report for the second.
The index listed a count per service and left the reader to add them up. It
now opens with the total and closes the table with it, counting only the
services it links.
The coverage pages led with a percentage and an "x of y" count, and every
table listed the modelled operations Spinifex does not implement, which reads
as a scorecard against AWS rather than a statement of what the platform serves.
A published page now opens with the operation count alone, tables the
operations the gateway serves, and carries a separate not-applicable table with
each reason inline instead of footnotes. The unimplemented operations and the
counts move to docs/coverage/internal-report.md, which no docs-site slug points
at. A service with no implemented operations has no publishable page.
ec2rolecreds implements aws.HandleFailRefreshCredentialsCacheStrategy, which
aws.CredentialsCache prefers over returning the refresh error. When the cached
credential is expired or within five minutes of expiring and the refresh fails,
it hands back the same access key, secret and session token with Expires pushed
a random five to fifteen minutes out, reported as a success. Every later
refresh fails the same way, so ecs-agent signs with a credential the control
plane has already rejected, indefinitely, and nothing surfaces it.
The cache picks its strategies by asserting on the provider's dynamic type, so
returning aws.CredentialsProvider from NewProvider does not disarm it. Wrap the
provider in a type whose method set omits HandleFailToRefresh, and the cache
falls through to returning the error. AdjustExpiresBy is forwarded on purpose:
it only fires under an ExpiryWindow, which no caller sets today, but dropping
it silently would change behaviour for whoever sets one.
Fixed in imdscreds rather than at the one exposed call site, so a future caller
that wraps this provider in a cache is safe by construction.
ecr-credential-provider was never affected: it calls Retrieve uncached.
Two comments claimed the hazard was ec2rolecreds capping Expires to an hour.
That cap only ever shortens a too-far-future expiry and is unrelated.
* refactor(test): route duplicated NATS bootstraps through testutil
Thirteen test sites hand-rolled a NATS or JetStream server bootstrap that
spinifex/testutil already provides, and one seedKV clone duplicated
testutil.SeedKV. Route them through the shared helpers.
Sites whose options genuinely diverge are left alone: fixed ports needed for
reconnect and restart scenarios, custom TLS and token auth, a custom
MaxPayload, and TestMain entry points that have no testing.TB for t.Cleanup.
The natgw and routetable seedKV clones use CreateOrUpdateKeyValue rather than
CreateKeyValue, so their failure behaviour differs and they stay as they are.
Also moves eight test files to external test packages where they declare no
helper used by a sibling in the same package. Coverage is unaffected: Go
compiles internal and external test packages into one binary per package, so
attribution is identical either way.
* chore(test): require new test files to use an external test package
Adds a preflight check that fails when a newly added test file declares an
in-package test. External test packages bind tests to the exported API and
let packages share test doubles, which an in-package double cannot be. Only
added files are checked, so the existing in-package tests are untouched.
e2e and integration suites are exempt: both share a package-level fixture
built in main_test.go, so their files cannot move individually. A file that
needs unexported access can opt out with a //test:in-package comment.
Takes iampolicy, masterkey and sigv4 from github.com/mulgadc/bluebottle rather
than predastore, so the control plane no longer depends on the S3 server for
shared auth and policy code.
* ci: skip building unused E2E suite binaries in e2e.yml's single/multi jobs
These two jobs immediately rebuild suite-aware binaries after Build
Install Artifacts and never consume the tar's _bin contents, so the
cross-compiled suite build was pure waste. build-install-artifacts.sh
gains an optional --skip-e2e-build flag (positional contract unchanged);
e2e-nightly.yml and the other callers keep building the full set.
* refactor: replace hand-rolled IMDS client with AWS SDK v2
internal/imdscreds, cmd/ecs-agent's metadata fetch and its credentials
provider used a hand-written IMDSv2 token/GET flow with manual RFC3339
expiry parsing. A malformed Expiration silently fell back to "never
expires" instead of failing.
Replace it with feature/ec2/imds, credentials/ec2rolecreds and
aws.CredentialsCache, following the internal/gwsign/signer.go
precedent. An unparseable Expiration is now a hard error from Fetch,
covered by TestFetch_BadExpirationErrors. Output shapes to callers are
unchanged, and IMDS base URLs ending in /latest keep working.
* refactor: derive predastore bind config from spinifex.toml in spx itself
predastore-start.sh awk-parsed spinifex.toml for the bind host/port and
node_id, exported them as env vars, then exec'd spx. Move that
derivation into predastoreStartCmd via derivePredastoreBind, which
reads the raw (non-DIAL-normalized) nodes.<node>.predastore section,
defaulting node_id to -1 (co-located) when the key is absent, same as
the script. An explicit --host/--port/--node-id flag or
SPINIFEX_PREDASTORE_* env var still overrides the derived value.
Delete the now-unused script and point the systemd unit's ExecStart
directly at spx service predastore start, matching the
spinifex-viperblock.service precedent.
A tool whose child dies by signal can still exit 0 through go tool, so
govulncheck printed 'ok' after being OOM-killed and preflight passed
without a vulnerability scan having run. Route lint, govulncheck and
nilaway through scripts/run-gate.sh, which fails on a non-zero exit or
on signal-death output.
* feat(eks): reconciler in-place control-plane restart on sustained degradation
Phase 1 of EKS CP recovery (siv-243). The ACTIVE-branch reconciler only
recorded health; a CP VM that died (OOM/host crash) left the cluster wedged
ACTIVE-but-unreachable with no recovery path. Add a CPInstanceControl surface
and bounded in-place restart: once an ACTIVE cluster's CP stays unhealthy past
a grace window, StartInstance is issued (backoff-spaced, capped) when the
instance is stopped/error, re-mounting the same root volume so embedded etcd
survives. Degradation stays AWS-faithful ACTIVE + a ClusterHealth issue rather
than a non-AWS DEGRADED status enum. cpControl nil keeps the reflect-only path.
Wiring of a real CPInstanceControl in the daemon and the error-state
manager-eviction fix are the remaining Phase-1 integration steps; Phase 2
(etcd snapshot cadence + restore-snapshot) is tracked separately.
* feat(eks): wire control-plane restart into the reconciler via instance service
Threads a CPInstanceControl into the per-cluster reconciler (siv-243 Phase 1).
cpControlAdapter binds the daemon instance service + account: InstanceState
reads the CP's EC2 state via DescribeInstances, StartInstance restarts a stopped
CP via StartStoppedInstance, which loads from the shared stopped store and
re-mounts the same root volume so embedded etcd survives. Wired as
EKSServiceDeps.CPControl = d.instanceService and passed through spawnReconciler.
Recovers a cleanly-stopped control plane co-located with the reconciler leader.
Remaining: DescribeInstances is node-local so a cross-node stopped CP needs an
aggregating describe; and an error/crashed CP (not in the stopped store) needs a
dedicated error-state restart path — both tracked as Phase-1 follow-ups.
* feat(eks): cross-node CP describe + error-state control-plane restart
The reconciler's control-plane recovery was wired to the node-local instance
service: DescribeInstances only saw the leader's node and StartStoppedInstance
only handled a cleanly-stopped CP from the shared KV. A crashed/OOM CP ends in
error state, still resident in its owning node's manager map (not the stopped
store), so recovery never reached it — the actual incident in the recovery plan.
Wire EKSServiceDeps.CPControl to a NATS-backed eksCPControl. DescribeInstances
fans out across every host (gateway scatter-gather) so the reconciler observes a
CP on any node. RecoverInstance drives the gateway StartInstances two-step:
ec2.cmd.<id> recovers a live error/running owner in place (the owner stays
subscribed through the crash; vmMgr.Start relaunches error->pending, re-mounting
the etcd root volume), falling back to ec2.start shared-KV rehydration for a
stopped CP.
The CP VM is ManagedBy=EKS, hidden from customer DescribeInstances; relax that
hide so the owning account resolves its own managed CP when targeting it by
explicit instance ID (bulk listings still hide it). EC2APIState already maps the
internal error state to the AWS closed-enum stopped, which cpRestartable accepts.
* fix(eks): reconcile CP as system account, not customer
The control-plane VMs are launched under the system account
(admin.SystemAccountID()), not the customer account that owns the cluster
record. spawnReconciler was binding cpControlAdapter to the customer account,
so InstanceState/RecoverInstance ran as the customer — which cannot see or own
its own cluster's system-owned CP VMs. DescribeInstances returned zero and the
reconciler logged 'control-plane instance not found', never restarting a wedged
CP. Bind the adapter to the system account instead.
Revert the earlier DescribeInstances managed-VM relaxation: it was based on the
mistaken assumption that CP VMs are customer-owned. Visibility is resolved by
the system-account adapter, so the explicit-ID escape hatch is dead code.
* fix(vm): pre-create mgmt tap on disk-boot restart path
The direct-boot branch pre-creates the management tap via NetworkPlumber.SetupTap
(owned by the daemon euid), but the disk-boot/else branch only appended the QEMU
tap,id=mgmt0 device without creating the tap. The daemon runs as a non-root user
without CAP_NET_ADMIN, so QEMU can only attach a tap it already owns; on the
disk-boot path QEMU tried to create /dev/net/tun itself and failed with
'Operation not permitted', failing every stopped-instance restart (StartInstances
returned ServerInternal). This blocked EKS control-plane recovery: the reconciler
found the stopped CP and issued the restart, but the VM never booted.
Call SetupTap for the mgmt tap in the disk-boot branch, mirroring direct-boot.
* feat(eks): recover all HA control-plane members, not just primary
The reconciler restarted only ControlPlaneInstanceID (the primary CP
member), so on an HA cluster the other server VMs stayed stopped and
etcd never regained quorum (1/3) — apiserver healthz stayed fail even
after recovery ran. maybeRecoverControlPlane now iterates every member
in ControlPlaneNodes (falling back to the scalar ID for single-node
clusters), restarting each stopped/error member per pass while
preserving the grace window, attempt cap, and backoff gates.
Refs mulga-siv-509
* fix(eks): keep cloud-init polling IMDS so primary CP gets its user-data
The eks-node system VM's user-data is delivered only via IMDS (the
NoCloud seed was retired in #411). IMDS (169.254.169.254) is reachable
only once mulga-mgmt-net brings up the data NIC via DHCP and the per-tap
br-imds datapath reconciles, which on the first-launched (primary) CP
can lag cloud-init's stock 120s probe window. With the default max_wait
cloud-init falls back to DataSourceNone, permanently losing the launch
user-data (no SPINIFEX_K3S_ROLE -> no k3s server) and wedging the cluster
in CREATING.
Bake a cloud.cfg.d drop-in that pins DataSourceEc2 and raises max_wait to
600s so cloud-init keeps polling until the datapath is ready instead of
falling through to None. Worst case a slow boot, never a lost role.
* fix(eks): warm IMDS instance-role creds before first SigV4 sign
On IMDS-credentials mode the on-VM SigV4 helpers (eks-token-webhook,
eks-gateway-publish, mulga-eks-state-report) sign via the SDK default
chain, which resolves in-VM IMDS instance-role creds. The per-tap
br-imds datapath lags VM boot by minutes (observed ~264s), and the SDK's
IMDS client fast-fails each probe (250ms dial), so early Retrieve calls
return 'context deadline exceeded' / 'no EC2 IMDS role found'.
eks-gateway-publish only retries for 30x5s=150s, shorter than the
datapath warm time, so it exhausts its budget and drops the
bootstrap/state publish -> cluster hangs in CREATING.
The credential chain is sound (the CP role trusts ec2.amazonaws.com and
AssumeRoleForInstance evaluates that principal); the failure is timing.
Add gwsign.Signer.EnsureCredentials and block in eksgw.New until the
instance-role creds first retrieve (2s interval, 10m ceiling aligned
with cloud-init max_wait), so every helper's first signed request
already holds creds instead of racing a cold datapath. Static-key mode
is unaffected.
* fix(eks): retry CP data-NIC DHCP until cross-host datapath is up
A single ~16s DHCP one-shot on the primary data NIC lost the race against
the cross-host OVN datapath (logical-port binding + per-tap reconcile on the
remote placement host), stranding the NIC IP-less so the Ec2 datasource never
reached IMDS and the CP member wedged. Add dhcp_acquire: retry the one-shot on
a budget aligned with cloud-init's Ec2 max_wait (600s), confirming an inet
address before success. Each attempt still exits leaving no daemon, so
cloud-init's later ENI management is unaffected.
* fix(eks): drop reconciler registry entry when Run self-exits
ReconcilerRegistry.Spawn only cleaned up its holder on ctx cancel
(Stop/StopAll). When a ClusterReconciler's Run returned on its own — a
terminal cluster state or a lost lease — nothing cancelled the ctx, so
the holder leaked and a later same-name create silently no-opped at the
head of Spawn, never reconciling to ACTIVE.
Thread a finished channel from RunClusterReconciler's Run goroutine back
to Spawn's watcher, which now cleans up on whichever fires first: an
external cancel or Run exiting. On self-exit it deletes its own holder
and releases the lease, so a fresh same-name create re-spawns and
reconciles.
* fix(eks): run mulga-mgmt-net in boot runlevel before cloud-init-local
CP user-data is delivered only via IMDS since the NoCloud seed was retired
(#411), so cloud-init DataSourceEc2 must reach 169.254.169.254. mulga-mgmt-net
DHCPs the data NIC that makes IMDS reachable, but it was added to the default
runlevel and only declared `before cloud-init` (the network-stage service).
cloud-init-local therefore ran its init-local Ec2 crawl first with no data-NIC
route and burned its whole probe window (~4 min) before mulga-mgmt-net brought
eth0 up, regressing CP bringup from ~1m50s to ~5m54s.
Add mulga-mgmt-net to the boot runlevel in setup.sh instead of default, drop it
from ENABLE_SERVICES, and order it `before cloud-init-local` so the data NIC and
IMDS route are up before the init-local crawl. Keeps the IMDS-only user-data
path; the raised Ec2 max_wait stays as a safety net for host br-imds datapath lag.
* fix(vm): refresh LaunchTime on crash-restart so watchdog spares relaunch
RestartCrashedInstance re-launched a crashed VM without refreshing
Instance.LaunchTime. The pending watchdog measures now-LaunchTime against
PendingWatchdogTimeout, so the stale original launch time tripped
launch_timeout on the next tick and terminated the crash-restart
mid-flight. Refresh LaunchTime under the state lock, mirroring the
daemon-restart restore path (restore.go), so the relaunch gets a fresh
timeout window. Adds a regression test asserting LaunchTime advances.
* feat(eks): reconcile control-plane member count after a lost member
The cluster reconciler self-heals a stopped/error control-plane VM in
place but never replaced a terminated/gone member, so an HA cluster ran
below haControlPlaneCount permanently. Add a member-count reconcile pass
that provisions one replacement CP joining the surviving etcd quorum.
- CPProvisioner + EKSServiceImpl.ProvisionReplacementCP: replay the
persisted create template on a free host, the single-node analogue of
the spread join branch, with rotating creds re-derived.
- ControlPlaneTemplate persisted on ClusterMeta at create (per-node and
rotating-cred fields cleared). SwapControlPlaneMember CAS swaps the dead
node for the replacement and refreshes the [0] scalar mirrors.
- maybeReplaceControlPlaneMember fires only when no member is restartable
and one is truly lost, a healthy majority survives, bounded by grace,
backoff and an attempt cap; placement excludes live-member hosts.
- Replacement prunes the superseded member from etcd once joined:
k3s-first-boot deletes the stale Node (EKS_ETCD_PRUNE_PEER_IP) so quorum
width returns to N. json tags added to K3sServerInput (now persisted).
* feat(eks): in-guest control-plane self-diagnosis on healthz fail
When a control-plane apiserver reports unhealthy, the reconciler only
saw healthz=fail with no cause, and the guest has no on-demand command
channel (serial root login is password-gated, no guest agent). Make the
CP self-diagnose and relay the cause through the one-way state report.
- mulga-eks-state-report.sh: diagnose() collects failing /readyz?verbose
subchecks, isolated /healthz/etcd reachability, and etcd-db-dir free
space; on fail write a verbose block to the host-captured serial
console and add a JSON-safe reason to the state payload.
- ServerStateReport: add Reason (omitempty; empty from a healthy CP or
an older AMI predating the field).
- cluster_reconciler observe(): surface the reason in the health issue.
- Tests: POSIX unit test for the agent (stubbed kubectl/df); Go tests
for the reason-bearing and terse health-issue paths.
* feat(eks): 15-min etcd snapshot cadence alongside nightly
Nightly alone is too coarse a recovery point for a young cluster. Install the
same mulga-eks-etcd-snapshot script into a second Alpine crond periodic dir
(/etc/periodic/15min) so it also runs on a ~15-minute RPO window; the script
picks its cadence tier from the dir it is invoked from ($0 sniff) and encodes
the tier in the snapshot key (etcd-{tier}-{ts}.snap), so a predastore S3
lifecycle rule can retain the frequent rolling window shallow and the nightly
copies deep. Server-role only (agents exit early); no restore-path change.
Add a self-contained POSIX unit test (stub k3s/curl/logger, symlinked crond
dirs to exercise the real $0 tier sniff) covering agent no-op, frequent + daily
key prefixes, local-snapshot prune, and quiet no-op without creds.
* feat(eks): auto-reform wedged etcd quorum via boot-time cluster-reset
An HA k3s control plane does not self-heal after a simultaneous full-CP
restart: embedded etcd never reforms quorum (etcd:unreachable sustained),
and in-place VM restarts cannot fix it. Add a last-resort escalation that
drives the documented k3s cluster-reset recovery, pull-based so the host
never needs an inbound channel into the guest.
Reconciler: maybeReformEtcdQuorum escalates only when every CP member is
VM-running but the health issue carries etcd:unreachable, the cluster is
HA, and a grace window has elapsed. It sets per-member recovery directives
(seed = cluster-reset, others = wipe-rejoin), epoch-bumped and keyed by
instance ID, then restarts the members. Bounded by grace/backoff/attempts;
a recovered cluster clears the directives once. Enabled via
WithEtcdResetRecovery alongside the existing CP control wiring.
Directive plane: per-member KV directive {epoch, action, snapshot} with
store helpers, an EKS service Get/Set pair over NATS-RPC, and an
internal-recovery gateway route (system-cred carve-out, same as
internal-addons). The CP server instance role gains eks:GetRecoveryDirective.
Guest: mulga-eks-k3s-recovery is a pre-k3s oneshot (server role only) that
fetches its directive through the AWS gateway (eks-gateway-fetch -resource
recovery), epoch-guards one-shot application, then runs
`k3s server --cluster-reset` (optionally restoring a predastore snapshot)
or wipes the local etcd datastore to rejoin the reset seed. Wired into the
image manifest, setup.sh perms, and the eks-node-role selector.
Tests: reconciler escalation gates + recovery clear, directive store epoch
increment + service validation, and a POSIX unit test for the guest agent.
* fix(eks): reboot running CP members to apply etcd cluster-reset directive
The etcd quorum-reformation escalation drove StartInstance on the seed and
followers, but the reset gate only fires when every member is VM-running. The
StartInstances path rejects a running instance with IncorrectInstanceState, so
the guest never re-entered its boot sequence and the cluster-reset/wipe-rejoin
directive was never applied — the wedged control plane stayed wedged.
Reboot the running members (QMP system_reset) instead, re-entering the boot
sequence so the on-VM recovery agent applies the directive. Adds RebootInstance
to the CP control surface (interface, adapter, NATS-backed daemon impl) and
switches maybeReformEtcdQuorum to it; the stopped/error in-place restart path
keeps StartInstance.
* fix(eks): graceful stop for etcd reset recovery
The etcd quorum-reformation reset path forced running CP members through a
hard QMP system_reset to re-enter the boot sequence and apply their recovery
directives. A hard reset skips a clean unmount: on the next boot e2fsck
cleared the recovery and k3s runlevel entries (dangling inodes), so the
cluster-reset never ran and k3s was unstartable — the reset defeated recovery.
Replace the reboot with a graceful StopInstance (QMP system_powerdown, clean
unmount, SIGKILL fallback). The existing in-place restart path
(maybeRecoverControlPlane) then boots each stopped member clean on a later
tick, and the on-VM recovery agent applies its directive against an intact
filesystem. Reuses existing lifecycle machinery; the stop also tolerates a
wedged QMP client via the SIGKILL fallback.
Swaps RebootInstance -> StopInstance across CPInstanceControl,
cpInstanceController, the cpControlAdapter, and the NATS-backed eksCPControl
(gateway RebootInstances -> StopInstances), plus the reset/recovery test fakes.
* fix(vm): retry QMP dial and unlink stale socket on relaunch
A QEMU killed with SIGKILL (crash or etcd-reset restart) leaves its
qmp-<id>.sock inode behind. On relaunch the startup os.Stat probe and the
QMP dial both raced that dead listener, so a single dial returned
'connection refused': StartStoppedInstance then reported ServerInternal and
left the instance KV-stopped even though the guest booted fully. The wedged
member stayed invisible to the reconciler's cpLive gate, stalling recovery.
Unlink the stale socket before launch so the inode that appears is the fresh
QEMU's, and retry the QMP connect with a short backoff to cover the window
where the socket is bound but not yet listening. Handshake failures are not
retried — a listener answered, so the error surfaces immediately.
* feat(eks-ui): surface cluster Health.Issues on the EKS console
DescribeCluster is AWS-faithful (Status stays ACTIVE, degradation lives in
Health.Issues), but the cluster detail page only rendered the status badge, so
a degraded control plane still showed as ACTIVE with no visible signal. Add a
destructive health badge next to the status and a Health row that lists the
issue messages, mirroring the addons tab. Healthy clusters read OK.
* chore: rebuild ui
* fix(viperblockd): stop mount-path daemon VB uploader to prevent checkpoint clobber
The daemon-side VB created in the ebs.mount handler is stored in
MountedVolume.VB for the whole mount, and its background chunk uploader was
only stopped at daemon shutdown. That state-only VB ran DrainToBackend every
30s, overwriting the live checkpoint the nbdkit plugin VB (the data-path
writer) keeps updating, leaving SeqNums inconsistent with sealed chunk
ciphertext and causing transient AEAD auth failures on cloned encrypted
volumes. Stop its uploader right after New, matching the other short-lived VB
call sites.
Also bump viperblock to the atomic chunk-ObjectID allocation fix.
* build(deps): pin viperblock to landed AEAD concurrency fix
Bump github.com/mulgadc/viperblock to the dev commit b08bea6 that
eliminates "cipher: message authentication failed" on encrypted volumes
under concurrent read-modify-write + background chunk drain. Removes the
local-path replace used while the fix was in flight.
joshsiv-mulga