Compare commits

...

16 Commits

Author SHA1 Message Date
Julian Sommer 09d23ae7dd fix: write MariaDB's startup spelling for time_zone into the option file
mariadbd has no time_zone startup option — the name only exists for SET — so
the resolved parameter group aborted the server while it parsed defaults, before
it opened the datadir. No pidfile was ever written and the reconciler failed
every instance at the 20-minute engine-health deadline.

ParameterSpec gains an option-file spelling, used only when rendering the
include. time_zone is the one setting that needs it, as default_time_zone, which
resolves @@global.time_zone to the same value. The customer-facing name is
unchanged: the API still reports time_zone and a live apply still issues
SET GLOBAL time_zone. Names read back out of a generated file return to the
catalog's namespace first, so a startup spelling cannot read as an unknown name
and report a pending restart nothing could clear.

setup.sh now feeds the catalog's resolved defaults to the image's own mariadbd
at build time, so a name the server refuses fails the build rather than every
instance built from it.
2026-08-12 10:14:30 +10:00
Julian Sommer 16aa888d10 fix: harden MariaDB guest bootstrap 2026-08-12 09:50:38 +10:00
Julian Sommer 8c5e5467e3 h 2026-08-11 16:48:46 +10:00
Julian Sommer 0cd749fdbf fixes 2026-08-11 16:32:12 +10:00
Julian Sommer 9bd14e00c9 feat: cover MariaDB end to end and document both engines
The rds suite gains three MariaDB tests. TestMariaDBInstance carries one
instance from create through a TLS connection, a password rotation, an attached
parameter group's dynamic and static edits, a stop/start and a class change,
re-asserting both overrides after every transition: an instance that came back
on catalog defaults is healthy, available and running a configuration its owner
did not choose, which no status reports. TestMariaDBSnapshotRestore proves the
snapshot carries MariaDB's own engine, version and port and that the restore
holds the rows as of the moment it was taken. TestMariaDBValidation boots
nothing and covers the unpinned versions, mysql, the cross-family group in both
directions and the implicit default.mariadb11.8.

The master password carries ' and \ at create and at rotation, the two
characters ValidateMasterUserPassword permits and the client cannot quote for
us, so rds-init's escaping and the agent's ALTER USER are under test against a
live server rather than a fixture.

The shared client VM installs mariadb-client alongside psql and opens 3306 as
well as 5432. The MariaDB client helpers avoid --ssl, which the 11.x client
deprecates: its notice would land in the same stream as the query's output,
which the guest exec folds together.

COMMANDS.md now covers both engines in one section, with a per-engine table of
pins, ports, families and limits, the mysql-is-not-an-alias note, the enumerated
MariaDB master grant, the InnoDB-only snapshot guarantee for non-transactional
tables, and the family-must-match rule. The Terraform quickstart stays
explicitly PostgreSQL and names what a MariaDB run would change.
2026-08-11 15:42:31 +10:00
Julian Sommer e47f4bcfca hu 2026-08-11 15:22:20 +10:00
Julian Sommer efa7d4e476 feat: offer MariaDB as an RDS engine
Registers MariaDB in the engine registry now that its image, bootstrap and
guest implementation exist, so a create naming it resolves the MariaDB AMI
rather than failing validation. default.mariadb11.8 joins default.postgres18
in DescribeDBParameterGroups, as AWS reports one default group per engine.
mysql stays rejected and is deliberately not aliased onto MariaDB.

The crash-consistent snapshot event becomes engine-aware. Its old wording
promised recovery from a write-ahead log, which is false for a MariaDB
database holding Aria or MyISAM tables: those have no redo log and can be
torn by a snapshot taken without a quiesce, which is exactly the case that
event reports.
2026-08-11 14:56:18 +10:00
Julian Sommer f5370a5d37 feat: implement the MariaDB guest engine behind the shared seam
Adds mariadbEngine alongside the PostgreSQL one: a three-stage probe that
separates a live-but-unreachable server from an absent one, ALTER USER password
rotation on stdin behind a session logging guard, a dynamic SET GLOBAL apply
with rollback of the installed file, file-comparison pending-restart against a
serving copy, last-known-good restore, service stop and restart, and BACKUP
STAGE quiesce on a held session with a bounded lock_wait_timeout.

The parameter-file machinery, service actions, quiesce hold bookkeeping and
client session are extracted so both engines share them, and a contract suite
runs the same cases against both through the interfaces. MariaDB is not
registered in the agent's engine table yet, so construction still refuses.
2026-08-11 14:42:17 +10:00
Julian Sommer 32ad16a24d step 8 2026-08-11 14:13:13 +10:00
Julian Sommer 2ad9d64c48 feat: add the MariaDB RDS image preset
Adds scripts/images/rds-mariadb/: the manifest with every mariadb package pinned =~11.8, setup.sh, the rds-datadir and rds-agent services, and the static include drop-in that points MariaDB at the generated configuration on the data volume. The AMI carries engine=mariadb, engine-version=11.8 and the data-volume contract tag, so resolveEngineAMI can select it.

The include is named zz- rather than 00-: MariaDB walks an include directory in byte order and takes the last occurrence of a setting, so a digit prefix would sort ahead of Alpine's packaged mariadb-server.cnf and let a distribution default beat a platform-owned one. setup.sh asserts the resulting order against the real directory at build time. The include target is baked empty as well, because MariaDB treats an includedir it cannot open as a fatal defaults-parsing error and a boot with no data volume would otherwise lose the client alongside the server.

The packaged drop-in also sets skip-networking, which would leave the endpoint resolving and the health probe passing while nothing on the customer ENI could connect. setup.sh disables it at the source and asserts no configuration file still does.

The mariadb service is deliberately not enabled: the packaged service only warns about an empty datadir rather than refusing, so an engine enabled before the bootstrap exists would launch mysqld_safe against an unbootstrapped volume.

rds-datadir moves to scripts/images/common/ rather than being copied. Volume selection, format authorization and mount are already engine-neutral; the mount point and owning user are the only per-engine facts and now come from each preset's own service. The script fails closed when either is unset, so no image can inherit another engine's layout by omission.
2026-08-11 13:39:49 +10:00
Julian Sommer b867300dee feat: add the MariaDB engine value and its parameter catalog
MariaDB 11.8 as an Engine with port 3306, its own reserved accounts,
an 80-character username limit and a 64-character database name limit,
alongside a curated parameter catalog of the same shape and size as the
PostgreSQL one. The size-derived defaults follow RDS's own formulas:
innodb_buffer_pool_size at three quarters of class memory and
max_connections on {DBInstanceClassMemory/12582880}, both with class
ceilings. Platform-owned settings are absent from the catalog rather
than present-and-unmodifiable.

It is deliberately not registered in engines. The image and guest
implementation land first, so no create can resolve an AMI nothing
builds.

The generic catalog machinery had two PostgreSQL assumptions baked into
it that a second engine cannot share. Boolean spellings are now carried
on the spec, because MariaDB refuses yes and no where PostgreSQL accepts
them, and the per-parameter string rules move onto a Validate hook, so
the timezone and datestyle checks live with PostgreSQL and sql_mode and
time_zone live with MariaDB.
2026-08-11 12:25:02 +10:00
Julian Sommer 598582dc37 feat: stamp the RDS data volume with the engine that wrote it
Another engine's datadir mounts cleanly and reads as uninitialised to
rds-init, which would initialise beside the customer's data and serve an
empty database that passes every health probe while the real data sits
unreferenced.

rds-init now writes the engine to ${DATA_MOUNT}/.spinifex-rds/engine
before the datadir is inspected, backfills it on attach for volumes
predating it, and refuses a stamp that disagrees.
2026-08-11 12:10:43 +10:00
Julian Sommer dafb001e06 feat: build the guest agent's engine from the image it is baked into
Define the composite engine interface and a newEngine() factory keyed by a per-engine layout table, so the agent's defaults, health probe and command implementation all come from the engine the image bakes rather than from PostgreSQL-shaped constants. Rename PGBin/PGData/PGUser to EngineBinDir/EngineDataDir/EngineUser, delete PGIsReady, and give engineProbe an injected state function with the pg_isready exit-code mapping moved into the PostgreSQL implementation.

setup.sh now stamps /etc/spinifex-rds/engine and buildAgentUserData emits RDS_ENGINE. The agent compares the two after registering and refuses to run when they disagree, reporting unhealthy before the bootstrap handoff is written so rds-init never unblocks and no datadir is touched.
2026-08-11 12:04:22 +10:00
Julian Sommer 4f5cde03bb feat: validate the initial database name at create and in the guest
DBName was passed through unvalidated, so a name such as my-db was accepted
at CreateDBInstance and only quoted into safety by psql inside the guest.
Engine gains a validateDBName rule and ValidateDBName, and rds-init re-checks
the same rule in load_handoff before initdb, where a failure would clear the
datadir and spend the one-shot master password.

PostgreSQL caps the name at NAMEDATALEN-1.
2026-08-11 11:48:29 +10:00
Julian Sommer 59ca1b33e6 feat: refuse a DB parameter group of another engine's family
Every path that binds a parameter group to an instance — create, modify,
snapshot restore, the deferred apply and group propagation — bottoms out
in resolveGroupParameters, which resolved against PostgreSQL whatever
the instance ran. It now takes the instance's engine and rejects a group
whose family is not that engine's with InvalidParameterCombination,
naming the group, its family and the family it needs. Each caller passes
the engine it already holds.

ModifyDBParameterGroup and DescribeDBParameters resolve the engine from
the target group's own family before validating or listing values, so a
parameter of one engine cannot be stored into another engine's group and
fail later against whichever instance next attached it.

The unrecognised-family rejection lists every supported family instead
of asserting postgres18 is the only one. An omitted family still takes
postgres18.
2026-08-11 11:38:05 +10:00
Julian Sommer 08189c581d refactor: move the RDS parameter catalog onto Engine
Completes the control-plane half of the engine seam ahead of a second
engine. Engine gains maxUsernameLen, its own parameter table and its own
cross-parameter checks, and the catalog accessors become methods on it,
so no path resolves parameters without naming an engine.

PostgreSQL's table, size-derived formulas and combination checks move to
engine_postgres.go. engines is also indexed by parameter-group family,
for the callers that hold only a family string. EnginePostgres() is
deleted: the guest's reserved-username re-check now resolves the engine
through LookupEngine once at agent startup.

No behavior change.
2026-08-11 11:05:52 +10:00
88 changed files with 9247 additions and 1468 deletions
+37 -31
View File
@@ -52,7 +52,7 @@ jobs:
# inputs.suites.
if: github.event_name != 'workflow_dispatch' || contains(github.event.inputs.suites, 'e2e-single') || contains(github.event.inputs.suites, 'e2e-iam') || contains(github.event.inputs.suites, 'e2e-cert') || contains(github.event.inputs.suites, 'e2e-eks') || contains(github.event.inputs.suites, 'e2e-ecs') || contains(github.event.inputs.suites, 'e2e-storagegrowth') || contains(github.event.inputs.suites, 'e2e-partialblock') || contains(github.event.inputs.suites, 'e2e-rds')
runs-on: [self-hosted, ci-single]
timeout-minutes: 50
timeout-minutes: 60
env:
TOFU_LOG: /tmp/e2e-tofu-single-${{ github.run_id }}.log
ARTIFACT_DIR: /tmp/spinifex-e2e-artifacts-${{ github.run_id }}
@@ -250,53 +250,58 @@ jobs:
EOF
ok "kubectl installed on VM"
# CreateDBInstance boots the DB VM from the rds-postgres AMI, so that image
# CreateDBInstance boots the DB VM from an RDS AMI, so both engines' images
# must be in the VM's image store before the suite runs — same split as the
# eks-node pair above: build on the runner (needs source + network), import
# on the VM (needs predastore/NATS). The suite's psql client needs no step
# here: it runs inside a client VM that installs postgresql-client from its
# own user-data, because a private-only endpoint is unreachable from the
# runner and from the cluster host alike.
- name: Build rds-postgres AMI (libguestfs, runner)
# on the VM (needs predastore/NATS). The suite's psql and mariadb clients
# need no step here: they run inside a client VM that installs them from
# its own user-data, because a private-only endpoint is unreachable from
# the runner and from the cluster host alike.
- name: Build RDS AMIs (libguestfs, runner)
if: contains(env.SUITES, 'e2e-rds')
working-directory: spinifex
run: |
set -euo pipefail
source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
banner "build rds-postgres AMI (libguestfs — no host devices)"
sudo apt-get update -qq
sudo apt-get install -y --no-install-recommends libguestfs-tools qemu-utils
export LIBGUESTFS_BACKEND=direct
sudo chmod 0644 /boot/vmlinuz-* || true
group_open "make build-rds-postgres-image"
make build-rds-postgres-image
group_close
ok "raw: $(du -h /tmp/rds-postgres-image-build/rds-postgres-alpine.raw | cut -f1)"
for IMAGE in rds-postgres rds-mariadb; do
banner "build ${IMAGE} AMI (libguestfs — no host devices)"
group_open "make build-${IMAGE}-image"
make "build-${IMAGE}-image"
group_close
ok "raw: $(du -h "/tmp/${IMAGE}-image-build/${IMAGE}-alpine.raw" | cut -f1)"
done
- name: Import rds-postgres AMI on VM
- name: Import RDS AMIs on VM
if: contains(env.SUITES, 'e2e-rds')
run: |
set -euo pipefail
source "${GITHUB_WORKSPACE}/spinifex/.github/scripts/ci-fmt.sh"
eval "$(grep -E '^(ALPINE_VERSION|AMI_NAME|SYSTEM_TAG)=' \
spinifex/scripts/images/rds-postgres/manifest.conf)"
# SYSTEM_TAG carries three tags, not one: managed-by marks it a system
# image and engine/engine-version are what an EngineVersion request
# resolves against. --tag is repeatable, so expand rather than pass the
# whole string as a single value.
TAG_ARGS=""
for tag in ${SYSTEM_TAG}; do TAG_ARGS="${TAG_ARGS} --tag ${tag}"; done
SSH_OPTS=(-o StrictHostKeyChecking=no -i "${{ steps.vm.outputs.ssh_key }}")
VM=tf-user@${{ steps.vm.outputs.wan_ip }}
banner "import rds-postgres AMI to VM image store"
scp "${SSH_OPTS[@]}" \
/tmp/rds-postgres-image-build/rds-postgres-alpine.raw \
"$VM":/tmp/rds-postgres-alpine.raw
ssh "${SSH_OPTS[@]}" "$VM" \
"spx admin images import --file /tmp/rds-postgres-alpine.raw \
--distro alpine --version ${ALPINE_VERSION} --arch x86_64 \
--boot-mode bios --ami-name ${AMI_NAME}${TAG_ARGS}" >/dev/null 2>&1
ok "rds-postgres AMI imported"
for IMAGE in rds-postgres rds-mariadb; do
eval "$(grep -E '^(ALPINE_VERSION|AMI_NAME|SYSTEM_TAG)=' \
"spinifex/scripts/images/${IMAGE}/manifest.conf")"
# SYSTEM_TAG carries four tags, not one: managed-by marks it a system
# image, engine/engine-version are what an EngineVersion request
# resolves against, and the contract tag gates the data-volume format
# grant. --tag is repeatable, so expand rather than pass the whole
# string as a single value.
TAG_ARGS=""
for tag in ${SYSTEM_TAG}; do TAG_ARGS="${TAG_ARGS} --tag ${tag}"; done
banner "import ${IMAGE} AMI to VM image store"
scp "${SSH_OPTS[@]}" \
"/tmp/${IMAGE}-image-build/${IMAGE}-alpine.raw" \
"$VM":"/tmp/${IMAGE}-alpine.raw"
ssh "${SSH_OPTS[@]}" "$VM" \
"spx admin images import --file /tmp/${IMAGE}-alpine.raw \
--distro alpine --version ${ALPINE_VERSION} --arch x86_64 \
--boot-mode bios --ami-name ${AMI_NAME}${TAG_ARGS}" >/dev/null 2>&1
ok "${IMAGE} AMI imported"
done
- name: Build E2E test binaries on runner
working-directory: spinifex/tests/e2e
@@ -383,9 +388,10 @@ jobs:
echo \"::group::suite \$SUITE\"; \
printf '\n%s%s━━ suite %s ━━%s\n' \"\$B\" \"\$C\" \"\$SUITE\" \"\$R\"; \
date -u +%Y-%m-%dT%H:%M:%SZ > '${ARTIFACT_DIR}/test-'\$SUITE'.start'; \
case \$SUITE in rds) TMO=45m ;; *) TMO=30m ;; esac; \
GITHUB_ACTIONS=true FORCE_COLOR=1 SPINIFEX_E2E=1 SPINIFEX_MODE=single \
ARTIFACT_DIR='${ARTIFACT_DIR}' \
./\$SUITE.test -test.v -test.timeout=30m ${TEST_RUN_FLAG} \
./\$SUITE.test -test.v -test.timeout=\$TMO ${TEST_RUN_FLAG} \
2>&1 | tee '${ARTIFACT_DIR}/test-'\$SUITE'.log' \
| sed -uE 's/^[[:space:]]+([a-zA-Z0-9_]+_test\.go:[0-9]+: )?//'; \
rc=\${PIPESTATUS[0]}; \
+4 -2
View File
@@ -2,18 +2,20 @@ name: Publish System AMIs
# Builds the system AMIs (libguestfs, on the runner) and publishes each
# compressed qcow2 + .sha256 sidecar to the Cloudflare R2 bucket that backs
# iso.mulgadc.com/system-ami/. Five images ship:
# iso.mulgadc.com/system-ami/. Six images ship:
# spinifex-eks-node Alpine EKS K3s node (server|agent, role at boot)
# spinifex-ecs-node Alpine ECS container instance
# spinifex-eks-node-gpu Ubuntu EKS K3s GPU worker (NVIDIA driver + toolkit)
# spinifex-ecs-node-gpu Ubuntu ECS GPU container instance
# spinifex-rds-postgres Alpine RDS PostgreSQL instance
# spinifex-rds-mariadb Alpine RDS MariaDB instance
# Operators then install them on any cluster with:
# spx admin images import --name spinifex-eks-node
# spx admin images import --name spinifex-ecs-node
# spx admin images import --name spinifex-eks-node-gpu
# spx admin images import --name spinifex-ecs-node-gpu
# spx admin images import --name spinifex-rds-postgres
# spx admin images import --name spinifex-rds-mariadb
#
# Manual-only: publishing overwrites the mirror artifact, so it is a deliberate
# action, not something every push should trigger. The `variant` input scopes
@@ -59,7 +61,7 @@ jobs:
set -euo pipefail
cpu='{"name":"spinifex-eks-node","manifest":"scripts/images/eks-node/manifest.conf"},{"name":"spinifex-ecs-node","manifest":"scripts/images/ecs-agent/manifest.conf"}'
gpu='{"name":"spinifex-eks-node-gpu","manifest":"scripts/images/eks-node-gpu/manifest.conf"},{"name":"spinifex-ecs-node-gpu","manifest":"scripts/images/ecs-agent-gpu/manifest.conf"}'
rds='{"name":"spinifex-rds-postgres","manifest":"scripts/images/rds-postgres/manifest.conf"}'
rds='{"name":"spinifex-rds-postgres","manifest":"scripts/images/rds-postgres/manifest.conf"},{"name":"spinifex-rds-mariadb","manifest":"scripts/images/rds-mariadb/manifest.conf"}'
case "${VARIANT}" in
cpu) items="${cpu}" ;;
gpu) items="${gpu}" ;;
+7 -1
View File
@@ -111,6 +111,12 @@ build-rds-postgres-image: ## Build the spinifex-rds-postgres AMI (Alpine + Postg
import-rds-postgres-image: ## Build + register the rds-postgres AMI (requires a running cluster)
$(MAKE) build-system-image IMAGE=rds-postgres IMPORT=1
build-rds-mariadb-image: ## Build the spinifex-rds-mariadb AMI (Alpine + MariaDB 11.8 + rds-init; IMPORT=1 to register)
$(MAKE) build-system-image IMAGE=rds-mariadb
import-rds-mariadb-image: ## Build + register the rds-mariadb AMI (requires a running cluster)
$(MAKE) build-system-image IMAGE=rds-mariadb IMPORT=1
MICROVM_OUT_DIR := build/microvm
MICROVM_ARTIFACTS := $(MICROVM_OUT_DIR)/vmlinuz $(MICROVM_OUT_DIR)/initramfs.cpio.gz
MICROVM_INPUTS := scripts/build-microvm-image.sh $(MICROVM_OUT_DIR)/init.sh $(MICROVM_OUT_DIR)/inittab bin/lb-agent
@@ -388,7 +394,7 @@ distro-arm64:
distro-clean:
rm -rf dist/
.PHONY: build build-ui build-installer build-lb-agent build-ecs-agent build-system-image build-eks-node-image import-eks-node-image publish-eks-node-image build-ecs-node-image import-ecs-node-image build-rds-postgres-image import-rds-postgres-image build-microvm-image install-microvm go_build preflight test test-cover test-race diff-coverage bench test-actions test-images test-harness test-integration generate-aws-model-coverage aws-model-coverage test-segscan-oracle manifest-check manifest-lint manifest-lint-update \
.PHONY: build build-ui build-installer build-lb-agent build-ecs-agent build-system-image build-eks-node-image import-eks-node-image publish-eks-node-image build-ecs-node-image import-ecs-node-image build-rds-postgres-image import-rds-postgres-image build-rds-mariadb-image import-rds-mariadb-image build-microvm-image install-microvm go_build preflight test test-cover test-race diff-coverage bench test-actions test-images test-harness test-integration generate-aws-model-coverage aws-model-coverage test-segscan-oracle manifest-check manifest-lint manifest-lint-update \
deploy reinstall clean \
install-system install-go install-aws quickinstall \
lint fix govulncheck nilaway \
+40 -8
View File
@@ -180,7 +180,7 @@ type Agent struct {
id identity
cp controlPlane
probe *engineProbe
engine *postgresEngine
engine engine
handoffWriter func(string, *handlers_rds.GetDBBootstrapConfigOutput) error
dataMountWaiter func(context.Context, string, string) error
@@ -195,10 +195,7 @@ type Agent struct {
// Assembles from already-built parts, so tests can pass fakes; New builds the
// production control plane and delegates here.
func newAgent(cfg config, cp controlPlane, probe *engineProbe) *Agent {
if cfg.DataMount == "" {
cfg.DataMount = defaultDataMount
}
func newAgent(cfg config, cp controlPlane, probe *engineProbe) (*Agent, error) {
if cfg.MountsFile == "" {
cfg.MountsFile = defaultMountsFile
}
@@ -211,11 +208,15 @@ func newAgent(cfg config, cp controlPlane, probe *engineProbe) *Agent {
cfg: cfg, id: id, cp: cp, probe: probe,
handoffWriter: writeHandoff, dataMountWaiter: waitForDataMount,
}
a.engine = newPostgresEngine(cfg, execCommandRunner, execSessionRunner, probe)
eng, err := newEngine(cfg, execCommandRunner, execSessionRunner, probe)
if err != nil {
return nil, err
}
a.engine = eng
a.hb = newHeartbeater(cp, probe, a.engine, handlers_rds.HeartbeatInterval)
a.cmd = newCommander(cp, newCommandRegistry(a.engine, newGuestStorage(cfg, execCommandRunner)), cfg.PollWait)
a.guard = newParamGuard(a.engine, probe, cp)
return a
return a, nil
}
// Does not wait for IMDS: the register loop rides out a datapath still coming up.
@@ -227,7 +228,11 @@ func New(cfg config) (*Agent, error) {
if err != nil {
return nil, err
}
return newAgent(cfg, cp, newEngineProbe(cfg, execProbeRunner)), nil
probe, err := newProbe(cfg, execProbeRunner)
if err != nil {
return nil, err
}
return newAgent(cfg, cp, probe)
}
func (a *Agent) Run(ctx context.Context) error {
@@ -245,6 +250,13 @@ func (a *Agent) Run(ctx context.Context) error {
}
a.hb.id, a.cmd.id, a.guard.id = a.id, a.id, a.id
// Ahead of the handoff, so rds-init never unblocks and no datadir is touched.
// An agent running on the wrong image would initialise one engine over
// another's data, coming up healthy and empty beside data nothing references.
if err := a.refuseEngineMismatch(ctx); err != nil {
return err
}
// Beating before the bootstrap keeps a stuck boot visible as a live VM with
// a down engine rather than as silence.
go a.hb.Run(ctx)
@@ -279,6 +291,26 @@ func (a *Agent) Run(ctx context.Context) error {
return nil
}
// The engine cloud-init says this VM was launched as, against the engine the
// image bakes. An empty expectation is no assertion rather than a disagreement:
// a VM launched before the control plane emitted RDS_ENGINE carries none.
func (a *Agent) refuseEngineMismatch(ctx context.Context) error {
expected := strings.ToLower(strings.TrimSpace(a.cfg.Engine))
if expected == "" || expected == a.cfg.BakedEngine {
return nil
}
message := fmt.Sprintf("this VM was launched as engine %q but its image bakes %q; refusing to bootstrap",
expected, a.cfg.BakedEngine)
slog.ErrorContext(ctx, "rds-agent: "+message)
// Reported before returning, so the control plane learns why the instance
// never came up rather than only that it never did.
if _, err := a.cp.SubmitState(ctx, a.id, handlers_rds.EngineHealthUnhealthy, message); err != nil {
slog.ErrorContext(ctx, "rds-agent: reporting the engine mismatch failed", "err", err)
}
return errors.New(message)
}
const dataMountPollInterval = time.Second
// waitForDataMount blocks the stateful agent loops until the kernel reports the
+84 -27
View File
@@ -168,18 +168,26 @@ func (f *fakeControlPlane) snapshotReplies() [][]handlers_rds.CommandReply {
return out
}
// testConfig points the agent's file output at a temp dir and its probe at a
// name no real binary answers to.
// testConfig stamps the engine a real rds-postgres image bakes, so the agent
// resolves the same guest layout it would in a guest, and points its file output
// at a temp dir.
func testConfig(t *testing.T) config {
t.Helper()
return config{
GatewayURL: "https://gw.test",
HandoffDir: filepath.Join(t.TempDir(), "spinifex-rds"),
EngineHost: defaultEngineHost,
EnginePort: defaultEnginePort,
PGIsReady: "pg_isready",
PollWait: time.Second,
cfg := testLoadConfig(t, enginePostgres)
cfg.GatewayURL = "https://gw.test"
cfg.HandoffDir = filepath.Join(t.TempDir(), "spinifex-rds")
cfg.PollWait = time.Second
return cfg
}
// newTestAgent builds the agent over a probe the test drives directly.
func newTestAgent(t *testing.T, cfg config, cp controlPlane, run probeRunner) *Agent {
t.Helper()
a, err := newAgent(cfg, cp, newPostgresProbe(cfg, run))
if err != nil {
t.Fatalf("newAgent: %v", err)
}
return a
}
// staticProbe is a probeRunner that always reports the same exit code.
@@ -272,7 +280,7 @@ func TestRun_RegistersThenDeliversBootstrap(t *testing.T) {
cp.bootstrapOut = bootstrapOutput(&password)
cfg := testConfig(t)
a := newAgent(cfg, cp, newEngineProbe(cfg, staticProbe(0)))
a := newTestAgent(t, cfg, cp, staticProbe(0))
if err := runAgent(t, a); err != nil {
t.Fatalf("Run: %v", err)
}
@@ -303,7 +311,7 @@ func TestRun_AttachModeNeedsNoPassword(t *testing.T) {
cp.bootstrapOut = bootstrapOutput(nil)
cfg := testConfig(t)
if err := runAgent(t, newAgent(cfg, cp, newEngineProbe(cfg, staticProbe(0)))); err != nil {
if err := runAgent(t, newTestAgent(t, cfg, cp, staticProbe(0))); err != nil {
t.Fatalf("Run: %v", err)
}
@@ -325,7 +333,7 @@ func TestRun_RetriesRegisterAndBootstrap(t *testing.T) {
cp.bootstrapErrs = []error{errors.New("ServerInternal")}
cfg := testConfig(t)
if err := runAgent(t, newAgent(cfg, cp, newEngineProbe(cfg, staticProbe(0)))); err != nil {
if err := runAgent(t, newTestAgent(t, cfg, cp, staticProbe(0))); err != nil {
t.Fatalf("Run: %v", err)
}
@@ -344,7 +352,7 @@ func TestBootstrap_RetriesHandoffWithoutRefetching(t *testing.T) {
password := "staged-secret"
cp.bootstrapOut = bootstrapOutput(&password)
cfg := testConfig(t)
a := newAgent(cfg, cp, newEngineProbe(cfg, staticProbe(0)))
a := newTestAgent(t, cfg, cp, staticProbe(0))
writes := 0
a.handoffWriter = func(_ string, got *handlers_rds.GetDBBootstrapConfigOutput) error {
@@ -385,7 +393,7 @@ func TestBootstrap_RetriesInitializeWithNoPassword(t *testing.T) {
}}
cp.bootstrapOut = bootstrapOutput(&password)
cfg := testConfig(t)
a := newAgent(cfg, cp, newEngineProbe(cfg, staticProbe(0)))
a := newTestAgent(t, cfg, cp, staticProbe(0))
var handed []*handlers_rds.GetDBBootstrapConfigOutput
a.handoffWriter = func(_ string, got *handlers_rds.GetDBBootstrapConfigOutput) error {
@@ -408,7 +416,7 @@ func TestAcknowledgeBootstrap_WaitsForTheReceiptThenRetiresIt(t *testing.T) {
cp := newFakeControlPlane()
cfg := testConfig(t)
cfg.DataMount = t.TempDir()
a := newAgent(cfg, cp, newEngineProbe(cfg, staticProbe(0)))
a := newTestAgent(t, cfg, cp, staticProbe(0))
a.id = identity{DBInstanceIdentifier: "db-resolved"}
a.pending = &pendingBootstrap{payloadID: testPayloadID, vmGeneration: 1, dataVolumeID: "vol-data-01"}
@@ -447,7 +455,7 @@ func TestAcknowledgeBootstrap_IgnoresAForeignReceipt(t *testing.T) {
cp := newFakeControlPlane()
cfg := testConfig(t)
cfg.DataMount = t.TempDir()
a := newAgent(cfg, cp, newEngineProbe(cfg, staticProbe(0)))
a := newTestAgent(t, cfg, cp, staticProbe(0))
a.id = identity{DBInstanceIdentifier: "db-resolved"}
a.pending = &pendingBootstrap{payloadID: testPayloadID, vmGeneration: 1}
writeReceipt(t, a, tc.payloadID, tc.dbID)
@@ -470,7 +478,7 @@ func TestAcknowledgeBootstrap_IsANoOpOnAttach(t *testing.T) {
cp := newFakeControlPlane()
cfg := testConfig(t)
cfg.DataMount = t.TempDir()
a := newAgent(cfg, cp, newEngineProbe(cfg, staticProbe(0)))
a := newTestAgent(t, cfg, cp, staticProbe(0))
if err := a.acknowledgeBootstrap(t.Context()); err != nil {
t.Fatalf("acknowledgeBootstrap: %v", err)
@@ -480,6 +488,52 @@ func TestAcknowledgeBootstrap_IsANoOpOnAttach(t *testing.T) {
}
}
// The image is the authority on which engine the agent runs. A VM launched as
// another engine has to stop ahead of the handoff: rds-init would otherwise
// initialise one engine over the other's datadir and come up healthy and empty
// beside data nothing references.
func TestRun_RefusesAnEngineTheImageDoesNotBake(t *testing.T) {
cp := newFakeControlPlane()
cp.bootstrapOut = bootstrapOutput(nil)
cfg := testConfig(t)
cfg.Engine = "mariadb"
a := newTestAgent(t, cfg, cp, staticProbe(0))
err := a.Run(t.Context())
if err == nil {
t.Fatal("Run returned nil for a VM launched as an engine this image does not bake")
}
if !strings.Contains(err.Error(), "mariadb") {
t.Errorf("error = %v, want it to name the engine the VM was launched as", err)
}
if _, statErr := os.Stat(filepath.Join(cfg.HandoffDir, handoffEnvFile)); !os.IsNotExist(statErr) {
t.Error("the agent wrote a bootstrap handoff for an engine it refused to run")
}
if n := len(cp.bootstrapReqs); n != 0 {
t.Errorf("bootstrap fetches = %d, want none before the refusal", n)
}
// Reported rather than only logged, so the control plane learns why the
// instance never came up rather than only that it never did.
states := cp.snapshotStates()
if len(states) != 1 || states[0].health != handlers_rds.EngineHealthUnhealthy {
t.Fatalf("submitted states = %+v, want a single unhealthy report", states)
}
}
// The same assertion when it agrees is not a gate: the boot goes on exactly as
// it does for a VM launched before the control plane sent one at all.
func TestRun_ProceedsWhenTheDeliveredEngineMatchesTheImage(t *testing.T) {
cp := newFakeControlPlane()
cp.bootstrapOut = bootstrapOutput(nil)
cfg := testConfig(t)
cfg.Engine = enginePostgres
if err := runAgent(t, newTestAgent(t, cfg, cp, staticProbe(0))); err != nil {
t.Fatalf("Run: %v", err)
}
}
// A configured identifier is asserted on the wire so a mis-provisioned VM is
// rejected by the gateway instead of adopting whatever it resolves to.
func TestRun_SendsConfiguredIdentifier(t *testing.T) {
@@ -489,7 +543,7 @@ func TestRun_SendsConfiguredIdentifier(t *testing.T) {
cfg := testConfig(t)
cfg.DBInstanceIdentifier = "db-configured"
cfg.EngineVersion = "18.1"
if err := runAgent(t, newAgent(cfg, cp, newEngineProbe(cfg, staticProbe(0)))); err != nil {
if err := runAgent(t, newTestAgent(t, cfg, cp, staticProbe(0))); err != nil {
t.Fatalf("Run: %v", err)
}
@@ -503,12 +557,12 @@ func TestRun_WaitsForDataMountBeforePollingCommands(t *testing.T) {
cp := newFakeControlPlane()
cp.bootstrapOut = bootstrapOutput(nil)
cfg := testConfig(t)
a := newAgent(cfg, cp, newEngineProbe(cfg, staticProbe(2)))
a := newTestAgent(t, cfg, cp, staticProbe(2))
waiting := make(chan struct{})
release := make(chan struct{})
a.dataMountWaiter = func(ctx context.Context, mountsFile, target string) error {
if mountsFile != defaultMountsFile || target != defaultDataMount {
if mountsFile != defaultMountsFile || target != cfg.DataMount {
t.Errorf("mount waiter got %q/%q, want configured defaults", mountsFile, target)
}
close(waiting)
@@ -580,7 +634,7 @@ func TestRun_HeartbeatIncludesBootstrapFailure(t *testing.T) {
cp.registerOut.HeartbeatIntervalSeconds = 0
cfg := testConfig(t)
a := newAgent(cfg, cp, newEngineProbe(cfg, staticProbe(2)))
a := newTestAgent(t, cfg, cp, staticProbe(2))
a.hb.interval = 10 * time.Millisecond
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
@@ -609,7 +663,7 @@ func TestRun_HeartbeatsEngineHealth(t *testing.T) {
cp.registerOut.HeartbeatIntervalSeconds = 0
cfg := testConfig(t)
a := newAgent(cfg, cp, newEngineProbe(cfg, staticProbe(0)))
a := newTestAgent(t, cfg, cp, staticProbe(0))
a.hb.interval = 10 * time.Millisecond
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
@@ -639,7 +693,7 @@ func TestRegister_AdoptsControlPlaneCadence(t *testing.T) {
cp.registerOut.HeartbeatIntervalSeconds = 45
cfg := testConfig(t)
a := newAgent(cfg, cp, newEngineProbe(cfg, staticProbe(0)))
a := newTestAgent(t, cfg, cp, staticProbe(0))
if err := runAgent(t, a); err != nil {
t.Fatalf("Run: %v", err)
}
@@ -656,12 +710,15 @@ func TestRun_HeartbeatReportsDownEngine(t *testing.T) {
cp.registerOut.HeartbeatIntervalSeconds = 0
cfg := testConfig(t)
probe := newEngineProbe(cfg, staticProbe(2))
probe := newPostgresProbe(cfg, staticProbe(2))
// The engine has served before, so a silent one now is a failure rather
// than a boot still in progress.
probe.seenHealthy = true
a := newAgent(cfg, cp, probe)
a, err := newAgent(cfg, cp, probe)
if err != nil {
t.Fatalf("newAgent: %v", err)
}
a.hb.interval = 10 * time.Millisecond
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
@@ -752,7 +809,7 @@ func TestAcknowledgeBootstrap_StopsOnATerminalDenial(t *testing.T) {
}}
cfg := testConfig(t)
cfg.DataMount = t.TempDir()
a := newAgent(cfg, cp, newEngineProbe(cfg, staticProbe(0)))
a := newTestAgent(t, cfg, cp, staticProbe(0))
a.id = identity{DBInstanceIdentifier: "db-resolved"}
a.pending = &pendingBootstrap{payloadID: testPayloadID, vmGeneration: 1}
writeReceipt(t, a, testPayloadID, "db-resolved")
@@ -780,7 +837,7 @@ func TestAcknowledgeBootstrap_DoesNotReportAnAbsentReceiptAsAFailure(t *testing.
cp := newFakeControlPlane()
cfg := testConfig(t)
cfg.DataMount = t.TempDir()
a := newAgent(cfg, cp, newEngineProbe(cfg, staticProbe(0)))
a := newTestAgent(t, cfg, cp, staticProbe(0))
a.id = identity{DBInstanceIdentifier: "db-resolved"}
a.pending = &pendingBootstrap{payloadID: testPayloadID, vmGeneration: 1}
+19 -6
View File
@@ -107,7 +107,11 @@ func writeHandoff(dir string, cfg *handlers_rds.GetDBBootstrapConfigOutput) erro
return fmt.Errorf("secure handoff dir %s: %w", dir, err)
}
if err := writeHandoffFile(dir, handoffParamsFile, renderParameters(cfg.Parameters)); err != nil {
params, err := renderParameters(cfg.Engine, cfg.Parameters)
if err != nil {
return err
}
if err := writeHandoffFile(dir, handoffParamsFile, params); err != nil {
return err
}
// Cert and key are written as a pair; half of one would start the engine
@@ -177,18 +181,27 @@ func shellQuote(s string) string {
return "'" + strings.ReplaceAll(s, "'", `'\''`) + "'"
}
// postgresql.conf syntax. Values are quoted so a setting with a space or unit
// suffix survives; the engine accepts quoted numerics and booleans too.
func renderParameters(params []handlers_rds.Parameter) string {
// postgresql.conf syntax, which MariaDB's option files also parse. Values are
// quoted so a setting with a space or unit suffix survives; the engine accepts
// quoted numerics and booleans too.
//
// Names are the engine's rather than the customer's: a setting the server takes
// only through SET has a different spelling at startup, and writing the wrong one
// leaves the engine refusing to boot on a file already on the data volume.
func renderParameters(engineName string, params []handlers_rds.Parameter) (string, error) {
engine, err := handlers_rds.LookupEngine(engineName)
if err != nil {
return "", fmt.Errorf("render the resolved parameters: %w", err)
}
var b strings.Builder
b.WriteString("# Resolved parameter group, written by rds-agent.\n")
for _, p := range params {
if p.Name == "" {
continue
}
fmt.Fprintf(&b, "%s = '%s'\n", p.Name, strings.ReplaceAll(p.Value, "'", "''"))
fmt.Fprintf(&b, "%s = '%s'\n", engine.OptionFileName(p.Name), strings.ReplaceAll(p.Value, "'", "''"))
}
return b.String()
return b.String(), nil
}
// Atomic: a temp file in the same directory, renamed over the target. The mode
+51 -2
View File
@@ -16,6 +16,7 @@ func TestWriteHandoff_WritesEveryFileRootOnly(t *testing.T) {
err := writeHandoff(dir, &handlers_rds.GetDBBootstrapConfigOutput{
Mode: handlers_rds.BootstrapModeInitialize,
Engine: "postgres",
MasterUsername: "master",
MasterUserPassword: &password,
DBName: "appdb",
@@ -82,6 +83,7 @@ func TestWriteHandoff_AttachOmitsPassword(t *testing.T) {
dir := filepath.Join(t.TempDir(), "handoff")
if err := writeHandoff(dir, &handlers_rds.GetDBBootstrapConfigOutput{
Mode: handlers_rds.BootstrapModeAttach,
Engine: "postgres",
MasterUsername: "master",
Port: 5432,
}); err != nil {
@@ -112,6 +114,7 @@ func TestWriteHandoff_QuotesShellMetacharacters(t *testing.T) {
if err := writeHandoff(dir, &handlers_rds.GetDBBootstrapConfigOutput{
Mode: handlers_rds.BootstrapModeInitialize,
Engine: "postgres",
MasterUsername: "master",
MasterUserPassword: &password,
Port: 5432,
@@ -132,6 +135,7 @@ func TestWriteHandoff_SkipsHalfATLSPair(t *testing.T) {
dir := filepath.Join(t.TempDir(), "handoff")
if err := writeHandoff(dir, &handlers_rds.GetDBBootstrapConfigOutput{
Mode: handlers_rds.BootstrapModeAttach,
Engine: "postgres",
MasterUsername: "master",
Port: 5432,
ServingCertificate: "CERT",
@@ -149,8 +153,9 @@ func TestWriteHandoff_SkipsHalfATLSPair(t *testing.T) {
func TestWriteHandoff_RejectsConfigWithNoMasterUser(t *testing.T) {
dir := filepath.Join(t.TempDir(), "handoff")
err := writeHandoff(dir, &handlers_rds.GetDBBootstrapConfigOutput{
Mode: handlers_rds.BootstrapModeAttach,
Port: 5432,
Mode: handlers_rds.BootstrapModeAttach,
Engine: "postgres",
Port: 5432,
})
if err == nil {
t.Fatal("writeHandoff accepted a config with no master username, want an error")
@@ -170,6 +175,7 @@ func TestWriteHandoff_TightensExistingDirectory(t *testing.T) {
if err := writeHandoff(dir, &handlers_rds.GetDBBootstrapConfigOutput{
Mode: handlers_rds.BootstrapModeAttach,
Engine: "postgres",
MasterUsername: "master",
Port: 5432,
}); err != nil {
@@ -219,3 +225,46 @@ func findLine(t *testing.T, body, prefix string) string {
t.Fatalf("no line starting %q in:\n%s", prefix, body)
return ""
}
// The include rds-init installs is what the engine starts on, so the spelling
// written here is the one that has to boot.
func TestRenderParameters_UsesTheEngineStartupSpelling(t *testing.T) {
params := []handlers_rds.Parameter{
{Name: "time_zone", Value: "SYSTEM"},
{Name: "max_connections", Value: "85"},
}
rendered, err := renderParameters("mariadb", params)
if err != nil {
t.Fatalf("renderParameters: %v", err)
}
if !strings.Contains(rendered, "default_time_zone = 'SYSTEM'\n") {
t.Errorf("time_zone was not written under its startup spelling:\n%s", rendered)
}
if strings.Contains(rendered, "\ntime_zone = ") {
t.Errorf("time_zone reached the option file, which mariadbd refuses:\n%s", rendered)
}
if !strings.Contains(rendered, "max_connections = '85'\n") {
t.Errorf("an unmapped parameter was not written unchanged:\n%s", rendered)
}
}
func TestRenderParameters_PostgresKeepsEveryName(t *testing.T) {
rendered, err := renderParameters("postgres", []handlers_rds.Parameter{
{Name: "shared_buffers", Value: "262144"},
})
if err != nil {
t.Fatalf("renderParameters: %v", err)
}
if !strings.Contains(rendered, "shared_buffers = '262144'\n") {
t.Errorf("postgres name was rewritten:\n%s", rendered)
}
}
// Failing closed beats writing a file the engine may refuse: the bad include
// would already be on the data volume by the time the server rejected it.
func TestRenderParameters_RefusesAnUnknownEngine(t *testing.T) {
if _, err := renderParameters("", nil); err == nil {
t.Fatal("renderParameters accepted a config naming no engine, want an error")
}
}
+83 -42
View File
@@ -1,7 +1,9 @@
package main
import (
"os"
"strconv"
"strings"
"time"
"github.com/mulgadc/spinifex/internal/guestenv"
@@ -12,17 +14,11 @@ const (
defaultGatewayCA = "/etc/spinifex-rds/gateway-ca.pem"
defaultHandoffDir = "/run/spinifex-rds"
defaultEngineHost = "127.0.0.1"
defaultEnginePort = 5432
defaultPGIsReady = "pg_isready"
// The guest layout rds-init lays down. Overridable for the same reason it is
// there: a sibling engine preset points at its own paths.
defaultPGBin = "/usr/libexec/postgresql18"
defaultPGData = "/var/lib/postgresql/18/data"
defaultSocketDir = "/run/postgresql"
defaultPGUser = "postgres"
defaultRCService = "rc-service"
defaultEngineSvcName = "postgresql"
// Where setup.sh stamps the engine the image bakes. The agent builds its
// engine implementation from this file rather than from anything delivered.
defaultEngineFile = "/etc/spinifex-rds/engine"
defaultRCService = "rc-service"
defaultEngineProbeTimeout = 5 * time.Second
// The long-poll window the agent asks the gateway to hold a request open
// for. The gateway caps it at 20s.
defaultPollWait = 20 * time.Second
@@ -44,19 +40,29 @@ type config struct {
HandoffDir string
EngineHost string
EnginePort int
// Overridable so a test or a sibling engine preset can point at its own.
PGIsReady string
PollWait time.Duration
PollWait time.Duration
// The engine the control plane launched this VM as, delivered by cloud-init,
// and the engine the image itself bakes. Two independent statements of the
// same fact: the agent refuses to bootstrap when they disagree.
Engine string
EngineFile string
BakedEngine string
// Where the command handlers reach the engine: the client binaries, the
// datadir the parameter file is installed into, the socket they connect
// over, the OS user they drop to, and the service the stop goes through.
PGBin string
PGData string
EngineBinDir string
EngineDataDir string
SocketDir string
PGUser string
EngineUser string
RCService string
EngineService string
// Where the engine records the pid its probe checks for liveness, for an
// engine whose own client cannot tell a server still recovering from one that
// is not running at all.
EnginePidFile string
EngineProbeTimeout time.Duration
// Where the data volume is mounted, and the two kernel surfaces a storage
// grow resolves its device from. Overridable so a test can point them at
@@ -75,19 +81,21 @@ func loadConfig(envFile string) config {
GatewayCA: get("RDS_GATEWAY_CA"),
Region: get("RDS_REGION"),
DBInstanceIdentifier: get("RDS_DB_INSTANCE_IDENTIFIER"),
Engine: get("RDS_ENGINE"),
EngineFile: get("RDS_ENGINE_FILE"),
EngineVersion: get("RDS_ENGINE_VERSION"),
HandoffDir: get("RDS_HANDOFF_DIR"),
EngineHost: get("RDS_ENGINE_HOST"),
PGIsReady: get("RDS_PG_ISREADY"),
PGBin: get("RDS_PG_BIN"),
PGData: get("RDS_PGDATA"),
EngineBinDir: get("RDS_ENGINE_BIN"),
EngineDataDir: get("RDS_ENGINE_DATA"),
SocketDir: get("RDS_SOCKET_DIR"),
PGUser: get("RDS_PG_USER"),
EngineUser: get("RDS_ENGINE_USER"),
RCService: get("RDS_RC_SERVICE"),
EngineService: get("RDS_ENGINE_SERVICE"),
EnginePidFile: get("RDS_ENGINE_PIDFILE"),
DataMount: get("RDS_DATA_MOUNT"),
EnginePort: defaultEnginePort,
PollWait: defaultPollWait,
EngineProbeTimeout: defaultEngineProbeTimeout,
}
if cfg.GatewayCA == "" {
cfg.GatewayCA = defaultGatewayCA
@@ -98,32 +106,20 @@ func loadConfig(envFile string) config {
if cfg.EngineHost == "" {
cfg.EngineHost = defaultEngineHost
}
if cfg.PGIsReady == "" {
cfg.PGIsReady = defaultPGIsReady
}
if cfg.PGBin == "" {
cfg.PGBin = defaultPGBin
}
if cfg.PGData == "" {
cfg.PGData = defaultPGData
}
if cfg.SocketDir == "" {
cfg.SocketDir = defaultSocketDir
}
if cfg.PGUser == "" {
cfg.PGUser = defaultPGUser
if cfg.EngineFile == "" {
cfg.EngineFile = defaultEngineFile
}
if cfg.RCService == "" {
cfg.RCService = defaultRCService
}
if cfg.EngineService == "" {
cfg.EngineService = defaultEngineSvcName
}
// The image's own stamp rather than the delivered engine: an agent that took
// its layout from the payload could run one engine's implementation against
// another engine's datadir, which is the failure the stamp exists to prevent.
cfg.BakedEngine = readBakedEngine(cfg.EngineFile)
cfg.applyLayout(engineLayouts[cfg.BakedEngine])
// rds-datadir reads the same RDS_DATA_MOUNT, so the two agree on where the
// volume landed without either asserting it to the other.
if cfg.DataMount == "" {
cfg.DataMount = defaultDataMount
}
cfg.MountsFile = defaultMountsFile
cfg.SysBlock = defaultSysBlock
// The authoritative port comes from the bootstrap config; this is only what
@@ -138,5 +134,50 @@ func loadConfig(envFile string) config {
cfg.PollWait = d
}
}
if v := get("RDS_ENGINE_PROBE_TIMEOUT"); v != "" {
if d, err := time.ParseDuration(v); err == nil && d > 0 {
cfg.EngineProbeTimeout = d
}
}
return cfg
}
// Fills in whatever the delivered configuration left unset. An override always
// wins, so a test can point any of these at a fixture. An unrecognised engine
// leaves the layout empty, and New refuses rather than guessing at one.
func (c *config) applyLayout(layout engineLayout) {
if c.EngineBinDir == "" {
c.EngineBinDir = layout.binDir
}
if c.EngineDataDir == "" {
c.EngineDataDir = layout.dataDir
}
if c.SocketDir == "" {
c.SocketDir = layout.socketDir
}
if c.EngineUser == "" {
c.EngineUser = layout.osUser
}
if c.EngineService == "" {
c.EngineService = layout.service
}
if c.EnginePidFile == "" {
c.EnginePidFile = layout.pidFile
}
if c.DataMount == "" {
c.DataMount = layout.dataMount
}
if c.EnginePort == 0 {
c.EnginePort = layout.port
}
}
// The engine the image bakes. An absent or unreadable stamp reads as empty,
// which New refuses: an agent that guessed would defeat the stamp.
func readBakedEngine(path string) string {
raw, err := os.ReadFile(path)
if err != nil {
return ""
}
return strings.ToLower(strings.TrimSpace(string(raw)))
}
+108 -9
View File
@@ -3,21 +3,34 @@ package main
import (
"os"
"path/filepath"
"strings"
"testing"
"time"
)
// Stamps the engine a real image bakes and loads the configuration against it,
// which is what selects the guest layout the agent's defaults come from.
func testLoadConfig(t *testing.T, engine string) config {
t.Helper()
stamp := filepath.Join(t.TempDir(), "engine")
if err := os.WriteFile(stamp, []byte(engine+"\n"), 0o444); err != nil {
t.Fatalf("write the engine stamp: %v", err)
}
t.Setenv("RDS_ENGINE_FILE", stamp)
return loadConfig(filepath.Join(t.TempDir(), "absent.env"))
}
func TestLoadConfig_Defaults(t *testing.T) {
cfg := loadConfig(filepath.Join(t.TempDir(), "absent.env"))
cfg := testLoadConfig(t, enginePostgres)
if cfg.GatewayCA != defaultGatewayCA || cfg.HandoffDir != defaultHandoffDir {
t.Errorf("CA/handoff = %q/%q, want the built-in defaults", cfg.GatewayCA, cfg.HandoffDir)
}
if cfg.EngineHost != defaultEngineHost || cfg.EnginePort != defaultEnginePort {
t.Errorf("engine endpoint = %s:%d, want the local default", cfg.EngineHost, cfg.EnginePort)
if cfg.EngineHost != defaultEngineHost || cfg.PollWait != defaultPollWait {
t.Errorf("engine host/pollWait = %q/%v, want the built-in defaults", cfg.EngineHost, cfg.PollWait)
}
if cfg.PGIsReady != defaultPGIsReady || cfg.PollWait != defaultPollWait {
t.Errorf("probe/pollWait = %q/%v, want the built-in defaults", cfg.PGIsReady, cfg.PollWait)
if cfg.EngineProbeTimeout != defaultEngineProbeTimeout {
t.Errorf("engine probe timeout = %v, want %v", cfg.EngineProbeTimeout, defaultEngineProbeTimeout)
}
// Identity is optional: the gateway resolves it from the caller's
// credentials, so an agent with no configured identifier is normal.
@@ -26,11 +39,94 @@ func TestLoadConfig_Defaults(t *testing.T) {
}
}
// The guest layout is the image's to state, not the control plane's: everything
// the agent reaches the engine through comes from the stamp it was built with.
func TestLoadConfig_TakesTheLayoutFromTheBakedEngine(t *testing.T) {
cfg := testLoadConfig(t, enginePostgres)
if cfg.BakedEngine != enginePostgres {
t.Fatalf("baked engine = %q, want %q", cfg.BakedEngine, enginePostgres)
}
layout := engineLayouts[enginePostgres]
if cfg.EngineBinDir != layout.binDir || cfg.EngineDataDir != layout.dataDir {
t.Errorf("bin/data = %q/%q, want the PostgreSQL layout", cfg.EngineBinDir, cfg.EngineDataDir)
}
if cfg.EngineUser != layout.osUser || cfg.EngineService != layout.service {
t.Errorf("user/service = %q/%q, want the PostgreSQL layout", cfg.EngineUser, cfg.EngineService)
}
if cfg.SocketDir != layout.socketDir || cfg.DataMount != layout.dataMount {
t.Errorf("socket/mount = %q/%q, want the PostgreSQL layout", cfg.SocketDir, cfg.DataMount)
}
if cfg.EnginePort != layout.port {
t.Errorf("port = %d, want the PostgreSQL default %d", cfg.EnginePort, layout.port)
}
}
// An image with no stamp cannot say which engine it is, and an agent that
// guessed would be the failure the stamp exists to prevent.
func TestNewEngine_RefusesAnImageWithNoEngineStamp(t *testing.T) {
t.Setenv("RDS_ENGINE_FILE", filepath.Join(t.TempDir(), "absent-engine"))
cfg := loadConfig(filepath.Join(t.TempDir(), "absent.env"))
if _, err := newProbe(cfg, staticProbe(0)); err == nil {
t.Error("newProbe built a probe for an image carrying no engine stamp")
}
if _, err := newEngine(cfg, nil, nil, nil); err == nil {
t.Error("newEngine built an engine for an image carrying no engine stamp")
}
}
func TestNewEngine_RefusesAnEngineItDoesNotImplement(t *testing.T) {
cfg := testLoadConfig(t, "oracle")
_, err := newEngine(cfg, nil, nil, nil)
if err == nil {
t.Fatal("newEngine built an engine this agent does not implement")
}
if !strings.Contains(err.Error(), "oracle") {
t.Errorf("error = %v, want it to name the engine the image bakes", err)
}
}
// The implementation is the agent's, but the definition it validates names and
// classifies parameters against is the control plane's, which now offers MariaDB.
func TestNewEngine_BuildsMariaDBForAMariaDBImage(t *testing.T) {
cfg := testLoadConfig(t, engineMariaDB)
probe, err := newProbe(cfg, staticProbe(0))
if err != nil {
t.Fatalf("newProbe: %v", err)
}
built, err := newEngine(cfg, nil, nil, probe)
if err != nil {
t.Fatalf("newEngine: %v", err)
}
if _, ok := built.(*mariadbEngine); !ok {
t.Errorf("newEngine built %T for a MariaDB image", built)
}
}
// The probe reads the pidfile Alpine's packaged service starts mariadbd with,
// because nothing in the generated configuration can move it: the service passes
// --pid-file on the command line, which beats an option file.
func TestLoadConfig_TakesTheMariaDBPidfileFromTheLayout(t *testing.T) {
cfg := testLoadConfig(t, engineMariaDB)
if cfg.EnginePidFile != engineLayouts[engineMariaDB].pidFile {
t.Errorf("pidfile = %q, want the MariaDB layout's", cfg.EnginePidFile)
}
if cfg.EnginePidFile == "" {
t.Error("an engine whose probe needs a pidfile was given none")
}
}
func TestLoadConfig_ReadsEnvFile(t *testing.T) {
t.Setenv("RDS_ENGINE_FILE", filepath.Join(t.TempDir(), "absent-engine"))
path := filepath.Join(t.TempDir(), "agent.env")
body := "RDS_GATEWAY_URL=https://gw.internal:9999\n" +
"RDS_REGION=ap-southeast-2\n" +
"RDS_DB_INSTANCE_IDENTIFIER=db-1\n" +
"RDS_ENGINE=postgres\n" +
"RDS_ENGINE_VERSION=18.1\n" +
"RDS_ENGINE_PORT=6543\n" +
"RDS_POLL_WAIT=5s\n"
@@ -45,19 +141,22 @@ func TestLoadConfig_ReadsEnvFile(t *testing.T) {
if cfg.DBInstanceIdentifier != "db-1" || cfg.EngineVersion != "18.1" {
t.Errorf("identity = %q/%q, want db-1/18.1", cfg.DBInstanceIdentifier, cfg.EngineVersion)
}
if cfg.Engine != enginePostgres {
t.Errorf("delivered engine = %q, want the one cloud-init carried", cfg.Engine)
}
if cfg.EnginePort != 6543 || cfg.PollWait != 5*time.Second {
t.Errorf("port/pollWait = %d/%v, want 6543/5s", cfg.EnginePort, cfg.PollWait)
}
}
// An unparseable override falls back to the default rather than leaving the
// agent with a zero port or a zero-length long poll.
// An unparseable override falls back to the layout's own default rather than
// leaving the agent with a zero port or a zero-length long poll.
func TestLoadConfig_IgnoresUnusableOverrides(t *testing.T) {
t.Setenv("RDS_ENGINE_PORT", "not-a-port")
t.Setenv("RDS_POLL_WAIT", "forever")
cfg := loadConfig(filepath.Join(t.TempDir(), "absent.env"))
if cfg.EnginePort != defaultEnginePort || cfg.PollWait != defaultPollWait {
cfg := testLoadConfig(t, enginePostgres)
if cfg.EnginePort != engineLayouts[enginePostgres].port || cfg.PollWait != defaultPollWait {
t.Errorf("port/pollWait = %d/%v, want the defaults", cfg.EnginePort, cfg.PollWait)
}
}
+69
View File
@@ -0,0 +1,69 @@
package main
import (
"os"
"path/filepath"
"strings"
"testing"
handlers_rds "github.com/mulgadc/spinifex/spinifex/handlers/rds"
)
// The include the image build validates against a real mariadbd. It is checked
// in rather than generated during the build because the build runs inside the
// libguestfs appliance, where there is no Go toolchain to resolve the catalog.
const mariadbDefaultParametersFixture = "../../scripts/images/rds-mariadb/default-parameters.cnf"
// Byte-for-byte what rds-init installs on a smallest-class instance that
// overrides nothing. Regenerate with RDS_UPDATE_FIXTURES=1 go test ./cmd/rds-agent/.
//
// Keeping it in step matters because setup.sh feeds exactly this file to
// mariadbd: a catalog that grew a name the server refuses at startup would
// otherwise reach a customer as an instance that never boots.
func TestMariaDBDefaultParametersFixture_MatchesTheCatalog(t *testing.T) {
engine, err := handlers_rds.LookupEngine("mariadb")
if err != nil {
t.Fatalf("LookupEngine: %v", err)
}
class := handlers_rds.SmallestInstanceClass()
params, err := engine.ResolveEffectiveParameters(class, nil)
if err != nil {
t.Fatalf("ResolveEffectiveParameters(%s): %v", class, err)
}
body, err := renderParameters("mariadb", params)
if err != nil {
t.Fatalf("renderParameters: %v", err)
}
want := mariadbParametersHeader + body
path := filepath.FromSlash(mariadbDefaultParametersFixture)
if os.Getenv("RDS_UPDATE_FIXTURES") == "1" {
if err := os.WriteFile(path, []byte(want), 0o644); err != nil {
t.Fatalf("update %s: %v", path, err)
}
return
}
got, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read %s: %v", path, err)
}
if string(got) != want {
t.Errorf("%s is stale for class %s.\nRegenerate with RDS_UPDATE_FIXTURES=1 go test ./cmd/rds-agent/\n\ngot:\n%s\nwant:\n%s",
path, class, got, want)
}
}
// The one thing the fixture exists to prove cannot regress without the image
// build catching it, asserted here too so a plain preflight names it.
func TestMariaDBDefaultParametersFixture_CarriesNoSetOnlyName(t *testing.T) {
got, err := os.ReadFile(filepath.FromSlash(mariadbDefaultParametersFixture))
if err != nil {
t.Fatalf("read the fixture: %v", err)
}
for _, name := range []string{"\ntime_zone ", "\ntx_isolation ", "\ninnodb_buffer_pool_instances "} {
if strings.Contains(string(got), name) {
t.Errorf("the default include carries %q, which mariadbd refuses at startup", name)
}
}
}
+126 -414
View File
@@ -3,25 +3,21 @@ package main
import (
"bytes"
"context"
"errors"
"fmt"
"os"
"os/exec"
"os/user"
"path/filepath"
"strconv"
"strings"
"sync"
"sync/atomic"
"syscall"
"time"
handlers_rds "github.com/mulgadc/spinifex/spinifex/handlers/rds"
)
// The in-guest half of the engine seam. Everything PostgreSQL-shaped that a
// control-plane command needs lives behind this, so the command registry above
// stays engine-agnostic.
// The in-guest half of the engine seam. Everything engine-shaped that a
// control-plane command needs lives behind this, so the command registry stays
// engine-agnostic.
type engineOps interface {
// Rotates the master role's password live. Never persisted anywhere in the
// guest: it exists only for the length of this call.
@@ -49,13 +45,119 @@ type parameterRecovery interface {
Restart(ctx context.Context) error
}
// Records the parameter file only after the probe observes the postmaster
// serving it. This is separate from apply, where static values are installed
// but not adopted until a later restart.
// Records the parameter file only after the probe observes the engine serving
// it. This is separate from apply, where static values are installed but not
// adopted until a later restart.
type servingParameterRecorder interface {
RecordServingParameters(ctx context.Context) error
}
// Everything one engine implementation owns. The narrow interfaces above are
// what the heartbeat, the commander and the rollback guard each hold; this is
// only what the agent has to construct.
type engine interface {
engineOps
parameterRecovery
servingParameterRecorder
// Follows the port the control plane assigned, once the bootstrap fetch
// lands. Called from a different goroutine than the commands that read it.
setPort(port int)
}
// The guest layout an engine preset lays down, keyed by the engine the image
// bakes. Every default that belongs to one engine lives here, so nothing
// outside an implementation names its paths, service, port or probe.
type engineLayout struct {
binDir string
dataDir string
socketDir string
osUser string
service string
dataMount string
port int
// Where the engine records the pid the probe checks for liveness. Empty for
// an engine whose probe answers without one.
pidFile string
// The engine's own liveness signal, which is the only part of engineProbe
// that is not shared.
newProbe func(cfg config, run probeRunner) *engineProbe
// The implementation every control-plane directive is served by. It fails
// when the control plane this agent was built against does not offer the
// engine the image bakes, rather than serving one it has no definition for.
newEngine func(cfg config, run commandRunner, startSess sessionRunner, probe *engineProbe) (engine, error)
}
// The names the control plane knows these implementations by, which are also
// what setup.sh stamps into each image.
const (
enginePostgres = "postgres"
engineMariaDB = "mariadb"
)
var engineLayouts = map[string]engineLayout{
enginePostgres: {
binDir: "/usr/libexec/postgresql18",
dataDir: "/var/lib/postgresql/18/data",
socketDir: "/run/postgresql",
osUser: "postgres",
service: "postgresql",
dataMount: "/var/lib/postgresql",
port: 5432,
newProbe: newPostgresProbe,
newEngine: func(cfg config, run commandRunner, startSess sessionRunner, probe *engineProbe) (engine, error) {
return newPostgresEngine(cfg, run, startSess, probe), nil
},
},
engineMariaDB: {
binDir: "/usr/bin",
dataDir: "/var/lib/mysql/data",
socketDir: "/run/mysqld",
osUser: "mysql",
service: "mariadb",
dataMount: "/var/lib/mysql",
port: 3306,
// Alpine's packaged service starts mysqld_safe with
// --pid-file=/run/mysqld/$RC_SVCNAME.pid, and a command-line option beats
// the configuration file, so this is the path rather than anything the
// platform drop-in could name. setup.sh asserts it against the packaged
// init script at build time.
pidFile: "/run/mysqld/mariadb.pid",
newProbe: newMariaDBProbe,
newEngine: newMariaDBEngineFromCatalog,
},
}
// Builds the health probe the image's engine supplies.
func newProbe(cfg config, run probeRunner) (*engineProbe, error) {
layout, err := layoutFor(cfg)
if err != nil {
return nil, err
}
return layout.newProbe(cfg, run), nil
}
// Builds the implementation the image bakes. Nothing here consults the control
// plane for which engine to be: the agent runs the engine it is made of, and the
// delivered engine is only ever checked against it.
func newEngine(cfg config, run commandRunner, startSess sessionRunner, probe *engineProbe) (engine, error) {
layout, err := layoutFor(cfg)
if err != nil {
return nil, err
}
return layout.newEngine(cfg, run, startSess, probe)
}
func layoutFor(cfg config) (engineLayout, error) {
if cfg.BakedEngine == "" {
return engineLayout{}, fmt.Errorf("this image carries no engine stamp at %s", cfg.EngineFile)
}
layout, ok := engineLayouts[cfg.BakedEngine]
if !ok {
return engineLayout{}, fmt.Errorf("this image bakes engine %q, which this agent does not implement", cfg.BakedEngine)
}
return layout, nil
}
// One child process. Env replaces the agent's own environment rather than
// extending it, so a secret placed here reaches only the process that needs it.
type command struct {
@@ -65,10 +167,14 @@ type command struct {
Stdin string
// The OS user to drop to before exec. Empty runs as the agent itself.
User string
// The statement that makes this client print the session sentinel. Read only
// when the command is started as a held session, whose reader has to know
// when the engine has finished each script it was fed.
SentinelStatement string
}
// Returns stdout. stderr is folded into the error, because psql reports the
// actual SQL failure there while exiting with a bare status.
// Returns stdout. stderr is folded into the error, because both clients report
// the actual SQL failure there while exiting with a bare status.
type commandRunner func(ctx context.Context, c command) (string, error)
func execCommandRunner(ctx context.Context, c command) (string, error) {
@@ -113,384 +219,16 @@ func lookupCredential(username string) (*syscall.Credential, error) {
return &syscall.Credential{Uid: uint32(uid), Gid: uint32(gid)}, nil
}
// The engine is reached over its unix socket under peer authentication, so the
// agent drops to the postgres OS user rather than holding a password of its own.
type postgresEngine struct {
run commandRunner
startSess sessionRunner
psql string
rcService string
service string
pgData string
socketDir string
osUser string
probe *engineProbe
// Set from the bootstrap config, on a different goroutine than the commands
// that read it.
port atomic.Int64
// Serializes parameter installs, serving snapshots and rollback restores so
// none can copy or replace an intermediate configuration.
paramMu sync.Mutex
repairTimeout time.Duration
repairPoll time.Duration
// The backup mode currently held, if any. Guarded because the expiry timer
// releases it on its own goroutine.
mu sync.Mutex
held *quiesceHold
}
var (
_ engineOps = (*postgresEngine)(nil)
_ parameterRecovery = (*postgresEngine)(nil)
_ servingParameterRecorder = (*postgresEngine)(nil)
)
const (
parameterRepairTimeout = 90 * time.Second
parameterRepairPoll = time.Second
)
func newPostgresEngine(cfg config, run commandRunner, startSess sessionRunner, probe *engineProbe) *postgresEngine {
e := &postgresEngine{
run: run,
startSess: startSess,
psql: filepath.Join(cfg.PGBin, "psql"),
rcService: cfg.RCService,
service: cfg.EngineService,
pgData: cfg.PGData,
socketDir: cfg.SocketDir,
osUser: cfg.PGUser,
probe: probe,
repairTimeout: parameterRepairTimeout,
repairPoll: parameterRepairPoll,
}
e.port.Store(int64(cfg.EnginePort))
return e
}
func (e *postgresEngine) setPort(port int) {
e.port.Store(int64(port))
}
// The role name and the password ride the environment and are re-quoted by
// psql, so neither reaches a shell word or an argv another process can read.
func (e *postgresEngine) SetPassword(ctx context.Context, username, password string) error {
if username == "" || password == "" {
return fmt.Errorf("set-password requires both %s and %s",
handlers_rds.CommandParamMasterUsername, handlers_rds.CommandParamMasterUserPassword)
}
// The statement below runs as the cluster superuser over the socket under
// peer auth, so a reserved name in the command payload would hand the
// customer the bootstrap superuser rather than rotate their own role.
if err := handlers_rds.EnginePostgres().ValidateUsernameNotReserved(username); err != nil {
return fmt.Errorf("refusing to set the password of a role the engine reserves: %w", err)
}
// psql interpolates the password into the ALTER ROLE before the server sees
// it, and these three are what would write it to the log — the last on any
// failure at its own default. SUSET, so the parameter group cannot win.
const sql = `SET log_statement = 'none';
SET log_min_duration_statement = -1;
SET log_min_error_statement = 'panic';
\getenv master RDS_MASTER_USERNAME
\getenv password RDS_MASTER_PASSWORD
ALTER ROLE :"master" WITH LOGIN PASSWORD :'password';
`
_, err := e.psqlRun(ctx, sql,
"RDS_MASTER_USERNAME="+username,
"RDS_MASTER_PASSWORD="+password,
)
if err != nil {
// psql echoes the failing statement, which here would carry the new
// password back to the control plane and into the event ring.
return fmt.Errorf("apply the master password: %s", redact(err.Error(), password))
}
return nil
}
// The include the resolved parameter set is rendered to, and the copy of the
// last one the engine accepted. Both live beside the data rather than in /etc,
// matching rds-init: a class change boots a fresh root volume, which would
// otherwise revert them when a fresh root volume is used.
const (
parametersFileName = "10-rds-parameters.conf"
// Deliberately not a .conf name: include_dir globs *.conf, so the rollback
// copy must not be read as a second set of settings.
lastGoodFileName = "10-rds-parameters.last-good"
)
// Installs the resolved set, validates it with the engine's own config parser
// before the engine ever adopts it, and reloads. A value the engine refuses is
// rolled back here rather than left on the data volume, where it would survive
// every VM replace and turn the next restart into a boot loop.
func (e *postgresEngine) ApplyParameters(ctx context.Context, params []handlers_rds.Parameter) ([]string, error) {
e.paramMu.Lock()
defer e.paramMu.Unlock()
initialState, _ := e.probe.state(ctx)
if initialState == engineRecovering {
return nil, errors.New("apply parameters while the engine is still starting or recovering")
}
if initialState == engineServing {
// A command can arrive before the first heartbeat. Seed the configuration
// currently being served before replacing its file in that window.
if err := e.recordServingParametersLocked(ctx); err != nil {
return nil, fmt.Errorf("record the parameters serving before the apply: %w", err)
}
}
// The check has to run against the file in place, because the engine parses
// the datadir's own include_dir. The window that leaves is closed from both
// ends: the rollback below, and the last-known-good restore the agent runs at
// boot when the engine does not come up.
_, restore, err := e.installParameters(params)
if err != nil {
return nil, err
}
if err := e.checkConfig(ctx); err != nil {
return nil, errors.Join(fmt.Errorf("the engine rejected the parameter set: %w", err), restore())
}
state, _ := e.probe.state(ctx)
switch state {
case engineAbsent:
return e.restartOnRepairSetLocked(ctx)
case engineRecovering:
return nil, errors.Join(errors.New("the engine entered startup or recovery during the parameter apply"), restore())
}
if _, err := e.psqlRun(ctx, "SELECT pg_reload_conf();\n"); err != nil {
// The engine may have gone down after the first probe. In that case start
// it on the new, parser-checked repair set rather than restoring the set
// that already left it unable to serve.
if state, _ := e.probe.state(ctx); state == engineAbsent {
return e.restartOnRepairSetLocked(ctx)
}
return nil, errors.Join(fmt.Errorf("reload the engine configuration: %w", err), restore())
}
return e.pendingRestartParameters(ctx)
}
func (e *postgresEngine) restartOnRepairSetLocked(ctx context.Context) ([]string, error) {
repairCtx, cancel := context.WithTimeout(ctx, e.repairTimeout)
defer cancel()
if err := e.Restart(repairCtx); err != nil {
return nil, fmt.Errorf("start the engine on the repaired parameter set: %w", err)
}
timer := time.NewTimer(0)
defer timer.Stop()
lastMessage := "engine did not respond"
for {
select {
case <-repairCtx.Done():
if ctx.Err() != nil {
return nil, fmt.Errorf("wait for the engine on the repaired parameter set: %w", ctx.Err())
}
return nil, fmt.Errorf("wait for the engine on the repaired parameter set: %s: %w", lastMessage, repairCtx.Err())
case <-timer.C:
}
state, message := e.probe.state(repairCtx)
if state == engineServing {
pending, err := e.pendingRestartParameters(repairCtx)
if err == nil {
return pending, nil
}
lastMessage = err.Error()
} else if message != "" {
lastMessage = message
}
timer.Reset(e.repairPoll)
}
}
func (e *postgresEngine) pendingRestartParameters(ctx context.Context) ([]string, error) {
out, err := e.psqlRun(ctx, "SELECT name FROM pg_settings WHERE pending_restart ORDER BY name;\n")
if err != nil {
return nil, fmt.Errorf("read the settings pending a restart: %w", err)
}
var pending []string
for line := range strings.SplitSeq(out, "\n") {
if name := strings.TrimSpace(line); name != "" {
pending = append(pending, name)
}
}
return pending, nil
}
func (e *postgresEngine) parametersPath() string {
return filepath.Join(e.pgData, "conf.d", parametersFileName)
}
func (e *postgresEngine) lastGoodPath() string {
return filepath.Join(e.pgData, "conf.d", lastGoodFileName)
}
// Written as root and handed to the engine's user, at the same path and mode
// rds-init installs it at, so a later boot overwrites rather than shadows it.
// Returns the content that was there and a func putting it back, so the caller
// can undo the install without re-rendering anything.
func (e *postgresEngine) installParameters(params []handlers_rds.Parameter) (previous []byte, restore func() error, err error) {
path := e.parametersPath()
// A missing file is the first-ever apply, and its rollback is a removal.
previous, readErr := os.ReadFile(path)
if readErr != nil && !os.IsNotExist(readErr) {
return nil, nil, fmt.Errorf("read the installed parameters at %s: %w", path, readErr)
}
existed := readErr == nil
if err := e.writeEngineFile(path, []byte(renderParameters(params))); err != nil {
return nil, nil, err
}
return previous, func() error {
if !existed {
if err := os.Remove(path); err != nil && !os.IsNotExist(err) {
return fmt.Errorf("withdraw the rejected parameters at %s: %w", path, err)
}
return nil
}
return e.writeEngineFile(path, previous)
}, nil
}
// Snapshots the include only when the running postmaster has no settings still
// pending a restart. The parameter mutex keeps an apply from replacing the file
// between that check and the copy.
func (e *postgresEngine) RecordServingParameters(ctx context.Context) error {
e.paramMu.Lock()
defer e.paramMu.Unlock()
return e.recordServingParametersLocked(ctx)
}
func (e *postgresEngine) recordServingParametersLocked(ctx context.Context) error {
if _, err := os.Stat(e.parametersPath()); err != nil {
if os.IsNotExist(err) {
return nil
}
return fmt.Errorf("inspect the serving parameters: %w", err)
}
pending, err := e.pendingRestartParameters(ctx)
if err != nil {
return err
}
if len(pending) > 0 {
return nil
}
content, err := os.ReadFile(e.parametersPath())
if err != nil {
return fmt.Errorf("read the serving parameters: %w", err)
}
lastGood, err := os.ReadFile(e.lastGoodPath())
if err == nil && bytes.Equal(content, lastGood) {
return nil
}
if err != nil && !os.IsNotExist(err) {
return fmt.Errorf("read the last known good parameters: %w", err)
}
return e.writeEngineFile(e.lastGoodPath(), content)
}
// Puts the last set the engine accepted back in place, for a restart that failed
// after a parameter change. The probe is checked again under the parameter lock
// so a repair that just brought the engine back cannot be reversed.
func (e *postgresEngine) RestoreLastKnownGoodParameters(ctx context.Context) (bool, error) {
e.paramMu.Lock()
defer e.paramMu.Unlock()
if state, _ := e.probe.state(ctx); state != engineAbsent {
return false, nil
}
lastGood, err := os.ReadFile(e.lastGoodPath())
if err != nil {
if os.IsNotExist(err) {
return false, nil
}
return false, fmt.Errorf("read the last known good parameters: %w", err)
}
current, err := os.ReadFile(e.parametersPath())
if err != nil && !os.IsNotExist(err) {
return false, fmt.Errorf("read the installed parameters: %w", err)
}
if bytes.Equal(current, lastGood) {
return false, nil
}
if err := e.writeEngineFile(e.parametersPath(), lastGood); err != nil {
return false, err
}
return true, nil
}
// Atomic: a temp file in the same directory, renamed over the target. The temp
// name deliberately does not end in .conf, so a crash between write and rename
// cannot leave the engine reading a half-written include.
func (e *postgresEngine) writeEngineFile(path string, content []byte) error {
tmp := path + ".new"
if err := os.WriteFile(tmp, content, 0o600); err != nil {
return fmt.Errorf("write %s: %w", tmp, err)
}
if err := e.chownToEngine(tmp); err != nil {
os.Remove(tmp)
return err
}
if err := os.Rename(tmp, path); err != nil {
os.Remove(tmp)
return fmt.Errorf("install %s: %w", path, err)
}
return nil
}
// The engine's own parser, run offline against the datadir. Reading one setting
// back is enough: postgres parses postgresql.conf and every include first, and
// exits non-zero naming the file and line of an unknown parameter or a value
// outside its range.
func (e *postgresEngine) checkConfig(ctx context.Context) error {
if _, err := e.run(ctx, command{
Name: filepath.Join(filepath.Dir(e.psql), "postgres"),
Args: []string{"-D", e.pgData, "-C", "shared_buffers"},
Env: []string{"PATH=" + defaultGuestPath},
User: e.osUser,
}); err != nil {
return err
}
return nil
}
// The engine reads its own config, so a root-owned file would be unreadable to
// it. A guest with no such user is a broken image, not something to work around.
func (e *postgresEngine) chownToEngine(path string) error {
credential, err := lookupCredential(e.osUser)
if err != nil {
return err
}
if err := os.Chown(path, int(credential.Uid), int(credential.Gid)); err != nil {
return fmt.Errorf("hand %s to %s: %w", path, e.osUser, err)
}
return nil
}
// Through the service manager rather than pg_ctl, so the supervisor records the
// engine as stopped and does not restart it underneath a VM that is going down.
func (e *postgresEngine) Stop(ctx context.Context) error {
return e.serviceAction(ctx, "stop")
}
// Only the parameter rollback calls this: a restart the control plane wants goes
// through RebootDBInstance, which cycles the VM.
func (e *postgresEngine) Restart(ctx context.Context) error {
return e.serviceAction(ctx, "restart")
}
func (e *postgresEngine) serviceAction(ctx context.Context, action string) error {
if _, err := e.run(ctx, command{
Name: e.rcService,
Args: []string{e.service, action},
// Through the service manager rather than the engine's own control tool, so the
// supervisor records the state and does not restart the engine underneath a VM
// that is going down.
func serviceAction(ctx context.Context, run commandRunner, rcService, service, action string) error {
if _, err := run(ctx, command{
Name: rcService,
Args: []string{service, action},
Env: []string{"PATH=" + defaultGuestPath},
}); err != nil {
return fmt.Errorf("%s the %s service: %w", action, e.service, err)
return fmt.Errorf("%s the %s service: %w", action, service, err)
}
return nil
}
@@ -498,32 +236,6 @@ func (e *postgresEngine) serviceAction(ctx context.Context, action string) error
// A minimal environment for a child that is not carrying secrets.
const defaultGuestPath = "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
// Feeds sql on stdin rather than as an argument, so a statement is never
// visible in the process table.
func (e *postgresEngine) psqlRun(ctx context.Context, sql string, env ...string) (string, error) {
return e.run(ctx, command{
Name: e.psql,
Args: e.psqlArgs(),
Env: append([]string{"PATH=" + defaultGuestPath}, env...),
Stdin: sql,
User: e.osUser,
})
}
// Reading the script from stdin is what lets one invocation serve both a
// one-shot run and a session held open across several statements.
func (e *postgresEngine) psqlArgs() []string {
return []string{
"--no-psqlrc", "--quiet", "--no-align", "--tuples-only",
"-v", "ON_ERROR_STOP=1",
"-h", e.socketDir,
"-p", strconv.FormatInt(e.port.Load(), 10),
"-U", e.osUser,
"-d", "postgres",
"-f", "-",
}
}
func redact(s, secret string) string {
if secret == "" {
return s
+329
View File
@@ -0,0 +1,329 @@
package main
import (
"context"
"errors"
"os"
"slices"
"strings"
"testing"
"time"
handlers_rds "github.com/mulgadc/spinifex/spinifex/handlers/rds"
)
// A probe the test drives directly, in place of whichever binaries the engine's
// own state function would have run. Each implementation's state function is
// covered on its own; what these cases exercise is what the engine does with the
// answer.
func drivenProbe(state *engineState) *engineProbe {
return newEngineProbe(0, func(context.Context, int64) (engineState, string) {
return *state, "driven by the test"
})
}
// One engine behind the interfaces the heartbeat, the commander and the rollback
// guard hold, plus the few facts a shared case needs to say what it means.
type engineHarness struct {
engine engine
runner *recordingRunner
params parameterStore
service string
// The state the probe reports, moved by the test rather than by the engine.
state *engineState
// Points the next quiesce at a session the test can inspect, and returns the
// commands those sessions were opened with.
useSession func(*fakeSession) *[]command
// A setting this engine's catalog really carries, so a shared case installs
// something the implementation would genuinely be handed.
parameter handlers_rds.Parameter
// What the engine's own include directive globs. Neither the rollback copy
// nor the serving copy may end in it.
includeSuffix string
}
func engineHarnesses() []struct {
name string
build func(*testing.T) *engineHarness
} {
return []struct {
name string
build func(*testing.T) *engineHarness
}{
{name: enginePostgres, build: buildPostgresHarness},
{name: engineMariaDB, build: buildMariaDBHarness},
}
}
func forEachEngine(t *testing.T, run func(*testing.T, *engineHarness)) {
t.Helper()
for _, fixture := range engineHarnesses() {
t.Run(fixture.name, func(t *testing.T) { run(t, fixture.build(t)) })
}
}
func buildPostgresHarness(t *testing.T) *engineHarness {
t.Helper()
runner := &recordingRunner{}
e := newTestEngine(t, runner.run)
state := engineServing
e.probe = drivenProbe(&state)
return &engineHarness{
engine: e, runner: runner, params: e.params, service: e.service, state: &state,
useSession: func(session *fakeSession) *[]command {
started := &[]command{}
e.startSess = func(_ context.Context, c command) (engineSession, error) {
*started = append(*started, c)
return session, nil
}
return started
},
parameter: handlers_rds.Parameter{Name: "work_mem", Value: "4096"},
includeSuffix: ".conf",
}
}
func buildMariaDBHarness(t *testing.T) *engineHarness {
t.Helper()
runner := &recordingRunner{}
e := newTestMariaDBEngine(t, runner.run)
state := engineServing
e.probe = drivenProbe(&state)
return &engineHarness{
engine: e, runner: runner, params: e.params, service: e.service, state: &state,
useSession: func(session *fakeSession) *[]command {
started := &[]command{}
e.startSess = func(_ context.Context, c command) (engineSession, error) {
*started = append(*started, c)
return session, nil
}
return started
},
parameter: handlers_rds.Parameter{Name: "max_connections", Value: "200"},
includeSuffix: ".cnf",
}
}
// The parameters land where rds-init installs them, so the next boot overwrites
// the file rather than shadowing it with a second copy.
func TestEngineContract_ApplyParametersInstallsWhereRdsInitDoes(t *testing.T) {
forEachEngine(t, func(t *testing.T, h *engineHarness) {
if _, err := h.engine.ApplyParameters(t.Context(), []handlers_rds.Parameter{h.parameter}); err != nil {
t.Fatalf("ApplyParameters: %v", err)
}
installed, err := os.ReadFile(h.params.installedPath())
if err != nil {
t.Fatalf("read the installed parameters: %v", err)
}
if !strings.Contains(string(installed), h.parameter.Name+" = '"+h.parameter.Value+"'") {
t.Errorf("installed parameters = %q, want the resolved value", installed)
}
if !strings.HasSuffix(h.params.installedPath(), h.includeSuffix) {
t.Errorf("the installed set at %s is not a file the engine includes", h.params.installedPath())
}
// Comparison material, not a second set of settings: an include glob that
// read either of these would apply a withdrawn configuration.
copies := []string{h.params.lastGoodPath()}
if h.params.serving != "" {
copies = append(copies, h.params.servingPath())
}
for _, path := range copies {
if strings.HasSuffix(path, h.includeSuffix) {
t.Errorf("the copy at %s would be included by the engine", path)
}
}
})
}
// An engine still starting or replaying has not finished adopting the set it is
// on, so replacing that set underneath it would be installing over a moving
// target.
func TestEngineContract_ApplyParametersRefusesWhileTheEngineIsRecovering(t *testing.T) {
forEachEngine(t, func(t *testing.T, h *engineHarness) {
*h.state = engineRecovering
if _, err := h.engine.ApplyParameters(t.Context(), []handlers_rds.Parameter{h.parameter}); err == nil {
t.Fatal("ApplyParameters succeeded against an engine that was still coming up")
}
if _, err := os.Stat(h.params.installedPath()); !os.IsNotExist(err) {
t.Errorf("the refused set was installed anyway (stat err = %v)", err)
}
})
}
// The boot-time half of the parameter safety net: the last set the engine
// accepted goes back only when the engine is genuinely down, and only once.
func TestEngineContract_RestoreLastKnownGoodOnlyRollsBackADownEngine(t *testing.T) {
forEachEngine(t, func(t *testing.T, h *engineHarness) {
accepted := h.parameter
if _, err := h.engine.ApplyParameters(t.Context(), []handlers_rds.Parameter{accepted}); err != nil {
t.Fatalf("ApplyParameters: %v", err)
}
if err := h.engine.RecordServingParameters(t.Context()); err != nil {
t.Fatalf("RecordServingParameters: %v", err)
}
if _, err := h.engine.ApplyParameters(t.Context(),
[]handlers_rds.Parameter{{Name: accepted.Name, Value: accepted.Value + "0"}}); err != nil {
t.Fatalf("second ApplyParameters: %v", err)
}
// A guard whose deadline expired while an apply brought the engine back must
// not reverse that repair.
restored, err := h.engine.RestoreLastKnownGoodParameters(t.Context())
if err != nil {
t.Fatalf("RestoreLastKnownGoodParameters against a serving engine: %v", err)
}
if restored {
t.Fatal("rolled the parameters back while the engine was serving")
}
*h.state = engineAbsent
restored, err = h.engine.RestoreLastKnownGoodParameters(t.Context())
if err != nil {
t.Fatalf("RestoreLastKnownGoodParameters: %v", err)
}
if !restored {
t.Fatal("did not roll back to a set that differs from the last accepted one")
}
installed, err := os.ReadFile(h.params.installedPath())
if err != nil {
t.Fatalf("read the installed parameters: %v", err)
}
if !strings.Contains(string(installed), accepted.Name+" = '"+accepted.Value+"'") {
t.Errorf("installed = %q, want the last accepted set", installed)
}
// Repeating a rollback would only churn a cluster that is failing for
// another reason.
restored, err = h.engine.RestoreLastKnownGoodParameters(t.Context())
if err != nil {
t.Fatalf("RestoreLastKnownGoodParameters: %v", err)
}
if restored {
t.Error("rolled back again when the installed set already matched")
}
})
}
// Through the service manager rather than the engine's own control tool, so the
// supervisor records the engine as stopped and does not restart it underneath a
// VM that is going down.
func TestEngineContract_StopGoesThroughTheServiceManager(t *testing.T) {
forEachEngine(t, func(t *testing.T, h *engineHarness) {
if err := h.engine.Stop(t.Context()); err != nil {
t.Fatalf("Stop: %v", err)
}
if len(h.runner.calls) != 1 {
t.Fatalf("ran %d commands, want the service stop", len(h.runner.calls))
}
call := h.runner.calls[0]
if !slices.Equal(call.Args, []string{h.service, "stop"}) {
t.Errorf("stop ran %s %v, want the service manager", call.Name, call.Args)
}
})
}
// One backup at a time, released by Unquiesce or by its own deadline, whichever
// comes first. The deadline is what keeps a control plane that dies mid-snapshot
// from leaving the engine held.
func TestEngineContract_QuiesceHoldsUntilReleased(t *testing.T) {
forEachEngine(t, func(t *testing.T, h *engineHarness) {
session := &fakeSession{}
started := h.useSession(session)
if err := h.engine.Quiesce(t.Context(), "orders-db-pre-upgrade", time.Minute); err != nil {
t.Fatalf("Quiesce: %v", err)
}
if len(*started) != 1 {
t.Fatalf("started %d sessions, want 1", len(*started))
}
if session.closes() != 0 {
t.Fatal("the session was closed while the backup was meant to be held")
}
err := h.engine.Quiesce(t.Context(), "second", time.Minute)
if err == nil {
t.Fatal("a second quiesce was accepted while the first was held")
}
if !strings.Contains(err.Error(), "orders-db-pre-upgrade") {
t.Errorf("error %q does not name the backup already holding the engine", err)
}
if len(*started) != 1 {
t.Errorf("started %d sessions, want the second refused before opening one", len(*started))
}
if err := h.engine.Unquiesce(t.Context()); err != nil {
t.Fatalf("Unquiesce: %v", err)
}
if session.closes() != 1 {
t.Errorf("closed %d times, want the session ended exactly once", session.closes())
}
})
}
// A hold that expired mid-snapshot means the snapshot was not taken against a
// held checkpoint, so the release reports it rather than succeeding silently.
func TestEngineContract_QuiesceReleasesItselfOnItsDeadline(t *testing.T) {
forEachEngine(t, func(t *testing.T, h *engineHarness) {
session := &fakeSession{}
h.useSession(session)
if err := h.engine.Quiesce(t.Context(), "orders-db-pre-upgrade", 10*time.Millisecond); err != nil {
t.Fatalf("Quiesce: %v", err)
}
deadline := time.Now().Add(2 * time.Second)
for session.closes() == 0 && time.Now().Before(deadline) {
time.Sleep(time.Millisecond)
}
if session.closes() != 1 {
t.Fatal("the backup session outlived its deadline")
}
err := h.engine.Unquiesce(t.Context())
if err == nil {
t.Fatal("Unquiesce succeeded after the hold expired")
}
if !strings.Contains(err.Error(), "expired") {
t.Errorf("error %q does not say the hold had expired", err)
}
})
}
// A failed start leaves nothing held, so a retry is not refused by a hold that
// never took effect.
func TestEngineContract_QuiesceEndsTheSessionWhenTheBackupWillNotStart(t *testing.T) {
forEachEngine(t, func(t *testing.T, h *engineHarness) {
session := &fakeSession{execErr: errors.New("the engine refused the backup")}
h.useSession(session)
if err := h.engine.Quiesce(t.Context(), "orders-db-pre-upgrade", time.Minute); err == nil {
t.Fatal("Quiesce succeeded against an engine that refused the backup")
}
if session.closes() != 1 {
t.Errorf("closed %d times, want the failed session ended", session.closes())
}
if err := h.engine.Unquiesce(t.Context()); err == nil {
t.Error("a failed quiesce left a hold behind")
}
})
}
// Every command the control plane can issue is served by both implementations,
// so a directive cannot be answered as unsupported on one engine and honoured on
// the other.
func TestEngineContract_ServesEveryRegisteredCommand(t *testing.T) {
forEachEngine(t, func(t *testing.T, h *engineHarness) {
registry := newCommandRegistry(h.engine, &fakeStorage{})
for _, name := range []string{
handlers_rds.CommandSetPassword, handlers_rds.CommandApplyParams, handlers_rds.CommandStopEngine,
handlers_rds.CommandGrowFilesystem, handlers_rds.CommandQuiesce, handlers_rds.CommandUnquiesce,
} {
if _, ok := registry[name]; !ok {
t.Errorf("command %q has no handler", name)
}
}
})
}
+638
View File
@@ -0,0 +1,638 @@
package main
import (
"context"
"errors"
"fmt"
"log/slog"
"os"
"path/filepath"
"slices"
"strconv"
"strings"
"sync"
"syscall"
"time"
handlers_rds "github.com/mulgadc/spinifex/spinifex/handlers/rds"
)
// The control plane's own statements about this engine, resolved once at
// startup. Held as functions rather than as the Engine value so nothing here
// re-derives a rule the API already owns.
type controlPlaneRules struct {
// The whole master-username rule rather than only the reserved set. MariaDB's
// client has no identifier quoting, so the name is interpolated into SQL by
// this process and every part of the rule is load-bearing here.
validateUsername func(username string) error
// Whether a setting takes effect only at a restart. Already customer-facing
// and authoritative, since the API refuses ApplyMethod=immediate on one.
isStatic func(name string) bool
// The catalog name behind a spelling read back out of an option file. Only a
// setting whose startup spelling differs from the customer's moves.
catalogName func(optionFileName string) string
}
func controlPlaneRulesFrom(meta handlers_rds.Engine) controlPlaneRules {
// Built once per engine: everything read back out of a generated file has to
// return to the catalog's namespace before it is classified or reported, or a
// startup spelling would read as an unknown name.
catalogNames := map[string]string{}
for _, name := range meta.CatalogParameterNames() {
if optionFileName := meta.OptionFileName(name); optionFileName != name {
catalogNames[optionFileName] = name
}
}
return controlPlaneRules{
validateUsername: meta.ValidateMasterUsername,
isStatic: func(name string) bool {
spec, ok := meta.LookupParameter(name)
// A name the catalog does not carry cannot be shown to have been adopted
// without a restart, and is never issued as a live SET GLOBAL either.
return !ok || spec.ApplyType == handlers_rds.ApplyTypeStatic
},
catalogName: func(optionFileName string) string {
if name, ok := catalogNames[optionFileName]; ok {
return name
}
return optionFileName
},
}
}
// The engine is reached over its unix socket as root, which the datadir's
// unix_socket plugin authenticates from the connecting process' own uid — the
// direct analogue of PostgreSQL's peer auth, and the reason the agent holds no
// password of its own.
type mariadbEngine struct {
quiesceState
rules controlPlaneRules
run commandRunner
startSess sessionRunner
client string
admin string
socket string
rcService string
service string
probe *engineProbe
params parameterStore
// Serializes parameter installs, serving snapshots and rollback restores so
// none can copy or replace an intermediate configuration.
paramMu sync.Mutex
repairTimeout time.Duration
repairPoll time.Duration
}
var _ engine = (*mariadbEngine)(nil)
const (
mariadbClientBinary = "mariadb"
mariadbAdminBinary = "mariadb-admin"
// The account mariadb-install-db creates for the unix_socket plugin, which
// rds-init keeps for the platform and never hands to the customer.
mariadbSuperuser = "root"
mariadbProbeConnectTimeoutSeconds = 3
// Named by the platform drop-in rds-init writes, so both halves reach the
// same socket without either asserting it to the other.
mariadbSocketFile = "mysqld.sock"
// The customer reaches the instance over its customer ENI, so the master is
// created and rotated on the wildcard host rather than on localhost.
mariadbMasterHost = "%"
)
// The generated drop-ins on the data volume. The rollback and serving copies
// deliberately do not end in .cnf: !includedir reads *.cnf, and neither is a
// second set of settings.
const (
mariadbParametersFile = "10-rds-parameters.cnf"
mariadbLastGoodFile = "10-rds-parameters.last-good"
mariadbServingFile = "10-rds-parameters.serving"
// MariaDB treats a setting before any group as a fatal parsing error, so
// every generated file carries the header rds-init writes.
mariadbParametersHeader = "[mysqld]\n"
)
// The layout's factory. The rules come from the control plane's own definition
// of this engine, so a build whose control plane does not offer MariaDB refuses
// to run this implementation rather than inventing a definition for it.
func newMariaDBEngineFromCatalog(cfg config, run commandRunner, startSess sessionRunner, probe *engineProbe) (engine, error) {
meta, err := handlers_rds.LookupEngine(engineMariaDB)
if err != nil {
return nil, fmt.Errorf("this image bakes %s, which this build's control plane does not offer: %w", engineMariaDB, err)
}
return newMariaDBEngine(cfg, controlPlaneRulesFrom(meta), run, startSess, probe), nil
}
func newMariaDBEngine(cfg config, rules controlPlaneRules, run commandRunner, startSess sessionRunner, probe *engineProbe) *mariadbEngine {
return &mariadbEngine{
rules: rules,
run: run,
startSess: startSess,
client: filepath.Join(cfg.EngineBinDir, mariadbClientBinary),
admin: filepath.Join(cfg.EngineBinDir, mariadbAdminBinary),
socket: mariadbSocketPath(cfg),
rcService: cfg.RCService,
service: cfg.EngineService,
probe: probe,
params: parameterStore{
// The mount point rather than the datadir one level inside it: the
// include directory has to outlive the sweep a failed bootstrap runs.
dir: filepath.Join(cfg.DataMount, "conf.d"),
installed: mariadbParametersFile,
lastGood: mariadbLastGoodFile,
serving: mariadbServingFile,
header: mariadbParametersHeader,
osUser: cfg.EngineUser,
engine: engineMariaDB,
},
repairTimeout: parameterRepairTimeout,
repairPoll: parameterRepairPoll,
}
}
func mariadbSocketPath(cfg config) string {
return filepath.Join(cfg.SocketDir, mariadbSocketFile)
}
// Nothing this implementation runs needs the assigned port: every connection it
// makes is over the engine's unix socket, and the platform drop-in rds-init
// writes is what puts the server on the port. The probe follows it separately.
func (e *mariadbEngine) setPort(int) {}
// Reports whether a pid is a live process. EPERM is a process this agent may not
// signal rather than one that is gone, which matters because a stale pidfile
// read as "alive" is far less dangerous than a live engine read as "absent".
type processLivenessFn func(pid int) bool
func processAlive(pid int) bool {
if pid <= 0 {
return false
}
err := syscall.Kill(pid, 0)
return err == nil || errors.Is(err, syscall.EPERM)
}
func newMariaDBProbe(cfg config, run probeRunner) *engineProbe {
return newEngineProbe(cfg.EnginePort, mariadbProbeState(cfg, run, processAlive))
}
// Three stages, because MariaDB has no single signal that separates an engine
// still coming up from one that is not there at all: during InnoDB crash
// recovery mariadbd opens neither its socket nor its port, so a ping fails
// exactly as it would against nothing.
//
// Losing that distinction would break the rollback guard, which treats a
// recovering engine as a reason to reset its deadline. An instance killed hard
// mid-write can spend minutes replaying its redo log; read as absent, the guard
// would roll the parameter file back and restart the server mid-recovery,
// discarding the work — a rollback that exists to break a boot loop would create
// one, on an instance whose parameters were never at fault.
//
// The port is not consulted. Both stages below reach the server over its unix
// socket, whose path does not move with the port.
func mariadbProbeState(cfg config, run probeRunner, alive processLivenessFn) probeStateFn {
pidFile, socket := cfg.EnginePidFile, mariadbSocketPath(cfg)
admin, client := filepath.Join(cfg.EngineBinDir, mariadbAdminBinary), filepath.Join(cfg.EngineBinDir, mariadbClientBinary)
connect := []string{
"--no-defaults", "--protocol=socket", "--socket=" + socket,
"--user=" + mariadbSuperuser,
fmt.Sprintf("--connect-timeout=%d", mariadbProbeConnectTimeoutSeconds),
}
probeTimeout := cfg.EngineProbeTimeout
if probeTimeout <= 0 {
probeTimeout = defaultEngineProbeTimeout
}
runBounded := func(ctx context.Context, name string, args ...string) (int, error) {
probeCtx, cancel := context.WithTimeout(ctx, probeTimeout)
defer cancel()
return run(probeCtx, name, args...)
}
return func(ctx context.Context, _ int64) (engineState, string) {
pid, err := readPidFile(pidFile)
switch {
case os.IsNotExist(err):
return engineAbsent, fmt.Sprintf("the engine has written no pidfile at %s", pidFile)
case err != nil:
return engineAbsent, fmt.Sprintf("the engine pidfile %s could not be read: %v", pidFile, err)
case !alive(pid):
return engineAbsent, fmt.Sprintf("the engine pidfile %s names pid %d, which is not running", pidFile, pid)
}
// The process is up from here on, so every remaining failure is an engine
// that is not serving yet rather than one that is gone — including a probe
// binary that will not run at all. Reporting absent against a live server
// would have the rollback guard restart one that is making progress.
switch code, err := runBounded(ctx, admin, append(slices.Clone(connect), "ping")...); {
case err != nil:
return engineRecovering, fmt.Sprintf("engine probe could not run: %v", err)
case code != 0:
return engineRecovering, "engine is not answering on its socket yet (startup or crash recovery)"
}
// ping answers successfully even on ER_ACCESS_DENIED, so on its own it
// certifies a server that may be unable to execute anything at all. The
// statement is a literal in the argv rather than on stdin because it
// carries nothing secret and a probe reads no result back.
query := append(slices.Clone(connect), "--batch", "--skip-column-names", "--execute=SELECT 1")
switch code, err := runBounded(ctx, client, query...); {
case err != nil:
return engineRecovering, fmt.Sprintf("engine probe could not run: %v", err)
case code != 0:
return engineRecovering, "engine answered its socket but would not execute a statement"
}
return engineServing, ""
}
}
func readPidFile(path string) (int, error) {
raw, err := os.ReadFile(path)
if err != nil {
return 0, err
}
pid, err := strconv.Atoi(strings.TrimSpace(string(raw)))
if err != nil {
return 0, fmt.Errorf("%s does not hold a pid: %w", path, err)
}
return pid, nil
}
func (e *mariadbEngine) Stop(ctx context.Context) error {
return serviceAction(ctx, e.run, e.rcService, e.service, "stop")
}
// Only the parameter rollback and a failed apply call this: a restart the
// control plane wants goes through RebootDBInstance, which cycles the VM.
//
// The serving copy is refreshed first, because the server is about to start on
// whatever is installed. That is exactly what rds-init does on every boot, and
// it is what keeps the two equal by construction whenever the engine starts.
func (e *mariadbEngine) Restart(ctx context.Context) error {
if err := e.recordServingCopy(); err != nil {
return err
}
return serviceAction(ctx, e.run, e.rcService, e.service, "restart")
}
// The installed set copied to the name the pending-restart comparison reads. A
// live SET GLOBAL apply deliberately does not touch it: that value was adopted
// without a restart, which is what "not pending" means.
func (e *mariadbEngine) recordServingCopy() error {
content, err := os.ReadFile(e.params.installedPath())
if err != nil {
if os.IsNotExist(err) {
return nil
}
return fmt.Errorf("read the installed parameters: %w", err)
}
return e.params.write(e.params.servingPath(), content)
}
// A value safe to interpolate into a single-quoted SQL literal. The quote is
// doubled, which holds in every sql_mode, and the backslash is doubled, which is
// why every statement built here pins a mode with backslash escapes on.
func sqlLiteral(value string) string {
return strings.ReplaceAll(strings.ReplaceAll(value, `\`, `\\`), `'`, `''`)
}
// Turns off the two logs that would otherwise copy a statement's text, and pins
// the mode the escaping above assumes. All three are session-scoped: a rotation
// must not leave the customer's own general log switched off behind it.
const mariadbSessionGuard = `SET SESSION sql_log_off = 1;
SET SESSION slow_query_log = 0;
SET SESSION sql_mode = 'NO_ENGINE_SUBSTITUTION';
`
// Rotates the master's password. The statement is built here rather than by the
// client, which has neither psql's identifier interpolation nor its parameter
// quoting, and it rides stdin so it is never visible in the process table.
func (e *mariadbEngine) SetPassword(ctx context.Context, username, password string) error {
if username == "" || password == "" {
return fmt.Errorf("set-password requires both %s and %s",
handlers_rds.CommandParamMasterUsername, handlers_rds.CommandParamMasterUserPassword)
}
// This runs as the server's own superuser over the socket, so a reserved name
// in the command payload would rotate the platform's account rather than the
// customer's. The rest of the rule matters just as much here, because the name
// is interpolated into the statement below rather than quoted by the client.
if err := e.rules.validateUsername(username); err != nil {
return fmt.Errorf("refusing to set the password of role %q: %w", username, err)
}
escaped := sqlLiteral(password)
sql := fmt.Sprintf("%sALTER USER '%s'@'%s' IDENTIFIED BY '%s';\n",
mariadbSessionGuard, username, mariadbMasterHost, escaped)
if _, err := e.clientRun(ctx, sql); err != nil {
// The client echoes the failing statement, which carries the escaped form
// of the password rather than the raw one — so redacting only the raw one
// would leak exactly the passwords that needed escaping.
return fmt.Errorf("apply the master password: %s", redact(redact(err.Error(), password), escaped))
}
return nil
}
// Installs the resolved set and applies the half of it a running server can
// adopt. MariaDB re-reads no configuration file while it is up, so SET GLOBAL is
// the only way an immediate apply reaches it; the static half waits in the
// installed file for the next start, and is what comes back as pending.
//
// There is no offline parser to check the set with first — MariaDB ships no
// equivalent of postgres -C — so the safety net for a static value the server
// will not start on is the boot-time rollback to the last accepted set.
func (e *mariadbEngine) ApplyParameters(ctx context.Context, params []handlers_rds.Parameter) ([]string, error) {
e.paramMu.Lock()
defer e.paramMu.Unlock()
initialState, _ := e.probe.state(ctx)
if initialState == engineRecovering {
return nil, errors.New("apply parameters while the engine is still starting or recovering")
}
if initialState == engineServing {
// A command can arrive before the first heartbeat. Seed the configuration
// currently being served before replacing its file in that window.
if err := e.recordServingParametersLocked(ctx); err != nil {
return nil, fmt.Errorf("record the parameters serving before the apply: %w", err)
}
}
restore, err := e.params.install(params)
if err != nil {
return nil, err
}
state, _ := e.probe.state(ctx)
switch state {
case engineAbsent:
return e.restartOnRepairSetLocked(ctx)
case engineRecovering:
return nil, errors.Join(errors.New("the engine entered startup or recovery during the parameter apply"), restore())
}
if err := e.applyDynamicParameters(ctx, params); err != nil {
// The engine may have gone down after the first probe. In that case start it
// on the newly installed set rather than restoring one that already left it
// unable to serve.
if state, _ := e.probe.state(ctx); state == engineAbsent {
return e.restartOnRepairSetLocked(ctx)
}
return nil, errors.Join(err, restore())
}
return e.pendingRestartParameters(ctx)
}
// The dynamic half of the set, in one invocation that stops at the first refusal
// so the server's own message names the setting it would not take.
//
// Only names the catalog carries are emitted, which is also what keeps the
// statement safe to build: a name is an identifier here, and the classification
// that selects it comes from the same table the API validated the request
// against. MariaDB cannot set several globals atomically, so a value set before
// a refusal stays live until the next restart, when the restored file wins.
func (e *mariadbEngine) applyDynamicParameters(ctx context.Context, params []handlers_rds.Parameter) error {
var b strings.Builder
b.WriteString("SET SESSION sql_mode = 'NO_ENGINE_SUBSTITUTION';\n")
applied := 0
for _, p := range params {
if p.Name == "" || e.rules.isStatic(p.Name) {
continue
}
fmt.Fprintf(&b, "SET GLOBAL %s = '%s';\n", p.Name, sqlLiteral(p.Value))
applied++
}
if applied == 0 {
return nil
}
if _, err := e.clientRun(ctx, b.String()); err != nil {
return fmt.Errorf("apply the dynamic parameters: %w", err)
}
return nil
}
func (e *mariadbEngine) restartOnRepairSetLocked(ctx context.Context) ([]string, error) {
return awaitRepairedEngine(ctx, e.probe, e.Restart, e.pendingRestartParameters, e.repairTimeout, e.repairPoll)
}
// MariaDB has no pending_restart. Nothing in information_schema reports a
// setting the server has stored but not adopted, so the answer is computed from
// the files instead: the catalog-static keys whose value in the installed
// drop-in differs from the one the server actually started on.
//
// Comparing live values against the desired set was rejected. mysqld silently
// rewrites values at startup — rounding the buffer pool and log file sizes, and
// downgrading max_connections when it cannot obtain enough file descriptors —
// and each rewrite would read as a permanent pending restart: un-clearable
// pending-reboot state, permanent drift, and a last-known-good copy frozen for
// good, silently disabling a recovery mechanism only exercised during an
// incident.
func (e *mariadbEngine) pendingRestartParameters(context.Context) ([]string, error) {
installed, err := readOptionFile(e.params.installedPath())
if err != nil {
if os.IsNotExist(err) {
// Nothing is installed, so nothing is waiting to be adopted.
return nil, nil
}
return nil, fmt.Errorf("read the installed parameters: %w", err)
}
serving, err := readOptionFile(e.params.servingPath())
if err != nil && !os.IsNotExist(err) {
return nil, fmt.Errorf("read the parameters the engine started on: %w", err)
}
// Back to the catalog's names before anything is classified or reported: the
// files are written in the engine's startup spellings, and the answer is a
// customer-facing list the API refuses ApplyMethod=immediate against.
installed = e.catalogKeyed(installed)
serving = e.catalogKeyed(serving)
// An absent serving copy is a set the engine has not started on: rds-init
// writes the two together on every boot, so the whole static half counts as
// pending rather than being promoted on the strength of a missing file.
var pending []string
for name, value := range installed {
if !e.rules.isStatic(name) {
continue
}
if served, ok := serving[name]; !ok || served != value {
pending = append(pending, name)
}
}
// A static setting the group stopped naming reverts to its default at the next
// start, which the running server has not adopted either.
for name := range serving {
if _, ok := installed[name]; !ok && e.rules.isStatic(name) {
pending = append(pending, name)
}
}
slices.Sort(pending)
return pending, nil
}
// Snapshots the installed set as the rollback target, but only when nothing in
// it is still waiting for a restart. The parameter mutex keeps an apply from
// replacing the file between that check and the copy.
func (e *mariadbEngine) RecordServingParameters(ctx context.Context) error {
e.paramMu.Lock()
defer e.paramMu.Unlock()
return e.recordServingParametersLocked(ctx)
}
func (e *mariadbEngine) recordServingParametersLocked(ctx context.Context) error {
return recordLastGood(ctx, e.params, e.pendingRestartParameters)
}
// Puts the last set the engine accepted back in place, for a start that failed
// after a parameter change.
func (e *mariadbEngine) RestoreLastKnownGoodParameters(ctx context.Context) (bool, error) {
e.paramMu.Lock()
defer e.paramMu.Unlock()
return restoreLastGood(ctx, e.params, e.probe)
}
// The same settings under the names the catalog and the API know them by, since
// a file is written in the engine's startup spellings and those are not always
// the customer's.
func (e *mariadbEngine) catalogKeyed(values map[string]string) map[string]string {
out := make(map[string]string, len(values))
for name, value := range values {
out[e.rules.catalogName(name)] = value
}
return out
}
// The subset of MariaDB's option-file syntax the generated drop-ins are written
// in: a group header, comments, and one `name = value` per line. Nothing else
// reaches these two files, because only rds-init and this agent write them.
func readOptionFile(path string) (map[string]string, error) {
raw, err := os.ReadFile(path)
if err != nil {
return nil, err
}
values := make(map[string]string)
for line := range strings.SplitSeq(string(raw), "\n") {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "#") || strings.HasPrefix(line, ";") || strings.HasPrefix(line, "[") {
continue
}
// A bare option such as skip-log-bin carries no value; it is still a setting
// whose presence or absence differs.
name, value, _ := strings.Cut(line, "=")
values[normaliseOptionName(name)] = strings.Trim(strings.TrimSpace(value), `"'`)
}
return values, nil
}
// MariaDB reads - and _ as the same character in an option name, so the two
// spellings of one setting must not compare as two settings.
func normaliseOptionName(name string) string {
return strings.ReplaceAll(strings.TrimSpace(name), "-", "_")
}
// How long any one BACKUP STAGE waits for the metadata locks it needs. Well
// under the control plane's own quiesce timeout, so a stage that cannot take its
// lock fails and is reported rather than being abandoned mid-wait with the
// request still queued in front of live traffic.
const mariadbQuiesceLockWait = 20 * time.Second
// MariaDB's own backup API rather than FLUSH TABLES WITH READ LOCK. FTWRL turns
// a snapshot into a write outage: the whole database is read-only until it is
// released, so a control plane that died after issuing quiesce would leave the
// customer read-only for the full hold, on a schedule and unattended. Its
// acquisition phase is unbounded too, with writes queueing behind a pending lock
// request. BACKUP STAGE blocks commits rather than all writes and its flush
// stage covers Aria and MyISAM, so it yields a better consistency point while
// being far less disruptive — and it is what mariadb-backup itself uses.
func (e *mariadbEngine) Quiesce(ctx context.Context, label string, hold time.Duration) error {
if err := validateQuiesceRequest(label, hold); err != nil {
return err
}
// Held across the whole start, so a second quiesce waits and then finds the
// first one's hold rather than opening a concurrent backup alongside it.
e.mu.Lock()
defer e.mu.Unlock()
if e.held != nil {
return fmt.Errorf("the engine is already quiesced for backup %s", e.held.label)
}
// Deliberately not the command's context: the session has to outlive the call
// that started it, and its own deadline is what bounds it instead.
session, err := e.startSess(context.WithoutCancel(ctx), command{
Name: e.client,
Args: e.clientArgs(),
Env: []string{"PATH=" + defaultGuestPath},
SentinelStatement: "SELECT '" + sessionSentinel + "';\n",
})
if err != nil {
return fmt.Errorf("open a backup session: %w", err)
}
// The stages run in order and on one connection: MariaDB releases the whole
// hold with the session that took it, which is what bounds a control plane
// that dies mid-snapshot. Fed one at a time so a stage that cannot take its
// locks is named rather than reported as a line number.
stages := []string{
fmt.Sprintf("SET SESSION lock_wait_timeout = %d;\n", int(mariadbQuiesceLockWait.Seconds())),
"BACKUP STAGE START;\n",
"BACKUP STAGE FLUSH;\n",
"BACKUP STAGE BLOCK_DDL;\n",
"BACKUP STAGE BLOCK_COMMIT;\n",
}
for _, sql := range stages {
if err := session.Exec(ctx, sql); err != nil {
if closeErr := session.Close(); closeErr != nil {
slog.Warn("rds-agent: closing a failed backup session", "err", closeErr)
}
return fmt.Errorf("put the engine into backup mode at %q: %w", strings.TrimSpace(sql), err)
}
}
e.beginHoldLocked(label, session, hold)
return nil
}
// Ends the backup cleanly. A missing hold is an error rather than a silent
// success: it means the deadline fired first, so the snapshot the control plane
// just took was not taken against a held stage.
func (e *mariadbEngine) Unquiesce(ctx context.Context) error {
held := e.takeHold()
if held == nil {
return errors.New("the engine is not quiesced; the backup hold had already expired")
}
// The release has to run on the session that took the stages, and the session
// is closed either way — the engine ends an unreleased backup with it.
execErr := held.session.Exec(ctx, "BACKUP STAGE END;\n")
closeErr := held.session.Close()
if execErr != nil {
return fmt.Errorf("take the engine out of backup mode: %w", errors.Join(execErr, closeErr))
}
if closeErr != nil {
return fmt.Errorf("close the backup session: %w", closeErr)
}
slog.Info("rds-agent: engine released from backup mode", "label", held.label)
return nil
}
// Feeds sql on stdin rather than as an argument, so a statement is never visible
// in the process table.
func (e *mariadbEngine) clientRun(ctx context.Context, sql string) (string, error) {
return e.run(ctx, command{
Name: e.client,
Args: e.clientArgs(),
Env: []string{"PATH=" + defaultGuestPath},
Stdin: sql,
})
}
// --no-defaults so no option file can move the connection this agent makes, and
// --unbuffered so a held session's output reaches the reader as each statement
// finishes rather than when a pipe buffer fills. No User: the agent is root, and
// root is what the datadir's unix_socket plugin authenticates.
func (e *mariadbEngine) clientArgs() []string {
return []string{
"--no-defaults", "--batch", "--skip-column-names", "--unbuffered",
"--protocol=socket", "--socket=" + e.socket, "--user=" + mariadbSuperuser,
}
}
+796
View File
@@ -0,0 +1,796 @@
package main
import (
"context"
"errors"
"fmt"
"os"
"os/user"
"path/filepath"
"slices"
"strconv"
"strings"
"testing"
"time"
handlers_rds "github.com/mulgadc/spinifex/spinifex/handlers/rds"
)
// Stands in for the control plane's own definition of the engine. It mirrors the
// production wiring exactly, including that a name the catalog does not carry
// counts as static: a rule the fake inverted would hide the failure it exists to
// prevent.
func testMariaDBRules() controlPlaneRules {
applyTypes := map[string]string{
"innodb_buffer_pool_size": handlers_rds.ApplyTypeStatic,
"innodb_log_file_size": handlers_rds.ApplyTypeStatic,
"max_connections": handlers_rds.ApplyTypeDynamic,
"long_query_time": handlers_rds.ApplyTypeDynamic,
"time_zone": handlers_rds.ApplyTypeDynamic,
}
reserved := []string{"root", "mysql", "mariadb.sys", "rdsadmin", "public"}
return controlPlaneRules{
validateUsername: func(username string) error {
lower := strings.ToLower(username)
if slices.Contains(reserved, lower) || strings.HasPrefix(lower, "mysql.") || strings.HasPrefix(lower, "mariadb.") {
return fmt.Errorf("MasterUsername %q is reserved by mariadb", username)
}
for _, r := range username {
if !isTestIdentifierRune(r) {
return fmt.Errorf("MasterUsername may contain only letters, digits and underscores")
}
}
return nil
},
isStatic: func(name string) bool {
applyType, ok := applyTypes[name]
return !ok || applyType == handlers_rds.ApplyTypeStatic
},
catalogName: func(optionFileName string) string {
if optionFileName == "default_time_zone" {
return "time_zone"
}
return optionFileName
},
}
}
func isTestIdentifierRune(r rune) bool {
return (r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z') || (r >= '0' && r <= '9') || r == '_'
}
func newTestMariaDBEngine(t *testing.T, run commandRunner) *mariadbEngine {
t.Helper()
cfg := testLoadConfig(t, engineMariaDB)
cfg.DataMount = t.TempDir()
if err := os.MkdirAll(filepath.Join(cfg.DataMount, "conf.d"), 0o750); err != nil {
t.Fatalf("create conf.d: %v", err)
}
// The guest's mysql user does not exist here, and the chown of the installed
// parameter file has to resolve to something real.
current, err := user.Current()
if err != nil {
t.Fatalf("resolve the current user: %v", err)
}
cfg.EngineUser = current.Username
state := engineServing
return newMariaDBEngine(cfg, testMariaDBRules(), run, nil, drivenProbe(&state))
}
// The wiring the guest takes its classification from, checked against a catalog
// that is registered: what a spec says about a parameter is what the guest must
// see, and an unlisted name must read as static rather than as dynamic.
func TestControlPlaneRulesFrom_TakesTheClassificationFromTheCatalog(t *testing.T) {
rules := controlPlaneRulesFrom(postgresEngineMeta)
for name, want := range map[string]bool{"shared_buffers": true, "work_mem": false} {
spec, ok := postgresEngineMeta.LookupParameter(name)
if !ok {
t.Fatalf("the catalog no longer carries %s", name)
}
if (spec.ApplyType == handlers_rds.ApplyTypeStatic) != want {
t.Fatalf("%s is %s in the catalog, so this test is asserting the wrong way round", name, spec.ApplyType)
}
if got := rules.isStatic(name); got != want {
t.Errorf("isStatic(%s) = %v, want %v", name, got, want)
}
}
if !rules.isStatic("not_a_parameter_any_catalog_carries") {
t.Error("a name the catalog does not carry read as dynamic, so it would be issued as a live SET GLOBAL")
}
if rules.validateUsername("postgres") == nil {
t.Error("the username rule does not refuse a reserved role")
}
}
// The client has no parameter quoting of its own, so the password is built into
// the statement here. It must reach the process on stdin and nowhere else.
func TestMariaDBEngine_SetPasswordKeepsTheSecretOutOfArgvAndEnv(t *testing.T) {
runner := &recordingRunner{}
engine := newTestMariaDBEngine(t, runner.run)
if err := engine.SetPassword(context.Background(), "mulgamaster", "n3w-pw"); err != nil {
t.Fatalf("SetPassword: %v", err)
}
if len(runner.calls) != 1 {
t.Fatalf("ran %d commands, want 1", len(runner.calls))
}
call := runner.calls[0]
for _, arg := range call.Args {
if strings.Contains(arg, "n3w-pw") {
t.Fatalf("argv %v carries the password", call.Args)
}
}
for _, env := range call.Env {
if strings.Contains(env, "n3w-pw") {
t.Fatalf("env %v carries the password, which the client would not read anyway", call.Env)
}
}
if !strings.Contains(call.Stdin, "ALTER USER 'mulgamaster'@'%' IDENTIFIED BY 'n3w-pw';") {
t.Errorf("SQL %q does not rotate the master on the wildcard host", call.Stdin)
}
}
// ValidateMasterUserPassword permits both of these, and the client offers no
// equivalent of psql's parameter quoting, so the escaping here is what keeps a
// password from ending the literal it is inside.
func TestMariaDBEngine_SetPasswordEscapesQuotesAndBackslashes(t *testing.T) {
runner := &recordingRunner{}
engine := newTestMariaDBEngine(t, runner.run)
if err := engine.SetPassword(context.Background(), "mulgamaster", `a'b\c`); err != nil {
t.Fatalf("SetPassword: %v", err)
}
sql := runner.calls[0].Stdin
if !strings.Contains(sql, `IDENTIFIED BY 'a''b\\c';`) {
t.Errorf("SQL %q does not escape both the quote and the backslash", sql)
}
// The escaping assumes a mode where backslash escapes are on, so the statement
// cannot inherit whatever sql_mode the customer's parameter group set.
if !strings.Contains(sql, "SET SESSION sql_mode = 'NO_ENGINE_SUBSTITUTION';") {
t.Errorf("SQL %q does not pin the mode its escaping assumes", sql)
}
}
// The password is interpolated into the statement, so a parameter group that
// turned the general or slow log on would copy it into a log file every snapshot
// then carries. Both are silenced for this session only: a rotation must not
// leave the customer's own logging switched off behind it.
func TestMariaDBEngine_SetPasswordSilencesStatementLoggingForItsSessionOnly(t *testing.T) {
runner := &recordingRunner{}
engine := newTestMariaDBEngine(t, runner.run)
if err := engine.SetPassword(context.Background(), "mulgamaster", "n3w-pw"); err != nil {
t.Fatalf("SetPassword: %v", err)
}
sql := runner.calls[0].Stdin
alter := strings.Index(sql, "ALTER USER")
if alter < 0 {
t.Fatalf("SQL %q does not alter the user", sql)
}
for _, guard := range []string{"SET SESSION sql_log_off = 1;", "SET SESSION slow_query_log = 0;"} {
at := strings.Index(sql, guard)
if at < 0 {
t.Errorf("SQL %q is missing %q", sql, guard)
continue
}
if at > alter {
t.Errorf("SQL %q sets %q only after the statement has already been logged", sql, guard)
}
}
if strings.Contains(sql, "SET GLOBAL general_log") || strings.Contains(sql, "SET GLOBAL slow_query_log") {
t.Errorf("SQL %q turns logging off globally, which outlives the rotation", sql)
}
}
// The client echoes the statement it failed on, which carries the escaped form
// of the password rather than the raw one — so redacting only the raw one would
// leak exactly the passwords that needed escaping.
func TestMariaDBEngine_SetPasswordRedactsBothFormsOfTheSecret(t *testing.T) {
secret := `a'b\c`
runner := &recordingRunner{err: errors.New(`ERROR 1064 near "IDENTIFIED BY 'a''b\\c'"`)}
engine := newTestMariaDBEngine(t, runner.run)
err := engine.SetPassword(context.Background(), "mulgamaster", secret)
if err == nil {
t.Fatal("SetPassword succeeded against a failing client")
}
if strings.Contains(err.Error(), secret) || strings.Contains(err.Error(), sqlLiteral(secret)) {
t.Errorf("error %q leaks the password", err)
}
if !strings.Contains(err.Error(), "[REDACTED]") {
t.Errorf("error %q does not mark the redaction", err)
}
}
// The apply runs as the server's own superuser over the socket, and the name is
// interpolated into the statement rather than quoted by the client, so both
// halves of the rule are refused before any SQL is built.
func TestMariaDBEngine_SetPasswordRefusesNamesTheEngineWillNotTake(t *testing.T) {
for _, username := range []string{"root", "mysql", "mariadb.sys", "rdsadmin", "PUBLIC", "mysql.infoschema", `ma'ster`} {
t.Run(username, func(t *testing.T) {
runner := &recordingRunner{}
engine := newTestMariaDBEngine(t, runner.run)
if err := engine.SetPassword(context.Background(), username, "n3w-pw"); err == nil {
t.Fatalf("SetPassword accepted %q", username)
}
if len(runner.calls) != 0 {
t.Errorf("ran %d commands for a refused name, want 0", len(runner.calls))
}
})
}
}
func TestMariaDBEngine_SetPasswordRejectsAnIncompleteCommand(t *testing.T) {
runner := &recordingRunner{}
engine := newTestMariaDBEngine(t, runner.run)
if err := engine.SetPassword(context.Background(), "mulgamaster", ""); err == nil {
t.Error("SetPassword accepted an empty password")
}
if len(runner.calls) != 0 {
t.Errorf("ran %d commands for an incomplete request, want 0", len(runner.calls))
}
}
// A running server re-reads no configuration file, so the dynamic half has to
// reach it as SET GLOBAL. The static half is installed and waits for a restart,
// which is what comes back as pending.
func TestMariaDBEngine_ApplyParametersSetsOnlyTheDynamicHalfLive(t *testing.T) {
runner := &recordingRunner{}
engine := newTestMariaDBEngine(t, runner.run)
pending, err := engine.ApplyParameters(context.Background(), []handlers_rds.Parameter{
{Name: "max_connections", Value: "200"},
{Name: "innodb_buffer_pool_size", Value: "1073741824"},
})
if err != nil {
t.Fatalf("ApplyParameters: %v", err)
}
installed, err := os.ReadFile(engine.params.installedPath())
if err != nil {
t.Fatalf("read the installed parameters: %v", err)
}
if !strings.HasPrefix(string(installed), "[mysqld]\n") {
t.Errorf("installed parameters = %q, want the group header rds-init writes", installed)
}
for _, want := range []string{"max_connections = '200'", "innodb_buffer_pool_size = '1073741824'"} {
if !strings.Contains(string(installed), want) {
t.Errorf("installed parameters = %q, want %q", installed, want)
}
}
if len(runner.calls) != 1 {
t.Fatalf("ran %d commands, want the one live apply", len(runner.calls))
}
sql := runner.calls[0].Stdin
if !strings.Contains(sql, "SET GLOBAL max_connections = '200';") {
t.Errorf("SQL %q does not set the dynamic value live", sql)
}
if strings.Contains(sql, "innodb_buffer_pool_size") {
t.Errorf("SQL %q sets a static value live, which the server would refuse", sql)
}
if !slices.Equal(pending, []string{"innodb_buffer_pool_size"}) {
t.Errorf("pending = %v, want the static setting the engine has not adopted", pending)
}
}
// A set with nothing dynamic in it must not open a connection at all: the
// installed file is the whole of the change.
func TestMariaDBEngine_ApplyParametersRunsNoStatementForAStaticOnlySet(t *testing.T) {
runner := &recordingRunner{}
engine := newTestMariaDBEngine(t, runner.run)
if _, err := engine.ApplyParameters(context.Background(),
[]handlers_rds.Parameter{{Name: "innodb_log_file_size", Value: "268435456"}}); err != nil {
t.Fatalf("ApplyParameters: %v", err)
}
if len(runner.calls) != 0 {
t.Errorf("ran %d commands for a static-only set, want 0", len(runner.calls))
}
}
// A value the server refuses leaves the installed file back where it was, so it
// does not sit on the data volume turning the next restart into a boot loop.
func TestMariaDBEngine_ApplyParametersRollsBackAValueTheServerRefuses(t *testing.T) {
runner := &recordingRunner{}
engine := newTestMariaDBEngine(t, runner.run)
if _, err := engine.ApplyParameters(context.Background(),
[]handlers_rds.Parameter{{Name: "max_connections", Value: "200"}}); err != nil {
t.Fatalf("first ApplyParameters: %v", err)
}
runner.err = errors.New("ERROR 1231 (42000): Variable 'max_connections' can't be set to the value of '0'")
if _, err := engine.ApplyParameters(context.Background(),
[]handlers_rds.Parameter{{Name: "max_connections", Value: "0"}}); err == nil {
t.Fatal("ApplyParameters succeeded against a value the server refused")
}
installed, err := os.ReadFile(engine.params.installedPath())
if err != nil {
t.Fatalf("read the installed parameters: %v", err)
}
if !strings.Contains(string(installed), "max_connections = '200'") {
t.Errorf("installed parameters = %q, want the previous accepted set restored", installed)
}
}
// The whole of the pending-restart answer, since MariaDB has none of its own:
// the installed drop-in against the one the server actually started on, over the
// static keys only.
func TestMariaDBEngine_PendingRestartComparesTheInstalledSetAgainstTheServingCopy(t *testing.T) {
engine := newTestMariaDBEngine(t, (&recordingRunner{}).run)
write := func(path, body string) {
t.Helper()
if err := os.WriteFile(path, []byte("[mysqld]\n"+body), 0o600); err != nil {
t.Fatalf("write %s: %v", path, err)
}
}
pending := func() []string {
t.Helper()
names, err := engine.pendingRestartParameters(t.Context())
if err != nil {
t.Fatalf("pendingRestartParameters: %v", err)
}
return names
}
// A fresh boot: rds-init wrote the two together, so nothing is waiting.
write(engine.params.installedPath(), "innodb_buffer_pool_size = '1073741824'\nmax_connections = '100'\n")
write(engine.params.servingPath(), "innodb_buffer_pool_size = '1073741824'\nmax_connections = '100'\n")
if names := pending(); len(names) != 0 {
t.Errorf("pending = %v, want nothing on a set the engine started on", names)
}
// A dynamic value the server took live is not pending: it was adopted without
// a restart, which is exactly what the serving copy not moving records.
write(engine.params.installedPath(), "innodb_buffer_pool_size = '1073741824'\nmax_connections = '200'\n")
if names := pending(); len(names) != 0 {
t.Errorf("pending = %v, want nothing for a value adopted live", names)
}
// A static value is.
write(engine.params.installedPath(), "innodb_buffer_pool_size = '2147483648'\nmax_connections = '200'\n")
if names := pending(); !slices.Equal(names, []string{"innodb_buffer_pool_size"}) {
t.Errorf("pending = %v, want the static setting", names)
}
// So is a static setting the group stopped naming, which reverts to its
// default at the next start.
write(engine.params.installedPath(), "max_connections = '200'\n")
if names := pending(); !slices.Equal(names, []string{"innodb_buffer_pool_size"}) {
t.Errorf("pending = %v, want the withdrawn static setting", names)
}
// MariaDB reads - and _ as the same character, so one setting spelled two ways
// is not two settings.
write(engine.params.installedPath(), "innodb-buffer-pool-size = '1073741824'\nmax_connections = '200'\n")
if names := pending(); len(names) != 0 {
t.Errorf("pending = %v, want the two spellings of one setting to compare equal", names)
}
}
// rds-init writes the serving copy beside the installed one on every boot, so a
// set with no copy beside it is one the engine has not started on. Promoting it
// would point the rollback at a configuration that has never served.
func TestMariaDBEngine_PendingRestartTreatsAMissingServingCopyAsUnadopted(t *testing.T) {
engine := newTestMariaDBEngine(t, (&recordingRunner{}).run)
if err := os.WriteFile(engine.params.installedPath(),
[]byte("[mysqld]\ninnodb_buffer_pool_size = '1073741824'\nmax_connections = '100'\n"), 0o600); err != nil {
t.Fatalf("write installed parameters: %v", err)
}
names, err := engine.pendingRestartParameters(t.Context())
if err != nil {
t.Fatalf("pendingRestartParameters: %v", err)
}
if !slices.Equal(names, []string{"innodb_buffer_pool_size"}) {
t.Errorf("pending = %v, want the static half of a set the engine never started on", names)
}
}
// The restart the agent performs is the only thing that moves the serving copy,
// which is what makes a fresh start clear whatever was pending.
func TestMariaDBEngine_RestartRefreshesTheServingCopy(t *testing.T) {
runner := &recordingRunner{}
engine := newTestMariaDBEngine(t, runner.run)
installed := []byte("[mysqld]\ninnodb_buffer_pool_size = '2147483648'\n")
if err := os.WriteFile(engine.params.installedPath(), installed, 0o600); err != nil {
t.Fatalf("write installed parameters: %v", err)
}
if err := os.WriteFile(engine.params.servingPath(), []byte("[mysqld]\ninnodb_buffer_pool_size = '1073741824'\n"), 0o600); err != nil {
t.Fatalf("write serving parameters: %v", err)
}
if err := engine.Restart(context.Background()); err != nil {
t.Fatalf("Restart: %v", err)
}
serving, err := os.ReadFile(engine.params.servingPath())
if err != nil {
t.Fatalf("read the serving copy: %v", err)
}
if string(serving) != string(installed) {
t.Errorf("serving copy = %q, want the set the engine was just started on", serving)
}
names, err := engine.pendingRestartParameters(t.Context())
if err != nil {
t.Fatalf("pendingRestartParameters: %v", err)
}
if len(names) != 0 {
t.Errorf("pending = %v, want a restart to have cleared it", names)
}
// The copy the comparison reads must never be parsed as a second set of
// settings by the engine's own include glob.
if strings.HasSuffix(engine.params.servingPath(), ".cnf") {
t.Errorf("the serving copy at %s would be included by the engine", engine.params.servingPath())
}
}
// The rollback target advances only once a restart has proved the installed
// static values can serve, not when a dynamic apply merely set them live.
func TestMariaDBEngine_LastKnownGoodWaitsForARestartToAdoptAStaticSet(t *testing.T) {
runner := &recordingRunner{}
engine := newTestMariaDBEngine(t, runner.run)
if _, err := engine.ApplyParameters(t.Context(),
[]handlers_rds.Parameter{{Name: "innodb_buffer_pool_size", Value: "1073741824"}}); err != nil {
t.Fatalf("ApplyParameters: %v", err)
}
if err := engine.RecordServingParameters(t.Context()); err != nil {
t.Fatalf("RecordServingParameters: %v", err)
}
if _, err := os.Stat(engine.params.lastGoodPath()); !os.IsNotExist(err) {
t.Errorf("a set the engine has not restarted onto became the rollback target (stat err = %v)", err)
}
if err := engine.Restart(t.Context()); err != nil {
t.Fatalf("Restart: %v", err)
}
if err := engine.RecordServingParameters(t.Context()); err != nil {
t.Fatalf("RecordServingParameters after the restart: %v", err)
}
lastGood, err := os.ReadFile(engine.params.lastGoodPath())
if err != nil {
t.Fatalf("read the last known good parameters: %v", err)
}
if !strings.Contains(string(lastGood), "innodb_buffer_pool_size = '1073741824'") {
t.Errorf("last known good = %q, want the set the engine restarted onto", lastGood)
}
}
// A dynamic apply is adopted without a restart, so it may advance the rollback
// target on the next healthy beat rather than waiting for one.
func TestMariaDBEngine_LastKnownGoodAdvancesOnADynamicApply(t *testing.T) {
runner := &recordingRunner{}
engine := newTestMariaDBEngine(t, runner.run)
if err := os.WriteFile(engine.params.servingPath(), []byte("[mysqld]\n"), 0o600); err != nil {
t.Fatalf("write serving parameters: %v", err)
}
if _, err := engine.ApplyParameters(t.Context(),
[]handlers_rds.Parameter{{Name: "max_connections", Value: "200"}}); err != nil {
t.Fatalf("ApplyParameters: %v", err)
}
if err := engine.RecordServingParameters(t.Context()); err != nil {
t.Fatalf("RecordServingParameters: %v", err)
}
lastGood, err := os.ReadFile(engine.params.lastGoodPath())
if err != nil {
t.Fatalf("read the last known good parameters: %v", err)
}
if !strings.Contains(string(lastGood), "max_connections = '200'") {
t.Errorf("last known good = %q, want the set the engine is already running", lastGood)
}
}
// A probe that reports what the test hands it, in place of whichever binaries
// the engine's own state function would have run.
func mariadbTestProbeConfig(t *testing.T) config {
t.Helper()
cfg := testLoadConfig(t, engineMariaDB)
cfg.EnginePidFile = filepath.Join(t.TempDir(), "mariadb.pid")
return cfg
}
func writePid(t *testing.T, path string, pid int) {
t.Helper()
if err := os.WriteFile(path, []byte(strconv.Itoa(pid)+"\n"), 0o644); err != nil {
t.Fatalf("write pidfile: %v", err)
}
}
// The three stages, and what each one separates. During InnoDB crash recovery
// mariadbd opens neither its socket nor its port, so nothing the client can ask
// distinguishes a server replaying its redo log from no server at all.
func TestMariaDBProbe_SeparatesRecoveringFromAbsent(t *testing.T) {
tests := []struct {
name string
writePid bool
alive bool
pingCode int
pingErr error
execCode int
want engineState
}{
{name: "no pidfile at all", want: engineAbsent},
{name: "pidfile names a dead process", writePid: true, want: engineAbsent},
{name: "process alive but socket silent", writePid: true, alive: true, pingCode: 1, want: engineRecovering},
{name: "process alive but probe cannot run", writePid: true, alive: true, pingErr: errors.New("no such file"), want: engineRecovering},
{name: "answers ping but executes nothing", writePid: true, alive: true, execCode: 1, want: engineRecovering},
{name: "serving", writePid: true, alive: true, want: engineServing},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
cfg := mariadbTestProbeConfig(t)
if tt.writePid {
writePid(t, cfg.EnginePidFile, 4242)
}
run := func(_ context.Context, name string, args ...string) (int, error) {
if strings.HasSuffix(name, mariadbAdminBinary) {
return tt.pingCode, tt.pingErr
}
if !slices.Contains(args, "--execute=SELECT 1") {
t.Errorf("the second stage ran %s %v, want a statement the server has to execute", name, args)
}
return tt.execCode, nil
}
stateFn := mariadbProbeState(cfg, run, func(int) bool { return tt.alive })
got, message := stateFn(context.Background(), int64(cfg.EnginePort))
if got != tt.want {
t.Errorf("state = %v, want %v (%s)", got, tt.want, message)
}
if got != engineServing && message == "" {
t.Error("a non-serving result carried no message explaining it")
}
})
}
}
// The failure the guard exists for still behaves correctly: a static value
// mariadbd will not accept makes it exit during startup, so the pidfile names a
// process that is gone and the deadline runs.
func TestMariaDBProbe_BoundsAStalledClient(t *testing.T) {
cfg := mariadbTestProbeConfig(t)
cfg.EngineProbeTimeout = 20 * time.Millisecond
writePid(t, cfg.EnginePidFile, 4242)
started := time.Now()
state, message := mariadbProbeState(cfg,
func(ctx context.Context, _ string, args ...string) (int, error) {
if !slices.Contains(args, "--connect-timeout=3") {
t.Errorf("probe args = %v, want a client connection timeout", args)
}
<-ctx.Done()
return -1, ctx.Err()
},
func(int) bool { return true })(t.Context(), int64(cfg.EnginePort))
if state != engineRecovering {
t.Errorf("state = %v, want recovering after a live engine's probe timed out", state)
}
if !strings.Contains(message, context.DeadlineExceeded.Error()) {
t.Errorf("message = %q, want the probe deadline", message)
}
if elapsed := time.Since(started); elapsed > time.Second {
t.Errorf("probe took %v, want the configured deadline to bound it", elapsed)
}
}
func TestMariaDBProbe_MapsToTheHealthTheControlPlaneReads(t *testing.T) {
cfg := mariadbTestProbeConfig(t)
writePid(t, cfg.EnginePidFile, 4242)
alive := true
serving := true
probe := newEngineProbe(cfg.EnginePort, mariadbProbeState(cfg,
func(context.Context, string, ...string) (int, error) {
if serving {
return 0, nil
}
return 1, nil
},
func(int) bool { return alive }))
if got, _ := probe.Check(context.Background()); got != handlers_rds.EngineHealthHealthy {
t.Fatalf("health = %q, want healthy", got)
}
serving = false
if got, _ := probe.Check(context.Background()); got != handlers_rds.EngineHealthStarting {
t.Errorf("health while recovering = %q, want starting", got)
}
alive = false
if got, _ := probe.Check(context.Background()); got != handlers_rds.EngineHealthUnhealthy {
t.Errorf("health after the engine went away = %q, want unhealthy", got)
}
}
// An instance killed hard mid-write can spend minutes replaying its redo log. If
// that read as absent, the guard would roll the parameter file back and restart
// the server mid-recovery — a rollback that exists to break a boot loop would
// create one, on an instance whose parameters were never at fault.
func TestMariaDBProbe_LiveButUnreachableEngineResetsTheRollbackDeadline(t *testing.T) {
cfg := mariadbTestProbeConfig(t)
writePid(t, cfg.EnginePidFile, 4242)
probe := newEngineProbe(cfg.EnginePort, mariadbProbeState(cfg,
func(context.Context, string, ...string) (int, error) { return 1, nil },
func(int) bool { return true }))
recovery := &fakeRecovery{restored: true}
guard := newTestGuard(recovery, probe)
// Bounded so the deadline-resetting loop cannot run forever if it stops
// resetting; the assertion is that the rollback was never reached.
ctx, cancel := context.WithTimeout(t.Context(), 200*time.Millisecond)
defer cancel()
guard.Run(ctx)
if recovery.restarts != 0 {
t.Errorf("restarts = %d, want none while the engine is still recovering", recovery.restarts)
}
}
// MariaDB releases a BACKUP STAGE with the connection that took it, so the whole
// sequence has to run on one held session — which is also what bounds a control
// plane that dies mid-snapshot.
func TestMariaDBEngine_QuiesceTakesTheBackupStagesOnAHeldSession(t *testing.T) {
session := &fakeSession{}
engine := newTestMariaDBEngine(t, (&recordingRunner{}).run)
var started []command
engine.startSess = func(_ context.Context, c command) (engineSession, error) {
started = append(started, c)
return session, nil
}
if err := engine.Quiesce(context.Background(), "orders-db-pre-upgrade", time.Minute); err != nil {
t.Fatalf("Quiesce: %v", err)
}
if len(started) != 1 {
t.Fatalf("started %d sessions, want 1", len(started))
}
if !slices.Contains(started[0].Args, "--unbuffered") {
t.Errorf("session argv %v does not flush after each statement, so the sentinel could sit in a pipe buffer", started[0].Args)
}
statements := session.statements()
want := []string{
"SET SESSION lock_wait_timeout = 20;",
"BACKUP STAGE START;",
"BACKUP STAGE FLUSH;",
"BACKUP STAGE BLOCK_DDL;",
"BACKUP STAGE BLOCK_COMMIT;",
}
if len(statements) != len(want) {
t.Fatalf("statements = %q, want the bounded stage sequence %q", statements, want)
}
for i, sql := range want {
if strings.TrimSpace(statements[i]) != sql {
t.Errorf("statement %d = %q, want %q", i, statements[i], sql)
}
}
// FLUSH TABLES WITH READ LOCK would make the whole database read-only for the
// length of the hold rather than blocking commits.
if strings.Contains(strings.Join(statements, " "), "FLUSH TABLES") {
t.Error("the quiesce took a read lock over the whole database")
}
if session.closes() != 0 {
t.Error("the session was closed while the backup was meant to be held")
}
}
// The stage bound has to leave the control plane's own quiesce timeout room for
// the whole sequence, or a stage abandoned mid-wait leaves its lock request
// queued in front of live traffic.
func TestMariaDBEngine_QuiesceLockWaitFitsInsideTheControlPlaneTimeout(t *testing.T) {
const stages = 4
if stages*mariadbQuiesceLockWait >= 2*time.Minute {
t.Errorf("%d stages of %s can outlast the control plane's quiesce timeout", stages, mariadbQuiesceLockWait)
}
}
func TestMariaDBEngine_UnquiesceEndsTheBackupAndTheSession(t *testing.T) {
session := &fakeSession{}
engine := newTestMariaDBEngine(t, (&recordingRunner{}).run)
engine.startSess = func(context.Context, command) (engineSession, error) { return session, nil }
if err := engine.Quiesce(context.Background(), "orders-db-pre-upgrade", time.Minute); err != nil {
t.Fatalf("Quiesce: %v", err)
}
if err := engine.Unquiesce(context.Background()); err != nil {
t.Fatalf("Unquiesce: %v", err)
}
statements := session.statements()
if last := strings.TrimSpace(statements[len(statements)-1]); last != "BACKUP STAGE END;" {
t.Errorf("last statement = %q, want the stage released", last)
}
if session.closes() != 1 {
t.Errorf("closed %d times, want the session ended exactly once", session.closes())
}
}
// The two halves of a time_zone apply use different names: SET GLOBAL takes the
// customer's, the option file the server's startup spelling. Getting either
// backwards is a statement the server refuses or a boot it refuses.
func TestMariaDBEngine_TimeZoneAppliesLiveByItsCustomerFacingName(t *testing.T) {
runner := &recordingRunner{}
engine := newTestMariaDBEngine(t, runner.run)
if _, err := engine.ApplyParameters(context.Background(), []handlers_rds.Parameter{
{Name: "time_zone", Value: "+10:00"},
}); err != nil {
t.Fatalf("ApplyParameters: %v", err)
}
if len(runner.calls) != 1 {
t.Fatalf("ran %d commands, want the one live apply", len(runner.calls))
}
sql := runner.calls[0].Stdin
if !strings.Contains(sql, "SET GLOBAL time_zone = '+10:00';") {
t.Errorf("SQL %q does not set time_zone under the name MariaDB accepts for SET", sql)
}
if strings.Contains(sql, "default_time_zone") {
t.Errorf("SQL %q names the startup spelling, which is not a settable variable", sql)
}
installed, err := os.ReadFile(engine.params.installedPath())
if err != nil {
t.Fatalf("read the installed parameters: %v", err)
}
if !strings.Contains(string(installed), "default_time_zone = '+10:00'") {
t.Errorf("installed parameters = %q, want the startup spelling", installed)
}
}
// A dynamic setting applied live deliberately leaves the serving copy alone, so
// the two files differ by design. Read back under the startup spelling the
// catalog does not carry, that difference would classify as static and report a
// pending restart that nothing could ever clear.
func TestMariaDBEngine_ALiveTimeZoneChangeLeavesNothingPendingRestart(t *testing.T) {
runner := &recordingRunner{}
engine := newTestMariaDBEngine(t, runner.run)
if err := os.WriteFile(engine.params.servingPath(),
[]byte("[mysqld]\ndefault_time_zone = 'SYSTEM'\n"), 0o600); err != nil {
t.Fatalf("seed the serving copy: %v", err)
}
pending, err := engine.ApplyParameters(context.Background(), []handlers_rds.Parameter{
{Name: "time_zone", Value: "+10:00"},
})
if err != nil {
t.Fatalf("ApplyParameters: %v", err)
}
if len(pending) != 0 {
t.Errorf("pending = %v, want nothing: time_zone is dynamic and was applied live", pending)
}
}
// The list is customer-facing, so a static setting still has to be reported
// under the name the customer set rather than the engine's startup spelling.
func TestMariaDBEngine_PendingRestartReportsCatalogNames(t *testing.T) {
runner := &recordingRunner{}
engine := newTestMariaDBEngine(t, runner.run)
if err := os.WriteFile(engine.params.servingPath(),
[]byte("[mysqld]\ninnodb_buffer_pool_size = '536870912'\ndefault_time_zone = 'SYSTEM'\n"), 0o600); err != nil {
t.Fatalf("seed the serving copy: %v", err)
}
pending, err := engine.ApplyParameters(context.Background(), []handlers_rds.Parameter{
{Name: "innodb_buffer_pool_size", Value: "1073741824"},
{Name: "time_zone", Value: "SYSTEM"},
})
if err != nil {
t.Fatalf("ApplyParameters: %v", err)
}
if !slices.Equal(pending, []string{"innodb_buffer_pool_size"}) {
t.Errorf("pending = %v, want only the static setting under its catalog name", pending)
}
}
+380
View File
@@ -0,0 +1,380 @@
package main
import (
"context"
"errors"
"fmt"
"log/slog"
"path/filepath"
"strconv"
"strings"
"sync"
"sync/atomic"
"time"
handlers_rds "github.com/mulgadc/spinifex/spinifex/handlers/rds"
)
// The engine is reached over its unix socket under peer authentication, so the
// agent drops to the postgres OS user rather than holding a password of its own.
type postgresEngine struct {
quiesceState
// The control plane's own metadata for this engine, resolved once at
// startup: the live password apply runs as the cluster superuser, so it
// re-checks the role name against the same reserved set the API validates.
meta handlers_rds.Engine
run commandRunner
startSess sessionRunner
psql string
rcService string
service string
pgData string
socketDir string
osUser string
probe *engineProbe
// Set from the bootstrap config, on a different goroutine than the commands
// that read it.
port atomic.Int64
params parameterStore
// Serializes parameter installs, serving snapshots and rollback restores so
// none can copy or replace an intermediate configuration.
paramMu sync.Mutex
repairTimeout time.Duration
repairPoll time.Duration
}
var _ engine = (*postgresEngine)(nil)
// pg_isready is what gives PostgreSQL three states for free: exit 0 is serving,
// exit 1 is a postmaster up and rejecting connections, and anything else is an
// engine that is not there at all. Resolved on PATH, where the client package
// puts it.
const postgresProbeBinary = "pg_isready"
func newPostgresProbe(cfg config, run probeRunner) *engineProbe {
return newEngineProbe(cfg.EnginePort, postgresProbeState(cfg.EngineHost, run))
}
func postgresProbeState(host string, run probeRunner) probeStateFn {
return func(ctx context.Context, port int64) (engineState, string) {
portArg := strconv.FormatInt(port, 10)
code, err := run(ctx, postgresProbeBinary, "-h", host, "-p", portArg, "-q")
switch {
case err != nil:
// A missing binary or broken image. Reporting healthy on the strength of
// nothing would hide it, so this reads as absent like an engine that did
// not answer.
return engineAbsent, fmt.Sprintf("engine probe could not run: %v", err)
case code == 0:
return engineServing, ""
case code == 1:
return engineRecovering, "engine is rejecting connections (startup or recovery)"
default:
return engineAbsent, fmt.Sprintf("engine did not respond on %s:%s", host, portArg)
}
}
}
// Resolved once, under the name the control plane knows this implementation by.
// A lookup failure is a mismatch between the agent and the control plane it was
// built against, so it fails at startup rather than at the first rotation.
var postgresEngineMeta = mustLookupEngine(enginePostgres)
func mustLookupEngine(name string) handlers_rds.Engine {
engine, err := handlers_rds.LookupEngine(name)
if err != nil {
panic("rds-agent: " + err.Error())
}
return engine
}
// The include the resolved parameter set is rendered to, and the copy of the
// last one the engine accepted. Both live beside the data rather than in /etc,
// matching rds-init: a class change boots a fresh root volume, which would
// otherwise revert them.
const (
postgresParametersFile = "10-rds-parameters.conf"
// Deliberately not a .conf name: include_dir globs *.conf, so the rollback
// copy must not be read as a second set of settings.
postgresLastGoodFile = "10-rds-parameters.last-good"
)
func newPostgresEngine(cfg config, run commandRunner, startSess sessionRunner, probe *engineProbe) *postgresEngine {
e := &postgresEngine{
meta: postgresEngineMeta,
run: run,
startSess: startSess,
psql: filepath.Join(cfg.EngineBinDir, "psql"),
rcService: cfg.RCService,
service: cfg.EngineService,
pgData: cfg.EngineDataDir,
socketDir: cfg.SocketDir,
osUser: cfg.EngineUser,
probe: probe,
params: parameterStore{
dir: filepath.Join(cfg.EngineDataDir, "conf.d"),
installed: postgresParametersFile,
lastGood: postgresLastGoodFile,
osUser: cfg.EngineUser,
engine: enginePostgres,
},
repairTimeout: parameterRepairTimeout,
repairPoll: parameterRepairPoll,
}
e.port.Store(int64(cfg.EnginePort))
return e
}
func (e *postgresEngine) setPort(port int) {
e.port.Store(int64(port))
}
// The role name and the password ride the environment and are re-quoted by
// psql, so neither reaches a shell word or an argv another process can read.
func (e *postgresEngine) SetPassword(ctx context.Context, username, password string) error {
if username == "" || password == "" {
return fmt.Errorf("set-password requires both %s and %s",
handlers_rds.CommandParamMasterUsername, handlers_rds.CommandParamMasterUserPassword)
}
// The statement below runs as the cluster superuser over the socket under
// peer auth, so a reserved name in the command payload would hand the
// customer the bootstrap superuser rather than rotate their own role.
if err := e.meta.ValidateUsernameNotReserved(username); err != nil {
return fmt.Errorf("refusing to set the password of a role the engine reserves: %w", err)
}
// psql interpolates the password into the ALTER ROLE before the server sees
// it, and these three are what would write it to the log — the last on any
// failure at its own default. SUSET, so the parameter group cannot win.
const sql = `SET log_statement = 'none';
SET log_min_duration_statement = -1;
SET log_min_error_statement = 'panic';
\getenv master RDS_MASTER_USERNAME
\getenv password RDS_MASTER_PASSWORD
ALTER ROLE :"master" WITH LOGIN PASSWORD :'password';
`
_, err := e.psqlRun(ctx, sql,
"RDS_MASTER_USERNAME="+username,
"RDS_MASTER_PASSWORD="+password,
)
if err != nil {
// psql echoes the failing statement, which here would carry the new
// password back to the control plane and into the event ring.
return fmt.Errorf("apply the master password: %s", redact(err.Error(), password))
}
return nil
}
// Installs the resolved set, validates it with the engine's own config parser
// before the engine ever adopts it, and reloads. A value the engine refuses is
// rolled back here rather than left on the data volume, where it would survive
// every VM replace and turn the next restart into a boot loop.
func (e *postgresEngine) ApplyParameters(ctx context.Context, params []handlers_rds.Parameter) ([]string, error) {
e.paramMu.Lock()
defer e.paramMu.Unlock()
initialState, _ := e.probe.state(ctx)
if initialState == engineRecovering {
return nil, errors.New("apply parameters while the engine is still starting or recovering")
}
if initialState == engineServing {
// A command can arrive before the first heartbeat. Seed the configuration
// currently being served before replacing its file in that window.
if err := e.recordServingParametersLocked(ctx); err != nil {
return nil, fmt.Errorf("record the parameters serving before the apply: %w", err)
}
}
// The check has to run against the file in place, because the engine parses
// the datadir's own include_dir. The window that leaves is closed from both
// ends: the rollback below, and the last-known-good restore the agent runs at
// boot when the engine does not come up.
restore, err := e.params.install(params)
if err != nil {
return nil, err
}
if err := e.checkConfig(ctx); err != nil {
return nil, errors.Join(fmt.Errorf("the engine rejected the parameter set: %w", err), restore())
}
state, _ := e.probe.state(ctx)
switch state {
case engineAbsent:
return e.restartOnRepairSetLocked(ctx)
case engineRecovering:
return nil, errors.Join(errors.New("the engine entered startup or recovery during the parameter apply"), restore())
}
if _, err := e.psqlRun(ctx, "SELECT pg_reload_conf();\n"); err != nil {
// The engine may have gone down after the first probe. In that case start
// it on the new, parser-checked repair set rather than restoring the set
// that already left it unable to serve.
if state, _ := e.probe.state(ctx); state == engineAbsent {
return e.restartOnRepairSetLocked(ctx)
}
return nil, errors.Join(fmt.Errorf("reload the engine configuration: %w", err), restore())
}
return e.pendingRestartParameters(ctx)
}
func (e *postgresEngine) restartOnRepairSetLocked(ctx context.Context) ([]string, error) {
return awaitRepairedEngine(ctx, e.probe, e.Restart, e.pendingRestartParameters, e.repairTimeout, e.repairPoll)
}
// The postmaster's own answer: a static setting it has stored but not adopted is
// what it reports here, so nothing in the guest has to classify one.
func (e *postgresEngine) pendingRestartParameters(ctx context.Context) ([]string, error) {
out, err := e.psqlRun(ctx, "SELECT name FROM pg_settings WHERE pending_restart ORDER BY name;\n")
if err != nil {
return nil, fmt.Errorf("read the settings pending a restart: %w", err)
}
var pending []string
for line := range strings.SplitSeq(out, "\n") {
if name := strings.TrimSpace(line); name != "" {
pending = append(pending, name)
}
}
return pending, nil
}
// Snapshots the include only when the running postmaster has no settings still
// pending a restart. The parameter mutex keeps an apply from replacing the file
// between that check and the copy.
func (e *postgresEngine) RecordServingParameters(ctx context.Context) error {
e.paramMu.Lock()
defer e.paramMu.Unlock()
return e.recordServingParametersLocked(ctx)
}
func (e *postgresEngine) recordServingParametersLocked(ctx context.Context) error {
return recordLastGood(ctx, e.params, e.pendingRestartParameters)
}
// Puts the last set the engine accepted back in place, for a restart that failed
// after a parameter change.
func (e *postgresEngine) RestoreLastKnownGoodParameters(ctx context.Context) (bool, error) {
e.paramMu.Lock()
defer e.paramMu.Unlock()
return restoreLastGood(ctx, e.params, e.probe)
}
// The engine's own parser, run offline against the datadir. Reading one setting
// back is enough: postgres parses postgresql.conf and every include first, and
// exits non-zero naming the file and line of an unknown parameter or a value
// outside its range.
func (e *postgresEngine) checkConfig(ctx context.Context) error {
if _, err := e.run(ctx, command{
Name: filepath.Join(filepath.Dir(e.psql), "postgres"),
Args: []string{"-D", e.pgData, "-C", "shared_buffers"},
Env: []string{"PATH=" + defaultGuestPath},
User: e.osUser,
}); err != nil {
return err
}
return nil
}
func (e *postgresEngine) Stop(ctx context.Context) error {
return serviceAction(ctx, e.run, e.rcService, e.service, "stop")
}
// Only the parameter rollback calls this: a restart the control plane wants goes
// through RebootDBInstance, which cycles the VM.
func (e *postgresEngine) Restart(ctx context.Context) error {
return serviceAction(ctx, e.run, e.rcService, e.service, "restart")
}
// Feeds sql on stdin rather than as an argument, so a statement is never
// visible in the process table.
func (e *postgresEngine) psqlRun(ctx context.Context, sql string, env ...string) (string, error) {
return e.run(ctx, command{
Name: e.psql,
Args: e.psqlArgs(),
Env: append([]string{"PATH=" + defaultGuestPath}, env...),
Stdin: sql,
User: e.osUser,
})
}
// Reading the script from stdin is what lets one invocation serve both a
// one-shot run and a session held open across several statements.
func (e *postgresEngine) psqlArgs() []string {
return []string{
"--no-psqlrc", "--quiet", "--no-align", "--tuples-only",
"-v", "ON_ERROR_STOP=1",
"-h", e.socketDir,
"-p", strconv.FormatInt(e.port.Load(), 10),
"-U", e.osUser,
"-d", "postgres",
"-f", "-",
}
}
// Puts the engine into backup mode: the datadir is checkpointed and the engine
// stops writing over the pages a snapshot is about to read. The hold is released
// by Unquiesce, or by its own deadline, whichever comes first.
func (e *postgresEngine) Quiesce(ctx context.Context, label string, hold time.Duration) error {
if err := validateQuiesceRequest(label, hold); err != nil {
return err
}
// Held across the whole start, so a second quiesce waits and then finds the
// first one's hold rather than opening a concurrent backup alongside it.
e.mu.Lock()
defer e.mu.Unlock()
if e.held != nil {
return fmt.Errorf("the engine is already quiesced for backup %s", e.held.label)
}
// Deliberately not the command's context: the session has to outlive the call
// that started it, and its own deadline is what bounds it instead.
session, err := e.startSess(context.WithoutCancel(ctx), command{
Name: e.psql,
Args: e.psqlArgs(),
Env: []string{"PATH=" + defaultGuestPath, "RDS_BACKUP_LABEL=" + label},
User: e.osUser,
SentinelStatement: `\echo ` + sessionSentinel + "\n",
})
if err != nil {
return fmt.Errorf("open a backup session: %w", err)
}
// fast forces an immediate checkpoint rather than spreading it over the
// checkpoint interval, which would hold the snapshot open for minutes.
const sql = `\getenv label RDS_BACKUP_LABEL
SELECT pg_backup_start(:'label', fast => true);
`
if err := session.Exec(ctx, sql); err != nil {
if closeErr := session.Close(); closeErr != nil {
slog.Warn("rds-agent: closing a failed backup session", "err", closeErr)
}
return fmt.Errorf("put the engine into backup mode: %w", err)
}
e.beginHoldLocked(label, session, hold)
return nil
}
// Ends the backup cleanly. A missing hold is an error rather than a silent
// success: it means the deadline fired first, so the snapshot the control plane
// just took was not taken against a held checkpoint.
func (e *postgresEngine) Unquiesce(ctx context.Context) error {
held := e.takeHold()
if held == nil {
return errors.New("the engine is not quiesced; the backup hold had already expired")
}
// The stop has to run on the session that started the backup, and the session
// is closed either way — the engine aborts an unstopped backup with it.
execErr := held.session.Exec(ctx, "SELECT pg_backup_stop();\n")
closeErr := held.session.Close()
if execErr != nil {
return fmt.Errorf("take the engine out of backup mode: %w", errors.Join(execErr, closeErr))
}
if closeErr != nil {
return fmt.Errorf("close the backup session: %w", closeErr)
}
slog.Info("rds-agent: engine released from backup mode", "label", held.label)
return nil
}
+8 -29
View File
@@ -141,9 +141,9 @@ func (r *recordingRunner) run(_ context.Context, c command) (string, error) {
func newTestEngine(t *testing.T, run commandRunner) *postgresEngine {
t.Helper()
cfg := loadConfig(filepath.Join(t.TempDir(), "absent.env"))
cfg.PGData = t.TempDir()
if err := os.MkdirAll(filepath.Join(cfg.PGData, "conf.d"), 0o700); err != nil {
cfg := testLoadConfig(t, enginePostgres)
cfg.EngineDataDir = t.TempDir()
if err := os.MkdirAll(filepath.Join(cfg.EngineDataDir, "conf.d"), 0o700); err != nil {
t.Fatalf("create conf.d: %v", err)
}
// The guest's postgres user does not exist here, and the chown of the
@@ -152,8 +152,8 @@ func newTestEngine(t *testing.T, run commandRunner) *postgresEngine {
if err != nil {
t.Fatalf("resolve the current user: %v", err)
}
cfg.PGUser = current.Username
return newPostgresEngine(cfg, run, nil, newEngineProbe(cfg, staticProbe(0)))
cfg.EngineUser = current.Username
return newPostgresEngine(cfg, run, nil, newPostgresProbe(cfg, staticProbe(0)))
}
// The password reaches psql through the environment and is re-quoted there,
@@ -341,7 +341,7 @@ func TestPostgresEngine_ApplyParametersRestartsOnARepairSetWhileEngineIsDown(t *
codes := []int{2, 2, 0}
probeCall := 0
cfg := testProbeConfig()
engine.probe = newEngineProbe(cfg, func(context.Context, string, ...string) (int, error) {
engine.probe = newPostgresProbe(cfg, func(context.Context, string, ...string) (int, error) {
code := codes[min(probeCall, len(codes)-1)]
probeCall++
return code, nil
@@ -372,7 +372,7 @@ func TestPostgresEngine_ApplyParametersRestartsOnARepairSetWhileEngineIsDown(t *
if !strings.Contains(runner.calls[2].Stdin, "pending_restart") {
t.Errorf("third call = %+v, want the pending-restart read", runner.calls[2])
}
installed, err := os.ReadFile(engine.parametersPath())
installed, err := os.ReadFile(engine.params.installedPath())
if err != nil {
t.Fatalf("read installed parameters: %v", err)
}
@@ -392,7 +392,7 @@ func TestPostgresEngine_ApplyParametersKeepsRepairSetWhenRestartTimesOut(t *test
if err == nil || !strings.Contains(err.Error(), "wait for the engine") {
t.Fatalf("ApplyParameters error = %v, want the repair wait failure", err)
}
installed, readErr := os.ReadFile(engine.parametersPath())
installed, readErr := os.ReadFile(engine.params.installedPath())
if readErr != nil {
t.Fatalf("read installed parameters: %v", readErr)
}
@@ -400,24 +400,3 @@ func TestPostgresEngine_ApplyParametersKeepsRepairSetWhenRestartTimesOut(t *test
t.Errorf("installed parameters = %q, want the checked repair set retained", installed)
}
}
// Through the service manager, so the supervisor records the engine as stopped
// and does not restart it underneath a VM that is going down.
func TestPostgresEngine_StopGoesThroughTheServiceManager(t *testing.T) {
runner := &recordingRunner{}
engine := newTestEngine(t, runner.run)
if err := engine.Stop(context.Background()); err != nil {
t.Fatalf("Stop: %v", err)
}
if len(runner.calls) != 1 {
t.Fatalf("ran %d commands, want 1", len(runner.calls))
}
call := runner.calls[0]
if filepath.Base(call.Name) != "rc-service" {
t.Errorf("ran %q, want rc-service", call.Name)
}
if len(call.Args) != 2 || call.Args[0] != "postgresql" || call.Args[1] != "stop" {
t.Errorf("args = %v, want [postgresql stop]", call.Args)
}
}
+17 -26
View File
@@ -3,9 +3,7 @@ package main
import (
"context"
"errors"
"fmt"
"os/exec"
"strconv"
"sync/atomic"
handlers_rds "github.com/mulgadc/spinifex/spinifex/handlers/rds"
@@ -20,6 +18,9 @@ func execProbeRunner(ctx context.Context, name string, args ...string) (int, err
if err == nil {
return 0, nil
}
if ctxErr := ctx.Err(); ctxErr != nil {
return -1, ctxErr
}
var exitErr *exec.ExitError
if errors.As(err, &exitErr) {
return exitErr.ExitCode(), nil
@@ -27,12 +28,16 @@ func execProbeRunner(ctx context.Context, name string, args ...string) (int, err
return -1, err
}
// Shells out to pg_isready rather than dialling the port, since only a startup
// exchange separates an engine that is serving from one still in recovery.
// What the engine reports about itself, which is the only part of the probe
// that differs between engines. The port is passed in rather than read from the
// engine, so the atomic below stays the one place it is followed from.
type probeStateFn func(ctx context.Context, port int64) (engineState, string)
// Engine-neutral: the latch, the port, the degraded mapping and the translation
// to a reported health are shared, and only the state function is supplied by
// the engine the image bakes.
type engineProbe struct {
run probeRunner
binary string
host string
stateFn probeStateFn
// Set from the bootstrap config once it lands, from a different goroutine
// than the heartbeat that reads it.
port atomic.Int64
@@ -41,9 +46,9 @@ type engineProbe struct {
seenHealthy bool
}
func newEngineProbe(cfg config, run probeRunner) *engineProbe {
p := &engineProbe{run: run, binary: cfg.PGIsReady, host: cfg.EngineHost}
p.port.Store(int64(cfg.EnginePort))
func newEngineProbe(port int, stateFn probeStateFn) *engineProbe {
p := &engineProbe{stateFn: stateFn}
p.port.Store(int64(port))
return p
}
@@ -60,7 +65,7 @@ type engineState int
const (
// Not answering at all, or the probe could not run.
engineAbsent engineState = iota
// The postmaster is up and rejecting connections: startup or crash recovery.
// The process is up but not yet serving: startup or crash recovery.
engineRecovering
engineServing
)
@@ -83,21 +88,7 @@ func (p *engineProbe) Check(ctx context.Context) (handlers_rds.EngineHealth, str
// The raw probe result. Safe to call from a goroutine other than the
// heartbeat's: it touches no latched state.
func (p *engineProbe) state(ctx context.Context) (engineState, string) {
port := strconv.FormatInt(p.port.Load(), 10)
code, err := p.run(ctx, p.binary, "-h", p.host, "-p", port, "-q")
switch {
case err != nil:
// A missing binary or broken image. Reporting healthy on the strength of
// nothing would hide it, so this reads as absent like an engine that did
// not answer.
return engineAbsent, fmt.Sprintf("engine probe could not run: %v", err)
case code == 0:
return engineServing, ""
case code == 1:
return engineRecovering, "engine is rejecting connections (startup or recovery)"
default:
return engineAbsent, fmt.Sprintf("engine did not respond on %s:%s", p.host, port)
}
return p.stateFn(ctx, p.port.Load())
}
// A non-answering engine: starting until it has answered once, unhealthy after.
+19 -5
View File
@@ -5,6 +5,7 @@ import (
"errors"
"strings"
"testing"
"time"
handlers_rds "github.com/mulgadc/spinifex/spinifex/handlers/rds"
)
@@ -28,7 +29,7 @@ func TestEngineProbe_MapsExitCodesToHealth(t *testing.T) {
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
probe := newEngineProbe(testProbeConfig(), func(context.Context, string, ...string) (int, error) {
probe := newPostgresProbe(testProbeConfig(), func(context.Context, string, ...string) (int, error) {
return tt.code, tt.runErr
})
probe.seenHealthy = tt.seenHealthy
@@ -51,7 +52,7 @@ func TestEngineProbe_MapsExitCodesToHealth(t *testing.T) {
// has to flip the first time the engine answers.
func TestEngineProbe_LatchesAfterFirstHealthy(t *testing.T) {
code := 0
probe := newEngineProbe(testProbeConfig(), func(context.Context, string, ...string) (int, error) {
probe := newPostgresProbe(testProbeConfig(), func(context.Context, string, ...string) (int, error) {
return code, nil
})
@@ -68,7 +69,7 @@ func TestEngineProbe_LatchesAfterFirstHealthy(t *testing.T) {
// must follow it rather than keep asking the default.
func TestEngineProbe_ProbesTheAssignedPort(t *testing.T) {
var gotArgs []string
probe := newEngineProbe(testProbeConfig(), func(_ context.Context, _ string, args ...string) (int, error) {
probe := newPostgresProbe(testProbeConfig(), func(_ context.Context, _ string, args ...string) (int, error) {
gotArgs = args
return 0, nil
})
@@ -81,6 +82,19 @@ func TestEngineProbe_ProbesTheAssignedPort(t *testing.T) {
}
}
func testProbeConfig() config {
return config{EngineHost: defaultEngineHost, EnginePort: defaultEnginePort, PGIsReady: defaultPGIsReady}
func TestExecProbeRunner_PreservesTheProbeDeadline(t *testing.T) {
ctx, cancel := context.WithTimeout(t.Context(), 20*time.Millisecond)
defer cancel()
code, err := execProbeRunner(ctx, "sh", "-c", "while :; do :; done")
if code != -1 {
t.Errorf("exit code = %d, want -1 for a timed-out probe", code)
}
if !errors.Is(err, context.DeadlineExceeded) {
t.Errorf("error = %v, want the context deadline", err)
}
}
func testProbeConfig() config {
return config{EngineHost: defaultEngineHost, EnginePort: engineLayouts[enginePostgres].port}
}
+4 -4
View File
@@ -17,7 +17,7 @@ func TestHeartbeater_FirstServingProbeSeedsLastKnownGood(t *testing.T) {
runner := &recordingRunner{}
engine := newTestEngine(t, runner.run)
content := []byte("work_mem = '4096'\n")
if err := os.WriteFile(engine.parametersPath(), content, 0o600); err != nil {
if err := os.WriteFile(engine.params.installedPath(), content, 0o600); err != nil {
t.Fatalf("write installed parameters: %v", err)
}
@@ -25,7 +25,7 @@ func TestHeartbeater_FirstServingProbeSeedsLastKnownGood(t *testing.T) {
h := newHeartbeater(cp, engine.probe, engine, 0)
h.beat(context.Background())
lastGood, err := os.ReadFile(engine.lastGoodPath())
lastGood, err := os.ReadFile(engine.params.lastGoodPath())
if err != nil {
t.Fatalf("read last known good: %v", err)
}
@@ -39,7 +39,7 @@ func TestHeartbeater_FirstServingProbeSeedsLastKnownGood(t *testing.T) {
}
func TestHeartbeater_BoundsBootstrapFailure(t *testing.T) {
h := newHeartbeater(newFakeControlPlane(), newEngineProbe(testProbeConfig(), staticProbe(2)), nil, 0)
h := newHeartbeater(newFakeControlPlane(), newPostgresProbe(testProbeConfig(), staticProbe(2)), nil, 0)
h.setBootstrapFailure("bootstrap fetch", errors.New(strings.Repeat("界", 1000)))
failure := h.bootstrapFailure.Load()
@@ -60,7 +60,7 @@ func TestHeartbeater_BoundsBootstrapFailure(t *testing.T) {
func TestHeartbeater_ChecksServingParametersOnEveryHealthyProbe(t *testing.T) {
code := 0
cfg := testProbeConfig()
probe := newEngineProbe(cfg, func(context.Context, string, ...string) (int, error) {
probe := newPostgresProbe(cfg, func(context.Context, string, ...string) (int, error) {
return code, nil
})
recorder := &countingServingRecorder{}
+223
View File
@@ -0,0 +1,223 @@
package main
import (
"bytes"
"context"
"fmt"
"os"
"path/filepath"
"time"
handlers_rds "github.com/mulgadc/spinifex/spinifex/handlers/rds"
)
// The generated parameter files and what it takes to install one. They live on
// the data volume rather than under /etc, at the same paths rds-init writes them
// to: a class change is a VM replacement, so a set under /etc would be discarded
// with the boot volume and the instance would silently revert to catalog
// defaults while the API still reported the customer's values.
type parameterStore struct {
dir string
// The include the resolved set is rendered to.
installed string
// The copy of the last set the engine accepted, which a boot loop is rolled
// back to. Deliberately not a name the engine's own include glob reads, so it
// is never parsed as a second set of settings.
lastGood string
// What the engine actually started on, for an engine that cannot be asked
// which of its settings are still pending a restart. Empty for one that can.
serving string
// Prepended to every file written here, for a configuration format that has
// no setting outside a group.
header string
// The engine parses its own configuration, so a root-owned file would be
// unreadable to it.
osUser string
// Names the engine whose option-file spellings these files are written in.
engine string
}
func (s parameterStore) installedPath() string { return filepath.Join(s.dir, s.installed) }
func (s parameterStore) lastGoodPath() string { return filepath.Join(s.dir, s.lastGood) }
func (s parameterStore) servingPath() string { return filepath.Join(s.dir, s.serving) }
// Byte-for-byte what rds-init writes for the same set, so the copies compared
// against it are comparing values rather than formatting.
func (s parameterStore) render(params []handlers_rds.Parameter) ([]byte, error) {
body, err := renderParameters(s.engine, params)
if err != nil {
return nil, err
}
return []byte(s.header + body), nil
}
// Installs the resolved set at the path rds-init uses, so a later boot
// overwrites rather than shadows it. Returns a func putting back whatever was
// there, so the caller can undo the install without re-rendering anything.
func (s parameterStore) install(params []handlers_rds.Parameter) (restore func() error, err error) {
path := s.installedPath()
// A missing file is the first-ever apply, and its rollback is a removal.
previous, readErr := os.ReadFile(path)
if readErr != nil && !os.IsNotExist(readErr) {
return nil, fmt.Errorf("read the installed parameters at %s: %w", path, readErr)
}
existed := readErr == nil
rendered, err := s.render(params)
if err != nil {
return nil, err
}
if err := s.write(path, rendered); err != nil {
return nil, err
}
return func() error {
if !existed {
if err := os.Remove(path); err != nil && !os.IsNotExist(err) {
return fmt.Errorf("withdraw the rejected parameters at %s: %w", path, err)
}
return nil
}
return s.write(path, previous)
}, nil
}
// Atomic: a temp file in the same directory, renamed over the target. The temp
// name deliberately does not end in the engine's include suffix, so a crash
// between write and rename cannot leave the engine reading a half-written file.
func (s parameterStore) write(path string, content []byte) error {
tmp := path + ".new"
if err := os.WriteFile(tmp, content, 0o600); err != nil {
return fmt.Errorf("write %s: %w", tmp, err)
}
if err := s.chownToEngine(tmp); err != nil {
os.Remove(tmp)
return err
}
if err := os.Rename(tmp, path); err != nil {
os.Remove(tmp)
return fmt.Errorf("install %s: %w", path, err)
}
return nil
}
// A guest with no engine user is a broken image, not something to work around.
func (s parameterStore) chownToEngine(path string) error {
credential, err := lookupCredential(s.osUser)
if err != nil {
return err
}
if err := os.Chown(path, int(credential.Uid), int(credential.Gid)); err != nil {
return fmt.Errorf("hand %s to %s: %w", path, s.osUser, err)
}
return nil
}
// Snapshots the installed set as the rollback target, but only once the engine
// is running all of it. A set with a setting still pending a restart has not
// been shown to serve, and promoting it would leave the rollback pointing at the
// configuration that is about to fail.
func recordLastGood(ctx context.Context, store parameterStore, pending pendingRestartFn) error {
if _, err := os.Stat(store.installedPath()); err != nil {
if os.IsNotExist(err) {
return nil
}
return fmt.Errorf("inspect the serving parameters: %w", err)
}
names, err := pending(ctx)
if err != nil {
return err
}
if len(names) > 0 {
return nil
}
content, err := os.ReadFile(store.installedPath())
if err != nil {
return fmt.Errorf("read the serving parameters: %w", err)
}
lastGood, err := os.ReadFile(store.lastGoodPath())
if err == nil && bytes.Equal(content, lastGood) {
return nil
}
if err != nil && !os.IsNotExist(err) {
return fmt.Errorf("read the last known good parameters: %w", err)
}
return store.write(store.lastGoodPath(), content)
}
// Puts the last set the engine accepted back in place, for a restart that failed
// after a parameter change. The probe is checked again under the caller's
// parameter lock, so a repair that just brought the engine back is not reversed.
func restoreLastGood(ctx context.Context, store parameterStore, probe *engineProbe) (bool, error) {
if state, _ := probe.state(ctx); state != engineAbsent {
return false, nil
}
lastGood, err := os.ReadFile(store.lastGoodPath())
if err != nil {
if os.IsNotExist(err) {
return false, nil
}
return false, fmt.Errorf("read the last known good parameters: %w", err)
}
current, err := os.ReadFile(store.installedPath())
if err != nil && !os.IsNotExist(err) {
return false, fmt.Errorf("read the installed parameters: %w", err)
}
if bytes.Equal(current, lastGood) {
return false, nil
}
if err := store.write(store.installedPath(), lastGood); err != nil {
return false, err
}
return true, nil
}
// The settings the engine has accepted but will not honour until it restarts.
// Each implementation answers it its own way, and both the apply and the serving
// snapshot are gated on the same answer.
type pendingRestartFn func(ctx context.Context) ([]string, error)
const (
parameterRepairTimeout = 90 * time.Second
parameterRepairPoll = time.Second
)
// Starts a down engine on the set that was just installed and waits for it to
// serve. Reached when the engine went away during an apply: the installed set is
// the newer of the two, so starting on it beats restoring one that already left
// the engine unable to serve.
func awaitRepairedEngine(ctx context.Context, probe *engineProbe, restart func(context.Context) error,
pending pendingRestartFn, timeout, poll time.Duration) ([]string, error) {
repairCtx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
if err := restart(repairCtx); err != nil {
return nil, fmt.Errorf("start the engine on the repaired parameter set: %w", err)
}
timer := time.NewTimer(0)
defer timer.Stop()
lastMessage := "engine did not respond"
for {
select {
case <-repairCtx.Done():
if ctx.Err() != nil {
return nil, fmt.Errorf("wait for the engine on the repaired parameter set: %w", ctx.Err())
}
return nil, fmt.Errorf("wait for the engine on the repaired parameter set: %s: %w", lastMessage, repairCtx.Err())
case <-timer.C:
}
state, message := probe.state(repairCtx)
if state == engineServing {
names, err := pending(repairCtx)
if err == nil {
return names, nil
}
lastMessage = err.Error()
} else if message != "" {
lastMessage = message
}
timer.Reset(poll)
}
}
+17 -18
View File
@@ -4,7 +4,6 @@ import (
"context"
"errors"
"os"
"path/filepath"
"slices"
"strings"
"testing"
@@ -50,7 +49,7 @@ func TestPostgresEngine_ApplyParametersRollsBackAValueTheEngineRejects(t *testin
t.Fatal("ApplyParameters succeeded against a config the engine rejected")
}
installed, err := os.ReadFile(engine.parametersPath())
installed, err := os.ReadFile(engine.params.installedPath())
if err != nil {
t.Fatalf("read the installed parameters: %v", err)
}
@@ -69,7 +68,7 @@ func TestPostgresEngine_ApplyParametersWithdrawsTheFirstRejectedSet(t *testing.T
[]handlers_rds.Parameter{{Name: "work_mem", Value: "0"}}); err == nil {
t.Fatal("ApplyParameters succeeded against a config the engine rejected")
}
if _, err := os.Stat(engine.parametersPath()); !os.IsNotExist(err) {
if _, err := os.Stat(engine.params.installedPath()); !os.IsNotExist(err) {
t.Errorf("the rejected include is still installed (stat err = %v)", err)
}
}
@@ -97,7 +96,7 @@ func TestPostgresEngine_ApplyBeforeFirstHeartbeatSeedsLastKnownGood(t *testing.T
runner := &pendingAfterReloadRunner{}
engine := newTestEngine(t, runner.run)
first := []byte("shared_buffers = '32768'\n")
if err := os.WriteFile(engine.parametersPath(), first, 0o600); err != nil {
if err := os.WriteFile(engine.params.installedPath(), first, 0o600); err != nil {
t.Fatalf("write serving parameter set: %v", err)
}
@@ -108,7 +107,7 @@ func TestPostgresEngine_ApplyBeforeFirstHeartbeatSeedsLastKnownGood(t *testing.T
if !slices.Equal(pending, []string{"shared_buffers"}) {
t.Errorf("pending = %v, want shared_buffers", pending)
}
lastGood, err := os.ReadFile(engine.lastGoodPath())
lastGood, err := os.ReadFile(engine.params.lastGoodPath())
if err != nil {
t.Fatalf("read last known good: %v", err)
}
@@ -153,10 +152,10 @@ func TestPostgresEngine_ServingSnapshotWaitsForParameterApply(t *testing.T) {
}
engine := newTestEngine(t, runner.run)
first := []byte("shared_buffers = '32768'\n")
if err := os.WriteFile(engine.parametersPath(), first, 0o600); err != nil {
if err := os.WriteFile(engine.params.installedPath(), first, 0o600); err != nil {
t.Fatalf("write serving parameter set: %v", err)
}
if err := os.WriteFile(engine.lastGoodPath(), first, 0o600); err != nil {
if err := os.WriteFile(engine.params.lastGoodPath(), first, 0o600); err != nil {
t.Fatalf("write last known good: %v", err)
}
@@ -192,7 +191,7 @@ func TestPostgresEngine_ServingSnapshotWaitsForParameterApply(t *testing.T) {
t.Error("serving snapshot completed while the parameter apply held its transaction")
}
lastGood, err := os.ReadFile(engine.lastGoodPath())
lastGood, err := os.ReadFile(engine.params.lastGoodPath())
if err != nil {
t.Fatalf("read last known good: %v", err)
}
@@ -216,7 +215,7 @@ func TestPostgresEngine_LastKnownGoodTracksTheServingSet(t *testing.T) {
}
assertLastGood := func(value string) {
t.Helper()
lastGood, err := os.ReadFile(engine.lastGoodPath())
lastGood, err := os.ReadFile(engine.params.lastGoodPath())
if err != nil {
t.Fatalf("read the last known good parameters: %v", err)
}
@@ -240,8 +239,8 @@ func TestPostgresEngine_LastKnownGoodTracksTheServingSet(t *testing.T) {
apply("16384")
assertLastGood("8192")
if strings.HasSuffix(engine.lastGoodPath(), ".conf") {
t.Errorf("the rollback copy at %s would be included by the engine", engine.lastGoodPath())
if strings.HasSuffix(engine.params.lastGoodPath(), ".conf") {
t.Errorf("the rollback copy at %s would be included by the engine", engine.params.lastGoodPath())
}
}
@@ -249,7 +248,7 @@ func TestPostgresEngine_RecordServingParametersSkipsPendingRestart(t *testing.T)
runner := &recordingRunner{}
engine := newTestEngine(t, runner.run)
first := []byte("shared_buffers = '32768'\n")
if err := os.WriteFile(engine.parametersPath(), first, 0o600); err != nil {
if err := os.WriteFile(engine.params.installedPath(), first, 0o600); err != nil {
t.Fatalf("write first parameter set: %v", err)
}
if err := engine.RecordServingParameters(t.Context()); err != nil {
@@ -257,14 +256,14 @@ func TestPostgresEngine_RecordServingParametersSkipsPendingRestart(t *testing.T)
}
second := []byte("shared_buffers = '65536'\n")
if err := os.WriteFile(engine.parametersPath(), second, 0o600); err != nil {
if err := os.WriteFile(engine.params.installedPath(), second, 0o600); err != nil {
t.Fatalf("write pending parameter set: %v", err)
}
runner.out = "shared_buffers\n"
if err := engine.RecordServingParameters(t.Context()); err != nil {
t.Fatalf("record with a pending restart: %v", err)
}
lastGood, err := os.ReadFile(engine.lastGoodPath())
lastGood, err := os.ReadFile(engine.params.lastGoodPath())
if err != nil {
t.Fatalf("read last known good: %v", err)
}
@@ -278,7 +277,7 @@ func TestPostgresEngine_RecordServingParametersSkipsPendingRestart(t *testing.T)
if err := engine.RecordServingParameters(t.Context()); err != nil {
t.Fatalf("record the restarted parameter set: %v", err)
}
lastGood, err = os.ReadFile(engine.lastGoodPath())
lastGood, err = os.ReadFile(engine.params.lastGoodPath())
if err != nil {
t.Fatalf("read advanced last known good: %v", err)
}
@@ -330,7 +329,7 @@ func TestPostgresEngine_RestoreLastKnownGoodParameters(t *testing.T) {
if !restored {
t.Fatal("did not restore a set that differs from the last known good")
}
installed, err := os.ReadFile(engine.parametersPath())
installed, err := os.ReadFile(engine.params.installedPath())
if err != nil {
t.Fatalf("read the installed parameters: %v", err)
}
@@ -369,8 +368,8 @@ func (f *fakeRecovery) Restart(context.Context) error {
// A probe whose result the test drives directly, standing in for pg_isready.
func stubProbe(t *testing.T, code int, err error) *engineProbe {
t.Helper()
cfg := loadConfig(filepath.Join(t.TempDir(), "absent.env"))
return newEngineProbe(cfg, func(context.Context, string, ...string) (int, error) {
cfg := testLoadConfig(t, enginePostgres)
return newPostgresProbe(cfg, func(context.Context, string, ...string) (int, error) {
return code, err
})
}
+72 -91
View File
@@ -15,10 +15,11 @@ import (
"time"
)
// PostgreSQL's non-exclusive backup API ties the backup to the session that
// started it: when that session ends, for any reason, the backup is aborted.
// That is what makes the hold self-expiring — a control plane that dies mid
// snapshot cannot leave the engine in backup mode forever.
// Both engines tie a backup hold to the session that took it: PostgreSQL aborts
// a non-exclusive backup when its session ends, and MariaDB releases a BACKUP
// STAGE with the connection that started it. That is what makes the hold
// self-expiring — a control plane that dies mid snapshot cannot leave the engine
// in backup mode forever.
type engineSession interface {
// Runs sql and waits for the engine to finish it.
Exec(ctx context.Context, sql string) error
@@ -31,7 +32,8 @@ type engineSession interface {
type sessionRunner func(ctx context.Context, c command) (engineSession, error)
// Written after every statement so the reader knows the engine is done with it.
// The statements this carries return LSNs, so it cannot appear in a result.
// The statements this carries return LSNs and backup stages, so it cannot appear
// in a result.
const sessionSentinel = "--rds-agent-ready--"
// The backup mode currently held, and the timer that ends it regardless of what
@@ -42,98 +44,65 @@ type quiesceHold struct {
expiry *time.Timer
}
// Puts the engine into backup mode: the datadir is checkpointed and the engine
// stops writing over the pages a snapshot is about to read. The hold is released
// by Unquiesce, or by its own deadline, whichever comes first.
func (e *postgresEngine) Quiesce(ctx context.Context, label string, hold time.Duration) error {
// The hold bookkeeping, which is the same for both engines: only the statements
// that take and release it differ, not the single-hold rule or the deadline that
// releases it whatever happens to the caller.
type quiesceState struct {
// Guarded because the expiry timer releases the hold on its own goroutine.
mu sync.Mutex
held *quiesceHold
}
// The two things every quiesce needs, checked before a session is opened so a
// malformed command costs the engine nothing.
func validateQuiesceRequest(label string, hold time.Duration) error {
if label == "" {
return errors.New("quiesce requires a backup label")
}
if hold <= 0 {
return errors.New("quiesce requires a positive hold deadline")
}
return nil
}
// Held across the whole start, so a second quiesce waits and then finds the
// first one's hold rather than opening a concurrent backup alongside it.
e.mu.Lock()
defer e.mu.Unlock()
if e.held != nil {
return fmt.Errorf("the engine is already quiesced for backup %s", e.held.label)
}
// Deliberately not the command's context: the session has to outlive the call
// that started it, and its own deadline is what bounds it instead.
session, err := e.startSess(context.WithoutCancel(ctx), command{
Name: e.psql,
Args: e.psqlArgs(),
Env: []string{"PATH=" + defaultGuestPath, "RDS_BACKUP_LABEL=" + label},
Stdin: "",
User: e.osUser,
})
if err != nil {
return fmt.Errorf("open a backup session: %w", err)
}
// fast forces an immediate checkpoint rather than spreading it over the
// checkpoint interval, which would hold the snapshot open for minutes.
const sql = `\getenv label RDS_BACKUP_LABEL
SELECT pg_backup_start(:'label', fast => true);
`
if err := session.Exec(ctx, sql); err != nil {
if closeErr := session.Close(); closeErr != nil {
slog.Warn("rds-agent: closing a failed backup session", "err", closeErr)
}
return fmt.Errorf("put the engine into backup mode: %w", err)
}
e.held = &quiesceHold{
// Records the hold and arms its deadline. Called with mu held, so a second
// quiesce waits and then finds this hold rather than opening a concurrent
// backup alongside it.
func (q *quiesceState) beginHoldLocked(label string, session engineSession, hold time.Duration) {
q.held = &quiesceHold{
label: label,
session: session,
expiry: time.AfterFunc(hold, func() { e.expireQuiesce(label) }),
expiry: time.AfterFunc(hold, func() { q.expire(label) }),
}
slog.Info("rds-agent: engine quiesced for backup", "label", label, "hold", hold)
return nil
}
// Ends the backup cleanly. A missing hold is an error rather than a silent
// success: it means the deadline fired first, so the snapshot the control plane
// just took was not taken against a held checkpoint.
func (e *postgresEngine) Unquiesce(ctx context.Context) error {
e.mu.Lock()
held := e.held
e.held = nil
e.mu.Unlock()
// Takes the hold off the engine, if one is still there, and disarms its
// deadline. The caller ends the session, because the statement that releases the
// backup has to run on it first.
func (q *quiesceState) takeHold() *quiesceHold {
q.mu.Lock()
defer q.mu.Unlock()
if held == nil {
return errors.New("the engine is not quiesced; the backup hold had already expired")
held := q.held
q.held = nil
if held != nil {
held.expiry.Stop()
}
held.expiry.Stop()
// The stop has to run on the session that started the backup, and the session
// is closed either way — the engine aborts an unstopped backup with it.
execErr := held.session.Exec(ctx, "SELECT pg_backup_stop();\n")
closeErr := held.session.Close()
if execErr != nil {
return fmt.Errorf("take the engine out of backup mode: %w", errors.Join(execErr, closeErr))
}
if closeErr != nil {
return fmt.Errorf("close the backup session: %w", closeErr)
}
slog.Info("rds-agent: engine released from backup mode", "label", held.label)
return nil
return held
}
// The deadline. Ending the session is enough — the engine aborts the backup it
// was holding — so this never has to reach the engine itself.
func (e *postgresEngine) expireQuiesce(label string) {
e.mu.Lock()
held := e.held
// The deadline. Ending the session is enough — both engines release the hold
// with it — so this never has to reach the engine itself.
func (q *quiesceState) expire(label string) {
q.mu.Lock()
held := q.held
if held == nil || held.label != label {
e.mu.Unlock()
q.mu.Unlock()
return
}
e.held = nil
e.mu.Unlock()
q.held = nil
q.mu.Unlock()
slog.Warn("rds-agent: backup hold expired; releasing the engine", "label", label)
if err := held.session.Close(); err != nil {
@@ -141,21 +110,27 @@ func (e *postgresEngine) expireQuiesce(label string) {
}
}
// A psql child with its stdin held open, so statements can be fed to it one at a
// time and the engine keeps seeing a single session.
type psqlSession struct {
// A client child with its stdin held open, so statements can be fed to it one at
// a time and the engine keeps seeing a single session.
type clientSession struct {
cmd *exec.Cmd
stdin io.WriteCloser
stdout *bufio.Reader
stderr *bytes.Buffer
// The statement that makes this client print the sentinel, which is the only
// part of a held session that differs between engines.
sentinel string
mu sync.Mutex
closed bool
}
var _ engineSession = (*psqlSession)(nil)
var _ engineSession = (*clientSession)(nil)
func execSessionRunner(ctx context.Context, c command) (engineSession, error) {
if c.SentinelStatement == "" {
return nil, fmt.Errorf("%s was started as a session with no sentinel statement", c.Name)
}
cmd := exec.CommandContext(ctx, c.Name, c.Args...)
cmd.Env = c.Env
if c.User != "" {
@@ -180,24 +155,30 @@ func execSessionRunner(ctx context.Context, c command) (engineSession, error) {
if err := cmd.Start(); err != nil {
return nil, fmt.Errorf("start %s: %w", c.Name, err)
}
return &psqlSession{cmd: cmd, stdin: stdin, stdout: bufio.NewReader(stdout), stderr: &stderr}, nil
return &clientSession{
cmd: cmd,
stdin: stdin,
stdout: bufio.NewReader(stdout),
stderr: &stderr,
sentinel: c.SentinelStatement,
}, nil
}
// Feeds sql and waits for the sentinel that follows it. ON_ERROR_STOP makes psql
// exit on a failed statement, so a sentinel that never arrives is the failure
// itself and the engine's own message is on stderr.
func (s *psqlSession) Exec(ctx context.Context, sql string) error {
// Feeds sql and waits for the sentinel that follows it. Both clients are run in
// a mode that exits on a failed statement, so a sentinel that never arrives is
// the failure itself and the engine's own message is on stderr.
func (s *clientSession) Exec(ctx context.Context, sql string) error {
s.mu.Lock()
defer s.mu.Unlock()
if s.closed {
return errors.New("the backup session is closed")
}
if _, err := io.WriteString(s.stdin, sql+"\\echo "+sessionSentinel+"\n"); err != nil {
if _, err := io.WriteString(s.stdin, sql+s.sentinel); err != nil {
return fmt.Errorf("%w: %s", err, strings.TrimSpace(s.stderr.String()))
}
// Read on its own goroutine so a psql that never answers costs the caller its
// deadline rather than blocking it forever.
// Read on its own goroutine so a client that never answers costs the caller
// its deadline rather than blocking it forever.
done := make(chan error, 1)
go func() { done <- s.awaitSentinel() }()
select {
@@ -208,7 +189,7 @@ func (s *psqlSession) Exec(ctx context.Context, sql string) error {
}
}
func (s *psqlSession) awaitSentinel() error {
func (s *clientSession) awaitSentinel() error {
for {
line, err := s.stdout.ReadString('\n')
if strings.TrimSpace(line) == sessionSentinel {
@@ -226,7 +207,7 @@ func (s *psqlSession) awaitSentinel() error {
// Closing stdin ends the script, and the wait reaps the child. A session that
// will not end is killed, because leaving it is what would keep the engine in
// backup mode.
func (s *psqlSession) Close() error {
func (s *clientSession) Close() error {
s.mu.Lock()
defer s.mu.Unlock()
if s.closed {
-76
View File
@@ -2,7 +2,6 @@ package main
import (
"context"
"errors"
"slices"
"strings"
"sync"
@@ -113,81 +112,6 @@ func TestPostgresEngine_UnquiesceStopsTheBackupAndEndsTheSession(t *testing.T) {
}
}
// A hold that expired mid-snapshot means the snapshot was not taken against a
// held checkpoint, so the release reports it rather than succeeding silently.
func TestPostgresEngine_UnquiesceWithoutAHoldIsAnError(t *testing.T) {
engine, _ := newQuiesceEngine(t, &fakeSession{})
err := engine.Unquiesce(context.Background())
if err == nil {
t.Fatal("Unquiesce succeeded with no hold to release")
}
if !strings.Contains(err.Error(), "expired") {
t.Errorf("error %q does not say the hold had expired", err)
}
}
// The engine aborts the backup when the session ends, which is what makes the
// hold self-expiring: a control plane that dies cannot leave it held forever.
func TestPostgresEngine_QuiesceReleasesItselfOnItsDeadline(t *testing.T) {
session := &fakeSession{}
engine, _ := newQuiesceEngine(t, session)
if err := engine.Quiesce(context.Background(), "orders-db-pre-upgrade", 10*time.Millisecond); err != nil {
t.Fatalf("Quiesce: %v", err)
}
deadline := time.Now().Add(2 * time.Second)
for session.closes() == 0 && time.Now().Before(deadline) {
time.Sleep(time.Millisecond)
}
if session.closes() != 1 {
t.Fatal("the backup session outlived its deadline")
}
// The hold is gone with it, so a later release is honest about that.
if err := engine.Unquiesce(context.Background()); err == nil {
t.Error("Unquiesce succeeded after the hold expired")
}
}
// One backup at a time: a second start would open a concurrent backup alongside
// the first rather than extending it.
func TestPostgresEngine_QuiesceRefusesASecondHold(t *testing.T) {
session := &fakeSession{}
engine, started := newQuiesceEngine(t, session)
if err := engine.Quiesce(context.Background(), "first", time.Minute); err != nil {
t.Fatalf("Quiesce: %v", err)
}
err := engine.Quiesce(context.Background(), "second", time.Minute)
if err == nil {
t.Fatal("a second quiesce was accepted while the first was held")
}
if !strings.Contains(err.Error(), "first") {
t.Errorf("error %q does not name the backup already holding the engine", err)
}
if len(*started) != 1 {
t.Errorf("started %d sessions, want the second refused before opening one", len(*started))
}
}
// A failed start leaves nothing held, so a retry is not refused by a hold that
// never took effect.
func TestPostgresEngine_QuiesceEndsTheSessionWhenTheBackupWillNotStart(t *testing.T) {
session := &fakeSession{execErr: errors.New(`ERROR: a backup is already in progress`)}
engine, _ := newQuiesceEngine(t, session)
if err := engine.Quiesce(context.Background(), "orders-db-pre-upgrade", time.Minute); err == nil {
t.Fatal("Quiesce succeeded against an engine that refused the backup")
}
if session.closes() != 1 {
t.Errorf("closed %d times, want the failed session ended", session.closes())
}
if engine.held != nil {
t.Error("a failed quiesce left a hold behind")
}
}
func TestCommandRegistry_QuiesceCarriesTheLabelAndDeadline(t *testing.T) {
engine := &fakeEngine{}
reply := newCommander(nil, newCommandRegistry(engine, &fakeStorage{}), 0).execute(context.Background(),
-2
View File
@@ -28,8 +28,6 @@ const (
defaultXFSGrowfs = "xfs_growfs"
defaultMountsFile = "/proc/mounts"
defaultSysBlock = "/sys/class/block"
// Where rds-datadir mounts the data volume; the datadir sits one level in.
defaultDataMount = "/var/lib/postgresql"
// growpart says this when the partition already fills the disk, which a
// resumed grow is the normal way to reach.
growpartNoChange = "NOCHANGE"
+7 -4
View File
@@ -61,12 +61,12 @@ func newTestStorage(t *testing.T, device, fstype string, run commandRunner) *gue
mounts := filepath.Join(dir, "mounts")
content := "proc /proc proc rw 0 0\n"
if device != "" {
content += device + " " + defaultDataMount + " " + fstype + " rw,relatime 0 0\n"
content += device + " " + testDataMount + " " + fstype + " rw,relatime 0 0\n"
}
if err := os.WriteFile(mounts, []byte(content), 0o600); err != nil {
t.Fatalf("write the mount table: %v", err)
}
cfg := config{DataMount: defaultDataMount, MountsFile: mounts, SysBlock: filepath.Join(dir, "block")}
cfg := config{DataMount: testDataMount, MountsFile: mounts, SysBlock: filepath.Join(dir, "block")}
if err := os.MkdirAll(cfg.SysBlock, 0o755); err != nil {
t.Fatalf("create the sysfs tree: %v", err)
}
@@ -93,6 +93,9 @@ func seedPartition(t *testing.T, g *guestStorage, disk, name, number string) {
// The data volume is formatted whole, so the usual grow is resize2fs against
// the device with no partition to push out first.
// Where rds-datadir mounts the data volume in the PostgreSQL preset.
const testDataMount = "/var/lib/postgresql"
func TestGrowFilesystem_ResizesAnExt4DeviceOnline(t *testing.T) {
log := &runLog{}
g := newTestStorage(t, "/dev/vdb", "ext4", log.run)
@@ -108,7 +111,7 @@ func TestGrowFilesystem_ResizesAnExt4DeviceOnline(t *testing.T) {
if args := log.calls[0].Args; len(args) != 1 || args[0] != "/dev/vdb" {
t.Errorf("resize2fs args = %v, want the device", args)
}
if !strings.Contains(message, "/dev/vdb") || !strings.Contains(message, defaultDataMount) {
if !strings.Contains(message, "/dev/vdb") || !strings.Contains(message, testDataMount) {
t.Errorf("reply message = %q, want the device and mount it grew", message)
}
}
@@ -125,7 +128,7 @@ func TestGrowFilesystem_GrowsXFSByMountPoint(t *testing.T) {
if got := log.names(); len(got) != 1 || got[0] != defaultXFSGrowfs {
t.Fatalf("ran %v, want just %s", got, defaultXFSGrowfs)
}
if args := log.calls[0].Args; len(args) != 1 || args[0] != defaultDataMount {
if args := log.calls[0].Args; len(args) != 1 || args[0] != testDataMount {
t.Errorf("xfs_growfs args = %v, want the mount point", args)
}
}
+37 -15
View File
@@ -845,16 +845,35 @@ Spinifex both stores externally-issued certificates (`import-certificate`) and i
---
## RDS (PostgreSQL)
## RDS (PostgreSQL and MariaDB)
Each DB instance is one dedicated system-owned VM running the engine directly, launched from the `spinifex-rds-postgres` AMI, tagged `spinifex:managed-by=rds` and therefore hidden from the customer's EC2 API. The engine is reached over a customer-account ENI injected into a subnet of the DB subnet group, so **the endpoint is private — reachable from inside the VPC only**, from any subnet of it rather than the endpoint ENI's own. `Endpoint.Address` is `{db-instance-identifier}.{account-id}.{region}.rds.{base-domain}` where northstar is configured, and the endpoint ENI's private IP where it is not; the IP is stable across VM replacement either way. Default port 5432.
Each DB instance is one dedicated system-owned VM running the engine directly, launched from the `spinifex-rds-postgres` or `spinifex-rds-mariadb` AMI, tagged `spinifex:managed-by=rds` and therefore hidden from the customer's EC2 API. The engine is reached over a customer-account ENI injected into a subnet of the DB subnet group, so **the endpoint is private — reachable from inside the VPC only**, from any subnet of it rather than the endpoint ENI's own. `Endpoint.Address` is `{db-instance-identifier}.{account-id}.{region}.rds.{base-domain}` where northstar is configured, and the endpoint ENI's private IP where it is not; the IP is stable across VM replacement either way.
- **Engine:** `postgres` 18 only — `EngineVersion` accepts `18` or `18.x`, and there is no in-place upgrade.
Everything below applies to both engines unless it names one. `Engine` is fixed at create — there is no in-place engine change, no cross-engine snapshot restore and no migration between the two.
| | `postgres` | `mariadb` |
|---|---|---|
| Version | 18 | 11.8 |
| `EngineVersion` accepted | `18`, or omitted | `11.8`, or omitted |
| Default port | 5432 | 3306 |
| Parameter-group family | `postgres18` | `mariadb11.8` |
| Implicit default group | `default.postgres18` | `default.mariadb11.8` |
| `DBName` maximum | 63 characters | 64 characters |
| `MasterUsername` maximum | 63 characters | 80 characters |
| Reserved usernames | `postgres`, `rdsadmin`, `rds_superuser`, and the `pg_` prefix | `root`, `mysql`, `mariadb.sys`, `rdsadmin`, `PUBLIC`, and the `mysql.` and `mariadb.` prefixes |
| Catalog size | 51 parameters | 54 parameters |
**`mysql` is not an accepted engine and is not an alias for `mariadb`.** MariaDB is offered under its own AWS engine name, exactly as AWS RDS offers it. Naming `mysql` fails with `InvalidParameterValue`; a client — including Terraform's `aws_db_instance` — must set `engine = "mariadb"`. Aliasing would report an engine and a version the instance is not running, and the discrepancy would propagate into `DescribeDBInstances`, parameter-group families and snapshot metadata.
- **Engine version:** pinned per engine, and an `EngineVersion` naming anything but the pin is rejected. A minor version such as `18.4` or `11.8.8` is rejected as well: the AMI does not promise any particular minor, so accepting one would be a promise the platform cannot keep. There is no in-place upgrade.
- **Instance classes:** `db.t3.{micro,small,medium,large}` and `db.m5.{large,xlarge}` — a naming facade over the platform's EC2 sizing table. Any other class is rejected at create.
- **Storage:** gp3 only, 20–65536 GiB, always encrypted with the cluster key. Grow-only, and a grow is **stop/start with downtime** — the volume cannot be resized while attached.
- **TLS:** offered, not enforced. The engine serves a per-instance certificate signed by the cluster CA (the `ca.pem` baked into AMIs) carrying both the ENI IP and the DNS name in its SAN set, so `sslmode=verify-full` works by name or by address. A deployment with no cluster CA starts the engine without TLS.
- **Master user:** administrative but **not a PostgreSQL superuser**, as on AWS. It gets `CREATEDB` and `CREATEROLE`, owns the initial database, and holds `rds_superuser` (`pg_monitor`, `pg_signal_backend`, `pg_checkpoint`) with `ADMIN OPTION`, so it can grant that set onward. It cannot `COPY ... FROM/TO PROGRAM`, touch server-side files, run `ALTER SYSTEM`, or install **untrusted** extensions — those need the cluster superuser, which no customer credential is. Trusted extensions install normally in a database it owns. `postgres`, `rdsadmin` and `rds_superuser` are rejected as `--master-username`.
- **Backups:** daily COW snapshots of the data volume inside `PreferredBackupWindow`. Retention defaults to 7 days and caps at 7; `0` disables automated backups. No point-in-time recovery.
- **`DBName`:** optional; an omitted name creates no initial database. When supplied it must begin with a letter and hold only letters, digits and underscores, within the per-engine limit above. The rule is narrower than either engine's own, because the guest interpolates the name into a `CREATE DATABASE`.
- **TLS:** offered, not enforced, on both engines. The engine serves a per-instance certificate signed by the cluster CA (the `ca.pem` baked into AMIs) carrying both the ENI IP and the DNS name in its SAN set, so PostgreSQL's `sslmode=verify-full` verifies by name or by address, and MariaDB's `--ssl-ca` plus `--ssl-verify-server-cert` verifies against the endpoint name. A deployment with no cluster CA starts the engine without TLS.
- **PostgreSQL master user:** administrative but **not a PostgreSQL superuser**, as on AWS. It gets `CREATEDB` and `CREATEROLE`, owns the initial database, and holds `rds_superuser` (`pg_monitor`, `pg_signal_backend`, `pg_checkpoint`) with `ADMIN OPTION`, so it can grant that set onward. It cannot `COPY ... FROM/TO PROGRAM`, touch server-side files, run `ALTER SYSTEM`, or install **untrusted** extensions — those need the cluster superuser, which no customer credential is. Trusted extensions install normally in a database it owns.
- **MariaDB master user:** created as `'<name>'@'%'` with global `RELOAD`, `PROCESS`, `SHOW DATABASES`, `REPLICATION SLAVE` and `REPLICATION CLIENT`, while application DML, DDL, routines, events, triggers and delegation are granted only on customer databases. MariaDB has no deny or partial-revoke mechanism, so granting those object privileges `ON *.*` would also grant them on `mysql.global_priv` and let the master acquire `SUPER`, `FILE` or `SHUTDOWN`. The initial `DBName` is granted automatically. Create another customer database with `CALL _spinifex_rds.create_database('name')`; plain `CREATE DATABASE`, arbitrary account administration and access to `mysql.*` are intentionally unavailable on the stock MariaDB build.
- **MariaDB storage engine:** `default_storage_engine` is platform-owned and pinned to **InnoDB**, and is absent from the parameter catalog — a deliberate divergence from AWS, which exposes it as modifiable. Only InnoDB keeps a redo log, so only InnoDB is covered by the snapshot guarantee. An explicit `ENGINE=Aria` or `ENGINE=MyISAM` on `CREATE TABLE` still works — the limitation is documented rather than prevented — but such a table has no redo log, so a snapshot taken without a successful quiesce can leave it inconsistent with no way back. Use InnoDB, as AWS also recommends.
- **Backups:** daily COW snapshots of the data volume inside `PreferredBackupWindow`. Retention defaults to 7 days and caps at 7; `0` disables automated backups. No point-in-time recovery, and MariaDB's binary log is off.
- **Availability:** single-AZ. Engine crashes restart in-guest and VM crashes on a live host restart via `ec2-health-restart`, but an instance whose host is lost is reported `failed` with a reason in `StatusInfos` and needs operator recovery.
Statuses: `creating`, `available`, `modifying`, `backing-up`, `rebooting`, `stopping`, `stopped`, `starting`, `deleting`, `failed`. (`recovering` is defined in the state machine but unreachable until auto-recovery lands.)
@@ -863,7 +882,7 @@ Statuses: `creating`, `available`, `modifying`, `backing-up`, `rebooting`, `stop
| Command | Implemented Flags | Missing Flags | Status |
|---------|-------------------|---------------|--------|
| `create-db-instance` | `--db-instance-identifier`, `--engine` (postgres), `--engine-version` (18), `--db-instance-class`, `--allocated-storage` (20–65536 GiB), `--storage-type` (gp3), `--master-username`, `--master-user-password`, `--db-name`, `--port` (1150–65535), `--db-subnet-group-name` (unnamed → the account's default VPC subnet), `--vpc-security-group-ids`, `--db-parameter-group-name`, `--backup-retention-period` (0–7, default 7), `--preferred-backup-window`, `--preferred-maintenance-window` (unnamed → assigned from the identifier), `--deletion-protection`, `--tags`, `--storage-encrypted` (true only) | `--auto-minor-version-upgrade`, `--copy-tags-to-snapshot`, Performance Insights / Enhanced Monitoring flags (all accepted, no-op); see "Rejected Parameters" for those that fail loudly | **DONE** |
| `create-db-instance` | `--db-instance-identifier`, `--engine` (postgres, mariadb), `--engine-version` (the engine's pin, or omitted), `--db-instance-class`, `--allocated-storage` (20–65536 GiB), `--storage-type` (gp3), `--master-username`, `--master-user-password`, `--db-name` (letters, digits and underscores, beginning with a letter, max 63 on postgres and 64 on mariadb), `--port` (1150–65535), `--db-subnet-group-name` (unnamed → the account's default VPC subnet), `--vpc-security-group-ids`, `--db-parameter-group-name`, `--backup-retention-period` (0–7, default 7), `--preferred-backup-window`, `--preferred-maintenance-window` (unnamed → assigned from the identifier), `--deletion-protection`, `--tags`, `--storage-encrypted` (true only) | `--auto-minor-version-upgrade`, `--copy-tags-to-snapshot`, Performance Insights / Enhanced Monitoring flags (all accepted, no-op); see "Rejected Parameters" for those that fail loudly | **DONE** |
| `describe-db-instances` | `--db-instance-identifier` | `--filters`, `--max-records`, `--marker` (parsed, not applied) | **DONE** |
| `modify-db-instance` | `--db-instance-identifier`, `--master-user-password`, `--allocated-storage` (grow only, stop/start), `--db-instance-class` (VM replace), `--db-parameter-group-name`, `--vpc-security-group-ids`, `--deletion-protection`, `--backup-retention-period`, `--preferred-backup-window`, `--preferred-maintenance-window`, `--apply-immediately` | `--new-db-instance-identifier`, `--engine-version`, `--db-port-number`, `--db-subnet-group-name`, `--max-allocated-storage`, `--ca-certificate-identifier` (all rejected, not ignored) | **DONE** |
| `delete-db-instance` | `--db-instance-identifier`, `--skip-final-snapshot`, `--final-db-snapshot-identifier` (exactly one is required) | `--delete-automated-backups` (no-op — automated snapshots are always purged with the instance) | **DONE** — `DeletionProtection` blocks the call; a final snapshot **retains** the data volume until that snapshot is deleted |
@@ -875,7 +894,7 @@ Statuses: `creating`, `available`, `modifying`, `backing-up`, `rebooting`, `stop
| Command | Implemented Flags | Missing Flags | Status |
|---------|-------------------|---------------|--------|
| `create-db-snapshot` | `--db-snapshot-identifier`, `--db-instance-identifier`, `--tags` | — | **DONE** — the agent quiesces the engine first; a failed quiesce falls back to a crash-consistent snapshot and writes a `DescribeEvents` warning |
| `create-db-snapshot` | `--db-snapshot-identifier`, `--db-instance-identifier`, `--tags` | — | **DONE** — the agent quiesces the engine first (`pg_backup_start` on postgres, `BACKUP STAGE` on mariadb); a failed quiesce falls back to a crash-consistent snapshot and writes a `DescribeEvents` warning naming what that engine does and does not recover |
| `describe-db-snapshots` | `--db-snapshot-identifier`, `--db-instance-identifier`, `--snapshot-type` (manual, automated) | `--filters`, `--dbi-resource-id`, `--include-shared`, `--include-public` (rejected), `--max-records`, `--marker` | **DONE** |
| `delete-db-snapshot` | `--db-snapshot-identifier` | — | **DONE** — refused while a restored volume still references the snapshot; the last snapshot released reclaims a retained data volume |
| `restore-db-instance-from-db-snapshot` | `--db-instance-identifier`, `--db-snapshot-identifier`, `--db-instance-class`, `--allocated-storage` (≥ the snapshot's), `--storage-type` (gp3), `--port`, `--db-subnet-group-name`, `--vpc-security-group-ids`, `--db-parameter-group-name`, `--deletion-protection`, `--tags`, `--engine` (must match the snapshot) | `--db-name` (rejected when it differs from the snapshot's), plus the create rejections | **DONE** — unnamed fields are inherited from the snapshot; the master password comes from the restored datadir |
@@ -903,15 +922,17 @@ Statuses: `creating`, `available`, `modifying`, `backing-up`, `rebooting`, `stop
| Command | Implemented Flags | Missing Flags | Status |
|---------|-------------------|---------------|--------|
| `create-db-parameter-group` | `--db-parameter-group-name`, `--db-parameter-group-family` (postgres18), `--description`, `--tags` | — | **DONE** — names beginning with `default.` are reserved; the implicit group is `default.postgres18` |
| `describe-db-parameter-groups` | `--db-parameter-group-name` | `--filters`, `--max-records`, `--marker` | **DONE** — the implicit default group is always listed |
| `create-db-parameter-group` | `--db-parameter-group-name`, `--db-parameter-group-family` (`postgres18` or `mariadb11.8`), `--description`, `--tags` | — | **DONE** — names beginning with `default.` are reserved. **`--db-parameter-group-family` defaults to `postgres18` when omitted**, so a MariaDB group must name its family explicitly or the mismatch surfaces one call later, at `create-db-instance` |
| `describe-db-parameter-groups` | `--db-parameter-group-name` | `--filters`, `--max-records`, `--marker` | **DONE** — every engine's implicit default group is always listed, so both `default.postgres18` and `default.mariadb11.8` appear in every account |
| `modify-db-parameter-group` | `--db-parameter-group-name`, `--parameters` (ParameterName, ParameterValue, ApplyMethod) | — | **DONE** — the whole batch is validated before anything is written and propagated to attached instances; `immediate` on a static parameter is rejected, as AWS does |
| `describe-db-parameters` | `--db-parameter-group-name`, `--source` (user, engine-default) | `--filters`, `--max-records`, `--marker` | **DONE** — 51-parameter PostgreSQL 18 catalog; memory defaults are computed per instance class and reported as literals |
| `describe-db-parameters` | `--db-parameter-group-name`, `--source` (user, engine-default) | `--filters`, `--max-records`, `--marker` | **DONE** — the catalog of the group's own family: 51 parameters on `postgres18`, 54 on `mariadb11.8`. Memory defaults are computed per instance class and reported as literals |
| `delete-db-parameter-group` | `--db-parameter-group-name` | — | **DONE** — refused for a default group and while any instance references it |
| `reset-db-parameter-group` | — | all | **NOT STARTED** (`InvalidAction`) |
A group takes effect on an instance when `modify-db-instance --db-parameter-group-name` attaches it — immediately with `--apply-immediately`, otherwise at the next maintenance window. Later group edits propagate to every attached instance. Dynamic parameters are written into the engine's config and reloaded live; static ones are recorded `pending-reboot` and applied by `reboot-db-instance`.
**A group's family must match the instance's engine.** A `postgres18` group cannot be attached to a MariaDB instance and a `mariadb11.8` group cannot be attached to a PostgreSQL one, on create, on modify, on a deferred apply or on a snapshot restore; the mismatch is refused with `InvalidParameterCombination`. The catalogs are per-family too, so `modify-db-parameter-group` will not store one engine's parameter name into the other's group. Each catalog is a curated, validated subset rather than the engine's full variable set, and the platform-owned settings — the port, the datadir, the socket, the bind address, the `ssl_*` family, and on MariaDB `secure_file_priv`, `log_bin`, `default_storage_engine` and the InnoDB buffer-pool chunk settings — are absent from it rather than present-and-unmodifiable, because they are not the customer's to set.
### RDS — Tags
| Command | Implemented Flags | Missing Flags | Status |
@@ -942,7 +963,8 @@ Policy: a parameter whose omission would create a false safety, security or avai
| `DBSecurityGroups` | EC2-Classic security groups — use `VpcSecurityGroupIds` |
| `DBClusterIdentifier`, `DBClusterSnapshotIdentifier` | Clustered engines are not offered |
| `EnableCloudwatchLogsExports` | Log export is not implemented |
| `EngineVersion` other than 18, `Engine` on modify | No in-place engine or version change |
| `EngineVersion` other than the engine's pin, `Engine` on modify | No in-place engine or version change |
| `Engine=mysql` (and Aurora engines) | Oracle MySQL is not offered; `mariadb` is a distinct engine, not an alias for it |
| `NewDBInstanceIdentifier` | The identifier is the DNS label and the KV key |
| `DBPortNumber`, `DBSubnetGroupName` on modify | Both would move the endpoint |
| `MaxAllocatedStorage` | Storage autoscaling is not implemented |
@@ -961,14 +983,14 @@ Recognised actions below return `OperationNotSupported`, so a client sees "not o
| Feature | Actions | Priority | Status |
|---------|---------|----------|--------|
| MySQL engine | — (second AMI preset + `Engine` implementation) | High | **NOT STARTED** |
| Point-in-time recovery (WAL archiving to predastore) | `restore-db-instance-to-point-in-time` | High | **NOT STARTED** |
| Oracle MySQL engine | — | Low | **NOT PLANNED** — MariaDB is offered instead, under its own `mariadb` engine name; see the note above on why it is not aliased onto `mysql` |
| Point-in-time recovery (WAL archiving to predastore on postgres, binlog on mariadb) | `restore-db-instance-to-point-in-time` | High | **NOT STARTED** |
| Auto-recovery from node loss (failure is detected, not repaired) | — | High | **NOT STARTED** |
| Read replicas | `create-db-instance-read-replica`, `promote-read-replica` | Medium | **NOT STARTED** |
| Aurora / DB clusters | `create-db-cluster`, `modify-db-cluster`, `delete-db-cluster`, `describe-db-clusters`, `failover-db-cluster` | Low | **NOT STARTED** |
| Option groups | `create-option-group`, `modify-option-group`, `delete-option-group`, `describe-option-groups` | Low | **NOT STARTED** |
| Engine-version / orderable-option data sources | `describe-db-engine-versions`, `describe-orderable-db-instance-options` | Low | **NOT STARTED** (`InvalidAction`) — not required by `aws_db_instance`; pin the class and version literally in Terraform |
| Multi-AZ standby, online (no-downtime) storage grow, storage autoscaling, IAM database auth, Performance Insights, Enhanced Monitoring, log exports, per-tenant private DNS zones, enforced TLS | — | — | **NOT STARTED** |
| Multi-AZ standby, online (no-downtime) storage grow, storage autoscaling, IAM database auth, Performance Insights, Enhanced Monitoring, log exports, per-tenant private DNS zones, enforced TLS, cross-engine snapshot restore, in-place migration between the two engines | — | — | **NOT STARTED** |
IAM: `AmazonRDSFullAccess` and `AmazonRDSReadOnlyAccess` are available as managed policies. They grant `rds:` verb prefixes rather than `rds:*`, because `rds:*` would also appear to grant the internal agent actions the gateway reserves for a DB VM's own role.
@@ -1,5 +1,5 @@
---
title: "RDS Quickstart"
title: "RDS Quickstart (PostgreSQL)"
description: "Stand up a managed PostgreSQL database on Spinifex with Terraform — a VPC, a DB subnet group, a parameter group, an aws_db_instance, and a client VM inside the VPC that connects to it with psql."
category: "Terraform Workbooks"
tags:
@@ -16,7 +16,7 @@ sections:
- troubleshooting
resources:
- title: "RDS CLI Reference"
url: "https://github.com/mulgadc/spinifex/blob/main/docs/COMMANDS.md#rds-postgresql"
url: "https://github.com/mulgadc/spinifex/blob/main/docs/COMMANDS.md#rds-postgresql-and-mariadb"
- title: "Terraform AWS Provider — aws_db_instance"
url: "https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/db_instance"
- title: "Spinifex Repository"
@@ -25,10 +25,12 @@ resources:
url: "https://opentofu.org/"
---
# Terraform: RDS Quickstart
# Terraform: RDS Quickstart (PostgreSQL)
> The smallest end-to-end RDS example on Spinifex: a VPC with a public client subnet and a private DB subnet, a DB subnet group, a parameter group, a PostgreSQL DB instance, and a client VM with `psql` already pointed at the endpoint.
This workbook is **PostgreSQL throughout** — the engine, the `postgres18` parameter group, port `5432`, the client package and the `~/.pgpass` credential are all PostgreSQL's. Spinifex also serves MariaDB 11.8 under `engine = "mariadb"`; running this workbook against it means changing `engine`, `engine_version`, the parameter-group `family` to `mariadb11.8`, the security-group port to `3306` and the client to `mariadb-client`. See the [RDS command reference](https://github.com/mulgadc/spinifex/blob/main/docs/COMMANDS.md#rds-postgresql-and-mariadb) for both engines' pins, families and limits. Note that `mysql` is not an accepted engine and is not an alias for `mariadb`.
## Overview
A Spinifex DB instance runs in a VM you never see. What lands in your VPC is a single **endpoint ENI** in one of the DB subnet group's subnets, so the endpoint is **always private** — there is no `publicly_accessible` mode, and a request for one is rejected. That is why this workbook builds a client VM: it is the only place from which the database can be reached.
@@ -127,7 +129,7 @@ psql "sslmode=verify-full sslrootcert=/home/ubuntu/ca.pem" \
| `spinifex_endpoint` | `https://127.0.0.1:9999` | Gateway as seen from the host running Terraform. |
| `instance_type` | `t3.small` | EC2 type for the **client** VM. |
| `db_instance_class` | `db.t3.micro` | DB class. One of the curated `db.*` subset. |
| `engine_version` | `18` | PostgreSQL major. `18` is the only version served. |
| `engine_version` | `18` | PostgreSQL major. `18` is the only version served, and a minor such as `18.4` is rejected. |
| `db_name` | `appdb` | Database created at bootstrap. |
| `db_username` | `appuser` | Master user. `postgres`, `rdsadmin`, `rds_superuser` and `pg_*` are reserved. |
| `db_password` | `QuickstartS3cret1` | Master password — override it. No `/`, `"`, `@` or spaces. |
+1
View File
@@ -55,6 +55,7 @@ Each subdirectory holds a `manifest.conf`, init scripts, and a `setup.sh` that `
| `eks-node/` | The unified Alpine EKS node AMI (K3s server *or* agent, selected at first boot). `setup.sh` installs the pinned, SHA-verified K3s binary and the role init scripts. |
| `eks-node-gpu/` | The Ubuntu EKS GPU worker AMI. Agent role only — GPU nodes never run the control plane — plus the NVIDIA GPU stack. |
| `rds-postgres/` | The `spinifex-rds-postgres` AMI. `setup.sh` sets exec bits on the OpenRC services and creates the root-only directory cloud-init drops the agent env file and gateway CA into. |
| `rds-mariadb/` | The `spinifex-rds-mariadb` AMI. Same layout as `rds-postgres/`, plus the `/etc/my.cnf.d` include drop-in and the `conf.d/mariadb` dependency that keeps the packaged service off an unbootstrapped datadir. |
| `ubuntu-gpu-{amd,nvidia}-setup.sh` | Chroot setup for GPU-capable Ubuntu guest images: ROCm CLI tools and AMD firmware, or the DKMS-built headless NVIDIA driver. Guest-side only — host VFIO setup is `spx admin gpu setup`. |
### `images/eks-node/` — in-guest helpers
@@ -1,10 +1,14 @@
#!/bin/sh
# rds-datadir — select, authorize, prepare, and mount the exact data volume
# assigned by the control plane before rds-init touches the PostgreSQL datadir.
# assigned by the control plane before rds-init touches the engine's datadir.
#
# Shared by every RDS image preset. The mount point and the OS user that owns it
# are the one engine-specific pair here, and both are supplied by the preset's
# own service rather than defaulted, so no image can inherit another's layout.
set -eu
DATA_MOUNT="${RDS_DATA_MOUNT:-/var/lib/postgresql}"
PG_USER="${RDS_PG_USER:-postgres}"
DATA_MOUNT="${RDS_DATA_MOUNT:-}"
ENGINE_USER="${RDS_ENGINE_USER:-}"
SYS_BLOCK="${RDS_SYS_BLOCK:-/sys/block}"
DEV_DIR="${RDS_DEV_DIR:-/dev}"
MOUNTS_FILE="${RDS_MOUNTS_FILE:-/proc/mounts}"
@@ -103,6 +107,14 @@ consume_format_authorization() {
}
main() {
# An unset layout is a broken image, not something to guess at: mounting the
# customer's volume where the engine is not looking would read as an empty
# datadir and bootstrap a second, empty database beside their data.
[ -n "${DATA_MOUNT}" ] ||
fail "RDS_DATA_MOUNT is not set; refusing to guess where this engine's datadir belongs"
[ -n "${ENGINE_USER}" ] ||
fail "RDS_ENGINE_USER is not set; refusing to guess which user owns this engine's datadir"
load_handoff
command -v blkid >/dev/null 2>&1 ||
@@ -184,8 +196,8 @@ main() {
mount -t "${_fstype}" "${_dev}" "${DATA_MOUNT}" ||
fail "could not mount ${_dev} at ${DATA_MOUNT}"
chown "${PG_USER}:${PG_USER}" "${DATA_MOUNT}" ||
fail "could not set ${PG_USER} ownership on ${DATA_MOUNT}"
chown "${ENGINE_USER}:${ENGINE_USER}" "${DATA_MOUNT}" ||
fail "could not set ${ENGINE_USER} ownership on ${DATA_MOUNT}"
chmod 0750 "${DATA_MOUNT}"
log "${DATA_MOUNT} ready on ${_dev}"
+63
View File
@@ -0,0 +1,63 @@
#!/bin/sh
set -eu
# rds-datadir is shared by every engine preset and refuses to guess a layout, so
# each preset's service is the only statement of where its volume mounts and who
# owns it. A preset that stopped exporting them would fail closed at boot; a
# preset that exported another engine's would mount the customer's volume where
# its engine is not looking, which reads as an empty datadir.
SCRIPT_DIR=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd)
IMAGES=$(CDPATH='' cd -- "${SCRIPT_DIR}/.." && pwd)
WORK=$(mktemp -d)
trap 'rm -rf "${WORK}"' EXIT
FAILS=0
fail() { echo "FAIL: $*" >&2; FAILS=$((FAILS + 1)); }
pass() { echo "ok: $*"; }
# Sourced rather than executed: start_pre is what OpenRC runs, and the values it
# exports are what reach the mount script.
layout() {
(
# shellcheck disable=SC1090
. "${IMAGES}/$1/rds-datadir.initd"
# Read by the sourced start_pre, which shellcheck cannot see.
export AGENT_ENV="$2"
start_pre
printf '%s %s\n' "${RDS_DATA_MOUNT:-}" "${RDS_ENGINE_USER:-}"
)
}
check_preset() {
_preset=$1
_mount=$2
_user=$3
_got=$(layout "${_preset}" "${WORK}/absent.env")
if [ "${_got}" = "${_mount} ${_user}" ]; then
pass "${_preset}: exports ${_mount} owned by ${_user}"
else
fail "${_preset}: exports '${_got}', want '${_mount} ${_user}'"
fi
# The control plane can move the mount point without a rebuild, so a
# delivered value has to win over the image's own default.
printf "RDS_DATA_MOUNT=%s\n" "${WORK}/delivered" > "${WORK}/agent.env"
_got=$(layout "${_preset}" "${WORK}/agent.env")
if [ "${_got}" = "${WORK}/delivered ${_user}" ]; then
pass "${_preset}: a delivered mount point wins"
else
fail "${_preset}: delivered mount point ignored, got '${_got}'"
fi
}
check_preset rds-postgres /var/lib/postgresql postgres
check_preset rds-mariadb /var/lib/mysql mysql
if [ "${FAILS}" -eq 0 ]; then
echo "rds-datadir.initd: all presets state their layout"
exit 0
fi
echo "FAILED: ${FAILS} case(s)"
exit 1
@@ -56,6 +56,7 @@ fail() { echo "FAIL: $*"; FAILS=$((FAILS + 1)); }
pass() { echo "ok: $*"; }
DATA_MOUNT="${WORK}/data"
ENGINE_USER="dbengine"
SYS_BLOCK="${WORK}/sys/block"
DEV_DIR="${WORK}/dev"
MOUNTS="${WORK}/mounts"
@@ -91,6 +92,8 @@ EOF
}
reset_state() {
DATA_MOUNT="${WORK}/data"
ENGINE_USER="dbengine"
rm -rf "${WORK}/sys" "${DEV_DIR}" "${DATA_MOUNT}" "$(dirname "${HANDOFF}")"
mkdir -p "${SYS_BLOCK}" "${DEV_DIR}" "${DATA_MOUNT}"
printf '/dev/vda1 / ext4 rw,relatime 0 0\n' > "${MOUNTS}"
@@ -104,6 +107,7 @@ reset_state() {
run() {
env RDS_DATA_MOUNT="${DATA_MOUNT}" \
RDS_ENGINE_USER="${ENGINE_USER}" \
RDS_SYS_BLOCK="${SYS_BLOCK}" \
RDS_DEV_DIR="${DEV_DIR}" \
RDS_MOUNTS_FILE="${MOUNTS}" \
@@ -152,6 +156,9 @@ if run_ok "authorized blank exact volume"; then
grep -q "mkfs.ext4 .*${DEV_DIR}/vdc" "${MKFS_CALLS}" && pass "authorized: exact disk formatted" || fail "authorized: exact disk not formatted"
grep -q -- '-m 0' "${MKFS_CALLS}" && pass "authorized: no reserved blocks" || fail "authorized: reserved blocks left on"
grep -q "mount -t ext4 ${DEV_DIR}/vdc ${DATA_MOUNT}" "${MOUNT_CALLS}" && pass "authorized: mounted" || fail "authorized: not mounted"
# The engine reads its own datadir, so the mount is handed to the preset's
# user rather than to whichever engine this script was first written for.
grep -q "chown ${ENGINE_USER}:${ENGINE_USER} ${DATA_MOUNT}" "${CHOWN_CALLS}" && pass "authorized: handed to the engine user" || fail "authorized: not handed to ${ENGINE_USER}"
grep -q '^RDS_MASTER_PASSWORD=' "${HANDOFF}" && pass "authorized: bootstrap material preserved" || fail "authorized: bootstrap material removed"
if grep -q '^RDS_FORMAT_AUTHORIZED=' "${HANDOFF}"; then fail "authorized: grant retained"; else pass "authorized: grant consumed"; fi
fi
@@ -309,6 +316,24 @@ printf '%s %s ext4 rw 0 0\n' "${DEV_DIR}/wrong" "${DATA_MOUNT}" >> "${MOUNTS}"
run_fails "already mounted wrong device" "not expected device"
nothing_formatted "already mounted wrong device"
# An image whose service states no layout must not fall back to another engine's
# paths: a volume mounted where the engine is not looking reads as an empty
# datadir, and the bootstrap that follows serves an empty database beside the
# customer's data. Both refusals come before the handoff is even read.
reset_state
add_disk vdb volexpected
write_handoff vol-expected volexpected 1 true
DATA_MOUNT=""
run_fails "no mount point" "RDS_DATA_MOUNT is not set"
nothing_formatted "no mount point"
reset_state
add_disk vdb volexpected
write_handoff vol-expected volexpected 1 true
ENGINE_USER=""
run_fails "no engine user" "RDS_ENGINE_USER is not set"
nothing_formatted "no engine user"
if [ "${FAILS}" -eq 0 ]; then
echo "PASS: all rds-datadir cases"
exit 0
@@ -0,0 +1,56 @@
[mysqld]
# Resolved parameter group, written by rds-agent.
autocommit = 'on'
character_set_server = 'utf8mb4'
collation_server = 'utf8mb4_general_ci'
connect_timeout = '10'
event_scheduler = 'off'
general_log = 'off'
group_concat_max_len = '1048576'
idle_transaction_timeout = '0'
innodb_adaptive_hash_index = 'off'
innodb_autoinc_lock_mode = '1'
innodb_buffer_pool_size = '805306368'
innodb_file_per_table = 'on'
innodb_flush_log_at_trx_commit = '1'
innodb_flush_method = 'o_direct'
innodb_flush_neighbors = '1'
innodb_io_capacity = '200'
innodb_io_capacity_max = '2000'
innodb_lock_wait_timeout = '50'
innodb_log_file_size = '100663296'
innodb_max_dirty_pages_pct = '90'
innodb_purge_threads = '4'
innodb_read_io_threads = '4'
innodb_stats_persistent = 'on'
innodb_strict_mode = 'on'
innodb_write_io_threads = '4'
interactive_timeout = '28800'
join_buffer_size = '262144'
key_buffer_size = '134217728'
lock_wait_timeout = '86400'
log_output = 'file'
log_queries_not_using_indexes = 'off'
log_warnings = '2'
long_query_time = '10'
max_allowed_packet = '16777216'
max_connect_errors = '100'
max_connections = '85'
max_heap_table_size = '16777216'
max_prepared_stmt_count = '16382'
max_statement_time = '0'
net_read_timeout = '30'
net_write_timeout = '60'
performance_schema = 'off'
read_buffer_size = '131072'
read_rnd_buffer_size = '262144'
slow_query_log = 'off'
sort_buffer_size = '2097152'
sql_mode = 'STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION'
table_definition_cache = '400'
table_open_cache = '2000'
thread_cache_size = '256'
default_time_zone = 'SYSTEM'
tmp_table_size = '16777216'
transaction_isolation = 'repeatable-read'
wait_timeout = '28800'
+59
View File
@@ -0,0 +1,59 @@
IMAGE_NAME="rds-mariadb"
IMAGE_DESCRIPTION="Spinifex RDS MariaDB instance — Alpine + MariaDB 11.8"
DISTRO="alpine"
ALPINE_VERSION="3.24.1"
# Engine binaries and per-boot diagnostics only. The datadir lives on the data
# volume, so the boot volume stays a disposable COW clone.
IMAGE_SIZE="2G"
# Alpine's mariadb package name carries no version, unlike postgresql18, so the
# fuzzy pin is what keeps a rebuild on this series: =~11.8 admits a patch release
# and refuses a series bump, which would otherwise produce an image tagged
# engine-version=11.8 running 11.9. mariadb-common carries my.cnf and the charset
# and error-message data; the client package carries the binaries the agent
# probes and rotates passwords with.
# blkid is a hard requirement — without it rds-datadir refuses to run rather
# than risk formatting a customer's datadir.
APK_PACKAGES="mariadb=~11.8 mariadb-client=~11.8 mariadb-common=~11.8 mariadb-openrc=~11.8 ca-certificates cloud-utils-growpart e2fsprogs-extra xfsprogs blkid openrc"
# rds-agent is the only built binary (static, FIPS boot like the host build).
BUILD_COMMANDS="CGO_ENABLED=0 GOFIPS140=v1.0.0 go build -ldflags \"-s -w\" -o bin/rds-agent ./cmd/rds-agent"
INSTALL_BINARIES="bin/rds-agent:/usr/local/bin/rds-agent"
# mulga-mgmt-net is shared with eks-node: this is a three-NIC system VM (system
# ENI, customer ENI, mgmt0), so the host's fw_cfg netcfg blob needs a consumer in
# the guest, or mgmt0 never gets its static address and rds-agent cannot reach
# the gateway, and the customer ENI never comes up so nothing reaches the DB.
INSTALL_FILES="\
scripts/images/common/rds-datadir:/usr/local/sbin/rds-datadir \
scripts/images/rds-mariadb/rds-datadir.initd:/etc/init.d/rds-datadir \
scripts/images/rds-mariadb/rds-init:/usr/local/sbin/rds-init \
scripts/images/rds-mariadb/rds-init.initd:/etc/init.d/rds-init \
scripts/images/rds-mariadb/rds-agent.initd:/etc/init.d/rds-agent \
scripts/images/common/mulga-mgmt-net.initd:/etc/init.d/mulga-mgmt-net \
scripts/images/common/mulga-mgmt-net-routes.initd:/etc/init.d/mulga-mgmt-net-routes \
scripts/images/common/mulga-mgmt-net.sh:/usr/local/sbin/mulga-mgmt-net \
scripts/images/rds-mariadb/zz-rds-include.cnf:/etc/my.cnf.d/zz-rds-include.cnf \
scripts/images/rds-mariadb/default-parameters.cnf:/usr/local/share/spinifex-rds/default-parameters.cnf \
scripts/images/rds-mariadb/mariadb.confd:/etc/conf.d/mariadb"
# Each service's own depend() asserts the ordering; this lists it to match.
# mulga-mgmt-net is NOT here: setup.sh adds it to the boot runlevel, since a
# default-runlevel entry would run after cloud-init-local and be too late. Its
# route-enforcement second pass is the opposite — it has to follow cloud-init, so
# the default runlevel is exactly where it belongs.
#
# The packaged mariadb service only warns about an empty datadir instead of
# refusing, so it is not ordering that keeps it off an unbootstrapped volume:
# /etc/conf.d/mariadb makes it need rds-init, which fails closed.
ENABLE_SERVICES="rds-datadir rds-agent rds-init mariadb mulga-mgmt-net-routes"
SETUP_SCRIPT="scripts/images/rds-mariadb/setup.sh"
AMI_NAME="spinifex-rds-mariadb"
# The contract tag prevents the control plane from issuing a format grant to an
# older image that still discovers generic vol* devices.
SYSTEM_TAG="spinifex:managed-by=rds engine=mariadb engine-version=11.8 rds-data-volume-contract=format-auth-v1"
+15
View File
@@ -0,0 +1,15 @@
# /etc/conf.d/mariadb — Spinifex RDS overrides for the stock Alpine mariadb
# service. The package ships no conf.d file of its own, so this adds one rather
# than replacing one.
# The packaged service's start_pre only prints "Datadir is empty or invalid" and
# starts anyway, so an rds-init that failed — an unreachable gateway, a data
# volume that did not attach, a datadir stamped for another engine — would be
# answered by launching mysqld_safe against whatever is mounted. OpenRC reads
# rc_need from here, so the engine depends on the bootstrap without the packaged
# init script being patched.
rc_need="rds-init"
# Everything else the service resolves from the configuration itself: datadir,
# socket and port come from the platform drop-in rds-init writes onto the data
# volume, which the packaged !includedir reads last.
@@ -0,0 +1,77 @@
#!/sbin/openrc-run
# rds-agent — Spinifex RDS in-guest agent. Reaches the control plane over
# TLS+SigV4 (never NATS), signing with IMDS instance-role credentials. Static
# config comes from /etc/spinifex-rds/agent.env, which carries no secrets.
description="Spinifex RDS instance agent"
command="/usr/local/bin/rds-agent"
command_background="yes"
pidfile="/run/rds-agent.pid"
output_log="/var/log/rds-agent.log"
error_log="/var/log/rds-agent.log"
start_stop_daemon_args="--wait 2000"
AGENT_ENV="/etc/spinifex-rds/agent.env"
# The agent retries indefinitely, so this only bounds how long the boot waits
# before letting rds-init fail visibly rather than hanging forever.
handoff_timeout="${RDS_HANDOFF_TIMEOUT:-120}"
# How much of the agent log to mirror to the console when the handoff times out.
# Enough to carry a few backed-off retries with their errors, not the whole boot.
handoff_tail="${RDS_HANDOFF_TAIL:-20}"
depend() {
need net
after localmount
before rds-datadir rds-init mariadb
}
start_pre() {
# An absent file is tolerated, so the agent still starts and logs why it
# cannot reach the gateway.
if [ -f "${AGENT_ENV}" ]; then
# shellcheck disable=SC1090
. "${AGENT_ENV}"
export RDS_GATEWAY_URL RDS_GATEWAY_CA RDS_REGION RDS_DB_INSTANCE_IDENTIFIER \
RDS_ENGINE RDS_ENGINE_VERSION RDS_HANDOFF_DIR RDS_ENGINE_HOST RDS_ENGINE_PORT \
RDS_POLL_WAIT RDS_DATA_MOUNT
fi
}
start_post() {
# Resolve this after start_pre loads agent.env so custom handoff directories
# are shared by the agent, this wait, and rds-init.
HANDOFF_ENV="${RDS_HANDOFF_DIR:-/run/spinifex-rds}/bootstrap.env"
# `before rds-init` only orders the starts; rds-init needs the fetched config
# itself. The agent writes bootstrap.env last and renames it into place, so
# its appearance means the whole handoff is there.
ebegin "Waiting for the RDS bootstrap handoff"
_waited=0
while [ ! -f "${HANDOFF_ENV}" ]; do
if [ "${_waited}" -ge "${handoff_timeout}" ]; then
eend 1 "no bootstrap handoff after ${handoff_timeout}s"
# Mirror the agent's own tail onto the console. The log lives inside a
# guest nothing can reach when this fails — an unreachable gateway is
# the usual cause — so the host-side console capture is the only way
# the dial error survives, and without it the failure reads as silence.
if [ -s "${output_log}" ]; then
einfo "last ${handoff_tail} lines of ${output_log}:"
tail -n "${handoff_tail}" "${output_log}" 2>/dev/null | while IFS= read -r _line; do
einfo " [rds-agent] ${_line}"
done
else
einfo "${output_log} is empty — the agent produced no output at all"
fi
# The agent keeps retrying, so it is left running: rds-init fails on
# the missing handoff and the engine stays down, which is what the
# control plane observes.
return 0
fi
sleep 1
_waited=$((_waited + 1))
done
eend 0
}
+48
View File
@@ -0,0 +1,48 @@
#!/bin/sh
set -eu
SCRIPT_DIR=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd)
CASE=$(mktemp -d)
trap 'rm -rf "${CASE}"' EXIT
source_time_handoff="${CASE}/source-time"
configured_handoff="${CASE}/configured"
mkdir -p "${configured_handoff}"
touch "${configured_handoff}/bootstrap.env"
{
printf 'RDS_HANDOFF_DIR=%s\n' "${configured_handoff}"
printf 'RDS_ENGINE=mariadb\n'
} >"${CASE}/agent.env"
# Simulate OpenRC sourcing the service before start_pre loads agent.env.
export RDS_HANDOFF_DIR="${source_time_handoff}"
# shellcheck source=rds-agent.initd
. "${SCRIPT_DIR}/rds-agent.initd"
export AGENT_ENV="${CASE}/agent.env"
export handoff_timeout=0
EEND_STATUS=
ebegin() { :; }
einfo() { :; }
eend() { EEND_STATUS=$1; }
start_pre
start_post
if [ "${HANDOFF_ENV}" != "${configured_handoff}/bootstrap.env" ]; then
echo "FAIL: handoff path did not use RDS_HANDOFF_DIR from agent.env" >&2
exit 1
fi
if [ "${EEND_STATUS}" != "0" ]; then
echo "FAIL: post-start wait did not find the configured handoff" >&2
exit 1
fi
# The agent checks this against the engine its own image bakes. Left out of the
# export list it would read as no assertion at all, so a VM launched as the
# wrong engine would bootstrap instead of refusing.
if [ "${RDS_ENGINE:-}" != "mariadb" ]; then
echo "FAIL: start_pre did not export RDS_ENGINE from agent.env" >&2
exit 1
fi
echo "rds-agent.initd: all tests passed"
@@ -0,0 +1,35 @@
#!/sbin/openrc-run
# rds-datadir — boot oneshot that discovers the hot-plugged data volume, formats
# it if authorized, and mounts it at the datadir mount point. It runs after the
# agent has written the authenticated handoff and before anything can use the
# MariaDB datadir. Soft ordering keeps a failed boot visible on the console.
description="Spinifex RDS data volume mount"
AGENT_ENV="/etc/spinifex-rds/agent.env"
depend() {
after localmount rds-agent
before rds-init mariadb
}
start_pre() {
# This image's layout, stated here because the mount script is shared by
# every engine preset. A delivered RDS_DATA_MOUNT still wins, so the control
# plane can move the mount point without rebuilding the image.
RDS_DATA_MOUNT="/var/lib/mysql"
RDS_ENGINE_USER="mysql"
if [ -f "${AGENT_ENV}" ]; then
# shellcheck disable=SC1090
. "${AGENT_ENV}"
fi
export RDS_HANDOFF_DIR RDS_DATA_MOUNT RDS_ENGINE_USER
}
start() {
ebegin "Mounting the RDS data volume"
# To the console for the same reason rds-init's is: the serial log is the
# only view of a boot that failed before the engine came up.
/usr/local/sbin/rds-datadir > /dev/console 2>&1
eend $?
}
+62
View File
@@ -0,0 +1,62 @@
#!/bin/sh
set -eu
# The engine this preset bakes, and the series it pins, are stated in places that
# have to agree: the stamp setup.sh writes, the manifest tags resolveEngineAMI
# selects on, the apk pin the packages are installed from, and the engine value
# the control plane launches instances with. A drift between them launches the
# wrong image, refuses the right one, or reports a version the VM is not running.
#
# The agent's layout table is checked by the postgres preset's copy of this test
# only, until this engine has a guest implementation to key it by.
SCRIPT_DIR=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd)
REPO_ROOT=$(CDPATH='' cd -- "${SCRIPT_DIR}/../../.." && pwd)
ENGINE="mariadb"
ENGINE_GO="${REPO_ROOT}/spinifex/handlers/rds/engine_mariadb.go"
FAILS=0
fail() { echo "FAIL: $*" >&2; FAILS=$((FAILS + 1)); }
stamped=$(sed -n "s|^printf '\\(.*\\)\\\\n' > /etc/spinifex-rds/engine\$|\\1|p" "${SCRIPT_DIR}/setup.sh")
if [ "${stamped}" != "${ENGINE}" ]; then
fail "setup.sh stamps engine '${stamped}', want '${ENGINE}'"
fi
if ! grep -q "engine=${ENGINE} " "${SCRIPT_DIR}/manifest.conf"; then
fail "manifest.conf SYSTEM_TAG does not carry engine=${ENGINE}"
grep '^SYSTEM_TAG' "${SCRIPT_DIR}/manifest.conf" >&2
fi
if ! grep -q "Name: *\"${ENGINE}\"" "${ENGINE_GO}"; then
fail "the control plane has no engine named ${ENGINE}"
fi
if ! grep -q "^ENGINE=\"${ENGINE}\"\$" "${SCRIPT_DIR}/rds-init"; then
fail "rds-init does not stamp the data volume '${ENGINE}'"
grep '^ENGINE=' "${SCRIPT_DIR}/rds-init" >&2
fi
# The tag is what an instance's recorded EngineVersion resolves an AMI by, so it
# has to be the same series the control plane pins and the packages install.
version=$(sed -n 's/^[[:space:]]*MajorVersion: *"\([^"]*\)".*/\1/p' "${ENGINE_GO}")
if [ -z "${version}" ]; then
fail "could not read MajorVersion from ${ENGINE_GO}"
else
if ! grep -q "engine-version=${version} " "${SCRIPT_DIR}/manifest.conf"; then
fail "manifest.conf SYSTEM_TAG does not carry engine-version=${version}"
grep '^SYSTEM_TAG' "${SCRIPT_DIR}/manifest.conf" >&2
fi
# Alpine's package name carries no version, so this fuzzy pin is the only
# thing standing between a rebuild and a silent series bump.
if ! grep -q "APK_PACKAGES=.*${ENGINE}=~${version} " "${SCRIPT_DIR}/manifest.conf"; then
fail "manifest.conf does not pin the ${ENGINE} packages to =~${version}"
grep '^APK_PACKAGES' "${SCRIPT_DIR}/manifest.conf" >&2
fi
fi
if [ "${FAILS}" -eq 0 ]; then
echo "rds-engine-stamp: all tests passed"
exit 0
fi
exit 1
+903
View File
@@ -0,0 +1,903 @@
#!/bin/sh
# rds-init — MariaDB bootstrap for the spinifex-rds-mariadb AMI.
#
# A boot oneshot ordered before the mariadb service. This is the MariaDB half of
# the engine seam: mariadb-install-db, the master grant, the datadir and the 3306
# default live here and nowhere else in the platform.
#
# Handoff contract, written by rds-agent into a root-owned tmpfs directory:
#
# /run/spinifex-rds/bootstrap.env 0600 KEY=value shell fragment:
# RDS_MODE=initialize|attach
# RDS_DB_INSTANCE_IDENTIFIER=<the DB instance this VM backs>
# RDS_MASTER_USERNAME=<master user name>
# RDS_MASTER_PASSWORD=<present only when RDS_MODE=initialize>
# RDS_DB_NAME=<initial database; optional, as on AWS>
# RDS_PORT=<listen port; defaults to 3306>
# RDS_DATA_VOLUME_ID / RDS_DATA_VOLUME_SERIAL=<expected volume identity>
# RDS_VM_GENERATION / RDS_FORMAT_AUTHORIZED=<create-only format grant>
# RDS_BOOTSTRAP_PENDING / RDS_PAYLOAD_ID=<the control plane is still
# waiting for this payload to be applied>
# /run/spinifex-rds/parameters.conf 0600 resolved parameter-group values, one
# name = 'value' per line
# /run/spinifex-rds/server.crt 0600 serving cert, cluster-CA-signed
# /run/spinifex-rds/server.key 0600 its private key
#
# Nothing there survives a reboot: the next boot re-fetches and the cert is
# re-minted rather than persisted.
#
# The data volume also carries an engine stamp, written before the datadir is
# inspected and compared on every boot, so another engine's volume is refused
# rather than initialised over.
#
# rds-init keys off the datadir, never off RDS_MODE. A datadir holding the mysql
# system database means skip mariadb-install-db and only refresh config, so a
# replaced VM comes back on its existing data volume without re-bootstrapping.
# RDS_BOOTSTRAP_PENDING is not an exception: it is a control-plane-resolved
# precondition compared against on-disk evidence, the same shape as
# RDS_FORMAT_AUTHORIZED.
set -eu
# The engine this image bakes, stamped onto the data volume and compared against
# it on every boot. Held here rather than read from /etc/spinifex-rds/engine:
# this script is the MariaDB half of the seam, so it is the same constant.
ENGINE="mariadb"
MARIADB_BIN="${RDS_MARIADB_BIN:-/usr/bin}"
ENGINE_USER="${RDS_ENGINE_USER:-mysql}"
# Mount point of the data volume, and the datadir one level inside it. The
# datadir is deliberately not the mount point itself: a failed bootstrap clears
# it, and the include directory, the engine stamp and the completion receipt all
# have to outlive that. It also keeps lost+found and the include directory out of
# a directory whose every subdirectory MariaDB lists as a database.
DATA_MOUNT="${RDS_DATA_MOUNT:-/var/lib/mysql}"
DATADIR="${RDS_DATADIR:-${DATA_MOUNT}/data}"
# Read through the image's baked /etc/my.cnf.d/zz-rds-include.cnf. Generated
# configuration lives here rather than under /etc because a class change is a VM
# replacement — fresh boot volume, same data volume — and parameters under /etc
# would be discarded by it while the API still reported the customer's values.
CONF_DIR="${RDS_CONF_DIR:-${DATA_MOUNT}/conf.d}"
PARAM_FILE="${CONF_DIR}/10-rds-parameters.cnf"
# What the server actually started on. The agent computes pending-restart by
# comparing the installed file against this one, so writing it here — before the
# engine starts — is what makes the two equal by construction on every boot.
# Deliberately not a .cnf name: !includedir reads *.cnf and this is comparison
# material, not a second set of settings.
SERVING_FILE="${CONF_DIR}/10-rds-parameters.serving"
PLATFORM_FILE="${CONF_DIR}/90-rds-init.cnf"
# Written between mariadb-install-db and a working master user. It lives on the
# data volume because no trap survives a crash or a hard kill, and it is what
# stops the next boot attaching to a datadir whose master user was never created.
BOOTSTRAP_SENTINEL="${DATADIR}/rds-bootstrap-incomplete"
# Proof that the master user was durably applied, written once the datadir and
# the user are both done. Deliberately outside DATADIR, which
# clear_unfinished_datadir rm -rf's.
RECEIPT_DIR="${RDS_RECEIPT_DIR:-${DATA_MOUNT}/.spinifex-rds/bootstrap}"
RECEIPT_FILE="${RECEIPT_DIR}/receipt.env"
# Which engine's data this volume holds. Outside DATADIR for the same reason the
# receipt is, and beside it rather than inside the bootstrap directory: it
# outlives any one bootstrap and is read before the datadir is ever inspected.
ENGINE_STAMP="${RDS_ENGINE_STAMP:-${DATA_MOUNT}/.spinifex-rds/engine}"
HANDOFF_DIR="${RDS_HANDOFF_DIR:-/run/spinifex-rds}"
ENV_FILE="${HANDOFF_DIR}/bootstrap.env"
SOCKET_DIR="${RDS_SOCKET_DIR:-/run/mysqld}"
SOCKET="${SOCKET_DIR}/mysqld.sock"
TLS_DIR="${RDS_TLS_DIR:-${SOCKET_DIR}/tls}"
LOG_DIR="${RDS_LOG_DIR:-/var/log/mysql}"
# The bootstrap server listens on a unix socket in this directory and nowhere
# else. It is 0700 and mysql-owned, so only mysql and root can reach the short
# window before the master user exists.
BOOTSTRAP_DIR="${RDS_BOOTSTRAP_DIR:-/run/rds-init}"
BOOTSTRAP_SOCKET="${BOOTSTRAP_DIR}/mysqld.sock"
BOOTSTRAP_PID="${BOOTSTRAP_DIR}/mysqld.pid"
# secure_file_priv names the one directory FILE-privileged statements may read
# and write. It must be a real directory: the empty string means no restriction
# at all. Root-owned and empty, on the boot volume, so it stays empty.
SECURE_FILE_DIR="${RDS_SECURE_FILE_DIR:-/var/lib/mysql-files}"
MOUNTS_FILE="${RDS_MOUNTS_FILE:-/proc/mounts}"
# How long the private bootstrap server gets to open its socket, as probes one
# BOOTSTRAP_POLL apart. InnoDB creates its redo log and system tablespace on this
# first start, so it is not instant.
BOOTSTRAP_PROBES="${RDS_BOOTSTRAP_PROBES:-60}"
BOOTSTRAP_POLL="${RDS_BOOTSTRAP_POLL:-1}"
# Failure cleanup gives the private server this long to honor TERM before KILL.
BOOTSTRAP_STOP_GRACE="${RDS_BOOTSTRAP_STOP_GRACE:-10}"
# The server truncates a longer name, so the database created would not be the
# one the customer asked for, and MariaDB maps the name onto a directory name.
MAX_DB_NAME_LEN=64
MAX_USERNAME_LEN=80
MANAGEMENT_DB="_spinifex_rds"
CREATE_DATABASE_PROCEDURE="create_database"
log() { echo "[rds-init] $*"; }
fail() { echo "[rds-init] ERROR: $*" >&2; exit 1; }
# Between these two, the datadir carries a mysql system database but no usable
# master user. _server_pid is the private bootstrap server, if one is running.
_created_datadir=0
_bootstrapped=0
_server_pid=""
# Runs from a trap rather than a statement, so neither `set -e` on an unguarded
# step between mariadb-install-db and a working master user nor a signal can skip
# it — either would leave a healthy-looking engine with no usable master user,
# unrepairable.
clear_unfinished_datadir() {
[ "${_created_datadir}" = "1" ] || return 0
[ "${_bootstrapped}" = "1" ] && return 0
# A server left holding a datadir that is about to be deleted keeps its
# socket and pidfile and blocks the mariadb service for the life of the boot.
kill_bootstrap_server
# The receipt is installed before the traps retire, so a sweep in between
# would leave one vouching for a datadir that is about to be deleted. The
# agent would acknowledge it and the control plane would destroy the only
# copy of the password this instance can ever be initialised with.
rm -f -- "${RECEIPT_FILE}"
if rm -rf -- "${DATADIR}" && [ ! -e "${DATADIR}" ]; then
echo "[rds-init] ERROR: bootstrap did not complete; datadir ${DATADIR} cleared" >&2
return 0
fi
echo "[rds-init] ERROR: bootstrap did not complete and datadir ${DATADIR} could not be cleared; recover it out of band before retrying" >&2
}
kill_bootstrap_server() {
[ -n "${_server_pid}" ] || return 0
_pid="${_server_pid}"
# wait has no timeout in POSIX sh. A watchdog bounds it and escalates, while
# this shell remains the one that waits and reaps its own child.
kill -TERM "${_pid}" 2>/dev/null || true
(
sleep "${BOOTSTRAP_STOP_GRACE}"
kill -KILL "${_pid}" 2>/dev/null || true
) &
_watchdog_pid=$!
# A signal exit is expected here: cleanup cares that the process is reaped
# and cannot touch the datadir, not whether its incomplete startup was clean.
wait "${_pid}" 2>/dev/null || true
_server_pid=""
kill -TERM "${_watchdog_pid}" 2>/dev/null || true
wait "${_watchdog_pid}" 2>/dev/null || true
}
load_handoff() {
[ -f "${ENV_FILE}" ] || fail "no bootstrap handoff at ${ENV_FILE} — rds-agent has not delivered the bootstrap config"
# shellcheck disable=SC1090
. "${ENV_FILE}"
RDS_MODE="${RDS_MODE:-}"
RDS_DB_INSTANCE_IDENTIFIER="${RDS_DB_INSTANCE_IDENTIFIER:-}"
RDS_MASTER_USERNAME="${RDS_MASTER_USERNAME:-}"
RDS_MASTER_PASSWORD="${RDS_MASTER_PASSWORD:-}"
RDS_DB_NAME="${RDS_DB_NAME:-}"
RDS_PORT="${RDS_PORT:-3306}"
RDS_VM_GENERATION="${RDS_VM_GENERATION:-}"
# Absent from an agent that predates the receipt protocol, which reads as
# "nothing is pending" and leaves every path below unchanged.
RDS_BOOTSTRAP_PENDING="${RDS_BOOTSTRAP_PENDING:-0}"
RDS_PAYLOAD_ID="${RDS_PAYLOAD_ID:-}"
[ -n "${RDS_MASTER_USERNAME}" ] || fail "bootstrap config has no RDS_MASTER_USERNAME"
# The name is interpolated into CREATE USER by the shell — the mariadb client
# has no identifier quoting of its own — so the rule the control plane
# applies is enforced here rather than trusted.
case "${RDS_MASTER_USERNAME}" in
[!A-Za-z]* | *[!A-Za-z0-9_]*)
fail "RDS_MASTER_USERNAME ${RDS_MASTER_USERNAME} must begin with a letter and hold only letters, digits and underscores; the control plane should have rejected it"
;;
esac
if [ "${#RDS_MASTER_USERNAME}" -gt "${MAX_USERNAME_LEN}" ]; then
fail "RDS_MASTER_USERNAME is longer than the ${MAX_USERNAME_LEN} characters MariaDB allows; the control plane should have rejected it"
fi
# root is the superuser mariadb-install-db creates and rds-init keeps for
# itself, mysql is the unix_socket account beside it, mariadb.sys owns the sys
# schema, rdsadmin is reserved as on AWS, and PUBLIC is a role name the server
# refuses to create a user under. Matched the same way the control plane
# matches them: anything it refuses must be refused here too, or the name
# reaches mariadb-install-db, fails at CREATE USER and spends the one-shot
# password on the way.
_master_lower=$(printf '%s' "${RDS_MASTER_USERNAME}" | tr '[:upper:]' '[:lower:]')
case "${_master_lower}" in
root | mysql | mariadb.sys | rdsadmin | public | mysql.* | mariadb.*)
fail "RDS_MASTER_USERNAME ${RDS_MASTER_USERNAME} is reserved by the platform; the control plane should have rejected it"
;;
esac
# The same rule the control plane applies. Here it is the barrier rather than
# defence in depth: the mariadb client has no identifier interpolation, so the
# name is built into CREATE DATABASE by the shell, and MariaDB maps a database
# name onto a directory name — which makes /, \, . and a trailing space
# structurally dangerous rather than merely awkward.
case "${RDS_DB_NAME}" in
'') ;;
[!A-Za-z]* | *[!A-Za-z0-9_]*)
fail "RDS_DB_NAME ${RDS_DB_NAME} must begin with a letter and hold only letters, digits and underscores; the control plane should have rejected it"
;;
esac
if [ "${#RDS_DB_NAME}" -gt "${MAX_DB_NAME_LEN}" ]; then
fail "RDS_DB_NAME is longer than the ${MAX_DB_NAME_LEN} characters MariaDB allows; the control plane should have rejected it"
fi
case "${RDS_PORT}" in
'' | *[!0-9]*) fail "bootstrap config has a non-numeric RDS_PORT: ${RDS_PORT}" ;;
esac
# redact_stream is line-oriented, so a password spanning a line boundary
# would match neither half and reach /dev/console in the clear. The control
# plane rejects one; refusing here too keeps that an asserted property of
# the handoff rather than a silent dependency on another service.
case "${RDS_MASTER_PASSWORD}" in
*'
'*) fail "the master password in the bootstrap config spans more than one line; the control plane should have rejected it" ;;
esac
# The form the password takes inside the SQL, computed once beside it. A
# statement echoed on failure carries that form and not the raw secret, so
# redacting only the raw one would put a trivially unescaped password on the
# console for exactly the passwords that needed escaping.
RDS_MASTER_PASSWORD_SQL=""
if [ -n "${RDS_MASTER_PASSWORD}" ]; then
RDS_MASTER_PASSWORD_SQL=$(sql_literal "${RDS_MASTER_PASSWORD}")
fi
log "bootstrap config loaded (mode=${RDS_MODE:-unset}, port=${RDS_PORT})"
}
# Every replace boots a fresh clone of this AMI, so a datadir off the data volume
# disappears at the next class change or recovery — silently, as the engine looks
# fine.
require_data_volume() {
if awk -v mnt="${DATA_MOUNT}" '$2 == mnt { found = 1 } END { exit !found }' "${MOUNTS_FILE}"; then
return 0
fi
[ "${RDS_ALLOW_LOCAL_DATADIR:-0}" = "1" ] ||
fail "${DATA_MOUNT} is not a mount point — the data volume is not attached; refusing to initialise on the boot volume"
log "warning: ${DATA_MOUNT} is not a mount point; data will not survive a VM replace (RDS_ALLOW_LOCAL_DATADIR=1)"
}
# True when the data volume holds anything that is not ours. The mount is what
# is inspected rather than ${DATADIR}: another engine lays its data out under its
# own name, so a PostgreSQL volume has content here while ${DATADIR} does not
# exist at all.
data_volume_has_content() {
[ -d "${DATA_MOUNT}" ] || return 1
# A mount that cannot be read is neither empty nor not: it is a refusal,
# because the caller decides from the answer whether to initialise.
ls -A -- "${DATA_MOUNT}" >/dev/null 2>&1 ||
fail "could not inspect ${DATA_MOUNT}; refusing to decide whether the data volume is ours"
# Globbed rather than split out of `ls`, so an entry carrying whitespace
# counts once. lost+found is the filesystem's, and the metadata and include
# directories are ours — none of the three says whose data the volume holds.
for _entry in "${DATA_MOUNT}"/* "${DATA_MOUNT}"/.[!.]* "${DATA_MOUNT}"/..?*; do
[ -e "${_entry}" ] || [ -L "${_entry}" ] || continue
case "${_entry##*/}" in
lost+found | .spinifex-rds | "${CONF_DIR##*/}") continue ;;
esac
return 0
done
return 1
}
# Written before the datadir is inspected, so a volume this image goes on to
# initialise carries the stamp from the first boot onwards. Fatal on failure:
# an unstamped volume is one the next boot cannot tell from another engine's.
write_engine_stamp() {
_stamp_dir=$(dirname "${ENGINE_STAMP}")
install -d -m 0700 -o "${ENGINE_USER}" -g "${ENGINE_USER}" "${_stamp_dir}" ||
fail "the datadir engine stamp directory ${_stamp_dir} could not be created; refusing to use a data volume whose engine cannot be recorded"
_tmp="${ENGINE_STAMP}.new"
printf '%s\n' "${ENGINE}" > "${_tmp}" ||
fail "the datadir engine stamp ${ENGINE_STAMP} could not be written; refusing to use a data volume whose engine cannot be recorded"
chown "${ENGINE_USER}:${ENGINE_USER}" "${_tmp}"
chmod 0600 "${_tmp}"
# Renamed rather than written in place, so a crash leaves either the old
# stamp or none — and none is decided on the datadir, where a torn one would
# be fatal.
sync "${_tmp}" 2>/dev/null || sync
mv -f "${_tmp}" "${ENGINE_STAMP}" ||
fail "the datadir engine stamp ${ENGINE_STAMP} could not be installed; refusing to use a data volume whose engine cannot be recorded"
sync "${_stamp_dir}" 2>/dev/null || sync
}
# The volume mounts cleanly whichever engine wrote it, and another engine's
# datadir reads as uninitialised here: rds-init would run mariadb-install-db
# beside the customer's data and serve an empty database that passes every health
# probe.
require_engine_stamp() {
if [ -e "${ENGINE_STAMP}" ]; then
_stamped=""
_stamp_read=0
_stamped=$(head -n 1 -- "${ENGINE_STAMP}") && _stamp_read=1
[ "${_stamp_read}" = "1" ] ||
fail "the datadir engine stamp ${ENGINE_STAMP} exists but could not be read; refusing to attach to a data volume whose engine cannot be established"
if [ "${_stamped}" != "${ENGINE}" ]; then
fail "the data volume holds a '${_stamped}' datadir but this image runs ${ENGINE}; refusing to touch it. Attach it to a '${_stamped}' instance, or recover the volume out of band."
fi
log "data volume is stamped ${ENGINE}"
return 0
fi
# No MariaDB volume in the field predates the stamp, so unlike PostgreSQL
# there is nothing to backfill onto: an unstamped volume with content on it
# is by definition not ours, and stamping it would turn the check into a
# rubber stamp on exactly the volume it exists to refuse.
if data_volume_has_content; then
fail "${DATA_MOUNT} holds data but the volume carries no ${ENGINE} engine stamp; refusing to touch it. Recover the volume out of band."
fi
log "data volume carries no engine stamp — recording ${ENGINE}"
write_engine_stamp
}
# Creates the system tables in an empty datadir. Root authenticates through the
# unix_socket plugin — the direct analogue of PostgreSQL's peer auth, and the
# reason no bootstrap password is needed.
initialise_datadir() {
# An empty datadir in `attach` mode means the data volume did not come back.
# Initialising there would report success and hand the customer an empty
# database in place of theirs.
if [ "${RDS_MODE}" = "attach" ]; then
fail "datadir ${DATADIR} holds no MariaDB system database but the bootstrap config is Mode=attach — the existing data volume is missing; refusing to create an empty database"
fi
[ -n "${RDS_MASTER_PASSWORD}" ] || fail "Mode=${RDS_MODE:-unset} carries no master password; cannot initialise the datadir"
install -d -m 0700 -o "${ENGINE_USER}" -g "${ENGINE_USER}" "${DATADIR}"
# Whether the datadir was empty beforehand gates the failure cleanup below.
# A torn write can leave the mysql system database missing over intact user
# data, and deleting then would destroy it in response to the one signal that
# it is still there.
_was_empty=0
if _contents=$(ls -A -- "${DATADIR}" 2>/dev/null); then
[ -z "${_contents}" ] && _was_empty=1
else
fail "could not inspect datadir ${DATADIR}; refusing to decide whether failure cleanup is safe"
fi
log "initialising the MariaDB datadir at ${DATADIR}"
# --no-defaults: the installed configuration includes the customer's own
# parameter file, and a static value this server will not start under would
# spend the one-shot master password on a datadir that never gets one. The
# real server starts on those parameters, where a rollback exists for them.
if ! "${MARIADB_BIN}/mariadb-install-db" --no-defaults \
--user="${ENGINE_USER}" --datadir="${DATADIR}" \
--auth-root-authentication-method=socket --skip-test-db; then
# A half-written datadir carries a mysql directory and would look
# initialised forever after, but only what this invocation created may be
# cleared.
if [ "${_was_empty}" = "1" ]; then
if rm -rf -- "${DATADIR}" && [ ! -e "${DATADIR}" ]; then
fail "mariadb-install-db failed; datadir cleared"
fi
fail "mariadb-install-db failed and datadir ${DATADIR} could not be cleared; recover it out of band before retrying"
fi
fail "mariadb-install-db failed and ${DATADIR} was not empty beforehand — refusing to clear it, since the contents may be a datadir whose system tables were lost; recover the data volume out of band"
fi
# Only a datadir built over an empty directory may be swept later.
_created_datadir="${_was_empty}"
trap clear_unfinished_datadir EXIT
# An EXIT trap does not run on a signal in a POSIX shell, so the same sweep
# is armed for the signals a shutdown, an ACPI powerdown or a host-side
# force-stop delivers. EXIT is disarmed first so the sweep runs once.
trap 'trap - EXIT; clear_unfinished_datadir; exit 1' INT TERM HUP
# No trap survives a crash or a SIGKILL, so the incomplete state is also
# recorded on the volume itself. Cleared once the master user exists.
printf '%s\n' \
'rds-init created this datadir and did not finish bootstrapping the master user.' \
'While this file exists rds-init refuses to start the engine: the datadir carries no' \
'usable master user, so it must not be attached to. Recover out of band.' \
> "${BOOTSTRAP_SENTINEL}"
chown "${ENGINE_USER}:${ENGINE_USER}" "${BOOTSTRAP_SENTINEL}"
chmod 0600 "${BOOTSTRAP_SENTINEL}"
}
# The resolved parameter group, prefixed with the group header. MariaDB parses
# each file in an include directory on its own and treats a setting before any
# group as a fatal error, so the header belongs to whoever writes the file rather
# than to the engine-neutral renderer that produced its body.
install_parameters() {
if [ -f "${HANDOFF_DIR}/parameters.conf" ]; then
_tmp="${CONF_DIR}/.10-rds-parameters.cnf.new"
{ printf '[mysqld]\n'; cat "${HANDOFF_DIR}/parameters.conf"; } > "${_tmp}" ||
fail "the resolved parameter group could not be written to ${PARAM_FILE}"
chown "${ENGINE_USER}:${ENGINE_USER}" "${_tmp}"
chmod 0600 "${_tmp}"
mv -f "${_tmp}" "${PARAM_FILE}"
record_serving_parameters
return 0
fi
# A handoff without parameters keeps the previous include: dropping it would
# silently reset the instance to engine defaults.
if [ -f "${PARAM_FILE}" ]; then
log "warning: no parameters in the handoff — keeping the parameters already installed"
record_serving_parameters
fi
}
# The engine is about to start on whatever is installed, so the two are equal by
# construction here. The agent refreshes this copy only after a restart it
# performs itself: a live SET GLOBAL apply deliberately leaves it alone, since
# that value was adopted without a restart, which is what "not pending" means.
record_serving_parameters() {
install -m 0600 -o "${ENGINE_USER}" -g "${ENGINE_USER}" "${PARAM_FILE}" "${SERVING_FILE}" ||
fail "the serving copy of the parameters could not be written to ${SERVING_FILE}"
}
# The key stays on tmpfs: it is re-minted every fetch, so persisting it would
# spread a private key into every snapshot.
install_tls() {
if [ -f "${HANDOFF_DIR}/server.crt" ] && [ -f "${HANDOFF_DIR}/server.key" ]; then
install -d -m 0700 -o "${ENGINE_USER}" -g "${ENGINE_USER}" "${TLS_DIR}"
install -m 0600 -o "${ENGINE_USER}" -g "${ENGINE_USER}" "${HANDOFF_DIR}/server.crt" "${TLS_DIR}/server.crt"
install -m 0600 -o "${ENGINE_USER}" -g "${ENGINE_USER}" "${HANDOFF_DIR}/server.key" "${TLS_DIR}/server.key"
rm -f "${HANDOFF_DIR}/server.crt" "${HANDOFF_DIR}/server.key"
SSL_MODE="on"
return 0
fi
# A re-run after the cert was consumed must not turn TLS off underneath
# clients connecting with --ssl-verify-server-cert.
if [ -f "${TLS_DIR}/server.crt" ] && [ -f "${TLS_DIR}/server.key" ]; then
SSL_MODE="on"
return 0
fi
log "warning: no serving cert in the handoff — the engine will start without TLS"
SSL_MODE="off"
}
# The 90- prefix sorts last within the include directory and MariaDB takes the
# last occurrence of a setting, so a resolved parameter can never move the
# endpoint's port, redirect the datadir or turn off TLS.
#
# innodb_buffer_pool_chunk_size and innodb_buffer_pool_instances are deliberately
# absent although they are platform-owned: 11.8 removed the second outright, so
# naming it is a startup failure rather than an override, and the first is
# deprecated with no effect. The granularity they used to pin is fixed by the
# server itself in this series.
write_platform_config() {
_ssl="$1"
_tmp="${CONF_DIR}/.90-rds-init.cnf.new"
cat > "${_tmp}" <<EOF
# Managed by rds-init — regenerated on every boot from the bootstrap config.
# Edits here are lost; instance parameters belong in the parameter group.
[mysqld]
port = ${RDS_PORT}
datadir = ${DATADIR}
socket = ${SOCKET}
bind_address = 0.0.0.0
# InnoDB is the only storage engine the snapshot guarantee holds for: it keeps a
# redo log and recovers from a crash-consistent snapshot, and Aria and MyISAM do
# not. An explicit ENGINE= clause still works, which keeps the exception
# deliberate and per-table rather than account-wide and invisible.
default_storage_engine = InnoDB
# Point-in-time recovery is not offered, so the binary log is off: it would cost
# write throughput and disk for a capability nothing reads.
skip-log-bin
# Defence in depth behind the master grant, which withholds FILE: an empty
# secure_file_priv would mean no restriction at all, so it names a root-owned
# directory that stays empty.
secure_file_priv = ${SECURE_FILE_DIR}
skip-symbolic-links
EOF
if [ "${_ssl}" = "on" ]; then
# Offered, not enforced, matching the PostgreSQL image's pg_hba: clients
# opt in. require_secure_transport is deliberately absent until enforcement
# lands for both engines at once. No ssl_ca, because client certificates
# are not required and it would only gate those.
cat >> "${_tmp}" <<EOF
ssl_cert = ${TLS_DIR}/server.crt
ssl_key = ${TLS_DIR}/server.key
EOF
fi
chown "${ENGINE_USER}:${ENGINE_USER}" "${_tmp}"
chmod 0600 "${_tmp}"
mv -f "${_tmp}" "${PLATFORM_FILE}"
}
# Everything refreshed on every boot: the resolved parameters and their serving
# copy, the serving cert and the platform's own settings.
install_runtime_config() {
install -d -m 0750 -o "${ENGINE_USER}" -g "${ENGINE_USER}" "${CONF_DIR}"
install -d -m 0755 -o "${ENGINE_USER}" -g "${ENGINE_USER}" "${SOCKET_DIR}"
install -d -m 0750 -o "${ENGINE_USER}" -g "${ENGINE_USER}" "${LOG_DIR}"
install -d -m 0700 "${SECURE_FILE_DIR}"
install_parameters
install_tls
write_platform_config "${SSL_MODE}"
}
# Brings the server up on a private socket with no TCP listener at all, so the
# master password can be set before the engine is reachable. --no-defaults for
# the same reason mariadb-install-db has it, and because the password below is
# escaped for a known sql_mode rather than the customer's.
start_bootstrap_server() {
install -d -m 0700 -o "${ENGINE_USER}" -g "${ENGINE_USER}" "${BOOTSTRAP_DIR}"
"${MARIADB_BIN}/mariadbd" --no-defaults \
--user="${ENGINE_USER}" --datadir="${DATADIR}" \
--skip-networking --socket="${BOOTSTRAP_SOCKET}" \
--pid-file="${BOOTSTRAP_PID}" --log-error="${LOG_DIR}/rds-init.log" &
_server_pid=$!
_probes=0
while ! bootstrap_admin ping > /dev/null 2>&1; do
# The server writes why it gave up to its error log, which is on the boot
# volume and readable after the fact; the console only gets the refusal.
if ! kill -0 "${_server_pid}" 2>/dev/null; then
_server_pid=""
fail "the bootstrap server exited before it opened ${BOOTSTRAP_SOCKET}; see ${LOG_DIR}/rds-init.log"
fi
if [ "${_probes}" -ge "${BOOTSTRAP_PROBES}" ]; then
fail "the bootstrap server did not open ${BOOTSTRAP_SOCKET} after ${BOOTSTRAP_PROBES} probes ${BOOTSTRAP_POLL}s apart; see ${LOG_DIR}/rds-init.log"
fi
sleep "${BOOTSTRAP_POLL}"
_probes=$((_probes + 1))
done
}
bootstrap_admin() {
"${MARIADB_BIN}/mariadb-admin" --no-defaults \
--socket="${BOOTSTRAP_SOCKET}" --user=root "$@"
}
# Returns the shutdown's status, then the server's own: a caller reading this
# must be able to tell "it refused to stop" from "it stopped badly", and both
# from a server still holding the datadir.
stop_bootstrap_server() {
[ -n "${_server_pid}" ] || return 0
bootstrap_admin shutdown || return 1
# `wait` rather than a kill -0 poll: the server is this shell's own child, so
# until it is reaped it stays a zombie that kill -0 reports as alive.
_wait_rc=0
wait "${_server_pid}" || _wait_rc=$?
_server_pid=""
return "${_wait_rc}"
}
# Prepended to every bootstrap script. The two logs that would copy the password
# out of the statement are turned off — global rather than session variables, and
# safe to set globally because nothing but rds-init can reach this server. The
# mode is pinned because the password is escaped below for a mode that has
# NO_BACKSLASH_ESCAPES off.
BOOTSTRAP_SQL_GUARD="SET GLOBAL general_log = 0;
SET GLOBAL slow_query_log = 0;
SET SESSION sql_mode = 'NO_ENGINE_SUBSTITUTION';"
# Replaces every occurrence of the secret with a placeholder. Parameter
# expansion rather than sed, whose script would carry the secret in an argv any
# process on the guest can read.
redact_stream() {
_secret="$1"
while IFS= read -r _line || [ -n "${_line}" ]; do
_out=""
while [ -n "${_secret}" ]; do
case "${_line}" in
*"${_secret}"*)
_out="${_out}${_line%%"${_secret}"*}[REDACTED]"
_line="${_line#*"${_secret}"}"
;;
*) break ;;
esac
done
printf '%s\n' "${_out}${_line}"
done
}
# A value safe to interpolate into a single-quoted SQL literal. The quote is
# doubled, which holds in every sql_mode, and the backslash is doubled, which is
# why the guard above pins one where backslash escapes are on. The secret reaches
# sed on stdin: an argv would be readable by every process in the guest.
sql_literal() {
printf '%s' "$1" | sed -e 's/\\/\\\\/g' -e "s/'/''/g"
}
# The script rides stdin and never an argv, and both streams are redacted before
# they reach the caller: the client echoes the statement it failed on, and the
# initd sends everything here to /dev/console, which the host captures off ttyS0.
mariadb_bootstrap() {
# A pipeline's status is its last command's and POSIX sh has no pipefail, so
# the client's own status leaves the subshell through the filesystem. The
# guard shifts the client's reported line numbers by the lines it adds.
_status_file="${BOOTSTRAP_DIR}/client.status"
rm -f "${_status_file}"
{
_client_rc=0
# --batch stops at the first error and exits non-zero, so a failed
# statement cannot leave the rest of the script running.
{ printf '%s\n' "${BOOTSTRAP_SQL_GUARD}"; cat; } |
"${MARIADB_BIN}/mariadb" --no-defaults --batch --skip-column-names \
--socket="${BOOTSTRAP_SOCKET}" --user=root ||
_client_rc=$?
printf '%s\n' "${_client_rc}" > "${_status_file}"
} 2>&1 | redact_stream "${RDS_MASTER_PASSWORD}" | redact_stream "${RDS_MASTER_PASSWORD_SQL}"
# Every way of not recovering the client's status is a failed bootstrap
# rather than a silent success, and each says so in its own words: blaming
# the client for a status file that could not be written would send an
# operator looking at a MariaDB that did exactly what it was asked.
#
# A torn write leaves the file created but empty, so the value is checked
# before it is returned: `return ""` is fatal in the guest's ash, and would
# kill rds-init here — past the sweep, before the diagnostics.
_client_status=1
if [ -f "${_status_file}" ]; then
_client_status=$(cat "${_status_file}" 2>/dev/null) || _client_status=""
case "${_client_status}" in
'' | *[!0-9]*)
echo "[rds-init] ERROR: the mariadb client ran but its exit status could not be read back from ${_status_file} (got '${_client_status}'); treating the bootstrap as failed" >&2
_client_status=1
;;
esac
rm -f "${_status_file}"
else
echo "[rds-init] ERROR: the mariadb client ran but its exit status was never recorded at ${_status_file}; treating the bootstrap as failed" >&2
fi
return "${_client_status}"
}
# 11.8 creates no anonymous account and --skip-test-db leaves no test database,
# so on a current image both of these are no-ops. They run anyway: the properties
# are asserted by the image tests, and an engine or packaging change that
# reintroduced either would otherwise reach a customer's instance.
remove_insecure_defaults() {
mariadb_bootstrap <<'SQL'
DELETE FROM mysql.global_priv WHERE User = '';
DROP DATABASE IF EXISTS test;
FLUSH PRIVILEGES;
SQL
}
# MariaDB has no DENY or partial revocation: an object privilege ON *.* also
# reaches mysql.global_priv. Application privileges are therefore database-scoped,
# while this protected definer procedure creates and grants future databases.
apply_master_user() {
log "creating the master user ${RDS_MASTER_USERNAME}"
mariadb_bootstrap <<SQL
CREATE USER IF NOT EXISTS '${RDS_MASTER_USERNAME}'@'%' IDENTIFIED BY '${RDS_MASTER_PASSWORD_SQL}';
ALTER USER '${RDS_MASTER_USERNAME}'@'%' IDENTIFIED BY '${RDS_MASTER_PASSWORD_SQL}';
GRANT RELOAD, PROCESS, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT
ON *.* TO '${RDS_MASTER_USERNAME}'@'%';
CREATE DATABASE IF NOT EXISTS \`${MANAGEMENT_DB}\`;
DROP PROCEDURE IF EXISTS \`${MANAGEMENT_DB}\`.\`${CREATE_DATABASE_PROCEDURE}\`;
DELIMITER //
CREATE DEFINER='root'@'localhost' PROCEDURE \`${MANAGEMENT_DB}\`.\`${CREATE_DATABASE_PROCEDURE}\`(
IN database_name VARCHAR(64) CHARACTER SET ascii
)
SQL SECURITY DEFINER
BEGIN
IF database_name IS NULL
OR BINARY database_name NOT REGEXP '^[A-Za-z][A-Za-z0-9_]{0,63}$'
OR LOWER(database_name) IN ('mysql', 'information_schema', 'performance_schema', 'sys', '${MANAGEMENT_DB}') THEN
SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'database name must be a non-system identifier of at most 64 characters';
END IF;
SET @rds_database = REPLACE(database_name, '\`', '\`\`');
SET @rds_statement = CONCAT('CREATE DATABASE IF NOT EXISTS \`', @rds_database, '\`');
PREPARE rds_create_database FROM @rds_statement;
EXECUTE rds_create_database;
DEALLOCATE PREPARE rds_create_database;
SET @rds_statement = CONCAT(
'GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, ',
'CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, ',
'CREATE ROUTINE, ALTER ROUTINE, EVENT, TRIGGER ON \`', @rds_database,
'\`.* TO ''${RDS_MASTER_USERNAME}''@''%'' WITH GRANT OPTION'
);
PREPARE rds_grant_database FROM @rds_statement;
EXECUTE rds_grant_database;
DEALLOCATE PREPARE rds_grant_database;
END//
DELIMITER ;
GRANT EXECUTE ON PROCEDURE \`${MANAGEMENT_DB}\`.\`${CREATE_DATABASE_PROCEDURE}\`
TO '${RDS_MASTER_USERNAME}'@'%';
SQL
}
# The resolved value of one setting in the installed parameter file, or nothing
# when the group does not carry it.
resolved_parameter() {
[ -f "${PARAM_FILE}" ] || return 0
sed -n "s/^[[:space:]]*$1[[:space:]]*=[[:space:]]*//p" "${PARAM_FILE}" |
tail -n 1 | tr -d "\"'[:space:]"
}
# The bootstrap server runs on --no-defaults, so it is not carrying the character
# set the real server will start with — and a database keeps the character set it
# was created under for good. Both values therefore come from the parameter file
# rather than from the server's compiled default.
database_charset_clause() {
_charset=$(resolved_parameter character_set_server)
_collation=$(resolved_parameter collation_server)
if [ -z "${_charset}" ]; then
_charset="utf8mb4"
_collation=""
fi
# Interpolated into CREATE DATABASE below, and read out of a file rather than
# validated as an identifier on the way in.
case "${_charset}${_collation}" in
*[!A-Za-z0-9_]*)
fail "the parameter group's character_set_server or collation_server is not a bare identifier; refusing to interpolate it into CREATE DATABASE"
;;
esac
if [ -n "${_collation}" ]; then
printf 'CHARACTER SET %s COLLATE %s' "${_charset}" "${_collation}"
return 0
fi
printf 'CHARACTER SET %s' "${_charset}"
}
# As on AWS, an instance created without a DBName gets no initial database. A
# supplied database is created under its resolved charset before the protected
# routine applies the same scope future databases receive.
create_database() {
log "creating database ${RDS_DB_NAME}"
_charset_clause=$(database_charset_clause)
mariadb_bootstrap <<SQL
CREATE DATABASE IF NOT EXISTS \`${RDS_DB_NAME}\` ${_charset_clause};
CALL \`${MANAGEMENT_DB}\`.\`${CREATE_DATABASE_PROCEDURE}\`('${RDS_DB_NAME}');
SQL
}
# Runs on a freshly initialised datadir only.
bootstrap_master() {
log "starting bootstrap server on ${BOOTSTRAP_DIR}"
start_bootstrap_server
_rc=0
remove_insecure_defaults || _rc=$?
if [ "${_rc}" -eq 0 ]; then
apply_master_user || _rc=$?
fi
if [ "${_rc}" -eq 0 ] && [ -n "${RDS_DB_NAME}" ]; then
create_database || _rc=$?
fi
# `set -e` is suspended for a function called from an AND-OR list, so the
# stop status is captured rather than propagated. A failed stop must also not
# abort before the bootstrap check below, which names the real cause.
_stop_rc=0
stop_bootstrap_server || _stop_rc=$?
# The datadir is swept by the traps, which also cover the steps here that
# abort under `set -e` before this check is ever reached.
if [ "${_rc}" -ne 0 ]; then
fail "bootstrapping the master user/database failed (rc=${_rc})"
fi
# Proof written while the traps are still armed, so a receipt that cannot be
# made durable clears the datadir instead of leaving one it cannot vouch for.
write_completion_receipt
# The master user exists from here, so the datadir is complete and nothing
# may clear it — including on the stop failure reported below.
mark_bootstrap_complete
# A server still holding the datadir blocks the mariadb service for the life
# of the boot, so this is a failure and not a warning.
if [ "${_stop_rc}" -ne 0 ]; then
fail "the bootstrap server did not stop (rc=${_stop_rc}) and still holds ${DATADIR}; the mariadb service cannot start until it is gone"
fi
}
# The value of one receipt key. Never sourced: the receipt is on the data volume,
# so a restored volume can carry one written by another instance, and executing
# it would run whatever that instance's control plane put there.
receipt_value() {
[ -f "${RECEIPT_FILE}" ] || return 0
sed -n "s/^$1=//p" "${RECEIPT_FILE}" 2>/dev/null | head -n 1
}
# Written between the master user and the retirement of the traps, so a receipt
# that cannot be made durable still clears the datadir rather than leaving a
# bootstrapped one nobody can prove.
#
# Fatal on failure, the mirror image of the sentinel removal below: a healthy
# engine whose receipt never landed would come up available while the agent
# blocked forever on a receipt that will never appear, leaving a working database
# that can never accept a password change or a parameter reload.
write_completion_receipt() {
install -d -m 0700 -o "${ENGINE_USER}" -g "${ENGINE_USER}" "${RECEIPT_DIR}" ||
fail "the master user exists but the bootstrap receipt directory ${RECEIPT_DIR} could not be created"
_tmp="${RECEIPT_FILE}.new"
{
echo "# Written by rds-init once the master user was applied. Holds no secrets."
echo "RDS_RECEIPT_PAYLOAD_ID=${RDS_PAYLOAD_ID}"
echo "RDS_RECEIPT_DB_INSTANCE_IDENTIFIER=${RDS_DB_INSTANCE_IDENTIFIER}"
echo "RDS_RECEIPT_VM_GENERATION=${RDS_VM_GENERATION}"
echo "RDS_RECEIPT_COMPLETED_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
} > "${_tmp}" ||
fail "the master user exists but the bootstrap receipt ${RECEIPT_FILE} could not be written"
chown "${ENGINE_USER}:${ENGINE_USER}" "${_tmp}"
chmod 0600 "${_tmp}"
# A receipt still in the page cache is lost by exactly the crash it exists to
# survive, so it is on disk before it is renamed into place.
sync "${_tmp}" 2>/dev/null || sync
mv -f "${_tmp}" "${RECEIPT_FILE}" ||
fail "the master user exists but the bootstrap receipt ${RECEIPT_FILE} could not be installed"
sync "${RECEIPT_DIR}" 2>/dev/null || sync
}
# A system database with no sentinel and no receipt is ambiguous from the datadir
# alone: either an instance that bootstrapped before receipts existed, which must
# attach, or one that applied the master user and crashed before the receipt was
# durable, which must not. Only the control plane's view of whether the payload
# is still staged separates the two, and that is what RDS_BOOTSTRAP_PENDING says.
require_bootstrap_receipt() {
[ "${RDS_BOOTSTRAP_PENDING}" = "1" ] || return 0
if [ -n "${RDS_PAYLOAD_ID}" ] &&
[ "$(receipt_value RDS_RECEIPT_PAYLOAD_ID)" = "${RDS_PAYLOAD_ID}" ] &&
[ "$(receipt_value RDS_RECEIPT_DB_INSTANCE_IDENTIFIER)" = "${RDS_DB_INSTANCE_IDENTIFIER}" ]; then
return 0
fi
fail "the control plane has no record of a completed bootstrap for this datadir; refusing to attach to a datadir whose master user cannot be proven. Recover the volume out of band, or delete and recreate the instance."
}
# Retires the guards armed at initialisation, in an order where no signal in
# between can clear a datadir that now carries a usable master user.
mark_bootstrap_complete() {
_bootstrapped=1
rm -f "${BOOTSTRAP_SENTINEL}" ||
fail "the master user exists but ${BOOTSTRAP_SENTINEL} could not be removed; the next boot will refuse to start until it is deleted"
trap - EXIT INT TERM HUP
}
# Drops the master password from the handoff once applied, so it is not left
# readable for the life of the boot. The rest is tmpfs and goes at reboot.
consume_secrets() {
[ -f "${ENV_FILE}" ] || return 0
_tmp="${ENV_FILE}.new"
install -m 0600 /dev/null "${_tmp}"
grep -v '^RDS_MASTER_PASSWORD=' "${ENV_FILE}" >> "${_tmp}" || true
mv -f "${_tmp}" "${ENV_FILE}"
}
load_handoff
require_data_volume
require_engine_stamp
if [ -d "${DATADIR}/mysql" ]; then
# The engine would start cleanly on such a datadir and pass every health
# probe with no master user to log in as. The sentinel says the bootstrap did
# not finish, not that the volume is empty, so nothing here clears it.
if [ -e "${BOOTSTRAP_SENTINEL}" ]; then
fail "datadir ${DATADIR} carries ${BOOTSTRAP_SENTINEL}: an earlier bootstrap was interrupted before the master user existed; recover the data volume out of band rather than starting the engine"
fi
require_bootstrap_receipt
log "datadir ${DATADIR} is already initialised — attaching"
install_runtime_config
else
initialise_datadir
install_runtime_config
bootstrap_master
fi
consume_secrets
log "bootstrap complete; handing off to the mariadb service"
+35
View File
@@ -0,0 +1,35 @@
#!/sbin/openrc-run
# rds-init — boot oneshot that bootstraps or attaches the MariaDB datadir before
# the engine starts.
#
# `before mariadb` orders the two, and /etc/conf.d/mariadb makes the engine need
# this oneshot: Alpine's packaged mariadb service only prints a warning when the
# datadir is empty and starts anyway, so ordering alone would let mysqld_safe
# launch against an unbootstrapped data volume. `after rds-agent`, not `need`, so
# a failed fetch fails this oneshot visibly rather than skipping it.
description="Spinifex RDS MariaDB bootstrap"
AGENT_ENV="/etc/spinifex-rds/agent.env"
depend() {
need net
after rds-agent rds-datadir
before mariadb
}
start_pre() {
if [ -f "${AGENT_ENV}" ]; then
# shellcheck disable=SC1090
. "${AGENT_ENV}"
export RDS_HANDOFF_DIR RDS_DATA_MOUNT
fi
}
start() {
ebegin "Bootstrapping the RDS MariaDB instance"
# The VM has no operator login, so the host-side serial log is the only view
# of a bootstrap that failed before the engine came up.
/usr/local/sbin/rds-init > /dev/console 2>&1
eend $?
}
File diff suppressed because it is too large Load Diff
+74
View File
@@ -0,0 +1,74 @@
#!/bin/sh
set -eu
# The generated parameter files live on the data volume and reach the server
# through one baked include. Three properties make that work, and each fails
# silently rather than loudly if it breaks: the include must be read after every
# packaged drop-in, it must point at the mount rds-datadir uses, and it must be
# named so MariaDB reads it at all.
SCRIPT_DIR=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd)
INCLUDE="zz-rds-include.cnf"
FAILS=0
fail() { echo "FAIL: $*" >&2; FAILS=$((FAILS + 1)); }
pass() { echo "ok: $*"; }
# MariaDB reads only .cnf and .ini out of an include directory. Any other suffix
# would leave the customer's parameters silently unread.
case "${INCLUDE}" in
*.cnf) pass "the include is named so MariaDB reads it" ;;
*) fail "${INCLUDE} is not a name MariaDB reads out of an include directory" ;;
esac
if [ ! -f "${SCRIPT_DIR}/${INCLUDE}" ]; then
fail "${INCLUDE} is missing from the preset"
fi
if ! grep -q "${INCLUDE}:/etc/my.cnf.d/${INCLUDE}" "${SCRIPT_DIR}/manifest.conf"; then
fail "manifest.conf does not install ${INCLUDE} into /etc/my.cnf.d"
fi
# Alpine's own drop-in is the file this has to outsort: MariaDB walks the include
# directory in byte order and takes the last occurrence of a setting, so a digit
# prefix would sort ahead of it and let a distribution default beat the platform.
packaged="mariadb-server.cnf"
if [ "$(printf '%s\n%s\n' "${INCLUDE}" "${packaged}" | LC_ALL=C sort | tail -n 1)" = "${INCLUDE}" ]; then
pass "the include is read after ${packaged}"
else
fail "${packaged} is read after ${INCLUDE}, so packaged defaults win"
fi
# setup.sh asserts the same property against the real directory at build time.
grep -q 'LC_ALL=C sort | tail -n 1' "${SCRIPT_DIR}/setup.sh" ||
fail "setup.sh does not assert the include is the last drop-in"
include_dir=$(sed -n 's/^!includedir[[:space:]]*//p' "${SCRIPT_DIR}/${INCLUDE}")
if [ -z "${include_dir}" ]; then
fail "${INCLUDE} declares no !includedir"
else
# The generated files survive a VM replacement only by living on the data
# volume, so the include has to sit under the mount rds-datadir provides.
data_mount=$(sed -n 's/^[[:space:]]*RDS_DATA_MOUNT="\([^"]*\)".*/\1/p' "${SCRIPT_DIR}/rds-datadir.initd")
case "${include_dir}" in
"${data_mount}"/*) pass "the include target is on the data volume at ${include_dir}" ;;
*) fail "${include_dir} is not under the data mount ${data_mount:-<unset>}" ;;
esac
# An !includedir MariaDB cannot open is a fatal defaults-parsing error, which
# would take the client down alongside the server on a boot with no volume.
grep -q "install -d .* ${include_dir}\$" "${SCRIPT_DIR}/setup.sh" ||
fail "setup.sh does not bake ${include_dir} into the image"
fi
# The packaged drop-in disables TCP. Left in place, the endpoint resolves and the
# health probe passes while nothing on the customer ENI can ever connect.
grep -q 's/\^skip-networking\$/' "${SCRIPT_DIR}/setup.sh" ||
fail "setup.sh does not disable the packaged skip-networking"
grep -q "skip\[-_\]networking" "${SCRIPT_DIR}/setup.sh" ||
fail "setup.sh does not assert that no configuration file disables TCP"
if [ "${FAILS}" -eq 0 ]; then
echo "rds-mycnf: all tests passed"
exit 0
fi
exit 1
+68
View File
@@ -0,0 +1,68 @@
#!/bin/sh
set -eu
SCRIPT_DIR=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd)
WORK=$(mktemp -d)
trap 'rm -rf "${WORK}"' EXIT
capture_dependencies() {
_script=$1
(
need() { echo "need $*"; }
after() { echo "after $*"; }
before() { echo "before $*"; }
# shellcheck disable=SC1090
. "${SCRIPT_DIR}/${_script}"
depend
)
}
assert_line() {
_file=$1
_line=$2
grep -Fxq "${_line}" "${_file}" || {
echo "FAIL: ${_file} missing dependency: ${_line}" >&2
cat "${_file}" >&2
exit 1
}
}
capture_dependencies rds-agent.initd > "${WORK}/agent"
capture_dependencies rds-datadir.initd > "${WORK}/datadir"
capture_dependencies rds-init.initd > "${WORK}/init"
assert_line "${WORK}/agent" "need net"
assert_line "${WORK}/agent" "after localmount"
assert_line "${WORK}/agent" "before rds-datadir rds-init mariadb"
assert_line "${WORK}/datadir" "after localmount rds-agent"
assert_line "${WORK}/datadir" "before rds-init mariadb"
assert_line "${WORK}/init" "need net"
assert_line "${WORK}/init" "after rds-agent rds-datadir"
assert_line "${WORK}/init" "before mariadb"
if grep -Fq "before rds-agent" "${WORK}/datadir"; then
echo "FAIL: rds-datadir still starts before the bootstrap handoff" >&2
exit 1
fi
# Ordering alone is not enough for this engine: Alpine's mariadb service only
# warns about an empty datadir and starts anyway, so a failed rds-init would be
# answered by launching mysqld_safe against an unbootstrapped volume. OpenRC
# reads the dependency out of the service's conf.d file.
grep -Fq 'rc_need="rds-init"' "${SCRIPT_DIR}/mariadb.confd" || {
echo "FAIL: /etc/conf.d/mariadb does not make the engine need rds-init" >&2
exit 1
}
grep -Fq "mariadb.confd:/etc/conf.d/mariadb" "${SCRIPT_DIR}/manifest.conf" || {
echo "FAIL: manifest.conf does not install mariadb.confd" >&2
exit 1
}
for svc in rds-init mariadb; do
grep -qE "^ENABLE_SERVICES=.*[\" ]${svc}[\" ]" "${SCRIPT_DIR}/manifest.conf" || {
echo "FAIL: manifest.conf does not enable ${svc}" >&2
grep '^ENABLE_SERVICES' "${SCRIPT_DIR}/manifest.conf" >&2
exit 1
}
done
echo "rds-openrc: dependency order tests passed"
+122
View File
@@ -0,0 +1,122 @@
#!/bin/sh
set -eu
# setup.sh — guest customisation for the spinifex-rds-mariadb AMI. Runs inside
# the libguestfs appliance under build-system-image.sh, after packages and
# INSTALL_FILES are placed.
# INSTALL_FILES land 0644; OpenRC requires 0755 on init scripts, and rds-init
# and rds-datadir are executed directly by their services.
chmod 0755 /etc/init.d/rds-datadir /etc/init.d/rds-init /etc/init.d/rds-agent \
/usr/local/sbin/rds-datadir /usr/local/sbin/rds-init
chmod 0755 /etc/init.d/mulga-mgmt-net /etc/init.d/mulga-mgmt-net-routes \
/usr/local/sbin/mulga-mgmt-net
# mulga-mgmt-net goes in the boot runlevel, not default (where ENABLE_SERVICES
# lands services). It applies mgmt0's static address, which rds-agent needs to
# reach the gateway at all, and DHCPs the data NIC so the init-local Ec2 crawl
# reaches IMDS; a default entry runs after cloud-init-local and is too late.
rc-update add mulga-mgmt-net boot
# Where cloud-init drops the agent's env file and the gateway CA. Created here
# so the delivery lands in a root-only directory.
install -d -m 0700 /etc/spinifex-rds
# The engine this image bakes. rds-agent builds its engine implementation from
# this file, so running the wrong implementation is structurally impossible
# rather than merely validated against; a VM launched as another engine is
# refused before anything touches the datadir.
printf 'mariadb\n' > /etc/spinifex-rds/engine
chmod 0444 /etc/spinifex-rds/engine
# Empty in the image: rds-datadir mounts over it at boot. The package already
# creates it 0750 mysql:mysql; restated so a packaging change cannot loosen it.
install -d -m 0750 -o mysql -g mysql /var/lib/mysql
# The include target, baked so it exists even when the data volume is not
# mounted. MariaDB treats an !includedir it cannot open as a fatal error while
# parsing defaults, which would take down the client alongside the server on
# exactly the boot an operator needs both to diagnose the missing volume.
install -d -m 0750 -o mysql -g mysql /var/lib/mysql/conf.d
# The generated configuration must be read last, or a packaged default would win
# over a platform-owned setting. Sorted in byte order, the way MariaDB itself
# walks an include directory, so a future package adding a later-sorting file
# fails the build rather than silently overriding the platform.
last_dropin=$(ls /etc/my.cnf.d | LC_ALL=C sort | tail -n 1)
if [ "${last_dropin}" != "zz-rds-include.cnf" ]; then
echo "[rds-mariadb-setup] /etc/my.cnf.d/${last_dropin} is read after the RDS include"
exit 1
fi
# Alpine's packaged server drop-in sets skip-networking, which would leave the
# instance reachable only over its unix socket — the endpoint would resolve, the
# health probe would pass, and nothing on the customer ENI could ever connect.
# Disabled at the source rather than countered from a later file, so the image
# does not depend on override ordering for the endpoint to work at all.
sed -i 's/^skip-networking$/# skip-networking — disabled: the customer reaches this instance over TCP/' \
/etc/my.cnf.d/mariadb-server.cnf
if grep -rEq '^[[:space:]]*skip[-_]networking' /etc/my.cnf /etc/my.cnf.d; then
echo "[rds-mariadb-setup] a packaged configuration file still disables TCP networking"
grep -rEn '^[[:space:]]*skip[-_]networking' /etc/my.cnf /etc/my.cnf.d
exit 1
fi
# The resolved defaults checked against this image's own mariadbd. A name the
# server refuses at startup is not a rejected setting: it aborts the boot with the
# generated file already on the data volume. cmd/rds-agent keeps this file in step.
#
# --verbose --help parses the option files and exits without touching a datadir,
# so it is the whole check. defaults-extra-file rather than the include directory,
# which stays empty in the image; it must be the first option mariadbd sees.
default_params=/usr/local/share/spinifex-rds/default-parameters.cnf
if [ ! -f "${default_params}" ]; then
echo "[rds-mariadb-setup] the catalog's default parameter set was not delivered to ${default_params}"
exit 1
fi
if ! mariadbd --defaults-extra-file="${default_params}" --user=mysql --verbose --help \
>/dev/null 2>/tmp/rds-defaults.err; then
echo "[rds-mariadb-setup] mariadbd refuses the catalog's default parameter set"
cat /tmp/rds-defaults.err
exit 1
fi
# Build-time only: nothing at runtime reads it, and an include the server could
# find twice is a way for a stale copy to shadow the customer's own set.
rm -f /tmp/rds-defaults.err "${default_params}"
rmdir /usr/local/share/spinifex-rds 2>/dev/null || true
# The packaged service starts on a warning when the datadir is empty, so the
# dependency is what keeps it off an unbootstrapped volume. Asserted here, since
# an image whose engine can come up beside a failed bootstrap is the one failure
# this preset must not have.
grep -q '^rc_need="rds-init"' /etc/conf.d/mariadb || {
echo "[rds-mariadb-setup] /etc/conf.d/mariadb does not make the engine need rds-init"
exit 1
}
# rds-agent's health probe reads this pid to tell a server replaying its redo log
# from one that is not running at all. Nothing generated can move the path: the
# packaged service passes --pid-file on mysqld_safe's command line, which beats
# any option file. A package that renamed it would leave every instance reporting
# an engine that is not there, and the rollback guard restarting a healthy one.
grep -q '^pidfile="/run/mysqld/\$RC_SVCNAME.pid"$' /etc/init.d/mariadb &&
grep -q -- '--pid-file=\$pidfile' /etc/init.d/mariadb || {
echo "[rds-mariadb-setup] /etc/init.d/mariadb no longer starts the engine with its pidfile at /run/mysqld/mariadb.pid"
grep -nE 'pidfile|command_args' /etc/init.d/mariadb
exit 1
}
# Bind /dev/console to the serial port so userspace boot output reaches ttyS0,
# which the orchestrator captures host-side. Linux makes the last console= the
# controlling one, and stock Alpine lists tty0 last — reorder so ttyS0 wins.
sed -i \
's|console=ttyS0,115200n8 console=ttyAMA0,115200n8 console=tty0|console=tty0 console=ttyAMA0,115200n8 console=ttyS0,115200n8|' \
/etc/update-extlinux.conf /boot/extlinux.conf
# Cut the boot-menu countdown from 10s to ~1s, a fixed tax on every VM start.
# Both the generator config (seconds) and the rendered output (1/10s) are patched
# so a regenerate keeps it; a small nonzero keeps the menu interruptible.
sed -i 's/^timeout=.*/timeout=1/' /etc/update-extlinux.conf
sed -i 's/^TIMEOUT[[:space:]].*/TIMEOUT 10/' /boot/extlinux.conf
echo "[rds-mariadb-setup] done"
@@ -0,0 +1,19 @@
# /etc/my.cnf.d/zz-rds-include.cnf — Spinifex RDS generated-configuration include.
#
# The generated files live on the data volume rather than under /etc. A class
# change is a VM replacement: a fresh boot volume from this AMI with the same
# data volume reattached. Parameters under /etc would be discarded by that
# replacement and the instance would silently revert to catalog defaults while
# the API still reported the customer's values.
#
# zz- so this is the last drop-in the packaged !includedir reads. MariaDB walks
# an include directory in byte order and takes the last occurrence of a setting,
# which is what makes a platform-owned value beat a distribution default — and a
# digit prefix would sort ahead of the packaged mariadb-server.cnf rather than
# after it. setup.sh asserts the resulting order at build time.
#
# The directory is also baked empty into this image. MariaDB treats an
# !includedir it cannot open as a fatal defaults-parsing error, so a boot with no
# data volume mounted would otherwise take the client down with the server.
!includedir /var/lib/mysql/conf.d
+1 -1
View File
@@ -23,7 +23,7 @@ INSTALL_BINARIES="bin/rds-agent:/usr/local/bin/rds-agent"
# the guest, or mgmt0 never gets its static address and rds-agent cannot reach
# the gateway, and the customer ENI never comes up so nothing reaches the DB.
INSTALL_FILES="\
scripts/images/rds-postgres/rds-datadir:/usr/local/sbin/rds-datadir \
scripts/images/common/rds-datadir:/usr/local/sbin/rds-datadir \
scripts/images/rds-postgres/rds-datadir.initd:/etc/init.d/rds-datadir \
scripts/images/rds-postgres/rds-init:/usr/local/sbin/rds-init \
scripts/images/rds-postgres/rds-init.initd:/etc/init.d/rds-init \
+2 -2
View File
@@ -35,8 +35,8 @@ start_pre() {
# shellcheck disable=SC1090
. "${AGENT_ENV}"
export RDS_GATEWAY_URL RDS_GATEWAY_CA RDS_REGION RDS_DB_INSTANCE_IDENTIFIER \
RDS_ENGINE_VERSION RDS_HANDOFF_DIR RDS_ENGINE_HOST RDS_ENGINE_PORT \
RDS_PG_ISREADY RDS_POLL_WAIT RDS_DATA_MOUNT
RDS_ENGINE RDS_ENGINE_VERSION RDS_HANDOFF_DIR RDS_ENGINE_HOST RDS_ENGINE_PORT \
RDS_POLL_WAIT RDS_DATA_MOUNT
fi
}
@@ -9,7 +9,10 @@ source_time_handoff="${CASE}/source-time"
configured_handoff="${CASE}/configured"
mkdir -p "${configured_handoff}"
touch "${configured_handoff}/bootstrap.env"
printf 'RDS_HANDOFF_DIR=%s\n' "${configured_handoff}" >"${CASE}/agent.env"
{
printf 'RDS_HANDOFF_DIR=%s\n' "${configured_handoff}"
printf 'RDS_ENGINE=postgres\n'
} >"${CASE}/agent.env"
# Simulate OpenRC sourcing the service before start_pre loads agent.env.
export RDS_HANDOFF_DIR="${source_time_handoff}"
@@ -34,5 +37,12 @@ if [ "${EEND_STATUS}" != "0" ]; then
echo "FAIL: post-start wait did not find the configured handoff" >&2
exit 1
fi
# The agent checks this against the engine its own image bakes. Left out of the
# export list it would read as no assertion at all, so a VM launched as the
# wrong engine would bootstrap instead of refusing.
if [ "${RDS_ENGINE:-}" != "postgres" ]; then
echo "FAIL: start_pre did not export RDS_ENGINE from agent.env" >&2
exit 1
fi
echo "rds-agent.initd: all tests passed"
@@ -14,11 +14,16 @@ depend() {
}
start_pre() {
# This image's layout, stated here because the mount script is shared by
# every engine preset. A delivered RDS_DATA_MOUNT still wins, so the control
# plane can move the mount point without rebuilding the image.
RDS_DATA_MOUNT="/var/lib/postgresql"
RDS_ENGINE_USER="postgres"
if [ -f "${AGENT_ENV}" ]; then
# shellcheck disable=SC1090
. "${AGENT_ENV}"
export RDS_HANDOFF_DIR RDS_DATA_MOUNT
fi
export RDS_HANDOFF_DIR RDS_DATA_MOUNT RDS_ENGINE_USER
}
start() {
+37
View File
@@ -0,0 +1,37 @@
#!/bin/sh
set -eu
# The engine this preset bakes is stated in four places that have to agree: the
# stamp setup.sh writes, the manifest tag resolveEngineAMI selects on, the name
# rds-agent's layout table is keyed by, and the stamp rds-init puts on the data
# volume. A drift between them launches the wrong image, refuses the right one,
# or stamps a volume with an engine that did not write it.
SCRIPT_DIR=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd)
REPO_ROOT=$(CDPATH='' cd -- "${SCRIPT_DIR}/../../.." && pwd)
ENGINE="postgres"
stamped=$(sed -n "s|^printf '\\(.*\\)\\\\n' > /etc/spinifex-rds/engine\$|\\1|p" "${SCRIPT_DIR}/setup.sh")
if [ "${stamped}" != "${ENGINE}" ]; then
echo "FAIL: setup.sh stamps engine '${stamped}', want '${ENGINE}'" >&2
exit 1
fi
if ! grep -q "engine=${ENGINE} " "${SCRIPT_DIR}/manifest.conf"; then
echo "FAIL: manifest.conf SYSTEM_TAG does not carry engine=${ENGINE}" >&2
grep '^SYSTEM_TAG' "${SCRIPT_DIR}/manifest.conf" >&2
exit 1
fi
if ! grep -q "enginePostgres = \"${ENGINE}\"" "${REPO_ROOT}/cmd/rds-agent/engine.go"; then
echo "FAIL: rds-agent does not implement an engine named ${ENGINE}" >&2
exit 1
fi
if ! grep -q "^ENGINE=\"${ENGINE}\"\$" "${SCRIPT_DIR}/rds-init"; then
echo "FAIL: rds-init does not stamp the data volume '${ENGINE}'" >&2
grep '^ENGINE=' "${SCRIPT_DIR}/rds-init" >&2
exit 1
fi
echo "rds-engine-stamp: all tests passed"
+79
View File
@@ -26,6 +26,10 @@
# Nothing there survives a reboot: the next boot re-fetches and the cert is
# re-minted rather than persisted.
#
# The data volume also carries an engine stamp, written before the datadir is
# inspected and compared on every boot, so another engine's volume is refused
# rather than initialised over.
#
# rds-init keys off the datadir, never off RDS_MODE. An initialised datadir
# means skip initdb and only refresh config, so a replaced VM comes back on its
# existing data volume without re-bootstrapping. RDS_BOOTSTRAP_PENDING is not an
@@ -37,6 +41,11 @@ PG_VERSION="${RDS_PG_VERSION:-18}"
PG_BIN="${RDS_PG_BIN:-/usr/libexec/postgresql${PG_VERSION}}"
PG_USER="${RDS_PG_USER:-postgres}"
# The engine this image bakes, stamped onto the data volume and compared against
# it on every boot. Held here rather than read from /etc/spinifex-rds/engine:
# this script is the PostgreSQL half of the seam, so it is the same constant.
ENGINE="postgres"
# Mount point of the data volume, and the datadir one level inside it. The
# datadir is deliberately not the mount point itself: initdb refuses a non-empty
# directory and a freshly formatted ext4 volume already holds lost+found.
@@ -54,6 +63,11 @@ BOOTSTRAP_SENTINEL="${PGDATA}/rds-bootstrap-incomplete"
RECEIPT_DIR="${RDS_RECEIPT_DIR:-${DATA_MOUNT}/.spinifex-rds/bootstrap}"
RECEIPT_FILE="${RECEIPT_DIR}/receipt.env"
# Which engine's data this volume holds. Outside PGDATA for the same reason the
# receipt is, and beside it rather than inside the bootstrap directory: it
# outlives any one bootstrap and is read before the datadir is ever inspected.
ENGINE_STAMP="${RDS_ENGINE_STAMP:-${DATA_MOUNT}/.spinifex-rds/engine}"
HANDOFF_DIR="${RDS_HANDOFF_DIR:-/run/spinifex-rds}"
ENV_FILE="${HANDOFF_DIR}/bootstrap.env"
SOCKET_DIR="${RDS_SOCKET_DIR:-/run/postgresql}"
@@ -71,6 +85,10 @@ MOUNTS_FILE="${RDS_MOUNTS_FILE:-/proc/mounts}"
# AWS, so a customer's `GRANT rds_superuser TO ...` works here too.
MASTER_GROUP_ROLE="rds_superuser"
# NAMEDATALEN-1: the server truncates a longer identifier, so the database
# created would not be the one the customer asked for.
MAX_DB_NAME_LEN=63
log() { echo "[rds-init] $*"; }
fail() { echo "[rds-init] ERROR: $*" >&2; exit 1; }
@@ -134,6 +152,20 @@ load_handoff() {
;;
esac
# The same rule the control plane applies, for the same reason the reserved
# name is re-checked: an unusable name reaches initdb, fails at CREATE
# DATABASE and spends the one-shot password on the way. psql quotes the
# identifier, so here this is defence in depth rather than the barrier.
case "${RDS_DB_NAME}" in
'') ;;
[!A-Za-z]* | *[!A-Za-z0-9_]*)
fail "RDS_DB_NAME ${RDS_DB_NAME} must begin with a letter and hold only letters, digits and underscores; the control plane should have rejected it"
;;
esac
if [ "${#RDS_DB_NAME}" -gt "${MAX_DB_NAME_LEN}" ]; then
fail "RDS_DB_NAME is longer than the ${MAX_DB_NAME_LEN} characters PostgreSQL allows; the control plane should have rejected it"
fi
case "${RDS_PORT}" in
'' | *[!0-9]*) fail "bootstrap config has a non-numeric RDS_PORT: ${RDS_PORT}" ;;
esac
@@ -162,6 +194,52 @@ require_data_volume() {
log "warning: ${DATA_MOUNT} is not a mount point; data will not survive a VM replace (RDS_ALLOW_LOCAL_DATADIR=1)"
}
# Written before the datadir is inspected, so a volume this image goes on to
# initialise carries the stamp from the first boot onwards. Fatal on failure:
# an unstamped volume is one the next boot cannot tell from another engine's.
write_engine_stamp() {
_stamp_dir=$(dirname "${ENGINE_STAMP}")
install -d -m 0700 -o "${PG_USER}" -g "${PG_USER}" "${_stamp_dir}" ||
fail "the datadir engine stamp directory ${_stamp_dir} could not be created; refusing to use a data volume whose engine cannot be recorded"
_tmp="${ENGINE_STAMP}.new"
printf '%s\n' "${ENGINE}" > "${_tmp}" ||
fail "the datadir engine stamp ${ENGINE_STAMP} could not be written; refusing to use a data volume whose engine cannot be recorded"
chown "${PG_USER}:${PG_USER}" "${_tmp}"
chmod 0600 "${_tmp}"
# Renamed rather than written in place, so a crash leaves either the old
# stamp or none — and none is backfilled, where a torn one would be fatal.
sync "${_tmp}" 2>/dev/null || sync
mv -f "${_tmp}" "${ENGINE_STAMP}" ||
fail "the datadir engine stamp ${ENGINE_STAMP} could not be installed; refusing to use a data volume whose engine cannot be recorded"
sync "${_stamp_dir}" 2>/dev/null || sync
}
# The volume mounts cleanly whichever engine wrote it, and another engine's
# datadir reads as uninitialised here: rds-init would initdb beside the
# customer's data and serve an empty database that passes every health probe.
require_engine_stamp() {
if [ -e "${ENGINE_STAMP}" ]; then
_stamped=""
_stamp_read=0
_stamped=$(head -n 1 -- "${ENGINE_STAMP}") && _stamp_read=1
[ "${_stamp_read}" = "1" ] ||
fail "the datadir engine stamp ${ENGINE_STAMP} exists but could not be read; refusing to attach to a data volume whose engine cannot be established"
if [ "${_stamped}" != "${ENGINE}" ]; then
fail "the data volume holds a '${_stamped}' datadir but this image runs ${ENGINE}; refusing to touch it. Attach it to a '${_stamped}' instance, or recover the volume out of band."
fi
log "data volume is stamped ${ENGINE}"
return 0
fi
# No stamp is either a volume predating it or one about to be initialised.
# Both are ours here — PostgreSQL had unstamped volumes in the field, so a
# non-empty datadir is backfilled rather than refused.
log "data volume carries no engine stamp — recording ${ENGINE}"
write_engine_stamp
}
# Runs initdb into an empty datadir and lays down the cluster config written
# once: the include_dir hook and pg_hba.
initialise_cluster() {
@@ -565,6 +643,7 @@ consume_secrets() {
load_handoff
require_data_volume
require_engine_stamp
if [ -s "${PGDATA}/PG_VERSION" ]; then
# The engine would start cleanly on such a datadir and pass every health
+130 -5
View File
@@ -160,11 +160,15 @@ EOF
# sync stub: a no-op, except that SYNC_KILL_PARENT signals rds-init from the
# directory sync that follows the receipt rename. That is the one window where
# an installed receipt and an armed sweep coexist.
# an installed receipt and an armed sweep coexist. Matched on the receipt
# directory by name: the engine stamp syncs its directory too, long before the
# traps are armed, and killing there would leave the window untested.
cat > "${STUBBIN}/sync" <<'EOF'
#!/bin/sh
if [ "${SYNC_KILL_PARENT:-0}" = "1" ] && [ -d "${1:-}" ]; then
kill -TERM "$(cat "${KILL_PID_FILE}")"
if [ "${SYNC_KILL_PARENT:-0}" = "1" ]; then
case "${1:-}" in
*/bootstrap) kill -TERM "$(cat "${KILL_PID_FILE}")" ;;
esac
fi
exit 0
EOF
@@ -182,6 +186,7 @@ PGDATA="${DATA_MOUNT}/18/data"
SENTINEL="${PGDATA}/rds-bootstrap-incomplete"
RECEIPT_DIR="${DATA_MOUNT}/.spinifex-rds/bootstrap"
RECEIPT="${RECEIPT_DIR}/receipt.env"
STAMP="${DATA_MOUNT}/.spinifex-rds/engine"
HANDOFF="${WORK}/run/spinifex-rds"
MOUNTS="${WORK}/mounts"
KILL_PID_FILE="${WORK}/rds-init.pid"
@@ -207,7 +212,10 @@ write_handoff() {
# password carrying shell metacharacters — or a newline — survive the
# handoff intact and reach the checks that exist for it.
[ -n "$2" ] && echo "RDS_MASTER_PASSWORD='$2'"
[ -n "$3" ] && echo "RDS_DB_NAME=$3"
# Quoted as rds-agent's shellQuote writes it, so a name carrying a space
# or a shell metacharacter survives the handoff and reaches the check
# that exists for it rather than being mangled on the way.
[ -n "$3" ] && echo "RDS_DB_NAME='$3'"
[ -n "${PENDING:-}" ] && echo "RDS_BOOTSTRAP_PENDING=${PENDING}"
[ -n "${PAYLOAD_ID:-}" ] && echo "RDS_PAYLOAD_ID=${PAYLOAD_ID}"
# Last, and unconditional: an optional line failing its test would make
@@ -239,7 +247,7 @@ reset_state() {
: > "${PGCTL_CALLS}"
: > "${PSQL_CALLS}"
unset INITDB_FAIL PG_HBA_AS_DIR PSQL_FAIL LS_FAIL RDS_ALLOW_LOCAL_DATADIR MASTER_USER || true
unset PENDING PAYLOAD_ID DB_ID RECEIPT_DIR_OVERRIDE PSQL_STATUS_CORRUPT || true
unset PENDING PAYLOAD_ID DB_ID RECEIPT_DIR_OVERRIDE PSQL_STATUS_CORRUPT STAMP_OVERRIDE || true
}
# run_env: invoke a command with every path knob pointed into the temp dir.
@@ -252,6 +260,7 @@ run_env() {
RDS_LOG_DIR="${WORK}/log" \
RDS_MOUNTS_FILE="${MOUNTS}" \
RDS_RECEIPT_DIR="${RECEIPT_DIR_OVERRIDE:-${RECEIPT_DIR}}" \
RDS_ENGINE_STAMP="${STAMP_OVERRIDE:-${STAMP}}" \
RDS_ALLOW_LOCAL_DATADIR="${RDS_ALLOW_LOCAL_DATADIR:-0}" \
INITDB_FAIL="${INITDB_FAIL:-0}" \
PG_HBA_AS_DIR="${PG_HBA_AS_DIR:-0}" \
@@ -811,6 +820,122 @@ for reserved in postgres rds_superuser rdsadmin pg_toast_owner PostGres; do
unset MASTER_USER
done
# --- Case 10: an initial database name the control plane would have refused ---
# The name is interpolated into a CREATE DATABASE, and a failure there is a
# failure after initdb: the datadir is cleared and the one-shot password is
# gone. Refusing before initdb is what keeps the create retryable.
for badname in 'my-db' 'my db' 'my/db' 'my.db' '1db' \
'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'; do
reset_state
write_handoff initialize 's3cr3t' "${badname}"
run_fails "bad-dbname-${badname}"
[ -s "${INITDB_CALLS}" ] \
&& fail "bad-dbname-${badname}: initdb ran before the name was refused" \
|| pass "bad-dbname-${badname}: refused before initdb"
done
# The rule is a rejection of malformed names, not of every name: the boundary
# length and an underscore have to keep working.
reset_state
write_handoff initialize 's3cr3t' \
'a_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'
write_parameters
if run_ok "dbname-at-the-limit"; then
grep -q 'CREATE DATABASE' "${PSQL_CALLS}" \
&& pass "dbname-at-the-limit: accepted" || fail "dbname-at-the-limit: no CREATE DATABASE"
fi
# --- Case 11: the data volume records which engine wrote it ---
# Another engine's datadir mounts cleanly and reads as uninitialised here, so
# without the stamp rds-init would initdb beside the customer's data and serve
# an empty database that passes every health probe while the real data sits
# unreferenced — until the first automated snapshot captures the wrong one.
reset_state
write_handoff initialize 's3cr3t' appdb
if run_ok "stamp-absent-empty"; then
[ "$(cat "${STAMP}" 2>/dev/null)" = "postgres" ] \
&& pass "stamp-absent-empty: fresh volume stamped postgres" \
|| fail "stamp-absent-empty: no stamp written at initialisation"
case "${STAMP}" in
"${PGDATA}"/*) fail "stamp-absent-empty: written inside the datadir the traps clear" ;;
*) pass "stamp-absent-empty: written outside PGDATA" ;;
esac
# --- Case 11a: a matching stamp attaches ---
: > "${INITDB_CALLS}"
write_handoff attach '' appdb
if run_ok "stamp-match"; then
[ -s "${INITDB_CALLS}" ] \
&& fail "stamp-match: re-ran initdb" || pass "stamp-match: attached"
fi
# --- Case 11b: a disagreeing stamp is fatal and touches nothing ---
echo 'customer table data' > "${PGDATA}/base-relation"
printf 'mariadb\n' > "${STAMP}"
: > "${INITDB_CALLS}"
run_fails "stamp-mismatch"
[ -s "${INITDB_CALLS}" ] \
&& fail "stamp-mismatch: initdb ran on another engine's volume" \
|| pass "stamp-mismatch: no initdb"
[ -f "${PGDATA}/base-relation" ] \
&& pass "stamp-mismatch: datadir preserved" || fail "stamp-mismatch: datadir cleared"
[ "$(cat "${STAMP}")" = "mariadb" ] \
&& pass "stamp-mismatch: the other engine's stamp is left as it stands" \
|| fail "stamp-mismatch: stamp overwritten"
grep -q "holds a 'mariadb' datadir" "${WORK}/out" \
&& pass "stamp-mismatch: refusal names both engines" \
|| fail "stamp-mismatch: no refusal naming the stamped engine"
# --- Case 11c: a stamp that cannot be read is not an absent one ---
# Reading it as absent would backfill our own engine over a volume whose
# engine was never established, turning the check into a rubber stamp.
rm -f "${STAMP}"
mkdir "${STAMP}"
run_fails "stamp-unreadable"
grep -q 'could not be read' "${WORK}/out" \
&& pass "stamp-unreadable: refusal names the unreadable stamp" \
|| fail "stamp-unreadable: no refusal message"
rmdir "${STAMP}"
# --- Case 11c2: a zero-length stamp does not read as a match ---
: > "${STAMP}"
run_fails "stamp-empty"
[ -f "${PGDATA}/base-relation" ] \
&& pass "stamp-empty: datadir preserved" || fail "stamp-empty: datadir cleared"
# --- Case 11d: an unstamped datadir with data in it is backfilled ---
# PostgreSQL volumes predate the stamp, so the check becomes total over time
# rather than refusing every instance created before it existed.
rm -f "${STAMP}"
: > "${INITDB_CALLS}"
if run_ok "stamp-absent-nonempty"; then
[ "$(cat "${STAMP}" 2>/dev/null)" = "postgres" ] \
&& pass "stamp-absent-nonempty: existing datadir gains a stamp" \
|| fail "stamp-absent-nonempty: stamp not backfilled"
[ -s "${INITDB_CALLS}" ] \
&& fail "stamp-absent-nonempty: re-ran initdb" || pass "stamp-absent-nonempty: attached"
[ -f "${PGDATA}/base-relation" ] \
&& pass "stamp-absent-nonempty: datadir preserved" \
|| fail "stamp-absent-nonempty: datadir cleared"
fi
fi
# --- Case 11e: a stamp that cannot be written is fatal, before initdb ---
# The refusal is before the one-shot password is spent, so the create is still
# retryable; continuing would leave a volume the next boot cannot identify.
reset_state
write_handoff initialize 's3cr3t' appdb
: > "${WORK}/not-a-dir-stamp"
STAMP_OVERRIDE="${WORK}/not-a-dir-stamp/engine"
export STAMP_OVERRIDE
run_fails "stamp-unwritable"
[ -s "${INITDB_CALLS}" ] \
&& fail "stamp-unwritable: initdb ran before the stamp was recorded" \
|| pass "stamp-unwritable: refused before initdb"
grep -q 'engine stamp' "${WORK}/out" \
&& pass "stamp-unwritable: refusal names the stamp" || fail "stamp-unwritable: no refusal message"
unset STAMP_OVERRIDE
if [ "${FAILS}" -eq 0 ]; then
echo "PASS: all rds-init cases"
exit 0
+7
View File
@@ -22,6 +22,13 @@ rc-update add mulga-mgmt-net boot
# so the delivery lands in a root-only directory.
install -d -m 0700 /etc/spinifex-rds
# The engine this image bakes. rds-agent builds its engine implementation from
# this file, so running the wrong implementation is structurally impossible
# rather than merely validated against; a VM launched as another engine is
# refused before anything touches the datadir.
printf 'postgres\n' > /etc/spinifex-rds/engine
chmod 0444 /etc/spinifex-rds/engine
# Empty in the image: rds-datadir mounts over it at boot. Postgres-owned so the
# engine can traverse it if the volume's own root is stricter.
install -d -m 0750 -o postgres -g postgres /var/lib/postgresql
+2 -1
View File
@@ -44,7 +44,7 @@ func (s *Service) CreateDBInstance(ctx context.Context, input *rds.CreateDBInsta
// does not exist leaves no record behind. The set is literals only: the
// class's memory has already been folded into every size-derived default, so
// the agent never sees a formula.
parameters, err := s.resolveGroupParameters(ctx, kv, accountID, req.DBParameterGroupName, req.InstanceClass)
parameters, err := s.resolveGroupParameters(ctx, kv, accountID, req.Engine, req.DBParameterGroupName, req.InstanceClass)
if err != nil {
return nil, err
}
@@ -99,6 +99,7 @@ func (s *Service) CreateDBInstance(ctx context.Context, input *rds.CreateDBInsta
GatewayCACert: s.deps.GatewayCACert,
Region: s.region,
DBInstanceIdentifier: req.Identifier,
Engine: req.Engine.Name,
EngineVersion: req.EngineVersion,
EnginePort: req.Port,
}),
+4
View File
@@ -594,6 +594,10 @@ func TestValidateCreateRequest_RejectsMalformedRequests(t *testing.T) {
{"PortTooLow", func(in *rds.CreateDBInstanceInput) { in.Port = aws.Int64(80) }, awserrors.ErrorInvalidParameterValue},
{"ReservedUsername", func(in *rds.CreateDBInstanceInput) { in.MasterUsername = aws.String("rdsadmin") }, awserrors.ErrorInvalidParameterValue},
{"ShortPassword", func(in *rds.CreateDBInstanceInput) { in.MasterUserPassword = aws.String("short") }, awserrors.ErrorInvalidParameterValue},
{"DBNameHyphen", func(in *rds.CreateDBInstanceInput) { in.DBName = aws.String("my-db") }, awserrors.ErrorInvalidParameterValue},
{"DBNameTooLong", func(in *rds.CreateDBInstanceInput) {
in.DBName = aws.String(strings.Repeat("a", 64))
}, awserrors.ErrorInvalidParameterValue},
}
for _, tc := range cases {
+105 -3
View File
@@ -23,10 +23,62 @@ type Engine struct {
reservedUsernames []string
// Prefixes the engine reserves for internal roles.
reservedUsernamePrefixes []string
// The engine's own identifier limit for a role name. The character rule is
// shared across engines; the length is not.
maxUsernameLen int
// The engine's rule for an initial database name, which the guest
// interpolates into a CREATE DATABASE.
validateDBName func(string) error
// The engine's parameter table, keyed by parameter name. The generic spec
// machinery is shared; only the table and its formulas are per-engine.
catalog map[string]ParameterSpec
// Cross-parameter checks a resolved set must satisfy, which are the
// combinations the engine itself would refuse to start under.
validateCombinations func([]Parameter) error
// What a snapshot taken without a quiesce actually recovers on restore, which
// is the engine's own guarantee and not a shared one.
crashRecoveryNote string
// The same guarantee stated for the next start rather than for a restore,
// which is what an engine that would not shut down cleanly gets.
uncleanStopNote string
}
var engines = map[string]Engine{
enginePostgres.Name: enginePostgres,
engineMariaDB.Name: engineMariaDB,
}
// The same registry keyed by parameter-group family, for the callers that hold
// only a family string and have no instance to derive an engine from. Family
// and engine are 1:1 by construction, so one registry serves both.
var enginesByFamily = indexEnginesByFamily()
func indexEnginesByFamily() map[string]Engine {
out := make(map[string]Engine, len(engines))
for _, engine := range engines {
// An engine registered without its own name rule would panic on the first
// create that names a database, rather than here where every build sees it.
if engine.validateDBName == nil {
panic("rds: engine " + engine.Name + " registers no DBName rule")
}
// An engine without these would tell a customer nothing about what a
// crash-consistent snapshot or an unclean stop of it recovers, on the two
// events that say so.
if engine.crashRecoveryNote == "" {
panic("rds: engine " + engine.Name + " registers no crash-recovery note")
}
if engine.uncleanStopNote == "" {
panic("rds: engine " + engine.Name + " registers no unclean-stop note")
}
family := engine.ParameterGroupFamily()
if _, exists := out[family]; exists {
panic("rds: parameter group family " + family + " is claimed by two engines")
}
out[family] = engine
}
return out
}
// The pinned version an AMI lookup resolves against, which is the major alone:
@@ -62,6 +114,27 @@ func SupportedEngines() []string {
return slices.Sorted(maps.Keys(engines))
}
// The engine a parameter group's family belongs to, for the paths that hold a
// group and no instance. A family naming no engine is a group written by a build
// that offered an engine this one does not.
func engineForFamily(family string) (Engine, error) {
engine, ok := enginesByFamily[normaliseFamily(family)]
if !ok {
return Engine{}, awserrors.Errorf(awserrors.ErrorInvalidParameterValue,
"DBParameterGroupFamily %q is not offered; supported families are %s",
family, strings.Join(SupportedParameterGroupFamilies(), ", "))
}
return engine, nil
}
func SupportedParameterGroupFamilies() []string {
return slices.Sorted(maps.Keys(enginesByFamily))
}
func normaliseFamily(family string) string {
return strings.ToLower(strings.TrimSpace(family))
}
// An empty version takes the pin. A supplied one must name the pinned major
// exactly, since the image does not promise any particular minor version.
func (e Engine) ValidateVersion(version string) error {
@@ -78,9 +151,9 @@ func (e Engine) ValidateMasterUsername(username string) error {
if username == "" {
return awserrors.Errorf(awserrors.ErrorInvalidParameterValue, "MasterUsername is required")
}
if len(username) > maxMasterUsernameLen {
if len(username) > e.maxUsernameLen {
return awserrors.Errorf(awserrors.ErrorInvalidParameterValue,
"MasterUsername must be at most %d characters", maxMasterUsernameLen)
"MasterUsername must be at most %d characters", e.maxUsernameLen)
}
if !isLetter(rune(username[0])) {
return awserrors.Errorf(awserrors.ErrorInvalidParameterValue, "MasterUsername must begin with a letter")
@@ -112,6 +185,36 @@ func (e Engine) ValidateUsernameNotReserved(username string) error {
return nil
}
// The initial database, which AWS leaves optional: an empty name creates no
// database at all rather than one named by default.
func (e Engine) ValidateDBName(name string) error {
return e.validateDBName(name)
}
// The shared rule, taking each engine's own identifier limit. The character set
// is narrower than either engine accepts, because it is also what makes the name
// safe to interpolate into the CREATE DATABASE rds-init builds inside the guest.
func dbNameRule(maxLen int) func(string) error {
return func(name string) error {
switch {
case name == "":
return nil
case len(name) > maxLen:
return awserrors.Errorf(awserrors.ErrorInvalidParameterValue,
"DBName must be at most %d characters", maxLen)
case !isLetter(rune(name[0])):
return awserrors.Errorf(awserrors.ErrorInvalidParameterValue, "DBName must begin with a letter")
}
for _, r := range name {
if !isLetter(r) && !isDigit(r) && r != '_' {
return awserrors.Errorf(awserrors.ErrorInvalidParameterValue,
"DBName may contain only letters, digits and underscores")
}
}
return nil
}
}
// Bounds and the printable-ASCII range AWS accepts. The password is never
// inspected beyond this and never stored in cleartext past the first bootstrap
// fetch.
@@ -140,7 +243,6 @@ func ValidateMasterUserPassword(password string) error {
}
const (
maxMasterUsernameLen = 63
minMasterPasswordLen = 8
maxMasterPasswordLen = 128
)
+507
View File
@@ -0,0 +1,507 @@
package handlers_rds
import (
"slices"
"strconv"
"strings"
"time"
"github.com/mulgadc/spinifex/spinifex/awserrors"
)
// MariaDB 11.8 is the current LTS series and the only MariaDB in the pinned
// Alpine base. MajorVersion is the version series the API pins to rather than an
// integer major — PostgreSQL's version axis is one integer and MariaDB's is two
// — so the family reads mariadb11.8, as AWS RDS's own family naming does.
var engineMariaDB = Engine{
Name: "mariadb",
MajorVersion: "11.8",
DefaultPort: 3306,
// root is the superuser mariadb-install-db creates and rds-init keeps for
// itself, mariadb.sys owns the sys schema views, mysql is created as
// mysql@localhost under unix_socket auth, rdsadmin is AWS's management role,
// and PUBLIC is a role name the server refuses to create a user under.
reservedUsernames: []string{"root", "mariadb.sys", "mysql", "rdsadmin", "public"},
reservedUsernamePrefixes: []string{"mysql.", "mariadb."},
// The engine's own limit rather than AWS's documented 16. A client written
// against real AWS stays inside 16 either way, so this accepts a strict
// superset; 16 would refuse a 20-character name the server handles fine.
maxUsernameLen: 80,
// MariaDB maps a database name onto a directory name and its client has no
// identifier interpolation, so rds-init builds CREATE DATABASE by shell
// interpolation and this rule is the barrier rather than defence in depth.
validateDBName: dbNameRule(64),
catalog: mariadbParameterCatalog,
validateCombinations: validateMariaDBParameterCombinations,
// Only InnoDB keeps a redo log. Aria and MyISAM have none, so a datadir torn
// mid-write can leave one of their tables inconsistent with no way back.
crashRecoveryNote: "InnoDB tables will recover from the redo log when it is restored; " +
"non-transactional tables such as Aria and MyISAM may be left inconsistent.",
uncleanStopNote: "InnoDB tables will recover from the redo log on the next start; " +
"non-transactional tables such as Aria and MyISAM may be left inconsistent.",
}
// The curated MariaDB 11.8 table, the same shape and approximate size as the
// PostgreSQL one: a genuinely validated subset rather than the engine's full
// variable set, because a static parameter the server refuses at startup is a
// boot loop with the bad config already on the data volume.
//
// The platform-owned settings (port, datadir, socket, bind_address, the ssl_*
// family, secure_file_priv, log_bin, default_storage_engine and the InnoDB
// buffer pool chunk and instance counts) are deliberately absent rather than
// present-and-unmodifiable, because they are not the customer's to set. The
// chunk settings are also load-bearing: the size-derived buffer pool default and
// the guest's file-comparison pending-restart both need that granularity fixed.
var mariadbParameterCatalog = withMariaDBBooleanSpellings(buildParameterCatalog(
// Connections and threads. max_connections is what a size-derived default
// matters most for: RDS's own formula is {DBInstanceClassMemory/12582880}.
ParameterSpec{
Name: "max_connections", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 10, Max: 100000,
DefaultFor: mariadbMaxConnectionsFor,
MaxFor: mariadbMaxConnectionsCeilingFor,
Description: "Maximum number of concurrent client connections.",
},
ParameterSpec{
Name: "max_connect_errors", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 1, Max: 4294967295, Default: "100",
Description: "Successive interrupted connections from a host before it is blocked.",
},
ParameterSpec{
Name: "max_allowed_packet", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 1024, Max: 1073741824, Default: "16777216", Unit: "B",
Description: "Maximum size of one packet or generated string, in bytes.",
},
ParameterSpec{
Name: "thread_cache_size", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 16384, Default: "256",
Description: "Threads the server keeps cached for reuse by new connections.",
},
ParameterSpec{
Name: "wait_timeout", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 1, Max: 31536000, Default: "28800", Unit: "s",
Description: "Seconds the server waits for activity on a non-interactive connection before closing it.",
},
ParameterSpec{
Name: "interactive_timeout", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 1, Max: 31536000, Default: "28800", Unit: "s",
Description: "Seconds the server waits for activity on an interactive connection before closing it.",
},
ParameterSpec{
Name: "connect_timeout", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 2, Max: 31536000, Default: "10", Unit: "s",
Description: "Seconds the server waits for a connection packet before responding with a bad handshake.",
},
ParameterSpec{
Name: "net_read_timeout", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 1, Max: 31536000, Default: "30", Unit: "s",
Description: "Seconds to wait for more data from a connection before aborting the read.",
},
ParameterSpec{
Name: "net_write_timeout", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 1, Max: 31536000, Default: "60", Unit: "s",
Description: "Seconds to wait for a block to be written to a connection before aborting the write.",
},
// Locks and statement timeouts. lock_wait_timeout also bounds how long a
// snapshot quiesce can hold the server, so its default is the engine's.
ParameterSpec{
Name: "lock_wait_timeout", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 1, Max: 31536000, Default: "86400", Unit: "s",
Description: "Seconds to wait for a metadata lock before failing the statement.",
},
ParameterSpec{
Name: "innodb_lock_wait_timeout", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 1, Max: 1073741824, Default: "50", Unit: "s",
Description: "Seconds an InnoDB transaction waits for a row lock before it is rolled back.",
},
ParameterSpec{
Name: "max_statement_time", DataType: paramTypeReal, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 31536000, Default: "0", Unit: "s",
Description: "Seconds a statement may run before it is aborted; 0 disables.",
},
ParameterSpec{
Name: "idle_transaction_timeout", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 31536000, Default: "0", Unit: "s",
Description: "Seconds an idle open transaction may live before its connection is killed; 0 disables.",
},
// InnoDB. The buffer pool is three quarters of class memory, which is RDS's
// own default and the setting whose being wrong makes the guest unbootable.
ParameterSpec{
Name: "innodb_buffer_pool_size", DataType: paramTypeInteger, ApplyType: ApplyTypeStatic,
IsModifiable: true, Min: 5242880, Max: 1099511627776, Unit: "B",
DefaultFor: innodbBufferPoolSizeFor,
MaxFor: innodbBufferPoolSizeCeilingFor,
Description: "Memory InnoDB caches table and index data in, in bytes.",
},
ParameterSpec{
Name: "innodb_log_file_size", DataType: paramTypeInteger, ApplyType: ApplyTypeStatic,
IsModifiable: true, Min: 4194304, Max: 549755813888, Default: "100663296", Unit: "B",
Description: "Size of the InnoDB redo log, in bytes.",
},
ParameterSpec{
Name: "innodb_flush_method", DataType: paramTypeEnum, ApplyType: ApplyTypeStatic,
IsModifiable: true,
Enum: []string{"fsync", "o_dsync", "littlesync", "nosync", "o_direct", "o_direct_no_fsync"},
Default: "o_direct",
Description: "How InnoDB flushes data and log files to the data volume.",
},
ParameterSpec{
Name: "innodb_flush_log_at_trx_commit", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 2, Default: "1",
Description: "When the redo log is written and flushed; 1 flushes at every commit.",
},
ParameterSpec{
Name: "innodb_read_io_threads", DataType: paramTypeInteger, ApplyType: ApplyTypeStatic,
IsModifiable: true, Min: 1, Max: 64, Default: "4",
Description: "Threads InnoDB uses for read prefetch.",
},
ParameterSpec{
Name: "innodb_write_io_threads", DataType: paramTypeInteger, ApplyType: ApplyTypeStatic,
IsModifiable: true, Min: 1, Max: 64, Default: "4",
Description: "Threads InnoDB uses to write from the buffer pool.",
},
ParameterSpec{
Name: "innodb_purge_threads", DataType: paramTypeInteger, ApplyType: ApplyTypeStatic,
IsModifiable: true, Min: 1, Max: 32, Default: "4",
Description: "Threads InnoDB uses to purge the undo history.",
},
ParameterSpec{
Name: "innodb_autoinc_lock_mode", DataType: paramTypeInteger, ApplyType: ApplyTypeStatic,
IsModifiable: true, Min: 0, Max: 2, Default: "1",
Description: "Locking mode for AUTO_INCREMENT allocation; 1 is consecutive.",
},
ParameterSpec{
Name: "innodb_io_capacity", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 100, Max: 4294967295, Default: "200",
Description: "I/O operations per second InnoDB assumes it may use for background work.",
},
ParameterSpec{
Name: "innodb_io_capacity_max", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 100, Max: 4294967295, Default: "2000",
Description: "I/O operations per second InnoDB may use when it is falling behind.",
},
ParameterSpec{
Name: "innodb_max_dirty_pages_pct", DataType: paramTypeReal, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 99.999, Default: "90",
Description: "Percentage of dirty pages in the buffer pool InnoDB flushes to stay under.",
},
ParameterSpec{
Name: "innodb_flush_neighbors", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 2, Default: "1",
Description: "Whether flushing a page also flushes its neighbours in the same extent.",
},
ParameterSpec{
Name: "innodb_file_per_table", DataType: paramTypeBoolean, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Default: "on",
Description: "Whether each InnoDB table gets its own tablespace file.",
},
ParameterSpec{
Name: "innodb_stats_persistent", DataType: paramTypeBoolean, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Default: "on",
Description: "Whether InnoDB optimizer statistics survive a restart.",
},
ParameterSpec{
Name: "innodb_strict_mode", DataType: paramTypeBoolean, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Default: "on",
Description: "Whether InnoDB raises errors rather than warnings for invalid table options.",
},
ParameterSpec{
Name: "innodb_adaptive_hash_index", DataType: paramTypeBoolean, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Default: "off",
Description: "Whether InnoDB maintains an in-memory hash index over frequently read pages.",
},
// Server memory outside InnoDB. Every per-session buffer here is allocated
// per connection, so a literal tuned for the large end multiplies by
// max_connections on the small one.
ParameterSpec{
Name: "sort_buffer_size", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 1024, Max: 4294967295, Default: "2097152", Unit: "B",
Description: "Per-session buffer for a sort that cannot use an index, in bytes.",
},
ParameterSpec{
Name: "join_buffer_size", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 128, Max: 4294967295, Default: "262144", Unit: "B",
Description: "Per-session buffer for a join that cannot use an index, in bytes.",
},
ParameterSpec{
Name: "read_buffer_size", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 8192, Max: 2147479552, Default: "131072", Unit: "B",
Description: "Per-session buffer for a sequential table scan, in bytes.",
},
ParameterSpec{
Name: "read_rnd_buffer_size", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 8192, Max: 2147483647, Default: "262144", Unit: "B",
Description: "Per-session buffer for reading rows in sorted order, in bytes.",
},
ParameterSpec{
Name: "tmp_table_size", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 1024, Max: 4294967295, Default: "16777216", Unit: "B",
Description: "Size an internal in-memory temporary table may reach before it is written to disk, in bytes.",
},
ParameterSpec{
Name: "max_heap_table_size", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 16384, Max: 4294967295, Default: "16777216", Unit: "B",
Description: "Maximum size of a user-created MEMORY table, in bytes.",
},
ParameterSpec{
Name: "key_buffer_size", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 4294967295, Default: "134217728", Unit: "B",
Description: "Buffer for MyISAM and Aria index blocks, in bytes.",
},
ParameterSpec{
Name: "table_open_cache", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 1, Max: 1048576, Default: "2000",
Description: "Open table handles the server caches across all connections.",
},
ParameterSpec{
Name: "table_definition_cache", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 400, Max: 2097152, Default: "400",
Description: "Table definitions the server caches.",
},
ParameterSpec{
Name: "max_prepared_stmt_count", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 1048576, Default: "16382",
Description: "Prepared statements the server allows across all connections.",
},
ParameterSpec{
Name: "group_concat_max_len", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 4, Max: 4294967295, Default: "1048576", Unit: "B",
Description: "Maximum length of a GROUP_CONCAT result, in bytes.",
},
// Off by default as MariaDB itself ships it: the instrumentation costs memory
// a small class does not have, and it cannot be turned on without a restart.
ParameterSpec{
Name: "performance_schema", DataType: paramTypeBoolean, ApplyType: ApplyTypeStatic,
IsModifiable: true, Default: "off",
Description: "Whether the Performance Schema instrumentation is enabled.",
},
// Logging. The destination and the log files are platform-owned; what is
// logged is the customer's.
ParameterSpec{
Name: "slow_query_log", DataType: paramTypeBoolean, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Default: "off",
Description: "Whether statements slower than long_query_time are logged.",
},
ParameterSpec{
Name: "long_query_time", DataType: paramTypeReal, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 31536000, Default: "10", Unit: "s",
Description: "Seconds a statement must run to reach the slow query log.",
},
ParameterSpec{
Name: "log_queries_not_using_indexes", DataType: paramTypeBoolean, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Default: "off",
Description: "Whether a statement that examines no index reaches the slow query log.",
},
ParameterSpec{
Name: "general_log", DataType: paramTypeBoolean, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Default: "off",
Description: "Whether every connection and statement is logged.",
},
ParameterSpec{
Name: "log_output", DataType: paramTypeEnum, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Enum: []string{"file", "table", "none"}, Default: "file",
Description: "Where the general and slow query logs are written.",
},
ParameterSpec{
Name: "log_warnings", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 4294967295, Default: "2",
Description: "Verbosity of the error log; 0 logs errors only.",
},
// SQL behaviour.
// The default deliberately omits NO_AUTO_CREATE_USER, which MariaDB has been
// retiring: the platform's own default has to be one every supported patch
// release parses, or a routine image rebuild becomes a boot loop.
ParameterSpec{
Name: "sql_mode", DataType: paramTypeString, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Validate: validateMariaDBSQLMode,
Default: "STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION",
Description: "Comma-separated SQL modes the server enforces.",
},
ParameterSpec{
Name: "character_set_server", DataType: paramTypeEnum, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Enum: []string{"latin1", "utf8mb3", "utf8mb4"}, Default: "utf8mb4",
Description: "Default character set for new databases.",
},
ParameterSpec{
Name: "collation_server", DataType: paramTypeEnum, ApplyType: ApplyTypeDynamic,
IsModifiable: true,
Enum: []string{
"latin1_bin", "latin1_swedish_ci",
"utf8mb3_bin", "utf8mb3_general_ci",
"utf8mb4_bin", "utf8mb4_general_ci", "utf8mb4_unicode_ci",
},
Default: "utf8mb4_general_ci",
Description: "Default collation for new databases, which must belong to character_set_server.",
},
// The image loads no time zone tables, so a named zone would resolve to
// nothing; SYSTEM and a fixed offset are what the server can actually apply.
//
// MariaDB has no time_zone startup option — the name only exists for SET — so
// an option file naming it aborts the server before it opens the datadir.
// default_time_zone is the same setting spelt the way startup accepts.
ParameterSpec{
Name: "time_zone", DataType: paramTypeString, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Default: "SYSTEM", Validate: validateMariaDBTimeZone,
optionFileName: "default_time_zone",
Description: "Server time zone, as SYSTEM or a fixed offset such as +10:00.",
},
ParameterSpec{
Name: "transaction_isolation", DataType: paramTypeEnum, ApplyType: ApplyTypeDynamic,
IsModifiable: true,
Enum: []string{"read-uncommitted", "read-committed", "repeatable-read", "serializable"},
Default: "repeatable-read",
Description: "Default transaction isolation level for new connections.",
},
ParameterSpec{
Name: "autocommit", DataType: paramTypeBoolean, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Default: "on",
Description: "Whether each statement outside an explicit transaction commits on its own.",
},
ParameterSpec{
Name: "event_scheduler", DataType: paramTypeEnum, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Enum: []string{"on", "off"}, Default: "off",
Description: "Whether the scheduled event executor thread runs.",
},
))
// MariaDB parses 0, 1, ON, OFF, TRUE and FALSE for a boolean system variable and
// refuses yes and no, which PostgreSQL accepts. Narrowing the whole catalog in
// one place keeps every boolean spec free of the distinction.
var mariadbBooleanSpellings = []string{"on", "off", "true", "false", "1", "0"}
func withMariaDBBooleanSpellings(catalog map[string]ParameterSpec) map[string]ParameterSpec {
for name, spec := range catalog {
if spec.DataType == paramTypeBoolean {
spec.Enum = mariadbBooleanSpellings
catalog[name] = spec
}
}
return catalog
}
// The InnoDB buffer pool ceiling, above the three-quarters default but below the
// whole guest: the pool is a single allocation the kernel will not overcommit
// its way out of, so a value at class memory is a server that never starts.
const bufferPoolCeilingNumerator, bufferPoolCeilingDenominator = 7, 8
// innodb_buffer_pool_size = {DBInstanceClassMemory*3/4}, in bytes, which is
// RDS's own default for this engine family.
func innodbBufferPoolSizeFor(memoryMiB int64) string {
return strconv.FormatInt(clampInt64(memoryMiB*mibToBytes/4*3, minBufferPoolBytes, maxBufferPoolBytes), 10)
}
func innodbBufferPoolSizeCeilingFor(memoryMiB int64) int64 {
ceiling := memoryMiB * mibToBytes / bufferPoolCeilingDenominator * bufferPoolCeilingNumerator
return clampInt64(ceiling, minBufferPoolBytes, maxBufferPoolBytes)
}
const (
minBufferPoolBytes = 5242880
maxBufferPoolBytes = 1099511627776
)
// max_connections = {DBInstanceClassMemory/12582880}. The floor keeps the
// smallest class above what the platform's own health probing, parameter apply
// and password rotation sessions need before a customer connects at all.
func mariadbMaxConnectionsFor(memoryMiB int64) string {
return strconv.FormatInt(mariadbMaxConnections(memoryMiB), 10)
}
func mariadbMaxConnectionsCeilingFor(memoryMiB int64) int64 {
return clampInt64(mariadbMaxConnections(memoryMiB)*4, 20, 100000)
}
func mariadbMaxConnections(memoryMiB int64) int64 {
return clampInt64(memoryMiB*mibToBytes/12582880, 20, 100000)
}
// The modes MariaDB 11.8 parses, including the combination modes that expand to
// several. A mode the server does not know is a startup failure, so the list is
// closed rather than a syntax check.
var mariadbSQLModes = []string{
"allow_invalid_dates", "ansi", "ansi_quotes", "db2", "empty_string_is_null",
"error_for_division_by_zero", "high_not_precedence", "ignore_bad_table_options",
"ignore_space", "maxdb", "mssql", "mysql323", "mysql40", "no_auto_value_on_zero",
"no_backslash_escapes", "no_dir_in_create", "no_engine_substitution",
"no_unsigned_subtraction", "no_zero_date", "no_zero_in_date", "only_full_group_by",
"oracle", "pad_char_to_full_length", "pipes_as_concat", "postgresql", "real_as_float",
"simultaneous_assignment", "strict_all_tables", "strict_trans_tables",
"time_round_fractional", "traditional",
}
func validateMariaDBSQLMode(value string) error {
for mode := range strings.SplitSeq(value, ",") {
mode = strings.ToLower(strings.TrimSpace(mode))
if mode == "" || !slices.Contains(mariadbSQLModes, mode) {
return awserrors.Errorf(awserrors.ErrorInvalidParameterValue,
"parameter sql_mode does not accept %q; it takes a comma-separated list of MariaDB SQL mode names", value)
}
}
return nil
}
// SYSTEM, or a signed offset between -14:00 and +14:00. A named zone is refused
// because the image loads no time zone tables for the server to resolve it from.
func validateMariaDBTimeZone(value string) error {
if strings.EqualFold(value, "SYSTEM") {
return nil
}
if offset, err := parseTimeZoneOffset(value); err == nil {
const maxOffsetMinutes = 14 * 60
if offset >= -maxOffsetMinutes && offset <= maxOffsetMinutes {
return nil
}
}
return awserrors.Errorf(awserrors.ErrorInvalidParameterValue,
"parameter time_zone does not accept %q; use SYSTEM or a fixed offset between -14:00 and +14:00", value)
}
func parseTimeZoneOffset(value string) (int, error) {
parsed, err := time.Parse("-07:00", value)
if err != nil {
return 0, awserrors.Errorf(awserrors.ErrorInvalidParameterValue,
"offset must use signed HH:MM syntax: %v", err)
}
_, offsetSeconds := parsed.Zone()
return offsetSeconds / int(time.Minute/time.Second), nil
}
func validateMariaDBParameterCombinations(params []Parameter) error {
values := resolvedValues(params)
// A collation outside the server's character set is one of the few settings
// mysqld refuses to start under rather than adjusting.
charset, err := resolvedString(values, "character_set_server")
if err != nil {
return err
}
collation, err := resolvedString(values, "collation_server")
if err != nil {
return err
}
if !strings.HasPrefix(collation, charset+"_") {
return awserrors.Errorf(awserrors.ErrorInvalidParameterValue,
"collation_server %q does not belong to character_set_server %q", collation, charset)
}
// Left inconsistent, the server silently raises the maximum to the steady
// rate, so the value a customer set is not the one the engine runs.
ioCapacity, err := resolvedInteger(values, "innodb_io_capacity")
if err != nil {
return err
}
ioCapacityMax, err := resolvedInteger(values, "innodb_io_capacity_max")
if err != nil {
return err
}
if ioCapacityMax < ioCapacity {
return awserrors.Errorf(awserrors.ErrorInvalidParameterValue,
"innodb_io_capacity_max must be at least innodb_io_capacity")
}
return nil
}
@@ -0,0 +1,455 @@
package handlers_rds
import (
"strconv"
"strings"
"testing"
"github.com/aws/aws-sdk-go/aws"
"github.com/mulgadc/spinifex/spinifex/awserrors"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// Offered only now that the image and the guest implementation exist: until
// they did, a create would have resolved an AMI nothing builds and left a volume
// and an ENI behind for an instance that could never become available.
func TestMariaDB_IsOffered(t *testing.T) {
engine, err := LookupEngine(" MariaDB ")
require.NoError(t, err, "the engine name is matched case-insensitively and trimmed")
assert.Equal(t, engineMariaDB.Name, engine.Name)
assert.Contains(t, SupportedEngines(), "mariadb")
byFamily, err := engineForFamily(engineMariaDB.ParameterGroupFamily())
require.NoError(t, err)
assert.Equal(t, engineMariaDB.Name, byFamily.Name)
assert.Contains(t, SupportedParameterGroupFamilies(), "mariadb11.8")
// mysql is a distinct AWS engine this platform does not offer, and MariaDB is
// deliberately not aliased onto it: the alias would report an engine and a
// version the instance is not running.
_, err = LookupEngine("mysql")
require.Error(t, err)
assert.Equal(t, awserrors.ErrorInvalidParameterValue, awserrors.ValidErrorCodeFromError(err))
}
// The whole point of registering the engine: a request naming it has to reach
// the MariaDB AMI carrying MariaDB's own port, family and guest configuration,
// not PostgreSQL's defaults with a different name on them.
func TestCreateDBInstance_LaunchesMariaDB(t *testing.T) {
h := newCreateHarness(t, testBaseDomain)
input := validCreateInput()
input.Engine = aws.String("mariadb")
out, err := h.svc.CreateDBInstance(t.Context(), input, testAccountID)
require.NoError(t, err)
require.NotNil(t, out.DBInstance)
assert.Equal(t, "mariadb", aws.StringValue(out.DBInstance.Engine))
assert.Equal(t, "11.8", aws.StringValue(out.DBInstance.EngineVersion))
require.NotNil(t, out.DBInstance.Endpoint)
assert.Equal(t, int64(3306), aws.Int64Value(out.DBInstance.Endpoint.Port),
"the engine's default port applies when the request names none")
rec := h.record(t, testDBInstanceID)
assert.Equal(t, "mariadb", rec.Engine)
assert.Equal(t, int64(3306), rec.Port)
assert.Equal(t, engineMariaDB.DefaultParameterGroupName(), rec.DBParameterGroupName)
// The AMI is selected by the engine's own tags, so a MariaDB create can never
// land on the PostgreSQL image however the two are published.
amiFilters := map[string]string{}
for _, f := range h.launch.images.filters {
amiFilters[aws.StringValue(f.Name)] = aws.StringValue(f.Values[0])
}
assert.Equal(t, "mariadb", amiFilters["tag:"+engineTagKey])
assert.Equal(t, "11.8", amiFilters["tag:"+engineVersionTagKey])
// The guest builds its engine implementation from the image, and refuses to
// run when cloud-init disagrees with what the image bakes.
require.NotNil(t, h.launch.launcher.input)
assert.Contains(t, h.launch.launcher.input.UserData, "RDS_ENGINE=mariadb")
assert.Contains(t, h.launch.launcher.input.UserData, "RDS_ENGINE_PORT=3306")
}
func TestMariaDB_Identity(t *testing.T) {
assert.Equal(t, "mariadb", engineMariaDB.Name)
assert.Equal(t, int64(3306), engineMariaDB.DefaultPort)
// The version series the API pins to, not an integer major: PostgreSQL's
// version axis is one integer and MariaDB's is two.
assert.Equal(t, "11.8", engineMariaDB.EngineVersion())
assert.Equal(t, "mariadb11.8", engineMariaDB.ParameterGroupFamily())
assert.Equal(t, "default.mariadb11.8", engineMariaDB.DefaultParameterGroupName())
for _, version := range []string{"", "11.8"} {
assert.NoError(t, engineMariaDB.ValidateVersion(version), "version %q", version)
}
for _, version := range []string{"11", "11.4", "11.8.8", "10.11", "12", "latest"} {
assert.Error(t, engineMariaDB.ValidateVersion(version), "version %q", version)
}
}
// The engine's own limit rather than AWS's documented 16, so a client written
// against real AWS is a strict subset and nothing that works there breaks here.
func TestMariaDB_ValidateMasterUsername(t *testing.T) {
for _, username := range []string{"appuser", "app_user1", "Orders", strings.Repeat("a", 80)} {
assert.NoError(t, engineMariaDB.ValidateMasterUsername(username), "username %q", username)
}
cases := map[string]string{
"empty": "",
"leading digit": "1user",
"hyphen": "app-user",
"dot": "app.user",
"too long": strings.Repeat("a", 81),
"install-db root": "root",
"cased root": "ROOT",
"sys schema owner": "mariadb.sys",
"socket account": "mysql",
"aws management": "rdsadmin",
// The server refuses CREATE USER PUBLIC, which would spend the one-shot
// bootstrap password on a guaranteed failure.
"role name": "PUBLIC",
}
for name, username := range cases {
t.Run(name, func(t *testing.T) {
err := engineMariaDB.ValidateMasterUsername(username)
require.Error(t, err)
assert.Equal(t, awserrors.ErrorInvalidParameterValue, awserrors.ValidErrorCodeFromError(err))
})
}
}
// The guest re-checks the name it is handed rather than trusting the control
// plane, and it reaches the reserved half on its own.
func TestMariaDB_ValidateUsernameNotReserved(t *testing.T) {
for _, username := range []string{"mysql.session", "MySQL.infoschema", "mariadb.sys", "mariadb.anything"} {
err := engineMariaDB.ValidateUsernameNotReserved(username)
require.Error(t, err, "username %q", username)
assert.Equal(t, awserrors.ErrorInvalidParameterValue, awserrors.ValidErrorCodeFromError(err))
}
assert.NoError(t, engineMariaDB.ValidateUsernameNotReserved("mysqladmin"),
"the prefix is mysql. with the dot, not the bare word")
}
// MariaDB maps a database onto a directory name and rds-init interpolates the
// name into CREATE DATABASE from the shell, so this rule is the actual barrier.
func TestMariaDB_ValidateDBName(t *testing.T) {
for _, name := range []string{"", "orders", "Orders_2", strings.Repeat("d", 64)} {
assert.NoError(t, engineMariaDB.ValidateDBName(name), "name %q", name)
}
for _, name := range []string{"my-db", "1db", "my db", "my/db", `my\db`, "my.db", "db ", strings.Repeat("d", 65)} {
err := engineMariaDB.ValidateDBName(name)
require.Error(t, err, "name %q", name)
assert.Equal(t, awserrors.ErrorInvalidParameterValue, awserrors.ValidErrorCodeFromError(err))
}
// Each engine takes its own identifier limit rather than a shared one.
assert.Error(t, enginePostgres.ValidateDBName(strings.Repeat("d", 64)))
assert.NoError(t, engineMariaDB.ValidateDBName(strings.Repeat("d", 64)))
}
// The two catalogs are separate tables, not one table with an engine column, so
// neither engine can be handed a setting the other's server would not parse.
func TestMariaDBCatalog_IsSeparateFromPostgres(t *testing.T) {
for _, name := range []string{"innodb_buffer_pool_size", "sql_mode", "key_buffer_size"} {
_, ok := enginePostgres.LookupParameter(name)
assert.False(t, ok, "PostgreSQL must not expose %s", name)
}
for _, name := range []string{"work_mem", "shared_buffers", "autovacuum", "wal_level", "datestyle"} {
_, ok := engineMariaDB.LookupParameter(name)
assert.False(t, ok, "MariaDB must not expose %s", name)
}
// max_connections exists in both, and each takes its own engine's bounds.
_, err := engineMariaDB.validateParameterValue("max_connections", "20000")
assert.NoError(t, err)
_, err = enginePostgres.validateParameterValue("max_connections", "20000")
assert.Error(t, err, "PostgreSQL's own ceiling is 5000")
}
// Absent rather than present-and-unmodifiable, because they are not the
// customer's to set: the endpoint, the agent's socket access, the serving
// certificate and the snapshot recovery guarantee all depend on them.
func TestMariaDBCatalog_OmitsPlatformOwnedSettings(t *testing.T) {
owned := []string{
"port", "datadir", "socket", "bind_address",
"ssl_ca", "ssl_cert", "ssl_key", "require_secure_transport",
"secure_file_priv", "skip_symbolic_links", "log_bin",
"default_storage_engine",
"innodb_buffer_pool_chunk_size", "innodb_buffer_pool_instances",
}
for _, name := range owned {
_, ok := engineMariaDB.LookupParameter(name)
assert.False(t, ok, "%s is platform-owned and must not be in the catalog", name)
}
}
// The size-derived defaults have to move with the class, or the formulas are
// decoration and one end of the supported range is mis-tuned.
func TestMariaDBCatalog_SizeDerivedDefaultsScaleWithClassMemory(t *testing.T) {
smallest, err := classMemoryMiB("db.t3.micro")
require.NoError(t, err)
largest, err := classMemoryMiB("db.m5.xlarge")
require.NoError(t, err)
for _, name := range []string{"innodb_buffer_pool_size", "max_connections"} {
spec, ok := engineMariaDB.LookupParameter(name)
require.True(t, ok, "%s is expected to carry a computed default", name)
require.NotNil(t, spec.DefaultFor, "%s should be size-derived", name)
require.NotNil(t, spec.MaxFor, "%s should carry a class ceiling", name)
small, err := strconv.ParseInt(spec.DefaultAt(smallest), 10, 64)
require.NoError(t, err)
large, err := strconv.ParseInt(spec.DefaultAt(largest), 10, 64)
require.NoError(t, err)
assert.Less(t, small, large, "%s does not grow with the class", name)
assert.LessOrEqual(t, small, spec.MaxFor(smallest), "%s defaults above its own class ceiling", name)
}
}
// Three quarters of class memory is RDS's own default for this engine family,
// and it is the setting whose being wrong stops the server from starting.
func TestMariaDBCatalog_BufferPoolIsThreeQuartersOfClassMemory(t *testing.T) {
memoryMiB, err := classMemoryMiB("db.m5.large")
require.NoError(t, err)
bytes, err := strconv.ParseInt(innodbBufferPoolSizeFor(memoryMiB), 10, 64)
require.NoError(t, err)
assert.Equal(t, memoryMiB*mibToBytes*3/4, bytes)
assert.Less(t, bytes, innodbBufferPoolSizeCeilingFor(memoryMiB),
"the ceiling has to leave headroom above the default")
assert.Less(t, innodbBufferPoolSizeCeilingFor(memoryMiB), memoryMiB*mibToBytes,
"a pool at class memory is a server that never starts")
}
// max_connections = {DBInstanceClassMemory/12582880}, which is RDS's own formula.
func TestMariaDBCatalog_MaxConnectionsFollowsTheRDSFormula(t *testing.T) {
memoryMiB, err := classMemoryMiB("db.m5.xlarge")
require.NoError(t, err)
assert.Equal(t, strconv.FormatInt(memoryMiB*mibToBytes/12582880, 10), mariadbMaxConnectionsFor(memoryMiB))
// The floor keeps the smallest class above what the platform's own probe,
// parameter apply and password rotation sessions need.
smallest, err := classMemoryMiB(SmallestInstanceClass())
require.NoError(t, err)
connections, err := strconv.ParseInt(mariadbMaxConnectionsFor(smallest), 10, 64)
require.NoError(t, err)
assert.GreaterOrEqual(t, connections, int64(20))
}
func TestMariaDBCatalog_RejectsBadValues(t *testing.T) {
cases := []struct {
name, param, value, want string
}{
{"UnknownName", "not_a_setting", "1", "is not a parameter this engine exposes"},
{"PostgresSetting", "work_mem", "16384", "is not a parameter this engine exposes"},
{"NotAnInteger", "max_connections", "many", "takes an integer"},
{"BelowRange", "max_connections", "1", "outside its allowed range"},
{"BufferPoolBelowMinimum", "innodb_buffer_pool_size", "1048576", "outside its allowed range"},
{"RedoLogBelowMinimum", "innodb_log_file_size", "1048576", "outside its allowed range"},
{"NotAReal", "long_query_time", "slow", "takes a number"},
{"RealOutOfRange", "innodb_max_dirty_pages_pct", "100", "outside its allowed range"},
{"NotInEnum", "log_output", "syslog", "does not accept"},
{"NotAnIsolationLevel", "transaction_isolation", "snapshot", "does not accept"},
{"UnknownSQLMode", "sql_mode", "STRICT_TRANS_TABLES,NOT_A_MODE", "does not accept"},
{"EmptySQLModeToken", "sql_mode", "STRICT_TRANS_TABLES,,ANSI_QUOTES", "does not accept"},
{"NamedTimeZone", "time_zone", "Australia/Sydney", "does not accept"},
{"OffsetOutOfRange", "time_zone", "+15:00", "does not accept"},
{"UnsignedOffset", "time_zone", "10:00", "does not accept"},
{"OffsetWithoutMinutes", "time_zone", "+10", "does not accept"},
{"OffsetWithShortHour", "time_zone", "+1:00", "does not accept"},
{"OffsetWithLongHour", "time_zone", "+0001:00", "does not accept"},
{"Empty", "sort_buffer_size", "", "empty value"},
{"Formula", "innodb_buffer_pool_size", "{DBInstanceClassMemory*3/4}", "is a formula"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
_, err := engineMariaDB.validateParameterValue(tc.param, tc.value)
require.Error(t, err)
assert.Equal(t, awserrors.ErrorInvalidParameterValue, awserrors.ValidErrorCodeFromError(err),
"the code has to survive resolution or the client sees a 500")
assert.Contains(t, err.Error(), tc.want)
})
}
}
func TestMariaDBCatalog_AcceptsEngineSpelledValues(t *testing.T) {
cases := []struct{ param, value string }{
{"sql_mode", "STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION"},
{"sql_mode", "ansi_quotes"},
{"sql_mode", "TRADITIONAL, ONLY_FULL_GROUP_BY"},
{"time_zone", "SYSTEM"},
{"time_zone", "system"},
{"time_zone", "+10:00"},
{"time_zone", "-05:30"},
{"time_zone", "+00:00"},
{"innodb_flush_method", "O_DIRECT"},
{"transaction_isolation", "READ-COMMITTED"},
{"log_output", "TABLE"},
}
for _, tc := range cases {
t.Run(tc.param+"/"+tc.value, func(t *testing.T) {
_, err := engineMariaDB.validateParameterValue(tc.param, tc.value)
assert.NoError(t, err)
})
}
}
// MariaDB refuses yes and no for a boolean system variable where PostgreSQL
// accepts them, so a value the API takes must be one mysqld will parse.
func TestMariaDBCatalog_TakesOnlyTheEnginesBooleanSpellings(t *testing.T) {
for _, value := range []string{"on", "OFF", "true", "False", "1", "0"} {
_, err := engineMariaDB.validateParameterValue("slow_query_log", value)
assert.NoError(t, err, "slow_query_log rejected %q, which the engine accepts", value)
}
for _, value := range []string{"yes", "no"} {
_, err := engineMariaDB.validateParameterValue("slow_query_log", value)
require.Error(t, err, "slow_query_log accepted %q, which the engine refuses", value)
assert.Contains(t, err.Error(), "takes a boolean")
}
spec, ok := engineMariaDB.LookupParameter("slow_query_log")
require.True(t, ok)
assert.Equal(t, "on,off,true,false,1,0", spec.AllowedValues(),
"a customer has to be able to read which spellings this engine takes")
}
func TestMariaDBCatalog_RejectsFatalCombinations(t *testing.T) {
tests := []struct {
name string
overrides map[string]string
want string
}{
{
name: "collation outside the server character set",
overrides: map[string]string{"character_set_server": "latin1"},
want: "does not belong to character_set_server",
},
{
name: "io capacity maximum below the steady rate",
overrides: map[string]string{"innodb_io_capacity": "4000"},
want: "innodb_io_capacity_max must be at least innodb_io_capacity",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
_, err := engineMariaDB.ResolveEffectiveParameters("db.m5.xlarge", tt.overrides)
require.Error(t, err)
assert.Equal(t, awserrors.ErrorInvalidParameterValue, awserrors.ValidErrorCodeFromError(err))
assert.Contains(t, err.Error(), tt.want)
})
}
_, err := engineMariaDB.ResolveEffectiveParameters("db.m5.xlarge", map[string]string{
"character_set_server": "latin1", "collation_server": "latin1_bin",
})
assert.NoError(t, err, "a consistent pair is the customer's to choose")
}
// The buffer pool is the one setting a large-class literal makes a smaller guest
// unbootable with, so its class ceiling has to bind.
func TestMariaDBCatalog_BoundsSizeDerivedOverridesToTheClass(t *testing.T) {
tests := []struct{ name, value string }{
{name: "innodb_buffer_pool_size", value: "12884901888"},
{name: "max_connections", value: "2000"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
_, err := engineMariaDB.ResolveEffectiveParameters("db.t3.micro", map[string]string{tt.name: tt.value})
require.Error(t, err)
assert.Equal(t, awserrors.ErrorInvalidParameterValue, awserrors.ValidErrorCodeFromError(err))
assert.Contains(t, err.Error(), "db.t3.micro")
assert.Contains(t, err.Error(), "ceiling")
_, err = engineMariaDB.ResolveEffectiveParameters("db.m5.xlarge", map[string]string{tt.name: tt.value})
assert.NoError(t, err, "the same literal should fit the larger class")
})
}
}
// The apply type is what decides whether a change is a SET GLOBAL or a restart,
// and the guest's pending-restart comparison reads the static keys out of the
// installed file. Getting one wrong is a permanent pending-reboot or a change
// the customer is told took effect and did not.
func TestMariaDBCatalog_ApplyTypesMatchTheEngine(t *testing.T) {
static := []string{
"innodb_buffer_pool_size", "innodb_log_file_size", "innodb_flush_method",
"innodb_read_io_threads", "innodb_write_io_threads", "innodb_purge_threads",
"innodb_autoinc_lock_mode", "performance_schema",
}
for _, name := range static {
spec, ok := engineMariaDB.LookupParameter(name)
require.True(t, ok, "%s", name)
assert.Equal(t, ApplyTypeStatic, spec.ApplyType, "%s", name)
}
dynamic := []string{
"max_connections", "max_allowed_packet", "innodb_flush_log_at_trx_commit",
"innodb_io_capacity", "key_buffer_size", "table_open_cache", "sql_mode",
"slow_query_log", "long_query_time", "time_zone", "transaction_isolation",
}
for _, name := range dynamic {
spec, ok := engineMariaDB.LookupParameter(name)
require.True(t, ok, "%s", name)
assert.Equal(t, ApplyTypeDynamic, spec.ApplyType, "%s", name)
}
}
func TestMariaDBCatalog_ResolvesEveryParameterAsALiteral(t *testing.T) {
resolved, err := engineMariaDB.ResolveEffectiveParameters("db.m5.large", map[string]string{"max_connections": "300"})
require.NoError(t, err)
require.Len(t, resolved, len(engineMariaDB.CatalogParameterNames()))
values := map[string]string{}
for i, param := range resolved {
if i > 0 {
assert.Less(t, resolved[i-1].Name, param.Name, "the resolved set is not sorted by name")
}
assert.NotContains(t, param.Value, "{", "%s reached the agent as a formula", param.Name)
assert.NotContains(t, param.Value, "DBInstanceClass", "%s reached the agent as a reference", param.Name)
values[param.Name] = param.Value
}
assert.Equal(t, "300", values["max_connections"], "the override did not win")
memoryMiB, err := classMemoryMiB("db.m5.large")
require.NoError(t, err)
assert.Equal(t, innodbBufferPoolSizeFor(memoryMiB), values["innodb_buffer_pool_size"],
"a parameter the group does not set should carry its computed default")
}
// The bug that kept every MariaDB instance from booting: mariadbd has no
// time_zone startup option, so an option file naming it aborts the server before
// it opens the datadir.
func TestMariaDBCatalog_TimeZoneUsesItsStartupSpellingInTheOptionFile(t *testing.T) {
assert.Equal(t, "default_time_zone", engineMariaDB.OptionFileName("time_zone"))
// The customer-facing name is what the API reports and what a live SET GLOBAL
// names, so the catalog must still carry it.
spec, ok := engineMariaDB.LookupParameter("time_zone")
require.True(t, ok)
assert.Equal(t, "time_zone", spec.Name)
assert.True(t, spec.IsModifiable)
}
func TestMariaDBCatalog_EveryOtherParameterKeepsItsName(t *testing.T) {
for _, name := range engineMariaDB.CatalogParameterNames() {
if name == "time_zone" {
continue
}
assert.Equal(t, name, engineMariaDB.OptionFileName(name),
"%s was given an option-file spelling but the fixture asserts none", name)
}
}
// PostgreSQL sets every one of its settings under the name the customer knows,
// so a mapping there would be a bug rather than a fix.
func TestPostgresCatalog_NoParameterIsRenamedForTheOptionFile(t *testing.T) {
for _, name := range enginePostgres.CatalogParameterNames() {
assert.Equal(t, name, enginePostgres.OptionFileName(name))
}
}
// A name the catalog does not carry is the customer's own, and the option file
// takes it unchanged rather than silently dropping it.
func TestMariaDBCatalog_OptionFileNamePassesAnUnknownNameThrough(t *testing.T) {
assert.Equal(t, "not_a_parameter", engineMariaDB.OptionFileName("not_a_parameter"))
}
+464 -4
View File
@@ -1,5 +1,16 @@
package handlers_rds
import (
"strconv"
"strings"
"time"
// The zone database Go's own LoadLocation reads, vendored into the binary so
// the timezone parameter validates identically wherever the daemon runs.
_ "time/tzdata"
"github.com/mulgadc/spinifex/spinifex/awserrors"
)
// PostgreSQL 18 is the pinned v1 major, matching the rds-postgres AMI preset.
// A new major is a new AMI plus a bump here, never a runtime upgrade.
var enginePostgres = Engine{
@@ -12,9 +23,458 @@ var enginePostgres = Engine{
// administrative privileges through.
reservedUsernames: []string{"rdsadmin", "postgres", "rds_superuser"},
reservedUsernamePrefixes: []string{"pg_"},
// NAMEDATALEN-1, where the engine's own limit and AWS's documented one
// coincide. A database name is the same identifier limit.
maxUsernameLen: 63,
validateDBName: dbNameRule(63),
catalog: postgresParameterCatalog,
validateCombinations: validatePostgresParameterCombinations,
// Every table is WAL-logged, so a torn datadir recovers in full.
crashRecoveryNote: "It will recover from its write-ahead log when it is restored.",
uncleanStopNote: "It will recover from its write-ahead log on the next start.",
}
// The pinned engine as a value, for the in-guest agent: it enforces the same
// reserved-role set before altering a role as the cluster superuser, and a
// lookup by name there could only fail silently into an empty set.
func EnginePostgres() Engine { return enginePostgres }
// The curated PostgreSQL 18 table: a genuinely validated subset rather than the
// engine's full ~350 GUCs, because a static parameter the engine refuses at
// startup is a boot loop with the bad config already on the data volume.
//
// The platform-owned settings (port, listen_addresses, ssl*,
// unix_socket_directories, data_directory) are deliberately absent rather than
// present-and-unmodifiable, because they are not the customer's to set.
var postgresParameterCatalog = buildParameterCatalog(
// Connections. max_connections is what a size-derived default matters most
// for: RDS's own formula is LEAST({DBInstanceClassMemory/9531392}, 5000).
ParameterSpec{
Name: "max_connections", DataType: paramTypeInteger, ApplyType: ApplyTypeStatic,
IsModifiable: true, Min: 6, Max: 5000,
DefaultFor: maxConnectionsFor,
MaxFor: maxConnectionsCeilingFor,
Description: "Maximum number of concurrent connections to the database server.",
},
ParameterSpec{
Name: "superuser_reserved_connections", DataType: paramTypeInteger, ApplyType: ApplyTypeStatic,
IsModifiable: true, Min: 0, Max: 100, Default: "3",
Description: "Connection slots reserved for superusers.",
},
ParameterSpec{
Name: "idle_in_transaction_session_timeout", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 2147483647, Default: "0", Unit: "ms",
Description: "Milliseconds an idle open transaction may live before it is aborted; 0 disables.",
},
ParameterSpec{
Name: "statement_timeout", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 2147483647, Default: "0", Unit: "ms",
Description: "Milliseconds a statement may run before it is aborted; 0 disables.",
},
ParameterSpec{
Name: "lock_timeout", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 2147483647, Default: "0", Unit: "ms",
Description: "Milliseconds to wait for a lock before failing the statement; 0 disables.",
},
ParameterSpec{
Name: "tcp_keepalives_idle", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 3600, Default: "0", Unit: "s",
Description: "Idle seconds before the server sends a TCP keepalive; 0 takes the system default.",
},
// Memory. Every one of these is a fraction of class memory on real RDS, so a
// literal tuned for the large end makes db.t3.micro fail to start.
ParameterSpec{
Name: "shared_buffers", DataType: paramTypeInteger, ApplyType: ApplyTypeStatic,
IsModifiable: true, Min: 16, Max: 4194304, Unit: "8kB",
DefaultFor: sharedBuffersFor,
MaxFor: sharedBuffersCeilingFor,
Description: "Shared memory buffers the server uses, in 8 kB blocks.",
},
ParameterSpec{
Name: "effective_cache_size", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 1, Max: 4194304, Unit: "8kB",
DefaultFor: effectiveCacheSizeFor,
MaxFor: effectiveCacheSizeCeilingFor,
Description: "Planner assumption about the disk cache available to one query, in 8 kB blocks.",
},
// Per sort or hash rather than per connection, so RDS leaves it a literal
// even in the memory group; a size-derived default here would multiply by
// max_connections and by the parallel workers under each of them.
ParameterSpec{
Name: "work_mem", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 64, Max: 2147483647, Default: "4096", Unit: "kB",
Description: "Memory one sort or hash operation may use before spilling to disk, in kB.",
},
ParameterSpec{
Name: "maintenance_work_mem", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 1024, Max: 2147483647, Unit: "kB",
DefaultFor: maintenanceWorkMemFor,
MaxFor: maintenanceWorkMemCeilingFor,
Description: "Memory a VACUUM, CREATE INDEX or ALTER TABLE ADD FOREIGN KEY may use, in kB.",
},
ParameterSpec{
Name: "temp_buffers", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 100, Max: 1073741823, Default: "1024", Unit: "8kB",
Description: "Per-session buffers for temporary tables, in 8 kB blocks.",
},
ParameterSpec{
Name: "max_prepared_transactions", DataType: paramTypeInteger, ApplyType: ApplyTypeStatic,
IsModifiable: true, Min: 0, Max: 262143, Default: "0",
Description: "Maximum simultaneously prepared transactions; 0 disables two-phase commit.",
},
// Parallelism. Bounded by the class's vCPU count on a real deployment, but
// PostgreSQL degrades gracefully when they exceed it, so literals are honest.
ParameterSpec{
Name: "max_worker_processes", DataType: paramTypeInteger, ApplyType: ApplyTypeStatic,
IsModifiable: true, Min: 0, Max: 262143, Default: "8",
Description: "Maximum background processes the server may start.",
},
ParameterSpec{
Name: "max_parallel_workers", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 1024, Default: "8",
Description: "Maximum workers the server may use for parallel operations.",
},
ParameterSpec{
Name: "max_parallel_workers_per_gather", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 1024, Default: "2",
Description: "Maximum parallel workers one Gather node may start.",
},
// WAL and checkpoints.
ParameterSpec{
Name: "wal_level", DataType: paramTypeEnum, ApplyType: ApplyTypeStatic,
IsModifiable: true, Enum: []string{"minimal", "replica", "logical"}, Default: "replica",
Description: "How much information is written to the WAL.",
},
ParameterSpec{
Name: "synchronous_commit", DataType: paramTypeEnum, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Enum: []string{"off", "local", "remote_write", "on", "remote_apply"}, Default: "on",
Description: "How much WAL processing must complete before a commit returns.",
},
// Alpine's PostgreSQL 18 package is built with both optional compression
// libraries, so every engine method is safe to expose.
ParameterSpec{
Name: "wal_compression", DataType: paramTypeEnum, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Enum: []string{"off", "pglz", "lz4", "zstd", "on"}, Default: "off",
Description: "Compression method for full-page images written to the WAL.",
},
// The image uses PostgreSQL's 16 MiB WAL segments; both size settings must
// hold at least two segments or startup rejects the configuration.
ParameterSpec{
Name: "max_wal_size", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 32, Max: 2097151, Default: "1024", Unit: "MB",
Description: "WAL size that triggers a checkpoint, in MB.",
},
ParameterSpec{
Name: "min_wal_size", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 32, Max: 2097151, Default: "80", Unit: "MB",
Description: "WAL size below which old segments are recycled rather than removed, in MB.",
},
ParameterSpec{
Name: "checkpoint_timeout", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 30, Max: 86400, Default: "300", Unit: "s",
Description: "Maximum seconds between automatic WAL checkpoints.",
},
ParameterSpec{
Name: "checkpoint_completion_target", DataType: paramTypeReal, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 1, Default: "0.9",
Description: "Fraction of the checkpoint interval a checkpoint's writes are spread over.",
},
ParameterSpec{
Name: "wal_buffers", DataType: paramTypeInteger, ApplyType: ApplyTypeStatic,
IsModifiable: true, Min: -1, Max: 262143, Default: "-1", Unit: "8kB",
Description: "Shared memory used for WAL not yet written, in 8 kB blocks; -1 derives it from shared_buffers.",
},
ParameterSpec{
Name: "max_wal_senders", DataType: paramTypeInteger, ApplyType: ApplyTypeStatic,
IsModifiable: true, Min: 0, Max: 262143, Default: "10",
Description: "Maximum simultaneously running WAL sender processes.",
},
ParameterSpec{
Name: "max_replication_slots", DataType: paramTypeInteger, ApplyType: ApplyTypeStatic,
IsModifiable: true, Min: 0, Max: 262143, Default: "10",
Description: "Maximum replication slots the server may define.",
},
// Autovacuum. Turning it off is the single most reliable way to take a
// PostgreSQL database down slowly, so it is offered but its default is on.
ParameterSpec{
Name: "autovacuum", DataType: paramTypeBoolean, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Default: "on",
Description: "Whether the autovacuum launcher runs.",
},
// PostgreSQL 18 moved this setting to SIGHUP; worker slot allocation is now
// controlled separately by the postmaster-only autovacuum_worker_slots.
ParameterSpec{
Name: "autovacuum_max_workers", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 1, Max: 262143, Default: "3",
Description: "Maximum autovacuum worker processes running at once.",
},
ParameterSpec{
Name: "autovacuum_naptime", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 1, Max: 2147483, Default: "60", Unit: "s",
Description: "Seconds between autovacuum runs on any one database.",
},
ParameterSpec{
Name: "autovacuum_vacuum_threshold", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 2147483647, Default: "50",
Description: "Row updates or deletes before a table is vacuumed.",
},
ParameterSpec{
Name: "autovacuum_vacuum_scale_factor", DataType: paramTypeReal, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 100, Default: "0.2",
Description: "Fraction of the table size added to autovacuum_vacuum_threshold.",
},
ParameterSpec{
Name: "autovacuum_analyze_threshold", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 2147483647, Default: "50",
Description: "Row inserts, updates or deletes before a table is analyzed.",
},
ParameterSpec{
Name: "autovacuum_analyze_scale_factor", DataType: paramTypeReal, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 100, Default: "0.1",
Description: "Fraction of the table size added to autovacuum_analyze_threshold.",
},
ParameterSpec{
Name: "autovacuum_vacuum_cost_limit", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: -1, Max: 10000, Default: "-1",
Description: "Cost budget one autovacuum worker spends before sleeping; -1 takes vacuum_cost_limit.",
},
// Planner.
// The engine's own ceiling on a planner cost is DBL_MAX, which is not a bound
// worth reporting: a cost past four digits already makes the plan choice
// insensitive to it, so the range stays one a customer can read.
ParameterSpec{
Name: "random_page_cost", DataType: paramTypeReal, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 10000, Default: "1.1",
Description: "Planner estimate of the cost of a non-sequentially fetched page.",
},
ParameterSpec{
Name: "seq_page_cost", DataType: paramTypeReal, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 10000, Default: "1",
Description: "Planner estimate of the cost of a sequentially fetched page.",
},
ParameterSpec{
Name: "effective_io_concurrency", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 1000, Default: "16",
Description: "Concurrent disk I/O operations the planner assumes are useful.",
},
ParameterSpec{
Name: "default_statistics_target", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 1, Max: 10000, Default: "100",
Description: "Default statistics target for table columns without one of their own.",
},
ParameterSpec{
Name: "jit", DataType: paramTypeBoolean, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Default: "on",
Description: "Whether JIT compilation may be used for qualifying queries.",
},
// Logging. The destination and the collector are platform-owned; what is
// logged is the customer's.
ParameterSpec{
Name: "log_min_duration_statement", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: -1, Max: 2147483647, Default: "-1", Unit: "ms",
Description: "Milliseconds a statement must run to be logged; 0 logs every statement, -1 disables.",
},
ParameterSpec{
Name: "log_statement", DataType: paramTypeEnum, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Enum: []string{"none", "ddl", "mod", "all"}, Default: "none",
Description: "Which SQL statements are written to the log.",
},
ParameterSpec{
Name: "log_min_messages", DataType: paramTypeEnum, ApplyType: ApplyTypeDynamic,
IsModifiable: true,
Enum: []string{"debug5", "debug4", "debug3", "debug2", "debug1", "info", "notice",
"warning", "error", "log", "fatal", "panic"},
Default: "warning",
Description: "Lowest message severity written to the server log.",
},
ParameterSpec{
Name: "log_connections", DataType: paramTypeBoolean, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Default: "off",
Description: "Whether each successful connection attempt is logged.",
},
ParameterSpec{
Name: "log_disconnections", DataType: paramTypeBoolean, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Default: "off",
Description: "Whether the end of each session is logged, with its duration.",
},
ParameterSpec{
Name: "log_lock_waits", DataType: paramTypeBoolean, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Default: "on",
Description: "Whether a session waiting longer than deadlock_timeout for a lock is logged.",
},
ParameterSpec{
Name: "log_temp_files", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: -1, Max: 2147483647, Default: "-1", Unit: "kB",
Description: "Size in kB above which a temporary file's removal is logged; 0 logs all, -1 disables.",
},
ParameterSpec{
Name: "log_autovacuum_min_duration", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: -1, Max: 2147483647, Default: "-1", Unit: "ms",
Description: "Milliseconds an autovacuum action must run to be logged; -1 disables.",
},
// Timeouts and locale.
ParameterSpec{
Name: "deadlock_timeout", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 1, Max: 2147483647, Default: "1000", Unit: "ms",
Description: "Milliseconds to wait on a lock before checking for a deadlock.",
},
ParameterSpec{
Name: "timezone", DataType: paramTypeString, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Default: "UTC", Validate: validatePostgresTimezone,
Description: "Time zone the server displays and interprets timestamps in.",
},
ParameterSpec{
Name: "datestyle", DataType: paramTypeString, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Default: "ISO, MDY", Validate: validatePostgresDateStyle,
Description: "Display format for date and time values.",
},
ParameterSpec{
Name: "track_activity_query_size", DataType: paramTypeInteger, ApplyType: ApplyTypeStatic,
IsModifiable: true, Min: 100, Max: 1048576, Default: "1024", Unit: "B",
Description: "Bytes of each running query pg_stat_activity retains.",
},
ParameterSpec{
Name: "track_io_timing", DataType: paramTypeBoolean, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Default: "on",
Description: "Whether block read and write times are collected.",
},
)
// Go's own zone database, which is vendored into the binary, so a name the API
// accepts is one the guest can load without a tzdata package.
func validatePostgresTimezone(value string) error {
if !strings.EqualFold(value, "Local") {
if _, err := time.LoadLocation(value); err == nil {
return nil
}
}
return awserrors.Errorf(awserrors.ErrorInvalidParameterValue,
"parameter timezone does not accept %q; use an IANA time zone name such as UTC or Australia/Sydney", value)
}
// One output style and one date order, in either order, which is the grammar
// PostgreSQL's DateStyle accepts.
func validatePostgresDateStyle(value string) error {
if validDateStyle(value) {
return nil
}
return awserrors.Errorf(awserrors.ErrorInvalidParameterValue,
"parameter datestyle does not accept %q; use one output style (ISO, SQL, Postgres, German) and one date order (MDY, DMY, YMD)", value)
}
func validDateStyle(value string) bool {
parts := strings.Split(value, ",")
if len(parts) == 0 || len(parts) > 2 {
return false
}
seenStyle, seenOrder := false, false
for _, part := range parts {
switch strings.ToLower(strings.TrimSpace(part)) {
case "iso", "sql", "postgres", "german":
if seenStyle {
return false
}
seenStyle = true
case "mdy", "dmy", "ymd":
if seenOrder {
return false
}
seenOrder = true
default:
return false
}
}
return seenStyle || seenOrder
}
// shared_buffers = {DBInstanceClassMemory/32768}, in 8 kB blocks — a quarter of
// class memory, which is RDS's own default.
func sharedBuffersFor(memoryMiB int64) string {
return strconv.FormatInt(clampInt64(memoryMiB*mibToBytes/32768, 16, 4194304), 10)
}
// effective_cache_size = {DBInstanceClassMemory/16384}, in 8 kB blocks — half of
// class memory, the planner's assumption about what the OS is caching.
func effectiveCacheSizeFor(memoryMiB int64) string {
return strconv.FormatInt(clampInt64(memoryMiB*mibToBytes/16384, 1, 4194304), 10)
}
// max_connections = LEAST({DBInstanceClassMemory/9531392}, 5000). The floor is
// what keeps db.t3.micro above superuser_reserved_connections plus the workers
// autovacuum and replication need, which is where the smallest class would
// otherwise fail to start.
func maxConnectionsFor(memoryMiB int64) string {
return strconv.FormatInt(clampInt64(memoryMiB*mibToBytes/9531392, 20, 5000), 10)
}
// maintenance_work_mem = {DBInstanceClassMemory/63963136}, in kB. Capped so a
// large class does not let autovacuum_max_workers reserve the whole machine.
func maintenanceWorkMemFor(memoryMiB int64) string {
return strconv.FormatInt(clampInt64(memoryMiB*mibToBytes/63963136*1024, 16384, 2097152), 10)
}
// Class ceilings deliberately leave substantial headroom above the defaults.
// They prevent obviously impossible large-class literals without prescribing
// production tuning for values the guest can support.
func sharedBuffersCeilingFor(memoryMiB int64) int64 {
return clampInt64(memoryMiB*1024/8*3/4, 16, 4194304)
}
func effectiveCacheSizeCeilingFor(memoryMiB int64) int64 {
return clampInt64(memoryMiB*1024/8, 1, 4194304)
}
func maxConnectionsCeilingFor(memoryMiB int64) int64 {
defaults := clampInt64(memoryMiB*mibToBytes/9531392, 20, 5000)
return clampInt64(defaults*4, 20, 5000)
}
func maintenanceWorkMemCeilingFor(memoryMiB int64) int64 {
return clampInt64(memoryMiB*1024/2, 16384, 2147483647)
}
func validatePostgresParameterCombinations(params []Parameter) error {
values := resolvedValues(params)
maxWALSenders, err := resolvedInteger(values, "max_wal_senders")
if err != nil {
return err
}
maxReplicationSlots, err := resolvedInteger(values, "max_replication_slots")
if err != nil {
return err
}
if values["wal_level"] == "minimal" && (maxWALSenders != 0 || maxReplicationSlots != 0) {
return awserrors.Errorf(awserrors.ErrorInvalidParameterValue,
"wal_level minimal requires max_wal_senders and max_replication_slots both to be 0")
}
maxConnections, err := resolvedInteger(values, "max_connections")
if err != nil {
return err
}
reservedConnections, err := resolvedInteger(values, "superuser_reserved_connections")
if err != nil {
return err
}
if reservedConnections >= maxConnections {
return awserrors.Errorf(awserrors.ErrorInvalidParameterValue,
"superuser_reserved_connections must be less than max_connections")
}
maxWorkerProcesses, err := resolvedInteger(values, "max_worker_processes")
if err != nil {
return err
}
const (
postgres18MaxBackends = 262143
postgres18AutovacuumWorkerSlots = 16
postgres18SpecialWorkerProcs = 2
)
backends := maxConnections + postgres18AutovacuumWorkerSlots + maxWorkerProcesses + maxWALSenders + postgres18SpecialWorkerProcs
if backends > postgres18MaxBackends {
return awserrors.Errorf(awserrors.ErrorInvalidParameterValue,
"max_connections, max_worker_processes and max_wal_senders reserve too many server processes")
}
return nil
}
+35 -2
View File
@@ -22,7 +22,7 @@ func TestLookupEngine(t *testing.T) {
_, err = LookupEngine("mysql")
require.Error(t, err)
assert.Contains(t, err.Error(), awserrors.ErrorInvalidParameterValue)
assert.Equal(t, []string{"postgres"}, SupportedEngines())
assert.Equal(t, []string{"mariadb", "postgres"}, SupportedEngines())
}
// A version other than the pinned one would be served by an image that is
@@ -54,7 +54,7 @@ func TestEngineValidateMasterUsername(t *testing.T) {
"empty": "",
"leading digit": "1user",
"hyphen": "app-user",
"too long": strings.Repeat("a", maxMasterUsernameLen+1),
"too long": strings.Repeat("a", enginePostgres.maxUsernameLen+1),
"reserved": "rdsadmin",
"reserved cased": "RDSAdmin",
// The cluster superuser and the group role the master's administrative
@@ -75,6 +75,39 @@ func TestEngineValidateMasterUsername(t *testing.T) {
}
}
// The name is interpolated into a CREATE DATABASE in the guest, so the rule is
// narrower than the engine's own: it is what makes the interpolation safe.
func TestEngineValidateDBName(t *testing.T) {
engine, err := LookupEngine("postgres")
require.NoError(t, err)
// Empty is not a rejection: AWS creates no initial database when none is named.
for _, name := range []string{"", "orders", "order_db1", "Orders"} {
assert.NoError(t, engine.ValidateDBName(name), "DBName %q", name)
}
cases := map[string]string{
"hyphen": "my-db",
"leading digit": "1db",
"space": "my db",
"dot": "my.db",
// MariaDB maps a database name onto a directory name, so these are
// structurally dangerous there rather than merely awkward.
"slash": "my/db",
"backslash": `my\db`,
"trailing space": "orders ",
"quote": "orders'",
"too long": strings.Repeat("a", 64),
}
for name, dbName := range cases {
t.Run(name, func(t *testing.T) {
err := engine.ValidateDBName(dbName)
require.Error(t, err)
assert.Contains(t, err.Error(), awserrors.ErrorInvalidParameterValue)
})
}
}
func TestValidateMasterUserPassword(t *testing.T) {
assert.NoError(t, ValidateMasterUserPassword("Sup3rSecret!"))
assert.NoError(t, ValidateMasterUserPassword(strings.Repeat("x", maxMasterPasswordLen)))
+215
View File
@@ -0,0 +1,215 @@
package handlers_rds
import (
"testing"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/rds"
"github.com/mulgadc/spinifex/spinifex/awserrors"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// A second engine that exists solely inside the test binary, so the cross-engine
// refusal is proven against the seam rather than against whichever engines the
// build happens to offer. Nothing about its catalog matters beyond being
// distinguishable from the shipped ones.
const (
testEngineName = "testdb"
testEngineGroup = "tuned-testdb"
)
var engineUnderTest = Engine{
Name: testEngineName,
MajorVersion: "1",
DefaultPort: 3306,
reservedUsernames: []string{"root"},
reservedUsernamePrefixes: []string{"testdb_"},
maxUsernameLen: 80,
validateDBName: dbNameRule(64),
catalog: buildParameterCatalog(
ParameterSpec{
Name: "buffer_size", DataType: paramTypeInteger, ApplyType: ApplyTypeStatic,
IsModifiable: true, Min: 1, Max: 1024, Default: "64", Unit: "MB",
Description: "Buffer the server allocates at startup, in MB.",
},
ParameterSpec{
Name: "connection_limit", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 1, Max: 1000, Default: "100",
Description: "Maximum concurrent connections to the server.",
},
),
validateCombinations: func([]Parameter) error { return nil },
crashRecoveryNote: "It will recover when it is restored.",
uncleanStopNote: "It will recover on the next start.",
}
// Registers the second engine for one test. Both indexes are rebuilt, because a
// family-knowing caller reads the one keyed by family.
func registerTestEngine(t *testing.T) {
t.Helper()
require.NotContains(t, engines, engineUnderTest.Name)
engines[engineUnderTest.Name] = engineUnderTest
enginesByFamily = indexEnginesByFamily()
t.Cleanup(func() {
delete(engines, engineUnderTest.Name)
enginesByFamily = indexEnginesByFamily()
})
}
func testEngineGroupInput(name string) *rds.CreateDBParameterGroupInput {
return &rds.CreateDBParameterGroupInput{
DBParameterGroupName: aws.String(name),
DBParameterGroupFamily: aws.String(engineUnderTest.ParameterGroupFamily()),
Description: aws.String("Tuned for the second engine"),
}
}
func requireFamilyMismatch(t *testing.T, err error) {
t.Helper()
require.Error(t, err)
assert.Equal(t, awserrors.ErrorInvalidParameterCombination, awserrors.ValidErrorCodeFromError(err),
"the code has to survive resolution or the client sees a 500")
}
// A group carries the settings of exactly one engine, so attaching one of
// another engine would boot the database on a configuration file it cannot
// parse.
func TestCreateDBInstance_RefusesAGroupOfAnotherEnginesFamily(t *testing.T) {
registerTestEngine(t)
h := newCreateHarness(t, testBaseDomain)
_, err := h.svc.CreateDBParameterGroup(t.Context(), testEngineGroupInput(testEngineGroup), testAccountID)
require.NoError(t, err)
_, err = h.svc.CreateDBParameterGroup(t.Context(), parameterGroupInput(testParameterGroup), testAccountID)
require.NoError(t, err)
input := validCreateInput()
input.DBParameterGroupName = aws.String(testEngineGroup)
_, err = h.svc.CreateDBInstance(t.Context(), input, testAccountID)
requireFamilyMismatch(t, err)
assert.Contains(t, err.Error(), enginePostgres.ParameterGroupFamily(), "the error names the family the instance needs")
assert.False(t, h.recordExists(t, testDBInstanceID), "a rejected create must reserve nothing")
// The other direction: neither engine may borrow the other's group.
input = validCreateInput()
input.Engine = aws.String(testEngineName)
input.DBParameterGroupName = aws.String(testParameterGroup)
_, err = h.svc.CreateDBInstance(t.Context(), input, testAccountID)
requireFamilyMismatch(t, err)
assert.Contains(t, err.Error(), engineUnderTest.ParameterGroupFamily())
assert.False(t, h.recordExists(t, testDBInstanceID))
}
func TestModifyDBInstance_RefusesAGroupOfAnotherEnginesFamily(t *testing.T) {
registerTestEngine(t)
h := newModifyHarness(t)
_, err := h.svc.CreateDBParameterGroup(t.Context(), testEngineGroupInput(testEngineGroup), testAccountID)
require.NoError(t, err)
seedInstance(t, h.svc, modifiableRecord())
input := modifyInput()
input.DBParameterGroupName = aws.String(testEngineGroup)
input.ApplyImmediately = aws.Bool(true)
_, err = h.svc.ModifyDBInstance(t.Context(), input, testAccountID)
requireFamilyMismatch(t, err)
assert.Equal(t, testDefaultGroup, h.record(t).DBParameterGroupName, "a rejected modify must change nothing")
assert.Empty(t, h.agent.received(), "nothing may reach the engine")
}
// The deferred half of a modify re-resolves at apply time, so the check has to
// hold there too rather than only at the request that recorded it.
func TestApplyPendingModifications_RefusesAGroupOfAnotherEnginesFamily(t *testing.T) {
registerTestEngine(t)
h := newModifyHarness(t)
h.agent.replyWith("")
_, err := h.svc.CreateDBParameterGroup(t.Context(), testEngineGroupInput(testEngineGroup), testAccountID)
require.NoError(t, err)
rec := modifyingRecord(&PendingModifiedValues{
DBParameterGroupName: testEngineGroup,
RequestedAt: time.Now().UTC(),
})
seedInstance(t, h.svc, rec)
err = h.svc.applyPendingModifications(t.Context(), h.kv(t), testAccountID, &rec)
requireFamilyMismatch(t, err)
assert.Empty(t, h.agent.received(), "nothing may reach the engine")
}
// A restore takes its engine from the snapshot rather than the request, so the
// group named alongside it is checked against what the restored data is.
func TestRestoreDBInstanceFromDBSnapshot_RefusesAGroupOfAnotherEnginesFamily(t *testing.T) {
registerTestEngine(t)
h := newSnapshotHarness(t, false)
h.seedSnapshot(t)
_, err := h.svc.CreateDBParameterGroup(t.Context(), testEngineGroupInput(testEngineGroup), testAccountID)
require.NoError(t, err)
input := restoreInput()
input.DBParameterGroupName = aws.String(testEngineGroup)
_, err = h.svc.RestoreDBInstanceFromDBSnapshot(t.Context(), input, testAccountID)
requireFamilyMismatch(t, err)
assert.False(t, h.instanceExists(t, testRestoredID), "a rejected restore must reserve nothing")
}
// Values are validated against the engine the target group's family names. The
// alternative stores one engine's setting into another's group and defers the
// failure to whichever instance next attaches it.
func TestModifyDBParameterGroup_ValidatesAgainstTheGroupsOwnEngine(t *testing.T) {
registerTestEngine(t)
h := newCreateHarness(t, testBaseDomain)
_, err := h.svc.CreateDBParameterGroup(t.Context(), testEngineGroupInput(testEngineGroup), testAccountID)
require.NoError(t, err)
_, err = h.svc.CreateDBParameterGroup(t.Context(), parameterGroupInput(testParameterGroup), testAccountID)
require.NoError(t, err)
_, err = h.svc.ModifyDBParameterGroup(t.Context(),
modifyParameters(testEngineGroup, parameter("work_mem", "16384", "")), testAccountID)
require.Error(t, err)
assert.Equal(t, awserrors.ErrorInvalidParameterValue, awserrors.ValidErrorCodeFromError(err))
_, err = h.svc.ModifyDBParameterGroup(t.Context(),
modifyParameters(testParameterGroup, parameter("connection_limit", "50", "")), testAccountID)
require.Error(t, err)
assert.Equal(t, awserrors.ErrorInvalidParameterValue, awserrors.ValidErrorCodeFromError(err))
_, err = h.svc.ModifyDBParameterGroup(t.Context(),
modifyParameters(testEngineGroup, parameter("connection_limit", "50", "")), testAccountID)
require.NoError(t, err, "a group takes its own engine's parameters")
assert.Equal(t, "50", aws.StringValue(describedParameters(t, h, testEngineGroup)["connection_limit"].ParameterValue))
}
func TestDescribeDBParameters_ListsTheGroupsOwnEngineCatalog(t *testing.T) {
registerTestEngine(t)
h := newCreateHarness(t, testBaseDomain)
_, err := h.svc.CreateDBParameterGroup(t.Context(), testEngineGroupInput(testEngineGroup), testAccountID)
require.NoError(t, err)
params := describedParameters(t, h, testEngineGroup)
require.Len(t, params, len(engineUnderTest.CatalogParameterNames()))
assert.NotContains(t, params, "work_mem", "a group must not report another engine's settings")
assert.Equal(t, "100", aws.StringValue(params["connection_limit"].ParameterValue))
assert.Equal(t, ParameterSourceEngineDefault, aws.StringValue(params["connection_limit"].Source))
}
// The rejection has to name what the client can use instead, which stops being
// one family as soon as a second engine registers.
func TestCreateDBParameterGroup_RejectionNamesEverySupportedFamily(t *testing.T) {
registerTestEngine(t)
h := newCreateHarness(t, testBaseDomain)
input := parameterGroupInput(testParameterGroup)
input.DBParameterGroupFamily = aws.String("mysql8")
_, err := h.svc.CreateDBParameterGroup(t.Context(), input, testAccountID)
require.Error(t, err)
assert.Equal(t, awserrors.ErrorInvalidParameterValue, awserrors.ValidErrorCodeFromError(err))
for _, family := range SupportedParameterGroupFamilies() {
assert.Contains(t, err.Error(), family)
}
assert.Equal(t, []string{
engineMariaDB.ParameterGroupFamily(),
enginePostgres.ParameterGroupFamily(),
engineUnderTest.ParameterGroupFamily(),
}, SupportedParameterGroupFamilies())
}
+19 -4
View File
@@ -233,19 +233,34 @@ func (s *Service) confirmVMStopped(ctx context.Context, accountID, instanceID st
}
// Asks the engine to shut down cleanly, and records an event rather than
// failing the call when it cannot. An engine that did not stop cleanly replays
// its WAL on the next start; refusing to stop the VM over it would leave the
// customer unable to stop an instance whose agent is wedged.
// failing the call when it cannot. Refusing to stop the VM over it would leave
// the customer unable to stop an instance whose agent is wedged.
func (s *Service) stopEngineOrRecordFallback(ctx context.Context, accountID string, rec *DBInstanceRecord, operation string) {
if err := s.stopEngine(ctx, accountID, rec.DBInstanceIdentifier); err != nil {
slog.WarnContext(ctx, "rds: graceful engine stop failed; continuing",
"dbInstance", rec.DBInstanceIdentifier, "operation", operation, "err", err)
s.RecordEvent(ctx, accountID, EventSourceTypeDBInstance, rec.DBInstanceIdentifier,
fmt.Sprintf("The database engine could not be shut down cleanly before %s; it will recover from its write-ahead log on the next start.", operation),
uncleanStopMessage(ctx, rec.Engine, operation),
EventCategoryNotification, EventCategoryAvailability)
}
}
// The half of the warning that holds for every engine. What the next start then
// recovers does not: PostgreSQL replays every table from its write-ahead log,
// MariaDB only its InnoDB ones.
func uncleanStopMessage(ctx context.Context, engineName, operation string) string {
warning := fmt.Sprintf("The database engine could not be shut down cleanly before %s.", operation)
engine, err := LookupEngine(engineName)
if err != nil {
// The VM is going down either way, so the customer still gets the half of
// the warning that does not depend on knowing the engine.
slog.ErrorContext(ctx, "rds: the DB instance names an engine this build does not offer",
"engine", engineName, "err", err)
return warning
}
return warning + " " + engine.uncleanStopNote
}
// Moves the instance into a transitional state under CAS and returns the record
// as written. from lists the statuses the operation is legal from, so a stop of
// an already-stopping instance is rejected as a state error rather than racing
+20
View File
@@ -679,3 +679,23 @@ func TestReconciler_MarksFailedWhenARestartOverrunsItsBound(t *testing.T) {
assert.Equal(t, StatusFailed, rec.Status)
assert.Contains(t, rec.FailureReason, "did not report healthy")
}
// What the next start recovers is the engine's own guarantee. Telling a MariaDB
// customer a write-ahead log will bring back an Aria or MyISAM table would be a
// false assurance about exactly the data most at risk.
func TestUncleanStopMessage_IsEngineAware(t *testing.T) {
postgres := uncleanStopMessage(t.Context(), "postgres", "stopping the instance")
assert.Contains(t, postgres, "could not be shut down cleanly before stopping the instance")
assert.Contains(t, postgres, "write-ahead log")
mariadb := uncleanStopMessage(t.Context(), "mariadb", "rebooting the instance")
assert.Contains(t, mariadb, "could not be shut down cleanly before rebooting the instance")
assert.Contains(t, mariadb, "InnoDB tables will recover")
assert.Contains(t, mariadb, "may be left inconsistent")
assert.NotContains(t, mariadb, "write-ahead log")
// The VM is going down either way, so an engine this build cannot resolve
// still gets the half of the warning that does not depend on knowing it.
unknown := uncleanStopMessage(t.Context(), "oracle", "stopping the instance")
assert.Equal(t, "The database engine could not be shut down cleanly before stopping the instance.", unknown)
}
+5 -1
View File
@@ -237,11 +237,15 @@ func (s *Service) planModify(ctx context.Context, input *rds.ModifyDBInstanceInp
if plan.ParameterGroup != "" {
targetGroup = plan.ParameterGroup
}
engine, err := LookupEngine(rec.Engine)
if err != nil {
return nil, err
}
kv, err := s.bucket(ctx, accountID)
if err != nil {
return nil, err
}
if _, err := s.resolveGroupParameters(ctx, kv, accountID, targetGroup, targetClass); err != nil {
if _, err := s.resolveGroupParameters(ctx, kv, accountID, engine, targetGroup, targetClass); err != nil {
return nil, err
}
}
+106 -498
View File
@@ -7,19 +7,15 @@ import (
"slices"
"strconv"
"strings"
"time"
_ "time/tzdata"
"unicode"
"github.com/mulgadc/spinifex/spinifex/awserrors"
"github.com/mulgadc/spinifex/spinifex/instancetypes"
)
// The static PostgreSQL 18 parameter catalog, following the EKS add-on catalog
// pattern: a curated in-binary table rather than the engine's full ~350 GUCs.
// A curated subset that is genuinely validated is worth more than the whole set
// passed through unchecked — a static parameter the engine refuses at startup
// is a boot loop, and the bad config is on the persistent data volume.
// The engine-neutral half of the parameter catalog: the spec type, its parsing
// and range checks, and the resolver. Each engine supplies its own table, its
// size-derived formulas and its combination checks; everything here is shared.
// When a parameter takes effect. Static settings are stored and reported
// pending-reboot; dynamic ones are adopted by a reload.
@@ -75,371 +71,22 @@ type ParameterSpec struct {
// unbounded below and above.
Min, Max float64
// The permitted values of an enum parameter, lowest-to-highest where the
// engine gives them an order.
// engine gives them an order. On a boolean it narrows the spellings the
// engine parses, which are not the same set for every engine.
Enum []string
// The engine's own unit suffix, reported in AllowedValues so a customer can
// see what an integer means. Empty for unitless settings.
Unit string
}
// The catalog, keyed by parameter name. Every entry here is one a customer has a
// real reason to tune; the platform-owned settings (port, listen_addresses,
// ssl*, unix_socket_directories, data_directory) are deliberately absent rather
// than present-and-unmodifiable, because they are not the customer's to set.
var parameterCatalog = buildParameterCatalog(
// Connections. max_connections is what a size-derived default matters most
// for: RDS's own formula is LEAST({DBInstanceClassMemory/9531392}, 5000).
ParameterSpec{
Name: "max_connections", DataType: paramTypeInteger, ApplyType: ApplyTypeStatic,
IsModifiable: true, Min: 6, Max: 5000,
DefaultFor: maxConnectionsFor,
MaxFor: maxConnectionsCeilingFor,
Description: "Maximum number of concurrent connections to the database server.",
},
ParameterSpec{
Name: "superuser_reserved_connections", DataType: paramTypeInteger, ApplyType: ApplyTypeStatic,
IsModifiable: true, Min: 0, Max: 100, Default: "3",
Description: "Connection slots reserved for superusers.",
},
ParameterSpec{
Name: "idle_in_transaction_session_timeout", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 2147483647, Default: "0", Unit: "ms",
Description: "Milliseconds an idle open transaction may live before it is aborted; 0 disables.",
},
ParameterSpec{
Name: "statement_timeout", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 2147483647, Default: "0", Unit: "ms",
Description: "Milliseconds a statement may run before it is aborted; 0 disables.",
},
ParameterSpec{
Name: "lock_timeout", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 2147483647, Default: "0", Unit: "ms",
Description: "Milliseconds to wait for a lock before failing the statement; 0 disables.",
},
ParameterSpec{
Name: "tcp_keepalives_idle", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 3600, Default: "0", Unit: "s",
Description: "Idle seconds before the server sends a TCP keepalive; 0 takes the system default.",
},
// The engine's own rule for a value the generic type and range checks cannot
// express, such as a zone name that must resolve or a comma-separated list of
// engine mode names. Runs last, on the trimmed value.
Validate func(value string) error
// Memory. Every one of these is a fraction of class memory on real RDS, so a
// literal tuned for the large end makes db.t3.micro fail to start.
ParameterSpec{
Name: "shared_buffers", DataType: paramTypeInteger, ApplyType: ApplyTypeStatic,
IsModifiable: true, Min: 16, Max: 4194304, Unit: "8kB",
DefaultFor: sharedBuffersFor,
MaxFor: sharedBuffersCeilingFor,
Description: "Shared memory buffers the server uses, in 8 kB blocks.",
},
ParameterSpec{
Name: "effective_cache_size", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 1, Max: 4194304, Unit: "8kB",
DefaultFor: effectiveCacheSizeFor,
MaxFor: effectiveCacheSizeCeilingFor,
Description: "Planner assumption about the disk cache available to one query, in 8 kB blocks.",
},
// Per sort or hash rather than per connection, so RDS leaves it a literal
// even in the memory group; a size-derived default here would multiply by
// max_connections and by the parallel workers under each of them.
ParameterSpec{
Name: "work_mem", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 64, Max: 2147483647, Default: "4096", Unit: "kB",
Description: "Memory one sort or hash operation may use before spilling to disk, in kB.",
},
ParameterSpec{
Name: "maintenance_work_mem", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 1024, Max: 2147483647, Unit: "kB",
DefaultFor: maintenanceWorkMemFor,
MaxFor: maintenanceWorkMemCeilingFor,
Description: "Memory a VACUUM, CREATE INDEX or ALTER TABLE ADD FOREIGN KEY may use, in kB.",
},
ParameterSpec{
Name: "temp_buffers", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 100, Max: 1073741823, Default: "1024", Unit: "8kB",
Description: "Per-session buffers for temporary tables, in 8 kB blocks.",
},
ParameterSpec{
Name: "max_prepared_transactions", DataType: paramTypeInteger, ApplyType: ApplyTypeStatic,
IsModifiable: true, Min: 0, Max: 262143, Default: "0",
Description: "Maximum simultaneously prepared transactions; 0 disables two-phase commit.",
},
// Parallelism. Bounded by the class's vCPU count on a real deployment, but
// PostgreSQL degrades gracefully when they exceed it, so literals are honest.
ParameterSpec{
Name: "max_worker_processes", DataType: paramTypeInteger, ApplyType: ApplyTypeStatic,
IsModifiable: true, Min: 0, Max: 262143, Default: "8",
Description: "Maximum background processes the server may start.",
},
ParameterSpec{
Name: "max_parallel_workers", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 1024, Default: "8",
Description: "Maximum workers the server may use for parallel operations.",
},
ParameterSpec{
Name: "max_parallel_workers_per_gather", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 1024, Default: "2",
Description: "Maximum parallel workers one Gather node may start.",
},
// WAL and checkpoints.
ParameterSpec{
Name: "wal_level", DataType: paramTypeEnum, ApplyType: ApplyTypeStatic,
IsModifiable: true, Enum: []string{"minimal", "replica", "logical"}, Default: "replica",
Description: "How much information is written to the WAL.",
},
ParameterSpec{
Name: "synchronous_commit", DataType: paramTypeEnum, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Enum: []string{"off", "local", "remote_write", "on", "remote_apply"}, Default: "on",
Description: "How much WAL processing must complete before a commit returns.",
},
// Alpine's PostgreSQL 18 package is built with both optional compression
// libraries, so every engine method is safe to expose.
ParameterSpec{
Name: "wal_compression", DataType: paramTypeEnum, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Enum: []string{"off", "pglz", "lz4", "zstd", "on"}, Default: "off",
Description: "Compression method for full-page images written to the WAL.",
},
// The image uses PostgreSQL's 16 MiB WAL segments; both size settings must
// hold at least two segments or startup rejects the configuration.
ParameterSpec{
Name: "max_wal_size", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 32, Max: 2097151, Default: "1024", Unit: "MB",
Description: "WAL size that triggers a checkpoint, in MB.",
},
ParameterSpec{
Name: "min_wal_size", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 32, Max: 2097151, Default: "80", Unit: "MB",
Description: "WAL size below which old segments are recycled rather than removed, in MB.",
},
ParameterSpec{
Name: "checkpoint_timeout", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 30, Max: 86400, Default: "300", Unit: "s",
Description: "Maximum seconds between automatic WAL checkpoints.",
},
ParameterSpec{
Name: "checkpoint_completion_target", DataType: paramTypeReal, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 1, Default: "0.9",
Description: "Fraction of the checkpoint interval a checkpoint's writes are spread over.",
},
ParameterSpec{
Name: "wal_buffers", DataType: paramTypeInteger, ApplyType: ApplyTypeStatic,
IsModifiable: true, Min: -1, Max: 262143, Default: "-1", Unit: "8kB",
Description: "Shared memory used for WAL not yet written, in 8 kB blocks; -1 derives it from shared_buffers.",
},
ParameterSpec{
Name: "max_wal_senders", DataType: paramTypeInteger, ApplyType: ApplyTypeStatic,
IsModifiable: true, Min: 0, Max: 262143, Default: "10",
Description: "Maximum simultaneously running WAL sender processes.",
},
ParameterSpec{
Name: "max_replication_slots", DataType: paramTypeInteger, ApplyType: ApplyTypeStatic,
IsModifiable: true, Min: 0, Max: 262143, Default: "10",
Description: "Maximum replication slots the server may define.",
},
// Autovacuum. Turning it off is the single most reliable way to take a
// PostgreSQL database down slowly, so it is offered but its default is on.
ParameterSpec{
Name: "autovacuum", DataType: paramTypeBoolean, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Default: "on",
Description: "Whether the autovacuum launcher runs.",
},
// PostgreSQL 18 moved this setting to SIGHUP; worker slot allocation is now
// controlled separately by the postmaster-only autovacuum_worker_slots.
ParameterSpec{
Name: "autovacuum_max_workers", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 1, Max: 262143, Default: "3",
Description: "Maximum autovacuum worker processes running at once.",
},
ParameterSpec{
Name: "autovacuum_naptime", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 1, Max: 2147483, Default: "60", Unit: "s",
Description: "Seconds between autovacuum runs on any one database.",
},
ParameterSpec{
Name: "autovacuum_vacuum_threshold", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 2147483647, Default: "50",
Description: "Row updates or deletes before a table is vacuumed.",
},
ParameterSpec{
Name: "autovacuum_vacuum_scale_factor", DataType: paramTypeReal, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 100, Default: "0.2",
Description: "Fraction of the table size added to autovacuum_vacuum_threshold.",
},
ParameterSpec{
Name: "autovacuum_analyze_threshold", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 2147483647, Default: "50",
Description: "Row inserts, updates or deletes before a table is analyzed.",
},
ParameterSpec{
Name: "autovacuum_analyze_scale_factor", DataType: paramTypeReal, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 100, Default: "0.1",
Description: "Fraction of the table size added to autovacuum_analyze_threshold.",
},
ParameterSpec{
Name: "autovacuum_vacuum_cost_limit", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: -1, Max: 10000, Default: "-1",
Description: "Cost budget one autovacuum worker spends before sleeping; -1 takes vacuum_cost_limit.",
},
// Planner.
// The engine's own ceiling on a planner cost is DBL_MAX, which is not a bound
// worth reporting: a cost past four digits already makes the plan choice
// insensitive to it, so the range stays one a customer can read.
ParameterSpec{
Name: "random_page_cost", DataType: paramTypeReal, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 10000, Default: "1.1",
Description: "Planner estimate of the cost of a non-sequentially fetched page.",
},
ParameterSpec{
Name: "seq_page_cost", DataType: paramTypeReal, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 10000, Default: "1",
Description: "Planner estimate of the cost of a sequentially fetched page.",
},
ParameterSpec{
Name: "effective_io_concurrency", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 0, Max: 1000, Default: "16",
Description: "Concurrent disk I/O operations the planner assumes are useful.",
},
ParameterSpec{
Name: "default_statistics_target", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 1, Max: 10000, Default: "100",
Description: "Default statistics target for table columns without one of their own.",
},
ParameterSpec{
Name: "jit", DataType: paramTypeBoolean, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Default: "on",
Description: "Whether JIT compilation may be used for qualifying queries.",
},
// Logging. The destination and the collector are platform-owned; what is
// logged is the customer's.
ParameterSpec{
Name: "log_min_duration_statement", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: -1, Max: 2147483647, Default: "-1", Unit: "ms",
Description: "Milliseconds a statement must run to be logged; 0 logs every statement, -1 disables.",
},
ParameterSpec{
Name: "log_statement", DataType: paramTypeEnum, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Enum: []string{"none", "ddl", "mod", "all"}, Default: "none",
Description: "Which SQL statements are written to the log.",
},
ParameterSpec{
Name: "log_min_messages", DataType: paramTypeEnum, ApplyType: ApplyTypeDynamic,
IsModifiable: true,
Enum: []string{"debug5", "debug4", "debug3", "debug2", "debug1", "info", "notice",
"warning", "error", "log", "fatal", "panic"},
Default: "warning",
Description: "Lowest message severity written to the server log.",
},
ParameterSpec{
Name: "log_connections", DataType: paramTypeBoolean, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Default: "off",
Description: "Whether each successful connection attempt is logged.",
},
ParameterSpec{
Name: "log_disconnections", DataType: paramTypeBoolean, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Default: "off",
Description: "Whether the end of each session is logged, with its duration.",
},
ParameterSpec{
Name: "log_lock_waits", DataType: paramTypeBoolean, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Default: "on",
Description: "Whether a session waiting longer than deadlock_timeout for a lock is logged.",
},
ParameterSpec{
Name: "log_temp_files", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: -1, Max: 2147483647, Default: "-1", Unit: "kB",
Description: "Size in kB above which a temporary file's removal is logged; 0 logs all, -1 disables.",
},
ParameterSpec{
Name: "log_autovacuum_min_duration", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: -1, Max: 2147483647, Default: "-1", Unit: "ms",
Description: "Milliseconds an autovacuum action must run to be logged; -1 disables.",
},
// Timeouts and locale.
ParameterSpec{
Name: "deadlock_timeout", DataType: paramTypeInteger, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Min: 1, Max: 2147483647, Default: "1000", Unit: "ms",
Description: "Milliseconds to wait on a lock before checking for a deadlock.",
},
ParameterSpec{
Name: "timezone", DataType: paramTypeString, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Default: "UTC",
Description: "Time zone the server displays and interprets timestamps in.",
},
ParameterSpec{
Name: "datestyle", DataType: paramTypeString, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Default: "ISO, MDY",
Description: "Display format for date and time values.",
},
ParameterSpec{
Name: "track_activity_query_size", DataType: paramTypeInteger, ApplyType: ApplyTypeStatic,
IsModifiable: true, Min: 100, Max: 1048576, Default: "1024", Unit: "B",
Description: "Bytes of each running query pg_stat_activity retains.",
},
ParameterSpec{
Name: "track_io_timing", DataType: paramTypeBoolean, ApplyType: ApplyTypeDynamic,
IsModifiable: true, Default: "on",
Description: "Whether block read and write times are collected.",
},
)
// The bytes of class memory an RDS parameter formula divides. Real RDS exposes
// it as {DBInstanceClassMemory}, and the resolver evaluates the formulas here
// rather than passing them to the engine.
const mibToBytes = 1024 * 1024
// shared_buffers = {DBInstanceClassMemory/32768}, in 8 kB blocks — a quarter of
// class memory, which is RDS's own default.
func sharedBuffersFor(memoryMiB int64) string {
return strconv.FormatInt(clampInt64(memoryMiB*mibToBytes/32768, 16, 4194304), 10)
}
// effective_cache_size = {DBInstanceClassMemory/16384}, in 8 kB blocks — half of
// class memory, the planner's assumption about what the OS is caching.
func effectiveCacheSizeFor(memoryMiB int64) string {
return strconv.FormatInt(clampInt64(memoryMiB*mibToBytes/16384, 1, 4194304), 10)
}
// max_connections = LEAST({DBInstanceClassMemory/9531392}, 5000). The floor is
// what keeps db.t3.micro above superuser_reserved_connections plus the workers
// autovacuum and replication need, which is where the smallest class would
// otherwise fail to start.
func maxConnectionsFor(memoryMiB int64) string {
return strconv.FormatInt(clampInt64(memoryMiB*mibToBytes/9531392, 20, 5000), 10)
}
// maintenance_work_mem = {DBInstanceClassMemory/63963136}, in kB. Capped so a
// large class does not let autovacuum_max_workers reserve the whole machine.
func maintenanceWorkMemFor(memoryMiB int64) string {
return strconv.FormatInt(clampInt64(memoryMiB*mibToBytes/63963136*1024, 16384, 2097152), 10)
}
// Class ceilings deliberately leave substantial headroom above the defaults.
// They prevent obviously impossible large-class literals without prescribing
// production tuning for values the guest can support.
func sharedBuffersCeilingFor(memoryMiB int64) int64 {
return clampInt64(memoryMiB*1024/8*3/4, 16, 4194304)
}
func effectiveCacheSizeCeilingFor(memoryMiB int64) int64 {
return clampInt64(memoryMiB*1024/8, 1, 4194304)
}
func maxConnectionsCeilingFor(memoryMiB int64) int64 {
defaults := clampInt64(memoryMiB*mibToBytes/9531392, 20, 5000)
return clampInt64(defaults*4, 20, 5000)
}
func maintenanceWorkMemCeilingFor(memoryMiB int64) int64 {
return clampInt64(memoryMiB*1024/2, 16384, 2147483647)
}
func clampInt64(v, lo, hi int64) int64 {
return min(max(v, lo), hi)
// The spelling the engine accepts for this setting in its option file, when
// that differs from the name the customer sets. Empty means the two are the
// same, which is every parameter but MariaDB's time_zone.
optionFileName string
}
// Indexes the specs by name and fails the build-equivalent — process start — on
@@ -467,15 +114,26 @@ func buildParameterCatalog(specs ...ParameterSpec) map[string]ParameterSpec {
// The catalog entry for a parameter name, or false when the engine has no such
// setting or it is one this platform does not expose.
func LookupParameter(name string) (ParameterSpec, bool) {
spec, ok := parameterCatalog[strings.ToLower(strings.TrimSpace(name))]
func (e Engine) LookupParameter(name string) (ParameterSpec, bool) {
spec, ok := e.catalog[strings.ToLower(strings.TrimSpace(name))]
return spec, ok
}
// The spelling to write into the engine's option file for a parameter the
// customer set. A name the server does not accept there is a boot loop with the
// bad file already on the data volume, so the two names are kept apart.
func (e Engine) OptionFileName(name string) string {
spec, ok := e.LookupParameter(name)
if !ok || spec.optionFileName == "" {
return name
}
return spec.optionFileName
}
// Sorted, so a describe returns the same order on every call and Terraform does
// not read a reshuffle as drift.
func CatalogParameterNames() []string {
return slices.Sorted(maps.Keys(parameterCatalog))
func (e Engine) CatalogParameterNames() []string {
return slices.Sorted(maps.Keys(e.catalog))
}
// The engine default for one parameter at one instance class: the literal, or
@@ -500,12 +158,24 @@ func (s ParameterSpec) AllowedValues() string {
case paramTypeEnum:
return strings.Join(s.Enum, ",")
case paramTypeBoolean:
return "on,off,true,false,yes,no,1,0"
return strings.Join(s.booleanSpellings(), ",")
default:
return ""
}
}
// The spellings the engine parses for a boolean. PostgreSQL takes all of these;
// MariaDB refuses yes and no, so its specs narrow the set rather than let the
// API accept a value mysqld will not parse.
var defaultBooleanSpellings = []string{"on", "off", "true", "false", "yes", "no", "1", "0"}
func (s ParameterSpec) booleanSpellings() []string {
if len(s.Enum) > 0 {
return s.Enum
}
return defaultBooleanSpellings
}
// Integral bounds print without a decimal point, so an integer parameter's range
// does not read as a real one's.
func formatBound(v float64) string {
@@ -549,8 +219,8 @@ func rejectFormulaValue(name, value string) error {
// catalog does not hold, one the platform owns, and one outside its own range
// are all rejected here — at the API, rather than by an engine that then refuses
// to start with the bad config already on the data volume.
func validateParameterValue(name, value string) (ParameterSpec, error) {
spec, ok := LookupParameter(name)
func (e Engine) validateParameterValue(name, value string) (ParameterSpec, error) {
spec, ok := e.LookupParameter(name)
if !ok {
return ParameterSpec{}, awserrors.Errorf(awserrors.ErrorInvalidParameterValue,
"%q is not a parameter this engine exposes", name)
@@ -586,8 +256,9 @@ func validateParameterValue(name, value string) (ParameterSpec, error) {
return ParameterSpec{}, rangeError(spec, value)
}
case paramTypeBoolean:
if !validBoolean(trimmed) {
return ParameterSpec{}, typeError(spec, value, "a boolean (on, off, true, false, yes, no, 1, 0)")
if !slices.Contains(spec.booleanSpellings(), strings.ToLower(trimmed)) {
return ParameterSpec{}, typeError(spec, value,
"a boolean ("+strings.Join(spec.booleanSpellings(), ", ")+")")
}
case paramTypeEnum:
if !slices.Contains(spec.Enum, strings.ToLower(trimmed)) {
@@ -595,7 +266,12 @@ func validateParameterValue(name, value string) (ParameterSpec, error) {
"parameter %s does not accept %q; allowed values are %s", spec.Name, value, strings.Join(spec.Enum, ", "))
}
case paramTypeString:
if err := validateStringParameter(spec, value, trimmed); err != nil {
if err := validateStringParameter(spec, trimmed); err != nil {
return ParameterSpec{}, err
}
}
if spec.Validate != nil {
if err := spec.Validate(trimmed); err != nil {
return ParameterSpec{}, err
}
}
@@ -604,7 +280,9 @@ func validateParameterValue(name, value string) (ParameterSpec, error) {
const maxStringParameterBytes = 1024
func validateStringParameter(spec ParameterSpec, value, trimmed string) error {
// The bounds every engine's free-form string shares. What a particular setting
// means is the engine's own rule, carried on the spec's Validate.
func validateStringParameter(spec ParameterSpec, trimmed string) error {
if len(trimmed) > maxStringParameterBytes {
return awserrors.Errorf(awserrors.ErrorInvalidParameterValue,
"parameter %s exceeds the maximum length of %d bytes", spec.Name, maxStringParameterBytes)
@@ -617,64 +295,9 @@ func validateStringParameter(spec ParameterSpec, value, trimmed string) error {
return awserrors.Errorf(awserrors.ErrorInvalidParameterValue,
"parameter %s cannot contain control characters", spec.Name)
}
switch spec.Name {
case "timezone":
if strings.EqualFold(trimmed, "Local") {
return invalidTimezone(value)
}
if _, err := time.LoadLocation(trimmed); err != nil {
return invalidTimezone(value)
}
case "datestyle":
if !validDateStyle(trimmed) {
return awserrors.Errorf(awserrors.ErrorInvalidParameterValue,
"parameter datestyle does not accept %q; use one output style (ISO, SQL, Postgres, German) and one date order (MDY, DMY, YMD)", value)
}
}
return nil
}
func invalidTimezone(value string) error {
return awserrors.Errorf(awserrors.ErrorInvalidParameterValue,
"parameter timezone does not accept %q; use an IANA time zone name such as UTC or Australia/Sydney", value)
}
func validDateStyle(value string) bool {
parts := strings.Split(value, ",")
if len(parts) == 0 || len(parts) > 2 {
return false
}
seenStyle, seenOrder := false, false
for _, part := range parts {
switch strings.ToLower(strings.TrimSpace(part)) {
case "iso", "sql", "postgres", "german":
if seenStyle {
return false
}
seenStyle = true
case "mdy", "dmy", "ymd":
if seenOrder {
return false
}
seenOrder = true
default:
return false
}
}
return seenStyle || seenOrder
}
// PostgreSQL's own boolean spellings, so a config a customer copied from the
// engine's documentation is accepted as the engine would accept it.
func validBoolean(value string) bool {
switch strings.ToLower(value) {
case "on", "off", "true", "false", "yes", "no", "1", "0":
return true
default:
return false
}
}
func typeError(spec ParameterSpec, value, want string) error {
return awserrors.Errorf(awserrors.ErrorInvalidParameterValue,
"parameter %s takes %s, not %q", spec.Name, want, value)
@@ -692,18 +315,22 @@ func rangeError(spec ParameterSpec, value string) error {
//
// Overrides are re-validated rather than trusted: a catalog whose bounds
// tightened must not keep handing the engine a value it would now reject.
func ResolveEffectiveParameters(instanceClass string, overrides map[string]string) ([]Parameter, error) {
func (e Engine) ResolveEffectiveParameters(instanceClass string, overrides map[string]string) ([]Parameter, error) {
if e.validateCombinations == nil {
return nil, awserrors.Errorf(awserrors.ErrorServerInternal,
"engine %s registers no parameter combination checks", e.Name)
}
memoryMiB, err := classMemoryMiB(instanceClass)
if err != nil {
return nil, err
}
names := CatalogParameterNames()
names := e.CatalogParameterNames()
resolved := make([]Parameter, 0, len(names))
for _, name := range names {
spec := parameterCatalog[name]
spec := e.catalog[name]
value := spec.DefaultAt(memoryMiB)
if override, ok := overrides[name]; ok {
if _, err := validateParameterValue(name, override); err != nil {
if _, err := e.validateParameterValue(name, override); err != nil {
return nil, err
}
if err := validateClassParameterValue(instanceClass, memoryMiB, spec, override); err != nil {
@@ -713,12 +340,54 @@ func ResolveEffectiveParameters(instanceClass string, overrides map[string]strin
}
resolved = append(resolved, Parameter{Name: name, Value: value})
}
if err := validateParameterCombinations(resolved); err != nil {
if err := e.validateCombinations(resolved); err != nil {
return nil, err
}
return resolved, nil
}
// The bytes of class memory an RDS parameter formula divides. Real RDS exposes
// it as {DBInstanceClassMemory}, and the resolver evaluates the formulas here
// rather than passing them to the engine.
const mibToBytes = 1024 * 1024
func clampInt64(v, lo, hi int64) int64 {
return min(max(v, lo), hi)
}
// Reads one setting out of a resolved set for a combination check. The set is
// every catalog name by construction, so an absent one is a catalog bug rather
// than anything a customer did.
func resolvedValues(params []Parameter) map[string]string {
values := make(map[string]string, len(params))
for _, param := range params {
values[param.Name] = strings.ToLower(strings.TrimSpace(param.Value))
}
return values
}
func resolvedString(values map[string]string, name string) (string, error) {
value, ok := values[name]
if !ok {
return "", awserrors.Errorf(awserrors.ErrorServerInternal,
"resolved parameter set is missing %s", name)
}
return value, nil
}
func resolvedInteger(values map[string]string, name string) (int64, error) {
value, err := resolvedString(values, name)
if err != nil {
return 0, err
}
n, err := strconv.ParseInt(value, 10, 64)
if err != nil {
return 0, awserrors.Errorf(awserrors.ErrorServerInternal,
"resolved parameter %s has invalid integer value %q", name, value)
}
return n, nil
}
func validateClassParameterValue(instanceClass string, memoryMiB int64, spec ParameterSpec, value string) error {
if spec.MaxFor == nil {
return nil
@@ -735,64 +404,3 @@ func validateClassParameterValue(instanceClass string, memoryMiB int64, spec Par
}
return nil
}
func validateParameterCombinations(params []Parameter) error {
values := make(map[string]string, len(params))
for _, param := range params {
values[param.Name] = strings.ToLower(strings.TrimSpace(param.Value))
}
maxWALSenders, err := resolvedInteger(values, "max_wal_senders")
if err != nil {
return err
}
maxReplicationSlots, err := resolvedInteger(values, "max_replication_slots")
if err != nil {
return err
}
if values["wal_level"] == "minimal" && (maxWALSenders != 0 || maxReplicationSlots != 0) {
return awserrors.Errorf(awserrors.ErrorInvalidParameterValue,
"wal_level minimal requires max_wal_senders and max_replication_slots both to be 0")
}
maxConnections, err := resolvedInteger(values, "max_connections")
if err != nil {
return err
}
reservedConnections, err := resolvedInteger(values, "superuser_reserved_connections")
if err != nil {
return err
}
if reservedConnections >= maxConnections {
return awserrors.Errorf(awserrors.ErrorInvalidParameterValue,
"superuser_reserved_connections must be less than max_connections")
}
maxWorkerProcesses, err := resolvedInteger(values, "max_worker_processes")
if err != nil {
return err
}
const (
postgres18MaxBackends = 262143
postgres18AutovacuumWorkerSlots = 16
postgres18SpecialWorkerProcs = 2
)
backends := maxConnections + postgres18AutovacuumWorkerSlots + maxWorkerProcesses + maxWALSenders + postgres18SpecialWorkerProcs
if backends > postgres18MaxBackends {
return awserrors.Errorf(awserrors.ErrorInvalidParameterValue,
"max_connections, max_worker_processes and max_wal_senders reserve too many server processes")
}
return nil
}
func resolvedInteger(values map[string]string, name string) (int64, error) {
value, ok := values[name]
if !ok {
return 0, awserrors.Errorf(awserrors.ErrorServerInternal,
"resolved parameter set is missing %s", name)
}
n, err := strconv.ParseInt(value, 10, 64)
if err != nil {
return 0, awserrors.Errorf(awserrors.ErrorServerInternal,
"resolved parameter %s has invalid integer value %q", name, value)
}
return n, nil
}
+63 -47
View File
@@ -10,46 +10,62 @@ import (
"github.com/stretchr/testify/require"
)
// Every engine's catalog, whether or not it is registered in engines: a catalog
// has to satisfy these invariants before the engine is offered, not once a
// customer can reach it.
var catalogEngines = []Engine{enginePostgres, engineMariaDB}
// The failure this guards is the one the phase opens by warning about, arriving
// through the default path rather than a customer typo: a catalog change that
// makes the smallest class unbootable has to fail here rather than in a create.
func TestParameterCatalog_DefaultsSatisfyTheirOwnConstraintsAtEveryClass(t *testing.T) {
for _, class := range SupportedInstanceClasses() {
t.Run(class, func(t *testing.T) {
memoryMiB, err := classMemoryMiB(class)
require.NoError(t, err)
for _, engine := range catalogEngines {
for _, class := range SupportedInstanceClasses() {
t.Run(engine.Name+"/"+class, func(t *testing.T) {
memoryMiB, err := classMemoryMiB(class)
require.NoError(t, err)
for _, name := range CatalogParameterNames() {
spec, ok := LookupParameter(name)
require.True(t, ok)
for _, name := range engine.CatalogParameterNames() {
spec, ok := engine.LookupParameter(name)
require.True(t, ok)
value := spec.DefaultAt(memoryMiB)
assert.NotEmpty(t, value, "%s has an empty default at %s", name, class)
// Validated by the same function a customer's value goes through, so
// a default cannot be something the API would reject.
_, err := validateParameterValue(name, value)
assert.NoError(t, err, "the default %q of %s is not a value %s accepts", value, name, class)
}
})
value := spec.DefaultAt(memoryMiB)
assert.NotEmpty(t, value, "%s has an empty default at %s", name, class)
// Validated by the same function a customer's value goes through, so
// a default cannot be something the API would reject.
_, err := engine.validateParameterValue(name, value)
assert.NoError(t, err, "the default %q of %s is not a value %s accepts", value, name, class)
}
// The defaults also have to satisfy the engine's combination checks,
// which no single parameter's own validation can see.
_, err = engine.ResolveEffectiveParameters(class, nil)
assert.NoError(t, err, "the untouched defaults do not resolve at %s", class)
})
}
}
}
// Every entry has to be a shape the resolver and the describe can both render,
// or a client reads a parameter with no type and no apply semantics.
func TestParameterCatalog_EntriesAreWellFormed(t *testing.T) {
for _, name := range CatalogParameterNames() {
spec, _ := LookupParameter(name)
for _, engine := range catalogEngines {
t.Run(engine.Name, func(t *testing.T) {
for _, name := range engine.CatalogParameterNames() {
spec, _ := engine.LookupParameter(name)
assert.Equal(t, name, spec.Name, "the catalog key and the spec name disagree")
assert.Contains(t, []string{ApplyTypeStatic, ApplyTypeDynamic}, spec.ApplyType, "%s", name)
assert.Contains(t, []string{paramTypeInteger, paramTypeReal, paramTypeBoolean, paramTypeString, paramTypeEnum},
spec.DataType, "%s", name)
if spec.DataType == paramTypeEnum {
assert.NotEmpty(t, spec.Enum, "%s is an enum with no allowed values", name)
}
if spec.DataType == paramTypeInteger || spec.DataType == paramTypeReal {
assert.Less(t, spec.Min, spec.Max, "%s has an empty range", name)
}
assert.Equal(t, name, spec.Name, "the catalog key and the spec name disagree")
assert.Contains(t, []string{ApplyTypeStatic, ApplyTypeDynamic}, spec.ApplyType, "%s", name)
assert.Contains(t, []string{paramTypeInteger, paramTypeReal, paramTypeBoolean, paramTypeString, paramTypeEnum},
spec.DataType, "%s", name)
if spec.DataType == paramTypeEnum {
assert.NotEmpty(t, spec.Enum, "%s is an enum with no allowed values", name)
}
if spec.DataType == paramTypeInteger || spec.DataType == paramTypeReal {
assert.Less(t, spec.Min, spec.Max, "%s has an empty range", name)
}
}
})
}
}
@@ -63,7 +79,7 @@ func TestParameterCatalog_SizeDerivedDefaultsScaleWithClassMemory(t *testing.T)
require.Less(t, smallest, largest)
for _, name := range []string{"shared_buffers", "effective_cache_size", "max_connections", "maintenance_work_mem"} {
spec, ok := LookupParameter(name)
spec, ok := enginePostgres.LookupParameter(name)
require.True(t, ok, "%s is expected to carry a computed default", name)
require.NotNil(t, spec.DefaultFor, "%s should be size-derived", name)
@@ -120,7 +136,7 @@ func TestValidateParameterValue_RejectsBadInput(t *testing.T) {
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
_, err := validateParameterValue(tc.param, tc.value)
_, err := enginePostgres.validateParameterValue(tc.param, tc.value)
require.Error(t, err)
assert.Equal(t, awserrors.ErrorInvalidParameterValue, awserrors.ValidErrorCodeFromError(err),
"the code has to survive resolution or the client sees a 500")
@@ -145,15 +161,15 @@ func TestValidateParameterValue_AcceptsPostgresStringValues(t *testing.T) {
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
_, err := validateParameterValue(tc.param, tc.value)
_, err := enginePostgres.validateParameterValue(tc.param, tc.value)
assert.NoError(t, err)
})
}
}
func TestValidateParameterValue_AcceptsEveryEnginesSpellingOfABoolean(t *testing.T) {
func TestValidateParameterValue_AcceptsEveryPostgresSpellingOfABoolean(t *testing.T) {
for _, value := range []string{"on", "OFF", "true", "False", "yes", "no", "1", "0"} {
_, err := validateParameterValue("autovacuum", value)
_, err := enginePostgres.validateParameterValue("autovacuum", value)
assert.NoError(t, err, "autovacuum rejected %q, which the engine accepts", value)
}
}
@@ -161,9 +177,9 @@ func TestValidateParameterValue_AcceptsEveryEnginesSpellingOfABoolean(t *testing
// A group's overrides are laid over the whole catalog, not merged into a subset:
// every parameter is present, and one the group does not set carries its default.
func TestResolveEffectiveParameters_OverlaysOverridesOnDefaults(t *testing.T) {
resolved, err := ResolveEffectiveParameters("db.t3.medium", map[string]string{"work_mem": "16384"})
resolved, err := enginePostgres.ResolveEffectiveParameters("db.t3.medium", map[string]string{"work_mem": "16384"})
require.NoError(t, err)
require.Len(t, resolved, len(CatalogParameterNames()))
require.Len(t, resolved, len(enginePostgres.CatalogParameterNames()))
values := map[string]string{}
for _, param := range resolved {
@@ -180,7 +196,7 @@ func TestResolveEffectiveParameters_OverlaysOverridesOnDefaults(t *testing.T) {
// Sorted output is what lets a re-resolve that changed nothing produce a
// byte-identical include, so an apply does not restart an engine for no reason.
func TestResolveEffectiveParameters_IsSortedAndCarriesNoFormula(t *testing.T) {
resolved, err := ResolveEffectiveParameters("db.m5.xlarge", nil)
resolved, err := enginePostgres.ResolveEffectiveParameters("db.m5.xlarge", nil)
require.NoError(t, err)
for i := 1; i < len(resolved); i++ {
@@ -195,7 +211,7 @@ func TestResolveEffectiveParameters_IsSortedAndCarriesNoFormula(t *testing.T) {
// A stored value the catalog would now reject must not keep being handed to the
// engine, so the resolve re-validates rather than trusting what was written.
func TestResolveEffectiveParameters_RevalidatesStoredOverrides(t *testing.T) {
_, err := ResolveEffectiveParameters("db.t3.medium", map[string]string{"max_connections": "999999"})
_, err := enginePostgres.ResolveEffectiveParameters("db.t3.medium", map[string]string{"max_connections": "999999"})
require.Error(t, err)
assert.Contains(t, err.Error(), "max_connections")
}
@@ -212,14 +228,14 @@ func TestResolveEffectiveParameters_BoundsSizeDerivedOverridesToTheClass(t *test
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
_, err := ResolveEffectiveParameters("db.t3.micro", map[string]string{tt.name: tt.value})
_, err := enginePostgres.ResolveEffectiveParameters("db.t3.micro", map[string]string{tt.name: tt.value})
require.Error(t, err)
assert.Equal(t, awserrors.ErrorInvalidParameterValue, awserrors.ValidErrorCodeFromError(err))
assert.Contains(t, err.Error(), "db.t3.micro")
assert.Contains(t, err.Error(), tt.value)
assert.Contains(t, err.Error(), "ceiling")
_, err = ResolveEffectiveParameters("db.m5.xlarge", map[string]string{tt.name: tt.value})
_, err = enginePostgres.ResolveEffectiveParameters("db.m5.xlarge", map[string]string{tt.name: tt.value})
assert.NoError(t, err, "the same literal should fit the larger class")
})
}
@@ -252,7 +268,7 @@ func TestResolveEffectiveParameters_RejectsFatalCombinations(t *testing.T) {
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
_, err := ResolveEffectiveParameters("db.m5.xlarge", tt.overrides)
_, err := enginePostgres.ResolveEffectiveParameters("db.m5.xlarge", tt.overrides)
require.Error(t, err)
assert.Equal(t, awserrors.ErrorInvalidParameterValue, awserrors.ValidErrorCodeFromError(err))
assert.Contains(t, err.Error(), tt.want)
@@ -261,22 +277,22 @@ func TestResolveEffectiveParameters_RejectsFatalCombinations(t *testing.T) {
}
func TestResolveEffectiveParameters_AcceptsMinimalWALWithoutReplication(t *testing.T) {
_, err := ResolveEffectiveParameters("db.t3.micro", map[string]string{
_, err := enginePostgres.ResolveEffectiveParameters("db.t3.micro", map[string]string{
"wal_level": "minimal", "max_wal_senders": "0", "max_replication_slots": "0",
})
require.NoError(t, err)
}
func TestParameterCatalog_Postgres18ApplyTypesAndBuiltCompressionMethods(t *testing.T) {
autovacuumWorkers, _ := LookupParameter("autovacuum_max_workers")
autovacuumWorkers, _ := enginePostgres.LookupParameter("autovacuum_max_workers")
assert.Equal(t, ApplyTypeDynamic, autovacuumWorkers.ApplyType)
walCompression, _ := LookupParameter("wal_compression")
walCompression, _ := enginePostgres.LookupParameter("wal_compression")
assert.ElementsMatch(t, []string{"off", "pglz", "lz4", "zstd", "on"}, walCompression.Enum)
}
func TestResolveEffectiveParameters_RejectsAnUnknownClass(t *testing.T) {
_, err := ResolveEffectiveParameters("db.x99.mega", nil)
_, err := enginePostgres.ResolveEffectiveParameters("db.x99.mega", nil)
require.Error(t, err)
assert.Equal(t, awserrors.ErrorInvalidParameterValue, awserrors.ValidErrorCodeFromError(err),
"the code has to survive resolution or the client sees a 500")
@@ -285,15 +301,15 @@ func TestResolveEffectiveParameters_RejectsAnUnknownClass(t *testing.T) {
// AllowedValues is what tells a customer what an integer means, so a unitful
// parameter has to report its unit rather than a bare pair of numbers.
func TestParameterSpec_AllowedValuesDescribesTheConstraint(t *testing.T) {
shared, _ := LookupParameter("shared_buffers")
shared, _ := enginePostgres.LookupParameter("shared_buffers")
assert.Equal(t, "16-4194304 (8kB)", shared.AllowedValues())
logStatement, _ := LookupParameter("log_statement")
logStatement, _ := enginePostgres.LookupParameter("log_statement")
assert.Equal(t, "none,ddl,mod,all", logStatement.AllowedValues())
timezone, _ := LookupParameter("timezone")
timezone, _ := enginePostgres.LookupParameter("timezone")
assert.Empty(t, timezone.AllowedValues(), "a free-form string has no constraint to report")
target, _ := LookupParameter("checkpoint_completion_target")
target, _ := enginePostgres.LookupParameter("checkpoint_completion_target")
assert.True(t, strings.HasPrefix(target.AllowedValues(), "0-1"), "got %q", target.AllowedValues())
}
+49 -26
View File
@@ -153,16 +153,24 @@ func (s *Service) ModifyDBParameterGroup(ctx context.Context, input *rds.ModifyD
"at most %d parameters may be modified in one request, got %d", maxParametersPerModify, len(input.Parameters))
}
updates, err := validateParameterUpdates(input.Parameters)
if err != nil {
return nil, err
}
kv, err := s.bucket(ctx, accountID)
if err != nil {
return nil, err
}
if _, _, err := getDBParameterGroup(ctx, kv, accountID, name); err != nil {
// The group is read before its values are checked because the engine that
// owns them comes from the group's family. Validating against anything else
// would store one engine's parameter into another's group and defer the
// failure to whichever instance next attached it.
rec, _, err := getDBParameterGroup(ctx, kv, accountID, name)
if err != nil {
return nil, err
}
engine, err := engineForFamily(rec.Family)
if err != nil {
return nil, err
}
updates, err := validateParameterUpdates(engine, input.Parameters)
if err != nil {
return nil, err
}
@@ -240,21 +248,28 @@ func (s *Service) DescribeDBParameters(ctx context.Context, input *rds.DescribeD
if err != nil {
return nil, err
}
if _, _, err := getDBParameterGroup(ctx, kv, accountID, name); err != nil {
rec, _, err := getDBParameterGroup(ctx, kv, accountID, name)
if err != nil {
return nil, err
}
// The catalog listed is the one the group's family names, so a group of one
// engine never reports another engine's settings.
engine, err := engineForFamily(rec.Family)
if err != nil {
return nil, err
}
overrides, err := ListDBParameterOverrides(ctx, kv, name)
if err != nil {
return nil, err
}
memoryMiB, err := classMemoryMiB(smallestInstanceClass())
memoryMiB, err := classMemoryMiB(SmallestInstanceClass())
if err != nil {
return nil, err
}
out := &rds.DescribeDBParametersOutput{}
for _, param := range CatalogParameterNames() {
spec, _ := LookupParameter(param)
for _, param := range engine.CatalogParameterNames() {
spec, _ := engine.LookupParameter(param)
override, isOverride := overrides[param]
value := override.Value
if !isOverride {
@@ -331,13 +346,13 @@ type parameterUpdate struct {
// Every entry is checked before any is written. A name repeated in one request
// keeps its last value, as AWS does.
func validateParameterUpdates(params []*rds.Parameter) ([]parameterUpdate, error) {
func validateParameterUpdates(engine Engine, params []*rds.Parameter) ([]parameterUpdate, error) {
byName := make(map[string]parameterUpdate, len(params))
for _, param := range params {
if param == nil {
return nil, awserrors.Errorf(awserrors.ErrorInvalidParameterValue, "a parameter entry is empty")
}
spec, err := validateParameterValue(aws.StringValue(param.ParameterName), aws.StringValue(param.ParameterValue))
spec, err := engine.validateParameterValue(aws.StringValue(param.ParameterName), aws.StringValue(param.ParameterValue))
if err != nil {
return nil, err
}
@@ -429,21 +444,19 @@ func engineForDefaultParameterGroup(name string) (Engine, bool) {
return Engine{}, false
}
// v1 pins one engine major, so the family is the pinned one or nothing. An empty
// family takes the pin rather than failing, since a client that omits it can
// only mean the one family this platform offers.
// An omitted family takes PostgreSQL's rather than failing, which AWS clients
// that predate a second engine depend on. The cost is that a group meant for
// another engine is created as a PostgreSQL one and is only refused a call
// later, when an instance of that engine tries to attach it.
func validateParameterGroupFamily(family string) (string, error) {
trimmed := strings.ToLower(strings.TrimSpace(family))
if trimmed == "" {
if normaliseFamily(family) == "" {
return enginePostgres.ParameterGroupFamily(), nil
}
for _, engine := range engines {
if trimmed == engine.ParameterGroupFamily() {
return engine.ParameterGroupFamily(), nil
}
engine, err := engineForFamily(family)
if err != nil {
return "", err
}
return "", awserrors.Errorf(awserrors.ErrorInvalidParameterValue,
"DBParameterGroupFamily %q is not offered; %s is the only supported family", family, enginePostgres.ParameterGroupFamily())
return engine.ParameterGroupFamily(), nil
}
func parameterSource(isOverride bool) string {
@@ -496,10 +509,20 @@ func (s *Service) projectParameterGroupRecord(rec *DBParameterGroupRecord) *rds.
// defaults evaluated at the class, overlaid with the group's stored overrides.
// A group that does not exist fails here rather than silently resolving to the
// bare defaults, which would run a database on settings nobody asked for.
func (s *Service) resolveGroupParameters(ctx context.Context, kv jetstream.KeyValue, accountID, group, instanceClass string) ([]Parameter, error) {
if _, _, err := getDBParameterGroup(ctx, kv, accountID, group); err != nil {
//
// Every path that binds a group to an instance comes through here — create,
// modify, restore, the deferred apply and group propagation — so the
// cross-engine refusal is one check rather than five.
func (s *Service) resolveGroupParameters(ctx context.Context, kv jetstream.KeyValue, accountID string, engine Engine, group, instanceClass string) ([]Parameter, error) {
rec, _, err := getDBParameterGroup(ctx, kv, accountID, group)
if err != nil {
return nil, err
}
if !strings.EqualFold(rec.Family, engine.ParameterGroupFamily()) {
return nil, awserrors.Errorf(awserrors.ErrorInvalidParameterCombination,
"DB parameter group %s is of family %s, which cannot be used by a %s DB instance; it requires a group of family %s",
group, rec.Family, engine.Name, engine.ParameterGroupFamily())
}
overrides, err := ListDBParameterOverrides(ctx, kv, group)
if err != nil {
return nil, err
@@ -508,5 +531,5 @@ func (s *Service) resolveGroupParameters(ctx context.Context, kv jetstream.KeyVa
for name, override := range overrides {
values[name] = override.Value
}
return ResolveEffectiveParameters(instanceClass, values)
return engine.ResolveEffectiveParameters(instanceClass, values)
}
+18 -12
View File
@@ -13,6 +13,7 @@ import (
const (
testParameterGroup = "tuned-pg"
testDefaultPG = "default.postgres18"
testDefaultMariaDB = "default.mariadb11.8"
)
func parameterGroupInput(name string) *rds.CreateDBParameterGroupInput {
@@ -80,7 +81,7 @@ func TestCreateDBParameterGroup_StoresAnEmptyGroup(t *testing.T) {
// A fresh group and the default group resolve to the same effective set,
// because a group holds overrides rather than a copy of the catalog.
params := describedParameters(t, h, testParameterGroup)
require.Len(t, params, len(CatalogParameterNames()))
require.Len(t, params, len(enginePostgres.CatalogParameterNames()))
for _, param := range params {
assert.Equal(t, ParameterSourceEngineDefault, aws.StringValue(param.Source))
}
@@ -139,10 +140,20 @@ func TestDescribeDBParameterGroups_ReportsTheImplicitDefault(t *testing.T) {
require.Len(t, named.DBParameterGroups, 1)
assert.Equal(t, testDefaultPG, aws.StringValue(named.DBParameterGroups[0].DBParameterGroupName))
// One per registered engine, as AWS does: an account that has never touched
// MariaDB still sees its default group, because the group is the engine's.
listed, err := h.svc.DescribeDBParameterGroups(t.Context(), &rds.DescribeDBParameterGroupsInput{}, testAccountID)
require.NoError(t, err)
require.Len(t, listed.DBParameterGroups, 1, "the default group is reported even with nothing created")
assert.Equal(t, testDefaultPG, aws.StringValue(listed.DBParameterGroups[0].DBParameterGroupName))
assert.Equal(t, []string{testDefaultMariaDB, testDefaultPG}, parameterGroupNames(listed),
"the default groups are reported even with nothing created")
}
func parameterGroupNames(out *rds.DescribeDBParameterGroupsOutput) []string {
var names []string
for _, group := range out.DBParameterGroups {
names = append(names, aws.StringValue(group.DBParameterGroupName))
}
return names
}
// Synthesised rather than stored, so it must appear exactly once alongside the
@@ -156,12 +167,7 @@ func TestDescribeDBParameterGroups_ListsCustomerGroupsBesideTheDefault(t *testin
listed, err := h.svc.DescribeDBParameterGroups(t.Context(), &rds.DescribeDBParameterGroupsInput{}, testAccountID)
require.NoError(t, err)
var names []string
for _, group := range listed.DBParameterGroups {
names = append(names, aws.StringValue(group.DBParameterGroupName))
}
assert.Equal(t, []string{"alpha", testDefaultPG, "zeta"}, names)
assert.Equal(t, []string{"alpha", testDefaultMariaDB, testDefaultPG, "zeta"}, parameterGroupNames(listed))
}
func TestDescribeDBParameterGroups_RejectsAnUnknownName(t *testing.T) {
@@ -441,7 +447,7 @@ func TestDescribeDBParameters_FiltersOnSource(t *testing.T) {
Source: aws.String(ParameterSourceEngineDefault),
}, testAccountID)
require.NoError(t, err)
assert.Len(t, defaults.Parameters, len(CatalogParameterNames())-1)
assert.Len(t, defaults.Parameters, len(enginePostgres.CatalogParameterNames())-1)
}
func TestDeleteDBParameterGroup_RemovesTheGroupAndItsValues(t *testing.T) {
@@ -527,7 +533,7 @@ func TestCreateDBInstance_ResolvesTheNamedGroupIntoTheBootstrapSet(t *testing.T)
rec := h.record(t, testDBInstanceID)
assert.Equal(t, testParameterGroup, rec.DBParameterGroupName)
require.Len(t, rec.Bootstrap.ResolvedParameters, len(CatalogParameterNames()))
require.Len(t, rec.Bootstrap.ResolvedParameters, len(enginePostgres.CatalogParameterNames()))
values := map[string]string{}
for _, param := range rec.Bootstrap.ResolvedParameters {
@@ -551,7 +557,7 @@ func TestCreateDBInstance_ResolvesTheDefaultGroupWhenNoneIsNamed(t *testing.T) {
rec := h.record(t, testDBInstanceID)
assert.Equal(t, testDefaultPG, rec.DBParameterGroupName)
assert.Len(t, rec.Bootstrap.ResolvedParameters, len(CatalogParameterNames()))
assert.Len(t, rec.Bootstrap.ResolvedParameters, len(enginePostgres.CatalogParameterNames()))
}
func TestCreateDBInstance_RejectsAnUnknownParameterGroup(t *testing.T) {
+8 -1
View File
@@ -143,7 +143,14 @@ func (s *Service) applyPendingModifications(ctx context.Context, kv jetstream.Ke
// the new group's overrides, so a parameter the old group set and the new one
// does not reverts to its default rather than lingering.
func (s *Service) applyParameterGroup(ctx context.Context, kv jetstream.KeyValue, accountID string, rec *DBInstanceRecord, group, instanceClass string) error {
resolved, err := s.resolveGroupParameters(ctx, kv, accountID, group, instanceClass)
engine, err := LookupEngine(rec.Engine)
if err != nil {
return err
}
// Reached from a deferred modify, whose group was checked at request time,
// and from group propagation, where the binding was checked at attach. A
// family mismatch here is corrupt state rather than a bad request.
resolved, err := s.resolveGroupParameters(ctx, kv, accountID, engine, group, instanceClass)
if err != nil {
return err
}
+1 -1
View File
@@ -382,7 +382,7 @@ func TestApplyPendingModifications_ParameterGroupChangeRevertsTheOldGroupsValues
for _, param := range issued[0].Parameters {
applied[param.Name] = param.Value
}
workMem, _ := LookupParameter("work_mem")
workMem, _ := enginePostgres.LookupParameter("work_mem")
assert.Equal(t, workMem.Default, applied["work_mem"],
"the old group's value should have reverted to the catalog default")
assert.False(t, h.record(t).ParametersRolledBack, "a successful corrected apply clears the rollback state")
+1
View File
@@ -180,6 +180,7 @@ func (s *Service) launchReplacementVM(ctx context.Context, accountID string, rec
GatewayCACert: s.deps.GatewayCACert,
Region: s.region,
DBInstanceIdentifier: rec.DBInstanceIdentifier,
Engine: rec.Engine,
EngineVersion: rec.EngineVersion,
EnginePort: rec.Port,
}),
+4 -1
View File
@@ -58,7 +58,9 @@ func (s *Service) RestoreDBInstanceFromDBSnapshot(ctx context.Context, input *rd
if err != nil {
return nil, err
}
parameters, err := s.resolveGroupParameters(ctx, kv, accountID, req.DBParameterGroupName, req.InstanceClass)
// resolveRestoreRequest forces the engine to the snapshot's, so a group of
// another engine is refused here however the request named it.
parameters, err := s.resolveGroupParameters(ctx, kv, accountID, req.Engine, req.DBParameterGroupName, req.InstanceClass)
if err != nil {
return nil, err
}
@@ -122,6 +124,7 @@ func (s *Service) RestoreDBInstanceFromDBSnapshot(ctx context.Context, input *rd
GatewayCACert: s.deps.GatewayCACert,
Region: s.region,
DBInstanceIdentifier: req.Identifier,
Engine: req.Engine.Name,
EngineVersion: req.EngineVersion,
EnginePort: req.Port,
}),
+1 -1
View File
@@ -40,7 +40,7 @@ func SupportedInstanceClasses() []string {
// The supported class with the least memory. Deliberately computed from the
// footprints rather than taken as the first name SupportedInstanceClasses
// reports: that list is sorted alphabetically, so its head is db.m5.large.
func smallestInstanceClass() string {
func SmallestInstanceClass() string {
smallest, least := "", int64(0)
for _, class := range SupportedInstanceClasses() {
memoryMiB, err := classMemoryMiB(class)
+2 -2
View File
@@ -62,9 +62,9 @@ func TestSupportedInstanceClasses_SortedAndComplete(t *testing.T) {
// memory, so reading the head of the sorted list would advertise a shared_buffers
// no small instance ever runs.
func TestSmallestInstanceClass_IsTheLeastMemory(t *testing.T) {
assert.Equal(t, "db.t3.micro", smallestInstanceClass())
assert.Equal(t, "db.t3.micro", SmallestInstanceClass())
least, err := classMemoryMiB(smallestInstanceClass())
least, err := classMemoryMiB(SmallestInstanceClass())
require.NoError(t, err)
for _, class := range SupportedInstanceClasses() {
memoryMiB, err := classMemoryMiB(class)
+19 -1
View File
@@ -153,7 +153,7 @@ func (s *Service) snapshotDataVolume(ctx context.Context, accountID string, rec
slog.WarnContext(ctx, "rds: the engine could not be quiesced; taking a crash-consistent snapshot",
"dbInstance", rec.DBInstanceIdentifier, "dbSnapshot", dbSnapshotIdentifier, "err", quiesceErr)
s.RecordEvent(ctx, accountID, EventSourceTypeDBInstance, rec.DBInstanceIdentifier,
"The database engine could not be quiesced before the snapshot; the snapshot is crash consistent and will recover from its write-ahead log when it is restored.",
crashConsistentSnapshotMessage(ctx, rec.Engine),
EventCategoryNotification, EventCategoryBackup)
} else {
defer s.releaseQuiesce(ctx, accountID, rec.DBInstanceIdentifier)
@@ -184,6 +184,24 @@ func (s *Service) snapshotDataVolume(ctx context.Context, accountID string, rec
return aws.StringValue(snapshot.SnapshotId), crashConsistent, nil
}
// The half of the warning that holds for every engine. What restoring such a
// snapshot then recovers does not: PostgreSQL replays every table from its
// write-ahead log, MariaDB only its InnoDB ones.
const crashConsistentSnapshotWarning = "The database engine could not be quiesced before the snapshot; " +
"the snapshot is crash consistent."
func crashConsistentSnapshotMessage(ctx context.Context, engineName string) string {
engine, err := LookupEngine(engineName)
if err != nil {
// The snapshot has already been taken, so the customer still gets the half
// of the warning that does not depend on knowing the engine.
slog.ErrorContext(ctx, "rds: the DB instance names an engine this build does not offer",
"engine", engineName, "err", err)
return crashConsistentSnapshotWarning
}
return crashConsistentSnapshotWarning + " " + engine.crashRecoveryNote
}
// Releases the quiesce on a context detached from the caller's, so a snapshot
// that failed because the request deadline expired still lets the engine out of
// backup mode rather than leaving it to the agent's own deadline.
+22 -1
View File
@@ -223,7 +223,28 @@ func TestCreateDBSnapshot_FallsBackToCrashConsistentWhenTheQuiesceFails(t *testi
// The customer is told, because a crash-consistent snapshot recovers rather
// than restores instantly.
assert.Contains(t, eventMessages(h.events(t, EventSourceTypeDBInstance, testDBID)),
"The database engine could not be quiesced before the snapshot; the snapshot is crash consistent and will recover from its write-ahead log when it is restored.")
"The database engine could not be quiesced before the snapshot; the snapshot is crash consistent. "+
"It will recover from its write-ahead log when it is restored.")
}
// What such a snapshot recovers is the engine's own guarantee. Telling a MariaDB
// customer a write-ahead log will bring back an Aria or MyISAM table would be a
// false assurance about exactly the data most at risk.
func TestCrashConsistentSnapshotMessage_IsEngineAware(t *testing.T) {
postgres := crashConsistentSnapshotMessage(t.Context(), "postgres")
assert.Contains(t, postgres, "crash consistent")
assert.Contains(t, postgres, "write-ahead log")
mariadb := crashConsistentSnapshotMessage(t.Context(), "mariadb")
assert.Contains(t, mariadb, "crash consistent")
assert.Contains(t, mariadb, "InnoDB tables will recover")
assert.Contains(t, mariadb, "may be left inconsistent")
assert.NotContains(t, mariadb, "write-ahead log")
// The snapshot has already been taken, so an engine this build cannot resolve
// still gets the half of the warning that does not depend on knowing it.
unknown := crashConsistentSnapshotMessage(t.Context(), "oracle")
assert.Equal(t, crashConsistentSnapshotWarning, unknown)
}
// The record only ever described a snapshot that now does not exist, so it goes
+6 -2
View File
@@ -20,8 +20,11 @@ type agentUserDataInput struct {
GatewayCACert string
Region string
DBInstanceIdentifier string
EngineVersion string
EnginePort int64
// The engine this VM is launched as. The agent checks it against the engine
// its own image bakes and refuses to bootstrap when the two disagree.
Engine string
EngineVersion string
EnginePort int64
}
type userDataFile struct {
@@ -36,6 +39,7 @@ func buildAgentUserData(in agentUserDataInput) string {
"RDS_GATEWAY_CA=" + agentGatewayCAPath,
"RDS_REGION=" + in.Region,
"RDS_DB_INSTANCE_IDENTIFIER=" + in.DBInstanceIdentifier,
"RDS_ENGINE=" + in.Engine,
"RDS_ENGINE_VERSION=" + in.EngineVersion,
fmt.Sprintf("RDS_ENGINE_PORT=%d", in.EnginePort),
}, "\n")
+4
View File
@@ -14,6 +14,7 @@ func testAgentUserDataInput() agentUserDataInput {
GatewayCACert: "-----BEGIN CERTIFICATE-----\nMIIB\n-----END CERTIFICATE-----\n",
Region: testRegion,
DBInstanceIdentifier: testDBID,
Engine: enginePostgres.Name,
EngineVersion: "18",
EnginePort: 5432,
}
@@ -29,6 +30,9 @@ func TestBuildAgentUserData(t *testing.T) {
assert.Contains(t, out, "RDS_GATEWAY_CA="+agentGatewayCAPath)
assert.Contains(t, out, "RDS_REGION="+testRegion)
assert.Contains(t, out, "RDS_DB_INSTANCE_IDENTIFIER="+testDBID)
// The agent checks this against the engine its own image bakes and refuses
// to bootstrap when the two disagree.
assert.Contains(t, out, "RDS_ENGINE=postgres")
assert.Contains(t, out, "RDS_ENGINE_VERSION=18")
assert.Contains(t, out, "RDS_ENGINE_PORT=5432")
+5 -1
View File
@@ -117,6 +117,10 @@ func (s *Service) validateCreateRequest(input *rds.CreateDBInstanceInput) (*vali
if err := ValidateMasterUserPassword(masterPassword); err != nil {
return nil, err
}
dbName := aws.StringValue(input.DBName)
if err := engine.ValidateDBName(dbName); err != nil {
return nil, err
}
port := engine.DefaultPort
if input.Port != nil {
@@ -168,7 +172,7 @@ func (s *Service) validateCreateRequest(input *rds.CreateDBInstanceInput) (*vali
Port: port,
MasterUsername: masterUsername,
MasterPassword: masterPassword,
DBName: aws.StringValue(input.DBName),
DBName: dbName,
SecurityGroupIDs: aws.StringValueSlice(input.VpcSecurityGroupIds),
DBSubnetGroupName: aws.StringValue(input.DBSubnetGroupName),
DBParameterGroupName: paramGroup,
+20
View File
@@ -534,6 +534,26 @@ var AvailableImages = map[string]Images{
"rds-data-volume-contract": "format-auth-v1",
},
},
// Same resolution path as rds-postgres — only the engine tags differ, which
// is what keeps a MariaDB request off the PostgreSQL image and vice versa.
"spinifex-rds-mariadb": {
Name: "spinifex-rds-mariadb",
Description: "Mulga RDS MariaDB image — Alpine 3.24.1 + MariaDB 11.8 + rds-init (install-db, master password, parameters and TLS applied at first boot)",
Distro: "alpine",
Version: "3.24.1",
Arch: "x86_64",
Platform: "Linux/UNIX",
CreatedAt: time.Date(2026, 8, 11, 0, 0, 0, 0, time.UTC),
URL: "https://iso.mulgadc.com/system-ami/spinifex-rds-mariadb-x86_64.qcow2",
Checksum: "https://iso.mulgadc.com/system-ami/spinifex-rds-mariadb-x86_64.qcow2.sha256",
ChecksumType: "sha256",
BootMode: "bios",
Tags: map[string]string{
"spinifex:managed-by": "rds", "engine": "mariadb", "engine-version": "11.8",
"rds-data-volume-contract": "format-auth-v1",
},
},
}
func ExtractDiskImageFromFile(imagepath string, tmpdir string) (diskimage string, err error) {
+22 -16
View File
@@ -21,9 +21,9 @@ import (
// So the client that proves a customer can reach the database has to be a VM in
// that subnet, driven over SSH — not psql on the machine running the test.
//
// The client is an Ubuntu gold image with the postgres client installed at boot,
// not the rds-postgres engine AMI: that image carries neither sshd nor
// cloud-init, so a VM launched from it would come up unreachable.
// The client is an Ubuntu gold image with both engines' clients installed at
// boot, not an engine AMI: those images carry neither sshd nor cloud-init, so a
// VM launched from one would come up unreachable.
const (
rdsClientUser = "ubuntu"
@@ -37,17 +37,19 @@ const (
// trust root. The engine AMI ships one; an Ubuntu client does not.
RDSClientCACertPath = "/home/ubuntu/spinifex-ca.pem"
// Postgres' assigned port, and the only one a DB instance publishes.
DBEnginePort = 5432
// Each engine's assigned port, and the only one an instance of it publishes.
PostgresEnginePort = 5432
MariaDBEnginePort = 3306
rdsClientSSHTimeout = 5 * time.Minute
rdsClientSetupTimeout = 5 * time.Minute
)
// rdsClientUserData installs the postgres client at first boot. Status and
// rdsClientUserData installs both engines' clients at first boot. Status and
// exit-code sentinels mirror the storagegrowth suite's protocol; the distinct
// exit codes separate "no apt egress" from "no such package", which otherwise
// both surface as a missing psql much later.
// exit codes separate "no apt egress" from "no such package", and one engine's
// client from the other's, which otherwise all surface as a missing binary much
// later.
var rdsClientUserData = fmt.Sprintf(`#!/bin/bash
set -u
trap 'echo $? > %[2]s' EXIT
@@ -57,6 +59,7 @@ echo running > %[1]s
export DEBIAN_FRONTEND=noninteractive
apt-get update -y || exit 10
apt-get install -y postgresql-client || exit 11
apt-get install -y mariadb-client || exit 12
echo done > %[1]s
`, rdsClientStatusFile, rdsClientExitCodeFile)
@@ -83,9 +86,12 @@ func RDSClientVM(t *testing.T, c *AWSClient, fx *Fixture, env *Env) SSHTarget {
// The default SG admits only same-SG members, so the runner's SSH and the
// client's connection to the DB endpoint both need opening explicitly. A
// caller supplying its own placement brings its own groups instead.
// caller supplying its own placement brings its own groups instead. Both
// engines' ports are opened here: the client VM is shared by every test that
// needs a connection, whichever engine it is connecting to.
EnsureDefaultSGOpen(t, c)
AuthorizeTCPIngress(t, c, vpc.SGID, DBEnginePort)
AuthorizeTCPIngress(t, c, vpc.SGID, PostgresEnginePort)
AuthorizeTCPIngress(t, c, vpc.SGID, MariaDBEnginePort)
return RDSClientVMIn(t, c, fx, env, RDSClientPlacement{SubnetID: vpc.SubnetID, SGID: vpc.SGID})
}
@@ -141,9 +147,9 @@ func RDSClientVMIn(t *testing.T, c *AWSClient, fx *Fixture, env *Env, p RDSClien
return tgt
}
// waitForRDSClientTooling polls the boot-time install's sentinels until psql is
// installed. A non-zero exit code is reported with the cloud-init log tail,
// since the interesting failure is always inside apt.
// waitForRDSClientTooling polls the boot-time install's sentinels until both
// clients are installed. A non-zero exit code is reported with the cloud-init
// log tail, since the interesting failure is always inside apt.
func waitForRDSClientTooling(t *testing.T, tgt SSHTarget) error {
t.Helper()
deadline := time.Now().Add(rdsClientSetupTimeout)
@@ -163,13 +169,13 @@ func waitForRDSClientTooling(t *testing.T, tgt SSHTarget) error {
if code != "" && code != "0" {
log, _ := GuestExec(tgt, "tail -40 /var/log/cloud-init-output.log 2>/dev/null || true")
return fmt.Errorf("client VM postgres-client install exited %s (status=%q):\n%s", code, status, log)
return fmt.Errorf("client VM database-client install exited %s (status=%q):\n%s", code, status, log)
}
if code == "0" && status == "done" {
return nil
}
if time.Now().After(deadline) {
return fmt.Errorf("client VM postgres-client install still %q after %s (last ssh error: %v)",
return fmt.Errorf("client VM database-client install still %q after %s (last ssh error: %v)",
status, rdsClientSetupTimeout, lastErr)
}
time.Sleep(5 * time.Second)
@@ -266,7 +272,7 @@ func TryPSQL(tgt SSHTarget, conn PSQLConn, sql string) (string, error) {
func psqlCommand(conn PSQLConn, sql string) string {
port := conn.Port
if port == 0 {
port = DBEnginePort
port = PostgresEnginePort
}
env := []string{
"PGPASSWORD=" + shellQuote(conn.Password),
+110
View File
@@ -0,0 +1,110 @@
//go:build e2e
package harness
import (
"strconv"
"strings"
"testing"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/rds"
)
// The MariaDB half of the client leg. The guest and its trust root are the
// shared ones RDSClientVM readies — only the wire protocol differs, so only the
// command line does.
// MariaDBConn addresses one mariadb invocation from inside the client guest.
// The client is MariaDB's own rather than MySQL's: the server's authentication
// plugins and its TLS options are what MariaDB's client is written against.
type MariaDBConn struct {
Host string
Port int64
User string
Password string
DBName string
// SSLRootCert is an in-guest path to the trust root, normally
// RDSClientCACertPath. Setting it is what turns TLS on: every option in the
// client's --ssl-* family implies an encrypted connection. --ssl itself is
// deliberately unused — the 11.x client deprecates it, and the deprecation
// notice would land in the same stream as the query's own output.
SSLRootCert string
// VerifyServerCert checks the serving certificate against the host being
// connected to, which is the other half of what verify-full means.
VerifyServerCert bool
}
// MariaDBConnFor builds a connection from an available DB instance's own
// reported endpoint, so a test cannot accidentally assert against an address the
// control plane never published.
func MariaDBConnFor(t *testing.T, instance *rds.DBInstance, user, password, dbName string) MariaDBConn {
t.Helper()
if instance == nil || instance.Endpoint == nil {
t.Fatalf("MariaDBConnFor: instance publishes no endpoint")
}
return MariaDBConn{
Host: aws.StringValue(instance.Endpoint.Address),
Port: aws.Int64Value(instance.Endpoint.Port),
User: user,
Password: password,
DBName: dbName,
}
}
// One mariadb call, including connect. Sized as psqlTimeout is: generous enough
// for a cold connection to an engine that has just finished starting, tight
// enough to fail inside a subtest.
const mariadbTimeout = 90 * time.Second
// MariaDB runs sql through the mariadb client in the client guest and returns
// its output.
//
// A failure fails the test, for the same reason PSQL's does: a green run that
// never opened a connection is the exact outcome this suite exists to prevent.
func MariaDB(t *testing.T, tgt SSHTarget, conn MariaDBConn, sql string) string {
t.Helper()
out, err := TryMariaDB(tgt, conn, sql)
if err != nil {
t.Fatalf("mariadb %s@%s:%d/%s: %v\n%s", conn.User, conn.Host, conn.Port, conn.DBName, err, out)
}
return out
}
// TryMariaDB is MariaDB without the t.Fatal, for the cases where the connection
// is supposed to fail: a retired password, a security group that does not admit
// 3306, a certificate that must not verify.
func TryMariaDB(tgt SSHTarget, conn MariaDBConn, sql string) (string, error) {
return GuestExecTimeout(tgt, mariadbCommand(conn, sql), mariadbTimeout)
}
// mariadbCommand renders the guest-side command line. The password goes through
// MYSQL_PWD rather than the client's argv, so it stays out of the guest's
// process list and out of any diagnostic that captures it.
//
// --batch makes the output tab-separated and unpadded, and in that mode the
// client aborts on the first failed statement instead of carrying on, which is
// what turns a broken statement into a non-zero exit rather than a warning.
func mariadbCommand(conn MariaDBConn, sql string) string {
port := conn.Port
if port == 0 {
port = MariaDBEnginePort
}
env := "MYSQL_PWD=" + shellQuote(conn.Password)
args := []string{
"mariadb", "--batch", "--skip-column-names", "--connect-timeout=30",
"--host", shellQuote(conn.Host),
"--port", strconv.FormatInt(port, 10),
"--user", shellQuote(conn.User),
"--database", shellQuote(conn.DBName),
}
if conn.SSLRootCert != "" {
args = append(args, "--ssl-ca="+shellQuote(conn.SSLRootCert))
}
if conn.VerifyServerCert {
args = append(args, "--ssl-verify-server-cert")
}
args = append(args, "--execute", shellQuote(sql))
return env + " " + strings.Join(args, " ")
}
+1 -1
View File
@@ -138,7 +138,7 @@ func TestConnectivity(t *testing.T) {
if _, err := harness.TryPSQL(client, byIP, "SELECT 1;"); err != nil {
return nil
}
return fmt.Errorf("%s still accepts connections through a group that does not admit %d", id, harness.DBEnginePort)
return fmt.Errorf("%s still accepts connections through a group that does not admit %d", id, harness.PostgresEnginePort)
}, sgSettleTimeout, 5*time.Second)
harness.Step(t, "Restoring the original groups on %q", id)
+2 -2
View File
@@ -233,8 +233,8 @@ func crossSubnetSSHIngress() *ec2.IpPermission {
func crossSubnetDBIngress() *ec2.IpPermission {
return &ec2.IpPermission{
IpProtocol: aws.String("tcp"),
FromPort: aws.Int64(harness.DBEnginePort),
ToPort: aws.Int64(harness.DBEnginePort),
FromPort: aws.Int64(harness.PostgresEnginePort),
ToPort: aws.Int64(harness.PostgresEnginePort),
IpRanges: []*ec2.IpRange{{CidrIp: aws.String(crossSubnetClientCIDR)}},
}
}
+18 -8
View File
@@ -34,11 +34,12 @@ import (
// plane publishes is region-qualified.
const defaultRegion = "ap-southeast-2"
// The instance spec every test in the suite creates from. The class is the
// floor and the storage is the API's own minimum: nothing here needs more, and
// each DB VM booted is charged against the phase's instance budget. Only a
// class-sensitive assertion names a bigger class, and only a grow names a
// bigger size.
// The PostgreSQL instance spec most of the suite creates from; the MariaDB
// cases carry their own engine, database name and credentials and share the
// class and storage. The class is the floor and the storage is the API's own
// minimum: nothing here needs more, and each DB VM booted is charged against the
// phase's instance budget. Only a class-sensitive assertion names a bigger
// class, and only a grow names a bigger size.
const (
dbInstancePfx = "rds-e2e"
dbEngine = "postgres"
@@ -306,18 +307,27 @@ func validCreateInput(id string) *rds.CreateDBInstanceInput {
// createDBInstance creates the suite's standard instance and returns the create
// response's own view of it — status creating, no endpoint yet.
func createDBInstance(t *testing.T, f *Fixture, id string, opts ...func(*rds.CreateDBInstanceInput)) *rds.DBInstance {
t.Helper()
return createDBInstanceFrom(t, f, validCreateInput(id), opts...)
}
// createDBInstanceFrom is createDBInstance for a request the caller assembled
// itself, which is what the MariaDB cases need: their engine, database name and
// credentials are not the suite's PostgreSQL defaults.
//
// Every test that owns an instance goes through here, because the three things a
// Every test that owns an instance bottoms out here, because the three things a
// test that boots a DB VM must not forget are registered here: the teardown, so
// a failed run does not charge the next one, the failure-only diagnostic bundle,
// without which "create timed out" names no owning phase, and the charge against
// the test's DB-VM reservation.
func createDBInstance(t *testing.T, f *Fixture, id string, opts ...func(*rds.CreateDBInstanceInput)) *rds.DBInstance {
func createDBInstanceFrom(t *testing.T, f *Fixture, in *rds.CreateDBInstanceInput,
opts ...func(*rds.CreateDBInstanceInput)) *rds.DBInstance {
t.Helper()
in := validCreateInput(id)
for _, opt := range opts {
opt(in)
}
id := aws.StringValue(in.DBInstanceIdentifier)
markDBCreateStarted(id)
out, err := f.AWS.RDS.CreateDBInstance(in) //nolint:staticcheck // e2e:allow-create — the instance under test
require.NoError(t, err, "create-db-instance %s", id)
+53
View File
@@ -0,0 +1,53 @@
//go:build e2e
package rds
import (
"testing"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// What the control plane resolves a MariaDB create against. The AMI name is not
// parsed for the engine — these tags are the whole contract.
const (
mariaAMIName = "spinifex-rds-mariadb"
mariaEngine = "mariadb"
mariaVersion = "11.8"
)
// TestMariaDBSystemImage asserts the CI job's build-and-import step produced an
// image the engine can actually be resolved from. It creates nothing, so it
// takes no DB-VM budget, and it is the cheapest possible failure for a missing
// or mis-tagged AMI: without it the first MariaDB create fails deep in launch
// with an error that names no cause.
func TestMariaDBSystemImage(t *testing.T) {
t.Parallel()
f := requireRDSFixture(t)
out, err := f.AWS.EC2.DescribeImages(&ec2.DescribeImagesInput{
Filters: []*ec2.Filter{{
Name: aws.String("name"),
Values: []*string{aws.String(mariaAMIName)},
}},
})
require.NoError(t, err, "DescribeImages")
require.Len(t, out.Images, 1, "expected exactly one %s image in the store", mariaAMIName)
image := out.Images[0]
assert.Equal(t, "available", aws.StringValue(image.State))
tags := make(map[string]string, len(image.Tags))
for _, tag := range image.Tags {
tags[aws.StringValue(tag.Key)] = aws.StringValue(tag.Value)
}
assert.Equal(t, "rds", tags["spinifex:managed-by"])
assert.Equal(t, mariaEngine, tags["engine"])
assert.Equal(t, mariaVersion, tags["engine-version"])
// Without this the control plane withholds the data-volume format grant and
// every create wedges waiting for a datadir the guest may not format.
assert.Equal(t, "format-auth-v1", tags["rds-data-volume-contract"])
}
+621
View File
@@ -0,0 +1,621 @@
//go:build e2e
package rds
import (
"fmt"
"strconv"
"strings"
"testing"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/ec2"
"github.com/aws/aws-sdk-go/service/rds"
"github.com/mulgadc/spinifex/tests/e2e/harness"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
const (
// The family a MariaDB parameter group belongs to, which is the engine and
// its pinned series exactly as AWS RDS names it.
mariaFamily = mariaEngine + mariaVersion
mariaDefaultGroup = "default." + mariaFamily
mariaInstancePfx = dbInstancePfx + "-maria"
mariaDBName = "orders"
mariaMasterUser = "appuser"
mariaMasterPassword = "e2eM4ria'DB\\Secret1"
// Both passwords carry the two characters ValidateMasterUserPassword permits
// and no MariaDB client can quote on our behalf, so rds-init's escaping and
// the agent's rotation are under test rather than assumed.
mariaRotatedPassword = "e2eR0tated\\Maria'2"
// The table every MariaDB assertion is made against: a stop, a start or a VM
// replacement that loses it has not preserved the datadir.
mariaTable = "e2e_mariadb"
mariaNote = "hello from the client VM"
// A dynamic parameter, adopted with SET GLOBAL, so the assertion does not
// have to wait out a restart. Neither this nor the static one below is a
// setting mysqld rewrites at startup, which would otherwise read as a value
// the customer set never reaching the engine.
mariaDynamicParameter = "wait_timeout"
mariaDynamicValue = "3600"
// A static one — the engine adopts it only on a restart, which is what makes
// it the parameter that can be pending-reboot.
mariaStaticParameter = "innodb_read_io_threads"
mariaStaticValue = "8"
)
// Versions that exist upstream but are not what the image runs: two other
// series, and a minor of the pinned one. Each has to be refused rather than
// served by an image running something else.
var mariaUnpinnedVersions = []string{"11.4", "10.11", "11.8.8"}
// The MariaDB create request, valid as it stands so a caller mutates only the
// field it cares about. Same class and storage as the PostgreSQL spec: the
// engine is what is under test, not the sizing.
func validMariaDBCreateInput(id string) *rds.CreateDBInstanceInput {
return &rds.CreateDBInstanceInput{
DBInstanceIdentifier: aws.String(id),
Engine: aws.String(mariaEngine),
DBInstanceClass: aws.String(dbClass),
AllocatedStorage: aws.Int64(dbStorageGiB),
DBName: aws.String(mariaDBName),
MasterUsername: aws.String(mariaMasterUser),
MasterUserPassword: aws.String(mariaMasterPassword),
}
}
// TestMariaDBInstance is the second engine's own leg of the suite: one instance
// carried from create through a client connection, a password rotation, an
// attached parameter group, a stop/start and a VM replacement.
//
// It is one test rather than several because every assertion in it is about the
// same database, and a DB VM is the suite's scarcest resource. The instance is
// consumed: it ends on a larger class with a rotated password and both
// overrides in force.
func TestMariaDBInstance(t *testing.T) {
f := requireRDSFixture(t)
t.Parallel()
// One instance throughout — the class change replaces the VM behind it rather
// than adding a second — reserved at the class it ends on, since that is the
// peak the replacement has to find room for.
reserveDBVMs(t, grownClass)
suffix := time.Now().Unix()
id := fmt.Sprintf("%s-%d", mariaInstancePfx, suffix)
paramGroup := fmt.Sprintf("%s-params-%d", mariaInstancePfx, suffix)
// Created before the instance so its teardown is registered first and LIFO
// runs it last: a group delete issued while the instance still holds it is
// refused.
createMariaDBParameterGroup(t, f, paramGroup)
harness.Phase(t, "Creating MariaDB instance %q", id)
created := createDBInstanceFrom(t, f, validMariaDBCreateInput(id), func(in *rds.CreateDBInstanceInput) {
in.DBParameterGroupName = aws.String(paramGroup)
})
assert.Equal(t, harness.DBInstanceCreating, aws.StringValue(created.DBInstanceStatus))
client := rdsClient(t, f)
system := f.SystemAWS(t)
instance := waitForAvailable(t, f, id)
// Reassigned by the rotation subtest, so every subtest after it connects with
// the credential the platform last installed rather than the one that created
// the instance.
conn := harness.MariaDBConnFor(t, instance, mariaMasterUser, mariaMasterPassword, mariaDBName)
endpoint := conn.Host
// Captured while the instance is up: the VM carries no per-instance tag, so
// after a replacement it can only be recognised by an ID held from before.
vmID := aws.StringValue(harness.DBInstanceVM(t, system, id).InstanceId)
dataVolumeID := aws.StringValue(harness.DBInstanceDataVolume(t, system, id).VolumeId)
privateIP := aws.StringValue(harness.DBEndpointENI(t, f.AWS, id).PrivateIpAddress)
t.Run("DescribesAsMariaDBOnItsOwnPort", func(t *testing.T) {
assert.Equal(t, mariaEngine, aws.StringValue(instance.Engine),
"the engine identifier is mariadb; mysql is not offered as an alias for it")
assert.Equal(t, mariaVersion, aws.StringValue(instance.EngineVersion))
assert.Equal(t, dbClass, aws.StringValue(instance.DBInstanceClass))
assert.Equal(t, int64(dbStorageGiB), aws.Int64Value(instance.AllocatedStorage))
assert.Equal(t, mariaMasterUser, aws.StringValue(instance.MasterUsername))
assert.True(t, aws.BoolValue(instance.StorageEncrypted))
assert.False(t, aws.BoolValue(instance.MultiAZ))
assert.False(t, aws.BoolValue(instance.PubliclyAccessible))
assert.Equal(t, paramGroup, dbParameterGroupName(instance))
require.NotNil(t, instance.Endpoint, "an available instance must publish an endpoint")
assert.Equal(t, int64(harness.MariaDBEnginePort), aws.Int64Value(instance.Endpoint.Port),
"MariaDB's default port is its own, not the other engine's")
})
// The first assertion that proves an engine is actually serving: every status
// before this one is the control plane's own account of itself.
t.Run("AClientInTheVPCRoundTripsARow", func(t *testing.T) {
harness.MariaDB(t, client, conn, fmt.Sprintf(
"CREATE TABLE %s (id int primary key, note text); INSERT INTO %s VALUES (1, '%s');",
mariaTable, mariaTable, mariaNote))
assert.Equal(t, mariaNote, mariaValue(t, client, conn,
fmt.Sprintf("SELECT note FROM %s WHERE id = 1;", mariaTable)),
"the row written over the endpoint must read back")
// default_storage_engine is platform-owned and pinned to InnoDB because
// the snapshot guarantee rests on it: a table created without an explicit
// ENGINE= clause has to land somewhere with a redo log.
assert.Equal(t, "InnoDB", mariaValue(t, client, conn, fmt.Sprintf(
"SELECT engine FROM information_schema.tables WHERE table_schema = '%s' AND table_name = '%s';",
mariaDBName, mariaTable)),
"a table created with no ENGINE clause must be InnoDB")
})
// Stock MariaDB cannot exclude mysql.* from a global object grant. The
// protected routine is the only path to privileges on another database.
t.Run("MasterPrivilegesCannotReachSystemAccounts", func(t *testing.T) {
for name, sql := range map[string]string{
"ReadGrantTable": "SELECT Priv FROM mysql.global_priv LIMIT 1;",
"WriteGrantTable": "UPDATE mysql.global_priv SET Priv = Priv WHERE User = 'root';",
"AlterRoot": "ALTER USER 'root'@'localhost' IDENTIFIED BY 'not-allowed';",
"DropRoutine": "DROP PROCEDURE `_spinifex_rds`.`create_database`;",
"PlainCreate": "CREATE DATABASE forbidden_plain_database;",
} {
t.Run(name, func(t *testing.T) {
out, err := harness.TryMariaDB(client, conn, sql)
require.Error(t, err, "master statement unexpectedly succeeded: %s", out)
assert.Contains(t, strings.ToLower(out), "denied")
})
}
harness.MariaDB(t, client, conn, "FLUSH PRIVILEGES;")
futureDB := "e2e_future_database"
harness.MariaDB(t, client, conn,
fmt.Sprintf("CALL `_spinifex_rds`.`create_database`('%s');", futureDB))
future := conn
future.DBName = futureDB
harness.MariaDB(t, client, future,
"CREATE TABLE future_table (id int primary key); INSERT INTO future_table VALUES (1);")
assert.Equal(t, "1", strings.TrimSpace(harness.MariaDB(t, client, future,
"SELECT id FROM future_table;")))
out, err := harness.TryMariaDB(client, conn,
"CALL `_spinifex_rds`.`create_database`('mysql');")
require.Error(t, err, "system database creation unexpectedly succeeded: %s", out)
assert.Contains(t, out, "non-system identifier")
})
// TLS is offered rather than enforced, so what is asserted is that a client
// asking for it gets it.
t.Run("AClientCanConnectOverTLS", func(t *testing.T) {
encrypted := conn
encrypted.SSLRootCert = harness.RDSClientCACertPath
assert.NotEmpty(t, mariaSessionCipher(t, client, encrypted),
"a connection made against the cluster CA must report a negotiated cipher")
// Verified by name only, in its own subtest because a cluster with no base
// domain has no name to verify against. The endpoint name and the ENI
// address are both in the certificate's SAN set, but the client's hostname
// check is written against the certificate's DNS names, so pinning the
// address here would assert something about the client's IP-SAN handling
// rather than about ours.
t.Run("AndTheCertificateVerifiesByName", func(t *testing.T) {
requireEndpointName(t, endpoint)
verified := encrypted
verified.VerifyServerCert = true
assert.NotEmpty(t, mariaSessionCipher(t, client, verified),
"the serving certificate must verify against the cluster CA under the endpoint name")
})
})
// The system keeps no cleartext password anywhere, so the only proof a
// rotation reached the engine is that the new credential authenticates and
// the old one does not.
t.Run("TheMasterPasswordRotates", func(t *testing.T) {
_, err := f.AWS.RDS.ModifyDBInstance(&rds.ModifyDBInstanceInput{
DBInstanceIdentifier: aws.String(id),
MasterUserPassword: aws.String(mariaRotatedPassword),
ApplyImmediately: aws.Bool(true),
})
require.NoError(t, err, "modify-db-instance master password")
rotated := conn
rotated.Password = mariaRotatedPassword
assert.Equal(t, mariaNote, mariaValue(t, client, rotated,
fmt.Sprintf("SELECT note FROM %s WHERE id = 1;", mariaTable)),
"the rotated password must reach the same database")
out, err := harness.TryMariaDB(client, conn, "SELECT 1;")
require.Error(t, err, "the retired password must not connect: %s", out)
assert.Contains(t, out, "Access denied",
"the old password must be refused by the engine, not by the network")
conn = rotated
})
// The family check is one funnel every binding path goes through, and modify
// is the one a customer reaches by hand. Non-disruptive: it is refused before
// anything is written, so the instance is left where it was found.
t.Run("APostgreSQLParameterGroupCannotBeAttached", func(t *testing.T) {
harness.ExpectError(t, "InvalidParameterCombination", func() error {
_, err := f.AWS.RDS.ModifyDBInstance(&rds.ModifyDBInstanceInput{
DBInstanceIdentifier: aws.String(id),
DBParameterGroupName: aws.String("default." + dbParameterGroupFamily),
ApplyImmediately: aws.Bool(true),
})
return err
})
current, err := harness.DescribeDBInstance(f.AWS, id)
require.NoError(t, err)
assert.Equal(t, paramGroup, dbParameterGroupName(current),
"a refused attachment must leave the instance on the group it already had")
})
// The only assertions that prove a resolved parameter reached the engine
// rather than merely being stored, one per apply method.
t.Run("AttachedGroupEditsReachTheRunningEngine", func(t *testing.T) {
require.NotEqual(t, mariaDynamicValue, mariaGlobal(t, client, conn, mariaDynamicParameter),
"the chosen dynamic value must differ from the catalog default")
setMariaDBParameter(t, f, paramGroup, mariaDynamicParameter, mariaDynamicValue, "immediate")
assert.Equal(t, mariaDynamicValue, mariaGlobal(t, client, conn, mariaDynamicParameter),
"an immediate edit to an attached group must reload without a reboot")
before := mariaGlobal(t, client, conn, mariaStaticParameter)
require.NotEqual(t, mariaStaticValue, before,
"the chosen static value must differ from the catalog default")
setMariaDBParameter(t, f, paramGroup, mariaStaticParameter, mariaStaticValue, "pending-reboot")
pending, err := harness.DescribeDBInstance(f.AWS, id)
require.NoError(t, err, "describe the pending static parameter")
require.NotEmpty(t, pending.DBParameterGroups)
assert.Equal(t, "pending-reboot", aws.StringValue(pending.DBParameterGroups[0].ParameterApplyStatus))
assert.Equal(t, before, mariaGlobal(t, client, conn, mariaStaticParameter),
"a static edit must not take effect before a reboot")
_, err = f.AWS.RDS.RebootDBInstance(&rds.RebootDBInstanceInput{DBInstanceIdentifier: aws.String(id)})
require.NoError(t, err, "reboot-db-instance for the static parameter")
instance = waitForAvailable(t, f, id)
require.NotEmpty(t, instance.DBParameterGroups)
assert.Equal(t, "in-sync", aws.StringValue(instance.DBParameterGroups[0].ParameterApplyStatus),
"the reboot applied the pending parameter, so nothing is outstanding")
assertMariaDBOverridesInForce(t, client, conn)
})
// A stop keeps the data volume, the customer ENI and its address, so a start
// comes back on the same datadir at the same endpoint. The VM is stopped
// rather than terminated — only a class change replaces it.
t.Run("StopAndStartKeepTheDataTheAddressAndTheOverrides", func(t *testing.T) {
_, err := f.AWS.RDS.StopDBInstance(&rds.StopDBInstanceInput{DBInstanceIdentifier: aws.String(id)})
require.NoError(t, err, "stop-db-instance")
stopped := harness.WaitForDBInstanceStatus(t, f.AWS, id, harness.DBInstanceStopped)
harness.WaitForInstanceState(t, system, vmID, ec2.InstanceStateNameStopped)
assert.Equal(t, dataVolumeID, aws.StringValue(harness.DBInstanceDataVolume(t, system, id).VolumeId),
"the data volume must be retained across a stop")
require.NotNil(t, stopped.Endpoint, "a stopped instance still reports where it will be when it comes back")
assert.Equal(t, endpoint, aws.StringValue(stopped.Endpoint.Address))
_, err = f.AWS.RDS.StartDBInstance(&rds.StartDBInstanceInput{DBInstanceIdentifier: aws.String(id)})
require.NoError(t, err, "start-db-instance")
started := waitForAvailable(t, f, id)
require.NotNil(t, started.Endpoint)
assert.Equal(t, endpoint, aws.StringValue(started.Endpoint.Address), "the endpoint must survive a stop/start")
assert.Equal(t, privateIP, aws.StringValue(harness.DBEndpointENI(t, f.AWS, id).PrivateIpAddress),
"the private address is what the endpoint name resolves to, so it must persist")
assert.Equal(t, mariaNote, mariaValue(t, client, conn,
fmt.Sprintf("SELECT note FROM %s WHERE id = 1;", mariaTable)),
"the row written before the stop must still be there")
assertMariaDBOverridesInForce(t, client, conn)
})
// The class moves, which replaces the VM. The endpoint is kept through it —
// the data volume and the customer ENI are re-attached to the replacement —
// and so are the parameters the customer set, which is what putting the
// generated configuration on the data volume exists to guarantee: nothing
// issues an apply on a plain replace, so a configuration living on the boot
// volume would be discarded with it and the engine would silently revert to
// catalog defaults.
t.Run("AClassChangeReplacesTheVMAndKeepsTheOverrides", func(t *testing.T) {
poolAtFloor := mariaGlobalInt(t, client, conn, "innodb_buffer_pool_size")
harness.Phase(t, "Changing the class of %q from %s to %s", id, dbClass, grownClass)
out, err := f.AWS.RDS.ModifyDBInstance(&rds.ModifyDBInstanceInput{
DBInstanceIdentifier: aws.String(id),
DBInstanceClass: aws.String(grownClass),
ApplyImmediately: aws.Bool(true),
})
require.NoError(t, err, "modify-db-instance class")
require.NotNil(t, out.DBInstance)
assert.Equal(t, harness.DBInstanceModifying, aws.StringValue(out.DBInstance.DBInstanceStatus))
changed := waitForAvailable(t, f, id)
assert.Equal(t, grownClass, aws.StringValue(changed.DBInstanceClass))
require.NotNil(t, changed.Endpoint)
assert.Equal(t, endpoint, aws.StringValue(changed.Endpoint.Address),
"the endpoint is the customer's handle on the instance and must survive the VM being replaced")
assert.Equal(t, privateIP, aws.StringValue(harness.DBEndpointENI(t, f.AWS, id).PrivateIpAddress),
"the ENI is re-attached to the replacement, so the address the name resolves to is unchanged")
assert.Equal(t, dataVolumeID, aws.StringValue(harness.DBInstanceDataVolume(t, system, id).VolumeId),
"the data volume is re-attached rather than rebuilt")
replacementID := aws.StringValue(harness.DBInstanceVM(t, system, id).InstanceId)
assert.NotEqual(t, vmID, replacementID, "a class change is delivered by replacing the VM")
harness.AssertVMGone(t, system, vmID)
assert.Equal(t, mariaNote, mariaValue(t, client, conn,
fmt.Sprintf("SELECT note FROM %s WHERE id = 1;", mariaTable)),
"the row written before the replacement must survive it")
assertMariaDBOverridesInForce(t, client, conn)
// The size-derived defaults are re-resolved against the class the instance
// is now on, so a replacement that carried the old configuration over
// wholesale would leave a 2 GiB instance running a 1 GiB buffer pool. The
// assertion is the direction rather than a literal: the server rounds the
// pool to its own granularity, which is not ours to predict.
assert.Greater(t, mariaGlobalInt(t, client, conn, "innodb_buffer_pool_size"), poolAtFloor,
"the buffer pool default must be re-resolved against %s", grownClass)
require.NotEmpty(t, changed.DBParameterGroups)
assert.Equal(t, "in-sync", aws.StringValue(changed.DBParameterGroups[0].ParameterApplyStatus),
"the replacement booted on the re-resolved set, so there is nothing left for a reboot to apply")
})
}
// TestMariaDBSnapshotRestore proves the snapshot path is engine-neutral where it
// claims to be and engine-aware where it must be: the snapshot carries MariaDB's
// own engine, version and port, and the instance restored from it comes up on
// that datadir holding the rows as of the moment it was taken and the
// credentials the source already had.
func TestMariaDBSnapshotRestore(t *testing.T) {
f := requireRDSFixture(t)
t.Parallel()
// The source and the instance restored from its snapshot, which have to be
// alive together for the restore to be provably a point in time.
reserveDBVMs(t, dbClass, dbClass)
suffix := time.Now().Unix()
sourceID := fmt.Sprintf("%s-snapsrc-%d", mariaInstancePfx, suffix)
restoredID := fmt.Sprintf("%s-restored-%d", mariaInstancePfx, suffix)
snapshotID := fmt.Sprintf("%s-snapshot-%d", mariaInstancePfx, suffix)
harness.Phase(t, "Creating source MariaDB instance %q", sourceID)
createDBInstanceFrom(t, f, validMariaDBCreateInput(sourceID))
client := rdsClient(t, f)
source := waitForAvailable(t, f, sourceID)
sourceConn := harness.MariaDBConnFor(t, source, mariaMasterUser, mariaMasterPassword, mariaDBName)
harness.MariaDB(t, client, sourceConn, fmt.Sprintf(
"CREATE TABLE %s (id int primary key, note text); INSERT INTO %s VALUES (1, '%s');",
snapshotTable, snapshotTable, rowBeforeSnapshot))
harness.Phase(t, "Snapshotting %q as %q", sourceID, snapshotID)
snapshot := createDBSnapshot(t, f, sourceID, snapshotID)
// The snapshot has to carry enough of the instance to rebuild it, because the
// instance may be gone by the time it is used — and a snapshot that recorded
// the wrong engine would be restorable onto an image that cannot read it.
assert.Equal(t, harness.DBSnapshotAvailable, aws.StringValue(snapshot.Status))
assert.Equal(t, mariaEngine, aws.StringValue(snapshot.Engine))
assert.Equal(t, mariaVersion, aws.StringValue(snapshot.EngineVersion))
assert.Equal(t, int64(harness.MariaDBEnginePort), aws.Int64Value(snapshot.Port))
assert.Equal(t, mariaMasterUser, aws.StringValue(snapshot.MasterUsername))
// Written after the snapshot and therefore the row that must be missing from
// the restore. Its absence is what separates a point-in-time copy from a
// second reference to the live volume.
harness.MariaDB(t, client, sourceConn, fmt.Sprintf(
"INSERT INTO %s VALUES (2, '%s');", snapshotTable, rowAfterSnapshot))
// The engine was available, so the quiesce had every opportunity to run and
// the crash-consistent fallback — whose MariaDB wording warns that
// non-transactional tables may not recover — must not have been reached.
t.Run("TheSnapshotWasTakenOverAQuiescedEngine", func(t *testing.T) {
events := dbInstanceEventMessages(t, f, sourceID)
assert.NotContains(t, strings.Join(events, "\n"), crashConsistentNotice,
"BACKUP STAGE must have held the engine; the crash-consistent fallback is for when it cannot")
})
t.Run("ARestoreHoldsTheDataAsOfTheSnapshot", func(t *testing.T) {
harness.Phase(t, "Restoring %q from %q", restoredID, snapshotID)
restored := restoreFromSnapshot(t, f, restoredID, snapshotID)
assert.Equal(t, harness.DBInstanceCreating, aws.StringValue(restored.DBInstanceStatus))
instance := waitForAvailable(t, f, restoredID)
assert.Equal(t, mariaEngine, aws.StringValue(instance.Engine))
assert.Equal(t, mariaMasterUser, aws.StringValue(instance.MasterUsername))
require.NotNil(t, instance.Endpoint)
assert.NotEqual(t, sourceConn.Host, aws.StringValue(instance.Endpoint.Address),
"the restore is its own instance and must publish its own endpoint")
// No password was supplied to the restore and none could be: the account
// and its hash came out of the datadir, so the source's credentials are
// the restore's.
conn := harness.MariaDBConnFor(t, instance, mariaMasterUser, mariaMasterPassword, mariaDBName)
assert.Equal(t, strconv.Itoa(snapshotRowsBefore),
mariaValue(t, client, conn, fmt.Sprintf("SELECT count(*) FROM %s;", snapshotTable)),
"the restore must hold the rows the snapshot captured and nothing written after it")
assert.Equal(t, rowBeforeSnapshot, mariaValue(t, client, conn,
fmt.Sprintf("SELECT note FROM %s WHERE id = 1;", snapshotTable)))
assert.Equal(t, "2", mariaValue(t, client, sourceConn,
fmt.Sprintf("SELECT count(*) FROM %s;", snapshotTable)),
"the source is untouched by the restore reading from its snapshot")
})
}
// TestMariaDBValidation is the second engine's negative matrix: the versions and
// the cross-engine parameter groups that must be refused, and the implicit
// default group registering MariaDB brings with it. It boots nothing, so it
// fails in seconds rather than after a VM.
func TestMariaDBValidation(t *testing.T) {
f := requireRDSFixture(t)
t.Parallel()
// Every create below is asserted to be refused, so in a passing run nothing
// boots. The reservation covers the run where one is wrongly accepted:
// expectCreateRefused deletes it, but the VM is charged to the suite until it
// does.
reserveDBVMs(t, dbClass)
suffix := time.Now().Unix()
rejectedID := fmt.Sprintf("%s-rejected-%d", mariaInstancePfx, suffix)
mariaGroup := fmt.Sprintf("%s-family-%d", mariaInstancePfx, suffix)
createMariaDBParameterGroup(t, f, mariaGroup)
// Implicit rather than stored: registering the engine is what makes it
// describable, and it must be so without ever having been created.
t.Run("TheDefaultMariaDBGroupIsImplicitAndCarriesTheCatalog", func(t *testing.T) {
out, err := f.AWS.RDS.DescribeDBParameterGroups(&rds.DescribeDBParameterGroupsInput{
DBParameterGroupName: aws.String(mariaDefaultGroup),
})
require.NoError(t, err, "describe-db-parameter-groups")
require.Len(t, out.DBParameterGroups, 1)
assert.Equal(t, mariaFamily, aws.StringValue(out.DBParameterGroups[0].DBParameterGroupFamily))
params := describeAllParameters(t, f, mariaDefaultGroup)
pool, ok := params["innodb_buffer_pool_size"]
require.True(t, ok, "the MariaDB catalog must be reachable through its own default group")
assert.Equal(t, "engine-default", aws.StringValue(pool.Source))
assert.Equal(t, "static", aws.StringValue(pool.ApplyType))
assert.NotContains(t, aws.StringValue(pool.ParameterValue), "{",
"a size-derived default must reach the customer as a literal")
assert.NotContains(t, params, "shared_buffers",
"the MariaDB catalog must not carry the other engine's parameters")
})
t.Run("AnUnpinnedEngineVersionIsRefused", func(t *testing.T) {
for _, version := range mariaUnpinnedVersions {
t.Run(version, func(t *testing.T) {
in := validMariaDBCreateInput(rejectedID)
in.EngineVersion = aws.String(version)
expectCreateRefused(t, f, f.AWS, "InvalidParameterValue", in)
})
}
})
// Aliasing mysql onto the MariaDB image would report an engine and a version
// the instance is not running, and the lie would propagate into the describe,
// the parameter-group family and every snapshot taken of it.
t.Run("MySQLIsNotAnAliasForMariaDB", func(t *testing.T) {
in := validMariaDBCreateInput(rejectedID)
in.Engine = aws.String(unsupportedDBEngine)
expectCreateRefused(t, f, f.AWS, "InvalidParameterValue", in)
})
// Both directions, because the check is one funnel and a group carrying the
// wrong engine's parameters would only fail once the engine refused to start.
t.Run("AParameterGroupOfTheOtherEnginesFamilyIsRefusedAtCreate", func(t *testing.T) {
mariaOnPostgres := validCreateInput(rejectedID)
mariaOnPostgres.DBParameterGroupName = aws.String(mariaGroup)
expectCreateRefused(t, f, f.AWS, "InvalidParameterCombination", mariaOnPostgres)
postgresOnMaria := validMariaDBCreateInput(rejectedID)
postgresOnMaria.DBParameterGroupName = aws.String("default." + dbParameterGroupFamily)
expectCreateRefused(t, f, f.AWS, "InvalidParameterCombination", postgresOnMaria)
})
// The other engine's parameter names must not be storable into a MariaDB
// group either: caught here, the customer learns of it on the group they got
// wrong rather than on the instance that later refuses to start.
t.Run("TheOtherEnginesParametersCannotBeStored", func(t *testing.T) {
harness.ExpectError(t, "InvalidParameterValue", func() error {
_, err := f.AWS.RDS.ModifyDBParameterGroup(&rds.ModifyDBParameterGroupInput{
DBParameterGroupName: aws.String(mariaGroup),
Parameters: []*rds.Parameter{{
ParameterName: aws.String("shared_buffers"),
ParameterValue: aws.String("32768"),
}},
})
return err
})
})
}
// An empty MariaDB parameter group and its teardown. Empty because what each
// test then stores into it is the assertion.
func createMariaDBParameterGroup(t *testing.T, f *Fixture, name string) {
t.Helper()
_, err := f.AWS.RDS.CreateDBParameterGroup(&rds.CreateDBParameterGroupInput{
DBParameterGroupName: aws.String(name),
DBParameterGroupFamily: aws.String(mariaFamily),
Description: aws.String("rds e2e mariadb parameter group"),
})
require.NoError(t, err, "create-db-parameter-group %s", name)
t.Cleanup(func() {
if _, err := f.AWS.RDS.DeleteDBParameterGroup(&rds.DeleteDBParameterGroupInput{
DBParameterGroupName: aws.String(name),
}); err != nil && !harness.ErrorCodeIs(err, "DBParameterGroupNotFound") {
t.Logf("cleanup: delete-db-parameter-group %s: %v", name, err)
}
})
}
func setMariaDBParameter(t *testing.T, f *Fixture, group, name, value, applyMethod string) {
t.Helper()
_, err := f.AWS.RDS.ModifyDBParameterGroup(&rds.ModifyDBParameterGroupInput{
DBParameterGroupName: aws.String(group),
Parameters: []*rds.Parameter{{
ParameterName: aws.String(name),
ParameterValue: aws.String(value),
ApplyMethod: aws.String(applyMethod),
}},
})
require.NoError(t, err, "modify-db-parameter-group %s: %s=%s", group, name, value)
}
// Both overrides as the running engine reports them. Asserted after every
// transition, because the failure this catches is silent: an instance that came
// back on catalog defaults is healthy, available and running a configuration
// its owner did not choose.
func assertMariaDBOverridesInForce(t *testing.T, tgt harness.SSHTarget, conn harness.MariaDBConn) {
t.Helper()
assert.Equal(t, mariaDynamicValue, mariaGlobal(t, tgt, conn, mariaDynamicParameter),
"the dynamic override must still be in force")
assert.Equal(t, mariaStaticValue, mariaGlobal(t, tgt, conn, mariaStaticParameter),
"the static override must still be in force")
}
// One global system variable as the running server reports it. GLOBAL rather
// than the session copy, since that is the scope a parameter group sets.
func mariaGlobal(t *testing.T, tgt harness.SSHTarget, conn harness.MariaDBConn, name string) string {
t.Helper()
return mariaValue(t, tgt, conn, "SELECT @@GLOBAL."+name+";")
}
func mariaGlobalInt(t *testing.T, tgt harness.SSHTarget, conn harness.MariaDBConn, name string) int64 {
t.Helper()
raw := mariaGlobal(t, tgt, conn, name)
value, err := strconv.ParseInt(raw, 10, 64)
if err != nil {
t.Fatalf("@@GLOBAL.%s reported %q, which is not an integer", name, raw)
}
return value
}
// The cipher the connection negotiated, empty when it is in the clear. SHOW
// rather than performance_schema, which the catalog leaves off by default.
func mariaSessionCipher(t *testing.T, tgt harness.SSHTarget, conn harness.MariaDBConn) string {
t.Helper()
_, cipher, _ := strings.Cut(mariaValue(t, tgt, conn, "SHOW SESSION STATUS LIKE 'Ssl_cipher';"), "\t")
return strings.TrimSpace(cipher)
}
// One scalar as the client printed it. The guest exec folds stderr into the same
// stream, so a client-side notice would otherwise be compared against the value:
// the answer is the last line the command produced.
func mariaValue(t *testing.T, tgt harness.SSHTarget, conn harness.MariaDBConn, sql string) string {
t.Helper()
out := harness.MariaDB(t, tgt, conn, sql)
lines := strings.Split(strings.TrimSpace(out), "\n")
return strings.TrimSpace(lines[len(lines)-1])
}
+1 -1
View File
@@ -88,7 +88,7 @@ func TestSnapshotRestore(t *testing.T) {
assert.Equal(t, dbEngine, aws.StringValue(snapshot.Engine))
assert.Equal(t, int64(dbStorageGiB), aws.Int64Value(snapshot.AllocatedStorage))
assert.Equal(t, dbMasterUser, aws.StringValue(snapshot.MasterUsername))
assert.Equal(t, int64(harness.DBEnginePort), aws.Int64Value(snapshot.Port))
assert.Equal(t, int64(harness.PostgresEnginePort), aws.Int64Value(snapshot.Port))
assert.True(t, aws.BoolValue(snapshot.Encrypted), "the snapshot inherits the data volume's encryption")
assert.Equal(t, int64(100), aws.Int64Value(snapshot.PercentProgress),
"a snapshot reported as available must not also report itself part-done")