mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-15 09:01:31 +00:00
Ben Duncan
18853cef31
* fix: stop the e2e analyser inventing failures and let a detach outlast its seal go-junit-report emits <error message="No test result found"> for a === RUN with no matching result line, and attaches the test's own captured log as the body. The analyser counted those as failures, so the baremetal cell reported 25 failures against a junit file recording 1, with log output reading as assertions. Separate the marker from a real error in both the parse and the roll-up, count the unresolved cases and say so in the summary. The gateway gave a detach 30s while the daemon budgets 180s for the ebs.unmount seal and 180s more for the QMP unplug chain, so any detach whose seal replayed a WAL was reported as ServerInternal while it went on to succeed. Also switch four nightly job gates from always() to !cancelled(): always() overrides cancellation, so a cancelled run kept starting jobs. * fix: serialise the e2e workflows on one group and retry a dropped libvirt e2e.yml had no concurrency block at all, so every push to dev could provision alongside every other push and alongside the nightly. All four workflows that build on these hypervisors now share e2e-hypervisors, queued rather than cancelled. A per-environment group is not expressible: GitHub picks the group before it picks a runner, and the environment is a property of the runner. tofu apply also failed outright on a transient libvirt connect, which fails a 45-minute job in 90 seconds and reports it as a test failure. retry_run retries only when that attempt's own output names a connection failure, so a real failure still fails once. Also collect the unfiltered journal from the matrix cells and the node journal from cell-21: the unit filter drops the kernel and OOM lines that explain why one node behaved unlike its siblings, and a passing run's journal is what makes a failing one readable. * fix: build the baremetal junit from the unfiltered suite log run-baremetal-e2e.sh streams the full suite output to suite-output.log and a grep-filtered summary to its stdout, which the workflow tees into test-baremetal.log. That filter anchors --- PASS at column 0 and Go indents every subtest result by four spaces, so the junit was generated from a log with no subtest results in it: go-junit-report saw each === RUN with no outcome and emitted <error message="No test result found">. 17 of cell-28's 39 testcases and 93 of cell-20's 120 were that. Regenerating cell-28 from suite-output.log gives 39 tests, 0 failures, 0 errors. * fix: converge released EIPs instead of trusting a fire-and-forget teardown A disassociate leaves the ENI running, so the orphan sweep's owning-port test never fires for it. The teardown then rests entirely on vpc.delete-nat, which is core NATS and is lost whenever vpcd is restarting — and the intent KV watch is UpdatesOnly, so the same window hides the write that would have driven a repair pass. The dnat_and_snat row and its host /32 route to a released public address survived until something else happened to touch them, which for a live ENI was nothing at all. Give the sweep the second test it was missing: AddEIP is the only writer of dnat_and_snat, so the floating IPs intent asks for are the complete set of addresses that may hold one, and a row for any other address is stale however alive its port. Ports and external IPs travel together in LiveEIPs, both unioned from the start-of-pass snapshot and the fresh re-read, so a mid-pass launch is still spared. The host unbind now runs even when the row had already gone, which is the state a crash between the OVN delete and the unbind leaves. Run that sweep on the startup pass too. Startup is the pass that follows the blind window, so skipping it deferred repair of exactly the changes that were missed. The port-group and ENI-port sweeps stay gated there for the in-flight create they were written for; the EIP sweep re-reads intent immediately before deciding and abandons the whole sweep if that read fails, which is the same protection by a stronger route. * fix: keep a release verify off the hypervisors an e2e run is using release.yml is the fifth workflow that provisions on the ci-single pool and had no concurrency block at all, so a tag push could land a tofu apply on the same libvirt host a nightly cell was mid-provision on. The group goes on the verify job rather than the workflow: the build jobs touch no hypervisor and should not queue behind a nightly. * fix: stop cell-27 handing a 50-minute suite a 50-minute job The rds suite is budgeted 50m in suite-sets.sh and cell-27 took the matrix default of 50 minutes for the whole job, so provisioning, install, the suite and teardown had to fit inside the suite's allowance alone. It never could: the last measured run spent 6 minutes reaching the suite and was killed 44 minutes into it, mid-TestAutomatedBackups. A job timeout reports "The operation was canceled" and nothing else, which is why this cell has no verdict in any nightly rather than a failure anyone could read. cell-27 gets 75 minutes, and the select job now refuses to start a run whose job budget does not clear its suite budget plus a setup allowance. The inner timeout firing first is the property that matters: go test says where it stopped and what failed, and the job timeout says nothing at all. cell-29 already had the same fix applied by hand; nothing stopped the next suite from repeating it. * fix: converge host EIP plumbing against intent, not against the NAT row The released-EIP sweep landed and did not fire. Its premise was that a stranded dnat_and_snat row is what a stranded host route can be found by, and on the path that actually strands one there is no row left. AddEIP scrubs the predecessor row sharing the private IP before installing the new one, so when reconcile re-added the instance's auto-assigned address it silently deleted the released EIP's row — and applyEIPs runs ahead of the sweep, which then walked a table with nothing stale in it while the /32 route and its proxy-ARP stayed up. Give the sweep the host's own inventory. ListEIPIngress reads the EIPs back from the FORWARD accepts EnsureEIPIngress stamps with its own comment, so a binding is found by what installed it rather than by a row some other path is free to delete. Anything plumbed that intent no longer asks for is torn down, which also covers a crash between the OVN delete and the unbind. A failed inventory fails the pass: an empty list and an unreadable one are indistinguishable, and reading the second as the first is a sweep that reports success having done nothing. Close the window inside the pass too. The predecessor scrub now reads the row it is about to drop and releases that address's host state, so the strand does not exist between applyEIPs and the sweep rather than merely being repaired a step later. * fix: retry the release verify's libvirt calls like every other job does The verify job drives the same libvirt provider on the same hypervisors as the e2e cells, where a dropped connection under concurrent load is the steady state and is retried. Here it was a hard failure that reads as a broken release. It was skipped earlier only because the job checks out mulga and not spinifex, so the shared helpers were not on disk — a sparse checkout of .github/scripts costs nothing and makes apply and destroy behave like their e2e counterparts. * fix: do not report a rebooting DB instance available on the pre-reboot agent's health RegisterDBInstance stamped a fresh LastSeen while leaving Agent.EngineHealth at the value the previous agent process reported. After a reboot the new agent registers before it has probed anything, so the record read as a healthy engine with a beat newer than the transition, and the reconciler moved the instance to available while PostgreSQL was still starting up. A client that connects the moment the status turns green then gets "the database system is starting up". A register now discards the previous process's verdict and records when this agent process started. The restart reconciler requires that registration to be newer than the transition, which also closes the window where a beat lands between the engine being stopped and the VM going down. * test: record the credential kubectl sent when an EKS call comes back Unauthorized The IRSA pod subtest's first kubectl is the gateway-CA configmap, which lands before the pod dumps are registered, so a 401 there left no evidence at all. A verbose read-only probe reports whether an Authorization header was sent, which separates a token the apiserver rejected from one the exec plugin never made. * fix: report a starting engine promptly, and budget CI jobs from the suite The agent waited a full heartbeat interval before its first beat, so a restart could not be observed as finished for up to 30 seconds after the engine was already serving. Beat at startup instead, and report every 5 seconds until the engine has answered once — the control plane is actively waiting on that beat, and the steady-state cadence it hands back is for a database already serving. The rds suite is the one most sensitive to what else the hypervisor is doing: its DB VMs serialise behind a 4 GiB semaphore, so its wall clock is set by how long it waits for the budget. The same pass ran 45m beside four nightly cells and overran 50m beside nine, panicking with two tests still running. Raise its budget to 60m. That budget was then larger than the jobs meant to contain it. A job cancelled on timeout reports only "The operation was canceled" and takes the artifacts with it, so the inner `go test` timeout must always fire first — and it could not for rds on e2e.yml, nor for storagefault or diskperf, which have asked for 90m and 180m inside a 60m job since they were added. Both workflows now resolve their job budget from suite-sets.sh rather than carrying a literal: e2e.yml from the largest requested suite, e2e-suites.yml per matrix leg, so a leg is bounded by its own suite and not by the longest one beside it. The default push set resolves to exactly the timeouts both workflows had before. * fix: check a permutation cell's job budget against the suites it runs The guard only saw cells that name a suite, so the nine that run a topology's nightly subset were never checked against it — the same class of mistake the guard exists to catch, in the majority of the cells. Bound each by the longest suite in its subset: max() and not sum(), because whichever suite hangs must panic on its own timeout first, and a sum would demand a budget the set has never come close to using. Every cell clears it today. * fix: report why a PostgreSQL engine would not start, and stop failing a stop that landed Three defects, all found by the first rds pass that ever ran deep enough to reach them. A wait that saw an instance reach `failed` called t.Fatalf from inside the polling condition, and testify runs that condition on its own goroutine. The Goexit killed the tick and left the collector empty, which reads as the condition being satisfied — so the wait returned successfully on an instance it had just declared broken, logged "reached status available" about one that was failed, and let the test carry on against it. Six conditions across four suites call Fatal or require the same way. EventuallyErr now runs its condition on the test's goroutine, which makes all six behave as written. A DB VM stop has two drivers: the API call and the reconciler resuming one whose caller it assumes has died. When they overlap the loser is told the VM is not in a state to be stopped — which is the state the stop was heading for — and the customer's StopDBInstance returned InvalidDBInstanceState for a stop that had just completed. Both paths now treat a refused command the way they already treat an unanswered one: it says nothing about the VM, so the fleet view settles it. PostgreSQL had no equivalent of the engine log tail MariaDB has carried since it was added, so a postmaster that refuses a setting produced "engine did not respond on /run/postgresql:5432" and nothing else — on a guest with no SSH, whose console records only that pg_ctl could not start the server. The probe now quotes the tail of the log the image already directs the postmaster to, and the reboot and modify paths carry the agent's reason into the failure the way the create path always has. The reason reaches the customer through StatusInfos. The rds suite's budget goes to 90m and cell-27's job to 100m. Its wall clock is what the DB-VM queue costs rather than what the tests do — single tests logged 11m, 22m and 26m waiting for budget — and the pass above still had three of seventeen tests running at 60m. * fix: keep a DB VM recoverable when its agent cannot start, and flush the volume before a snapshot An rds-agent that failed to construct exited, so a transient IMDS credential resolution took the guest down for good: rds-datadir and rds-init cascade on the missing handoff, the engine never starts, and the control plane sees only silence until it declares the instance failed ten minutes later. Construction is retried like every other boot step instead. OpenRC skips start_post when the start itself fails, which is the one case with no diagnostic at all — the console carried nothing but "died". The log mirror moves out of the handoff wait so a failed start uses it too. A snapshot quiesced the engine and nothing else. The engine repairs its own files from the WAL, but postgresql.conf sits beside them and it does not: a restore came up on a torn tail and the postmaster stopped on a syntax error at the unterminated include hook. The quiesce now flushes the whole data mount, and the hook is renamed into place rather than appended, so a boot that finds a half written one replaces it instead of appending after its open quote. * fix: flush the config cloud-init wrote before a reset can take it A DB instance rebooted seconds after it went available came back with no RDS_GATEWAY_URL, so rds-agent could not reach the control plane and the engine never started. cloud-init writes agent.env once, per-instance, and never again; a reboot is a QMP system_reset with no guest shutdown, so the file was still dirty page cache when the machine reset under it and the next boot read nothing. The handoff wait now syncs, which is the point at which the agent has proved it read that file. * fix: stop dropping an EIP teardown published while vpcd is restarting DisassociateAddress publishes vpc.delete-nat fire-and-forget. On run 34046316024 vpcd disconnected from NATS at 03:51:26.73 and its replacement subscribed at 03:51:27.09; the publish landed at 03:51:26.92, in the gap, and went nowhere. ReleaseAddress then returned the address to the pool two minutes later with its /32 host route still delivering to the guest that had held it. vpc.delete-nat now uses request-reply like vpc.add-nat, and retries only on no-responders — which returns immediately, so the cost is paid solely when nothing is listening. A teardown that still fails is logged as an error naming the address rather than lost silently. Neither is fatal to the caller: the reconciler's orphan sweep remains the backstop. vpc.add-nat on the EIP path stays fire-and-forget. The shared helper blocks it on the flows barrier for up to 45s, and associating an address should not inherit that. Two reconcile passes in that run had the address absent from intent and did not sweep its host route, and the journal cannot say why: the sweep logged only when it acted, so a route it kept was indistinguishable from one it never saw. It now logs the bindings it found, kept and wanted. * ci: render one run-level table for the e2e nightly Reading a nightly meant opening nineteen jobs, and the Actions UI does not separate a cell that failed from one cancelled out from under it — a distinction every investigation of this suite has needed, and one the run 34046316024 analysis had to reach for the jobs API to get. A final job, needs: every cell and if: always(), renders cell, topology, result, duration, runner and per-cell test counts into the step summary, with run-level totals, the three slowest cells and the failing test names. Result, duration and runner come from the jobs API so a cell that died before writing junit still gets a row. Counts come from a new junit-only artifact per cell rather than the full cell artifact, so the roll-up does not pull nineteen journals it has no use for. Verified against run 34046316024: 14 success, 2 cancelled, 1 failure, with the two cancellations shown as cancelled rather than folded into failure. * fix: assert EC2's actual EIP semantics, and probe EKS auth on an authenticated path natuplink asserted that an instance's auto-assigned public IP is still plumbed after an EIP associated to the same instance is torn down. EC2 does the opposite: associating an EIP releases the auto-assigned address back into the pool, and it cannot be reused. Our datapath already does that — run 34059879666 shows 192.168.0.72's ingress removed at 07:09:44.000, 36ms before .73's was installed — so the test was asserting behaviour neither AWS nor we have. It now asserts the release at the point it happens, and that no public delivery remains after the teardown. This surfaced only once the teardown itself started working: the run's other assertions passed, including .73's removal, which was the failure before. Our records still disagree with our datapath — the ENI keeps the released address, which is why termination tears the old one down twice and never the current one. That is the remaining half of mulga-7zc9w and is not fixed here; it needs an IPAM release on the associate path. Separately, AuthDiagnostics read /version, which is anonymously readable, so its 200 was consistent with both branches it claimed to distinguish. It now runs auth whoami, reads kube-system, and invokes the credential plugin directly, reporting exit code, stderr and whether a token was produced — never the token, since CI logs outlive it. * test: capture per-guest state when a spread guest never answers SSH TestMultinodeSpread's only failure dump is dumpPlacementNATDiag, which collects OVN, OVS, ARP and route state aimed at the NAT and EIP half of the test. The failure it actually hits is a guest that never accepted SSH — run 34040324968, the guest on node3 at 10.100.2.5 — and for that it recorded no console, no port binding and no tap state. There was no way to tell from the artifact whether the guest booted at all. Now dumps, per guest: the console, the NB logical switch port, the SB port binding, the OVS interface backing its tap, and br-int's port list. Probed on every node rather than the expected one, because a port bound on the wrong chassis is exactly what a hosting-node-only dump cannot show. Covers all three guests, not just the failing one, so a working guest is available as the control. * ci: sample hypervisor load either side of every cell Eight runner names map onto three hypervisors, so cells that never share a runner still share a host: when cell 10's SSH wait expired on run 34040324968, cells 22, 17, 18 and 28 were concurrently live, two of them multi-node. Nothing in the artifacts measured host load, which left co-tenancy as an explanation that could be neither confirmed nor ruled out. Records loadavg, MemAvailable, SwapFree, the cpu line carrying steal, and the QEMU process count before and after each cell, into the cell artifact. * ci: note that main's missing concurrency block is why the nightly runs ungrouped Three refs carry three different groupings: main has no concurrency block at all, dev uses e2e-nightly, this branch uses e2e-hypervisors. Scheduled runs load the workflow from the default branch, so every cron nightly runs ungrouped and cannot queue against anything — which is how a dispatch of 34050630627 ran alongside the scheduled 34046316024 and cost it cells 24 and 27. e2e-hypervisors is the right group and deliberately broader than dev's, since e2e.yml, e2e-suites.yml and e2e-ddil.yml draw on the same runner pools. Nothing on a feature branch can fix this; the block has to reach main.
925 lines
43 KiB
Bash
925 lines
43 KiB
Bash
#!/bin/sh
|
|
# rds-init — PostgreSQL bootstrap for the spinifex-rds-postgres AMI.
|
|
#
|
|
# A boot oneshot ordered before the postgresql service. This is the PostgreSQL
|
|
# half of the engine seam: initdb, ALTER ROLE, the datadir and the 5432 default
|
|
# live here and nowhere else in the platform.
|
|
#
|
|
# Handoff contract, written by rds-agent into a root-owned tmpfs directory:
|
|
#
|
|
# /run/spinifex-rds/bootstrap.env 0600 KEY=value shell fragment:
|
|
# RDS_MODE=initialize|attach
|
|
# RDS_DB_INSTANCE_IDENTIFIER=<the DB instance this VM backs>
|
|
# RDS_MASTER_USERNAME=<master role name>
|
|
# RDS_MASTER_PASSWORD=<present only when RDS_MODE=initialize>
|
|
# RDS_DB_NAME=<initial database; optional, as on AWS>
|
|
# RDS_PORT=<listen port; defaults to 5432>
|
|
# RDS_LISTEN_ADDRESS=<the customer ENI address the engine binds>
|
|
# RDS_CLIENT_CIDR=<the customer VPC range client authentication admits>
|
|
# RDS_DATA_VOLUME_ID / RDS_DATA_VOLUME_SERIAL=<expected volume identity>
|
|
# RDS_VM_GENERATION / RDS_FORMAT_AUTHORIZED=<create-only format grant>
|
|
# RDS_BOOTSTRAP_PENDING / RDS_PAYLOAD_ID=<the control plane is still
|
|
# waiting for this payload to be applied>
|
|
# /run/spinifex-rds/parameters.conf 0600 resolved parameter-group values in
|
|
# postgresql.conf syntax
|
|
# /run/spinifex-rds/server.crt 0600 serving cert, cluster-CA-signed
|
|
# /run/spinifex-rds/server.key 0600 its private key
|
|
#
|
|
# Nothing there survives a reboot: the next boot re-fetches and the cert is
|
|
# re-minted rather than persisted.
|
|
#
|
|
# The data volume also carries an engine stamp, written before the datadir is
|
|
# inspected and compared on every boot, so another engine's volume is refused
|
|
# rather than initialised over.
|
|
#
|
|
# rds-init keys off the datadir, never off RDS_MODE. An initialised datadir
|
|
# means skip initdb and only refresh config, so a replaced VM comes back on its
|
|
# existing data volume without re-bootstrapping. RDS_BOOTSTRAP_PENDING is not an
|
|
# exception: it is a control-plane-resolved precondition compared against
|
|
# on-disk evidence, the same shape as RDS_FORMAT_AUTHORIZED.
|
|
set -eu
|
|
|
|
PG_VERSION="${RDS_PG_VERSION:-18}"
|
|
PG_BIN="${RDS_PG_BIN:-/usr/libexec/postgresql${PG_VERSION}}"
|
|
PG_USER="${RDS_PG_USER:-postgres}"
|
|
|
|
# The engine this image bakes, stamped onto the data volume and compared against
|
|
# it on every boot. Held here rather than read from /etc/spinifex-rds/engine:
|
|
# this script is the PostgreSQL half of the seam, so it is the same constant.
|
|
ENGINE="postgres"
|
|
|
|
# Mount point of the data volume, and the datadir one level inside it. The
|
|
# datadir is deliberately not the mount point itself: initdb refuses a non-empty
|
|
# directory and a freshly formatted ext4 volume already holds lost+found.
|
|
DATA_MOUNT="${RDS_DATA_MOUNT:-/var/lib/postgresql}"
|
|
PGDATA="${RDS_PGDATA:-${DATA_MOUNT}/${PG_VERSION}/data}"
|
|
|
|
# The TLS enforcement rule, included by the generated pg_hba when present. A
|
|
# relative include resolves against the directory holding the referencing file,
|
|
# so it sits inside PGDATA and rides the data volume through a VM replacement.
|
|
HBA_DIR="${PGDATA}/hba.d"
|
|
HBA_FORCE_SSL_INCLUDE="hba.d/20-rds-force-ssl.conf"
|
|
|
|
# The resolved parameter group, and the one setting in it this script reads
|
|
# rather than only installs. PostgreSQL has no server setting for requiring TLS,
|
|
# so the value is inert in the file and becomes behaviour only as the rule above.
|
|
PARAMETERS_FILE="${PGDATA}/conf.d/10-rds-parameters.conf"
|
|
TLS_PARAM="rds.force_ssl"
|
|
|
|
# Written between initdb and a working master role. It lives on the data volume
|
|
# because no trap survives a crash or a hard kill, and it is what stops the next
|
|
# boot attaching to a datadir whose master role was never created.
|
|
BOOTSTRAP_SENTINEL="${PGDATA}/rds-bootstrap-incomplete"
|
|
|
|
# Proof that the master role was durably applied, written once initdb and the
|
|
# role are both done. Deliberately outside PGDATA, which clear_unfinished_datadir
|
|
# rm -rf's: no initdb-adjacent path can touch it and PGDATA stays a stock datadir.
|
|
RECEIPT_DIR="${RDS_RECEIPT_DIR:-${DATA_MOUNT}/.spinifex-rds/bootstrap}"
|
|
RECEIPT_FILE="${RECEIPT_DIR}/receipt.env"
|
|
|
|
# Which engine's data this volume holds. Outside PGDATA for the same reason the
|
|
# receipt is, and beside it rather than inside the bootstrap directory: it
|
|
# outlives any one bootstrap and is read before the datadir is ever inspected.
|
|
ENGINE_STAMP="${RDS_ENGINE_STAMP:-${DATA_MOUNT}/.spinifex-rds/engine}"
|
|
|
|
HANDOFF_DIR="${RDS_HANDOFF_DIR:-/run/spinifex-rds}"
|
|
ENV_FILE="${HANDOFF_DIR}/bootstrap.env"
|
|
SOCKET_DIR="${RDS_SOCKET_DIR:-/run/postgresql}"
|
|
TLS_DIR="${RDS_TLS_DIR:-${SOCKET_DIR}/tls}"
|
|
LOG_DIR="${RDS_LOG_DIR:-/var/log/postgresql}"
|
|
|
|
# The bootstrap server listens on a unix socket in this directory and nowhere
|
|
# else. It is 0700 and postgres-owned, so only postgres and root can reach the
|
|
# short trust-authenticated window before the master password exists.
|
|
BOOTSTRAP_DIR="${RDS_BOOTSTRAP_DIR:-/run/rds-init}"
|
|
|
|
MOUNTS_FILE="${RDS_MOUNTS_FILE:-/proc/mounts}"
|
|
|
|
# How long the bind address may take to appear on a local interface. The lease
|
|
# is taken in the boot runlevel on a 600s budget and this script runs in the
|
|
# default one, so the wait is a guard rather than an expected delay.
|
|
LISTEN_ADDRESS_TIMEOUT="${RDS_LISTEN_ADDRESS_TIMEOUT:-120}"
|
|
|
|
# The group role the master's administrative privileges hang off. Named as on
|
|
# AWS, so a customer's `GRANT rds_superuser TO ...` works here too.
|
|
MASTER_GROUP_ROLE="rds_superuser"
|
|
|
|
# The Ochre vector store appliance is the one platform (non-tenant) consumer of
|
|
# this image, provisioned with this fixed master name. Tenant instances take a
|
|
# different master, so the pgvector install below is scoped to the appliance.
|
|
OCHRE_APPLIANCE_MASTER="ochre_vector_admin"
|
|
|
|
# NAMEDATALEN-1: the server truncates a longer identifier, so the database
|
|
# created would not be the one the customer asked for.
|
|
MAX_DB_NAME_LEN=63
|
|
|
|
log() { echo "[rds-init] $*"; }
|
|
fail() { echo "[rds-init] ERROR: $*" >&2; exit 1; }
|
|
|
|
# Between these two, the datadir carries PG_VERSION but no usable master role.
|
|
_created_datadir=0
|
|
_bootstrapped=0
|
|
|
|
# Runs from a trap rather than a statement, so neither `set -e` on an unguarded
|
|
# step between initdb and a working master role nor a signal can skip it — either
|
|
# would leave a healthy-looking engine with no usable master role, unrepairable.
|
|
clear_unfinished_datadir() {
|
|
[ "${_created_datadir}" = "1" ] || return 0
|
|
[ "${_bootstrapped}" = "1" ] && return 0
|
|
# The receipt is installed before the traps retire, so a sweep in between
|
|
# would leave one vouching for a datadir that is about to be deleted. The
|
|
# agent would acknowledge it and the control plane would destroy the only
|
|
# copy of the password this instance can ever be initialised with.
|
|
rm -f -- "${RECEIPT_FILE}"
|
|
if rm -rf -- "${PGDATA}" && [ ! -e "${PGDATA}" ]; then
|
|
echo "[rds-init] ERROR: bootstrap did not complete; datadir ${PGDATA} cleared" >&2
|
|
return 0
|
|
fi
|
|
echo "[rds-init] ERROR: bootstrap did not complete and datadir ${PGDATA} could not be cleared; recover it out of band before retrying" >&2
|
|
}
|
|
|
|
# The server refuses to run as root.
|
|
as_pg() { su "${PG_USER}" -c "$1"; }
|
|
|
|
# A dotted-quad IPv4 literal. postgresql.conf would also take a hostname here,
|
|
# but the control plane sends an address, and accepting a name would reintroduce
|
|
# the resolution step the bind exists to remove.
|
|
is_ipv4() {
|
|
case "$1" in
|
|
'' | *[!0-9.]*) return 1 ;;
|
|
esac
|
|
_rest="$1"
|
|
_octets=0
|
|
while [ -n "${_rest}" ]; do
|
|
_octet="${_rest%%.*}"
|
|
case "${_rest}" in
|
|
*.*) _rest="${_rest#*.}" ;;
|
|
*) _rest="" ;;
|
|
esac
|
|
# A leading zero reads as octal to some parsers and decimal to others,
|
|
# so the two would disagree about which address was bound.
|
|
case "${_octet}" in
|
|
0 | [1-9] | [1-9][0-9] | [1-9][0-9][0-9]) ;;
|
|
*) return 1 ;;
|
|
esac
|
|
[ "${_octet}" -le 255 ] || return 1
|
|
_octets=$((_octets + 1))
|
|
done
|
|
[ "${_octets}" = 4 ]
|
|
}
|
|
|
|
is_ipv4_cidr() {
|
|
case "$1" in
|
|
*/*) ;;
|
|
*) return 1 ;;
|
|
esac
|
|
_prefix="${1#*/}"
|
|
case "${_prefix}" in
|
|
'' | *[!0-9]*) return 1 ;;
|
|
esac
|
|
[ "${#_prefix}" -le 2 ] && [ "${_prefix}" -le 32 ] || return 1
|
|
is_ipv4 "${1%%/*}"
|
|
}
|
|
|
|
# The engine binds one address and fails to start if it is not there yet, and an
|
|
# engine that will not start for an unexplained reason is the worst way to learn
|
|
# that DHCP lost its race. Bounded, then fatal with the address in the message.
|
|
wait_for_listen_address() {
|
|
_deadline=$(($(date +%s) + LISTEN_ADDRESS_TIMEOUT))
|
|
while :; do
|
|
if ip -4 -o addr show 2>/dev/null |
|
|
awk -v want="${RDS_LISTEN_ADDRESS}" \
|
|
'{ split($4, addr, "/"); if (addr[1] == want) found = 1 } END { exit !found }'; then
|
|
return 0
|
|
fi
|
|
[ "$(date +%s)" -lt "${_deadline}" ] || break
|
|
sleep 1
|
|
done
|
|
fail "the endpoint address ${RDS_LISTEN_ADDRESS} did not appear on any interface within ${LISTEN_ADDRESS_TIMEOUT}s; the engine binds it and nothing else, so refusing to start"
|
|
}
|
|
|
|
load_handoff() {
|
|
[ -f "${ENV_FILE}" ] || fail "no bootstrap handoff at ${ENV_FILE} — rds-agent has not delivered the bootstrap config"
|
|
|
|
# shellcheck disable=SC1090
|
|
. "${ENV_FILE}"
|
|
|
|
RDS_MODE="${RDS_MODE:-}"
|
|
RDS_DB_INSTANCE_IDENTIFIER="${RDS_DB_INSTANCE_IDENTIFIER:-}"
|
|
RDS_MASTER_USERNAME="${RDS_MASTER_USERNAME:-}"
|
|
RDS_MASTER_PASSWORD="${RDS_MASTER_PASSWORD:-}"
|
|
RDS_DB_NAME="${RDS_DB_NAME:-}"
|
|
RDS_PORT="${RDS_PORT:-5432}"
|
|
RDS_LISTEN_ADDRESS="${RDS_LISTEN_ADDRESS:-}"
|
|
RDS_CLIENT_CIDR="${RDS_CLIENT_CIDR:-}"
|
|
RDS_VM_GENERATION="${RDS_VM_GENERATION:-}"
|
|
# Absent from an agent that predates the receipt protocol, which reads as
|
|
# "nothing is pending" and leaves every path below unchanged.
|
|
RDS_BOOTSTRAP_PENDING="${RDS_BOOTSTRAP_PENDING:-0}"
|
|
RDS_PAYLOAD_ID="${RDS_PAYLOAD_ID:-}"
|
|
|
|
[ -n "${RDS_MASTER_USERNAME}" ] || fail "bootstrap config has no RDS_MASTER_USERNAME"
|
|
|
|
# The master role is created NOSUPERUSER, so a master named after the
|
|
# bootstrap superuser would leave the cluster without one and strip rds-agent
|
|
# of the privileged SQL it reaches over peer auth — unrepairable, since the
|
|
# datadir bootstraps once.
|
|
#
|
|
# The same set the control plane reserves, matched the same way: anything it
|
|
# refuses must be refused here too, or the name reaches initdb, fails at role
|
|
# creation and spends the one-shot password on the way.
|
|
_master_lower=$(printf '%s' "${RDS_MASTER_USERNAME}" | tr '[:upper:]' '[:lower:]')
|
|
case "${_master_lower}" in
|
|
"${PG_USER}" | "${MASTER_GROUP_ROLE}" | rdsadmin | pg_*)
|
|
fail "RDS_MASTER_USERNAME ${RDS_MASTER_USERNAME} is reserved by the platform; the control plane should have rejected it"
|
|
;;
|
|
esac
|
|
|
|
# Scopes the platform pgvector install to the appliance; a tenant master
|
|
# takes a different name and skips it.
|
|
RDS_IS_OCHRE_APPLIANCE=0
|
|
[ "${_master_lower}" = "${OCHRE_APPLIANCE_MASTER}" ] && RDS_IS_OCHRE_APPLIANCE=1
|
|
|
|
# The same rule the control plane applies, for the same reason the reserved
|
|
# name is re-checked: an unusable name reaches initdb, fails at CREATE
|
|
# DATABASE and spends the one-shot password on the way. psql quotes the
|
|
# identifier, so here this is defence in depth rather than the barrier.
|
|
case "${RDS_DB_NAME}" in
|
|
'') ;;
|
|
[!A-Za-z]* | *[!A-Za-z0-9_]*)
|
|
fail "RDS_DB_NAME ${RDS_DB_NAME} must begin with a letter and hold only letters, digits and underscores; the control plane should have rejected it"
|
|
;;
|
|
esac
|
|
if [ "${#RDS_DB_NAME}" -gt "${MAX_DB_NAME_LEN}" ]; then
|
|
fail "RDS_DB_NAME is longer than the ${MAX_DB_NAME_LEN} characters PostgreSQL allows; the control plane should have rejected it"
|
|
fi
|
|
|
|
case "${RDS_PORT}" in
|
|
'' | *[!0-9]*) fail "bootstrap config has a non-numeric RDS_PORT: ${RDS_PORT}" ;;
|
|
esac
|
|
|
|
# Neither falls back to a wildcard. The engine is reachable over the customer
|
|
# ENI and nowhere else, and a handoff that cannot say which address that is
|
|
# has no safe reading — the DB VM's other two NICs carry no security group
|
|
# the customer can see, let alone set.
|
|
[ -n "${RDS_LISTEN_ADDRESS}" ] || fail "bootstrap config has no RDS_LISTEN_ADDRESS; refusing to start an engine that would answer on every interface"
|
|
is_ipv4 "${RDS_LISTEN_ADDRESS}" ||
|
|
fail "bootstrap config has a non-IPv4 RDS_LISTEN_ADDRESS: ${RDS_LISTEN_ADDRESS}"
|
|
[ -n "${RDS_CLIENT_CIDR}" ] || fail "bootstrap config has no RDS_CLIENT_CIDR; refusing to start an engine that would accept clients from anywhere"
|
|
is_ipv4_cidr "${RDS_CLIENT_CIDR}" ||
|
|
fail "bootstrap config has a non-IPv4 RDS_CLIENT_CIDR: ${RDS_CLIENT_CIDR}"
|
|
|
|
# redact_stream is line-oriented, so a password spanning a line boundary
|
|
# would match neither half and reach /dev/console in the clear. The control
|
|
# plane rejects one; refusing here too keeps that an asserted property of
|
|
# the handoff rather than a silent dependency on another service.
|
|
case "${RDS_MASTER_PASSWORD}" in
|
|
*'
|
|
'*) fail "the master password in the bootstrap config spans more than one line; the control plane should have rejected it" ;;
|
|
esac
|
|
|
|
log "bootstrap config loaded (mode=${RDS_MODE:-unset}, port=${RDS_PORT})"
|
|
}
|
|
|
|
# Every replace boots a fresh clone of this AMI, so a datadir off the data volume
|
|
# disappears at the next class change or recovery — silently, as the engine looks
|
|
# fine.
|
|
require_data_volume() {
|
|
if awk -v mnt="${DATA_MOUNT}" '$2 == mnt { found = 1 } END { exit !found }' "${MOUNTS_FILE}"; then
|
|
return 0
|
|
fi
|
|
[ "${RDS_ALLOW_LOCAL_DATADIR:-0}" = "1" ] ||
|
|
fail "${DATA_MOUNT} is not a mount point — the data volume is not attached; refusing to initialise on the boot volume"
|
|
log "warning: ${DATA_MOUNT} is not a mount point; data will not survive a VM replace (RDS_ALLOW_LOCAL_DATADIR=1)"
|
|
}
|
|
|
|
# Written before the datadir is inspected, so a volume this image goes on to
|
|
# initialise carries the stamp from the first boot onwards. Fatal on failure:
|
|
# an unstamped volume is one the next boot cannot tell from another engine's.
|
|
write_engine_stamp() {
|
|
_stamp_dir=$(dirname "${ENGINE_STAMP}")
|
|
install -d -m 0700 -o "${PG_USER}" -g "${PG_USER}" "${_stamp_dir}" ||
|
|
fail "the datadir engine stamp directory ${_stamp_dir} could not be created; refusing to use a data volume whose engine cannot be recorded"
|
|
|
|
_tmp="${ENGINE_STAMP}.new"
|
|
printf '%s\n' "${ENGINE}" > "${_tmp}" ||
|
|
fail "the datadir engine stamp ${ENGINE_STAMP} could not be written; refusing to use a data volume whose engine cannot be recorded"
|
|
chown "${PG_USER}:${PG_USER}" "${_tmp}"
|
|
chmod 0600 "${_tmp}"
|
|
|
|
# Renamed rather than written in place, so a crash leaves either the old
|
|
# stamp or none — and none is backfilled, where a torn one would be fatal.
|
|
sync "${_tmp}" 2>/dev/null || sync
|
|
mv -f "${_tmp}" "${ENGINE_STAMP}" ||
|
|
fail "the datadir engine stamp ${ENGINE_STAMP} could not be installed; refusing to use a data volume whose engine cannot be recorded"
|
|
sync "${_stamp_dir}" 2>/dev/null || sync
|
|
}
|
|
|
|
# The volume mounts cleanly whichever engine wrote it, and another engine's
|
|
# datadir reads as uninitialised here: rds-init would initdb beside the
|
|
# customer's data and serve an empty database that passes every health probe.
|
|
require_engine_stamp() {
|
|
if [ -e "${ENGINE_STAMP}" ]; then
|
|
_stamped=""
|
|
_stamp_read=0
|
|
_stamped=$(head -n 1 -- "${ENGINE_STAMP}") && _stamp_read=1
|
|
[ "${_stamp_read}" = "1" ] ||
|
|
fail "the datadir engine stamp ${ENGINE_STAMP} exists but could not be read; refusing to attach to a data volume whose engine cannot be established"
|
|
if [ "${_stamped}" != "${ENGINE}" ]; then
|
|
fail "the data volume holds a '${_stamped}' datadir but this image runs ${ENGINE}; refusing to touch it. Attach it to a '${_stamped}' instance, or recover the volume out of band."
|
|
fi
|
|
log "data volume is stamped ${ENGINE}"
|
|
return 0
|
|
fi
|
|
|
|
# No stamp is either a volume predating it or one about to be initialised.
|
|
# Both are ours here — PostgreSQL had unstamped volumes in the field, so a
|
|
# non-empty datadir is backfilled rather than refused.
|
|
log "data volume carries no engine stamp — recording ${ENGINE}"
|
|
write_engine_stamp
|
|
}
|
|
|
|
# Runs initdb into an empty datadir. The cluster config — the include_dir hook,
|
|
# pg_hba and the parameters — is laid down by install_runtime_config, which both
|
|
# the initialise and the attach path run.
|
|
initialise_cluster() {
|
|
# An empty datadir in `attach` mode means the data volume did not come back.
|
|
# Creating a cluster there would report success and hand the customer an
|
|
# empty database in place of theirs.
|
|
if [ "${RDS_MODE}" = "attach" ]; then
|
|
fail "datadir ${PGDATA} is empty but the bootstrap config is Mode=attach — the existing data volume is missing; refusing to create an empty cluster"
|
|
fi
|
|
[ -n "${RDS_MASTER_PASSWORD}" ] || fail "Mode=${RDS_MODE:-unset} carries no master password; cannot initialise the cluster"
|
|
|
|
install -d -m 0700 -o "${PG_USER}" -g "${PG_USER}" "$(dirname "${PGDATA}")" "${PGDATA}"
|
|
|
|
# Whether the datadir was empty beforehand gates the failure cleanup below.
|
|
# A torn write can leave a missing PG_VERSION over intact data, and deleting
|
|
# then would destroy it in response to the one signal that it is still there.
|
|
_was_empty=0
|
|
if _contents=$(ls -A -- "${PGDATA}" 2>/dev/null); then
|
|
[ -z "${_contents}" ] && _was_empty=1
|
|
else
|
|
fail "could not inspect datadir ${PGDATA}; refusing to decide whether failure cleanup is safe"
|
|
fi
|
|
|
|
log "initialising PostgreSQL ${PG_VERSION} cluster at ${PGDATA}"
|
|
# ICU collation, not libc: musl has no locale support, so a libc-provider
|
|
# cluster would silently behave as C. Local peer auth keeps root/postgres
|
|
# administration working once the bootstrap server is gone.
|
|
if ! as_pg "'${PG_BIN}/initdb' --pgdata='${PGDATA}' -E UTF-8 \
|
|
--locale-provider=icu --icu-locale=en-001-x-icu --data-checksums \
|
|
--auth-local=peer --auth-host=scram-sha-256"; then
|
|
# A half-written datadir carries PG_VERSION and would look initialised
|
|
# forever after, but only what this invocation created may be cleared.
|
|
if [ "${_was_empty}" = "1" ]; then
|
|
if rm -rf -- "${PGDATA}" && [ ! -e "${PGDATA}" ]; then
|
|
fail "initdb failed; datadir cleared"
|
|
fi
|
|
fail "initdb failed and datadir ${PGDATA} could not be cleared; recover it out of band before retrying"
|
|
fi
|
|
fail "initdb failed and ${PGDATA} was not empty beforehand — refusing to clear it, since the contents may be a datadir whose PG_VERSION was lost; recover the data volume out of band"
|
|
fi
|
|
|
|
# Only a datadir initdb built over an empty directory may be swept later.
|
|
_created_datadir="${_was_empty}"
|
|
trap clear_unfinished_datadir EXIT
|
|
# An EXIT trap does not run on a signal in a POSIX shell, so the same sweep
|
|
# is armed for the signals a shutdown, an ACPI powerdown or a host-side
|
|
# force-stop delivers. EXIT is disarmed first so the sweep runs once.
|
|
trap 'trap - EXIT; clear_unfinished_datadir; exit 1' INT TERM HUP
|
|
|
|
# No trap survives a crash or a SIGKILL, so the incomplete state is also
|
|
# recorded on the volume itself. Cleared once the master role exists.
|
|
printf '%s\n' \
|
|
'rds-init created this datadir and did not finish bootstrapping the master role.' \
|
|
'While this file exists rds-init refuses to start the engine: the datadir carries no' \
|
|
'usable master role, so it must not be attached to. Recover out of band.' \
|
|
> "${BOOTSTRAP_SENTINEL}"
|
|
chown "${PG_USER}:${PG_USER}" "${BOOTSTRAP_SENTINEL}"
|
|
chmod 0600 "${BOOTSTRAP_SENTINEL}"
|
|
}
|
|
|
|
# Parameters live beside the data rather than in /etc, since a class change boots
|
|
# a fresh root volume and would quietly revert them to engine defaults.
|
|
install_parameters() {
|
|
if [ -f "${HANDOFF_DIR}/parameters.conf" ]; then
|
|
install -m 0600 -o "${PG_USER}" -g "${PG_USER}" \
|
|
"${HANDOFF_DIR}/parameters.conf" "${PARAMETERS_FILE}"
|
|
return 0
|
|
fi
|
|
# A handoff without parameters keeps the previous include: dropping it would
|
|
# silently reset the instance to engine defaults.
|
|
if [ -f "${PARAMETERS_FILE}" ]; then
|
|
log "warning: no parameters in the handoff — keeping the parameters already installed"
|
|
fi
|
|
}
|
|
|
|
# Sets TLS_ENFORCE from the installed parameter file rather than the handoff's:
|
|
# install_parameters keeps the previous file when a handoff carries none, so the
|
|
# installed one is what the engine reads and what this has to agree with.
|
|
#
|
|
# An absent key is enforce. That is the ordinary state of an instance whose
|
|
# resolved set predates the parameter, and it is the whole of the migration — one
|
|
# begins requiring TLS at this boot with no control-plane work at all.
|
|
derive_tls_enforcement() {
|
|
TLS_ENFORCE=1
|
|
[ -f "${PARAMETERS_FILE}" ] || return 0
|
|
|
|
# Every value in the file is written quoted, and the quotes are optional here
|
|
# so the format is not a second thing to keep in step.
|
|
_value=$(sed -n "s/^[[:space:]]*${TLS_PARAM}[[:space:]]*=[[:space:]]*'\{0,1\}\([^'[:space:]]*\).*/\1/p" \
|
|
"${PARAMETERS_FILE}" | tail -n 1)
|
|
[ -n "${_value}" ] || return 0
|
|
|
|
case "${_value}" in
|
|
1) TLS_ENFORCE=1 ;;
|
|
0) TLS_ENFORCE=0 ;;
|
|
# The control plane canonicalises every boolean to 1 or 0, so anything else
|
|
# means the file was written by something other than the platform. Reading
|
|
# an unparsable security setting as off is the one choice not open here.
|
|
*) fail "the installed parameters set ${TLS_PARAM} to '${_value}', which is neither 1 nor 0; refusing to start the engine on a TLS setting that cannot be read" ;;
|
|
esac
|
|
}
|
|
|
|
# A configuration that asks for TLS is never quietly served in the clear. Not a
|
|
# transient-failure risk: load_handoff already required a successful bootstrap
|
|
# fetch to get this far, and that fetch carries a certificate whenever a CA
|
|
# exists.
|
|
require_tls_for_enforcement() {
|
|
[ "${TLS_ENFORCE}" = "1" ] || return 0
|
|
if [ "${SSL_MODE}" != "on" ]; then
|
|
fail "${TLS_PARAM} requires TLS of client connections but no serving certificate was delivered and none is installed; refusing to start an engine that would reject every client"
|
|
fi
|
|
}
|
|
|
|
# Enforcement is the presence of this file and nothing else: its content is a
|
|
# constant, so the copy written here and the one rds-agent writes in Go have no
|
|
# template to drift apart on.
|
|
write_tls_enforcement() {
|
|
_rule="${PGDATA}/${HBA_FORCE_SSL_INCLUDE}"
|
|
if [ "${TLS_ENFORCE}" != "1" ]; then
|
|
# A restored data volume carries hba.d with it, so a rule left from the
|
|
# snapshot would keep enforcing under a group that turns enforcement off.
|
|
rm -f "${_rule}" ||
|
|
fail "the TLS enforcement rule ${_rule} could not be removed; the engine would keep requiring TLS that the parameter group does not"
|
|
return 0
|
|
fi
|
|
|
|
_tmp="${HBA_DIR}/.20-rds-force-ssl.conf.new"
|
|
cat > "${_tmp}" <<EOF
|
|
# Managed by the platform: the rule that requires TLS of client connections.
|
|
# Written by rds-init at boot and by rds-agent on a parameter apply.
|
|
hostnossl all all 0.0.0.0/0 reject
|
|
hostnossl all all ::/0 reject
|
|
EOF
|
|
chown "${PG_USER}:${PG_USER}" "${_tmp}"
|
|
chmod 0600 "${_tmp}"
|
|
|
|
# mv onto a directory of that name would move the temp file inside it and
|
|
# report success, leaving the include pointing at a directory.
|
|
if [ -d "${_rule}" ]; then
|
|
fail "${_rule} is a directory rather than the TLS enforcement rule; refusing to start the engine on client authentication rds-init does not control"
|
|
fi
|
|
mv -f "${_tmp}" "${_rule}" ||
|
|
fail "the TLS enforcement rule ${_rule} could not be installed"
|
|
}
|
|
|
|
# The key stays on tmpfs: it is re-minted every fetch, so persisting it would
|
|
# spread a private key into every snapshot.
|
|
install_tls() {
|
|
if [ -f "${HANDOFF_DIR}/server.crt" ] && [ -f "${HANDOFF_DIR}/server.key" ]; then
|
|
install -d -m 0700 -o "${PG_USER}" -g "${PG_USER}" "${TLS_DIR}"
|
|
install -m 0600 -o "${PG_USER}" -g "${PG_USER}" "${HANDOFF_DIR}/server.crt" "${TLS_DIR}/server.crt"
|
|
install -m 0600 -o "${PG_USER}" -g "${PG_USER}" "${HANDOFF_DIR}/server.key" "${TLS_DIR}/server.key"
|
|
rm -f "${HANDOFF_DIR}/server.crt" "${HANDOFF_DIR}/server.key"
|
|
SSL_MODE="on"
|
|
return 0
|
|
fi
|
|
# A re-run after the cert was consumed must not turn TLS off underneath
|
|
# clients connecting with sslmode=require.
|
|
if [ -f "${TLS_DIR}/server.crt" ] && [ -f "${TLS_DIR}/server.key" ]; then
|
|
SSL_MODE="on"
|
|
return 0
|
|
fi
|
|
log "warning: no serving cert in the handoff — the engine will start without TLS"
|
|
SSL_MODE="off"
|
|
}
|
|
|
|
# The whole file, regenerated every boot rather than appended to initdb's.
|
|
# pg_hba is first-match-wins, so initdb's loopback lines sort above anything
|
|
# appended and no rule appended at the end could ever govern a local client.
|
|
#
|
|
# The include path is double-quoted because pg_hba quotes with " and not ':
|
|
# single quotes would become part of the filename, and include_if_exists skips a
|
|
# file it cannot find without a word, which reads as enforcing and does not.
|
|
write_pg_hba() {
|
|
_hba="${PGDATA}/pg_hba.conf"
|
|
_tmp="${PGDATA}/.pg_hba.conf.new"
|
|
cat > "${_tmp}" <<EOF
|
|
# Managed by rds-init — regenerated on every boot. Edits here are lost.
|
|
#
|
|
# Clients reach the instance over its customer ENI, gated by the DB security
|
|
# group as on AWS RDS, and authenticate with scram-sha-256. The host rule is
|
|
# scoped to the customer VPC rather than the DB subnet: an app subnet reaching a
|
|
# DB subnet arrives carrying its own address, with no NAT in between. The engine
|
|
# does not bind loopback, so initdb's own 127.0.0.1 and ::1 lines govern nothing
|
|
# and are not carried over. The local socket is how rds-init and rds-agent run
|
|
# privileged SQL.
|
|
local all all peer
|
|
include_if_exists "${HBA_FORCE_SSL_INCLUDE}"
|
|
host all all ${RDS_CLIENT_CIDR} scram-sha-256
|
|
EOF
|
|
chown "${PG_USER}:${PG_USER}" "${_tmp}"
|
|
chmod 0600 "${_tmp}"
|
|
|
|
# mv onto a directory of that name would move the temp file inside it and
|
|
# report success, leaving initdb's rules in force with nothing to say so.
|
|
if [ -d "${_hba}" ]; then
|
|
fail "${_hba} is a directory rather than PostgreSQL's client authentication file; refusing to start the engine on rules rds-init does not control"
|
|
fi
|
|
# Renamed rather than written in place: a torn pg_hba locks every client and
|
|
# rds-agent out of the instance at once.
|
|
mv -f "${_tmp}" "${_hba}" ||
|
|
fail "the client authentication file ${_hba} could not be installed"
|
|
}
|
|
|
|
# The 90- prefix sorts last within include_dir and the last setting wins, so a
|
|
# resolved parameter can never move the endpoint's port, widen the bind, turn off
|
|
# TLS or lower the protocol floor.
|
|
#
|
|
# listen_addresses is the customer ENI and nothing else — not the wildcard, and
|
|
# not loopback either. The DB VM's other two NICs sit on the shared RDS system
|
|
# VPC and the host's management bridge, neither of which any customer security
|
|
# group governs, so a socket bound there is reachable across accounts.
|
|
#
|
|
# The floor is written whether or not a cert was delivered: it costs nothing with
|
|
# ssl off, and making it conditional would mean a boot without a cert leaves the
|
|
# next reload — which does not re-run rds-init — with no floor at all.
|
|
write_rds_conf() {
|
|
_ssl="$1"
|
|
_tmp="${PGDATA}/conf.d/.90-rds-init.conf.new"
|
|
cat > "${_tmp}" <<EOF
|
|
# Managed by rds-init — regenerated on every boot from the bootstrap config.
|
|
# Edits here are lost; instance parameters belong in the parameter group.
|
|
listen_addresses = '${RDS_LISTEN_ADDRESS}'
|
|
port = ${RDS_PORT}
|
|
unix_socket_directories = '${SOCKET_DIR}'
|
|
ssl = ${_ssl}
|
|
ssl_min_protocol_version = 'TLSv1.3'
|
|
EOF
|
|
if [ "${_ssl}" = "on" ]; then
|
|
cat >> "${_tmp}" <<EOF
|
|
ssl_cert_file = '${TLS_DIR}/server.crt'
|
|
ssl_key_file = '${TLS_DIR}/server.key'
|
|
EOF
|
|
fi
|
|
chown "${PG_USER}:${PG_USER}" "${_tmp}"
|
|
chmod 0600 "${_tmp}"
|
|
mv -f "${_tmp}" "${PGDATA}/conf.d/90-rds-init.conf"
|
|
}
|
|
|
|
# postgresql.conf is on the data volume, so the hook is asserted every boot
|
|
# rather than only at initdb — a restored volume may predate it.
|
|
#
|
|
# Rewritten and renamed rather than appended to. It is the only file rds-init
|
|
# puts on that volume without the engine's knowledge, so a torn tail is the one
|
|
# corruption the engine cannot repair from its own WAL: the postmaster stops on
|
|
# a syntax error and the instance never comes up.
|
|
install_include_hook() {
|
|
_conf="${PGDATA}/postgresql.conf"
|
|
_tmp="${PGDATA}/.postgresql.conf.new"
|
|
|
|
if grep -q "^include_dir = 'conf.d'" "${_conf}" 2>/dev/null; then
|
|
return 0
|
|
fi
|
|
# Any half-written hook this script left behind goes rather than being kept:
|
|
# the guard above does not match one, so appending after its unterminated
|
|
# quote would stop the postmaster on a file it could not have started from.
|
|
grep -v "^include_dir = 'conf" "${_conf}" 2>/dev/null >"${_tmp}" || :
|
|
printf "\n# Managed by rds-init: parameter-group values (10-) then platform settings (90-).\ninclude_dir = 'conf.d'\n" \
|
|
>>"${_tmp}"
|
|
|
|
chown "${PG_USER}:${PG_USER}" "${_tmp}"
|
|
chmod 0600 "${_tmp}"
|
|
mv -f "${_tmp}" "${_conf}" ||
|
|
fail "the include hook could not be installed into ${_conf}"
|
|
}
|
|
|
|
# Everything refreshed on every boot: the include hook, the client
|
|
# authentication rules, the resolved parameters, the serving cert, the TLS
|
|
# enforcement those parameters ask for and the platform's own connectivity
|
|
# settings.
|
|
install_runtime_config() {
|
|
install -d -m 0700 -o "${PG_USER}" -g "${PG_USER}" "${PGDATA}/conf.d" "${HBA_DIR}"
|
|
install -d -m 1775 -o "${PG_USER}" -g "${PG_USER}" "${SOCKET_DIR}"
|
|
install -d -m 0755 -o "${PG_USER}" -g "${PG_USER}" "${LOG_DIR}"
|
|
|
|
install_include_hook
|
|
|
|
write_pg_hba
|
|
install_parameters
|
|
install_tls
|
|
derive_tls_enforcement
|
|
require_tls_for_enforcement
|
|
write_tls_enforcement
|
|
write_rds_conf "${SSL_MODE}"
|
|
}
|
|
|
|
# Brings the cluster up on a private socket with trust auth and no TLS, so the
|
|
# master password can be set before the engine is reachable. Never listens on TCP.
|
|
start_bootstrap_server() {
|
|
install -d -m 0700 -o "${PG_USER}" -g "${PG_USER}" "${BOOTSTRAP_DIR}"
|
|
printf 'local all all trust\n' > "${BOOTSTRAP_DIR}/pg_hba.conf"
|
|
chown "${PG_USER}:${PG_USER}" "${BOOTSTRAP_DIR}/pg_hba.conf"
|
|
chmod 0600 "${BOOTSTRAP_DIR}/pg_hba.conf"
|
|
|
|
as_pg "'${PG_BIN}/pg_ctl' --pgdata='${PGDATA}' --wait --timeout=60 \
|
|
--log='${LOG_DIR}/rds-init.log' \
|
|
-o \"-c listen_addresses='' -c unix_socket_directories='${BOOTSTRAP_DIR}' \
|
|
-c unix_socket_permissions=0700 -c hba_file='${BOOTSTRAP_DIR}/pg_hba.conf' -c ssl=off\" \
|
|
start"
|
|
}
|
|
|
|
# The trust-authenticated pg_hba is removed by the caller, not here: a caller
|
|
# reading this function's status must be reading pg_ctl's.
|
|
stop_bootstrap_server() {
|
|
as_pg "'${PG_BIN}/pg_ctl' --pgdata='${PGDATA}' --wait --timeout=60 --mode=fast stop"
|
|
}
|
|
|
|
# Prepended to every bootstrap script. psql interpolates the password into the
|
|
# ALTER ROLE before the server sees it, and these three are what would then write
|
|
# it to the log: log_min_error_statement does so on any failure at its own
|
|
# default. All are SUSET and set for this session only, so nothing the customer
|
|
# puts in the parameter group overrides them.
|
|
BOOTSTRAP_LOG_GUARD="SET log_statement = 'none';
|
|
SET log_min_duration_statement = -1;
|
|
SET log_min_error_statement = 'panic';"
|
|
|
|
# Replaces every occurrence of the secret with a placeholder. Parameter
|
|
# expansion rather than sed, whose script would carry the secret in an argv any
|
|
# process on the guest can read.
|
|
redact_stream() {
|
|
_secret="$1"
|
|
while IFS= read -r _line || [ -n "${_line}" ]; do
|
|
_out=""
|
|
while [ -n "${_secret}" ]; do
|
|
case "${_line}" in
|
|
*"${_secret}"*)
|
|
_out="${_out}${_line%%"${_secret}"*}[REDACTED]"
|
|
_line="${_line#*"${_secret}"}"
|
|
;;
|
|
*) break ;;
|
|
esac
|
|
done
|
|
printf '%s\n' "${_out}${_line}"
|
|
done
|
|
}
|
|
|
|
# Identifiers and the password ride the environment and are re-quoted by psql,
|
|
# never reaching a shell word or argv. Both streams are redacted before they
|
|
# reach the caller: psql echoes the statement it failed on, and the initd sends
|
|
# everything here to /dev/console, which the host captures off ttyS0.
|
|
psql_bootstrap() {
|
|
# A pipeline's status is its last command's and POSIX sh has no pipefail, so
|
|
# psql's own status leaves the subshell through the filesystem. The guard
|
|
# shifts psql's <stdin> line numbers by the number of lines it adds.
|
|
_status_file="${BOOTSTRAP_DIR}/psql.status"
|
|
rm -f "${_status_file}"
|
|
{
|
|
_psql_rc=0
|
|
{ printf '%s\n' "${BOOTSTRAP_LOG_GUARD}"; cat; } |
|
|
RDS_MASTER_USERNAME="${RDS_MASTER_USERNAME}" \
|
|
RDS_MASTER_PASSWORD="${RDS_MASTER_PASSWORD}" \
|
|
RDS_DB_NAME="${RDS_DB_NAME}" \
|
|
RDS_GROUP_ROLE="${MASTER_GROUP_ROLE}" \
|
|
"${PG_BIN}/psql" --no-psqlrc --quiet --no-align --tuples-only \
|
|
-v ON_ERROR_STOP=1 \
|
|
-h "${BOOTSTRAP_DIR}" -p "${RDS_PORT}" -U "${PG_USER}" -d postgres -f - ||
|
|
_psql_rc=$?
|
|
printf '%s\n' "${_psql_rc}" > "${_status_file}"
|
|
} 2>&1 | redact_stream "${RDS_MASTER_PASSWORD}"
|
|
|
|
# Every way of not recovering psql's status is a failed bootstrap rather
|
|
# than a silent success, and each says so in its own words: blaming psql for
|
|
# a status file that could not be written would send an operator looking at
|
|
# a PostgreSQL that did exactly what it was asked.
|
|
#
|
|
# A torn write leaves the file created but empty, so the value is checked
|
|
# before it is returned: `return ""` is fatal in the guest's ash, and would
|
|
# kill rds-init here — past the sweep, before the diagnostics.
|
|
_psql_status=1
|
|
if [ -f "${_status_file}" ]; then
|
|
_psql_status=$(cat "${_status_file}" 2>/dev/null) || _psql_status=""
|
|
case "${_psql_status}" in
|
|
'' | *[!0-9]*)
|
|
echo "[rds-init] ERROR: psql ran but its exit status could not be read back from ${_status_file} (got '${_psql_status}'); treating the bootstrap as failed" >&2
|
|
_psql_status=1
|
|
;;
|
|
esac
|
|
rm -f "${_status_file}"
|
|
else
|
|
echo "[rds-init] ERROR: psql ran but its exit status was never recorded at ${_status_file}; treating the bootstrap as failed" >&2
|
|
fi
|
|
return "${_psql_status}"
|
|
}
|
|
|
|
# Runs on a freshly initialised cluster only.
|
|
bootstrap_master() {
|
|
log "starting bootstrap server on ${BOOTSTRAP_DIR}"
|
|
start_bootstrap_server
|
|
|
|
_rc=0
|
|
apply_master_role || _rc=$?
|
|
if [ "${_rc}" -eq 0 ] && [ -n "${RDS_DB_NAME}" ]; then
|
|
create_database || _rc=$?
|
|
fi
|
|
if [ "${_rc}" -eq 0 ]; then
|
|
create_platform_extensions || _rc=$?
|
|
fi
|
|
|
|
# `set -e` is suspended for a function called from an AND-OR list, so the
|
|
# stop status is captured rather than propagated. A failed stop must also not
|
|
# abort before the bootstrap check below, which names the real cause.
|
|
_stop_rc=0
|
|
stop_bootstrap_server || _stop_rc=$?
|
|
rm -f "${BOOTSTRAP_DIR}/pg_hba.conf"
|
|
|
|
# The datadir is swept by the traps, which also cover the steps here that
|
|
# abort under `set -e` before this check is ever reached.
|
|
if [ "${_rc}" -ne 0 ]; then
|
|
fail "bootstrapping the master role/database failed (rc=${_rc})"
|
|
fi
|
|
|
|
# Proof written while the traps are still armed, so a receipt that cannot be
|
|
# made durable clears the datadir instead of leaving one it cannot vouch for.
|
|
write_completion_receipt
|
|
|
|
# The master role exists from here, so the datadir is complete and nothing
|
|
# may clear it — including on the stop failure reported below.
|
|
mark_bootstrap_complete
|
|
|
|
# A postmaster still holding the datadir blocks the postgresql service for
|
|
# the life of the boot, so this is a failure and not a warning.
|
|
if [ "${_stop_rc}" -ne 0 ]; then
|
|
fail "the bootstrap server did not stop (rc=${_stop_rc}) and still holds ${PGDATA}; the postgresql service cannot start until it is gone"
|
|
fi
|
|
}
|
|
|
|
# The value of one receipt key. Never sourced: the receipt is on the data volume,
|
|
# so a restored volume can carry one written by another instance, and executing
|
|
# it would run whatever that instance's control plane put there.
|
|
receipt_value() {
|
|
[ -f "${RECEIPT_FILE}" ] || return 0
|
|
sed -n "s/^$1=//p" "${RECEIPT_FILE}" 2>/dev/null | head -n 1
|
|
}
|
|
|
|
# Written between the master role and the retirement of the traps, so a receipt
|
|
# that cannot be made durable still clears the datadir rather than leaving a
|
|
# bootstrapped one nobody can prove.
|
|
#
|
|
# Fatal on failure, the mirror image of the sentinel removal below: a healthy
|
|
# engine whose receipt never landed would come up available while the agent
|
|
# blocked forever on a receipt that will never appear, leaving a working database
|
|
# that can never accept a password change or a parameter reload.
|
|
write_completion_receipt() {
|
|
install -d -m 0700 -o "${PG_USER}" -g "${PG_USER}" "${RECEIPT_DIR}" ||
|
|
fail "the master role exists but the bootstrap receipt directory ${RECEIPT_DIR} could not be created"
|
|
|
|
_tmp="${RECEIPT_FILE}.new"
|
|
{
|
|
echo "# Written by rds-init once the master role was applied. Holds no secrets."
|
|
echo "RDS_RECEIPT_PAYLOAD_ID=${RDS_PAYLOAD_ID}"
|
|
echo "RDS_RECEIPT_DB_INSTANCE_IDENTIFIER=${RDS_DB_INSTANCE_IDENTIFIER}"
|
|
echo "RDS_RECEIPT_VM_GENERATION=${RDS_VM_GENERATION}"
|
|
echo "RDS_RECEIPT_COMPLETED_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
|
} > "${_tmp}" ||
|
|
fail "the master role exists but the bootstrap receipt ${RECEIPT_FILE} could not be written"
|
|
chown "${PG_USER}:${PG_USER}" "${_tmp}"
|
|
chmod 0600 "${_tmp}"
|
|
|
|
# A receipt still in the page cache is lost by exactly the crash it exists to
|
|
# survive, so it is on disk before it is renamed into place.
|
|
sync "${_tmp}" 2>/dev/null || sync
|
|
mv -f "${_tmp}" "${RECEIPT_FILE}" ||
|
|
fail "the master role exists but the bootstrap receipt ${RECEIPT_FILE} could not be installed"
|
|
sync "${RECEIPT_DIR}" 2>/dev/null || sync
|
|
}
|
|
|
|
# PG_VERSION with no sentinel and no receipt is ambiguous from the datadir alone:
|
|
# either a legacy instance that bootstrapped before receipts existed, which must
|
|
# attach, or one that applied the master role and crashed before the receipt was
|
|
# durable, which must not. Only the control plane's view of whether the payload
|
|
# is still staged separates the two, and that is what RDS_BOOTSTRAP_PENDING says.
|
|
require_bootstrap_receipt() {
|
|
[ "${RDS_BOOTSTRAP_PENDING}" = "1" ] || return 0
|
|
if [ -n "${RDS_PAYLOAD_ID}" ] &&
|
|
[ "$(receipt_value RDS_RECEIPT_PAYLOAD_ID)" = "${RDS_PAYLOAD_ID}" ] &&
|
|
[ "$(receipt_value RDS_RECEIPT_DB_INSTANCE_IDENTIFIER)" = "${RDS_DB_INSTANCE_IDENTIFIER}" ]; then
|
|
return 0
|
|
fi
|
|
fail "the control plane has no record of a completed bootstrap for this datadir; refusing to attach to a datadir whose master role cannot be proven. Recover the volume out of band, or delete and recreate the instance."
|
|
}
|
|
|
|
# Retires the guards armed at initdb, in an order where no signal in between can
|
|
# clear a datadir that now carries a usable master role.
|
|
mark_bootstrap_complete() {
|
|
_bootstrapped=1
|
|
rm -f "${BOOTSTRAP_SENTINEL}" ||
|
|
fail "the master role exists but ${BOOTSTRAP_SENTINEL} could not be removed; the next boot will refuse to start until it is deleted"
|
|
trap - EXIT INT TERM HUP
|
|
}
|
|
|
|
# The master role is administrative but deliberately not a PostgreSQL SUPERUSER,
|
|
# as on AWS: a superuser reaches outside the database entirely — COPY FROM
|
|
# PROGRAM, pg_read_file/pg_write_file, untrusted languages — so master
|
|
# credentials would be a shell in the DB VM rather than a database privilege.
|
|
# The cluster's own superuser stays the bootstrap role, which only rds-init and
|
|
# rds-agent reach, over the unix socket under peer auth.
|
|
apply_master_role() {
|
|
log "creating the master role ${RDS_MASTER_USERNAME}"
|
|
# PG16 hardened CREATEROLE: it can neither grant itself SUPERUSER nor take
|
|
# membership in a predefined role it does not administer, so CREATEROLE here
|
|
# is not a route back to the privileges above.
|
|
psql_bootstrap <<'SQL'
|
|
\getenv master RDS_MASTER_USERNAME
|
|
\getenv password RDS_MASTER_PASSWORD
|
|
\getenv grp RDS_GROUP_ROLE
|
|
SELECT NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = :'grp') AS create_group \gset
|
|
\if :create_group
|
|
CREATE ROLE :"grp" NOLOGIN;
|
|
\endif
|
|
GRANT pg_monitor, pg_signal_backend, pg_checkpoint TO :"grp";
|
|
SELECT NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = :'master') AS create_role \gset
|
|
\if :create_role
|
|
CREATE ROLE :"master" WITH LOGIN NOSUPERUSER CREATEDB CREATEROLE;
|
|
\endif
|
|
ALTER ROLE :"master" WITH LOGIN NOSUPERUSER CREATEDB CREATEROLE PASSWORD :'password';
|
|
GRANT :"grp" TO :"master" WITH ADMIN OPTION;
|
|
SQL
|
|
}
|
|
|
|
# As on AWS, an instance created without a DBName gets no initial database.
|
|
create_database() {
|
|
log "creating database ${RDS_DB_NAME}"
|
|
psql_bootstrap <<'SQL'
|
|
\getenv master RDS_MASTER_USERNAME
|
|
\getenv dbname RDS_DB_NAME
|
|
SELECT NOT EXISTS (SELECT 1 FROM pg_database WHERE datname = :'dbname') AS create_db \gset
|
|
\if :create_db
|
|
CREATE DATABASE :"dbname" OWNER :"master";
|
|
\endif
|
|
SQL
|
|
}
|
|
|
|
# pgvector is baked in but left uninstalled so tenant databases stay free of it.
|
|
# The appliance master is NOSUPERUSER, so the bootstrap superuser installs the
|
|
# extension into a read-only extensions schema and grants the master the database
|
|
# and role privileges its daemon needs to create a per-account schema and role.
|
|
# Scoped to the appliance master alone; tenant masters skip this entirely.
|
|
create_platform_extensions() {
|
|
[ "${RDS_IS_OCHRE_APPLIANCE:-0}" = "1" ] || return 0
|
|
log "configuring pgvector and platform privileges for the Ochre vector appliance"
|
|
psql_bootstrap <<SQL
|
|
CREATE SCHEMA IF NOT EXISTS extensions;
|
|
CREATE EXTENSION IF NOT EXISTS vector SCHEMA extensions;
|
|
GRANT USAGE ON SCHEMA extensions TO PUBLIC;
|
|
GRANT CREATE ON DATABASE postgres TO "${OCHRE_APPLIANCE_MASTER}";
|
|
ALTER ROLE "${OCHRE_APPLIANCE_MASTER}" SET createrole_self_grant = 'set, inherit';
|
|
SQL
|
|
}
|
|
|
|
# Drops the master password from the handoff once applied, so it is not left
|
|
# readable for the life of the boot. The rest is tmpfs and goes at reboot.
|
|
consume_secrets() {
|
|
[ -f "${ENV_FILE}" ] || return 0
|
|
_tmp="${ENV_FILE}.new"
|
|
install -m 0600 /dev/null "${_tmp}"
|
|
grep -v '^RDS_MASTER_PASSWORD=' "${ENV_FILE}" >> "${_tmp}" || true
|
|
mv -f "${_tmp}" "${ENV_FILE}"
|
|
}
|
|
|
|
load_handoff
|
|
wait_for_listen_address
|
|
require_data_volume
|
|
require_engine_stamp
|
|
|
|
if [ -s "${PGDATA}/PG_VERSION" ]; then
|
|
# The engine would start cleanly on such a datadir and pass every health
|
|
# probe with no master role to log in as. The sentinel says the bootstrap did
|
|
# not finish, not that the volume is empty, so nothing here clears it.
|
|
if [ -e "${BOOTSTRAP_SENTINEL}" ]; then
|
|
fail "datadir ${PGDATA} carries ${BOOTSTRAP_SENTINEL}: an earlier bootstrap was interrupted before the master role existed; recover the data volume out of band rather than starting the engine"
|
|
fi
|
|
require_bootstrap_receipt
|
|
log "datadir ${PGDATA} is already initialised — attaching"
|
|
install_runtime_config
|
|
else
|
|
initialise_cluster
|
|
install_runtime_config
|
|
bootstrap_master
|
|
fi
|
|
|
|
consume_secrets
|
|
log "bootstrap complete; handing off to the postgresql service"
|