gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-11 23:56:53 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(targets): harden Postgres/MySQL SQL backends (pool timeouts, error classification, idempotency) (#4500)

Browse Source 
fix(targets): harden Postgres/MySQL SQL backends (pools, error class, idempotency)

Postgres (postgres.rs):
- Add deadpool wait/create/recycle timeouts + tokio-postgres connect_timeout,
  and wrap every pool.get() in a Tokio hard-limit timeout so an unreachable
  broker/DB can no longer block send_body/probe_table/is_active forever;
  checkout timeouts map to TargetError::Timeout to trigger store replay.
- namespace format now DELETEs the row on s3:ObjectRemoved:* events instead of
  UPSERTing, keeping the table consistent with the object lifecycle.
- map_pg_error: SQLSTATE class 40 (serialization_failure 40001,
  deadlock_detected 40P01, transaction rollback) is now transient (Timeout,
  retryable) instead of a permanent Request. Extracted map_pg_sqlstate for
  unit-testable classification.

MySQL (mysql.rs):
- Wrap pool.get_conn() in a Tokio checkout timeout across insert/init/liveness
  paths so an unreachable server cannot block the delivery thread.
- Add an event_id idempotency key: tables created by the target now carry an
  event_id VARCHAR(255) PRIMARY KEY, inserts use ON DUPLICATE KEY UPDATE, and
  replays reuse the stable store key so a lost-ack replay no longer appends a
  duplicate audit row. Legacy two-column tables are detected and fall back to
  the non-idempotent insert with a warning (backward compatible).
- redact_mysql_dsn splits on the last '@' so a password containing '@' no
  longer leaks its tail into Debug output.
- Disconnect the stale pool before dropping it on inline TLS reload instead of
  leaking its connections.
- Cache TLS file mtimes and only recompute the inline fingerprint when a cert
  file changes, avoiding a 3-file read+hash on every checkout.

Adds unit tests for SQLSTATE classification, namespace delete SQL, removal-event
detection, DSN redaction with '@' in the password, and the MySQL insert/DDL
builders.

Relates to rustfs/backlog#976
Relates to rustfs/backlog#973
Relates to rustfs/backlog#983

Co-authored-by: heihutu <heihutu@gmail.com>
This commit is contained in:
houseme
2026-07-09 01:00:44 +08:00
committed by GitHub
parent 73a30178f5
commit e008cc5dae
2 changed files with 404 additions and 59 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/targets/src/target/postgres.rs
+148 -22
View File
@@ -42,12 +42,14 @@ use crate::{
},
};
use async_trait::async_trait;
use deadpool_postgres::{Manager, ManagerConfig, Pool, RecyclingMethod};
use deadpool_postgres::{Client as PooledClient, Manager, ManagerConfig, Pool, RecyclingMethod, Runtime, Timeouts};
use rustfs_config::{POSTGRES_DSN_STRING, POSTGRES_TLS_CA, POSTGRES_TLS_CLIENT_CERT, POSTGRES_TLS_CLIENT_KEY};
use rustfs_s3_types::EventName;
use rustfs_tls_runtime::{load_certs, load_private_key};
use std::fmt;
use std::path::Path;
use std::sync::Arc;
use std::time::Duration;
use tokio_postgres::Config;
use tokio_postgres_rustls::MakeRustlsConnect;
use tracing::{info, instrument, warn};
@@ -56,6 +58,44 @@ use uuid::Uuid;
const TARGET_LOG_KEY_FIELD: &str = "Key";
/// Bounds the underlying TCP connect + startup handshake for a new backend
/// connection so an unreachable server cannot block a pool slot forever.
const POSTGRES_CONNECT_TIMEOUT: Duration = Duration::from_secs(10);
/// Maximum time `pool.get()` waits for a free slot before returning a timeout.
const POSTGRES_POOL_WAIT_TIMEOUT: Duration = Duration::from_secs(15);
/// Maximum time to create a brand-new pooled connection.
const POSTGRES_POOL_CREATE_TIMEOUT: Duration = Duration::from_secs(15);
/// Maximum time to recycle (health-check) an idle pooled connection.
const POSTGRES_POOL_RECYCLE_TIMEOUT: Duration = Duration::from_secs(10);
/// Absolute ceiling on a single checkout, wrapping `pool.get()` in a Tokio
/// timeout as a belt-and-suspenders guard on top of the deadpool timeouts.
const POSTGRES_POOL_CHECKOUT_HARD_LIMIT: Duration = Duration::from_secs(20);
/// Returns `true` for any `s3:ObjectRemoved:*` event.
///
/// Used by the `namespace` format so that object deletions remove the row
/// instead of leaving stale state behind via an UPSERT.
fn is_object_removed_event(event: &EventName) -> bool {
event.as_str().starts_with("s3:ObjectRemoved")
}
/// Checks out a client from the pool, wrapping `pool.get()` in a Tokio timeout.
///
/// The deadpool wait/create timeouts already bound the checkout, but the outer
/// timeout guarantees a hard ceiling even if a lower layer misbehaves. Any
/// timeout maps to `TargetError::Timeout`, which is a connectivity error so the
/// queue store retains the payload for replay.
async fn checkout_client(pool: &Pool, context: &str) -> Result<PooledClient, TargetError> {
match tokio::time::timeout(POSTGRES_POOL_CHECKOUT_HARD_LIMIT, pool.get()).await {
Ok(Ok(client)) => Ok(client),
Ok(Err(e)) => Err(map_pool_error(e, context)),
Err(_) => Err(TargetError::Timeout(format!(
"{context}: pool checkout exceeded {}s hard limit",
POSTGRES_POOL_CHECKOUT_HARD_LIMIT.as_secs()
))),
}
}
/// Output format selection for the PostgreSQL target.
///
/// - `Namespace`: single-row UPSERT per object key (MinIO `namespace` style).
@@ -388,6 +428,13 @@ pub fn namespace_upsert_sql(schema: &str, table: &str) -> String {
)
}
/// SQL for the `namespace` format on object removal. Deletes the row keyed on
/// the object key so the `namespace` table stays consistent with the object
/// lifecycle instead of retaining stale state after a delete.
pub fn namespace_delete_sql(schema: &str, table: &str) -> String {
format!("DELETE FROM {} WHERE key = $1", qualified_table(schema, table))
}
/// SQL for the `access` format. Append-only with `event_id` as PK so that
/// store-replay scenarios silently skip duplicates while distinct events still
/// land as separate rows.
@@ -472,6 +519,9 @@ pub fn build_pool(args: &PostgresArgs) -> Result<Pool, TargetError> {
.port(parsed.port)
.user(&parsed.user)
.dbname(&parsed.database)
// Bound the TCP connect + startup handshake so an unreachable backend
// cannot block a pool slot indefinitely.
.connect_timeout(POSTGRES_CONNECT_TIMEOUT)
.options(format!("-c search_path={}", parsed.schema));
if let Some(password) = parsed.password.as_deref()
&& !password.is_empty()
@@ -491,30 +541,59 @@ pub fn build_pool(args: &PostgresArgs) -> Result<Pool, TargetError> {
Manager::from_config(pg_config, tokio_postgres::NoTls, manager_config)
};
// Explicit wait/create/recycle timeouts guarantee that `pool.get()` always
// returns within a bounded time when the broker/DB is unreachable, instead
// of blocking the delivery thread forever. A Tokio runtime is required for
// deadpool to honor these timeouts.
Pool::builder(manager)
.runtime(Runtime::Tokio1)
.timeouts(Timeouts {
wait: Some(POSTGRES_POOL_WAIT_TIMEOUT),
create: Some(POSTGRES_POOL_CREATE_TIMEOUT),
recycle: Some(POSTGRES_POOL_RECYCLE_TIMEOUT),
})
.build()
.map_err(|e| TargetError::Configuration(format!("failed to build PostgreSQL pool: {e}")))
}
/// Classifies a PostgreSQL SQLSTATE code into the proper `TargetError` variant.
///
/// Split out from [`map_pg_error`] so the SQLSTATE-to-variant mapping can be
/// unit-tested without constructing an opaque `tokio_postgres::Error`.
///
/// Classification is by SQLSTATE class (first two characters):
/// - `08` connection exception → `NotConnected` (retry, keep in store).
/// - `28` invalid authorization → `Authentication` (permanent, surfaced).
/// - `23`/`42` integrity/syntax → `Configuration` (permanent, surfaced).
/// - `40` transaction rollback (`40001` serialization_failure,
/// `40P01` deadlock_detected, …) → `Timeout`, a transient/retryable error:
/// the transaction should be retried rather than dropped.
/// - anything else → `Request` (treated as permanent/ambiguous).
fn map_pg_sqlstate(code: &str, detail: &str) -> TargetError {
match code.get(..2).unwrap_or("") {
"08" => TargetError::NotConnected,
"28" => TargetError::Authentication(detail.to_string()),
"23" | "42" => TargetError::Configuration(detail.to_string()),
"40" => TargetError::Timeout(detail.to_string()),
_ => TargetError::Request(detail.to_string()),
}
}
/// Maps a `tokio_postgres::Error` to the proper `TargetError` variant.
///
/// Connection-class errors (SQLSTATE 08, closed connection, IO) become
/// `NotConnected` so the queue store retains the payload for replay.
/// Schema and constraint problems (SQLSTATE 23, 42) become `Configuration`
/// so they are surfaced to the operator without endless retry.
/// Transaction-rollback errors (SQLSTATE class 40, e.g. serialization failure
/// or deadlock) become `Timeout` so they are retried transiently.
pub fn map_pg_error(err: &tokio_postgres::Error, context: &str) -> TargetError {
if err.is_closed() {
return TargetError::NotConnected;
}
if let Some(db_err) = err.as_db_error() {
let class = db_err.code().code().get(..2).unwrap_or("");
return match class {
"08" => TargetError::NotConnected,
"28" => TargetError::Authentication(format!("{context}: {db_err}")),
"23" | "42" => TargetError::Configuration(format!("{context}: {db_err}")),
"40" => TargetError::Request(format!("{context}: {db_err}")),
_ => TargetError::Request(format!("{context}: {db_err}")),
};
let detail = format!("{context}: {db_err}");
return map_pg_sqlstate(db_err.code().code(), &detail);
}
TargetError::NotConnected
}
@@ -564,6 +643,7 @@ where
/// otherwise the inline fingerprint path is used as a fallback.
tls_adapter: Option<TlsReloadAdapter<Pool>>,
namespace_sql: String,
namespace_delete_sql: String,
access_sql: String,
store: Option<Box<dyn Store<QueuedPayload, Error = StoreError, Key = Key> + Send + Sync>>,
delivery_counters: Arc<TargetDeliveryCounters>,
@@ -582,6 +662,7 @@ where
tls_state: Arc::clone(&self.tls_state),
tls_adapter: self.tls_adapter.clone(),
namespace_sql: self.namespace_sql.clone(),
namespace_delete_sql: self.namespace_delete_sql.clone(),
access_sql: self.access_sql.clone(),
store: self.store.as_ref().map(|s| s.boxed_clone()),
delivery_counters: Arc::clone(&self.delivery_counters),
@@ -607,6 +688,7 @@ where
Ok(Self {
id: target_id,
namespace_sql: namespace_upsert_sql(&args.schema, &args.table),
namespace_delete_sql: namespace_delete_sql(&args.schema, &args.table),
access_sql: access_insert_sql(&args.schema, &args.table),
args,
pool: Arc::new(parking_lot::Mutex::new(pool)),
@@ -641,10 +723,7 @@ where
}
let pool = self.pool.lock().clone();
let client = pool
.get()
.await
.map_err(|e| map_pool_error(e, "PostgreSQL pool checkout failed"))?;
let client = checkout_client(&pool, "PostgreSQL pool checkout failed").await?;
let payload: serde_json::Value =
serde_json::from_slice(body).map_err(|e| TargetError::Serialization(format!("Failed to parse JSON payload: {e}")))?;
@@ -652,6 +731,12 @@ where
let key = resolve_payload_key(&payload, meta);
let result = match self.args.format {
// For the single-row `namespace` format, an object removal must
// delete the row rather than UPSERT it, otherwise stale state
// lingers in the table after the object is gone.
PostgresFormat::Namespace if is_object_removed_event(&meta.event_name) => {
client.execute(&self.namespace_delete_sql, &[&key]).await
}
PostgresFormat::Namespace => client.execute(&self.namespace_sql, &[&key, &payload]).await,
PostgresFormat::Access => {
let event_name_str = meta.event_name.to_string();
@@ -675,10 +760,7 @@ where
/// configured: events buffer in the store until the schema is fixed.
async fn probe_table(&self) -> Result<(), TargetError> {
let pool = self.pool.lock().clone();
let client = pool
.get()
.await
.map_err(|e| map_pool_error(e, "PostgreSQL pool checkout failed during init probe"))?;
let client = checkout_client(&pool, "PostgreSQL pool checkout failed during init probe").await?;
let sql = table_probe_sql(&self.args.schema, &self.args.table);
client
.execute(sql.as_str(), &[])
@@ -737,12 +819,9 @@ where
return Ok(false);
}
match tokio::time::timeout(std::time::Duration::from_secs(10), async {
match tokio::time::timeout(Duration::from_secs(10), async {
let pool = self.pool.lock().clone();
let client = pool
.get()
.await
.map_err(|e| map_pool_error(e, "PostgreSQL pool checkout failed"))?;
let client = checkout_client(&pool, "PostgreSQL pool checkout failed").await?;
client
.execute("SELECT 1", &[])
.await
@@ -1091,6 +1170,53 @@ mod tests {
assert!(sql.contains("$4::jsonb"));
}
#[test]
fn namespace_delete_targets_row_by_key() {
let sql = namespace_delete_sql("public", "events");
assert!(sql.starts_with("DELETE FROM"));
assert!(sql.contains(r#""public"."events""#));
assert!(sql.contains("WHERE key = $1"));
}
#[test]
fn is_object_removed_event_matches_all_removed_variants() {
assert!(is_object_removed_event(&EventName::ObjectRemovedDelete));
assert!(is_object_removed_event(&EventName::ObjectRemovedDeleteMarkerCreated));
assert!(is_object_removed_event(&EventName::ObjectRemovedDeleteAllVersions));
assert!(is_object_removed_event(&EventName::ObjectRemovedAll));
assert!(!is_object_removed_event(&EventName::ObjectCreatedPut));
assert!(!is_object_removed_event(&EventName::ObjectAccessedGet));
}
#[test]
fn map_pg_sqlstate_classifies_transaction_rollback_as_transient() {
// 40001 serialization_failure and 40P01 deadlock_detected are transient
// and must be retried, not dropped as permanent failures.
assert!(matches!(map_pg_sqlstate("40001", "ctx: serialization"), TargetError::Timeout(_)));
assert!(matches!(map_pg_sqlstate("40P01", "ctx: deadlock"), TargetError::Timeout(_)));
assert!(matches!(map_pg_sqlstate("40000", "ctx: rollback"), TargetError::Timeout(_)));
}
#[test]
fn map_pg_sqlstate_classifies_connection_and_permanent_errors() {
assert!(matches!(map_pg_sqlstate("08006", "ctx"), TargetError::NotConnected));
assert!(matches!(map_pg_sqlstate("08001", "ctx"), TargetError::NotConnected));
assert!(matches!(map_pg_sqlstate("28P01", "ctx: auth"), TargetError::Authentication(_)));
assert!(matches!(map_pg_sqlstate("23505", "ctx: unique"), TargetError::Configuration(_)));
assert!(matches!(map_pg_sqlstate("42P01", "ctx: undefined_table"), TargetError::Configuration(_)));
// Unknown class stays permanent (ambiguous → surfaced as Request).
assert!(matches!(map_pg_sqlstate("22001", "ctx: data"), TargetError::Request(_)));
assert!(matches!(map_pg_sqlstate("", "ctx: empty"), TargetError::Request(_)));
}
#[test]
fn transient_pg_errors_are_connectivity_errors() {
// Transaction-rollback errors must be treated as connectivity errors so
// the queue store retains the payload for replay instead of dropping it.
assert!(crate::target::is_connectivity_error(&map_pg_sqlstate("40001", "ctx")));
assert!(crate::target::is_connectivity_error(&map_pg_sqlstate("40P01", "ctx")));
}
#[test]
fn table_probe_does_not_select_rows() {
let sql = table_probe_sql("public", "events");