mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-29 16:37:07 +00:00
feat(table-catalog): add distributed maintenance scheduling (#4257)
* feat(table-catalog): add distributed maintenance scheduling * fix(table-catalog): address scheduler review feedback --------- Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com> Co-authored-by: houseme <housemecn@gmail.com>
This commit is contained in:
@@ -86,14 +86,15 @@ catalog extension.
|
|||||||
| Snapshot expiration planning | Supported | Produces expiration plans with retained and candidate snapshots. |
|
| Snapshot expiration planning | Supported | Produces expiration plans with retained and candidate snapshots. |
|
||||||
| Snapshot expiration commit | Preview / controlled | Can manually commit safe snapshot expiration through the catalog. Stale plans fail closed. |
|
| Snapshot expiration commit | Preview / controlled | Can manually commit safe snapshot expiration through the catalog. Stale plans fail closed. |
|
||||||
| Manifest/data/delete reachability cleanup | Supported | Reads manifest-list and manifest Avro references, reports reachable objects, and deletes only unreferenced table objects that pass the safety window. |
|
| Manifest/data/delete reachability cleanup | Supported | Reads manifest-list and manifest Avro references, reports reachable objects, and deletes only unreferenced table objects that pass the safety window. |
|
||||||
| Maintenance worker run endpoint | Preview / controlled | Supports run-once execution, current-job backpressure, retry deferral, lease expiry recovery, and heartbeat updates. |
|
| Maintenance scheduler run endpoint | Preview / controlled | Lets an external scheduler durably queue one maintenance job per table, reuse an active queued job, and recover expired queued leases before requeuing. |
|
||||||
| Maintenance scheduler guardrails | Preview / controlled | Exposes disabled, paused, ready, active-job backpressure, retry deferral, quarantine boundary, recommended actions, and recent maintenance job audit timeline state for external schedulers and operators. |
|
| Maintenance worker run endpoint | Preview / controlled | Supports queued-job claim, run-once execution, current-job backpressure, retry deferral, lease expiry recovery, and heartbeat updates. |
|
||||||
|
| Maintenance scheduler guardrails | Preview / controlled | Exposes disabled, paused, ready, queued-job handoff, active-job backpressure, retry deferral, quarantine boundary, recommended actions, and recent maintenance job audit timeline state for external schedulers and operators. |
|
||||||
| Maintenance audit events | Preview / controlled | Job reports and scheduler job summaries include structured audit events for planning, worker transitions, heartbeats, lease expiry recovery, and mutating quarantine operations. |
|
| Maintenance audit events | Preview / controlled | Job reports and scheduler job summaries include structured audit events for planning, worker transitions, heartbeats, lease expiry recovery, and mutating quarantine operations. |
|
||||||
| Maintenance quarantine operations | Preview / controlled | Lets operators inspect, release, retry, or abandon the current quarantined maintenance job without moving the table pointer. |
|
| Maintenance quarantine operations | Preview / controlled | Lets operators inspect, release, retry, or abandon the current quarantined maintenance job without moving the table pointer. |
|
||||||
| Compaction planning | Preview / controlled | Plans partition-local and sort-order-local binpack candidates for Parquet files and does not mix data files from different partition directories or sort orders in one rewrite group. |
|
| Compaction planning | Preview / controlled | Plans partition-local and sort-order-local binpack candidates for Parquet files and does not mix data files from different partition directories or sort orders in one rewrite group. |
|
||||||
| Delete-file or row-level compaction planning | Preview / controlled | Manifests with position or equality delete files produce machine-readable row-level planning and force the compaction report into manual review before any rewrite can run. |
|
| Delete-file or row-level compaction planning | Preview / controlled | Manifests with position or equality delete files produce machine-readable row-level planning and force the compaction report into manual review before any rewrite can run. |
|
||||||
| Compaction commit | Preview / controlled | Can commit a safe partition-local Parquet rewrite through the catalog while preserving Iceberg data file sort order IDs in the rewritten manifest. |
|
| Compaction commit | Preview / controlled | Can commit a safe partition-local Parquet rewrite through the catalog while preserving Iceberg data file sort order IDs in the rewritten manifest. |
|
||||||
| Built-in periodic scheduler | Not claimed | Operators can trigger worker runs, but continuous in-process scheduling is not claimed. |
|
| Built-in periodic scheduler | Not claimed | Operators can trigger scheduler and worker ticks, but continuous in-process scheduling is not claimed. |
|
||||||
| Delete-file or row-level compaction execution | Not claimed | RustFS does not rewrite delete files or execute row-level compaction; those cases remain manual-review maintenance items. |
|
| Delete-file or row-level compaction execution | Not claimed | RustFS does not rewrite delete files or execute row-level compaction; those cases remain manual-review maintenance items. |
|
||||||
|
|
||||||
## Recovery And Strong Backing Matrix
|
## Recovery And Strong Backing Matrix
|
||||||
@@ -162,7 +163,7 @@ RustFS does not currently claim:
|
|||||||
- full MinIO AIStor Tables private extension parity
|
- full MinIO AIStor Tables private extension parity
|
||||||
- full Cloudflare R2 Data Catalog interoperability
|
- full Cloudflare R2 Data Catalog interoperability
|
||||||
- full Alibaba OSS Tables interoperability
|
- full Alibaba OSS Tables interoperability
|
||||||
- built-in periodic maintenance scheduling; external schedulers can inspect scheduler guardrails, but RustFS does not claim a continuous in-process scheduler
|
- built-in periodic maintenance scheduling; external schedulers can queue maintenance jobs and workers can claim them, but RustFS does not claim a continuous in-process scheduler
|
||||||
- active-active multi-region table writes
|
- active-active multi-region table writes
|
||||||
- multi-table transactions
|
- multi-table transactions
|
||||||
- no-long-term-data-credential table bootstrap
|
- no-long-term-data-credential table bootstrap
|
||||||
|
|||||||
@@ -74,6 +74,7 @@ const TABLE_CATALOG_NAMESPACE_RESOURCE_ROOT: &str = "namespaces";
|
|||||||
const TABLE_CATALOG_TABLE_RESOURCE_ROOT: &str = "tables";
|
const TABLE_CATALOG_TABLE_RESOURCE_ROOT: &str = "tables";
|
||||||
const TABLE_CATALOG_VIEW_RESOURCE_ROOT: &str = "views";
|
const TABLE_CATALOG_VIEW_RESOURCE_ROOT: &str = "views";
|
||||||
const TABLE_CATALOG_ADMIN_OPERATION_SLOW_LOG_THRESHOLD: StdDuration = StdDuration::from_secs(2);
|
const TABLE_CATALOG_ADMIN_OPERATION_SLOW_LOG_THRESHOLD: StdDuration = StdDuration::from_secs(2);
|
||||||
|
const DEFAULT_TABLE_MAINTENANCE_SCHEDULER_ID: &str = "rustfs-maintenance-scheduler";
|
||||||
const DEFAULT_TABLE_MAINTENANCE_WORKER_ID: &str = "rustfs-maintenance-worker";
|
const DEFAULT_TABLE_MAINTENANCE_WORKER_ID: &str = "rustfs-maintenance-worker";
|
||||||
const EXTERNAL_CATALOG_BRIDGE_STATUS_UNCONFIGURED: &str = "bridge-unconfigured";
|
const EXTERNAL_CATALOG_BRIDGE_STATUS_UNCONFIGURED: &str = "bridge-unconfigured";
|
||||||
const EXTERNAL_CATALOG_BRIDGE_STATUS_CONFIGURED: &str = "bridge-configured";
|
const EXTERNAL_CATALOG_BRIDGE_STATUS_CONFIGURED: &str = "bridge-configured";
|
||||||
@@ -136,6 +137,7 @@ const TABLE_CATALOG_ENDPOINTS: &[&str] = &[
|
|||||||
"PUT /{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/config",
|
"PUT /{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/config",
|
||||||
"GET /{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/jobs/{job}",
|
"GET /{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/jobs/{job}",
|
||||||
"GET /{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/scheduler",
|
"GET /{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/scheduler",
|
||||||
|
"POST /{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/scheduler/run",
|
||||||
"POST /{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/worker/run",
|
"POST /{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/worker/run",
|
||||||
"POST /{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/jobs/{job}/heartbeat",
|
"POST /{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/jobs/{job}/heartbeat",
|
||||||
"POST /{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/jobs/{job}/quarantine",
|
"POST /{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/jobs/{job}/quarantine",
|
||||||
@@ -182,6 +184,7 @@ static GET_TABLE_MAINTENANCE_CONFIG_HANDLER: GetTableMaintenanceConfigHandler =
|
|||||||
static PUT_TABLE_MAINTENANCE_CONFIG_HANDLER: PutTableMaintenanceConfigHandler = PutTableMaintenanceConfigHandler {};
|
static PUT_TABLE_MAINTENANCE_CONFIG_HANDLER: PutTableMaintenanceConfigHandler = PutTableMaintenanceConfigHandler {};
|
||||||
static GET_TABLE_MAINTENANCE_JOB_HANDLER: GetTableMaintenanceJobHandler = GetTableMaintenanceJobHandler {};
|
static GET_TABLE_MAINTENANCE_JOB_HANDLER: GetTableMaintenanceJobHandler = GetTableMaintenanceJobHandler {};
|
||||||
static GET_TABLE_MAINTENANCE_SCHEDULER_HANDLER: GetTableMaintenanceSchedulerHandler = GetTableMaintenanceSchedulerHandler {};
|
static GET_TABLE_MAINTENANCE_SCHEDULER_HANDLER: GetTableMaintenanceSchedulerHandler = GetTableMaintenanceSchedulerHandler {};
|
||||||
|
static RUN_TABLE_MAINTENANCE_SCHEDULER_HANDLER: RunTableMaintenanceSchedulerHandler = RunTableMaintenanceSchedulerHandler {};
|
||||||
static RUN_TABLE_MAINTENANCE_WORKER_HANDLER: RunTableMaintenanceWorkerHandler = RunTableMaintenanceWorkerHandler {};
|
static RUN_TABLE_MAINTENANCE_WORKER_HANDLER: RunTableMaintenanceWorkerHandler = RunTableMaintenanceWorkerHandler {};
|
||||||
static HEARTBEAT_TABLE_MAINTENANCE_JOB_HANDLER: HeartbeatTableMaintenanceJobHandler = HeartbeatTableMaintenanceJobHandler {};
|
static HEARTBEAT_TABLE_MAINTENANCE_JOB_HANDLER: HeartbeatTableMaintenanceJobHandler = HeartbeatTableMaintenanceJobHandler {};
|
||||||
static TABLE_MAINTENANCE_QUARANTINE_HANDLER: TableMaintenanceQuarantineHandler = TableMaintenanceQuarantineHandler {};
|
static TABLE_MAINTENANCE_QUARANTINE_HANDLER: TableMaintenanceQuarantineHandler = TableMaintenanceQuarantineHandler {};
|
||||||
@@ -357,6 +360,19 @@ struct TableMetadataMaintenanceRequest {
|
|||||||
commit_compaction: bool,
|
commit_compaction: bool,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Default, Deserialize)]
|
||||||
|
#[serde(deny_unknown_fields)]
|
||||||
|
struct TableMaintenanceSchedulerRunRequest {
|
||||||
|
#[serde(default, rename = "scheduler-id")]
|
||||||
|
scheduler_id: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl TableMaintenanceSchedulerRunRequest {
|
||||||
|
fn scheduler_id(&self) -> &str {
|
||||||
|
self.scheduler_id.as_deref().unwrap_or(DEFAULT_TABLE_MAINTENANCE_SCHEDULER_ID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
#[derive(Debug, Deserialize)]
|
#[derive(Debug, Deserialize)]
|
||||||
#[serde(deny_unknown_fields)]
|
#[serde(deny_unknown_fields)]
|
||||||
struct TableMaintenanceWorkerRunRequest {
|
struct TableMaintenanceWorkerRunRequest {
|
||||||
@@ -951,6 +967,11 @@ fn register_table_catalog_prefix_routes(r: &mut S3Router<AdminOperation>, prefix
|
|||||||
format!("{prefix}/{{warehouse}}/namespaces/{{namespace}}/tables/{{table}}/maintenance/scheduler").as_str(),
|
format!("{prefix}/{{warehouse}}/namespaces/{{namespace}}/tables/{{table}}/maintenance/scheduler").as_str(),
|
||||||
AdminOperation(&GET_TABLE_MAINTENANCE_SCHEDULER_HANDLER),
|
AdminOperation(&GET_TABLE_MAINTENANCE_SCHEDULER_HANDLER),
|
||||||
)?;
|
)?;
|
||||||
|
r.insert(
|
||||||
|
Method::POST,
|
||||||
|
format!("{prefix}/{{warehouse}}/namespaces/{{namespace}}/tables/{{table}}/maintenance/scheduler/run").as_str(),
|
||||||
|
AdminOperation(&RUN_TABLE_MAINTENANCE_SCHEDULER_HANDLER),
|
||||||
|
)?;
|
||||||
r.insert(
|
r.insert(
|
||||||
Method::POST,
|
Method::POST,
|
||||||
format!("{prefix}/{{warehouse}}/namespaces/{{namespace}}/tables/{{table}}/maintenance/worker/run").as_str(),
|
format!("{prefix}/{{warehouse}}/namespaces/{{namespace}}/tables/{{table}}/maintenance/worker/run").as_str(),
|
||||||
@@ -5237,6 +5258,32 @@ impl Operation for GetTableMaintenanceSchedulerHandler {
|
|||||||
|
|
||||||
pub struct RunTableMaintenanceWorkerHandler {}
|
pub struct RunTableMaintenanceWorkerHandler {}
|
||||||
|
|
||||||
|
#[async_trait::async_trait]
|
||||||
|
impl Operation for RunTableMaintenanceSchedulerHandler {
|
||||||
|
async fn call(&self, req: S3Request<Body>, params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
|
||||||
|
let warehouse = warehouse_from_params(¶ms)?;
|
||||||
|
let namespace = namespace_from_params(¶ms)?;
|
||||||
|
let table = table_name_from_params(¶ms)?;
|
||||||
|
let resource = TableCatalogResource::table(&warehouse, &namespace, &table);
|
||||||
|
authorize_table_catalog_resource_request(&req, &resource, AdminAction::RunTableMaintenanceAction).await?;
|
||||||
|
ensure_table_bucket_enabled(&warehouse).await?;
|
||||||
|
let request = read_json_body_or_default::<TableMaintenanceSchedulerRunRequest>(req.input).await?;
|
||||||
|
let store = table_catalog_store()?;
|
||||||
|
let response = store
|
||||||
|
.run_table_maintenance_scheduler_once(
|
||||||
|
&warehouse,
|
||||||
|
&namespace.public_name(),
|
||||||
|
&table,
|
||||||
|
request.scheduler_id().to_string(),
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
.map_err(catalog_store_error)?;
|
||||||
|
build_json_response(StatusCode::OK, &response)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub struct RunTableMaintenanceSchedulerHandler {}
|
||||||
|
|
||||||
#[async_trait::async_trait]
|
#[async_trait::async_trait]
|
||||||
impl Operation for RunTableMaintenanceWorkerHandler {
|
impl Operation for RunTableMaintenanceWorkerHandler {
|
||||||
async fn call(&self, req: S3Request<Body>, params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
|
async fn call(&self, req: S3Request<Body>, params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
|
||||||
@@ -5698,6 +5745,7 @@ mod tests {
|
|||||||
("PutTableMaintenanceConfigHandler", "AdminAction::SetTableLifecycleAction"),
|
("PutTableMaintenanceConfigHandler", "AdminAction::SetTableLifecycleAction"),
|
||||||
("GetTableMaintenanceJobHandler", "AdminAction::GetTableLifecycleAction"),
|
("GetTableMaintenanceJobHandler", "AdminAction::GetTableLifecycleAction"),
|
||||||
("GetTableMaintenanceSchedulerHandler", "AdminAction::GetTableLifecycleAction"),
|
("GetTableMaintenanceSchedulerHandler", "AdminAction::GetTableLifecycleAction"),
|
||||||
|
("RunTableMaintenanceSchedulerHandler", "AdminAction::RunTableMaintenanceAction"),
|
||||||
("TableMaintenanceQuarantineHandler", "AdminAction::RunTableMaintenanceAction"),
|
("TableMaintenanceQuarantineHandler", "AdminAction::RunTableMaintenanceAction"),
|
||||||
("ExportTableCatalogHandler", "AdminAction::GetTableMetadataAction"),
|
("ExportTableCatalogHandler", "AdminAction::GetTableMetadataAction"),
|
||||||
("ImportTableCatalogHandler", "AdminAction::RegisterTableAction"),
|
("ImportTableCatalogHandler", "AdminAction::RegisterTableAction"),
|
||||||
@@ -5753,6 +5801,7 @@ mod tests {
|
|||||||
("PutTableMaintenanceConfigHandler", "AdminAction::SetTableLifecycleAction"),
|
("PutTableMaintenanceConfigHandler", "AdminAction::SetTableLifecycleAction"),
|
||||||
("GetTableMaintenanceJobHandler", "AdminAction::GetTableLifecycleAction"),
|
("GetTableMaintenanceJobHandler", "AdminAction::GetTableLifecycleAction"),
|
||||||
("GetTableMaintenanceSchedulerHandler", "AdminAction::GetTableLifecycleAction"),
|
("GetTableMaintenanceSchedulerHandler", "AdminAction::GetTableLifecycleAction"),
|
||||||
|
("RunTableMaintenanceSchedulerHandler", "AdminAction::RunTableMaintenanceAction"),
|
||||||
("TableMaintenanceQuarantineHandler", "AdminAction::RunTableMaintenanceAction"),
|
("TableMaintenanceQuarantineHandler", "AdminAction::RunTableMaintenanceAction"),
|
||||||
("ExportTableCatalogHandler", "AdminAction::GetTableMetadataAction"),
|
("ExportTableCatalogHandler", "AdminAction::GetTableMetadataAction"),
|
||||||
("ImportTableCatalogHandler", "AdminAction::RegisterTableAction"),
|
("ImportTableCatalogHandler", "AdminAction::RegisterTableAction"),
|
||||||
@@ -5810,6 +5859,7 @@ mod tests {
|
|||||||
"PutTableMaintenanceConfigHandler",
|
"PutTableMaintenanceConfigHandler",
|
||||||
"GetTableMaintenanceJobHandler",
|
"GetTableMaintenanceJobHandler",
|
||||||
"GetTableMaintenanceSchedulerHandler",
|
"GetTableMaintenanceSchedulerHandler",
|
||||||
|
"RunTableMaintenanceSchedulerHandler",
|
||||||
"RunTableMaintenanceWorkerHandler",
|
"RunTableMaintenanceWorkerHandler",
|
||||||
"HeartbeatTableMaintenanceJobHandler",
|
"HeartbeatTableMaintenanceJobHandler",
|
||||||
"TableMaintenanceQuarantineHandler",
|
"TableMaintenanceQuarantineHandler",
|
||||||
@@ -5920,6 +5970,7 @@ mod tests {
|
|||||||
let _: &PutTableMaintenanceConfigHandler = &PUT_TABLE_MAINTENANCE_CONFIG_HANDLER;
|
let _: &PutTableMaintenanceConfigHandler = &PUT_TABLE_MAINTENANCE_CONFIG_HANDLER;
|
||||||
let _: &GetTableMaintenanceJobHandler = &GET_TABLE_MAINTENANCE_JOB_HANDLER;
|
let _: &GetTableMaintenanceJobHandler = &GET_TABLE_MAINTENANCE_JOB_HANDLER;
|
||||||
let _: &GetTableMaintenanceSchedulerHandler = &GET_TABLE_MAINTENANCE_SCHEDULER_HANDLER;
|
let _: &GetTableMaintenanceSchedulerHandler = &GET_TABLE_MAINTENANCE_SCHEDULER_HANDLER;
|
||||||
|
let _: &RunTableMaintenanceSchedulerHandler = &RUN_TABLE_MAINTENANCE_SCHEDULER_HANDLER;
|
||||||
let _: &TableMaintenanceQuarantineHandler = &TABLE_MAINTENANCE_QUARANTINE_HANDLER;
|
let _: &TableMaintenanceQuarantineHandler = &TABLE_MAINTENANCE_QUARANTINE_HANDLER;
|
||||||
let _: &ExportTableCatalogHandler = &EXPORT_TABLE_CATALOG_HANDLER;
|
let _: &ExportTableCatalogHandler = &EXPORT_TABLE_CATALOG_HANDLER;
|
||||||
let _: &ImportTableCatalogHandler = &IMPORT_TABLE_CATALOG_HANDLER;
|
let _: &ImportTableCatalogHandler = &IMPORT_TABLE_CATALOG_HANDLER;
|
||||||
@@ -5959,6 +6010,7 @@ mod tests {
|
|||||||
assert_operation::<PutTableMaintenanceConfigHandler>();
|
assert_operation::<PutTableMaintenanceConfigHandler>();
|
||||||
assert_operation::<GetTableMaintenanceJobHandler>();
|
assert_operation::<GetTableMaintenanceJobHandler>();
|
||||||
assert_operation::<GetTableMaintenanceSchedulerHandler>();
|
assert_operation::<GetTableMaintenanceSchedulerHandler>();
|
||||||
|
assert_operation::<RunTableMaintenanceSchedulerHandler>();
|
||||||
assert_operation::<RunTableMaintenanceWorkerHandler>();
|
assert_operation::<RunTableMaintenanceWorkerHandler>();
|
||||||
assert_operation::<HeartbeatTableMaintenanceJobHandler>();
|
assert_operation::<HeartbeatTableMaintenanceJobHandler>();
|
||||||
assert_operation::<TableMaintenanceQuarantineHandler>();
|
assert_operation::<TableMaintenanceQuarantineHandler>();
|
||||||
@@ -6033,6 +6085,33 @@ mod tests {
|
|||||||
assert_eq!(compaction.max_rewrite_bytes_per_job, 10_737_418_240);
|
assert_eq!(compaction.max_rewrite_bytes_per_job, 10_737_418_240);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn table_maintenance_scheduler_run_request_uses_stable_default_scheduler_id() {
|
||||||
|
let request: TableMaintenanceSchedulerRunRequest =
|
||||||
|
serde_json::from_value(serde_json::json!({})).expect("scheduler run request should parse");
|
||||||
|
|
||||||
|
assert_eq!(request.scheduler_id(), "rustfs-maintenance-scheduler");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn table_maintenance_scheduler_run_request_empty_body_uses_default_scheduler_id() {
|
||||||
|
let request: TableMaintenanceSchedulerRunRequest = read_json_body_or_default(Body::empty())
|
||||||
|
.await
|
||||||
|
.expect("bodyless scheduler run should use the default scheduler id");
|
||||||
|
|
||||||
|
assert_eq!(request.scheduler_id(), "rustfs-maintenance-scheduler");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn table_maintenance_scheduler_run_request_accepts_scheduler_id() {
|
||||||
|
let request: TableMaintenanceSchedulerRunRequest = serde_json::from_value(serde_json::json!({
|
||||||
|
"scheduler-id": "scheduler-a"
|
||||||
|
}))
|
||||||
|
.expect("scheduler run request should parse scheduler id");
|
||||||
|
|
||||||
|
assert_eq!(request.scheduler_id(), "scheduler-a");
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn table_maintenance_worker_run_request_uses_stable_default_worker_id() {
|
fn table_maintenance_worker_run_request_uses_stable_default_worker_id() {
|
||||||
let request: TableMaintenanceWorkerRunRequest =
|
let request: TableMaintenanceWorkerRunRequest =
|
||||||
|
|||||||
@@ -835,6 +835,12 @@ pub const ADMIN_ROUTE_POLICY_SPECS: &[AdminRouteSpec] = &[
|
|||||||
GET_TABLE_LIFECYCLE,
|
GET_TABLE_LIFECYCLE,
|
||||||
RouteRiskLevel::Sensitive,
|
RouteRiskLevel::Sensitive,
|
||||||
),
|
),
|
||||||
|
admin(
|
||||||
|
HttpMethod::Post,
|
||||||
|
"/iceberg/v1/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/scheduler/run",
|
||||||
|
RUN_TABLE_MAINTENANCE,
|
||||||
|
RouteRiskLevel::High,
|
||||||
|
),
|
||||||
admin(
|
admin(
|
||||||
HttpMethod::Post,
|
HttpMethod::Post,
|
||||||
"/iceberg/v1/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/worker/run",
|
"/iceberg/v1/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/worker/run",
|
||||||
@@ -1094,6 +1100,12 @@ pub const ADMIN_ROUTE_POLICY_SPECS: &[AdminRouteSpec] = &[
|
|||||||
GET_TABLE_LIFECYCLE,
|
GET_TABLE_LIFECYCLE,
|
||||||
RouteRiskLevel::Sensitive,
|
RouteRiskLevel::Sensitive,
|
||||||
),
|
),
|
||||||
|
admin(
|
||||||
|
HttpMethod::Post,
|
||||||
|
"/_iceberg/v1/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/scheduler/run",
|
||||||
|
RUN_TABLE_MAINTENANCE,
|
||||||
|
RouteRiskLevel::High,
|
||||||
|
),
|
||||||
admin(
|
admin(
|
||||||
HttpMethod::Post,
|
HttpMethod::Post,
|
||||||
"/_iceberg/v1/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/worker/run",
|
"/_iceberg/v1/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/worker/run",
|
||||||
@@ -1342,7 +1354,7 @@ mod tests {
|
|||||||
let table_specs = ADMIN_ROUTE_POLICY_SPECS
|
let table_specs = ADMIN_ROUTE_POLICY_SPECS
|
||||||
.iter()
|
.iter()
|
||||||
.filter(|spec| spec.path().starts_with("/iceberg/v1") || spec.path().starts_with("/_iceberg/v1"));
|
.filter(|spec| spec.path().starts_with("/iceberg/v1") || spec.path().starts_with("/_iceberg/v1"));
|
||||||
assert_eq!(table_specs.count(), 88);
|
assert_eq!(table_specs.count(), 90);
|
||||||
assert_action(HttpMethod::Put, "/iceberg/v1/buckets/{warehouse}", SET_TABLE_BUCKET);
|
assert_action(HttpMethod::Put, "/iceberg/v1/buckets/{warehouse}", SET_TABLE_BUCKET);
|
||||||
assert_action(HttpMethod::Get, "/_iceberg/v1/buckets/{warehouse}", GET_TABLE_BUCKET);
|
assert_action(HttpMethod::Get, "/_iceberg/v1/buckets/{warehouse}", GET_TABLE_BUCKET);
|
||||||
assert_action(HttpMethod::Get, "/iceberg/v1/{warehouse}/namespaces", GET_TABLE_NAMESPACE);
|
assert_action(HttpMethod::Get, "/iceberg/v1/{warehouse}/namespaces", GET_TABLE_NAMESPACE);
|
||||||
@@ -1477,6 +1489,16 @@ mod tests {
|
|||||||
"/_iceberg/v1/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/scheduler",
|
"/_iceberg/v1/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/scheduler",
|
||||||
GET_TABLE_LIFECYCLE,
|
GET_TABLE_LIFECYCLE,
|
||||||
);
|
);
|
||||||
|
assert_action(
|
||||||
|
HttpMethod::Post,
|
||||||
|
"/iceberg/v1/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/scheduler/run",
|
||||||
|
RUN_TABLE_MAINTENANCE,
|
||||||
|
);
|
||||||
|
assert_action(
|
||||||
|
HttpMethod::Post,
|
||||||
|
"/_iceberg/v1/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/scheduler/run",
|
||||||
|
RUN_TABLE_MAINTENANCE,
|
||||||
|
);
|
||||||
assert_action(
|
assert_action(
|
||||||
HttpMethod::Post,
|
HttpMethod::Post,
|
||||||
"/iceberg/v1/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/worker/run",
|
"/iceberg/v1/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/worker/run",
|
||||||
|
|||||||
@@ -424,6 +424,11 @@ fn expected_admin_route_matrix() -> Vec<RouteMatrixEntry> {
|
|||||||
"/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/scheduler",
|
"/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/scheduler",
|
||||||
"/analytics/namespaces/sales/tables/orders/maintenance/scheduler",
|
"/analytics/namespaces/sales/tables/orders/maintenance/scheduler",
|
||||||
),
|
),
|
||||||
|
table_route_sample(
|
||||||
|
Method::POST,
|
||||||
|
"/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/scheduler/run",
|
||||||
|
"/analytics/namespaces/sales/tables/orders/maintenance/scheduler/run",
|
||||||
|
),
|
||||||
table_route_sample(
|
table_route_sample(
|
||||||
Method::POST,
|
Method::POST,
|
||||||
"/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/worker/run",
|
"/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/worker/run",
|
||||||
@@ -608,6 +613,11 @@ fn expected_admin_route_matrix() -> Vec<RouteMatrixEntry> {
|
|||||||
"/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/scheduler",
|
"/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/scheduler",
|
||||||
"/analytics/namespaces/sales/tables/orders/maintenance/scheduler",
|
"/analytics/namespaces/sales/tables/orders/maintenance/scheduler",
|
||||||
),
|
),
|
||||||
|
compat_table_route_sample(
|
||||||
|
Method::POST,
|
||||||
|
"/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/scheduler/run",
|
||||||
|
"/analytics/namespaces/sales/tables/orders/maintenance/scheduler/run",
|
||||||
|
),
|
||||||
compat_table_route_sample(
|
compat_table_route_sample(
|
||||||
Method::POST,
|
Method::POST,
|
||||||
"/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/worker/run",
|
"/{warehouse}/namespaces/{namespace}/tables/{table}/maintenance/worker/run",
|
||||||
@@ -864,6 +874,11 @@ fn test_register_routes_cover_representative_admin_paths() {
|
|||||||
Method::GET,
|
Method::GET,
|
||||||
&table_catalog_path("/analytics/namespaces/sales/tables/orders/maintenance/scheduler"),
|
&table_catalog_path("/analytics/namespaces/sales/tables/orders/maintenance/scheduler"),
|
||||||
);
|
);
|
||||||
|
assert_route(
|
||||||
|
&router,
|
||||||
|
Method::POST,
|
||||||
|
&table_catalog_path("/analytics/namespaces/sales/tables/orders/maintenance/scheduler/run"),
|
||||||
|
);
|
||||||
assert_route(
|
assert_route(
|
||||||
&router,
|
&router,
|
||||||
Method::POST,
|
Method::POST,
|
||||||
@@ -1021,6 +1036,11 @@ fn test_register_routes_cover_representative_admin_paths() {
|
|||||||
Method::GET,
|
Method::GET,
|
||||||
&compat_table_catalog_path("/analytics/namespaces/sales/tables/orders/maintenance/scheduler"),
|
&compat_table_catalog_path("/analytics/namespaces/sales/tables/orders/maintenance/scheduler"),
|
||||||
);
|
);
|
||||||
|
assert_route(
|
||||||
|
&router,
|
||||||
|
Method::POST,
|
||||||
|
&compat_table_catalog_path("/analytics/namespaces/sales/tables/orders/maintenance/scheduler/run"),
|
||||||
|
);
|
||||||
assert_route(
|
assert_route(
|
||||||
&router,
|
&router,
|
||||||
Method::POST,
|
Method::POST,
|
||||||
|
|||||||
+823
-31
File diff suppressed because it is too large
Load Diff
@@ -230,9 +230,9 @@ The smoke test also probes catalog-backed advanced Iceberg surfaces:
|
|||||||
`main` cannot be deleted
|
`main` cannot be deleted
|
||||||
- Iceberg views support basic create, list, load, replace, existence check, and
|
- Iceberg views support basic create, list, load, replace, existence check, and
|
||||||
drop routes with persisted view metadata and view-scoped authorization
|
drop routes with persisted view metadata and view-scoped authorization
|
||||||
- metadata maintenance supports safe dry-run planning, controlled worker
|
- metadata maintenance supports safe dry-run planning, controlled scheduler
|
||||||
execution checks, and a scheduler status report with disabled, paused,
|
queueing, worker execution checks, and a scheduler status report with disabled,
|
||||||
backpressure, retry, quarantine, and audit-timeline state; job reports expose
|
paused, queued, backpressure, retry, quarantine, and audit-timeline state; job reports expose
|
||||||
structured audit events for planning, worker transitions, heartbeat updates,
|
structured audit events for planning, worker transitions, heartbeat updates,
|
||||||
lease expiry, and mutating quarantine operations; quarantined jobs can be
|
lease expiry, and mutating quarantine operations; quarantined jobs can be
|
||||||
inspected, released, retried, or abandoned through an operator endpoint
|
inspected, released, retried, or abandoned through an operator endpoint
|
||||||
@@ -338,10 +338,10 @@ Unsupported behavior is documented instead of hidden behind internal errors. The
|
|||||||
current unsupported inventory is:
|
current unsupported inventory is:
|
||||||
|
|
||||||
- credential vending: automated after table bootstrap with exact-prefix validation and a data-plane scope probe; full no-long-term-data-credential bootstrap is not claimed
|
- credential vending: automated after table bootstrap with exact-prefix validation and a data-plane scope probe; full no-long-term-data-credential bootstrap is not claimed
|
||||||
- background maintenance worker: controlled run-once, heartbeat, quarantine operation, and scheduler status endpoints are registered; disabled/paused/backpressure/retry/quarantine/audit-timeline state and per-job audit events are machine-readable; continuous in-process scheduling is not claimed
|
- background maintenance worker: controlled scheduler run, worker run-once, heartbeat, quarantine operation, and scheduler status endpoints are registered; disabled/paused/queued/backpressure/retry/quarantine/audit-timeline state and per-job audit events are machine-readable; continuous in-process scheduling is not claimed
|
||||||
- manifest/data reachability cleanup: metadata maintenance reads manifest-list and manifest Avro references, reports manifest/data/delete reachability, and deletes only unreferenced table objects that pass the safety window
|
- manifest/data reachability cleanup: metadata maintenance reads manifest-list and manifest Avro references, reports manifest/data/delete reachability, and deletes only unreferenced table objects that pass the safety window
|
||||||
- snapshot expiration dry-run planning and manual catalog commit: supported through metadata maintenance reports
|
- snapshot expiration dry-run planning and manual catalog commit: supported through metadata maintenance reports
|
||||||
- automatic maintenance scheduling: external scheduler hook supported through the worker run endpoint and scheduler status report; built-in periodic scheduling is not claimed
|
- automatic maintenance scheduling: external scheduler hook supported through the scheduler run endpoint, worker run endpoint, and scheduler status report; built-in periodic scheduling is not claimed
|
||||||
- compaction rewrite: controlled run-once support for partition-local and sort-order-preserving Parquet binpack through metadata maintenance; manifests with position or equality delete files produce machine-readable row-level planning and fail closed before rewrite; built-in periodic scheduling, delete-file rewrite, and row-level compaction execution are not claimed
|
- compaction rewrite: controlled run-once support for partition-local and sort-order-preserving Parquet binpack through metadata maintenance; manifests with position or equality delete files produce machine-readable row-level planning and fail closed before rewrite; built-in periodic scheduling, delete-file rewrite, and row-level compaction execution are not claimed
|
||||||
- row-level delete/update/merge commits: standard catalog commit validates append, overwrite, delete, and replace snapshot manifests for table-warehouse scope, referenced object existence, current-live-file deletes, and stale add/delete conflicts; end-to-end SQL DML client coverage remains a compatibility validation item
|
- row-level delete/update/merge commits: standard catalog commit validates append, overwrite, delete, and replace snapshot manifests for table-warehouse scope, referenced object existence, current-live-file deletes, and stale add/delete conflicts; end-to-end SQL DML client coverage remains a compatibility validation item
|
||||||
- external catalog bridges: metadata import/register and operator-supplied metadata pointer sync are supported for Polaris/Glue/DLF/Hive identity boundaries; online vendor SDK polling and policy mirroring are not claimed
|
- external catalog bridges: metadata import/register and operator-supplied metadata pointer sync are supported for Polaris/Glue/DLF/Hive identity boundaries; online vendor SDK polling and policy mirroring are not claimed
|
||||||
|
|||||||
@@ -183,7 +183,7 @@ UNSUPPORTED_INVENTORY: list[dict[str, str]] = [
|
|||||||
"status": "controlled-run-once-supported",
|
"status": "controlled-run-once-supported",
|
||||||
"roadmap_area": "maintenance-worker",
|
"roadmap_area": "maintenance-worker",
|
||||||
"catalog_endpoint": "GET /v1/{prefix}/namespaces/{namespace}/tables/{table}/maintenance/scheduler",
|
"catalog_endpoint": "GET /v1/{prefix}/namespaces/{namespace}/tables/{table}/maintenance/scheduler",
|
||||||
"expected_behavior": "background-enabled maintenance can be driven by the worker run endpoint and inspected through the scheduler status endpoint with disabled/paused state, current-job backpressure, retry deferral, quarantine boundary, audit timeline, per-job audit events, lease expiry recovery, heartbeat updates, and operator quarantine inspect/release/retry/abandon actions; built-in periodic scheduling is not claimed",
|
"expected_behavior": "background-enabled maintenance can be queued by the scheduler run endpoint, claimed by the worker run endpoint, and inspected through the scheduler status endpoint with disabled/paused state, queued-job handoff, current-job backpressure, retry deferral, quarantine boundary, audit timeline, per-job audit events, lease expiry recovery, heartbeat updates, and operator quarantine inspect/release/retry/abandon actions; built-in periodic scheduling is not claimed",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"capability": "manifest-data-reachability-cleanup",
|
"capability": "manifest-data-reachability-cleanup",
|
||||||
@@ -1050,6 +1050,7 @@ def run_view_probe(args: argparse.Namespace, deps: RuntimeDeps) -> None:
|
|||||||
def run_maintenance_probe(args: argparse.Namespace, deps: RuntimeDeps) -> None:
|
def run_maintenance_probe(args: argparse.Namespace, deps: RuntimeDeps) -> None:
|
||||||
config_path = table_endpoint_path(args, "/maintenance/config")
|
config_path = table_endpoint_path(args, "/maintenance/config")
|
||||||
scheduler_path = table_endpoint_path(args, "/maintenance/scheduler")
|
scheduler_path = table_endpoint_path(args, "/maintenance/scheduler")
|
||||||
|
scheduler_run_path = table_endpoint_path(args, "/maintenance/scheduler/run")
|
||||||
signed_rest_request(args, deps, "PUT", config_path, default_maintenance_config())
|
signed_rest_request(args, deps, "PUT", config_path, default_maintenance_config())
|
||||||
config = signed_rest_request(args, deps, "GET", config_path)
|
config = signed_rest_request(args, deps, "GET", config_path)
|
||||||
if config.get("version") != TABLE_MAINTENANCE_CONFIG_VERSION:
|
if config.get("version") != TABLE_MAINTENANCE_CONFIG_VERSION:
|
||||||
@@ -1083,7 +1084,7 @@ def run_maintenance_probe(args: argparse.Namespace, deps: RuntimeDeps) -> None:
|
|||||||
if quarantine.get("action") != "INSPECT" or not isinstance(quarantine.get("report"), dict):
|
if quarantine.get("action") != "INSPECT" or not isinstance(quarantine.get("report"), dict):
|
||||||
raise RuntimeError("maintenance quarantine endpoint did not return an inspection report")
|
raise RuntimeError("maintenance quarantine endpoint did not return an inspection report")
|
||||||
scheduler = signed_rest_request(args, deps, "GET", scheduler_path)
|
scheduler = signed_rest_request(args, deps, "GET", scheduler_path)
|
||||||
expected_scheduler_statuses = {"READY", "DISABLED", "PAUSED", "BACKPRESSURED", "RETRY_DEFERRED", "QUARANTINED"}
|
expected_scheduler_statuses = {"READY", "QUEUED", "DISABLED", "PAUSED", "BACKPRESSURED", "RETRY_DEFERRED", "QUARANTINED"}
|
||||||
if scheduler.get("status") not in expected_scheduler_statuses:
|
if scheduler.get("status") not in expected_scheduler_statuses:
|
||||||
raise RuntimeError("maintenance scheduler endpoint did not return a stable scheduler status")
|
raise RuntimeError("maintenance scheduler endpoint did not return a stable scheduler status")
|
||||||
scheduler_timeline = scheduler.get("audit_timeline")
|
scheduler_timeline = scheduler.get("audit_timeline")
|
||||||
@@ -1092,9 +1093,23 @@ def run_maintenance_probe(args: argparse.Namespace, deps: RuntimeDeps) -> None:
|
|||||||
if scheduler_timeline and not isinstance(scheduler_timeline[0].get("audit-events"), list):
|
if scheduler_timeline and not isinstance(scheduler_timeline[0].get("audit-events"), list):
|
||||||
raise RuntimeError("maintenance scheduler audit timeline did not include job audit events")
|
raise RuntimeError("maintenance scheduler audit timeline did not include job audit events")
|
||||||
|
|
||||||
|
scheduler_config = default_maintenance_config()
|
||||||
|
scheduler_config["background-enabled"] = True
|
||||||
|
scheduler_config["worker-paused"] = False
|
||||||
|
signed_rest_request(args, deps, "PUT", config_path, scheduler_config)
|
||||||
|
scheduler_run = signed_rest_request(args, deps, "POST", scheduler_run_path, {"scheduler-id": "pyiceberg-smoke-scheduler"})
|
||||||
|
scheduler_run_job = scheduler_run.get("report", {}).get("job")
|
||||||
|
if not isinstance(scheduler_run_job, dict) or scheduler_run_job.get("status") != "QUEUED":
|
||||||
|
raise RuntimeError("maintenance scheduler run endpoint did not queue a maintenance job")
|
||||||
|
if scheduler_run_job.get("scheduler-id") != "pyiceberg-smoke-scheduler":
|
||||||
|
raise RuntimeError("maintenance scheduler run endpoint did not preserve scheduler identity")
|
||||||
|
scheduler_run_report = scheduler_run.get("scheduler")
|
||||||
|
if not isinstance(scheduler_run_report, dict) or scheduler_run_report.get("status") != "QUEUED":
|
||||||
|
raise RuntimeError("maintenance scheduler run endpoint did not return queued scheduler state")
|
||||||
|
|
||||||
worker = signed_rest_request(args, deps, "POST", table_endpoint_path(args, "/maintenance/worker/run"), {})
|
worker = signed_rest_request(args, deps, "POST", table_endpoint_path(args, "/maintenance/worker/run"), {})
|
||||||
worker_job = worker.get("job")
|
worker_job = worker.get("job")
|
||||||
expected_statuses = {"DISABLED", "PAUSED", "FAILED", "SUCCESSFUL", "RUNNING"}
|
expected_statuses = {"DISABLED", "PAUSED", "FAILED", "SUCCESSFUL", "RUNNING", "QUEUED"}
|
||||||
if not isinstance(worker_job, dict) or worker_job.get("status") not in expected_statuses:
|
if not isinstance(worker_job, dict) or worker_job.get("status") not in expected_statuses:
|
||||||
raise RuntimeError("maintenance worker endpoint did not return a stable job status")
|
raise RuntimeError("maintenance worker endpoint did not return a stable job status")
|
||||||
if not isinstance(worker.get("audit-events"), list):
|
if not isinstance(worker.get("audit-events"), list):
|
||||||
|
|||||||
Reference in New Issue
Block a user