Compare commits

...

161 Commits

Author SHA1 Message Date
github-actions[bot] 5f92297108 chore: release 1.20.1 2026-09-03 15:41:20 +00:00
Charles GTE d1821f7045 Merge pull request #102 from Portabase/fix/cron-crash
fix: cron crash for databases setup from dashboard
2026-09-03 17:38:53 +02:00
charles-gauthereau 0fe4d50cbd fix: docker-compose.yml 2026-09-03 17:38:35 +02:00
charles-gauthereau 1f29466a28 fix: cron crash for databases setup from dashboard 2026-09-03 17:22:03 +02:00
github-actions[bot] b74aaa0bbc chore: release 1.20.0 2026-08-28 15:42:27 +00:00
Charles GTE 7b5e5b2c78 Merge pull request #101 from Portabase/feat/retry-system
feat: retry-system
2026-08-28 17:40:20 +02:00
charles-gauthereau 99f1ef2081 fix: refactoring 2026-08-28 17:25:30 +02:00
charles-gauthereau 42c3c5945a fix: refactoring 2026-08-28 17:06:23 +02:00
charles-gauthereau db0f87c2e9 Merge branch 'main' into feat/retry-system 2026-08-28 16:54:51 +02:00
github-actions[bot] f95f0aa73a chore: release 1.19.2 2026-08-28 07:59:57 +00:00
Charles GTE 2177dfd44b Merge pull request #100 from Portabase/fix/kill-on-drop
fix: kill_on_drop on firebird, mariadb, mysql, redis, valkey
2026-08-28 09:57:51 +02:00
charles-gauthereau 0a53eec184 fix: kill_on_drop on firebird, mariadb, mysql, redis, valkey 2026-08-28 09:43:08 +02:00
charles-gauthereau 87f33af772 fix: wire retry env vars into helm configmap, dedupe terminal retry log
helm/templates/env-configmap.yaml never listed RETRY_ATTEMPTS and
RETRY_BACKOFF_MS even though values.yaml gained them, so --set
env.RETRY_ATTEMPTS=N was silently ignored by Kubernetes deployments.
Add both keys in the same explicit style as the existing entries.

src/utils/retry.rs logged its own "failed after N attempts" error on
exhaustion, on top of the terminal log each call site already writes,
producing two error entries per failure. Worse, it changed a log
level: FileLock::acquire's "backup_already_in_progress" bails through
the combinator, which now logged it as error before runner.rs got a
chance to reclassify it as the routine warn it always was. A manual
backup colliding with a scheduled one would show up as a hard error
on the dashboard instead of the harmless warn it used to be, breaking
the "fails exactly as it does today" guarantee for job records.

Drop the combinator's terminal error log and give download_backup its
own terminal error log so all three call sites (runner, uploader,
downloader) own their failure logging uniformly. Update the two tests
that asserted the removed message to assert the new behavior instead.
2026-08-27 22:46:49 +02:00
charles-gauthereau b6a120fcbf feat: retry the restore backup download
Extracts the download body to download_once and makes download_backup a
retry wrapper around it. This path had no retry at all before, so a
single dropped connection failed the whole restore job.

Retrying is safe because File::create truncates and the target filename
is derived from Content-Disposition or the URL, so it is stable across
attempts. There is no Range resume: a download that fails at 90% starts
over.
2026-08-27 19:01:10 +02:00
charles-gauthereau 5298d82576 feat: retry storage uploads
Wraps provider.upload in the retry combinator. No provider changes are
needed: each one builds its upload stream from the file on disk inside
upload(), so every attempt gets a fresh handle and a fresh nonce.

Result<UploadResult, UploadResult> is collapsed with an or-pattern so
the last attempt's error and metadata survive into the existing failure
branch. A missing backup file short-circuits into Err rather than
returning early, which skips the retry without skipping the
backup_upload_status(failed) call that closes the server-side record.

backup_upload_init and backup_upload_status are left unwrapped; they
are control-plane calls, not storage uploads.
2026-08-27 18:53:06 +02:00
charles-gauthereau b0da2e40a3 feat: retry the database backup
Each attempt now dumps into its own tmp_path/attempt-{n} directory,
which is removed when the attempt fails. Without the per-attempt
directory pg_dump -Fd would refuse every retry, because it will not
write into a directory a previous attempt left behind; removing it on
failure keeps peak disk at one attempt's artifacts rather than five.

A backup blocked by a concurrent job is retried before surfacing the
same backup_already_in_progress code, since FileLock reports it as an
ordinary error and the combinator cannot tell it apart.

Reshapes retry()'s bound from the native AsyncFnMut sugar to the
classic F: FnMut(u32) -> Fut, Fut: Future<Output = Result<T, E>> + Send
shape (the pattern tokio-retry and backoff both use). AsyncFnMut's
produced future is a lifetime-quantified associated type
(F::CallRefFuture<'_>) that cannot be named or bounded as Send on
stable Rust, so wiring a retried call through it into a future that
eventually gets polled inside tokio::spawn (dispatcher.rs, via
execute_backup) made rustc's opaque-type Send inference fail with
"implementation of Send is not general enough" at the spawn site,
several call layers away from the actual retry call. Naming Fut as its
own type parameter lets Send be asserted on it directly instead, which
resolves cleanly. The combinator's control flow, log messages, and
formats are unchanged; only the bound and the call sites' closure
shape (async |x| { } becomes |x| async { }, with shared references
bound outside a move closure so the inner async move block only moves
Copy references, not the originals) are affected.
2026-08-27 18:39:41 +02:00
charles-gauthereau 55e20d48e7 feat: configurable retry policy and combinator
Adds RETRY_ATTEMPTS (3..=5, default 3) and RETRY_BACKOFF_MS
(100..=30000, default 1000) to Settings, validated with the same
panic-on-invalid contract as POOLING and CHUNK_SIZE_MB.

The combinator logs every failed attempt and any late success through
the JobLogger it borrows, so retries reach the server on the existing
job-log path with no API change. It borrows rather than clones the Arc
so Arc::try_unwrap in the backup executor keeps working. Backoff is
exponential with equal jitter, because the uploader retries storages
concurrently and would otherwise retry them in lockstep.
2026-08-27 18:13:17 +02:00
github-actions[bot] f7f5f7e141 chore: release 1.19.1 2026-08-21 17:09:15 +00:00
Charles GTE 2170f96a72 Merge pull request #99 from Portabase/fix/config-file
fix: config directory creation
2026-08-21 19:06:48 +02:00
Charles GTE 298d46ba81 fix: config directory creation 2026-08-21 13:50:40 +02:00
github-actions[bot] 6537e9df53 chore: release 1.19.0 2026-08-20 11:56:59 +00:00
Charles GTE b8d869d5a6 Merge pull request #97 from Portabase/feat/dashboard-databases
feat: dashboard-databases
2026-08-20 13:54:46 +02:00
Charles GTE 90941ea67d fix 2026-08-20 13:41:14 +02:00
charles-gauthereau 046d593e2b fix 2026-08-18 18:20:21 +02:00
charles-gauthereau cf9a59a138 fix: config.rs 2026-08-18 13:07:22 +02:00
charles-gauthereau 2464f6dfb0 Merge branch 'main' into feat/dashboard-databases
# Conflicts:
#	docker-compose.yml
#	src/services/config.rs
2026-08-18 13:05:56 +02:00
github-actions[bot] 069067ca55 chore: release 1.18.6 2026-08-17 09:10:08 +00:00
Charles GTE 04b654d219 Merge pull request #96 from Portabase/fix/mongo-cloud
fix: mongodb cloud cluster issue
2026-08-17 11:07:35 +02:00
charles-gauthereau d83511cb64 fix: mongodb cloud cluster issue 2026-08-17 10:52:21 +02:00
charles-gauthereau ca294e968c fix 2026-08-13 22:02:17 +02:00
charles-gauthereau 1be88ffdb9 Merge branch 'main' into feat/dashboard-databases
# Conflicts:
#	docker-compose.yml
2026-08-13 21:56:38 +02:00
charles-gauthereau 9d393a96a4 fix 2026-08-13 21:55:59 +02:00
charles-gauthereau 044bf80633 feat(agent): ingest dashboard databases via merged cycle + cache 2026-08-13 18:47:18 +02:00
charles-gauthereau 0a6eb6db22 feat(dashboard_config): atomic cache load/persist 2026-08-13 18:40:56 +02:00
charles-gauthereau b26ff81889 feat(dashboard_config): merge (dashboard-wins) + collect_configs 2026-08-13 18:39:54 +02:00
charles-gauthereau 4e2f29f4ca feat(status): decrypt dashboard config_ciphertext into resolved_config 2026-08-13 18:38:15 +02:00
charles-gauthereau d1c8df4cac refactor(config): extract build_config, add load_optional + Serialize 2026-08-13 18:35:52 +02:00
github-actions[bot] 30f83bafcf chore: release 1.18.5 2026-07-26 09:06:42 +00:00
Charles GTE de106c835e Merge pull request #93 from Portabase/fix/s3-storage
fix: s3 logs
2026-07-26 11:04:35 +02:00
Charles GTE 23d6822ddc fix: s3 logs 2026-07-26 11:04:07 +02:00
github-actions[bot] fe1d74945f chore: release 1.18.4 2026-07-25 14:03:12 +00:00
Charles GTE 424a646385 Merge pull request #92 from Portabase/fix/windows-build
fix: windows-build
2026-07-25 16:00:57 +02:00
Charles GTE 0ef4bba5d7 fix: docker-compose.yml 2026-07-25 15:45:56 +02:00
Charles GTE 6548140eaf fix: windows build 2026-07-25 15:19:46 +02:00
github-actions[bot] c9725c381e chore: release 1.18.3 2026-07-24 16:46:51 +00:00
Charles GTE 03695b1897 Merge pull request #91 from Portabase/fix/firebird
fix: firebird
2026-07-24 18:44:20 +02:00
charles-gauthereau 80b1c0dac0 fix: firebird with -g for gbak 2026-07-24 18:29:04 +02:00
charles-gauthereau 5ededb3764 Merge branch 'main' into dev 2026-07-24 17:45:46 +02:00
github-actions[bot] 54cfd673e4 chore: release 1.18.2 2026-07-23 18:04:50 +00:00
Charles GTE 39dbeb5d30 Merge pull request #90 from Portabase/fix/ghcr-ci-cd
fix: ghcr.yml
2026-07-23 20:02:28 +02:00
charles-gauthereau 380cea79fb fix: ghcr.yml 2026-07-23 20:02:01 +02:00
github-actions[bot] 65e4538c2c chore: release 1.18.1 2026-07-23 17:47:38 +00:00
Charles GTE 86a706bca5 Merge pull request #89 from Portabase/fix/ghcr-ci-cd
fix: ghcr.yml
2026-07-23 19:45:23 +02:00
charles-gauthereau e8bda60f16 fix: ghcr.yml 2026-07-23 18:47:41 +02:00
charles-gauthereau 034912b176 Merge branch 'main' into dev 2026-07-23 18:24:18 +02:00
github-actions[bot] 4fd50ccda4 chore: release 1.18.0 2026-07-23 15:33:29 +00:00
Charles GTE 84c6b765c8 Merge pull request #88 from Portabase/fix/postgres-provider
fix: postgres-provider
2026-07-23 17:31:02 +02:00
charles-gauthereau 9790837200 fix: postgres 2026-07-23 17:14:27 +02:00
charles-gauthereau 8827234344 refactor(postgres): split restore into restore/ module (run, prepare, command, toc) 2026-07-23 16:40:56 +02:00
charles-gauthereau b3417304bb fix(postgres): match real pg_restore -l schema line in toc_creates_public_schema 2026-07-23 10:06:01 +02:00
charles-gauthereau 5f4006b1ca feat(postgres): sniff dump format by magic bytes 2026-07-23 09:56:46 +02:00
charles-gauthereau f24b1ec3e4 feat(postgres): wire clean_mode=drop_database with preflight gate 2026-07-23 09:51:49 +02:00
charles-gauthereau bb1b16d0f2 feat(postgres): add drop_and_recreate_database + preflight + FORCE/retry 2026-07-23 09:43:24 +02:00
charles-gauthereau 2aca036ee4 feat(postgres): wire clean_mode none/clean/drop_schemas into restore 2026-07-23 09:36:25 +02:00
charles-gauthereau 2a72a55e57 feat(postgres): detect CREATE SCHEMA public in archive TOC 2026-07-23 09:31:50 +02:00
charles-gauthereau ae63a0df50 feat(postgres): add drop_all_schemas + recreate_public_schema 2026-07-23 09:27:36 +02:00
charles-gauthereau 560c5e02f2 feat(postgres): add RestoreCleanMode option parsing 2026-07-23 09:24:21 +02:00
charles-gauthereau ca17bf9cb9 feat(postgres): add quote_ident/quote_literal for DDL 2026-07-23 09:20:15 +02:00
charles-gauthereau 9fbaab6feb refactor(postgres): unify FC/FD restore via run_pg_restore, validate-before-destroy 2026-07-23 09:15:32 +02:00
charles-gauthereau 1289497d98 refactor(postgres): extract prepare_archive, validate before restore 2026-07-23 09:09:19 +02:00
charles-gauthereau 27c55dbfd8 refactor(postgres): drive async restore work via captured Tokio Handle 2026-07-23 09:01:50 +02:00
charles-gauthereau 45e3acd709 fix(postgres): add pg_restore_binary_name() for Windows restore 2026-07-23 08:58:49 +02:00
charles-gauthereau a93054518a Merge branch 'main' into dev 2026-07-22 00:06:17 +02:00
Charles GTE baf2edc829 fix (#87)
Co-authored-by: charles-gauthereau <charles.gauthereau@soluce-technologies.com>
2026-07-21 23:58:11 +02:00
github-actions[bot] f050fa35e0 chore: release 1.17.1 2026-07-21 21:29:30 +00:00
Charles GTE 90d55f7630 fix: backup folder name (#86)
* fix: support configurable backup file prefix (#83)

* fix: backup folder name

* fix: ghcr publish agent image

---------

Co-authored-by: Antonin Jousson <18756890+Antoninj@users.noreply.github.com>
Co-authored-by: charles-gauthereau <charles.gauthereau@soluce-technologies.com>
2026-07-21 23:27:10 +02:00
charles-gauthereau c186d57105 fix: ghcr publish agent image 2026-07-21 23:10:40 +02:00
charles-gauthereau 37b4c1fe6d fix: backup folder name 2026-07-21 23:05:59 +02:00
Antonin Jousson 92a3324ca2 fix: support configurable backup file prefix (#83) 2026-07-21 21:50:11 +02:00
github-actions[bot] 82ffb71496 chore: release 1.17.0 2026-07-11 10:08:21 +00:00
Charles GTE 20b3b58024 Merge pull request #80 from Portabase/fix/storage-encryption
fix/storage-encryption
2026-07-11 12:05:59 +02:00
charles-gauthereau 6cddc73565 Merge branch 'main' into fix/storage-encryption
# Conflicts:
#	docker-compose.yml
2026-07-10 18:49:01 +02:00
github-actions[bot] fd3b0076c0 chore: release 1.16.3 2026-07-09 09:11:23 +00:00
Charles GTE 334ab51125 fix: cron (#79)
* fix: cron log

---------

Co-authored-by: charles-gauthereau <charles.gauthereau@soluce-technologies.com>
2026-07-09 11:03:55 +02:00
github-actions[bot] 4df3fe2d0c chore: release 1.16.2 2026-07-09 07:38:49 +00:00
Charles GTE ec7715ca26 fix: cron (#78)
* fix: docker-compose.yml

* fix: cron

---------

Co-authored-by: charles-gauthereau <charles.gauthereau@soluce-technologies.com>
2026-07-09 09:36:53 +02:00
charles-gauthereau 32d4e22196 fix: status.rs 2026-07-08 22:24:50 +02:00
charles-gauthereau f277c5485f feat: decrypt encrypted storages in status ping 2026-07-08 20:52:17 +02:00
charles-gauthereau 585e0bea72 feat: accept encrypted storages markers in DatabaseStatus 2026-07-08 20:47:20 +02:00
charles-gauthereau ee97c6056a feat: add decrypt_json_gcm for encrypted status storages 2026-07-08 20:43:01 +02:00
charles-gauthereau b37a4f2180 fix: docker-compose.yml 2026-07-07 18:43:30 +02:00
github-actions[bot] 1d15f40662 chore: release 1.16.1 2026-07-04 10:26:50 +00:00
Charles GTE 324a2ea3f2 fix: security (#75) 2026-07-04 12:24:37 +02:00
github-actions[bot] 9ec92af6c7 chore: release 1.16.0 2026-07-03 06:16:42 +00:00
Charles GTE 1dafdca2a0 Merge pull request #74 from Portabase/feat/docker-volume-backup-restore
feat: docker-volume-backup-restore
2026-07-03 08:14:30 +02:00
charles-gauthereau d12817a960 fix: codecov.yml 2026-07-02 22:42:02 +02:00
charles-gauthereau 608b82a254 fix: test docker_volume.rs 2026-07-02 22:08:55 +02:00
charles-gauthereau bbecca92f9 fix: databases.json 2026-07-02 21:51:17 +02:00
charles-gauthereau b2b2733c4d feat: docker-volume provider
refactor: move build_tar to utils::compress

refactor: move choose_restore_path to utils::common

docs: remove docker-volume README section

fix: keep archive path for multi-file non-docker-volume restores

docs: trim docker-volume README section to essentials

refactor: run docker-volume backup/restore inside spawn_blocking like other providers

docs: enable docker socket, add volume example and security notes

feat: sweep orphaned ephemeral helper containers on startup

feat: docker-volume clean-replace restore via upload_to_container

fix: serialize env-var access in docker-volume tests to avoid setenv/getenv UB

feat: docker-volume backup via download_from_container

feat: docker-volume ping via inspect_volume

fix: return extraction dir for multi-file restore archives

feat: gzip already-tar inputs directly instead of double-wrapping

feat: docker helper (client, self-image, container lifecycle, sweep)

feat: docker-volume provider
2026-07-02 21:45:11 +02:00
github-actions[bot] 2c257c5a58 chore: release 1.15.0 2026-06-29 17:14:03 +00:00
Charles GTE c24c0d7058 Merge pull request #73 from Portabase/fix/postgres-ownership
fix: postgres-ownership
2026-06-29 19:12:01 +02:00
charles-gauthereau 34f353b68f fix: postgres keep_ownership flag 2026-06-29 19:10:49 +02:00
charles-gauthereau ee51996401 chore: document keep_ownership option in example databases.json 2026-06-29 18:54:01 +02:00
charles-gauthereau 694b463787 feat: conditionally omit --no-owner/--no-privileges in pg_restore based on keep_ownership option 2026-06-29 18:52:37 +02:00
charles-gauthereau aed1e86956 feat: add generic options map to DatabaseConfig
Add `options: HashMap<String, serde_json::Value>` to DatabaseConfig and
`options: Option<HashMap<...>>` to InputDatabaseConfig so callers can
pass arbitrary per-database options (e.g. keep_ownership for pg_dumpall).
Update all DatabaseConfig struct literals in test files and add three
new config-service unit tests covering present, absent, and non-bool values.
2026-06-29 18:49:01 +02:00
charles-gauthereau 6388aff1e3 fix: postgres cluster dump command 2026-06-29 17:10:11 +02:00
github-actions[bot] 39a77b18a4 chore: release 1.14.1 2026-06-29 13:22:10 +00:00
Charles GTE e62167182e Merge pull request #72 from Portabase/fix/postgres-cluster
fix: postgres cluster dump command
2026-06-29 15:19:54 +02:00
charles-gauthereau 4be54a614e fix: postgres cluster dump command 2026-06-29 15:18:52 +02:00
github-actions[bot] c5eaa5086c chore: release 1.14.0 2026-06-27 19:16:46 +00:00
Charles GTE 610d443afc Dev (#69)
* fix: pgdump-cluster (#68)

* feat: add as_str/from_str to PostgresDumpFormat

* feat: resolve pg_dumpall/psql binary names

* feat: add include_globals field to database config

* feat: add pg_dumpall/psql globals dump and apply

* feat: add postgres backup bundle (manifest + build + resolve)

* feat: bundle globals into postgres backup when include_globals is set

* feat: replay globals before pg_restore when backup archive is a bundle

* refactor: bind FD restore tempdir guard once to clear unused warnings

* docs: demonstrate include_globals in sample databases.json

* chore: silence test-only re-export warning in non-test builds

* revert: remove include_globals feature, restore plain pg_dump/pg_restore

* feat: add pg_dumpall/psql binary names and is_superuser check

* feat: add postgresql-cluster db type and config parsing

* feat: pg_dumpall cluster backup and psql restore

* feat: route postgresql-cluster through PostgresClusterDatabase

* docs: add postgresql-cluster sample to databases.json

* refactor: split cluster mode into cluster/ module (backup, restore, database)

* test: mirror cluster tests into src/tests/domain/cluster/

* feat: add-gcs-storage-provider (#70)

* fix: refactoring

* chore: add google-cloud-storage and google-cloud-auth deps

* feat: add GCS provider config model

* feat: add GCS credential, client, and stream-source helpers

StreamSource bridges build_stream's Send-only byte stream into the SDK's
StreamingSource (which send_buffered requires to be Send+Sync+'static) via a
bounded mpsc channel, avoiding any change to the shared UploadStream type.

* feat: implement GCS StorageProvider upload

* feat: register google-cloud-storage provider in factory

* test: GCS upload roundtrip against fake-gcs-server

* fix: format GCS bucket as projects/_/buckets/<name> in upload_with_client

write_object rejects the bare bucket id with "malformed bucket name"; the
production provider passed config.bucket_name unformatted, so real uploads
would always fail. Format once in upload_with_client so prod and the
fake-gcs-server test share the corrected path.

* style: cargo fmt GCS provider files

* fix: gcs

* fix: azure (#71)
2026-06-27 21:14:21 +02:00
Charles GTE 16a37e033e fix: azure (#71) 2026-06-27 20:51:47 +02:00
Charles GTE 29ca5a317b feat: add-gcs-storage-provider (#70)
* fix: refactoring

* chore: add google-cloud-storage and google-cloud-auth deps

* feat: add GCS provider config model

* feat: add GCS credential, client, and stream-source helpers

StreamSource bridges build_stream's Send-only byte stream into the SDK's
StreamingSource (which send_buffered requires to be Send+Sync+'static) via a
bounded mpsc channel, avoiding any change to the shared UploadStream type.

* feat: implement GCS StorageProvider upload

* feat: register google-cloud-storage provider in factory

* test: GCS upload roundtrip against fake-gcs-server

* fix: format GCS bucket as projects/_/buckets/<name> in upload_with_client

write_object rejects the bare bucket id with "malformed bucket name"; the
production provider passed config.bucket_name unformatted, so real uploads
would always fail. Format once in upload_with_client so prod and the
fake-gcs-server test share the corrected path.

* style: cargo fmt GCS provider files

* fix: gcs
2026-06-27 20:37:22 +02:00
Charles GTE 01f5d34c96 fix: pgdump-cluster (#68)
* feat: add as_str/from_str to PostgresDumpFormat

* feat: resolve pg_dumpall/psql binary names

* feat: add include_globals field to database config

* feat: add pg_dumpall/psql globals dump and apply

* feat: add postgres backup bundle (manifest + build + resolve)

* feat: bundle globals into postgres backup when include_globals is set

* feat: replay globals before pg_restore when backup archive is a bundle

* refactor: bind FD restore tempdir guard once to clear unused warnings

* docs: demonstrate include_globals in sample databases.json

* chore: silence test-only re-export warning in non-test builds

* revert: remove include_globals feature, restore plain pg_dump/pg_restore

* feat: add pg_dumpall/psql binary names and is_superuser check

* feat: add postgresql-cluster db type and config parsing

* feat: pg_dumpall cluster backup and psql restore

* feat: route postgresql-cluster through PostgresClusterDatabase

* docs: add postgresql-cluster sample to databases.json

* refactor: split cluster mode into cluster/ module (backup, restore, database)

* test: mirror cluster tests into src/tests/domain/cluster/
2026-06-27 17:23:57 +02:00
github-actions[bot] 0b7b7ef59b chore: release 1.13.1 2026-06-25 12:58:08 +00:00
Charles GTE 038c5523b7 fix: download-crash-large-files (#67)
* fix(restore): add ProgressTracker for download progress logging

* fix(restore): stream backup download to disk instead of buffering in RAM

* chore(restore): add download diagnostics + exact byte logging

* refactor(restore): drop ProgressTracker, keep streaming download

* feat(restore): drive download progress from RestoreInfo.size

* fix(restore): accept integer or string for RestoreInfo.size

* fix(restore): show raw bytes in download progress log

* fix(restore): collapse progress to highest milestone per chunk

* fix: downloader.rs
2026-06-25 14:55:49 +02:00
github-actions[bot] ea010ce713 chore: release 1.13.0 2026-06-21 14:41:22 +00:00
Charles GTE 8897568281 Merge pull request #66 from Portabase/feat/az-blob-storage-2
feat: az-blob-storage
2026-06-21 16:39:12 +02:00
Charles GTE 66cad4e12e fix: az blog storage tests 2026-06-21 16:22:51 +02:00
Charles GTE 221ed4e7e1 feat: add azure blob storage support 2026-06-20 19:38:00 +02:00
Charles GTE 84735ac399 feat(storage): spike Azure Blob SAS auth + block round-trip against Azurite
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-20 17:49:54 +02:00
Charles GTE f7639de096 build: add azure_storage_blob and azure_core dependencies 2026-06-20 17:39:10 +02:00
github-actions[bot] e504d09cb3 chore: release 1.12.3 2026-06-20 13:00:11 +00:00
Charles GTE 0363b300a4 Merge pull request #65 from Portabase/dev
Dev
2026-06-20 14:58:03 +02:00
Charles GTE bf6e7d41ab fix: Dockerfile 2026-06-20 14:40:37 +02:00
Charles GTE 2c7065ae95 Merge pull request #64 from pcrosthwaite/feature-backup-timeout-on-large-db
Support large MySQL and MariaDb databases
2026-06-20 14:38:26 +02:00
Charles GTE 1fece577cc fix: tests and config.rs and backup.rs args 2026-06-20 14:36:09 +02:00
Charles GTE 273475fa3b Merge branch 'dev' into feature-backup-timeout-on-large-db 2026-06-20 13:02:14 +02:00
Charles GTE 628c017584 Merge pull request #63 from tobfel/tf-main
docs: add Windows build and run guide / and fix(postgres): resolve pg_dump/pg_restore path cross-platform
2026-06-20 12:53:31 +02:00
Charles GTE f02218708a fix: windows-release.yml 2026-06-20 12:52:53 +02:00
Charles GTE 16152328b0 fix: extract the env variable to the settings.rs and some refactoring, and the trigger in windows-release.yml 2026-06-20 12:31:01 +02:00
Peter Crosthwaite 7328827435 Fix mysqldump timeout on large database backups
- Add mysql:8.4 client tools stage to Dockerfile so the real mysqldump
  binary (with --network-timeout) takes PATH precedence over MariaDB's
  mysqldump symlink, which lacks this flag
- Use --network-timeout in MySQL backup command; sets net_read_timeout and
  net_write_timeout to 86400s for the duration of the dump
- Simplify config: replace net_read_timeout/net_write_timeout/max_allowed_packet
  with a single max_packet_size field (optional, defaults to 512M)
- Log mysqldump/mariadb-dump client version as a debug message before each backup
2026-06-20 18:06:21 +10:00
Peter Crosthwaite 348eaac81b Added config options for setting timeouts on mysql/mariadb 2026-06-20 07:51:18 +10:00
tf 0f6c93ecd0 fix encoding 2026-06-19 16:31:21 +02:00
tf befec0deae Windows Build workflow 2026-06-19 16:01:45 +02:00
tf 2052ab0ff5 fix(postgres): resolve pg_dump/pg_restore path cross-platform 2026-06-19 15:27:45 +02:00
tf 461e92d67e docs: add Windows build and run guide 2026-06-19 15:22:24 +02:00
github-actions[bot] 23678bf2d6 chore: release 1.12.2 2026-06-19 10:00:23 +00:00
Charles GTE 02105a8171 Merge pull request #62 from Portabase/fix/password-slash
fix: password-slash
2026-06-19 11:58:26 +02:00
charles-gauthereau f94656a39b fix: postgres password slash 2026-06-19 11:29:51 +02:00
Charles GTE b94ff4e987 Merge branch 'main' into dev 2026-06-09 19:18:38 +02:00
github-actions[bot] dc32c442f3 chore: release 1.12.1 2026-06-07 18:06:37 +00:00
Charles GTE 1453851555 Merge pull request #61 from Portabase/fix/cpu-ram-crash
fix: cpu-ram-crash
2026-06-07 20:04:25 +02:00
Charles GTE 531a25f292 Merge branch 'main' into dev 2026-05-24 16:52:40 +02:00
Charles GTE 740ee43038 fix: add gbak in production Dockerfile 2026-05-17 18:26:29 +02:00
Charles GTE 1fccc0bc19 fix: databases.json 2026-05-15 22:17:02 +02:00
Charles GTE 2c15ea64ae Merge branch 'main' into dev 2026-05-15 22:09:41 +02:00
Charles GTE ac8f7fd8d8 Merge branch 'main' into dev 2026-04-29 22:41:16 +02:00
Charles GTE 6edf2890f1 Merge remote-tracking branch 'origin/dev' into dev 2026-04-26 17:42:52 +02:00
Charles GTE 5f579690ff Merge branch 'main' into dev 2026-04-26 17:42:46 +02:00
charles-gauthereau 5c35375df7 Merge branch 'main' into dev 2026-04-08 19:57:08 +02:00
charles-gauthereau 61c7224104 chore: docker-compose.databases.yml 2026-04-02 19:28:20 +02:00
charles-gauthereau d7958e03b9 Merge branch 'main' into dev 2026-04-01 17:25:32 +02:00
Charles GTE b269d98d3c Merge branch 'main' into dev 2026-03-29 09:32:25 +02:00
Charles GTE 2d44b62844 Merge branch 'main' into dev 2026-03-28 17:23:31 +01:00
Charles GTE 092f760431 Merge branch 'main' into dev 2026-03-27 22:25:48 +01:00
Charles GTE 2c5c805308 Merge branch 'main' into dev 2026-03-21 10:20:08 +01:00
Charles GTE 38a3274c43 Merge branch 'main' into dev 2026-03-21 08:34:41 +01:00
Charles GTE bb45c1961f Merge branch 'main' into dev 2026-03-18 19:11:42 +01:00
charles-gauthereau ee7016d1fa Merge branch 'main' into dev
# Conflicts:
#	docker-compose.yml
2026-03-16 14:36:05 +01:00
charlesgauthereau c46c301371 fix: s3 port type mismatch, add string_or_number_to_string deserializer 2026-03-16 08:55:16 +01:00
120 changed files with 7088 additions and 590 deletions
+1 -1
View File
@@ -36,7 +36,7 @@ jobs:
agent-test bash -c " agent-test bash -c "
mkdir -p /app/coverage && mkdir -p /app/coverage &&
rm -rf /app/target/* /app/coverage/* && rm -rf /app/target/* /app/coverage/* &&
cargo test --verbose && cargo test --verbose -- --test-threads=2 &&
sync sync
" "
+134
View File
@@ -0,0 +1,134 @@
name: GHCR Publish
on:
workflow_call:
inputs:
version:
required: true
type: string
ref:
required: true
type: string
add_latest:
required: false
type: boolean
default: false
dockerfile:
required: false
type: string
default: "./docker/Dockerfile"
target:
required: false
type: string
default: "prod"
permissions:
contents: read
packages: write
jobs:
build:
name: Build ${{ matrix.platform }}
runs-on: ${{ matrix.platform == 'linux/amd64' && 'ubuntu-latest' || 'ubuntu-24.04-arm' }}
strategy:
fail-fast: false
matrix:
platform: [ linux/amd64, linux/arm64 ]
steps:
- name: Prepare vars
id: prep
run: |
ARCH="${{ matrix.platform == 'linux/amd64' && 'amd64' || 'arm64' }}"
echo "arch=$ARCH" >> "$GITHUB_OUTPUT"
echo "image=ghcr.io/${GITHUB_REPOSITORY_OWNER,,}/agent" >> "$GITHUB_OUTPUT"
- name: Checkout
uses: actions/checkout@v4
with:
ref: ${{ inputs.ref }}
fetch-depth: 1
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Log in to GHCR
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build and push by digest
id: build
uses: docker/build-push-action@v6
with:
context: .
file: ${{ inputs.dockerfile }}
platforms: ${{ matrix.platform }}
target: ${{ inputs.target }}
provenance: false
outputs: type=image,name=${{ steps.prep.outputs.image }},push-by-digest=true,name-canonical=true,push=true
cache-from: type=gha,scope=ghcr-${{ steps.prep.outputs.arch }}
cache-to: type=gha,mode=max,scope=ghcr-${{ steps.prep.outputs.arch }},ignore-error=true
- name: Export digest
env:
DIGEST: ${{ steps.build.outputs.digest }}
run: |
mkdir -p /tmp/digests
touch "/tmp/digests/${DIGEST#sha256:}"
- name: Upload digest
uses: actions/upload-artifact@v4
with:
name: digest-${{ steps.prep.outputs.arch }}
path: /tmp/digests/*
if-no-files-found: error
retention-days: 1
merge:
name: Create multi-arch manifest
runs-on: ubuntu-latest
needs: build
steps:
- name: Prepare vars
id: prep
run: echo "image=ghcr.io/${GITHUB_REPOSITORY_OWNER,,}/agent" >> "$GITHUB_OUTPUT"
- name: Download digests
uses: actions/download-artifact@v4
with:
path: /tmp/digests
pattern: digest-*
merge-multiple: true
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Log in to GHCR
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract Docker metadata
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ steps.prep.outputs.image }}
tags: |
type=semver,pattern={{version}},value=${{ inputs.version }}
type=semver,pattern={{major}}.{{minor}},value=${{ inputs.version }}
type=semver,pattern={{major}},value=${{ inputs.version }}
type=raw,value=latest,enable=${{ inputs.add_latest }}
- name: Create and push manifest list
working-directory: /tmp/digests
run: |
docker buildx imagetools create \
$(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
$(printf '${{ steps.prep.outputs.image }}@sha256:%s ' *)
- name: Inspect
run: docker buildx imagetools inspect ${{ steps.prep.outputs.image }}:${{ inputs.version }}
+22
View File
@@ -100,6 +100,15 @@ jobs:
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }} DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }} DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
publish-docker-ghcr:
needs: create-release
if: ${{ needs.create-release.result == 'success' }}
uses: ./.github/workflows/ghcr.yml
with:
version: ${{ needs.create-release.outputs.version }}
ref: ${{ needs.create-release.outputs.version }}
add_latest: true
publish-helm: publish-helm:
needs: create-release needs: create-release
if: ${{ needs.create-release.result == 'success' }} if: ${{ needs.create-release.result == 'success' }}
@@ -109,11 +118,24 @@ jobs:
secrets: secrets:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
build-windows:
needs: create-release
if: ${{ needs.create-release.result == 'success' }}
uses: ./.github/workflows/windows-release.yml
with:
version: ${{ needs.create-release.outputs.version }}
ref: ${{ needs.create-release.outputs.version }}
draft_tag: ${{ needs.create-release.outputs.draft_tag }}
secrets:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
finalize-release: finalize-release:
needs: needs:
- create-release - create-release
- publish-docker - publish-docker
- publish-docker-ghcr
- publish-helm - publish-helm
- build-windows
runs-on: ubuntu-latest runs-on: ubuntu-latest
outputs: outputs:
release_tag: ${{ steps.publish_release_step.outputs.release_tag }} release_tag: ${{ steps.publish_release_step.outputs.release_tag }}
+100
View File
@@ -0,0 +1,100 @@
name: Build Windows release
on:
workflow_call:
inputs:
version:
description: 'Release version (git tag), e.g. 1.18.4'
type: string
required: false
ref:
description: 'Git ref to check out and build'
type: string
required: false
draft_tag:
description: 'Draft GitHub release tag to attach the asset to (e.g. untagged-xxxx). Empty = skip upload.'
type: string
required: false
secrets:
GH_TOKEN:
required: false
workflow_dispatch:
inputs:
ref:
description: 'Git ref to check out and build'
type: string
required: false
jobs:
build-windows:
runs-on: windows-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
ref: ${{ inputs.ref || github.ref }}
- name: Set up Rust toolchain (MSVC)
uses: dtolnay/rust-toolchain@stable
with:
targets: x86_64-pc-windows-msvc
- name: Cache cargo build
uses: Swatinem/rust-cache@v2
- name: Cache vcpkg installed packages
uses: actions/cache@v4
with:
path: C:\vcpkg\installed
key: vcpkg-openssl-x64-windows-v1
- name: Install OpenSSL (x64) via vcpkg
shell: pwsh
run: |
# windows-latest ships vcpkg preinstalled; the install is a no-op when the
# package is restored from cache.
& "$env:VCPKG_INSTALLATION_ROOT\vcpkg.exe" install openssl:x64-windows
'VCPKG_ROOT=C:\vcpkg' | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append
'OPENSSL_DIR=C:\vcpkg\installed\x64-windows' | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append
- name: Build (cargo release)
shell: pwsh
run: cargo build --release --bin app
- name: Prepare artifact zip
id: prepare_artifact
shell: pwsh
env:
RELEASE_VERSION: ${{ inputs.version }}
run: |
$tag = $env:RELEASE_VERSION
if (-not $tag) { $tag = $env:GITHUB_SHA }
$exe = "target\release\app.exe"
if (-not (Test-Path $exe)) { Write-Error "Built binary $exe not found in target/release"; exit 1 }
$outDir = "artifact"
New-Item -ItemType Directory -Path $outDir -Force | Out-Null
# Ship under the package name, not the internal bin name "app".
Copy-Item -Path $exe -Destination "$outDir\portabase-agent.exe"
$zipName = "windows-release-$tag.zip"
if (Test-Path $zipName) { Remove-Item $zipName }
Compress-Archive -Path "$outDir\*" -DestinationPath $zipName -Force
"zip=$zipName" | Out-File -FilePath $env:GITHUB_OUTPUT -Encoding utf8 -Append
- name: Upload build artifact
uses: actions/upload-artifact@v4
with:
name: windows-release
path: ${{ steps.prepare_artifact.outputs.zip }}
- name: Attach asset to draft release
if: ${{ inputs.draft_tag != '' }}
shell: pwsh
env:
GH_TOKEN: ${{ secrets.GH_TOKEN }}
run: |
gh release upload "${{ inputs.draft_tag }}" "${{ steps.prepare_artifact.outputs.zip }}" --clobber
+2 -1
View File
@@ -6,4 +6,5 @@
.env .env
.claude .claude
/docs
/docs
+1 -1
View File
@@ -27,5 +27,5 @@ keywords:
- self-hosted - self-hosted
- portabase - portabase
license: Apache-2.0 license: Apache-2.0
version: 1.12.0 version: 1.20.1
date-released: '2026-02-24' date-released: '2026-02-24'
Generated
+641 -81
View File
File diff suppressed because it is too large Load Diff
+8 -2
View File
@@ -1,6 +1,6 @@
[package] [package]
name = "portabase-agent" name = "portabase-agent"
version = "1.12.0" version = "1.20.1"
edition = "2024" edition = "2024"
[dependencies] [dependencies]
@@ -35,10 +35,14 @@ rand = "0.9.2"
bytes = "1.11.0" bytes = "1.11.0"
async-stream = "0.3.6" async-stream = "0.3.6"
uuid = { version = "1.20.0", features = ["v4"] } uuid = { version = "1.20.0", features = ["v4"] }
tokio-util = { version = "0.7.18", features = ["compat"] } tokio-util = { version = "0.7.18", features = ["compat", "io"] }
tiberius = { version = "0.12", default-features = false, features = ["rustls", "chrono"] } tiberius = { version = "0.12", default-features = false, features = ["rustls", "chrono"] }
aws-config = "1.8.13" aws-config = "1.8.13"
aws-sdk-s3 = { version = "1.122.0", features = ["behavior-version-latest"] } aws-sdk-s3 = { version = "1.122.0", features = ["behavior-version-latest"] }
azure_core = "1.0.0"
azure_storage_blob = "1.0.0"
google-cloud-storage = "1.15"
google-cloud-auth = "1.13"
async-compression = { version = "0.4.37", features = ["tokio", "gzip"] } async-compression = { version = "0.4.37", features = ["tokio", "gzip"] }
tokio-tar = "0.3.1" tokio-tar = "0.3.1"
oauth2 = "5.0.0" oauth2 = "5.0.0"
@@ -53,6 +57,8 @@ testcontainers = "0.27.1"
testcontainers-modules = { version = "0.15.0", features = ["postgres", "redis", "valkey", "mysql", "mariadb", "mongo"] } testcontainers-modules = { version = "0.15.0", features = ["postgres", "redis", "valkey", "mysql", "mariadb", "mongo"] }
postgres = "0.19.12" postgres = "0.19.12"
url = "2.5.8" url = "2.5.8"
percent-encoding = "2.3.2"
bollard = "0.20.0"
[dev-dependencies] [dev-dependencies]
tokio = { version = "1", features = ["full"] } tokio = { version = "1", features = ["full"] }
+20
View File
@@ -10,6 +10,19 @@
"host": "db-postgres", "host": "db-postgres",
"generated_id": "16678159-ff7e-4c97-8c83-0adeff214681" "generated_id": "16678159-ff7e-4c97-8c83-0adeff214681"
}, },
{
"name": "Test database 1 - PostgreSQL - BIS",
"database": "devdb2",
"type": "postgresql",
"username": "devuser2",
"password": "changeme2",
"port": 5432,
"host": "db-postgres-2",
"generated_id": "16678159-ff7e-5697-8c83-0adeff214681",
"options": {
"keep_ownership": true
}
},
{ {
"name": "Test database 2 - MariaDB", "name": "Test database 2 - MariaDB",
"database": "mariadb", "database": "mariadb",
@@ -111,6 +124,13 @@
"port": 1433, "port": 1433,
"host": "db-mssql", "host": "db-mssql",
"generated_id": "16706125-ff7e-4c97-8c83-0adeff214682" "generated_id": "16706125-ff7e-4c97-8c83-0adeff214682"
},
{
"name": "Test database 14 - Docker Volume",
"type": "docker-volume",
"volume_name": "databases_sqlite-data",
"generated_id": "16706126-ff7e-4c97-8c83-0adeff214690",
"container_name": "db-sqlite"
} }
] ]
} }
+15
View File
@@ -15,6 +15,20 @@ services:
networks: networks:
- portabase - portabase
db-postgres-2:
container_name: db-postgres-2
image: postgres:17-alpine
ports:
- "5438:5432"
volumes:
- postgres-data-2:/var/lib/postgresql/data
environment:
- POSTGRES_DB=devdb2
- POSTGRES_USER=devuser2
- POSTGRES_PASSWORD=changeme2
networks:
- portabase
db-mariadb: db-mariadb:
container_name: db-mariadb container_name: db-mariadb
image: mariadb:latest image: mariadb:latest
@@ -179,6 +193,7 @@ services:
volumes: volumes:
postgres-data: postgres-data:
postgres-data-2:
mariadb-data: mariadb-data:
mysql-data: mysql-data:
mongodb-data: mongodb-data:
+6 -47
View File
@@ -1,13 +1,14 @@
services: services:
rust-app: rust-app:
# build: build:
# context: . context: .
# dockerfile: docker/Dockerfile dockerfile: docker/Dockerfile
# target: prod target: prod
image: portabase/agent:latest # image: portabase/agent:latest
container_name: rust-prod container_name: rust-prod
volumes: volumes:
- ./databases.json:/config/config.json - ./databases.json:/config/config.json
- /var/run/docker.sock:/var/run/docker.sock
environment: environment:
LOG: info LOG: info
TZ: "Europe/Paris" TZ: "Europe/Paris"
@@ -18,48 +19,6 @@ services:
networks: networks:
- portabase - portabase
db-mongodb-auth:
container_name: db-mongodb-auth
image: mongo:latest
ports:
- "27082:27017"
environment:
MONGO_INITDB_ROOT_USERNAME: root
MONGO_INITDB_ROOT_PASSWORD: rootpassword
MONGO_INITDB_DATABASE: testdbauth
command: mongod --auth
networks:
- portabase
volumes:
- mongodb-data-auth:/data/db
healthcheck:
test: [ "CMD", "mongo", "--eval", "db.adminCommand('ping')" ]
interval: 5s
timeout: 5s
retries: 10
db-mongodb:
container_name: db-mongodb
image: mongo:latest
ports:
- "27083:27017"
volumes:
- mongodb-data:/data/db
healthcheck:
test: [ "CMD", "mongosh", "--eval", "db.adminCommand('ping')" ]
interval: 5s
timeout: 5s
retries: 10
environment:
MONGO_INITDB_DATABASE: testdb
networks:
- portabase
volumes:
mongodb-data:
mongodb-data-auth:
networks: networks:
portabase: portabase:
name: portabase_network name: portabase_network
+11 -2
View File
@@ -11,22 +11,31 @@ services:
- cargo-git:/usr/local/cargo/git - cargo-git:/usr/local/cargo/git
- ./databases.json:/config/config.json - ./databases.json:/config/config.json
#- ./databases.toml:/config/config.toml #- ./databases.toml:/config/config.toml
#- /var/run/docker.sock:/var/run/docker.sock - /var/run/docker.sock:/var/run/docker.sock
# - cargo-target:/app/target # - cargo-target:/app/target
- databases_sqlite-data:/sqlite-data/workspace/data - databases_sqlite-data:/sqlite-data/workspace/data
- ./scripts/sqlite/test-db:/sqlite-data-2/workspace/data - ./scripts/sqlite/test-db:/sqlite-data-2/workspace/data
# - /bigdisk:/scratch
environment: environment:
APP_ENV: development APP_ENV: development
LOG: debug LOG: debug
TZ: "Europe/Paris" TZ: "Europe/Paris"
EDGE_KEY: "eyJzZXJ2ZXJVcmwiOiJodHRwOi8vbG9jYWxob3N0Ojg4ODciLCJhZ2VudElkIjoiMGZiNDYyMmUtMTMxNS00MzMxLTlkMTMtZWMzMjAyZjZiNTIwIiwibWFzdGVyS2V5QjY0IjoiMUh0djdtWCtYVkJxL0IzUEV2WDlZZjlQeUdVZW5oRHlXemo5THRqNW90WT0ifQ==" # TMPDIR: /scratch
EDGE_KEY: "eyJzZXJ2ZXJVcmwiOiJodHRwOi8vbG9jYWxob3N0Ojg4ODciLCJhZ2VudElkIjoiOWMxMzM5NjItMGE5OC00MmRkLTk1NjUtOTA5ZTkyYTI5N2VkIiwibWFzdGVyS2V5QjY0IjoiMUh0djdtWCtYVkJxL0IzUEV2WDlZZjlQeUdVZW5oRHlXemo5THRqNW90WT0ifQ=="
#CHUNK_SIZE_MB: "1" #CHUNK_SIZE_MB: "1"
#POOLING: 1 #POOLING: 1
#RETRY_ATTEMPTS: 3
#RETRY_BACKOFF_MS: 1000
#DATABASES_CONFIG_FILE: "config.toml" #DATABASES_CONFIG_FILE: "config.toml"
extra_hosts: extra_hosts:
- "localhost:host-gateway" - "localhost:host-gateway"
networks: networks:
- portabase - portabase
cpus: "1.50"
mem_limit: 4g
memswap_limit: 4g
pids_limit: 512
volumes: volumes:
cargo-registry: cargo-registry:
+20
View File
@@ -1,3 +1,11 @@
# =========================
# MySQL client tools
# =========================
FROM mysql:8.4 AS mysql-client-tools
RUN mkdir -p /mysql-exports/bin /mysql-exports/lib \
&& cp /usr/bin/mysqldump /mysql-exports/bin/ \
&& find /usr/lib -name "libmysqlclient.so.21*" -exec cp {} /mysql-exports/lib/ \;
# ========================= # =========================
# Base image (shared) # Base image (shared)
# ========================= # =========================
@@ -68,6 +76,13 @@ RUN if [ "$TARGETARCH" = "amd64" ]; then \
COPY assets/tools/${TARGETARCH}/mongodb/ /usr/local/mongodb/ COPY assets/tools/${TARGETARCH}/mongodb/ /usr/local/mongodb/
RUN chmod +x /usr/local/mongodb/bin/* RUN chmod +x /usr/local/mongodb/bin/*
# =========================
# MySQL real mysqldump binary
# =========================
COPY --from=mysql-client-tools /mysql-exports/bin/mysqldump /usr/local/bin/mysqldump
COPY --from=mysql-client-tools /mysql-exports/lib/ /usr/local/lib/
RUN chmod +x /usr/local/bin/mysqldump && ldconfig
WORKDIR /app WORKDIR /app
# ========================= # =========================
@@ -127,6 +142,8 @@ RUN curl -sSL https://dot.net/v1/dotnet-install.sh -o /tmp/dotnet-install.sh \
WORKDIR /app WORKDIR /app
RUN mkdir -p /config
COPY --from=builder /app/target/release/app /usr/local/bin/app COPY --from=builder /app/target/release/app /usr/local/bin/app
COPY --from=builder /app/version.env /app/version.env COPY --from=builder /app/version.env /app/version.env
COPY entrypoint.sh /entrypoint.sh COPY entrypoint.sh /entrypoint.sh
@@ -135,6 +152,9 @@ RUN chmod +x /entrypoint.sh
COPY --from=base /usr/lib/postgresql/ /usr/lib/postgresql/ COPY --from=base /usr/lib/postgresql/ /usr/lib/postgresql/
COPY --from=base /usr/local/mongodb/bin/ /usr/local/mongodb/bin/ COPY --from=base /usr/local/mongodb/bin/ /usr/local/mongodb/bin/
COPY --from=base /root/.dotnet/tools/ /root/.dotnet/tools/ COPY --from=base /root/.dotnet/tools/ /root/.dotnet/tools/
COPY --from=mysql-client-tools /mysql-exports/bin/mysqldump /usr/local/bin/mysqldump
COPY --from=mysql-client-tools /mysql-exports/lib/ /usr/local/lib/
RUN chmod +x /usr/local/bin/mysqldump && ldconfig
ENV PATH="$PATH:/usr/local/dotnet:/root/.dotnet/tools" ENV PATH="$PATH:/usr/local/dotnet:/root/.dotnet/tools"
ENV APP_ENV=production ENV APP_ENV=production
+3 -1
View File
@@ -7,4 +7,6 @@ data:
TZ: {{ .Values.env.TZ | quote }} TZ: {{ .Values.env.TZ | quote }}
POLLING: {{ .Values.env.POLLING | quote }} POLLING: {{ .Values.env.POLLING | quote }}
APP_ENV: {{ .Values.env.APP_ENV | quote }} APP_ENV: {{ .Values.env.APP_ENV | quote }}
LOG: {{ .Values.env.LOG | quote }} LOG: {{ .Values.env.LOG | quote }}
RETRY_ATTEMPTS: {{ .Values.env.RETRY_ATTEMPTS | quote }}
RETRY_BACKOFF_MS: {{ .Values.env.RETRY_BACKOFF_MS | quote }}
+2
View File
@@ -11,6 +11,8 @@ env:
POLLING: "5" POLLING: "5"
APP_ENV: "production" APP_ENV: "production"
LOG: "info" LOG: "info"
RETRY_ATTEMPTS: "3"
RETRY_BACKOFF_MS: "1000"
resources: resources:
limits: limits:
+8
View File
@@ -45,6 +45,14 @@ seed-firebird:
echo "SELECT RDB\$RELATION_NAME FROM RDB\$RELATIONS WHERE RDB\$SYSTEM_FLAG = 0 AND RDB\$VIEW_BLR IS NULL;" \ echo "SELECT RDB\$RELATION_NAME FROM RDB\$RELATIONS WHERE RDB\$SYSTEM_FLAG = 0 AND RDB\$VIEW_BLR IS NULL;" \
| docker exec -i db-firebird isql -user alice -password fake_password /var/lib/firebird/data/mirror.fdb | docker exec -i db-firebird isql -user alice -password fake_password /var/lib/firebird/data/mirror.fdb
seed-firebird-large:
echo "Seeding Firebird..."
docker exec -i db-firebird isql -user alice -password fake_password /var/lib/firebird/data/mirror.fdb < ./scripts/firebird/seed-large.sql
echo "Verifying Firebird tables..."
echo "SELECT RDB\$RELATION_NAME FROM RDB\$RELATIONS WHERE RDB\$SYSTEM_FLAG = 0 AND RDB\$VIEW_BLR IS NULL;" \
| docker exec -i db-firebird isql -user alice -password fake_password /var/lib/firebird/data/mirror.fdb
seed-mssql: seed-mssql:
echo "Seeding MSSQL..." echo "Seeding MSSQL..."
docker exec -i rust-dev sqlcmd -S "db-mssql,1433" -U sa -P "$MSSQL_SA_PASSWORD" -N disable -i /app/scripts/mssql/seed.sql docker exec -i rust-dev sqlcmd -S "db-mssql,1433" -U sa -P "$MSSQL_SA_PASSWORD" -N disable -i /app/scripts/mssql/seed.sql
+259
View File
@@ -0,0 +1,259 @@
SET SQL DIALECT 3;
SET BAIL ON;
SET AUTODDL OFF;
CREATE TABLE users (
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
email VARCHAR(255) NOT NULL UNIQUE,
name VARCHAR(255),
payload BLOB SUB_TYPE TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
COMMIT;
INSERT INTO users (email, name, payload)
VALUES ('alice@example.com', 'Alice', 'Alice seed data');
INSERT INTO users (email, name, payload)
VALUES ('bob@example.com', 'Bob', 'Bob seed data');
COMMIT;
/*
* Each procedure call generates approximately 128 MiB:
*
* 128 rows
* × 128 chunks per row
* × 8191 bytes per chunk
* = approximately 128 MiB
*
* 40 calls = approximately 5 GiB.
*/
SET TERM ^;
CREATE PROCEDURE seed_users_batch (
p_rows INTEGER,
p_chunks_per_row INTEGER
)
AS
DECLARE VARIABLE v_row_index INTEGER;
DECLARE VARIABLE v_chunk_index INTEGER;
DECLARE VARIABLE v_uuid VARCHAR(36);
DECLARE VARIABLE v_chunk VARCHAR(8191);
DECLARE VARIABLE v_payload BLOB SUB_TYPE TEXT;
BEGIN
v_row_index = 0;
WHILE (v_row_index < p_rows) DO
BEGIN
v_payload = NULL;
v_chunk_index = 0;
WHILE (v_chunk_index < p_chunks_per_row) DO
BEGIN
/*
* Generate a different chunk to avoid producing a completely
* uniform BLOB.
*/
v_chunk = RPAD(
UUID_TO_CHAR(GEN_UUID()),
8191,
UUID_TO_CHAR(GEN_UUID())
);
v_payload = BLOB_APPEND(v_payload, v_chunk);
v_chunk_index = v_chunk_index + 1;
END
v_uuid = UUID_TO_CHAR(GEN_UUID());
INSERT INTO users (
email,
name,
payload
)
VALUES (
:v_uuid || '@example.test',
'Seed User ' || :v_uuid,
:v_payload
);
v_row_index = v_row_index + 1;
END
END^
SET TERM ;^
/* Batch 01 — approximately 128 MiB */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 02 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 03 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 04 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 05 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 06 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 07 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 08 — approximately 1 GiB total */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 09 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 10 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 11 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 12 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 13 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 14 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 15 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 16 — approximately 2 GiB total */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 17 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 18 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 19 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 20 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 21 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 22 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 23 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 24 — approximately 3 GiB total */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 25 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 26 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 27 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 28 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 29 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 30 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 31 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 32 — approximately 4 GiB total */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 33 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 34 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 35 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 36 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 37 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 38 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 39 */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
/* Batch 40 — approximately 5 GiB total */
EXECUTE PROCEDURE seed_users_batch(128, 128);
COMMIT;
DROP PROCEDURE seed_users_batch;
COMMIT;
SELECT
COUNT(*) AS user_count,
CAST(SUM(OCTET_LENGTH(payload)) / 1073741824.0 AS DECIMAL(18, 2))
AS payload_size_gib
FROM users;
COMMIT;
+30 -8
View File
@@ -2,13 +2,16 @@
use crate::core::context::Context; use crate::core::context::Context;
use crate::services::backup::BackupService; use crate::services::backup::BackupService;
use crate::services::config::ConfigService; use crate::services::config::{ConfigService, DatabaseConfig};
use crate::services::cron::CronService; use crate::services::cron::CronService;
use crate::services::dashboard_config::{collect_configs, load_cache, merge, persist_cache};
use crate::services::restore::RestoreService; use crate::services::restore::RestoreService;
use crate::services::status::StatusService; use crate::services::status::StatusService;
use crate::settings::CONFIG;
use crate::utils::common::BackupMethod; use crate::utils::common::BackupMethod;
use std::path::PathBuf;
use std::sync::Arc; use std::sync::Arc;
use tracing::info; use tracing::{error, info, warn};
pub struct Agent { pub struct Agent {
ctx: Arc<Context>, ctx: Arc<Context>,
@@ -17,6 +20,8 @@ pub struct Agent {
cron_service: CronService, cron_service: CronService,
backup_service: BackupService, backup_service: BackupService,
restore_service: RestoreService, restore_service: RestoreService,
dashboard_cache: Vec<DatabaseConfig>,
cache_path: PathBuf,
} }
impl Agent { impl Agent {
@@ -28,6 +33,9 @@ impl Agent {
let backup_service = BackupService::new(ctx.clone()); let backup_service = BackupService::new(ctx.clone());
let restore_service = RestoreService::new(ctx.clone()); let restore_service = RestoreService::new(ctx.clone());
let cache_path = PathBuf::from(&CONFIG.data_path).join("dashboard_databases.json");
let dashboard_cache = load_cache(&cache_path);
Agent { Agent {
ctx, ctx,
config_service, config_service,
@@ -35,19 +43,33 @@ impl Agent {
cron_service, cron_service,
backup_service, backup_service,
restore_service, restore_service,
dashboard_cache,
cache_path,
} }
} }
pub async fn run(&mut self, method: BackupMethod) -> Result<(), Box<dyn std::error::Error>> { pub async fn run(&mut self, method: BackupMethod) -> Result<(), Box<dyn std::error::Error>> {
let config = self.config_service.load(None)?; let local = self.config_service.load_optional(None);
let ping_result = self.status_service.ping(&config.databases).await?;
let merged_in = merge(&local.databases, &self.dashboard_cache);
let ping_result = self.status_service.ping(&merged_in.databases).await?;
self.dashboard_cache = collect_configs(&ping_result);
if let Err(e) = persist_cache(&self.cache_path, &self.dashboard_cache) {
error!("Failed to persist dashboard cache: {e}");
}
let merged = merge(&local.databases, &self.dashboard_cache);
for db in ping_result.databases.iter() { for db in ping_result.databases.iter() {
let database = config let Some(database) = merged
.databases .databases
.iter() .iter()
.find(|cfg_db| cfg_db.generated_id == db.generated_id) .find(|cfg_db| cfg_db.generated_id == db.generated_id)
.unwrap(); else {
warn!("No config for returned database {}; skipping", db.generated_id);
continue;
};
info!( info!(
"Generated Id: {} | backup action: {} | restore action: {} | Database Name: {}", "Generated Id: {} | backup action: {} | restore action: {} | Database Name: {}",
db.generated_id, db.data.backup.action, db.data.restore.action, database.name, db.generated_id, db.data.backup.action, db.data.restore.action, database.name,
@@ -59,14 +81,14 @@ impl Agent {
.backup_service .backup_service
.dispatch( .dispatch(
&db.generated_id, &db.generated_id,
&config, &merged,
method.clone(), method.clone(),
&db.storages, &db.storages,
db.encrypt, db.encrypt,
) )
.await; .await;
} else if db.data.restore.action { } else if db.data.restore.action {
let _ = self.restore_service.dispatch(db, &config).await; let _ = self.restore_service.dispatch(db, &merged).await;
} }
} }
+68
View File
@@ -0,0 +1,68 @@
use crate::domain::docker_volume::docker::{
client, create_helper, remove_helper, resolve_helper_image, start_container, stop_container,
};
use crate::services::backup::logger::JobLogger;
use crate::services::config::DatabaseConfig;
use anyhow::{Context, Result};
use bollard::query_parameters::DownloadFromContainerOptions;
use futures_util::StreamExt;
use std::path::PathBuf;
use std::sync::Arc;
use std::time::Instant;
use tokio::fs::File;
use tokio::io::AsyncWriteExt;
pub async fn run(cfg: DatabaseConfig, backup_dir: PathBuf, logger: Arc<JobLogger>) -> Result<PathBuf> {
tokio::task::spawn_blocking(move || -> Result<PathBuf> {
futures::executor::block_on(async move {
logger.log("info", format!("Starting docker-volume backup for {}", cfg.name));
let docker = client()?;
let image = resolve_helper_image(&docker).await?;
logger.log("debug", format!("Helper image: {image}"));
if let Some(name) = &cfg.container_name {
logger.log("info", format!("Stopping container {name} for consistent backup"));
stop_container(&docker, name).await?;
}
let result = async {
let helper = create_helper(&docker, &image, &cfg.volume_name, &cfg.generated_id, true, None).await?;
let file_path = backup_dir.join(format!("{}.tar", cfg.generated_id));
let start = Instant::now();
let dl_opts = DownloadFromContainerOptions { path: "/vol".to_string() };
let mut stream = docker.download_from_container(&helper.id, Some(dl_opts));
let mut out = File::create(&file_path)
.await
.with_context(|| format!("Failed to create backup file {}", file_path.display()))?;
let mut bytes_written: u64 = 0;
while let Some(chunk) = stream.next().await {
let chunk = chunk.context("Error streaming volume archive from Docker")?;
bytes_written += chunk.len() as u64;
out.write_all(&chunk).await?;
}
out.flush().await?;
let duration_ms = start.elapsed().as_millis() as f64;
logger.log_command("docker download_from_container", None, Some(0), Some(duration_ms));
logger.log("info", format!("Volume backup wrote {bytes_written} bytes to {}", file_path.display()));
remove_helper(&docker, &helper.id).await;
anyhow::Ok(file_path)
}
.await;
if let Some(name) = &cfg.container_name {
if let Err(e) = start_container(&docker, name).await {
logger.log("error", format!("Failed to restart container {name}: {e}"));
}
}
result
})
})
.await?
}
+45
View File
@@ -0,0 +1,45 @@
use anyhow::Result;
use async_trait::async_trait;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use super::{backup, ping, restore};
use crate::domain::factory::Database;
use crate::services::backup::logger::JobLogger;
use crate::services::config::DatabaseConfig;
use crate::utils::locks::{DbOpLock, FileLock};
pub struct DockerVolumeDatabase {
cfg: DatabaseConfig,
}
impl DockerVolumeDatabase {
pub fn new(cfg: DatabaseConfig) -> Self {
Self { cfg }
}
}
#[async_trait]
impl Database for DockerVolumeDatabase {
fn file_extension(&self) -> &'static str {
".tar"
}
async fn ping(&self) -> Result<bool> {
ping::run(self.cfg.clone()).await
}
async fn backup(&self, dir: &Path, logger: Arc<JobLogger>) -> Result<PathBuf> {
FileLock::acquire(&self.cfg.generated_id, DbOpLock::Backup.as_str()).await?;
let res = backup::run(self.cfg.clone(), dir.to_path_buf(), logger).await;
FileLock::release(&self.cfg.generated_id).await?;
res
}
async fn restore(&self, file: &Path, logger: Arc<JobLogger>) -> Result<()> {
FileLock::acquire(&self.cfg.generated_id, DbOpLock::Restore.as_str()).await?;
let res = restore::run(self.cfg.clone(), file.to_path_buf(), logger).await;
FileLock::release(&self.cfg.generated_id).await?;
res
}
}
+169
View File
@@ -0,0 +1,169 @@
#![allow(dead_code)]
use anyhow::{Context, Result};
use bollard::Docker;
use bollard::models::{ContainerCreateBody, HostConfig};
use bollard::query_parameters::{
CreateContainerOptions, InspectContainerOptions, ListContainersOptions,
RemoveContainerOptions, StartContainerOptions, StopContainerOptions,
};
use std::collections::HashMap;
use tracing::{info, warn};
use uuid::Uuid;
pub const EPHEMERAL_LABEL: &str = "io.portabase.ephemeral";
const HELPER_MOUNT: &str = "/vol";
pub fn client() -> Result<Docker> {
Docker::connect_with_defaults().context("Failed to connect to Docker daemon socket")
}
pub fn parse_container_id(mountinfo: &str, cgroup: &str) -> Option<String> {
for src in [mountinfo, cgroup] {
for line in src.lines() {
for marker in ["/containers/", "/docker/"] {
if let Some(idx) = line.find(marker) {
let rest = &line[idx + marker.len()..];
let id: String = rest.chars().take_while(|c| c.is_ascii_hexdigit()).collect();
if id.len() >= 64 {
return Some(id[..64].to_string());
}
}
}
}
}
None
}
pub async fn resolve_helper_image(docker: &Docker) -> Result<String> {
if let Ok(img) = std::env::var("PORTABASE_HELPER_IMAGE") {
if !img.trim().is_empty() {
return Ok(img);
}
}
let mountinfo = std::fs::read_to_string("/proc/self/mountinfo").unwrap_or_default();
let cgroup = std::fs::read_to_string("/proc/self/cgroup").unwrap_or_default();
let id = parse_container_id(&mountinfo, &cgroup).context(
"Could not determine own container id; set PORTABASE_HELPER_IMAGE to a locally-present image",
)?;
let info = docker
.inspect_container(&id, None::<InspectContainerOptions>)
.await
.with_context(|| format!("Failed to inspect self container {id}"))?;
info.image
.context("Self container inspection returned no image reference")
}
pub struct Helper {
pub id: String,
}
pub async fn create_helper(
docker: &Docker,
image: &str,
volume_name: &str,
generated_id: &str,
read_only: bool,
cmd: Option<Vec<String>>,
) -> Result<Helper> {
let bind = format!(
"{volume_name}:{HELPER_MOUNT}{}",
if read_only { ":ro" } else { "" }
);
let mut labels = HashMap::new();
labels.insert(EPHEMERAL_LABEL.to_string(), "true".to_string());
labels.insert("com.docker.compose.project".to_string(), String::new());
labels.insert("com.docker.compose.service".to_string(), String::new());
labels.insert("com.docker.compose.oneoff".to_string(), String::new());
let name = format!(
"portabase-vol-{generated_id}-{}",
&Uuid::new_v4().to_string()[..8]
);
let body = ContainerCreateBody {
image: Some(image.to_string()),
cmd,
labels: Some(labels),
host_config: Some(HostConfig {
binds: Some(vec![bind]),
auto_remove: Some(false),
..Default::default()
}),
..Default::default()
};
let opts = CreateContainerOptions {
name: Some(name),
..Default::default()
};
let res = docker
.create_container(Some(opts), body)
.await
.with_context(|| format!("Failed to create helper container for volume {volume_name}"))?;
Ok(Helper { id: res.id })
}
pub async fn remove_helper(docker: &Docker, id: &str) {
let stop_opts = StopContainerOptions {
t: Some(2),
..Default::default()
};
let _ = docker.stop_container(id, Some(stop_opts)).await;
if let Ok(info) = docker
.inspect_container(id, None::<InspectContainerOptions>)
.await
{
let name = info.name.unwrap_or_default();
let name = name.trim_start_matches('/');
let code = info.state.and_then(|s| s.exit_code).unwrap_or_default();
info!("Helper container {name} exited with code {code}");
}
let opts = RemoveContainerOptions {
force: true,
..Default::default()
};
if let Err(e) = docker.remove_container(id, Some(opts)).await {
warn!("Failed to remove helper container {id}: {e}");
}
}
pub async fn stop_container(docker: &Docker, name: &str) -> Result<()> {
docker
.stop_container(name, None::<StopContainerOptions>)
.await
.with_context(|| format!("Failed to stop container {name}"))
}
pub async fn start_container(docker: &Docker, name: &str) -> Result<()> {
docker
.start_container(name, None::<StartContainerOptions>)
.await
.with_context(|| format!("Failed to start container {name}"))
}
pub async fn sweep_ephemeral(docker: &Docker) -> Result<usize> {
let mut filters = HashMap::new();
filters.insert("label".to_string(), vec![format!("{EPHEMERAL_LABEL}=true")]);
let opts = ListContainersOptions {
all: true,
filters: Some(filters),
..Default::default()
};
let list = docker.list_containers(Some(opts)).await?;
let mut removed = 0;
for c in list {
if let Some(id) = c.id {
remove_helper(docker, &id).await;
removed += 1;
}
}
Ok(removed)
}
+5
View File
@@ -0,0 +1,5 @@
pub mod backup;
pub mod database;
pub mod docker;
pub mod ping;
pub mod restore;
+12
View File
@@ -0,0 +1,12 @@
use crate::domain::docker_volume::docker::client;
use crate::services::config::DatabaseConfig;
use anyhow::Result;
pub async fn run(cfg: DatabaseConfig) -> Result<bool> {
let docker = client()?;
match docker.inspect_volume(&cfg.volume_name).await {
Ok(_) => Ok(true),
Err(bollard::errors::Error::DockerResponseServerError { status_code: 404, .. }) => Ok(false),
Err(e) => Err(e.into()),
}
}
+102
View File
@@ -0,0 +1,102 @@
use crate::domain::docker_volume::docker::{
client, create_helper, remove_helper, resolve_helper_image, start_container, stop_container,
};
use crate::services::backup::logger::JobLogger;
use crate::services::config::DatabaseConfig;
use anyhow::{Context, Result};
use bollard::exec::StartExecResults;
use bollard::models::ExecConfig;
use bollard::query_parameters::UploadToContainerOptions;
use futures_util::StreamExt;
use std::path::PathBuf;
use std::sync::Arc;
use std::time::Instant;
pub async fn run(cfg: DatabaseConfig, archive: PathBuf, logger: Arc<JobLogger>) -> Result<()> {
tokio::task::spawn_blocking(move || -> Result<()> {
futures::executor::block_on(async move {
logger.log("info", format!("Starting docker-volume restore for {}", cfg.name));
let docker = client()?;
let image = resolve_helper_image(&docker).await?;
logger.log("debug", format!("Restore archive: {}", archive.display()));
if let Some(name) = &cfg.container_name {
logger.log("info", format!("Stopping container {name} for restore"));
stop_container(&docker, name).await?;
}
let result = async {
let helper = create_helper(
&docker,
&image,
&cfg.volume_name,
&cfg.generated_id,
false,
Some(vec![
"sh".into(),
"-c".into(),
"trap 'exit 0' TERM; sleep 2147483647 & wait".into(),
]),
)
.await?;
start_container(&docker, &helper.id).await?;
let exec = docker
.create_exec(
&helper.id,
ExecConfig {
cmd: Some(vec![
"sh".to_string(),
"-c".to_string(),
"rm -rf /vol/* /vol/.[!.]* 2>/dev/null || true".to_string(),
]),
attach_stdout: Some(true),
attach_stderr: Some(true),
..Default::default()
},
)
.await
.context("Failed to create wipe exec")?;
if let StartExecResults::Attached { mut output, .. } =
docker.start_exec(&exec.id, None).await.context("Failed to run wipe exec")?
{
while output.next().await.is_some() {}
}
let start = Instant::now();
let file = tokio::fs::File::open(&archive)
.await
.with_context(|| format!("Failed to open {}", archive.display()))?;
let stream = tokio_util::io::ReaderStream::new(file);
let up_opts = UploadToContainerOptions { path: "/".to_string(), ..Default::default() };
docker
.upload_to_container(&helper.id, Some(up_opts), bollard::body_try_stream(stream))
.await
.context("Failed to upload volume archive")?;
let duration_ms = start.elapsed().as_millis() as f64;
logger.log_command("docker upload_to_container", None, Some(0), Some(duration_ms));
remove_helper(&docker, &helper.id).await;
logger.log("info", format!("Volume restore completed for {}", cfg.name));
anyhow::Ok(())
}
.await;
if let Some(name) = &cfg.container_name {
if let Err(e) = start_container(&docker, name).await {
logger.log("error", format!("Failed to restart container {name}: {e}"));
}
}
result
})
})
.await?
}
+6
View File
@@ -1,5 +1,7 @@
use crate::domain::docker_volume::database::DockerVolumeDatabase;
use crate::domain::mongodb::database::MongoDatabase; use crate::domain::mongodb::database::MongoDatabase;
use crate::domain::mysql::database::MySQLDatabase; use crate::domain::mysql::database::MySQLDatabase;
use crate::domain::postgres::cluster::database::PostgresClusterDatabase;
use crate::domain::postgres::database::PostgresDatabase; use crate::domain::postgres::database::PostgresDatabase;
use crate::domain::postgres::{detect_format_from_file, detect_format_from_size}; use crate::domain::postgres::{detect_format_from_file, detect_format_from_size};
use crate::domain::redis::database::RedisDatabase; use crate::domain::redis::database::RedisDatabase;
@@ -31,6 +33,7 @@ impl DatabaseFactory {
let format = detect_format_from_size(&cfg).await; let format = detect_format_from_size(&cfg).await;
Arc::new(PostgresDatabase::new(cfg, format)) Arc::new(PostgresDatabase::new(cfg, format))
} }
DbType::PostgresqlCluster => Arc::new(PostgresClusterDatabase::new(cfg)),
DbType::Mysql => Arc::new(MySQLDatabase::new(cfg)), DbType::Mysql => Arc::new(MySQLDatabase::new(cfg)),
DbType::Mariadb => Arc::new(MariaDBDatabase::new(cfg)), DbType::Mariadb => Arc::new(MariaDBDatabase::new(cfg)),
DbType::MongoDB => Arc::new(MongoDatabase::new(cfg)), DbType::MongoDB => Arc::new(MongoDatabase::new(cfg)),
@@ -39,6 +42,7 @@ impl DatabaseFactory {
DbType::Valkey => Arc::new(ValkeyDatabase::new(cfg)), DbType::Valkey => Arc::new(ValkeyDatabase::new(cfg)),
DbType::Firebird => Arc::new(FirebirdDatabase::new(cfg)), DbType::Firebird => Arc::new(FirebirdDatabase::new(cfg)),
DbType::Mssql => Arc::new(MssqlDatabase::new(cfg)), DbType::Mssql => Arc::new(MssqlDatabase::new(cfg)),
DbType::DockerVolume => Arc::new(DockerVolumeDatabase::new(cfg)),
} }
} }
@@ -48,6 +52,7 @@ impl DatabaseFactory {
let format = detect_format_from_file(restore_file); let format = detect_format_from_file(restore_file);
Arc::new(PostgresDatabase::new(cfg, format)) Arc::new(PostgresDatabase::new(cfg, format))
} }
DbType::PostgresqlCluster => Arc::new(PostgresClusterDatabase::new(cfg)),
DbType::Mysql => Arc::new(MySQLDatabase::new(cfg)), DbType::Mysql => Arc::new(MySQLDatabase::new(cfg)),
DbType::Mariadb => Arc::new(MariaDBDatabase::new(cfg)), DbType::Mariadb => Arc::new(MariaDBDatabase::new(cfg)),
DbType::MongoDB => Arc::new(MongoDatabase::new(cfg)), DbType::MongoDB => Arc::new(MongoDatabase::new(cfg)),
@@ -56,6 +61,7 @@ impl DatabaseFactory {
DbType::Valkey => Arc::new(ValkeyDatabase::new(cfg)), DbType::Valkey => Arc::new(ValkeyDatabase::new(cfg)),
DbType::Firebird => Arc::new(FirebirdDatabase::new(cfg)), DbType::Firebird => Arc::new(FirebirdDatabase::new(cfg)),
DbType::Mssql => Arc::new(MssqlDatabase::new(cfg)), DbType::Mssql => Arc::new(MssqlDatabase::new(cfg)),
DbType::DockerVolume => Arc::new(DockerVolumeDatabase::new(cfg)),
} }
} }
} }
+1
View File
@@ -23,6 +23,7 @@ pub async fn run(
let start = Instant::now(); let start = Instant::now();
let output = Command::new("gbak") let output = Command::new("gbak")
.arg("-b") .arg("-b")
.arg("-g")
.arg("-v") .arg("-v")
.arg("-user").arg(&cfg.username) .arg("-user").arg(&cfg.username)
.arg("-password").arg(&cfg.password) .arg("-password").arg(&cfg.password)
+1
View File
@@ -20,6 +20,7 @@ pub async fn run(cfg: DatabaseConfig) -> anyhow::Result<bool> {
.stdin(Stdio::piped()) .stdin(Stdio::piped())
.stdout(Stdio::piped()) .stdout(Stdio::piped())
.stderr(Stdio::piped()) .stderr(Stdio::piped())
.kill_on_drop(true)
.spawn()?; .spawn()?;
let query = b"SELECT 1 FROM RDB$DATABASE;\nQUIT;\n"; let query = b"SELECT 1 FROM RDB$DATABASE;\nQUIT;\n";
+6 -1
View File
@@ -33,6 +33,11 @@ pub async fn run(
let _mariadb_dump = select_mariadb_path(&version).join("mariadb-dump"); let _mariadb_dump = select_mariadb_path(&version).join("mariadb-dump");
logger.log("debug", format!("Using mariadb-dump at {}", _mariadb_dump.display())); logger.log("debug", format!("Using mariadb-dump at {}", _mariadb_dump.display()));
if let Ok(out) = Command::new("mariadb-dump").arg("--version").output() {
logger.log("debug", format!("mariadb-dump client: {}", String::from_utf8_lossy(&out.stdout).trim()));
}
logger.log("info", format!("Running mariadb-dump for {}", cfg.name)); logger.log("info", format!("Running mariadb-dump for {}", cfg.name));
let start = Instant::now(); let start = Instant::now();
@@ -50,7 +55,7 @@ pub async fn run(
.arg("--skip-add-drop-table") .arg("--skip-add-drop-table")
.arg("--compress") .arg("--compress")
.arg("--verbose") .arg("--verbose")
.arg("--max-allowed-packet=512M") .arg(format!("--max-allowed-packet={}", cfg.max_packet_size))
.arg("--net-buffer-length=16K") .arg("--net-buffer-length=16K")
.arg("--default-character-set=utf8mb4") .arg("--default-character-set=utf8mb4")
.arg(&cfg.database) .arg(&cfg.database)
+2 -1
View File
@@ -12,7 +12,8 @@ pub async fn run(cfg: DatabaseConfig, env: HashMap<String, String>) -> anyhow::R
.arg("--user") .arg("--user")
.arg(cfg.username) .arg(cfg.username)
.arg("ping") .arg("ping")
.envs(env); .envs(env)
.kill_on_drop(true);
let result = timeout(Duration::from_secs(10), cmd.output()).await; let result = timeout(Duration::from_secs(10), cmd.output()).await;
+2 -8
View File
@@ -2,7 +2,6 @@ use crate::services::backup::logger::JobLogger;
use crate::services::config::DatabaseConfig; use crate::services::config::DatabaseConfig;
use anyhow::{Context, Result}; use anyhow::{Context, Result};
use std::fs::File; use std::fs::File;
use std::io::{Read, Write};
use std::path::PathBuf; use std::path::PathBuf;
use std::process::Command; use std::process::Command;
use std::sync::Arc; use std::sync::Arc;
@@ -12,11 +11,8 @@ pub async fn run(cfg: DatabaseConfig, restore_file: PathBuf, logger: Arc<JobLogg
let handle = tokio::task::spawn_blocking(move || -> Result<()> { let handle = tokio::task::spawn_blocking(move || -> Result<()> {
logger.log("info", format!("Starting restore for database {}", cfg.name)); logger.log("info", format!("Starting restore for database {}", cfg.name));
let mut sql_content = String::new();
let mut file = File::open(&restore_file) let mut file = File::open(&restore_file)
.with_context(|| format!("Failed to open restore file {}", restore_file.display()))?; .with_context(|| format!("Failed to open restore file {}", restore_file.display()))?;
file.read_to_string(&mut sql_content)
.with_context(|| format!("Failed to read restore file {}", restore_file.display()))?;
let drop_create_cmd = format!( let drop_create_cmd = format!(
"DROP DATABASE IF EXISTS `{0}`; CREATE DATABASE `{0}`;", "DROP DATABASE IF EXISTS `{0}`; CREATE DATABASE `{0}`;",
@@ -66,10 +62,8 @@ pub async fn run(cfg: DatabaseConfig, restore_file: PathBuf, logger: Arc<JobLogg
.with_context(|| format!("Failed to start MariaDB restore for {}", cfg.name))?; .with_context(|| format!("Failed to start MariaDB restore for {}", cfg.name))?;
let mut stdin = child.stdin.take().context("Failed to open child stdin")?; let mut stdin = child.stdin.take().context("Failed to open child stdin")?;
stdin std::io::copy(&mut file, &mut stdin)
.write_all(sql_content.as_bytes()) .context("Failed to stream SQL content to MariaDB stdin")?;
.context("Failed to write SQL content to MariaDB stdin")?;
stdin.flush()?;
drop(stdin); drop(stdin);
let output = child let output = child
+1
View File
@@ -1,3 +1,4 @@
pub mod docker_volume;
pub mod factory; pub mod factory;
mod mongodb; mod mongodb;
pub mod mysql; pub mod mysql;
+106 -11
View File
@@ -1,6 +1,13 @@
use crate::services::config::DatabaseConfig; use crate::services::config::DatabaseConfig;
use anyhow::Result; use anyhow::Result;
use mongodb::Client; use mongodb::Client;
use percent_encoding::{utf8_percent_encode, AsciiSet, NON_ALPHANUMERIC};
const USERINFO_ENCODE: &AsciiSet = &NON_ALPHANUMERIC
.remove(b'-')
.remove(b'_')
.remove(b'.')
.remove(b'~');
pub async fn connect(cfg: DatabaseConfig) -> Result<Client> { pub async fn connect(cfg: DatabaseConfig) -> Result<Client> {
let uri = get_mongo_uri(cfg)?; let uri = get_mongo_uri(cfg)?;
@@ -16,19 +23,40 @@ pub fn select_mongo_path() -> std::path::PathBuf {
} }
pub fn get_mongo_uri(cfg: DatabaseConfig) -> Result<String> { pub fn get_mongo_uri(cfg: DatabaseConfig) -> Result<String> {
if cfg.username.is_empty() || cfg.password.is_empty() { Ok(build_mongo_uri(&cfg, true))
Ok(format!(
"mongodb://{}:{}/{}",
cfg.host, cfg.port, cfg.database
))
} else {
Ok(format!(
"mongodb://{}:{}@{}:{}/{}?authSource=admin",
cfg.username, cfg.password, cfg.host, cfg.port, cfg.database
))
}
} }
pub fn build_mongo_uri(cfg: &DatabaseConfig, include_db: bool) -> String {
let is_srv = cfg.port == 0;
let scheme = if is_srv { "mongodb+srv" } else { "mongodb" };
let has_auth = !cfg.username.is_empty() && !cfg.password.is_empty();
let credentials = if has_auth {
format!(
"{}:{}@",
utf8_percent_encode(&cfg.username, USERINFO_ENCODE),
utf8_percent_encode(&cfg.password, USERINFO_ENCODE)
)
} else {
String::new()
};
let authority = if is_srv {
cfg.host.clone()
} else {
format!("{}:{}", cfg.host, cfg.port)
};
let path = if include_db {
format!("/{}", cfg.database)
} else {
"/".to_string()
};
let query = if has_auth { "?authSource=admin" } else { "" };
format!("{}://{}{}{}{}", scheme, credentials, authority, path, query)
}
pub fn extract_db_name(dry_output: &str) -> Option<String> { pub fn extract_db_name(dry_output: &str) -> Option<String> {
let mut dbs = std::collections::HashSet::new(); let mut dbs = std::collections::HashSet::new();
@@ -43,3 +71,70 @@ pub fn extract_db_name(dry_output: &str) -> Option<String> {
} }
dbs.into_iter().next() dbs.into_iter().next()
} }
#[cfg(test)]
mod tests {
use super::*;
use crate::services::config::{DatabaseConfig, DbType};
use std::collections::HashMap;
fn cfg(host: &str, port: u16, user: &str, pass: &str) -> DatabaseConfig {
DatabaseConfig {
name: "t".into(),
database: "mydb".into(),
db_type: DbType::MongoDB,
username: user.into(),
password: pass.into(),
port,
host: host.into(),
generated_id: "id".into(),
path: String::new(),
max_packet_size: String::new(),
volume_name: String::new(),
container_name: None,
options: HashMap::new(),
}
}
#[test]
fn standard_with_auth() {
let c = cfg("localhost", 27017, "user", "pass");
assert_eq!(
build_mongo_uri(&c, true),
"mongodb://user:pass@localhost:27017/mydb?authSource=admin"
);
}
#[test]
fn standard_no_auth() {
let c = cfg("localhost", 27017, "", "");
assert_eq!(build_mongo_uri(&c, true), "mongodb://localhost:27017/mydb");
}
#[test]
fn srv_with_auth() {
let c = cfg("cluster.example.mongodb.net", 0, "user", "pass");
assert_eq!(
build_mongo_uri(&c, true),
"mongodb+srv://user:pass@cluster.example.mongodb.net/mydb?authSource=admin"
);
}
#[test]
fn srv_no_db_for_dryrun() {
let c = cfg("cluster.example.mongodb.net", 0, "user", "pass");
assert_eq!(
build_mongo_uri(&c, false),
"mongodb+srv://user:pass@cluster.example.mongodb.net/?authSource=admin"
);
}
#[test]
fn encodes_special_chars_in_credentials() {
let c = cfg("cluster.example.mongodb.net", 0, "user", "p@ss:w/rd?");
assert_eq!(
build_mongo_uri(&c, true),
"mongodb+srv://user:p%40ss%3Aw%2Frd%3F@cluster.example.mongodb.net/mydb?authSource=admin"
);
}
}
+5 -1
View File
@@ -19,7 +19,11 @@ pub async fn run(cfg: DatabaseConfig) -> Result<bool> {
Ok(_) => Ok(true), Ok(_) => Ok(true),
Err(e) => { Err(e) => {
error!("--- MongoDB Connection Error Details ---"); error!("--- MongoDB Connection Error Details ---");
error!("Target Host: {}:{}", cfg.host, cfg.port); if cfg.port == 0 {
error!("Target Host: {} (srv)", cfg.host);
} else {
error!("Target Host: {}:{}", cfg.host, cfg.port);
}
error!("Error Kind: {:?}", e.kind); error!("Error Kind: {:?}", e.kind);
error!("Full Error: {}", e); error!("Full Error: {}", e);
error!("Check you database network connectivity"); error!("Check you database network connectivity");
+4 -8
View File
@@ -1,4 +1,6 @@
use crate::domain::mongodb::connection::{extract_db_name, get_mongo_uri, select_mongo_path}; use crate::domain::mongodb::connection::{
build_mongo_uri, extract_db_name, get_mongo_uri, select_mongo_path,
};
use crate::services::backup::logger::JobLogger; use crate::services::backup::logger::JobLogger;
use crate::services::config::DatabaseConfig; use crate::services::config::DatabaseConfig;
use anyhow::{Context, Result}; use anyhow::{Context, Result};
@@ -16,13 +18,7 @@ pub async fn run(cfg: DatabaseConfig, restore_file: PathBuf, logger: Arc<JobLogg
let dry_start = Instant::now(); let dry_start = Instant::now();
let dry_run = Command::new(&mongorestore) let dry_run = Command::new(&mongorestore)
.arg(format!( .arg(format!("--uri={}", build_mongo_uri(&cfg, false)))
"--uri={}",
format!(
"mongodb://{}:{}@{}:{}/?authSource=admin",
cfg.username, cfg.password, cfg.host, cfg.port
)
))
.arg(format!("--archive={}", restore_file.display())) .arg(format!("--archive={}", restore_file.display()))
.arg("--gzip") .arg("--gzip")
.arg("--dryRun") .arg("--dryRun")
+8
View File
@@ -31,6 +31,10 @@ pub async fn run(
let file_path = backup_dir.join(format!("{}{}", cfg.generated_id, file_extension)); let file_path = backup_dir.join(format!("{}{}", cfg.generated_id, file_extension));
if let Ok(out) = Command::new("mysqldump").arg("--version").output() {
logger.log("debug", format!("mysqldump client: {}", String::from_utf8_lossy(&out.stdout).trim()));
}
logger.log("info", format!("Running mysqldump for {}", cfg.name)); logger.log("info", format!("Running mysqldump for {}", cfg.name));
let start = Instant::now(); let start = Instant::now();
@@ -43,11 +47,15 @@ pub async fn run(
.arg("--triggers") .arg("--triggers")
.arg("--verbose") .arg("--verbose")
.arg("--single-transaction") .arg("--single-transaction")
.arg("--set-gtid-purged=OFF")
.arg("--no-tablespaces")
.arg("--quick") .arg("--quick")
.arg("--skip-lock-tables") .arg("--skip-lock-tables")
.arg("--skip-add-drop-table") .arg("--skip-add-drop-table")
.arg("--no-create-db") .arg("--no-create-db")
.arg("--default-character-set=utf8mb4") .arg("--default-character-set=utf8mb4")
.arg("--network-timeout")
.arg(format!("--max-allowed-packet={}", cfg.max_packet_size))
.arg(&cfg.database) .arg(&cfg.database)
.arg("-r").arg(&file_path) .arg("-r").arg(&file_path)
.envs(env) .envs(env)
+2 -1
View File
@@ -12,7 +12,8 @@ pub async fn run(cfg: DatabaseConfig, env: HashMap<String, String>) -> anyhow::R
.arg("--user") .arg("--user")
.arg(cfg.username) .arg(cfg.username)
.arg("ping") .arg("ping")
.envs(env); .envs(env)
.kill_on_drop(true);
let result = timeout(Duration::from_secs(10), cmd.output()).await; let result = timeout(Duration::from_secs(10), cmd.output()).await;
+2 -8
View File
@@ -2,7 +2,6 @@ use crate::services::backup::logger::JobLogger;
use crate::services::config::DatabaseConfig; use crate::services::config::DatabaseConfig;
use anyhow::{Context, Result}; use anyhow::{Context, Result};
use std::fs::File; use std::fs::File;
use std::io::{Read, Write};
use std::path::PathBuf; use std::path::PathBuf;
use std::process::Command; use std::process::Command;
use std::sync::Arc; use std::sync::Arc;
@@ -12,11 +11,8 @@ pub async fn run(cfg: DatabaseConfig, restore_file: PathBuf, logger: Arc<JobLogg
let handle = tokio::task::spawn_blocking(move || -> Result<()> { let handle = tokio::task::spawn_blocking(move || -> Result<()> {
logger.log("info", format!("Starting restore for database {}", cfg.name)); logger.log("info", format!("Starting restore for database {}", cfg.name));
let mut sql_content = String::new();
let mut file = File::open(&restore_file) let mut file = File::open(&restore_file)
.with_context(|| format!("Failed to open restore file {}", restore_file.display()))?; .with_context(|| format!("Failed to open restore file {}", restore_file.display()))?;
file.read_to_string(&mut sql_content)
.with_context(|| format!("Failed to read restore file {}", restore_file.display()))?;
let drop_create_cmd = format!( let drop_create_cmd = format!(
"DROP DATABASE IF EXISTS `{0}`; CREATE DATABASE `{0}`;", "DROP DATABASE IF EXISTS `{0}`; CREATE DATABASE `{0}`;",
@@ -66,10 +62,8 @@ pub async fn run(cfg: DatabaseConfig, restore_file: PathBuf, logger: Arc<JobLogg
.with_context(|| format!("Failed to start mysql restore for {}", cfg.name))?; .with_context(|| format!("Failed to start mysql restore for {}", cfg.name))?;
let mut stdin = child.stdin.take().context("Failed to open child stdin")?; let mut stdin = child.stdin.take().context("Failed to open child stdin")?;
stdin std::io::copy(&mut file, &mut stdin)
.write_all(sql_content.as_bytes()) .context("Failed to stream SQL content to mysql stdin")?;
.context("Failed to write SQL content to mysql stdin")?;
stdin.flush()?;
drop(stdin); drop(stdin);
let output = child let output = child
+13 -11
View File
@@ -1,4 +1,5 @@
use anyhow::Result; use anyhow::Result;
use std::collections::HashMap;
use std::path::PathBuf; use std::path::PathBuf;
use std::process::Command; use std::process::Command;
use std::sync::Arc; use std::sync::Arc;
@@ -13,6 +14,7 @@ pub async fn run(
cfg: DatabaseConfig, cfg: DatabaseConfig,
format: PostgresDumpFormat, format: PostgresDumpFormat,
backup_dir: PathBuf, backup_dir: PathBuf,
env: HashMap<String, String>,
logger: Arc<JobLogger>, logger: Arc<JobLogger>,
) -> Result<PathBuf> { ) -> Result<PathBuf> {
tokio::task::spawn_blocking(move || -> Result<PathBuf> { tokio::task::spawn_blocking(move || -> Result<PathBuf> {
@@ -37,18 +39,18 @@ pub async fn run(
logger.log("info", format!("Running FC backup for {}", cfg.name)); logger.log("info", format!("Running FC backup for {}", cfg.name));
let file_path = backup_dir.join(format!("{}.dump", cfg.generated_id)); let file_path = backup_dir.join(format!("{}.dump", cfg.generated_id));
let url = format!(
"postgresql://{}:{}@{}:{}/{}",
cfg.username, cfg.password, cfg.host, cfg.port, cfg.database
);
let start = Instant::now(); let start = Instant::now();
let output = Command::new(&pg_dump) let output = Command::new(&pg_dump)
.arg("--dbname").arg(&url) .arg("--host").arg(&cfg.host)
.arg("--port").arg(cfg.port.to_string())
.arg("--username").arg(&cfg.username)
.arg("--dbname").arg(&cfg.database)
.arg("-Fc") .arg("-Fc")
.arg("-f").arg(&file_path) .arg("-f").arg(&file_path)
.arg("-v") .arg("-v")
.arg("--compress=3") .arg("--compress=3")
.envs(env)
.output(); .output();
let duration_ms = start.elapsed().as_millis() as f64; let duration_ms = start.elapsed().as_millis() as f64;
@@ -87,19 +89,19 @@ pub async fn run(
return Err(e.into()); return Err(e.into());
} }
let url = format!( let cmd_label = format!("pg_dump -Fd {}@{}:{}/{}", cfg.username, cfg.host, cfg.port, cfg.database);
"postgresql://{}:{}@{}:{}/{}",
cfg.username, cfg.password, cfg.host, cfg.port, cfg.database
);
let cmd_label = format!("pg_dump -Fd {}", url);
let start = Instant::now(); let start = Instant::now();
let output = Command::new(&pg_dump) let output = Command::new(&pg_dump)
.arg("--dbname").arg(&url) .arg("--host").arg(&cfg.host)
.arg("--port").arg(cfg.port.to_string())
.arg("--username").arg(&cfg.username)
.arg("--dbname").arg(&cfg.database)
.arg("-Fd") .arg("-Fd")
.arg("-j").arg("4") .arg("-j").arg("4")
.arg("-f").arg(&dump_dir) .arg("-f").arg(&dump_dir)
.arg("-v") .arg("-v")
.envs(env)
.output(); .output();
let duration_ms = start.elapsed().as_millis() as f64; let duration_ms = start.elapsed().as_millis() as f64;
+25
View File
@@ -0,0 +1,25 @@
use crate::services::config::DatabaseConfig;
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum RestoreCleanMode {
None,
Clean,
DropSchemas,
DropDatabase,
}
impl RestoreCleanMode {
pub fn from_config(cfg: &DatabaseConfig) -> (Self, Option<String>) {
match cfg.options.get("clean_mode").and_then(|v| v.as_str()) {
None | Some("clean") => (Self::Clean, None),
Some("none") => (Self::None, None),
Some("drop_schemas") => (Self::DropSchemas, None),
Some("drop_database") => (Self::DropDatabase, None),
Some(other) => (Self::Clean, Some(other.to_string())),
}
}
pub fn uses_pg_restore_clean(self) -> bool {
matches!(self, Self::Clean)
}
}
+81
View File
@@ -0,0 +1,81 @@
use anyhow::Result;
use std::collections::HashMap;
use std::path::PathBuf;
use std::process::Command;
use std::sync::Arc;
use std::time::Instant;
use super::super::connection::{
is_superuser, pg_dumpall_binary_name, select_pg_path, server_version,
};
use crate::services::backup::logger::JobLogger;
use crate::services::config::DatabaseConfig;
pub async fn run(
cfg: DatabaseConfig,
backup_dir: PathBuf,
env: HashMap<String, String>,
logger: Arc<JobLogger>,
) -> Result<PathBuf> {
tokio::task::spawn_blocking(move || -> Result<PathBuf> {
logger.log("info", format!("Starting cluster backup for {}", cfg.name));
let version = match futures::executor::block_on(server_version(&cfg)) {
Ok(v) => v,
Err(e) => {
logger.log("error", format!("Failed to get server version for {}: {:?}", cfg.name, e));
return Err(e.into());
}
};
match futures::executor::block_on(is_superuser(&cfg)) {
Ok(true) => {}
Ok(false) => {
logger.log("error", format!("postgresql-cluster backup requires a superuser role for {}", cfg.name));
anyhow::bail!("postgresql-cluster backup requires a superuser role for {}", cfg.name);
}
Err(e) => {
logger.log("error", format!("Failed to check superuser status for {}: {:?}", cfg.name, e));
return Err(e.into());
}
}
let pg_dumpall = select_pg_path(&version).join(pg_dumpall_binary_name());
let file_path = backup_dir.join(format!("{}.sql", cfg.generated_id));
logger.log("info", format!("Running pg_dumpall for cluster {} via {:?}", cfg.name, pg_dumpall));
let start = Instant::now();
let output = Command::new(&pg_dumpall)
.arg("--host").arg(&cfg.host)
.arg("--port").arg(cfg.port.to_string())
.arg("--username").arg(&cfg.username)
.arg("--clean")
.arg("--if-exists")
.arg("-v")
.arg("-f").arg(&file_path)
.envs(env)
.output();
let duration_ms = start.elapsed().as_millis() as f64;
match output {
Ok(o) => {
let stderr = String::from_utf8_lossy(&o.stderr).to_string();
let exit_code = o.status.code().unwrap_or(-1);
if o.status.success() {
logger.log_command("pg_dumpall", if stderr.is_empty() { None } else { Some(stderr) }, Some(0), Some(duration_ms));
logger.log("info", format!("Cluster backup completed for {} at {:?}", cfg.name, file_path));
Ok(file_path)
} else {
logger.log_command("pg_dumpall", Some(stderr), Some(exit_code), Some(duration_ms));
anyhow::bail!("Cluster backup (pg_dumpall) failed for {}", cfg.name);
}
}
Err(e) => {
logger.log_command("pg_dumpall", Some(e.to_string()), Some(-1), Some(duration_ms));
Err(e.into())
}
}
})
.await?
}
+53
View File
@@ -0,0 +1,53 @@
use anyhow::Result;
use async_trait::async_trait;
use std::collections::HashMap;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use super::super::ping;
use super::{backup, restore};
use crate::domain::factory::Database;
use crate::services::backup::logger::JobLogger;
use crate::services::config::DatabaseConfig;
use crate::utils::locks::{DbOpLock, FileLock};
pub struct PostgresClusterDatabase {
pub cfg: DatabaseConfig,
}
impl PostgresClusterDatabase {
pub fn new(cfg: DatabaseConfig) -> Self {
Self { cfg }
}
fn build_env(&self) -> HashMap<String, String> {
let mut envs = std::env::vars().collect::<HashMap<_, _>>();
envs.insert("PGPASSWORD".to_string(), self.cfg.password.to_string());
envs
}
}
#[async_trait]
impl Database for PostgresClusterDatabase {
fn file_extension(&self) -> &'static str {
".sql"
}
async fn ping(&self) -> Result<bool> {
ping::run(self.cfg.clone()).await
}
async fn backup(&self, dir: &Path, logger: Arc<JobLogger>) -> Result<PathBuf> {
FileLock::acquire(&self.cfg.generated_id, DbOpLock::Backup.as_str()).await?;
let res = backup::run(self.cfg.clone(), dir.to_path_buf(), self.build_env(), logger).await;
FileLock::release(&self.cfg.generated_id).await?;
res
}
async fn restore(&self, file: &Path, logger: Arc<JobLogger>) -> Result<()> {
FileLock::acquire(&self.cfg.generated_id, DbOpLock::Restore.as_str()).await?;
let res = restore::run(self.cfg.clone(), file.to_path_buf(), self.build_env(), logger).await;
FileLock::release(&self.cfg.generated_id).await?;
res
}
}
+3
View File
@@ -0,0 +1,3 @@
pub mod backup;
pub mod database;
pub mod restore;
+85
View File
@@ -0,0 +1,85 @@
use anyhow::Result;
use std::collections::HashMap;
use std::path::PathBuf;
use std::process::Command;
use std::sync::Arc;
use std::time::Instant;
use super::super::connection::{is_superuser, psql_binary_name, select_pg_path, server_version, terminate_all_connections};
use crate::services::backup::logger::JobLogger;
use crate::services::config::DatabaseConfig;
pub async fn run(
cfg: DatabaseConfig,
restore_file: PathBuf,
env: HashMap<String, String>,
logger: Arc<JobLogger>,
) -> Result<()> {
let handle = tokio::runtime::Handle::current();
tokio::task::spawn_blocking(move || -> Result<()> {
logger.log("info", format!("Starting cluster restore for {}", cfg.name));
let version = match handle.block_on(server_version(&cfg)) {
Ok(v) => v,
Err(e) => {
logger.log("error", format!("Failed to get server version for {}: {:?}", cfg.name, e));
return Err(e.into());
}
};
match handle.block_on(is_superuser(&cfg)) {
Ok(true) => {}
Ok(false) => {
logger.log("error", format!("postgresql-cluster restore requires a superuser role for {}", cfg.name));
anyhow::bail!("postgresql-cluster restore requires a superuser role for {}", cfg.name);
}
Err(e) => {
logger.log("error", format!("Failed to check superuser status for {}: {:?}", cfg.name, e));
return Err(e.into());
}
}
let psql = select_pg_path(&version).join(psql_binary_name());
if let Err(e) = handle.block_on(terminate_all_connections(&cfg)) {
logger.log("error", format!("Failed to terminate connections for cluster {}: {:?}", cfg.name, e));
return Err(e.into());
}
logger.log("info", format!("All user database connections terminated for cluster {}", cfg.name));
logger.log("info", format!("Replaying cluster dump for {} via {:?}", cfg.name, psql));
let start = Instant::now();
let output = Command::new(&psql)
.arg("--host").arg(&cfg.host)
.arg("--port").arg(cfg.port.to_string())
.arg("--username").arg(&cfg.username)
.arg("--dbname").arg("postgres")
.arg("-f").arg(&restore_file)
.envs(env)
.output();
let duration_ms = start.elapsed().as_millis() as f64;
match output {
Ok(o) => {
let stderr = String::from_utf8_lossy(&o.stderr).to_string();
let stdout = String::from_utf8_lossy(&o.stdout).to_string();
let combined = format!("{}{}", stdout, stderr);
let exit_code = o.status.code().unwrap_or(-1);
if o.status.success() {
logger.log_command("psql", if combined.is_empty() { None } else { Some(combined) }, Some(0), Some(duration_ms));
logger.log("info", format!("Cluster restore completed for {}", cfg.name));
Ok(())
} else {
logger.log_command("psql", if combined.is_empty() { None } else { Some(combined) }, Some(exit_code), Some(duration_ms));
anyhow::bail!("Cluster restore (psql) failed for {}", cfg.name);
}
}
Err(e) => {
logger.log_command("psql", Some(e.to_string()), Some(-1), Some(duration_ms));
Err(e.into())
}
}
})
.await?
}
+294 -6
View File
@@ -1,18 +1,23 @@
use crate::domain::postgres::format::PostgresDumpFormat; use crate::domain::postgres::format::PostgresDumpFormat;
use crate::services::config::DatabaseConfig; use crate::services::config::DatabaseConfig;
use crate::settings::CONFIG;
use anyhow::Result; use anyhow::Result;
use std::path::Path; use std::path::Path;
use tokio_postgres::{Client, NoTls}; use tokio_postgres::{Client, Config, NoTls};
use tracing::{error, info}; use tracing::{error, info};
pub async fn connect(cfg: &DatabaseConfig) -> Result<Client> { pub async fn connect(cfg: &DatabaseConfig) -> Result<Client> {
info!("Connecting to postgres database {}:{}", cfg.host, cfg.port); info!("Connecting to postgres database {}:{}", cfg.host, cfg.port);
let dsn = format!(
"host={} port={} user={} password={} dbname={}",
cfg.host, cfg.port, cfg.username, cfg.password, cfg.database
);
let (client, connection) = tokio_postgres::connect(&dsn, NoTls).await?; let mut config = Config::new();
config
.host(&cfg.host)
.port(cfg.port)
.user(&cfg.username)
.password(&cfg.password)
.dbname(&cfg.database);
let (client, connection) = config.connect(NoTls).await?;
tokio::spawn(async move { tokio::spawn(async move {
if let Err(e) = connection.await { if let Err(e) = connection.await {
error!("Postgres connection error: {}", e); error!("Postgres connection error: {}", e);
@@ -28,11 +33,133 @@ pub async fn server_version(cfg: &DatabaseConfig) -> Result<String> {
Ok(version) Ok(version)
} }
pub async fn server_version_major(cfg: &DatabaseConfig) -> Result<u32> {
let v = server_version(cfg).await?;
Ok(v.split(['.', ' '])
.next()
.and_then(|s| s.parse().ok())
.unwrap_or(17))
}
pub async fn is_superuser(cfg: &DatabaseConfig) -> Result<bool> {
let client = connect(cfg).await?;
let is_super: bool = client
.query_one("SELECT current_setting('is_superuser') = 'on';", &[])
.await?
.get(0);
Ok(is_super)
}
pub async fn can_drop_database(cfg: &DatabaseConfig) -> Result<bool> {
let client = connect(cfg).await?;
let row = client
.query_one(
"SELECT r.rolsuper OR (r.rolcreatedb AND pg_catalog.pg_has_role(current_user, d.datdba, 'USAGE')) \
FROM pg_roles r, pg_database d \
WHERE r.rolname = current_user AND d.datname = current_database()",
&[],
)
.await?;
Ok(row.get(0))
}
pub fn select_pg_path(version: &str) -> std::path::PathBuf { pub fn select_pg_path(version: &str) -> std::path::PathBuf {
select_pg_path_with(version, &CONFIG.pg_bin_dir)
}
pub(crate) fn select_pg_path_with(version: &str, pg_bin_dir: &str) -> std::path::PathBuf {
let major = version.split('.').next().unwrap_or("17"); let major = version.split('.').next().unwrap_or("17");
if !pg_bin_dir.is_empty() {
return pg_bin_dir.into();
}
let candidates: Vec<std::path::PathBuf> = if cfg!(target_os = "windows") {
vec![
// Default install path used by the official EDB Windows installer
format!(r"C:\Program Files\PostgreSQL\{major}\bin").into(),
format!(r"C:\Program Files (x86)\PostgreSQL\{major}\bin").into(),
]
} else if cfg!(target_os = "macos") {
vec![
// Homebrew on Apple Silicon
format!("/opt/homebrew/opt/postgresql@{major}/bin").into(),
// Homebrew on Intel
format!("/usr/local/opt/postgresql@{major}/bin").into(),
// Postgres.app
format!("/Applications/Postgres.app/Contents/Versions/{major}/bin").into(),
]
} else {
vec![
// Debian/Ubuntu packages
format!("/usr/lib/postgresql/{major}/bin").into(),
// Common RPM-based distro layout
format!("/usr/pgsql-{major}/bin").into(),
]
};
if let Some(found) = candidates.into_iter().find(|p| pg_dump_exists_in(p)) {
return found;
}
if let Some(dir) = find_pg_dump_in_path() {
return dir;
}
format!("/usr/lib/postgresql/{}/bin", major).into() format!("/usr/lib/postgresql/{}/bin", major).into()
} }
pub(crate) fn pg_dump_binary_name() -> &'static str {
if cfg!(target_os = "windows") {
"pg_dump.exe"
} else {
"pg_dump"
}
}
pub(crate) fn pg_dumpall_binary_name() -> &'static str {
if cfg!(target_os = "windows") {
"pg_dumpall.exe"
} else {
"pg_dumpall"
}
}
pub(crate) fn psql_binary_name() -> &'static str {
if cfg!(target_os = "windows") {
"psql.exe"
} else {
"psql"
}
}
pub(crate) fn pg_restore_binary_name() -> &'static str {
if cfg!(target_os = "windows") {
"pg_restore.exe"
} else {
"pg_restore"
}
}
pub(crate) fn quote_ident(s: &str) -> String {
format!("\"{}\"", s.replace('"', "\"\""))
}
pub(crate) fn quote_literal(s: &str) -> String {
format!("'{}'", s.replace('\'', "''"))
}
pub(crate) fn pg_dump_exists_in(dir: &std::path::Path) -> bool {
dir.join(pg_dump_binary_name()).is_file()
}
fn find_pg_dump_in_path() -> Option<std::path::PathBuf> {
let path_var = std::env::var_os("PATH")?;
std::env::split_paths(&path_var).find(|dir| pg_dump_exists_in(dir))
}
pub async fn terminate_connections(cfg: &DatabaseConfig) -> Result<()> { pub async fn terminate_connections(cfg: &DatabaseConfig) -> Result<()> {
let mut admin = cfg.clone(); let mut admin = cfg.clone();
admin.database = "postgres".to_string().into(); admin.database = "postgres".to_string().into();
@@ -54,6 +181,128 @@ pub async fn terminate_connections(cfg: &DatabaseConfig) -> Result<()> {
Ok(()) Ok(())
} }
pub async fn terminate_all_connections(cfg: &DatabaseConfig) -> Result<()> {
let mut admin = cfg.clone();
admin.database = "postgres".to_string().into();
let client = connect(&admin).await?;
client
.execute(
r#"
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE datname NOT IN ('postgres', 'template0', 'template1')
AND pid <> pg_backend_pid();
"#,
&[],
)
.await?;
Ok(())
}
pub async fn drop_and_recreate_database(cfg: &DatabaseConfig) -> Result<()> {
let mut admin_cfg = cfg.clone();
admin_cfg.database = "postgres".to_string();
let admin = connect(&admin_cfg).await?;
let row = admin
.query_opt(
r#"
SELECT pg_encoding_to_char(encoding), datcollate, datctype,
pg_get_userbyid(datdba), datistemplate
FROM pg_database WHERE datname = $1
"#,
&[&cfg.database],
)
.await?;
let (encoding, collate, ctype, owner) = match &row {
Some(r) => (
r.get::<_, String>(0),
r.get::<_, String>(1),
r.get::<_, String>(2),
r.get::<_, String>(3),
),
None => ("UTF8".into(), "C".into(), "C".into(), cfg.username.clone()),
};
if let Some(r) = &row {
if r.get::<_, bool>(4) {
anyhow::bail!("Refusing to drop template database {}", cfg.database);
}
}
let db = quote_ident(&cfg.database);
if let Err(e) = admin
.batch_execute(&format!("ALTER DATABASE {db} WITH ALLOW_CONNECTIONS false"))
.await
{
tracing::warn!("ALLOW_CONNECTIONS false failed for {}: {e}", cfg.database);
}
let major = server_version_major(&admin_cfg).await?;
let drop_stmt = if major >= 13 {
format!("DROP DATABASE IF EXISTS {db} WITH (FORCE)")
} else {
format!("DROP DATABASE IF EXISTS {db}")
};
let mut last_err = None;
let mut dropped = false;
for _ in 0..3 {
let _ = terminate_connections(cfg).await;
match admin.batch_execute(&drop_stmt).await {
Ok(()) => {
dropped = true;
break;
}
Err(e) => {
last_err = Some(e);
tokio::time::sleep(std::time::Duration::from_millis(500)).await;
}
}
}
if !dropped {
let _ = admin
.batch_execute(&format!("ALTER DATABASE {db} WITH ALLOW_CONNECTIONS true"))
.await;
return Err(last_err
.map(anyhow::Error::from)
.unwrap_or_else(|| anyhow::anyhow!("DROP DATABASE {} failed", cfg.database)));
}
admin
.batch_execute(&format!(
"CREATE DATABASE {db} OWNER {} TEMPLATE template0 ENCODING {} LC_COLLATE {} LC_CTYPE {}",
quote_ident(&owner),
quote_literal(&encoding),
quote_literal(&collate),
quote_literal(&ctype),
))
.await?;
Ok(())
}
pub fn sniff_format(restore_file: &Path) -> Result<PostgresDumpFormat> {
use std::io::Read;
let mut f = std::fs::File::open(restore_file)?;
let mut magic = [0u8; 5];
let n = f.read(&mut magic)?;
let head = &magic[..n];
if head.starts_with(b"PGDMP") {
Ok(PostgresDumpFormat::Fc)
} else if head.starts_with(&[0x1f, 0x8b]) {
Ok(PostgresDumpFormat::Fd)
} else {
anyhow::bail!("Unrecognized dump format for {:?}", restore_file)
}
}
pub fn detect_format_from_file(restore_file: &Path) -> PostgresDumpFormat { pub fn detect_format_from_file(restore_file: &Path) -> PostgresDumpFormat {
match restore_file.extension().and_then(|e| e.to_str()) { match restore_file.extension().and_then(|e| e.to_str()) {
Some("dump") => PostgresDumpFormat::Fc, Some("dump") => PostgresDumpFormat::Fc,
@@ -63,11 +312,50 @@ pub fn detect_format_from_file(restore_file: &Path) -> PostgresDumpFormat {
} }
} }
pub async fn drop_all_schemas(cfg: &DatabaseConfig) -> Result<Vec<String>> {
let client = connect(cfg).await?;
let rows = client
.query(
r#"
SELECT nspname FROM pg_namespace
WHERE nspname NOT IN ('pg_catalog', 'information_schema', 'pg_toast')
AND nspname NOT LIKE 'pg\_temp\_%'
AND nspname NOT LIKE 'pg\_toast\_temp\_%'
ORDER BY nspname
"#,
&[],
)
.await?;
let schemas: Vec<String> = rows.iter().map(|r| r.get::<_, String>(0)).collect();
for s in &schemas {
client
.batch_execute(&format!("DROP SCHEMA IF EXISTS {} CASCADE", quote_ident(s)))
.await?;
}
client
.batch_execute("SELECT lo_unlink(oid) FROM pg_largeobject_metadata")
.await
.ok();
Ok(schemas)
}
pub async fn recreate_public_schema(cfg: &DatabaseConfig, owner: &str) -> Result<()> {
let client = connect(cfg).await?;
client
.batch_execute(&format!(
"CREATE SCHEMA IF NOT EXISTS public AUTHORIZATION {}; GRANT USAGE ON SCHEMA public TO PUBLIC;",
quote_ident(owner)
))
.await?;
Ok(())
}
pub async fn detect_format_from_size(cfg: &DatabaseConfig) -> PostgresDumpFormat { pub async fn detect_format_from_size(cfg: &DatabaseConfig) -> PostgresDumpFormat {
info!( info!(
"Detecting database format {:?} - {:?}", "Detecting database format {:?} - {:?}",
cfg.name, cfg.generated_id cfg.name, cfg.generated_id
); );
let client = match connect(cfg).await { let client = match connect(cfg).await {
Ok(c) => c, Ok(c) => c,
Err(_) => return PostgresDumpFormat::Fc, Err(_) => return PostgresDumpFormat::Fc,
+23 -3
View File
@@ -1,8 +1,8 @@
use anyhow::Result; use anyhow::Result;
use async_trait::async_trait; use async_trait::async_trait;
use std::collections::HashMap;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use std::sync::Arc; use std::sync::Arc;
use super::{backup, format::PostgresDumpFormat, ping, restore}; use super::{backup, format::PostgresDumpFormat, ping, restore};
use crate::domain::factory::Database; use crate::domain::factory::Database;
use crate::services::backup::logger::JobLogger; use crate::services::backup::logger::JobLogger;
@@ -18,6 +18,12 @@ impl PostgresDatabase {
pub fn new(cfg: DatabaseConfig, format: PostgresDumpFormat) -> Self { pub fn new(cfg: DatabaseConfig, format: PostgresDumpFormat) -> Self {
Self { cfg, format } Self { cfg, format }
} }
fn build_env(&self) -> HashMap<String, String> {
let mut envs = std::env::vars().collect::<HashMap<_, _>>();
envs.insert("PGPASSWORD".to_string(), self.cfg.password.to_string());
envs
}
} }
#[async_trait] #[async_trait]
@@ -35,14 +41,28 @@ impl Database for PostgresDatabase {
async fn backup(&self, dir: &Path, logger: Arc<JobLogger>) -> Result<PathBuf> { async fn backup(&self, dir: &Path, logger: Arc<JobLogger>) -> Result<PathBuf> {
FileLock::acquire(&self.cfg.generated_id, DbOpLock::Backup.as_str()).await?; FileLock::acquire(&self.cfg.generated_id, DbOpLock::Backup.as_str()).await?;
let res = backup::run(self.cfg.clone(), self.format, dir.to_path_buf(), logger).await; let res = backup::run(
self.cfg.clone(),
self.format,
dir.to_path_buf(),
self.build_env(),
logger,
)
.await;
FileLock::release(&self.cfg.generated_id).await?; FileLock::release(&self.cfg.generated_id).await?;
res res
} }
async fn restore(&self, file: &Path, logger: Arc<JobLogger>) -> Result<()> { async fn restore(&self, file: &Path, logger: Arc<JobLogger>) -> Result<()> {
FileLock::acquire(&self.cfg.generated_id, DbOpLock::Restore.as_str()).await?; FileLock::acquire(&self.cfg.generated_id, DbOpLock::Restore.as_str()).await?;
let res = restore::run(self.cfg.clone(), self.format, file.to_path_buf(), logger).await; let res = restore::run(
self.cfg.clone(),
self.format,
file.to_path_buf(),
self.build_env(),
logger,
)
.await;
FileLock::release(&self.cfg.generated_id).await?; FileLock::release(&self.cfg.generated_id).await?;
res res
} }
+1 -1
View File
@@ -1,4 +1,4 @@
#[derive(Clone, Copy)] #[derive(Clone, Copy, PartialEq, Debug)]
pub enum PostgresDumpFormat { pub enum PostgresDumpFormat {
Fc, Fc,
Fd, Fd,
+5 -3
View File
@@ -1,8 +1,10 @@
pub mod backup; pub mod backup;
mod connection; pub(crate) mod cluster;
pub(crate) mod clean_mode;
pub(crate) mod connection;
pub mod database; pub mod database;
mod format; pub(crate) mod format;
mod ping; mod ping;
mod restore; pub(crate) mod restore;
pub use connection::{detect_format_from_file, detect_format_from_size}; pub use connection::{detect_format_from_file, detect_format_from_size};
-198
View File
@@ -1,198 +0,0 @@
use anyhow::Result;
use std::path::PathBuf;
use std::process::Command;
use std::sync::Arc;
use std::time::Instant;
use super::connection::{select_pg_path, server_version, terminate_connections};
use super::format::PostgresDumpFormat;
use crate::services::backup::logger::JobLogger;
use crate::services::config::DatabaseConfig;
pub async fn run(
cfg: DatabaseConfig,
format: PostgresDumpFormat,
restore_file: PathBuf,
logger: Arc<JobLogger>,
) -> Result<()> {
tokio::task::spawn_blocking(move || -> Result<()> {
logger.log("info", format!("Starting restore for database {}", cfg.name));
let version = match futures::executor::block_on(server_version(&cfg)) {
Ok(v) => {
logger.log("debug", format!("Postgres version detected: {}", v));
v
}
Err(e) => {
logger.log("error", format!("Failed to get server version for {}: {:?}", cfg.name, e));
return Err(e.into());
}
};
let pg_restore = select_pg_path(&version).join("pg_restore");
logger.log("debug", format!("Using pg_restore at {:?}", pg_restore));
if let Err(e) = futures::executor::block_on(terminate_connections(&cfg)) {
logger.log("error", format!("Failed to terminate connections for {}: {:?}", cfg.name, e));
return Err(e.into());
}
logger.log("info", format!("Connections terminated for database {}", cfg.name));
let url = format!(
"postgresql://{}:{}@{}:{}/{}",
cfg.username, cfg.password, cfg.host, cfg.port, cfg.database
);
match format {
PostgresDumpFormat::Fc => {
logger.log("info", format!("Running FC restore for {}", cfg.name));
let start = Instant::now();
let output = Command::new(&pg_restore)
.arg("--no-owner")
.arg("--no-privileges")
.arg("--clean")
.arg("--if-exists")
// .arg("--create")
.arg("--dbname")
.arg(&url)
.arg("-v")
.arg(&restore_file)
.env("PGPASSWORD", &cfg.password)
.output();
let duration_ms = start.elapsed().as_millis() as f64;
match output {
Ok(o) => {
let stderr = String::from_utf8_lossy(&o.stderr).to_string();
let stdout = String::from_utf8_lossy(&o.stdout).to_string();
let combined = format!("{}{}", stdout, stderr);
let exit_code = o.status.code().unwrap_or(-1);
if o.status.success() {
logger.log_command("pg_restore", if combined.is_empty() { None } else { Some(combined) }, Some(0), Some(duration_ms));
logger.log("info", format!("FC restore completed successfully for {}", cfg.name))
} else {
logger.log_command("pg_restore", if combined.is_empty() { None } else { Some(combined) }, Some(exit_code), Some(duration_ms));
logger.log("error", format!("FC restore failed with status {:?} for {}", o.status, cfg.name));
anyhow::bail!("Postgres restore failed for {}", cfg.name);
}
}
Err(e) => {
logger.log_command("pg_restore", Some(e.to_string()), Some(-1), Some(duration_ms));
logger.log("error", format!("Error executing pg_restore for {}: {:?}", cfg.name, e));
return Err(e.into());
}
}
}
PostgresDumpFormat::Fd => {
logger.log("info", format!("Running FD restore for {}", cfg.name));
let tar_gz = match std::fs::File::open(&restore_file) {
Ok(f) => f,
Err(e) => {
logger.log("error", format!(
"Failed to open restore file {:?} for {}: {:?}",
restore_file, cfg.name, e
));
return Err(e.into());
}
};
logger.log("info", format!("tar_gz {:?}", tar_gz));
let dec = flate2::read::GzDecoder::new(tar_gz);
let mut archive = tar::Archive::new(dec);
let tmp_dir = match tempfile::TempDir::new() {
Ok(d) => d,
Err(e) => {
logger.log("error", format!(
"Failed to create temporary directory for FD restore of {}: {:?}",
cfg.name, e
));
return Err(e.into());
}
};
if let Err(e) = archive.unpack(tmp_dir.path()) {
logger.log("error", format!("Failed to unpack FD archive for {}: {:?}", cfg.name, e));
return Err(e.into());
}
logger.log("debug", format!("Listing contents of temp dir: {}", tmp_dir.path().display()));
for entry in std::fs::read_dir(tmp_dir.path())? {
if let Ok(entry) = entry {
let path = entry.path();
let file_type = entry.file_type()?;
logger.log("debug", format!(
" - {} | is_dir: {} | is_file: {}",
path.display(),
file_type.is_dir(),
file_type.is_file()
));
}
}
let dump_dir = if tmp_dir.path().join("toc.dat").exists() {
tmp_dir.path().to_path_buf()
} else {
std::fs::read_dir(tmp_dir.path())?
.filter_map(|e| e.ok())
.find(|entry| entry.path().join("toc.dat").exists())
.map(|e| e.path())
.ok_or_else(|| anyhow::anyhow!("Invalid FD archive: toc.dat not found"))?
};
let start = Instant::now();
let output = Command::new(&pg_restore)
.arg("--no-owner")
.arg("--no-privileges")
.arg("--clean")
.arg("--if-exists")
// .arg("--create")
.arg("--dbname")
.arg(&url)
.arg("-v")
.arg("-j")
.arg("4")
.arg(dump_dir)
.env("PGPASSWORD", &cfg.password)
.output();
let duration_ms = start.elapsed().as_millis() as f64;
match output {
Ok(o) => {
let stderr = String::from_utf8_lossy(&o.stderr).to_string();
let stdout = String::from_utf8_lossy(&o.stdout).to_string();
let combined = format!("{}{}", stdout, stderr);
let exit_code = o.status.code().unwrap_or(-1);
if o.status.success() {
logger.log_command("pg_restore", if combined.is_empty() { None } else { Some(combined) }, Some(0), Some(duration_ms));
logger.log("info", format!("FD restore completed successfully for {}", cfg.name))
} else {
logger.log_command("pg_restore", if combined.is_empty() { None } else { Some(combined) }, Some(exit_code), Some(duration_ms));
logger.log("error", format!("FD restore failed with status {:?} for {}", o.status, cfg.name));
anyhow::bail!("Postgres FD restore failed for {}", cfg.name);
}
}
Err(e) => {
logger.log_command("pg_restore", Some(e.to_string()), Some(-1), Some(duration_ms));
logger.log("error", format!("Error executing pg_restore for {}: {:?}", cfg.name, e));
return Err(e.into());
}
}
}
}
logger.log("info", format!("Restore finished for database {}", cfg.name));
Ok(())
})
.await?
}
+41
View File
@@ -0,0 +1,41 @@
use anyhow::Result;
use std::process::Command;
use std::time::Instant;
use crate::services::backup::logger::JobLogger;
use crate::services::config::DatabaseConfig;
pub(crate) fn run_pg_restore(
mut cmd: Command,
logger: &JobLogger,
cfg: &DatabaseConfig,
) -> Result<()> {
let start = Instant::now();
let output = cmd.output();
let duration_ms = start.elapsed().as_millis() as f64;
match output {
Ok(o) => {
let stderr = String::from_utf8_lossy(&o.stderr).to_string();
let stdout = String::from_utf8_lossy(&o.stdout).to_string();
let combined = format!("{}{}", stdout, stderr);
let exit_code = o.status.code().unwrap_or(-1);
let payload = if combined.is_empty() { None } else { Some(combined) };
if o.status.success() {
logger.log_command("pg_restore", payload, Some(0), Some(duration_ms));
logger.log("info", format!("Restore completed successfully for {}", cfg.name));
Ok(())
} else {
logger.log_command("pg_restore", payload, Some(exit_code), Some(duration_ms));
logger.log("error", format!("Restore failed with status {:?} for {}", o.status, cfg.name));
anyhow::bail!("Postgres restore failed for {}", cfg.name);
}
}
Err(e) => {
logger.log_command("pg_restore", Some(e.to_string()), Some(-1), Some(duration_ms));
logger.log("error", format!("Error executing pg_restore for {}: {:?}", cfg.name, e));
Err(e.into())
}
}
}
+9
View File
@@ -0,0 +1,9 @@
mod command;
mod prepare;
mod run;
mod toc;
pub use run::run;
pub(crate) use command::run_pg_restore;
pub(crate) use prepare::prepare_archive;
pub(crate) use toc::toc_creates_public_schema;
+67
View File
@@ -0,0 +1,67 @@
use anyhow::Result;
use std::path::{Path, PathBuf};
use std::process::Command;
use crate::domain::postgres::connection::sniff_format;
use crate::domain::postgres::format::PostgresDumpFormat;
use crate::services::backup::logger::JobLogger;
pub(crate) struct PreparedArchive {
path: PathBuf,
_tmp: Option<tempfile::TempDir>,
toc: String,
}
impl PreparedArchive {
pub(crate) fn path(&self) -> &Path {
&self.path
}
pub(crate) fn toc(&self) -> &str {
&self.toc
}
}
pub(crate) fn prepare_archive(
format: PostgresDumpFormat,
restore_file: &Path,
pg_restore: &Path,
logger: &JobLogger,
) -> Result<PreparedArchive> {
let sniffed = sniff_format(restore_file)?;
if sniffed != format {
logger.log("warn", format!("Declared format {:?} != sniffed {:?}; using sniffed", format, sniffed));
}
let format = sniffed;
let (path, tmp) = match format {
PostgresDumpFormat::Fc => (restore_file.to_path_buf(), None),
PostgresDumpFormat::Fd => {
let tar_gz = std::fs::File::open(restore_file)?;
let dec = flate2::read::GzDecoder::new(tar_gz);
let mut archive = tar::Archive::new(dec);
let tmp_dir = tempfile::TempDir::new()?;
archive.unpack(tmp_dir.path())?;
let dump_dir = if tmp_dir.path().join("toc.dat").exists() {
tmp_dir.path().to_path_buf()
} else {
std::fs::read_dir(tmp_dir.path())?
.filter_map(|e| e.ok())
.find(|entry| entry.path().join("toc.dat").exists())
.map(|e| e.path())
.ok_or_else(|| anyhow::anyhow!("Invalid FD archive: toc.dat not found"))?
};
(dump_dir, Some(tmp_dir))
}
};
let toc_out = Command::new(pg_restore).arg("-l").arg(&path).output()?;
if !toc_out.status.success() {
let stderr = String::from_utf8_lossy(&toc_out.stderr).to_string();
logger.log("error", format!("pg_restore -l failed: {}", stderr));
anyhow::bail!("Archive validation failed (pg_restore -l): {}", stderr);
}
let toc = String::from_utf8_lossy(&toc_out.stdout).to_string();
Ok(PreparedArchive { path, _tmp: tmp, toc })
}
+108
View File
@@ -0,0 +1,108 @@
use anyhow::Result;
use std::collections::HashMap;
use std::path::PathBuf;
use std::process::Command;
use std::sync::Arc;
use super::{prepare_archive, run_pg_restore, toc_creates_public_schema};
use crate::domain::postgres::clean_mode::RestoreCleanMode;
use crate::domain::postgres::connection::{
can_drop_database, drop_all_schemas, drop_and_recreate_database, pg_restore_binary_name,
recreate_public_schema, select_pg_path, server_version, terminate_connections,
};
use crate::domain::postgres::format::PostgresDumpFormat;
use crate::services::backup::logger::JobLogger;
use crate::services::config::DatabaseConfig;
pub async fn run(
cfg: DatabaseConfig,
format: PostgresDumpFormat,
restore_file: PathBuf,
env: HashMap<String, String>,
logger: Arc<JobLogger>,
) -> Result<()> {
let handle = tokio::runtime::Handle::current();
tokio::task::spawn_blocking(move || -> Result<()> {
logger.log("info", format!("Starting restore for database {}", cfg.name));
let version = match handle.block_on(server_version(&cfg)) {
Ok(v) => {
logger.log("debug", format!("Postgres version detected: {}", v));
v
}
Err(e) => {
logger.log("error", format!("Failed to get server version for {}: {:?}", cfg.name, e));
return Err(e.into());
}
};
let pg_restore = select_pg_path(&version).join(pg_restore_binary_name());
logger.log("debug", format!("Using pg_restore at {:?}", pg_restore));
let keep_ownership = cfg.options
.get("keep_ownership")
.and_then(|v| v.as_bool())
.unwrap_or(false);
if keep_ownership {
logger.log("info", format!("Restoring ownership and privileges for {}", cfg.name));
} else {
logger.log("info", format!("Stripping ownership and privileges for {} (--no-owner --no-privileges)", cfg.name));
}
let (mode, bad_value) = RestoreCleanMode::from_config(&cfg);
if let Some(v) = bad_value {
logger.log("warn", format!("Unknown clean_mode '{}' for {}, falling back to 'clean'", v, cfg.name));
}
let prepared = prepare_archive(format, &restore_file, &pg_restore, &logger)?;
match mode {
RestoreCleanMode::DropSchemas => {
handle.block_on(terminate_connections(&cfg))?;
let owner = cfg.username.clone();
let dropped = handle.block_on(drop_all_schemas(&cfg))?;
logger.log("warn", format!("clean_mode=drop_schemas dropped schemas {:?} in {}", dropped, cfg.database));
if !toc_creates_public_schema(prepared.toc()) {
handle.block_on(recreate_public_schema(&cfg, &owner))?;
}
}
RestoreCleanMode::DropDatabase => {
if !handle.block_on(can_drop_database(&cfg))? {
anyhow::bail!(
"clean_mode=drop_database requires CREATEDB + ownership on {}; use clean_mode=drop_schemas instead",
cfg.database
);
}
logger.log("warn", format!("clean_mode=drop_database DROPPING database {} before restore", cfg.database));
handle.block_on(drop_and_recreate_database(&cfg))?;
}
RestoreCleanMode::Clean | RestoreCleanMode::None => {
handle.block_on(terminate_connections(&cfg))?;
}
}
let mut cmd = Command::new(&pg_restore);
if !keep_ownership {
cmd.args(["--no-owner", "--no-privileges"]);
}
if mode.uses_pg_restore_clean() {
cmd.args(["--clean", "--if-exists"]);
}
cmd.arg("--host").arg(&cfg.host)
.arg("--port").arg(cfg.port.to_string())
.arg("--username").arg(&cfg.username)
.arg("--dbname").arg(&cfg.database)
.arg("-v");
if matches!(format, PostgresDumpFormat::Fd) {
cmd.arg("-j").arg("4");
}
cmd.arg(prepared.path()).envs(env);
run_pg_restore(cmd, &logger, &cfg)?;
logger.log("info", format!("Restore finished for database {}", cfg.name));
Ok(())
})
.await?
}
+8
View File
@@ -0,0 +1,8 @@
pub(crate) fn toc_creates_public_schema(toc: &str) -> bool {
toc.lines().any(|l| {
l.split(" SCHEMA - ")
.nth(1)
.and_then(|rest| rest.split_whitespace().next())
== Some("public")
})
}
+2
View File
@@ -21,6 +21,8 @@ pub async fn run(cfg: DatabaseConfig) -> Result<bool> {
cmd.arg("PING"); cmd.arg("PING");
cmd.kill_on_drop(true);
debug!("Command Ping Redis: {:?}", cmd); debug!("Command Ping Redis: {:?}", cmd);
let result = timeout(Duration::from_secs(10), cmd.output()).await; let result = timeout(Duration::from_secs(10), cmd.output()).await;
+1
View File
@@ -20,6 +20,7 @@ pub async fn run(cfg: DatabaseConfig) -> Result<bool> {
} }
cmd.arg("PING"); cmd.arg("PING");
cmd.kill_on_drop(true);
debug!("Command Ping Valkey: {:?}", cmd); debug!("Command Ping Valkey: {:?}", cmd);
+9
View File
@@ -22,6 +22,15 @@ async fn main() {
eprintln!("Failed to clean locks on startup: {:?}", e); eprintln!("Failed to clean locks on startup: {:?}", e);
} }
match crate::domain::docker_volume::docker::client() {
Ok(docker) => match crate::domain::docker_volume::docker::sweep_ephemeral(&docker).await {
Ok(n) if n > 0 => tracing::info!("Removed {n} orphaned ephemeral helper container(s)"),
Ok(_) => {}
Err(e) => tracing::warn!("Ephemeral helper sweep failed: {e}"),
},
Err(e) => tracing::debug!("Docker socket unavailable, skipping helper sweep: {e}"),
}
tokio::join!(ping_server(), async { tokio::join!(ping_server(), async {
let conn = redis_client::redis_connection().await; let conn = redis_client::redis_connection().await;
scheduler::scheduler_loop(conn).await; scheduler::scheduler_loop(conn).await;
+18 -1
View File
@@ -1,6 +1,7 @@
#![allow(dead_code)] #![allow(dead_code)]
use crate::utils::deserializer::deserialize_snake_case; use crate::services::config::DatabaseConfig;
use crate::utils::deserializer::{deserialize_snake_case, string_or_number_to_string};
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use toml::Value; use toml::Value;
@@ -24,6 +25,8 @@ pub struct DatabaseStorage {
#[serde(deserialize_with = "deserialize_snake_case")] #[serde(deserialize_with = "deserialize_snake_case")]
pub config: Value, pub config: Value,
pub provider: String, pub provider: String,
#[serde(default, rename = "folderName")]
pub folder_name: Option<String>,
} }
#[derive(Debug, Deserialize)] #[derive(Debug, Deserialize)]
@@ -31,7 +34,19 @@ pub struct DatabaseStatus {
pub dbms: String, pub dbms: String,
#[serde(rename = "generatedId")] #[serde(rename = "generatedId")]
pub generated_id: String, pub generated_id: String,
#[serde(default)]
pub storages: Vec<DatabaseStorage>, pub storages: Vec<DatabaseStorage>,
#[serde(default)]
pub storages_encrypted: Option<bool>,
#[serde(default)]
pub storages_ciphertext: Option<String>,
#[serde(default)]
pub config_encrypted: Option<bool>,
#[serde(default)]
pub config_ciphertext: Option<String>,
/// Filled in memory after decrypting `config_ciphertext`; never on the wire.
#[serde(skip)]
pub resolved_config: Option<DatabaseConfig>,
pub encrypt: bool, pub encrypt: bool,
pub data: DatabaseData, pub data: DatabaseData,
} }
@@ -54,4 +69,6 @@ pub struct RestoreInfo {
pub file: Option<String>, pub file: Option<String>,
#[serde(rename = "metaFile")] #[serde(rename = "metaFile")]
pub meta_file: Option<String>, pub meta_file: Option<String>,
#[serde(default, deserialize_with = "string_or_number_to_string")]
pub size: Option<String>,
} }
+7
View File
@@ -1,6 +1,7 @@
#![allow(dead_code)] #![allow(dead_code)]
use crate::services::config::DbType; use crate::services::config::DbType;
use std::fmt::{self, Display, Formatter};
use std::path::PathBuf; use std::path::PathBuf;
#[derive(Debug, Clone)] #[derive(Debug, Clone)]
@@ -20,3 +21,9 @@ pub struct UploadResult {
pub remote_file_path: Option<String>, pub remote_file_path: Option<String>,
pub total_size: Option<u64>, pub total_size: Option<u64>,
} }
impl Display for UploadResult {
fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result {
write!(f, "{}", self.error.as_deref().unwrap_or("unknown error"))
}
}
+26 -1
View File
@@ -4,6 +4,7 @@ use super::service::BackupService;
use crate::domain::factory::DatabaseFactory; use crate::domain::factory::DatabaseFactory;
use crate::services::config::DatabaseConfig; use crate::services::config::DatabaseConfig;
use crate::utils::retry::{RetryPolicy, retry};
use anyhow::Result; use anyhow::Result;
use std::path::Path; use std::path::Path;
@@ -39,7 +40,31 @@ impl BackupService {
}); });
} }
match db.backup(tmp_path, Arc::clone(&logger)).await { let policy = RetryPolicy::default();
let db_ref = &db;
let logger_ref = &logger;
let outcome = retry("Database backup", &logger, &policy, move |attempt| {
let dir = tmp_path.join(format!("attempt-{attempt}"));
async move {
if let Err(e) = tokio::fs::create_dir_all(&dir).await {
return Err(anyhow::Error::from(e));
}
match db_ref.backup(&dir, Arc::clone(logger_ref)).await {
Ok(f) => Ok(f),
Err(e) => {
let _ = tokio::fs::remove_dir_all(&dir).await;
Err(e)
}
}
}
})
.await;
match outcome {
Ok(file) => Ok(BackupResult { Ok(file) => Ok(BackupResult {
generated_id, generated_id,
db_type, db_type,
+58 -8
View File
@@ -4,6 +4,7 @@ use super::service::BackupService;
use crate::services::api::models::agent::status::DatabaseStorage; use crate::services::api::models::agent::status::DatabaseStorage;
use crate::services::storage; use crate::services::storage;
use crate::utils::common::BackupMethod; use crate::utils::common::BackupMethod;
use crate::utils::retry::{RetryPolicy, retry};
use anyhow::{Result, bail}; use anyhow::{Result, bail};
use futures::future::join_all; use futures::future::join_all;
use std::sync::Arc; use std::sync::Arc;
@@ -97,15 +98,44 @@ impl BackupService {
/* /*
STORAGE UPLOAD STORAGE UPLOAD
*/ */
let upload_result = provider let policy = RetryPolicy::default();
.upload(
ctx_clone.clone(), let attempt_result = if result_clone.backup_file.is_none() {
result_clone, logger_clone.log("error", format!("Missing backup file for storage {}", storage_id));
method,
&storage, Err(UploadResult {
Some(encrypt), storage_id: storage_id.clone(),
success: false,
error: Some("Missing backup file path".into()),
remote_file_path: None,
total_size: None,
})
} else {
retry(
&format!("Upload to storage {storage_id}"),
&logger_clone,
&policy,
|_| async {
let r = provider
.upload(
ctx_clone.clone(),
result_clone.clone(),
method,
&storage,
Some(encrypt),
&backup_storage_id,
)
.await;
if r.success { Ok(r) } else { Err(r) }
},
) )
.await; .await
};
let upload_result = match attempt_result {
Ok(r) | Err(r) => r,
};
let status = if upload_result.success { "success" } else { "failed" }; let status = if upload_result.success { "success" } else { "failed" };
@@ -115,6 +145,26 @@ impl BackupService {
storage_id, storage_id,
upload_result.error.as_deref().unwrap_or("unknown error") upload_result.error.as_deref().unwrap_or("unknown error")
)); ));
if let Err(err) = ctx_clone
.api
.backup_upload_status(
ctx_clone.edge_key.agent_id.clone(),
generated_id.clone(),
backup_storage_id.clone(),
status,
String::new(),
0u64,
backup_id,
)
.await
{
logger_clone.log("error", format!(
"Failed-status update failed for {}: {}",
storage_id, err
));
}
return upload_result; return upload_result;
} }
+151 -99
View File
@@ -1,8 +1,9 @@
#![allow(dead_code)] #![allow(dead_code)]
use crate::core::context::Context; use crate::core::context::Context;
use serde::Deserialize; use serde::{Deserialize, Serialize};
use serde_json; use serde_json;
use std::collections::HashMap;
use std::fs::File; use std::fs::File;
use std::io::Read; use std::io::Read;
use std::path::Path; use std::path::Path;
@@ -11,18 +12,22 @@ use toml;
use tracing::info; use tracing::info;
use uuid::Uuid; use uuid::Uuid;
#[derive(Debug, Deserialize, Clone)] #[derive(Debug, Serialize, Deserialize, Clone)]
#[serde(rename_all = "lowercase")] #[serde(rename_all = "lowercase")]
pub enum DbType { pub enum DbType {
Mysql, Mysql,
Mariadb, Mariadb,
Postgresql, Postgresql,
#[serde(rename = "postgresql-cluster")]
PostgresqlCluster,
MongoDB, MongoDB,
Sqlite, Sqlite,
Redis, Redis,
Valkey, Valkey,
Firebird, Firebird,
Mssql, Mssql,
#[serde(rename = "docker-volume")]
DockerVolume,
} }
impl DbType { impl DbType {
@@ -31,18 +36,20 @@ impl DbType {
DbType::Mysql => "mysql", DbType::Mysql => "mysql",
DbType::Mariadb => "mariadb", DbType::Mariadb => "mariadb",
DbType::Postgresql => "postgresql", DbType::Postgresql => "postgresql",
DbType::PostgresqlCluster => "postgresql-cluster",
DbType::MongoDB => "mongodb", DbType::MongoDB => "mongodb",
DbType::Sqlite => "sqlite", DbType::Sqlite => "sqlite",
DbType::Redis => "redis", DbType::Redis => "redis",
DbType::Valkey => "valkey", DbType::Valkey => "valkey",
DbType::Firebird => "firebird", DbType::Firebird => "firebird",
DbType::Mssql => "mssql", DbType::Mssql => "mssql",
DbType::DockerVolume => "docker-volume",
} }
} }
} }
#[allow(dead_code)] #[allow(dead_code)]
#[derive(Debug, Deserialize, Clone)] #[derive(Debug, Serialize, Deserialize, Clone)]
pub struct DatabaseConfig { pub struct DatabaseConfig {
pub name: String, pub name: String,
pub database: String, pub database: String,
@@ -54,10 +61,14 @@ pub struct DatabaseConfig {
pub host: String, pub host: String,
pub generated_id: String, pub generated_id: String,
pub path: String, pub path: String,
pub max_packet_size: String,
pub volume_name: String,
pub container_name: Option<String>,
pub options: HashMap<String, serde_json::Value>,
} }
#[allow(dead_code)] #[allow(dead_code)]
#[derive(Debug, Deserialize, Clone)] #[derive(Debug, Serialize, Deserialize, Clone)]
pub struct DatabasesConfig { pub struct DatabasesConfig {
pub databases: Vec<DatabaseConfig>, pub databases: Vec<DatabaseConfig>,
} }
@@ -75,6 +86,10 @@ pub struct InputDatabaseConfig {
pub host: Option<String>, pub host: Option<String>,
pub generated_id: String, pub generated_id: String,
pub path: Option<String>, pub path: Option<String>,
pub max_packet_size: Option<String>,
pub volume_name: Option<String>,
pub container_name: Option<String>,
pub options: Option<HashMap<String, serde_json::Value>>,
} }
#[allow(dead_code)] #[allow(dead_code)]
@@ -83,6 +98,106 @@ pub struct InputDatabasesConfig {
pub databases: Vec<InputDatabaseConfig>, pub databases: Vec<InputDatabaseConfig>,
} }
fn required<T: Clone>(opt: &Option<T>, db_name: &str, field_name: &str) -> Result<T, String> {
match opt {
Some(v) => Ok(v.clone()),
None => Err(format!(
"Missing required field '{}' for database '{}'",
field_name, db_name
)),
}
}
fn optional<T: Clone + Default>(opt: &Option<T>) -> T {
opt.clone().unwrap_or_default()
}
pub fn build_config(db: InputDatabaseConfig) -> Result<DatabaseConfig, String> {
if Uuid::parse_str(&db.generated_id).is_err() {
return Err(format!("Invalid UUID for database '{}'", db.name));
}
let username = match db.db_type {
DbType::Postgresql
| DbType::PostgresqlCluster
| DbType::Mysql
| DbType::Mariadb
| DbType::Mssql => required(&db.username, &db.name, "username")?,
_ => optional(&db.username),
};
let password = match db.db_type {
DbType::Postgresql
| DbType::PostgresqlCluster
| DbType::Mysql
| DbType::Mariadb
| DbType::Mssql => required(&db.password, &db.name, "password")?,
_ => optional(&db.password),
};
let host = match db.db_type {
DbType::Postgresql
| DbType::PostgresqlCluster
| DbType::Mysql
| DbType::Mariadb
| DbType::MongoDB
| DbType::Redis
| DbType::Firebird
| DbType::Valkey
| DbType::Mssql => required(&db.host, &db.name, "host")?,
DbType::Sqlite | DbType::DockerVolume => optional(&db.host),
};
let port = match db.db_type {
DbType::Postgresql
| DbType::PostgresqlCluster
| DbType::Mysql
| DbType::Mariadb
| DbType::Redis
| DbType::Firebird
| DbType::Valkey
| DbType::Mssql => required(&db.port, &db.name, "port")?,
DbType::MongoDB | DbType::Sqlite | DbType::DockerVolume => db.port.unwrap_or(0),
};
let database_name = match db.db_type {
DbType::Sqlite | DbType::Redis | DbType::Valkey | DbType::DockerVolume => {
optional(&db.database)
}
DbType::PostgresqlCluster => db
.database
.clone()
.unwrap_or_else(|| "postgres".to_string()),
_ => required(&db.database, &db.name, "database")?,
};
let path_val = match db.db_type {
DbType::Sqlite => required(&db.path, &db.name, "path")?,
_ => optional(&db.path),
};
let max_packet_size = match db.db_type {
DbType::Mysql | DbType::Mariadb => db.max_packet_size.unwrap_or_else(|| "512M".to_string()),
_ => String::new(),
};
let volume_name = match db.db_type {
DbType::DockerVolume => required(&db.volume_name, &db.name, "volume_name")?,
_ => optional(&db.volume_name),
};
Ok(DatabaseConfig {
name: db.name,
database: database_name,
db_type: db.db_type,
username,
password,
host,
port,
generated_id: db.generated_id,
path: path_val,
max_packet_size,
volume_name,
container_name: db.container_name.clone(),
options: db.options.unwrap_or_default(),
})
}
pub struct ConfigService { pub struct ConfigService {
ctx: Arc<Context>, ctx: Arc<Context>,
} }
@@ -92,16 +207,19 @@ impl ConfigService {
ConfigService { ctx } ConfigService { ctx }
} }
pub fn load(&self, file_path: Option<&str>) -> Result<DatabasesConfig, String> { fn resolve_path(file_path: Option<&str>) -> String {
let path: String = if let Some(fp) = file_path { match file_path {
fp.to_string() Some(fp) => fp.to_string(),
} else { None => format!(
format!(
"{}/{}", "{}/{}",
crate::settings::CONFIG.data_path, crate::settings::CONFIG.data_path,
crate::settings::CONFIG.databases_config_file crate::settings::CONFIG.databases_config_file
) ),
}; }
}
pub fn load(&self, file_path: Option<&str>) -> Result<DatabasesConfig, String> {
let path = Self::resolve_path(file_path);
info!("Loading databases config from: {}", path); info!("Loading databases config from: {}", path);
@@ -135,99 +253,33 @@ impl ConfigService {
_ => return Err("Unsupported config file format. Use .json or .toml".to_string()), _ => return Err("Unsupported config file format. Use .json or .toml".to_string()),
}; };
fn required<T: Clone>(
opt: &Option<T>,
db_name: &str,
field_name: &str,
) -> Result<T, String> {
match opt {
Some(v) => Ok(v.clone()),
None => {
let msg = format!(
"Missing required field '{}' for database '{}'",
field_name, db_name
);
Err(msg)
}
}
}
fn optional<T: Clone>(opt: &Option<T>) -> T
where
T: Default,
{
opt.clone().unwrap_or_default()
}
let mut databases = Vec::with_capacity(input_config.databases.len()); let mut databases = Vec::with_capacity(input_config.databases.len());
for db in input_config.databases { for db in input_config.databases {
if Uuid::parse_str(&db.generated_id).is_err() { databases.push(build_config(db)?);
return Err(format!("Invalid UUID for database '{}'", db.name));
}
let username = match db.db_type {
DbType::Postgresql | DbType::Mysql | DbType::Mariadb | DbType::Mssql => {
required(&db.username, &db.name, "username")?
}
_ => optional(&db.username),
};
let password = match db.db_type {
DbType::Postgresql | DbType::Mysql | DbType::Mariadb | DbType::Mssql => {
required(&db.password, &db.name, "password")?
}
_ => optional(&db.password),
};
let host = match db.db_type {
DbType::Postgresql
| DbType::Mysql
| DbType::Mariadb
| DbType::MongoDB
| DbType::Redis
| DbType::Firebird
| DbType::Valkey
| DbType::Mssql => required(&db.host, &db.name, "host")?,
DbType::Sqlite => optional(&db.host),
};
let port = match db.db_type {
DbType::Postgresql
| DbType::Mysql
| DbType::Mariadb
| DbType::MongoDB
| DbType::Redis
| DbType::Firebird
| DbType::Valkey
| DbType::Mssql => required(&db.port, &db.name, "port")?,
DbType::Sqlite => db.port.unwrap_or(0),
};
let database_name = match db.db_type {
DbType::Sqlite | DbType::Redis | DbType::Valkey => optional(&db.database),
_ => required(&db.database, &db.name, "database")?,
};
let path_val = match db.db_type {
DbType::Sqlite => required(&db.path, &db.name, "path")?,
_ => optional(&db.path),
};
databases.push(DatabaseConfig {
name: db.name,
database: database_name,
db_type: db.db_type,
username,
password,
host,
port,
generated_id: db.generated_id,
path: path_val,
});
} }
info!("Databases: {} instances loaded", databases.len()); info!("Databases: {} instances loaded", databases.len());
Ok(DatabasesConfig { databases }) Ok(DatabasesConfig { databases })
} }
pub fn load_optional(&self, file_path: Option<&str>) -> DatabasesConfig {
let path = Self::resolve_path(file_path);
if !Path::new(&path).exists() {
info!(
"No local databases config at {}; using dashboard-defined databases only",
path
);
return DatabasesConfig {
databases: Vec::new(),
};
}
self.load(file_path).unwrap_or_else(|e| {
tracing::warn!(
"Local databases config unavailable ({e}); continuing with dashboard-defined databases only"
);
DatabasesConfig { databases: Vec::new() }
})
}
} }
+56
View File
@@ -0,0 +1,56 @@
#![allow(dead_code)]
use crate::services::api::models::agent::status::PingResult;
use crate::services::config::{DatabaseConfig, DatabasesConfig};
use std::path::Path;
pub fn merge(local: &[DatabaseConfig], dashboard: &[DatabaseConfig]) -> DatabasesConfig {
let mut databases: Vec<DatabaseConfig> = local.to_vec();
for d in dashboard {
if let Some(slot) = databases
.iter_mut()
.find(|c| c.generated_id == d.generated_id)
{
*slot = d.clone();
} else {
databases.push(d.clone());
}
}
DatabasesConfig { databases }
}
pub fn collect_configs(ping: &PingResult) -> Vec<DatabaseConfig> {
ping.databases
.iter()
.filter_map(|db| db.resolved_config.clone())
.collect()
}
pub fn load_cache(path: &Path) -> Vec<DatabaseConfig> {
let contents = match std::fs::read_to_string(path) {
Ok(c) => c,
Err(_) => return Vec::new(),
};
match serde_json::from_str::<DatabasesConfig>(&contents) {
Ok(cfg) => cfg.databases,
Err(e) => {
tracing::warn!("Dashboard cache at {:?} is corrupt ({e}); ignoring", path);
Vec::new()
}
}
}
pub fn persist_cache(path: &Path, databases: &[DatabaseConfig]) -> std::io::Result<()> {
let wrapper = DatabasesConfig {
databases: databases.to_vec(),
};
let json = serde_json::to_string_pretty(&wrapper)
.map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e))?;
if let Some(parent) = path.parent() {
std::fs::create_dir_all(parent)?;
}
let tmp = path.with_extension("json.tmp");
std::fs::write(&tmp, json)?;
std::fs::rename(&tmp, path)?;
Ok(())
}
+2 -1
View File
@@ -2,6 +2,7 @@ pub mod api;
pub mod backup; pub mod backup;
pub mod config; pub mod config;
pub mod cron; pub mod cron;
pub mod dashboard_config;
pub mod restore; pub mod restore;
pub mod status; pub mod status;
mod storage; pub mod storage;
+13 -9
View File
@@ -1,18 +1,19 @@
use super::service::RestoreService; use super::service::RestoreService;
use crate::utils::compress::decompress_large_tar_gz; use crate::utils::compress::decompress_large_tar_gz;
use crate::utils::file::decrypt_file_stream_gcm; use crate::utils::file::decrypt_file_stream_gcm;
use anyhow::Result; use anyhow::Result;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use std::sync::Arc; use std::sync::Arc;
use crate::services::backup::logger::JobLogger; use crate::services::backup::logger::JobLogger;
use crate::services::config::DbType;
use crate::utils::common::choose_restore_path;
impl RestoreService { impl RestoreService {
pub async fn prepare_archive( pub async fn prepare_archive(
&self, &self,
downloaded_file: PathBuf, downloaded_file: PathBuf,
tmp_path: &Path, tmp_path: &Path,
db_type: &DbType,
logger: Arc<JobLogger> logger: Arc<JobLogger>
) -> Result<PathBuf> { ) -> Result<PathBuf> {
logger.log("info", "Start preparing backup archive".to_string()); logger.log("info", "Start preparing backup archive".to_string());
@@ -59,6 +60,13 @@ impl RestoreService {
archive = decrypted; archive = decrypted;
} }
if matches!(db_type, DbType::DockerVolume) {
let raw_tar = tmp_path.join("volume.tar");
crate::utils::compress::gunzip_to_file(archive.as_path(), &raw_tar).await?;
logger.log("info", format!("Docker volume archive gunzipped to {}", raw_tar.display()));
return Ok(raw_tar);
}
logger.log("info", format!("Decompressing archive {}", archive.display())); logger.log("info", format!("Decompressing archive {}", archive.display()));
let files = match decompress_large_tar_gz(archive.as_path(), tmp_path).await { let files = match decompress_large_tar_gz(archive.as_path(), tmp_path).await {
@@ -76,12 +84,8 @@ impl RestoreService {
logger.log("info", format!("Archive prepared, {} file(s) extracted", files.len())); logger.log("info", format!("Archive prepared, {} file(s) extracted", files.len()));
if files.len() == 1 { let chosen = choose_restore_path(&files, tmp_path, &archive);
logger.log("debug", format!("Using single extracted file: {}", files[0].display())); logger.log("debug", format!("Restore source resolved to: {}", chosen.display()));
Ok(files[0].clone()) Ok(chosen)
} else {
logger.log("debug", format!("Multiple files extracted, using archive root: {}", archive.display()));
Ok(archive)
}
} }
} }
+6 -1
View File
@@ -20,6 +20,8 @@ impl RestoreService {
return; return;
}; };
let expected_size = db.data.restore.size.clone();
let service = Self { let service = Self {
ctx: self.ctx.clone(), ctx: self.ctx.clone(),
}; };
@@ -27,7 +29,10 @@ impl RestoreService {
let db_cfg = cfg.clone(); let db_cfg = cfg.clone();
tokio::spawn(async move { tokio::spawn(async move {
if let Err(e) = service.execute_restore(db_cfg, file_to_restore).await { if let Err(e) = service
.execute_restore(db_cfg, file_to_restore, expected_size)
.await
{
error!("Restore failed: {}", e); error!("Restore failed: {}", e);
} }
}); });
+109 -5
View File
@@ -1,20 +1,69 @@
use super::service::RestoreService; use super::service::RestoreService;
use anyhow::Result; use anyhow::Result;
use futures::StreamExt;
use reqwest::{Client, Url}; use reqwest::{Client, Url};
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use std::sync::Arc; use std::sync::Arc;
use std::time::Instant;
use tokio::io::AsyncWriteExt;
use crate::services::backup::logger::JobLogger; use crate::services::backup::logger::JobLogger;
use crate::utils::retry::{RetryPolicy, retry};
fn human_size(bytes: u64) -> String {
if bytes >= 1024 * 1024 {
format!("{} MB", bytes / 1024 / 1024)
} else if bytes >= 1024 {
format!("{} KB", bytes / 1024)
} else {
format!("{bytes} B")
}
}
impl RestoreService { impl RestoreService {
pub async fn download_backup(&self, file_url: &str, tmp_path: &Path, logger: Arc<JobLogger>) -> Result<PathBuf> { pub async fn download_backup(
&self,
file_url: &str,
tmp_path: &Path,
logger: Arc<JobLogger>,
expected_size: Option<String>,
) -> Result<PathBuf> {
let policy = RetryPolicy::default();
let logger_ref = &logger;
let outcome = retry("Backup download", &logger, &policy, move |_| {
let expected = expected_size.clone();
async move {
self.download_once(file_url, tmp_path, Arc::clone(logger_ref), expected)
.await
}
})
.await;
if let Err(e) = &outcome {
logger.log("error", format!("Download failed: {e}"));
}
outcome
}
pub async fn download_once(
&self,
file_url: &str,
tmp_path: &Path,
logger: Arc<JobLogger>,
expected_size: Option<String>,
) -> Result<PathBuf> {
logger.log("info", "Start downloading backup archive".to_string()); logger.log("info", "Start downloading backup archive".to_string());
let client = Client::new(); let client = Client::new();
let response = client.get(file_url).send().await?; let response = client.get(file_url).send().await?;
let status = response.status();
if !response.status().is_success() { if !status.is_success() {
logger.log("error", "Failed to download".to_string()); logger.log("error", "Failed to download".to_string());
anyhow::bail!("download failed"); anyhow::bail!("download failed");
} }
@@ -39,11 +88,66 @@ impl RestoreService {
let path = tmp_path.join(&filename); let path = tmp_path.join(&filename);
let bytes = response.bytes().await?; let total = expected_size
.as_deref()
.and_then(|s| s.trim().parse::<u64>().ok())
.filter(|&n| n > 0);
tokio::fs::write(&path, &bytes).await?; logger.log(
"info",
format!(
"Downloading backup '{}' ({})",
filename,
total.map(human_size).unwrap_or_else(|| "unknown size".to_string())
),
);
let start = Instant::now();
let mut file = tokio::fs::File::create(&path).await?;
let mut stream = response.bytes_stream();
let mut downloaded: u64 = 0;
let mut next_pct: u64 = 10;
while let Some(chunk) = stream.next().await {
let chunk = chunk?;
file.write_all(&chunk).await?;
downloaded += chunk.len() as u64;
if let Some(total) = total {
let pct = (downloaded.saturating_mul(100) / total).min(100);
let milestone = pct / 10 * 10;
if milestone >= next_pct {
logger.log(
"info",
format!(
"Download progress: {}% ({} / {} bytes)",
milestone, downloaded, total
),
);
next_pct = milestone + 10;
}
}
}
file.flush().await?;
if downloaded == 0 {
logger.log(
"warn",
format!("Downloaded 0 bytes (status {status}); backup body was empty"),
);
}
logger.log(
"info",
format!(
"Backup downloaded to {} ( {} bytes in {:.1}s)",
path.display(),
downloaded,
start.elapsed().as_secs_f64()
),
);
logger.log("info", format!("Backup downloaded to {}", path.display()));
Ok(path) Ok(path)
} }
} }
+12 -3
View File
@@ -7,7 +7,12 @@ use std::time::Instant;
use tempfile::TempDir; use tempfile::TempDir;
impl RestoreService { impl RestoreService {
pub async fn execute_restore(&self, cfg: DatabaseConfig, file_url: String) -> Result<()> { pub async fn execute_restore(
&self,
cfg: DatabaseConfig,
file_url: String,
expected_size: Option<String>,
) -> Result<()> {
let logger = Arc::new(JobLogger::new()); let logger = Arc::new(JobLogger::new());
let start = Instant::now(); let start = Instant::now();
@@ -18,9 +23,13 @@ impl RestoreService {
logger.log("info", format!("Created temp directory {}", tmp_path.display())); logger.log("info", format!("Created temp directory {}", tmp_path.display()));
let downloaded = self.download_backup(&file_url, tmp_path, Arc::clone(&logger)).await?; let downloaded = self
.download_backup(&file_url, tmp_path, Arc::clone(&logger), expected_size)
.await?;
let backup_file = self.prepare_archive(downloaded, tmp_path, Arc::clone(&logger)).await?; let backup_file = self
.prepare_archive(downloaded, tmp_path, &cfg.db_type, Arc::clone(&logger))
.await?;
let result = self.run_restore(cfg, backup_file, Arc::clone(&logger)).await?; let result = self.run_restore(cfg, backup_file, Arc::clone(&logger)).await?;
+51 -9
View File
@@ -1,16 +1,39 @@
#![allow(dead_code)] #![allow(dead_code)]
use crate::core::context::Context; use crate::core::context::Context;
use crate::domain::factory::DatabaseFactory;
use crate::services::api::endpoints::status::DatabasePayload; use crate::services::api::endpoints::status::DatabasePayload;
use crate::services::api::models::agent::status::DatabaseStatus;
use crate::services::api::models::agent::status::DatabaseStorage;
use crate::services::api::models::agent::status::PingResult; use crate::services::api::models::agent::status::PingResult;
use crate::services::config::DatabaseConfig; use crate::services::config::{build_config, DatabaseConfig, InputDatabaseConfig};
use crate::settings::CONFIG; use crate::settings::CONFIG;
use crate::utils::file::decrypt_json_gcm;
use futures_util::future::try_join_all;
use reqwest::Client; use reqwest::Client;
use std::error::Error; use std::error::Error;
use std::sync::Arc; use std::sync::Arc;
use futures_util::future::try_join_all;
use tracing::info; use tracing::info;
use crate::domain::factory::DatabaseFactory;
pub fn resolve_dashboard_config(
status: &mut DatabaseStatus,
master_key_b64: &str,
) -> Result<(), String> {
if status.config_encrypted != Some(true) {
return Ok(());
}
let ciphertext = status
.config_ciphertext
.as_deref()
.ok_or("config_encrypted set but config_ciphertext missing")?;
let plaintext = decrypt_json_gcm(ciphertext, master_key_b64)
.map_err(|e| format!("Failed to decrypt config: {e}"))?;
let input: InputDatabaseConfig = serde_json::from_slice(&plaintext)
.map_err(|e| format!("Failed to parse decrypted config: {e}"))?;
status.resolved_config = Some(build_config(input)?);
Ok(())
}
pub struct StatusService { pub struct StatusService {
ctx: Arc<Context>, ctx: Arc<Context>,
@@ -28,12 +51,12 @@ impl StatusService {
pub async fn ping(&self, databases: &[DatabaseConfig]) -> Result<PingResult, Box<dyn Error>> { pub async fn ping(&self, databases: &[DatabaseConfig]) -> Result<PingResult, Box<dyn Error>> {
let edge_key = &self.ctx.edge_key; let edge_key = &self.ctx.edge_key;
let databases_payload: Vec<DatabasePayload> = try_join_all( let databases_payload: Vec<DatabasePayload> =
databases.into_iter().map(|db| async move { try_join_all(databases.into_iter().map(|db| async move {
let db_engine = DatabaseFactory::create_for_backup(db.clone()).await; let db_engine = DatabaseFactory::create_for_backup(db.clone()).await;
let reachable = db_engine.ping().await?; let reachable = db_engine.ping().await?;
info!("Ping {} => {:?}",db.name, reachable); info!("Ping {} => {:?}", db.name, reachable);
Ok::<DatabasePayload, anyhow::Error>(DatabasePayload { Ok::<DatabasePayload, anyhow::Error>(DatabasePayload {
name: &db.name, name: &db.name,
@@ -41,16 +64,35 @@ impl StatusService {
generated_id: &db.generated_id, generated_id: &db.generated_id,
ping_status: reachable, ping_status: reachable,
}) })
}) }))
).await?; .await?;
let version_str = CONFIG.app_version.as_str(); let version_str = CONFIG.app_version.as_str();
let result = self let mut result = self
.ctx .ctx
.api .api
.agent_status(&edge_key.agent_id, &version_str, databases_payload) .agent_status(&edge_key.agent_id, &version_str, databases_payload)
.await? .await?
.unwrap(); .unwrap();
for db in result.databases.iter_mut() {
if db.storages_encrypted == Some(true) {
let ciphertext = db
.storages_ciphertext
.as_deref()
.ok_or("storages_encrypted set but storages_ciphertext missing")?;
let plaintext = decrypt_json_gcm(ciphertext, &edge_key.master_key_b64)
.map_err(|e| format!("Failed to decrypt storages: {e}"))?;
db.storages = serde_json::from_slice::<Vec<DatabaseStorage>>(&plaintext)
.map_err(|e| format!("Failed to parse decrypted storages: {e}"))?;
}
if let Err(e) = resolve_dashboard_config(db, &edge_key.master_key_b64) {
tracing::warn!("Skipping dashboard config for {}: {e}", db.generated_id);
}
}
Ok(result) Ok(result)
} }
} }
+7
View File
@@ -5,6 +5,8 @@ use crate::services::api::models::agent::status::DatabaseStorage;
use crate::services::backup::models::{BackupResult, UploadResult}; use crate::services::backup::models::{BackupResult, UploadResult};
use crate::utils::common::BackupMethod; use crate::utils::common::BackupMethod;
use async_trait::async_trait; use async_trait::async_trait;
use providers::azure_blob;
use providers::google_cloud_storage;
use providers::google_drive; use providers::google_drive;
use providers::local; use providers::local;
use providers::s3; use providers::s3;
@@ -20,6 +22,7 @@ pub trait StorageProvider: Send + Sync {
method: BackupMethod, method: BackupMethod,
config: &DatabaseStorage, config: &DatabaseStorage,
encrypt: Option<bool>, encrypt: Option<bool>,
backup_storage_id: &str,
) -> UploadResult; ) -> UploadResult;
} }
@@ -31,7 +34,11 @@ pub fn get_provider(storage: &DatabaseStorage) -> Option<Box<dyn StorageProvider
match storage.provider.as_str() { match storage.provider.as_str() {
"local" => Some(Box::new(local::LocalProvider {})), "local" => Some(Box::new(local::LocalProvider {})),
"s3" => Some(Box::new(s3::S3Provider {})), "s3" => Some(Box::new(s3::S3Provider {})),
"blob" => Some(Box::new(azure_blob::AzureBlobProvider {})),
"google-drive" => Some(Box::new(google_drive::GoogleDriveProvider {})), "google-drive" => Some(Box::new(google_drive::GoogleDriveProvider {})),
"google-cloud-storage" => Some(Box::new(
google_cloud_storage::GoogleCloudStorageProvider {},
)),
_ => { _ => {
error!("Unknown storage provider: {}", storage.provider); error!("Unknown storage provider: {}", storage.provider);
None None
@@ -0,0 +1,174 @@
use anyhow::{Context as _, Result, anyhow};
use base64::Engine;
use base64::engine::general_purpose::STANDARD;
use chrono::{Duration, Utc};
use openssl::hash::MessageDigest;
use openssl::pkey::PKey;
use openssl::sign::Signer;
use url::Url;
use azure_core::http::RequestContent;
use azure_storage_blob::clients::{BlobClient, BlockBlobClient};
use azure_storage_blob::models::BlockLookupList;
use bytes::{Bytes, BytesMut};
use futures::{Stream, StreamExt};
use std::pin::Pin;
use tracing::info;
#[derive(Debug, Clone)]
pub struct ResolvedAzure {
pub account_name: String,
pub account_key: String,
pub blob_endpoint: String,
}
#[derive(Clone, Copy)]
pub enum SasResource {
Blob,
#[allow(dead_code)]
Container,
}
impl SasResource {
fn code(self) -> &'static str {
match self { SasResource::Blob => "b", SasResource::Container => "c" }
}
}
pub(crate) const SAS_VERSION: &str = "2022-11-02";
pub(crate) fn hmac_sha256_b64(key: &[u8], data: &str) -> Result<String> {
let pkey = PKey::hmac(key).context("hmac key")?;
let mut signer = Signer::new(MessageDigest::sha256(), &pkey).context("signer")?;
signer.update(data.as_bytes()).context("signer update")?;
let sig = signer.sign_to_vec().context("sign")?;
Ok(STANDARD.encode(sig))
}
pub fn build_service_sas(
resolved: &ResolvedAzure,
canonical_resource: &str,
resource: SasResource,
permissions: &str,
) -> Result<Vec<(String, String)>> {
let key = STANDARD
.decode(&resolved.account_key)
.map_err(|_| anyhow!("account key is not valid base64"))?;
let signed_start = String::new();
let signed_expiry = (Utc::now() + Duration::hours(1))
.format("%Y-%m-%dT%H:%M:%SZ").to_string();
let signed_protocol = "https,http"; // Azurite is http
let signed_resource = resource.code();
let string_to_sign = format!(
"{sp}\n{st}\n{se}\n{canon}\n{si}\n{sip}\n{spr}\n{sv}\n{sr}\n{snap}\n{enc}\n{rscc}\n{rscd}\n{rsce}\n{rscl}\n{rsct}",
sp = permissions, st = signed_start, se = signed_expiry, canon = canonical_resource,
si = "", sip = "", spr = signed_protocol, sv = SAS_VERSION, sr = signed_resource,
snap = "", enc = "", rscc = "", rscd = "", rsce = "", rscl = "", rsct = "",
);
let sig = hmac_sha256_b64(&key, &string_to_sign)?;
Ok(vec![
("sv".into(), SAS_VERSION.into()),
("sr".into(), signed_resource.into()),
("sp".into(), permissions.into()),
("se".into(), signed_expiry),
("spr".into(), signed_protocol.into()),
("sig".into(), sig),
])
}
pub fn build_sas_url(
resolved: &ResolvedAzure,
container: &str,
blob: &str,
resource: SasResource,
permissions: &str,
) -> Result<Url> {
let canonical = if blob.is_empty() {
format!("/blob/{}/{}", resolved.account_name, container)
} else {
format!("/blob/{}/{}/{}", resolved.account_name, container, blob)
};
let pairs = build_service_sas(resolved, &canonical, resource, permissions)?;
let base = if blob.is_empty() {
format!("{}/{}", resolved.blob_endpoint.trim_end_matches('/'), container)
} else {
format!("{}/{}/{}", resolved.blob_endpoint.trim_end_matches('/'), container, blob)
};
let mut url = Url::parse(&base).context("invalid blob endpoint/url")?;
{
let mut qp = url.query_pairs_mut();
for (k, v) in pairs { qp.append_pair(&k, &v); }
}
Ok(url)
}
pub const BLOCK_SIZE: usize = 100 * 1024 * 1024;
type ByteStream = Pin<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>>;
async fn stage_block(
bbc: &BlockBlobClient,
index: u32,
block: Bytes,
block_ids: &mut Vec<Vec<u8>>,
) -> Result<()> {
let raw_id = format!("{index:032}").into_bytes();
let len = block.len() as u64;
bbc.stage_block(&raw_id, len, RequestContent::from(block.to_vec()), None)
.await
.map_err(|e| anyhow!("stage_block {index} failed: {e}"))?;
block_ids.push(raw_id);
info!("staged azure block {index} ({len} bytes)");
Ok(())
}
pub async fn upload_stream_to_azure(
resolved: &ResolvedAzure,
container: &str,
blob: &str,
mut body: ByteStream,
block_size: usize,
) -> Result<()> {
let url = build_sas_url(resolved, container, blob, SasResource::Blob, "cw")?;
let blob_client = BlobClient::new(url, None, None).context("blob client")?;
let bbc = blob_client.block_blob_client();
let mut buffer = BytesMut::with_capacity(block_size);
let mut block_ids: Vec<Vec<u8>> = Vec::new();
let mut index: u32 = 0;
while let Some(item) = body.next().await {
let bytes = item.context("stream error during upload")?;
buffer.extend_from_slice(&bytes);
while buffer.len() >= block_size {
let block = buffer.split_to(block_size).freeze();
stage_block(&bbc, index, block, &mut block_ids).await?;
index += 1;
}
}
if !buffer.is_empty() {
let block = buffer.split().freeze();
stage_block(&bbc, index, block, &mut block_ids).await?;
}
if block_ids.is_empty() {
stage_block(&bbc, 0, Bytes::new(), &mut block_ids).await?;
}
let block_list = BlockLookupList {
latest: Some(block_ids),
..Default::default()
};
bbc.commit_block_list(block_list.try_into()?, None)
.await
.map_err(|e| anyhow!("commit_block_list failed: {e}"))?;
Ok(())
}
@@ -0,0 +1,135 @@
pub mod helpers;
pub(crate) mod models;
use crate::core::context::Context;
use crate::services::api::models::agent::status::DatabaseStorage;
use crate::services::backup::models::{BackupResult, UploadResult};
use crate::services::storage::StorageProvider;
use crate::services::storage::providers::azure_blob::helpers::{BLOCK_SIZE, upload_stream_to_azure};
use crate::services::storage::providers::azure_blob::models::AzureBlobProviderConfig;
use crate::utils::common::BackupMethod;
use crate::utils::file::{full_file_name, full_file_path};
use crate::utils::stream::build_stream;
use async_trait::async_trait;
use std::sync::Arc;
use tokio::fs;
use tracing::{error, info};
pub struct AzureBlobProvider {}
#[async_trait]
impl StorageProvider for AzureBlobProvider {
async fn upload(
&self,
ctx: Arc<Context>,
result: BackupResult,
_method: BackupMethod,
storage: &DatabaseStorage,
encrypt: Option<bool>,
_backup_storage_id: &str,
) -> UploadResult {
let Some(file_path) = result.backup_file else {
return UploadResult {
storage_id: storage.id.clone(),
success: false,
error: Some("Missing backup file path".to_string()),
remote_file_path: None,
total_size: None,
};
};
let total_size = match fs::metadata(&file_path).await {
Ok(meta) => meta.len(),
Err(e) => {
error!("Failed to get file size: {}", e);
return UploadResult {
storage_id: storage.id.clone(),
success: false,
error: Some(e.to_string()),
remote_file_path: None,
total_size: None,
};
}
};
let encrypt = encrypt.unwrap_or(false);
let upload = match build_stream(&file_path, encrypt, &ctx.edge_key.master_key_b64).await {
Ok(u) => u,
Err(e) => {
error!("Stream build failed: {}", e);
return UploadResult {
storage_id: storage.id.clone(),
success: false,
error: Some(e.to_string()),
remote_file_path: None,
total_size: None,
};
}
};
let config: AzureBlobProviderConfig = match storage.clone().config.try_into() {
Ok(c) => c,
Err(e) => {
return UploadResult {
storage_id: storage.id.clone(),
success: false,
error: Some(e.to_string()),
remote_file_path: None,
total_size: None,
};
}
};
let resolved = match config.resolve() {
Ok(r) => r,
Err(e) => {
return UploadResult {
storage_id: storage.id.clone(),
success: false,
error: Some(e.to_string()),
remote_file_path: None,
total_size: None,
};
}
};
let file_name = full_file_name(encrypt);
let remote_file_path = full_file_path(&file_name, storage.folder_name.as_deref());
info!(
"Starting block upload to azure blob {}/{}",
config.container_name, remote_file_path
);
match upload_stream_to_azure(
&resolved,
&config.container_name,
&remote_file_path,
upload.stream,
BLOCK_SIZE,
)
.await
{
Ok(_) => {
info!("Azure blob upload successful: {}", remote_file_path);
UploadResult {
storage_id: storage.id.clone(),
success: true,
error: None,
remote_file_path: Some(remote_file_path),
total_size: Some(total_size),
}
}
Err(e) => {
error!("Azure blob upload failed: {:?}", e);
UploadResult {
storage_id: storage.id.clone(),
success: false,
error: Some(e.to_string()),
remote_file_path: None,
total_size: None,
}
}
}
}
}
@@ -0,0 +1,106 @@
use crate::services::storage::providers::azure_blob::helpers::ResolvedAzure;
use anyhow::{Result, anyhow};
use serde::{Deserialize, Serialize};
use url::Url;
#[derive(Debug, Deserialize, Serialize)]
pub struct AzureBlobProviderConfig {
#[serde(default)]
pub account_name: String,
#[serde(default)]
pub account_key: String,
pub container_name: String,
#[serde(default)]
pub auth_mode: Option<String>,
#[serde(default)]
pub connection_string: String,
#[serde(default)]
pub endpoint_url: Option<String>,
}
fn parse_connection_string(cs: &str) -> std::collections::HashMap<String, String> {
cs.split(';')
.filter(|s| !s.trim().is_empty())
.filter_map(|pair| {
let mut it = pair.splitn(2, '=');
let k = it.next()?.trim().to_string();
let v = it.next()?.trim().to_string();
Some((k, v))
})
.collect()
}
pub(crate) fn ensure_account_in_endpoint(endpoint: &str, account: &str) -> String {
let trimmed = endpoint.trim_end_matches('/');
if account.is_empty() {
return trimmed.to_string();
}
if let Ok(url) = Url::parse(trimmed) {
let host = url.host_str().unwrap_or("");
if host.contains(account) {
return trimmed.to_string();
}
let path = url.path().trim_matches('/');
if path == account || path.starts_with(&format!("{account}/")) {
return trimmed.to_string();
}
}
format!("{trimmed}/{account}")
}
impl AzureBlobProviderConfig {
pub fn resolve(&self) -> Result<ResolvedAzure> {
let mode = self.auth_mode.as_deref().unwrap_or("").trim();
let has_connection_string = !self.connection_string.trim().is_empty();
if mode == "connectionString" || (mode.is_empty() && has_connection_string) {
if !has_connection_string {
return Err(anyhow!(
"authMode is connectionString but connectionString is empty"
));
}
let map = parse_connection_string(&self.connection_string);
let account_name = map
.get("AccountName")
.cloned()
.unwrap_or_else(|| self.account_name.clone());
let account_key = map
.get("AccountKey")
.cloned()
.unwrap_or_else(|| self.account_key.clone());
let blob_endpoint = map
.get("BlobEndpoint")
.cloned()
.ok_or_else(|| anyhow!("connection string missing BlobEndpoint"))?;
return Ok(ResolvedAzure {
account_name,
account_key,
blob_endpoint,
});
}
if self.account_name.trim().is_empty() {
return Err(anyhow!("accountName required for accountKey auth"));
}
if self.account_key.trim().is_empty() {
return Err(anyhow!("accountKey required for accountKey auth"));
}
let blob_endpoint = match self
.endpoint_url
.as_deref()
.map(str::trim)
.filter(|s| !s.is_empty())
{
Some(endpoint) => ensure_account_in_endpoint(endpoint, &self.account_name),
None => format!("https://{}.blob.core.windows.net", self.account_name),
};
Ok(ResolvedAzure {
account_name: self.account_name.clone(),
account_key: self.account_key.clone(),
blob_endpoint,
})
}
}
@@ -0,0 +1,122 @@
use crate::services::storage::providers::google_cloud_storage::models::GoogleCloudStorageProviderConfig;
use anyhow::{Context, Result};
use bytes::Bytes;
use futures::Stream;
use futures::StreamExt;
use google_cloud_auth::credentials::Credentials;
use google_cloud_storage::client::Storage;
use google_cloud_storage::streaming_source::{SizeHint, StreamingSource};
use std::pin::Pin;
pub fn build_credentials(cfg: &GoogleCloudStorageProviderConfig) -> Result<Credentials> {
// Service-account JSON stores the PEM with `\n` escape sequences. When the key is
// carried through config as a JSON string those can arrive as literal two-char `\n`
// sequences rather than real newlines, so the PEM parser finds no `-----BEGIN-----`
// line ("no items found"). Normalize them back to real newlines. A PEM that already
// has real newlines contains no literal `\n` pairs, so this is a no-op for it.
let private_key = cfg.private_key.replace("\\n", "\n");
let key = serde_json::json!({
"type": "service_account",
"project_id": cfg.project_id,
"client_email": cfg.client_email,
"private_key": private_key,
"private_key_id": "",
"token_uri": "https://oauth2.googleapis.com/token",
"universe_domain": "googleapis.com",
});
google_cloud_auth::credentials::service_account::Builder::new(key)
.build()
.context("failed to build GCS service account credentials")
}
pub async fn build_client(cfg: &GoogleCloudStorageProviderConfig) -> Result<Storage> {
let endpoint = cfg.api_endpoint.as_deref().filter(|s| !s.trim().is_empty());
// A custom endpoint means a local emulator (fake-gcs-server), which does not verify
// credentials. Use anonymous creds so a dummy/empty `private_key` in the emulator
// config doesn't trip the service-account PEM parser. Real GCS still uses the
// service-account key built from config.
let builder = if let Some(ep) = endpoint {
let creds = google_cloud_auth::credentials::anonymous::Builder::new().build();
Storage::builder()
.with_credentials(creds)
.with_endpoint(ep.to_string())
} else {
Storage::builder().with_credentials(build_credentials(cfg)?)
};
builder.build().await.context("failed to build GCS client")
}
/// Bridges `build_stream`'s `Send`-only byte stream into the SDK's `StreamingSource`
/// (which `send_buffered` requires to be `Send + Sync + 'static`) via a bounded mpsc
/// channel. Also reports an exact `size_hint`: the SDK picks single-shot vs resumable
/// upload from `size_hint().upper()` — an unknown bound forces resumable unconditionally
/// (see `upload_with_client`).
pub struct StreamSource {
rx: tokio::sync::mpsc::Receiver<Result<Bytes, std::io::Error>>,
total_size: u64,
}
impl StreamSource {
pub fn from_stream(
mut stream: Pin<Box<dyn Stream<Item = Result<Bytes, std::io::Error>> + Send>>,
total_size: u64,
) -> Self {
let (tx, rx) = tokio::sync::mpsc::channel(8);
tokio::spawn(async move {
while let Some(item) = stream.next().await {
if tx.send(item).await.is_err() {
break;
}
}
});
StreamSource { rx, total_size }
}
}
impl StreamingSource for StreamSource {
type Error = std::io::Error;
async fn next(&mut self) -> Option<Result<Bytes, Self::Error>> {
self.rx.recv().await
}
// Report the exact size so the SDK can choose single-shot uploads. The default
// impl returns an unknown bound, which forces the resumable path unconditionally.
async fn size_hint(&self) -> Result<SizeHint, Self::Error> {
Ok(SizeHint::with_exact(self.total_size))
}
}
pub async fn upload_with_client(
client: &Storage,
bucket: &str,
object: &str,
source: StreamSource,
force_single_shot: bool,
) -> Result<()> {
// `write_object` uses gRPC-style resource names: the bucket must be passed as
// `projects/_/buckets/<name>`, not the bare bucket id.
let bucket_resource = format!("projects/_/buckets/{bucket}");
let mut builder = client.write_object(bucket_resource, object, source);
// Resumable uploads follow a server-generated `Location` URL. When pointed at a
// custom `apiEndpoint` on a non-443 port, the SDK's transport drops the port from
// the `Host` header (google-cloud-gax-internal `host.rs`), so emulators that build
// the `Location` from `Host` hand back a portless URL the SDK then hangs on. A
// single-shot upload issues one request to the configured endpoint (no `Location`
// to follow), sidestepping the bug. We force it only for custom endpoints; against
// real GCS we keep resumable (bounded memory + resume on large backups).
if force_single_shot {
builder = builder.with_resumable_upload_threshold(usize::MAX);
}
builder
.send_buffered()
.await
.context("GCS write_object failed")?;
Ok(())
}
@@ -0,0 +1,149 @@
pub mod helpers;
mod models;
use crate::core::context::Context;
use crate::services::api::models::agent::status::DatabaseStorage;
use crate::services::backup::models::{BackupResult, UploadResult};
use crate::services::storage::StorageProvider;
use crate::services::storage::providers::google_cloud_storage::helpers::{
StreamSource, build_client, upload_with_client,
};
use crate::services::storage::providers::google_cloud_storage::models::GoogleCloudStorageProviderConfig;
use crate::utils::common::BackupMethod;
use crate::utils::file::{full_file_name, full_file_path};
use crate::utils::stream::build_stream;
use async_trait::async_trait;
use std::sync::Arc;
use tokio::fs;
use tracing::{error, info};
pub struct GoogleCloudStorageProvider {}
#[async_trait]
impl StorageProvider for GoogleCloudStorageProvider {
async fn upload(
&self,
ctx: Arc<Context>,
result: BackupResult,
_method: BackupMethod,
storage: &DatabaseStorage,
encrypt: Option<bool>,
_backup_storage_id: &str,
) -> UploadResult {
let Some(file_path) = result.backup_file else {
return UploadResult {
storage_id: storage.id.clone(),
success: false,
error: Some("Missing backup file path".to_string()),
remote_file_path: None,
total_size: None,
};
};
let total_size = match fs::metadata(&file_path).await {
Ok(meta) => meta.len(),
Err(e) => {
error!("Failed to get file size: {}", e);
return UploadResult {
storage_id: storage.id.clone(),
success: false,
error: Some(e.to_string()),
remote_file_path: None,
total_size: None,
};
}
};
let encrypt = encrypt.unwrap_or(false);
let upload = match build_stream(&file_path, encrypt, &ctx.edge_key.master_key_b64).await {
Ok(u) => u,
Err(e) => {
error!("Stream build failed: {}", e);
return UploadResult {
storage_id: storage.id.clone(),
success: false,
error: Some(e.to_string()),
remote_file_path: None,
total_size: None,
};
}
};
let config: GoogleCloudStorageProviderConfig = match storage.clone().config.try_into() {
Ok(c) => c,
Err(e) => {
return UploadResult {
storage_id: storage.id.clone(),
success: false,
error: Some(e.to_string()),
remote_file_path: None,
total_size: None,
};
}
};
let file_name = full_file_name(encrypt);
info!("Uploading file {}", file_name);
let remote_file_path = full_file_path(&file_name, storage.folder_name.as_deref());
let client = match build_client(&config).await {
Ok(c) => c,
Err(e) => {
error!("GCS client build failed: {:?}", e);
return UploadResult {
storage_id: storage.id.clone(),
success: false,
error: Some(e.to_string()),
remote_file_path: None,
total_size: None,
};
}
};
let source = StreamSource::from_stream(upload.stream, total_size);
// A custom apiEndpoint (self-hosted / emulator) on a non-443 port trips an
// upstream SDK bug in the resumable-upload path; force single-shot for it.
let force_single_shot = config
.api_endpoint
.as_deref()
.is_some_and(|s| !s.trim().is_empty());
info!(
"Starting GCS upload to {}/{} (single_shot={})",
config.bucket_name, remote_file_path, force_single_shot
);
match upload_with_client(
&client,
&config.bucket_name,
&remote_file_path,
source,
force_single_shot,
)
.await
{
Ok(_) => {
info!("GCS upload successful: {}", remote_file_path);
UploadResult {
storage_id: storage.id.clone(),
success: true,
error: None,
remote_file_path: Some(remote_file_path),
total_size: Some(total_size),
}
}
Err(e) => {
error!("GCS upload failed: {:?}", e);
UploadResult {
storage_id: storage.id.clone(),
success: false,
error: Some(e.to_string()),
remote_file_path: None,
total_size: Some(total_size),
}
}
}
}
}
@@ -0,0 +1,11 @@
use serde::{Deserialize, Serialize};
#[derive(Debug, Deserialize, Serialize)]
pub struct GoogleCloudStorageProviderConfig {
pub project_id: String,
pub bucket_name: String,
pub client_email: String,
pub private_key: String,
#[serde(default)]
pub api_endpoint: Option<String>,
}
@@ -26,6 +26,7 @@ impl StorageProvider for GoogleDriveProvider {
_method: BackupMethod, _method: BackupMethod,
storage: &DatabaseStorage, storage: &DatabaseStorage,
encrypt: Option<bool>, encrypt: Option<bool>,
_backup_storage_id: &str,
) -> UploadResult { ) -> UploadResult {
let Some(file_path) = result.backup_file else { let Some(file_path) = result.backup_file else {
return UploadResult { return UploadResult {
@@ -84,7 +85,7 @@ impl StorageProvider for GoogleDriveProvider {
info!("Uploading file {}", file_name); info!("Uploading file {}", file_name);
let remote_file_path = full_file_path(&file_name); let remote_file_path = full_file_path(&file_name, storage.folder_name.as_deref());
match upload_stream_to_google_drive( match upload_stream_to_google_drive(
&config, &config,
+6 -1
View File
@@ -23,6 +23,7 @@ impl StorageProvider for LocalProvider {
method: BackupMethod, method: BackupMethod,
storage: &DatabaseStorage, storage: &DatabaseStorage,
encrypt: Option<bool>, encrypt: Option<bool>,
backup_storage_id: &str,
) -> UploadResult { ) -> UploadResult {
let Some(file_path) = result.backup_file else { let Some(file_path) = result.backup_file else {
return UploadResult { return UploadResult {
@@ -37,7 +38,7 @@ impl StorageProvider for LocalProvider {
let encrypt = encrypt.unwrap_or(false); let encrypt = encrypt.unwrap_or(false);
let file_name = full_file_name(encrypt); let file_name = full_file_name(encrypt);
let remote_file_path = full_file_path(&file_name); let remote_file_path = full_file_path(&file_name, storage.folder_name.as_deref());
let total_size = match fs::metadata(&file_path).await { let total_size = match fs::metadata(&file_path).await {
Ok(meta) => meta.len(), Ok(meta) => meta.len(),
@@ -82,6 +83,10 @@ impl StorageProvider for LocalProvider {
"X-Generated-Id", "X-Generated-Id",
HeaderValue::from_str(&result.generated_id).unwrap(), HeaderValue::from_str(&result.generated_id).unwrap(),
); );
extra_headers.insert(
"X-Backup-Storage-Id",
HeaderValue::from_str(backup_storage_id).unwrap(),
);
extra_headers.insert("X-Status", HeaderValue::from_str(&result.status).unwrap()); extra_headers.insert("X-Status", HeaderValue::from_str(&result.status).unwrap());
extra_headers.insert( extra_headers.insert(
"X-Method", "X-Method",
+2
View File
@@ -1,3 +1,5 @@
pub mod azure_blob;
pub mod google_cloud_storage;
pub mod google_drive; pub mod google_drive;
pub mod local; pub mod local;
pub mod s3; pub mod s3;
+14 -7
View File
@@ -13,7 +13,9 @@ use aws_config::retry::RetryConfig;
use aws_sdk_s3 as s3; use aws_sdk_s3 as s3;
use aws_sdk_s3::config::BehaviorVersion; use aws_sdk_s3::config::BehaviorVersion;
use aws_sdk_s3::config::Region; use aws_sdk_s3::config::Region;
use aws_sdk_s3::config::RequestChecksumCalculation;
use aws_sdk_s3::config::retry::ReconnectMode; use aws_sdk_s3::config::retry::ReconnectMode;
use aws_sdk_s3::error::DisplayErrorContext;
use aws_sdk_s3::primitives::ByteStream; use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart}; use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart};
use futures::StreamExt; use futures::StreamExt;
@@ -34,6 +36,7 @@ impl StorageProvider for S3Provider {
_method: BackupMethod, _method: BackupMethod,
storage: &DatabaseStorage, storage: &DatabaseStorage,
encrypt: Option<bool>, encrypt: Option<bool>,
_backup_storage_id: &str,
) -> UploadResult { ) -> UploadResult {
let Some(file_path) = result.backup_file else { let Some(file_path) = result.backup_file else {
return UploadResult { return UploadResult {
@@ -134,6 +137,7 @@ impl StorageProvider for S3Provider {
.credentials_provider(credentials) .credentials_provider(credentials)
.region(region) .region(region)
.force_path_style(true) .force_path_style(true)
.request_checksum_calculation(RequestChecksumCalculation::WhenRequired)
.endpoint_url(endpoint) .endpoint_url(endpoint)
.behavior_version(BehaviorVersion::latest()) .behavior_version(BehaviorVersion::latest())
.build(); .build();
@@ -147,7 +151,7 @@ impl StorageProvider for S3Provider {
info!("Uploading file {}", file_name); info!("Uploading file {}", file_name);
let bucket = &config.bucket_name; let bucket = &config.bucket_name;
let remote_file_path = full_file_path(&file_name); let remote_file_path = full_file_path(&file_name, storage.folder_name.as_deref());
info!("S3 key {:}", remote_file_path); info!("S3 key {:}", remote_file_path);
info!( info!(
"Starting multipart upload to s3://{}/{}", "Starting multipart upload to s3://{}/{}",
@@ -163,11 +167,12 @@ impl StorageProvider for S3Provider {
{ {
Ok(r) => r, Ok(r) => r,
Err(e) => { Err(e) => {
error!("Failed to create multipart upload: {}", e); let detail = DisplayErrorContext(&e).to_string();
error!("Failed to create multipart upload: {}", detail);
return UploadResult { return UploadResult {
storage_id: storage.id.clone(), storage_id: storage.id.clone(),
success: false, success: false,
error: Some(e.to_string()), error: Some(detail),
remote_file_path: None, remote_file_path: None,
total_size: None, total_size: None,
}; };
@@ -250,7 +255,8 @@ impl StorageProvider for S3Provider {
} }
} }
Err(e) => { Err(e) => {
error!("Failed to upload part {}: {}", part_number, e); let detail = DisplayErrorContext(&e).to_string();
error!("Failed to upload part {}: {}", part_number, detail);
let _ = client let _ = client
.abort_multipart_upload() .abort_multipart_upload()
.bucket(bucket) .bucket(bucket)
@@ -261,7 +267,7 @@ impl StorageProvider for S3Provider {
return UploadResult { return UploadResult {
storage_id: storage.id.clone(), storage_id: storage.id.clone(),
success: false, success: false,
error: Some(e.to_string()), error: Some(detail),
remote_file_path: None, remote_file_path: None,
total_size: None, total_size: None,
}; };
@@ -316,7 +322,8 @@ impl StorageProvider for S3Provider {
} }
} }
Err(e) => { Err(e) => {
error!("Failed to complete multipart upload: {}", e); let detail = DisplayErrorContext(&e).to_string();
error!("Failed to complete multipart upload: {}", detail);
let _ = client let _ = client
.abort_multipart_upload() .abort_multipart_upload()
.bucket(bucket) .bucket(bucket)
@@ -327,7 +334,7 @@ impl StorageProvider for S3Provider {
UploadResult { UploadResult {
storage_id: storage.id.clone(), storage_id: storage.id.clone(),
success: false, success: false,
error: Some(e.to_string()), error: Some(detail),
remote_file_path: None, remote_file_path: None,
total_size: None, total_size: None,
} }
+25 -1
View File
@@ -11,10 +11,13 @@ pub struct Settings {
pub edge_key: String, pub edge_key: String,
pub databases_config_file: String, pub databases_config_file: String,
pub data_path: String, pub data_path: String,
pub pg_bin_dir: String,
pub pooling: usize, pub pooling: usize,
pub timezone: String, pub timezone: String,
pub log: String, pub log: String,
pub chunk_size: usize, // bytes pub chunk_size: usize, // bytes
pub retry_attempts: u32,
pub retry_backoff_ms: u64,
} }
impl Settings { impl Settings {
@@ -48,6 +51,24 @@ impl Settings {
let chunk_size = chunk_size_mb * 1024 * 1024; let chunk_size = chunk_size_mb * 1024 * 1024;
let retry_attempts = env::var("RETRY_ATTEMPTS")
.unwrap_or_else(|_| "3".to_string())
.parse::<u32>()
.expect("RETRY_ATTEMPTS must be a valid positive integer");
if retry_attempts < 3 || retry_attempts > 5 {
panic!("RETRY_ATTEMPTS must be between 3 and 5");
}
let retry_backoff_ms = env::var("RETRY_BACKOFF_MS")
.unwrap_or_else(|_| "1000".to_string())
.parse::<u64>()
.expect("RETRY_BACKOFF_MS must be a valid positive integer");
if retry_backoff_ms < 100 || retry_backoff_ms > 30_000 {
panic!("RETRY_BACKOFF_MS must be between 100 and 30000 milliseconds");
}
let tz = env::var("TZ").unwrap_or_else(|_| "UTC".to_string()); let tz = env::var("TZ").unwrap_or_else(|_| "UTC".to_string());
Self { Self {
@@ -59,10 +80,13 @@ impl Settings {
databases_config_file: env::var("DATABASES_CONFIG_FILE") databases_config_file: env::var("DATABASES_CONFIG_FILE")
.unwrap_or_else(|_| "config.json".into()), .unwrap_or_else(|_| "config.json".into()),
data_path: env::var("DATA_PATH").unwrap_or_else(|_| "/config".into()), data_path: env::var("DATA_PATH").unwrap_or_else(|_| "/config".into()),
pg_bin_dir: env::var("PG_BIN_DIR").unwrap_or_default(),
pooling: pooling_seconds, pooling: pooling_seconds,
timezone: tz, timezone: tz,
log: env::var("LOG").unwrap_or_else(|_| "info".into()), log: env::var("LOG").unwrap_or_else(|_| "info".into()),
chunk_size chunk_size,
retry_attempts,
retry_backoff_ms,
} }
} }
} }
+53
View File
@@ -0,0 +1,53 @@
use super::{env_for, start_cluster};
use crate::domain::postgres::{cluster, connection};
use crate::services::backup::logger::JobLogger;
use crate::tests::init_tracing_for_test;
use std::sync::Arc;
use tempfile::TempDir;
#[tokio::test]
async fn produces_sql_with_roles_and_databases() {
init_tracing_for_test();
let (_c, cfg) = start_cluster("testuser").await;
let dir = TempDir::new().unwrap();
let logger = Arc::new(JobLogger::new());
let sql = cluster::backup::run(cfg.clone(), dir.path().to_path_buf(), env_for(&cfg), logger)
.await
.unwrap();
assert!(sql.is_file());
let contents = std::fs::read_to_string(&sql).unwrap();
assert!(contents.contains("CREATE ROLE"), "expected CREATE ROLE in dump");
assert!(
contents.contains("CREATE DATABASE") || contents.contains("\\connect"),
"expected database statements in dump"
);
}
#[tokio::test]
async fn requires_superuser() {
init_tracing_for_test();
let (_c, super_cfg) = start_cluster("testuser").await;
// Create a NON-superuser login role on the cluster.
let client = connection::connect(&super_cfg).await.unwrap();
client
.batch_execute("CREATE ROLE appuser LOGIN PASSWORD 'changeme' NOSUPERUSER;")
.await
.unwrap();
let mut weak = super_cfg.clone();
weak.username = "appuser".to_string();
let dir = TempDir::new().unwrap();
let logger = Arc::new(JobLogger::new());
let err = cluster::backup::run(weak.clone(), dir.path().to_path_buf(), env_for(&weak), logger)
.await
.unwrap_err();
assert!(
err.to_string().contains("superuser"),
"expected a superuser error, got: {err}"
);
}
+33
View File
@@ -0,0 +1,33 @@
use crate::domain::factory::DatabaseFactory;
use crate::services::config::{DatabaseConfig, DbType};
use std::path::Path;
fn cluster_config() -> DatabaseConfig {
DatabaseConfig {
name: "cluster".to_string(),
database: "postgres".to_string(),
db_type: DbType::PostgresqlCluster,
username: "postgres".to_string(),
password: "changeme".to_string(),
port: 5432,
host: "localhost".to_string(),
generated_id: "40875631-e3d2-4dfe-a26b-2a347ecc64fd".to_string(),
path: String::new(),
max_packet_size: String::new(),
volume_name: String::new(),
container_name: None,
options: std::collections::HashMap::new(),
}
}
#[tokio::test]
async fn factory_routes_cluster_for_backup_with_sql_extension() {
let db = DatabaseFactory::create_for_backup(cluster_config()).await;
assert_eq!(db.file_extension(), ".sql");
}
#[tokio::test]
async fn factory_routes_cluster_for_restore_with_sql_extension() {
let db = DatabaseFactory::create_for_restore(cluster_config(), Path::new("dump.sql")).await;
assert_eq!(db.file_extension(), ".sql");
}
+50
View File
@@ -0,0 +1,50 @@
mod backup;
mod database;
mod restore;
use crate::services::config::{DatabaseConfig, DbType};
use std::collections::HashMap;
use testcontainers::runners::AsyncRunner;
use testcontainers::{ContainerAsync, ImageExt};
use testcontainers_modules::postgres::Postgres;
use url::Host;
async fn start_cluster(user: &str) -> (ContainerAsync<Postgres>, DatabaseConfig) {
let container = Postgres::default()
.with_env_var("POSTGRES_DB", "postgres")
.with_env_var("POSTGRES_USER", user)
.with_env_var("POSTGRES_PASSWORD", "changeme")
.with_tag("17")
.start()
.await
.unwrap();
let host = container
.get_host()
.await
.unwrap_or(Host::parse("127.0.0.1").unwrap());
let port = container.get_host_port_ipv4(5432).await.unwrap_or(5432);
let config = DatabaseConfig {
name: format!("cluster-{}", user),
database: "postgres".to_string(),
db_type: DbType::PostgresqlCluster,
username: user.to_string(),
password: "changeme".to_string(),
port,
host: host.to_string(),
generated_id: "40875631-e3d2-4dfe-a26b-2a347ecc64fd".to_string(),
path: "".to_string(),
max_packet_size: "".to_string(),
volume_name: "".to_string(),
container_name: None,
options: std::collections::HashMap::new(),
};
(container, config)
}
fn env_for(cfg: &DatabaseConfig) -> HashMap<String, String> {
let mut env = std::env::vars().collect::<HashMap<_, _>>();
env.insert("PGPASSWORD".to_string(), cfg.password.clone());
env
}
+83
View File
@@ -0,0 +1,83 @@
use super::{env_for, start_cluster};
use crate::domain::postgres::{cluster, connection};
use crate::services::backup::logger::JobLogger;
use crate::tests::init_tracing_for_test;
use std::sync::Arc;
use tempfile::TempDir;
#[tokio::test]
async fn backup_restore_round_trip_preserves_ownership() {
init_tracing_for_test();
// Source cluster A: seed a role + a table owned by that role.
let (_a, src) = start_cluster("testuser").await;
let client = connection::connect(&src).await.unwrap();
client
.batch_execute(
"CREATE ROLE appowner LOGIN PASSWORD 'changeme' NOSUPERUSER;\n\
CREATE TABLE owned_tbl (id int);\n\
ALTER TABLE owned_tbl OWNER TO appowner;",
)
.await
.unwrap();
let dir = TempDir::new().unwrap();
let sql = cluster::backup::run(src.clone(), dir.path().to_path_buf(), env_for(&src), Arc::new(JobLogger::new()))
.await
.unwrap();
// Target cluster B: fresh, same bootstrap user.
let (_b, mut dst) = start_cluster("testuser").await;
cluster::restore::run(dst.clone(), sql.clone(), env_for(&dst), Arc::new(JobLogger::new()))
.await
.unwrap();
// Verify the seeded role exists and the table's owner was preserved on B.
dst.database = "postgres".to_string();
let bclient = connection::connect(&dst).await.unwrap();
let role_exists: bool = bclient
.query_one("SELECT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'appowner');", &[])
.await
.unwrap()
.get(0);
assert!(role_exists, "appowner role must be recreated on the target");
let owner: String = bclient
.query_one(
"SELECT tableowner FROM pg_tables WHERE tablename = 'owned_tbl';",
&[],
)
.await
.unwrap()
.get(0);
assert_eq!(owner, "appowner", "table ownership must be preserved");
}
#[tokio::test]
async fn requires_superuser() {
init_tracing_for_test();
let (_c, super_cfg) = start_cluster("testuser").await;
// A non-superuser login role must be rejected before psql runs.
let client = connection::connect(&super_cfg).await.unwrap();
client
.batch_execute("CREATE ROLE appuser LOGIN PASSWORD 'changeme' NOSUPERUSER;")
.await
.unwrap();
let mut weak = super_cfg.clone();
weak.username = "appuser".to_string();
// The superuser pre-check happens before the dump file is read, so a
// non-existent restore path is fine — it must never be touched.
let missing = std::path::PathBuf::from("/nonexistent/cluster.sql");
let err = cluster::restore::run(weak.clone(), missing, env_for(&weak), Arc::new(JobLogger::new()))
.await
.unwrap_err();
assert!(
err.to_string().contains("superuser"),
"expected a superuser error, got: {err}"
);
}
+279
View File
@@ -0,0 +1,279 @@
use crate::domain::docker_volume::docker::parse_container_id;
static ENV_GUARD: tokio::sync::Mutex<()> = tokio::sync::Mutex::const_new(());
#[test]
fn parse_container_id_from_mountinfo_line() {
let id = "a".repeat(64);
let mountinfo = format!(
"1234 1000 0:50 / /etc/hostname rw shared:1 - ext4 /var/lib/docker/containers/{id}/hostname rw"
);
assert_eq!(parse_container_id(&mountinfo, ""), Some(id));
}
#[test]
fn parse_container_id_from_cgroup_v1() {
let id = "b".repeat(64);
let cgroup = format!("12:memory:/docker/{id}\n11:cpu:/docker/{id}\n");
assert_eq!(parse_container_id("", &cgroup), Some(id));
}
#[test]
fn parse_container_id_none_on_cgroup_v2() {
assert_eq!(parse_container_id("", "0::/\n"), None);
}
#[tokio::test]
async fn docker_volume_ping_true_for_existing_volume() {
use crate::domain::docker_volume::docker::client;
use bollard::models::VolumeCreateRequest;
use bollard::query_parameters::RemoveVolumeOptions;
let docker = client().expect("docker daemon required for this test");
let vol = format!("portabase-test-{}", uuid::Uuid::new_v4());
docker
.create_volume(VolumeCreateRequest { name: Some(vol.clone()), ..Default::default() })
.await
.unwrap();
let cfg = volume_config(&vol);
let reachable = crate::domain::docker_volume::ping::run(cfg).await.unwrap();
assert!(reachable);
let missing = volume_config("portabase-does-not-exist-xyz");
assert!(!crate::domain::docker_volume::ping::run(missing).await.unwrap());
docker.remove_volume(&vol, None::<RemoveVolumeOptions>).await.ok();
}
fn volume_config(volume_name: &str) -> crate::services::config::DatabaseConfig {
use crate::services::config::{DatabaseConfig, DbType};
DatabaseConfig {
name: "vol-test".to_string(),
database: "".to_string(),
db_type: DbType::DockerVolume,
username: "".to_string(),
password: "".to_string(),
port: 0,
host: "".to_string(),
generated_id: uuid::Uuid::new_v4().to_string(),
path: "".to_string(),
max_packet_size: "".to_string(),
volume_name: volume_name.to_string(),
container_name: None,
options: std::collections::HashMap::new(),
}
}
async fn ensure_image(docker: &bollard::Docker, image: &str) {
use bollard::query_parameters::CreateImageOptionsBuilder;
use futures_util::StreamExt;
let (name, tag) = image.split_once(':').unwrap_or((image, "latest"));
let opts = CreateImageOptionsBuilder::default()
.from_image(name)
.tag(tag)
.build();
let mut stream = docker.create_image(Some(opts), None, None);
while let Some(item) = stream.next().await {
item.unwrap();
}
}
async fn seed_volume(docker: &bollard::Docker, volume: &str, filename: &str, content: &str) {
use bollard::models::{ContainerCreateBody, HostConfig};
use bollard::query_parameters::{
CreateContainerOptions, RemoveContainerOptions, StartContainerOptions,
WaitContainerOptions,
};
use futures_util::StreamExt;
ensure_image(docker, "busybox").await;
let body = ContainerCreateBody {
image: Some("busybox".to_string()),
cmd: Some(vec![
"sh".into(),
"-c".into(),
format!("printf '%s' '{content}' > /vol/{filename}"),
]),
host_config: Some(HostConfig {
binds: Some(vec![format!("{volume}:/vol")]),
..Default::default()
}),
..Default::default()
};
let created = docker
.create_container(None::<CreateContainerOptions>, body)
.await
.unwrap();
docker.start_container(&created.id, None::<StartContainerOptions>).await.unwrap();
let mut wait = docker.wait_container(&created.id, None::<WaitContainerOptions>);
while wait.next().await.is_some() {}
docker
.remove_container(&created.id, Some(RemoveContainerOptions { force: true, ..Default::default() }))
.await
.ok();
}
#[tokio::test]
async fn docker_volume_backup_captures_files() {
use crate::domain::docker_volume::docker::client;
use bollard::models::VolumeCreateRequest;
use bollard::query_parameters::RemoveVolumeOptions;
let _env_guard = ENV_GUARD.lock().await;
unsafe { std::env::set_var("PORTABASE_HELPER_IMAGE", "busybox"); }
let docker = client().expect("docker daemon required");
let vol = format!("portabase-test-{}", uuid::Uuid::new_v4());
docker
.create_volume(VolumeCreateRequest { name: Some(vol.clone()), ..Default::default() })
.await
.unwrap();
seed_volume(&docker, &vol, "hello.txt", "backup-me").await;
let tmp = tempfile::TempDir::new().unwrap();
let cfg = volume_config(&vol);
let logger = std::sync::Arc::new(crate::services::backup::logger::JobLogger::new());
let tar = crate::domain::docker_volume::backup::run(cfg, tmp.path().to_path_buf(), logger)
.await
.unwrap();
assert!(tar.is_file());
let names = tar_entry_names(&tar).await;
assert!(names.iter().any(|n| n.ends_with("hello.txt")), "entries: {names:?}");
docker.remove_volume(&vol, None::<RemoveVolumeOptions>).await.ok();
}
async fn tar_entry_names(tar_path: &std::path::Path) -> Vec<String> {
use tokio_stream::StreamExt;
let f = tokio::fs::File::open(tar_path).await.unwrap();
let mut archive = tokio_tar::Archive::new(f);
let mut names = Vec::new();
let mut entries = archive.entries().unwrap();
while let Some(e) = entries.next().await {
let e = e.unwrap();
names.push(e.path().unwrap().to_string_lossy().to_string());
}
names
}
#[tokio::test]
async fn docker_volume_restore_is_clean_replace() {
use crate::domain::docker_volume::docker::client;
use bollard::models::VolumeCreateRequest;
use bollard::query_parameters::RemoveVolumeOptions;
let _env_guard = ENV_GUARD.lock().await;
unsafe { std::env::set_var("PORTABASE_HELPER_IMAGE", "busybox"); }
let docker = client().expect("docker daemon required");
let vol = format!("portabase-test-{}", uuid::Uuid::new_v4());
docker
.create_volume(VolumeCreateRequest { name: Some(vol.clone()), ..Default::default() })
.await
.unwrap();
seed_volume(&docker, &vol, "keeper.txt", "original").await;
let tmp = tempfile::TempDir::new().unwrap();
let logger = std::sync::Arc::new(crate::services::backup::logger::JobLogger::new());
let tar = crate::domain::docker_volume::backup::run(
volume_config(&vol),
tmp.path().to_path_buf(),
logger.clone(),
)
.await
.unwrap();
seed_volume(&docker, &vol, "drift.txt", "added-later").await;
// Restore uploads the raw Docker tar directly.
crate::domain::docker_volume::restore::run(volume_config(&vol), tar.clone(), logger)
.await
.unwrap();
let listing = list_volume(&docker, &vol).await;
assert!(listing.contains("keeper.txt"), "listing: {listing}");
assert!(!listing.contains("drift.txt"), "clean-replace failed, listing: {listing}");
docker.remove_volume(&vol, None::<RemoveVolumeOptions>).await.ok();
}
async fn list_volume(docker: &bollard::Docker, volume: &str) -> String {
use bollard::models::{ContainerCreateBody, HostConfig};
use bollard::query_parameters::{
CreateContainerOptions, LogsOptions, RemoveContainerOptions, StartContainerOptions,
WaitContainerOptions,
};
use tokio_stream::StreamExt;
ensure_image(docker, "busybox").await;
let body = ContainerCreateBody {
image: Some("busybox".to_string()),
cmd: Some(vec!["sh".into(), "-c".into(), "ls -A /vol".into()]),
host_config: Some(HostConfig {
binds: Some(vec![format!("{volume}:/vol")]),
..Default::default()
}),
..Default::default()
};
let created = docker.create_container(None::<CreateContainerOptions>, body).await.unwrap();
docker.start_container(&created.id, None::<StartContainerOptions>).await.unwrap();
let mut wait = docker.wait_container(&created.id, None::<WaitContainerOptions>);
while wait.next().await.is_some() {}
let mut logs = docker.logs(
&created.id,
Some(LogsOptions { stdout: true, stderr: false, ..Default::default() }),
);
let mut out = String::new();
while let Some(chunk) = logs.next().await {
if let Ok(l) = chunk {
out.push_str(&l.to_string());
}
}
docker
.remove_container(&created.id, Some(RemoveContainerOptions { force: true, ..Default::default() }))
.await
.ok();
out
}
#[tokio::test]
async fn sweep_removes_labeled_helpers() {
use crate::domain::docker_volume::docker::{client, create_helper, sweep_ephemeral, EPHEMERAL_LABEL};
use bollard::models::VolumeCreateRequest;
use bollard::query_parameters::{ListContainersOptions, RemoveVolumeOptions};
use std::collections::HashMap;
let _env_guard = ENV_GUARD.lock().await;
unsafe { std::env::set_var("PORTABASE_HELPER_IMAGE", "busybox"); }
let docker = client().expect("docker daemon required");
let vol = format!("portabase-test-{}", uuid::Uuid::new_v4());
docker
.create_volume(VolumeCreateRequest { name: Some(vol.clone()), ..Default::default() })
.await
.unwrap();
ensure_image(&docker, "busybox").await;
let helper = create_helper(&docker, "busybox", &vol, "sweep-test", true, None).await.unwrap();
let removed = sweep_ephemeral(&docker).await.unwrap();
assert!(removed >= 1);
let mut filters = HashMap::new();
filters.insert("label".to_string(), vec![format!("{EPHEMERAL_LABEL}=true")]);
let remaining = docker
.list_containers(Some(ListContainersOptions { all: true, filters: Some(filters), ..Default::default() }))
.await
.unwrap();
assert!(remaining.iter().all(|c| c.id.as_deref() != Some(helper.id.as_str())));
docker.remove_volume(&vol, None::<RemoveVolumeOptions>).await.ok();
}
+4
View File
@@ -39,6 +39,10 @@ async fn create_config() -> (ContainerAsync<GenericImage>, DatabaseConfig) {
host, host,
generated_id: "3c445eb4-c2c6-4bde-a423-ee1385dcf6d2".to_string(), generated_id: "3c445eb4-c2c6-4bde-a423-ee1385dcf6d2".to_string(),
path: "".to_string(), path: "".to_string(),
max_packet_size: "".to_string(),
volume_name: "".to_string(),
container_name: None,
options: std::collections::HashMap::new(),
}; };
(container, config) (container, config)
+4
View File
@@ -31,6 +31,10 @@ async fn create_config() -> (ContainerAsync<Mariadb>, DatabaseConfig) {
host: host.to_string(), host: host.to_string(),
generated_id: "3c4b4eb4-c2c6-4bde-a423-ee1385dcf6d2".to_string(), generated_id: "3c4b4eb4-c2c6-4bde-a423-ee1385dcf6d2".to_string(),
path: "".to_string(), path: "".to_string(),
max_packet_size: "512M".to_string(),
volume_name: "".to_string(),
container_name: None,
options: std::collections::HashMap::new(),
}; };
(container, config) (container, config)
+2
View File
@@ -2,7 +2,9 @@ mod mariadb;
mod mongodb; mod mongodb;
mod mysql; mod mysql;
mod postgres; mod postgres;
mod cluster;
mod redis; mod redis;
mod valkey; mod valkey;
mod firebird; mod firebird;
mod mssql; mod mssql;
mod docker_volume;
+4
View File
@@ -29,6 +29,10 @@ async fn create_config() -> (ContainerAsync<Mongo>, DatabaseConfig) {
host: host.to_string(), host: host.to_string(),
generated_id: "96d30a9f-ff4b-47c9-aaab-f3147bb34f16".to_string(), generated_id: "96d30a9f-ff4b-47c9-aaab-f3147bb34f16".to_string(),
path: "".to_string(), path: "".to_string(),
max_packet_size: "".to_string(),
volume_name: "".to_string(),
container_name: None,
options: std::collections::HashMap::new(),
}; };
(container, config) (container, config)
+4
View File
@@ -54,6 +54,10 @@ fn make_config(host: String, port: u16, database: &str, generated_id: &str) -> D
host, host,
generated_id: generated_id.to_string(), generated_id: generated_id.to_string(),
path: "".to_string(), path: "".to_string(),
max_packet_size: "".to_string(),
volume_name: "".to_string(),
container_name: None,
options: std::collections::HashMap::new(),
} }
} }
+4
View File
@@ -31,6 +31,10 @@ async fn create_config() -> (ContainerAsync<Mysql>, DatabaseConfig) {
host: host.to_string(), host: host.to_string(),
generated_id: "0f1bb8f2-35a0-4c91-8098-e36873d3ce31".to_string(), generated_id: "0f1bb8f2-35a0-4c91-8098-e36873d3ce31".to_string(),
path: "".to_string(), path: "".to_string(),
max_packet_size: "512M".to_string(),
volume_name: "".to_string(),
container_name: None,
options: std::collections::HashMap::new(),
}; };
(container, config) (container, config)
+830
View File
@@ -1,9 +1,14 @@
use crate::domain::factory::DatabaseFactory; use crate::domain::factory::DatabaseFactory;
use crate::domain::postgres::connection::{pg_restore_binary_name, select_pg_path, server_version};
use crate::domain::postgres::format::PostgresDumpFormat;
use crate::domain::postgres::restore::prepare_archive;
use crate::services::backup::logger::JobLogger;
use crate::services::config::{DatabaseConfig, DbType}; use crate::services::config::{DatabaseConfig, DbType};
use crate::tests::init_tracing_for_test; use crate::tests::init_tracing_for_test;
use crate::utils::compress::{compress_to_tar_gz_large, decompress_large_tar_gz}; use crate::utils::compress::{compress_to_tar_gz_large, decompress_large_tar_gz};
use oauth2::url; use oauth2::url;
use std::path::PathBuf; use std::path::PathBuf;
use std::sync::Arc;
use tempfile::TempDir; use tempfile::TempDir;
use testcontainers::runners::AsyncRunner; use testcontainers::runners::AsyncRunner;
use testcontainers::{ContainerAsync, ImageExt}; use testcontainers::{ContainerAsync, ImageExt};
@@ -38,6 +43,10 @@ async fn create_config() -> (ContainerAsync<Postgres>, DatabaseConfig) {
host: host.to_string(), host: host.to_string(),
generated_id: "40875631-e3d2-4dfe-a26b-2a347ecc64fd".to_string(), generated_id: "40875631-e3d2-4dfe-a26b-2a347ecc64fd".to_string(),
path: "".to_string(), path: "".to_string(),
max_packet_size: "".to_string(),
volume_name: "".to_string(),
container_name: None,
options: std::collections::HashMap::new(),
}; };
(container, config) (container, config)
@@ -55,6 +64,44 @@ async fn postgres_ping_test() {
assert_eq!(reachable, true); assert_eq!(reachable, true);
} }
#[tokio::test]
async fn is_superuser_detects_superuser_role() {
init_tracing_for_test();
let (_container, config) = create_config().await;
let is_super = crate::domain::postgres::connection::is_superuser(&config)
.await
.unwrap();
assert!(is_super);
}
#[tokio::test]
async fn can_drop_database_false_for_unprivileged_role() {
init_tracing_for_test();
let (_container, admin) = create_config().await;
let a = crate::domain::postgres::connection::connect(&admin)
.await
.unwrap();
a.batch_execute("DROP ROLE IF EXISTS lowpriv; CREATE ROLE lowpriv LOGIN PASSWORD 'x';")
.await
.unwrap();
let mut low = admin.clone();
low.username = "lowpriv".into();
low.password = "x".into();
assert_eq!(
crate::domain::postgres::connection::can_drop_database(&low)
.await
.unwrap(),
false
);
}
#[tokio::test] #[tokio::test]
async fn postgres_backup_restore_test() { async fn postgres_backup_restore_test() {
init_tracing_for_test(); init_tracing_for_test();
@@ -107,3 +154,786 @@ async fn postgres_backup_restore_test() {
} }
} }
} }
#[tokio::test]
async fn postgres_password_with_slash_test() {
init_tracing_for_test();
let special_password = "ch/ange:me@1";
let container = Postgres::default()
.with_env_var("POSTGRES_DB", "testdb")
.with_env_var("POSTGRES_USER", "testuser")
.with_env_var("POSTGRES_PASSWORD", special_password)
.with_tag("17")
.start()
.await
.unwrap();
let host = container
.get_host()
.await
.unwrap_or(Host::parse("127.0.0.1").unwrap());
let port = container.get_host_port_ipv4(5432).await.unwrap_or(5432);
let config = DatabaseConfig {
name: "My test Postgres Database with slash password".to_string(),
database: "testdb".to_string(),
db_type: DbType::Postgresql,
username: "testuser".to_string(),
password: special_password.to_string(),
port,
host: host.to_string(),
generated_id: "5a1f0e3c-9b8a-4a8e-9b1b-0a1c2d3e4f5a".to_string(),
path: "".to_string(),
max_packet_size: "".to_string(),
volume_name: "".to_string(),
container_name: None,
options: std::collections::HashMap::new(),
};
let db = DatabaseFactory::create_for_backup(config.clone()).await;
let reachable = db.ping().await.unwrap_or(false);
assert_eq!(reachable, true);
}
fn pg_dump_env(config: &DatabaseConfig) -> std::collections::HashMap<String, String> {
let mut env = std::env::vars().collect::<std::collections::HashMap<_, _>>();
env.insert("PGPASSWORD".to_string(), config.password.clone());
env
}
#[tokio::test]
async fn prepare_archive_fd_locates_toc_dir() {
init_tracing_for_test();
let (_container, config) = create_config().await;
let temp_dir = TempDir::new().unwrap();
let backup_path = crate::domain::postgres::backup::run(
config.clone(),
PostgresDumpFormat::Fd,
temp_dir.path().to_path_buf(),
pg_dump_env(&config),
Arc::new(JobLogger::new()),
)
.await
.unwrap();
let compression = compress_to_tar_gz_large(&backup_path, Arc::new(JobLogger::new()))
.await
.unwrap();
assert!(compression.compressed_path.is_file());
let version = server_version(&config).await.unwrap();
let pg_restore = select_pg_path(&version).join(pg_restore_binary_name());
let logger = JobLogger::new();
let prepared = prepare_archive(
PostgresDumpFormat::Fd,
&compression.compressed_path,
&pg_restore,
&logger,
)
.unwrap();
assert!(prepared.path().join("toc.dat").exists());
assert!(!prepared.toc().is_empty());
}
#[tokio::test]
async fn prepare_archive_fc_returns_file_path_unchanged() {
init_tracing_for_test();
let (_container, config) = create_config().await;
let temp_dir = TempDir::new().unwrap();
let backup_path = crate::domain::postgres::backup::run(
config.clone(),
PostgresDumpFormat::Fc,
temp_dir.path().to_path_buf(),
pg_dump_env(&config),
Arc::new(JobLogger::new()),
)
.await
.unwrap();
assert!(backup_path.is_file());
let version = server_version(&config).await.unwrap();
let pg_restore = select_pg_path(&version).join(pg_restore_binary_name());
let logger = JobLogger::new();
let prepared = prepare_archive(PostgresDumpFormat::Fc, &backup_path, &pg_restore, &logger).unwrap();
assert_eq!(prepared.path(), backup_path.as_path());
assert!(!prepared.toc().is_empty());
}
#[tokio::test]
async fn restore_run_unified_fc_roundtrip() {
init_tracing_for_test();
let (_container, config) = create_config().await;
let client = crate::domain::postgres::connection::connect(&config)
.await
.unwrap();
client.execute("CREATE TABLE t(id int);", &[]).await.unwrap();
let temp_dir = TempDir::new().unwrap();
let dump_file = crate::domain::postgres::backup::run(
config.clone(),
PostgresDumpFormat::Fc,
temp_dir.path().to_path_buf(),
pg_dump_env(&config),
Arc::new(JobLogger::new()),
)
.await
.unwrap();
assert!(dump_file.is_file());
let format = crate::domain::postgres::connection::detect_format_from_file(&dump_file);
let result = crate::domain::postgres::restore::run(
config.clone(),
format,
dump_file,
pg_dump_env(&config),
Arc::new(JobLogger::new()),
)
.await;
assert!(result.is_ok(), "restore::run failed: {:?}", result);
}
#[tokio::test]
async fn drop_all_schemas_removes_user_schema() {
init_tracing_for_test();
let (_container, config) = create_config().await;
let client = crate::domain::postgres::connection::connect(&config)
.await
.unwrap();
client
.batch_execute("CREATE SCHEMA IF NOT EXISTS extra_ns; CREATE TABLE IF NOT EXISTS extra_ns.t(id int);")
.await
.unwrap();
let dropped = crate::domain::postgres::connection::drop_all_schemas(&config)
.await
.unwrap();
assert!(dropped.iter().any(|s| s == "extra_ns"));
let client = crate::domain::postgres::connection::connect(&config)
.await
.unwrap();
let row = client
.query_one(
"SELECT count(*) FROM pg_namespace WHERE nspname = 'extra_ns'",
&[],
)
.await
.unwrap();
let n: i64 = row.get(0);
assert_eq!(n, 0);
}
#[tokio::test]
async fn restore_drop_schemas_removes_extra_objects() {
init_tracing_for_test();
let (_container, config) = create_config().await;
let client = crate::domain::postgres::connection::connect(&config)
.await
.unwrap();
client
.batch_execute("CREATE TABLE base_t(id int);")
.await
.unwrap();
let temp_dir = TempDir::new().unwrap();
let dump_file = crate::domain::postgres::backup::run(
config.clone(),
PostgresDumpFormat::Fc,
temp_dir.path().to_path_buf(),
pg_dump_env(&config),
Arc::new(JobLogger::new()),
)
.await
.unwrap();
let client = crate::domain::postgres::connection::connect(&config)
.await
.unwrap();
client
.batch_execute("CREATE TABLE orphan_only_here(id int);")
.await
.unwrap();
let mut cfg = config.clone();
cfg.options
.insert("clean_mode".into(), serde_json::json!("drop_schemas"));
let format = crate::domain::postgres::connection::detect_format_from_file(&dump_file);
let result = crate::domain::postgres::restore::run(
cfg.clone(),
format,
dump_file,
pg_dump_env(&cfg),
Arc::new(JobLogger::new()),
)
.await;
assert!(result.is_ok(), "restore::run failed: {:?}", result);
let client = crate::domain::postgres::connection::connect(&config)
.await
.unwrap();
let n: i64 = client
.query_one(
"SELECT count(*) FROM information_schema.tables WHERE table_name = 'orphan_only_here'",
&[],
)
.await
.unwrap()
.get(0);
assert_eq!(n, 0);
let n: i64 = client
.query_one(
"SELECT count(*) FROM information_schema.tables WHERE table_name = 'base_t'",
&[],
)
.await
.unwrap()
.get(0);
assert_eq!(n, 1);
}
#[tokio::test]
async fn restore_clean_leaves_divergent_object() {
init_tracing_for_test();
let (_container, config) = create_config().await;
let client = crate::domain::postgres::connection::connect(&config)
.await
.unwrap();
client
.batch_execute("CREATE TABLE base_t(id int);")
.await
.unwrap();
let temp_dir = TempDir::new().unwrap();
let dump_file = crate::domain::postgres::backup::run(
config.clone(),
PostgresDumpFormat::Fc,
temp_dir.path().to_path_buf(),
pg_dump_env(&config),
Arc::new(JobLogger::new()),
)
.await
.unwrap();
let client = crate::domain::postgres::connection::connect(&config)
.await
.unwrap();
client
.batch_execute("CREATE TABLE survives_clean(id int);")
.await
.unwrap();
let mut cfg = config.clone();
cfg.options
.insert("clean_mode".into(), serde_json::json!("clean"));
let format = crate::domain::postgres::connection::detect_format_from_file(&dump_file);
let result = crate::domain::postgres::restore::run(
cfg.clone(),
format,
dump_file,
pg_dump_env(&cfg),
Arc::new(JobLogger::new()),
)
.await;
assert!(result.is_ok(), "restore::run failed: {:?}", result);
let client = crate::domain::postgres::connection::connect(&config)
.await
.unwrap();
let n: i64 = client
.query_one(
"SELECT count(*) FROM information_schema.tables WHERE table_name = 'survives_clean'",
&[],
)
.await
.unwrap()
.get(0);
assert_eq!(n, 1, "clean mode is not a reset; divergent object survives");
}
#[tokio::test]
async fn restore_unknown_clean_mode_falls_back() {
init_tracing_for_test();
let (_container, config) = create_config().await;
let temp_dir = TempDir::new().unwrap();
let dump_file = crate::domain::postgres::backup::run(
config.clone(),
PostgresDumpFormat::Fc,
temp_dir.path().to_path_buf(),
pg_dump_env(&config),
Arc::new(JobLogger::new()),
)
.await
.unwrap();
let mut cfg = config.clone();
cfg.options
.insert("clean_mode".into(), serde_json::json!("wat"));
let format = crate::domain::postgres::connection::detect_format_from_file(&dump_file);
let logger = Arc::new(JobLogger::new());
let result = crate::domain::postgres::restore::run(
cfg.clone(),
format,
dump_file,
pg_dump_env(&cfg),
logger.clone(),
)
.await;
assert!(result.is_ok(), "restore::run failed: {:?}", result);
let entries = Arc::try_unwrap(logger)
.expect("logger should have a single owner after run() completes")
.into_entries();
assert!(entries
.iter()
.any(|e| e.message.contains("Unknown clean_mode 'wat'")));
}
#[tokio::test]
async fn drop_database_preflight_preserves_data_when_unprivileged() {
init_tracing_for_test();
let (_container, admin) = create_config().await;
let a = crate::domain::postgres::connection::connect(&admin)
.await
.unwrap();
a.batch_execute("DROP ROLE IF EXISTS lowpriv2; CREATE ROLE lowpriv2 LOGIN PASSWORD 'x';")
.await
.unwrap();
a.batch_execute("CREATE TABLE IF NOT EXISTS keep_me(id int);")
.await
.unwrap();
let temp_dir = TempDir::new().unwrap();
let dump_file = crate::domain::postgres::backup::run(
admin.clone(),
PostgresDumpFormat::Fc,
temp_dir.path().to_path_buf(),
pg_dump_env(&admin),
Arc::new(JobLogger::new()),
)
.await
.unwrap();
let mut low = admin.clone();
low.username = "lowpriv2".into();
low.password = "x".into();
low.options
.insert("clean_mode".into(), serde_json::json!("drop_database"));
let format = crate::domain::postgres::connection::detect_format_from_file(&dump_file);
let res = crate::domain::postgres::restore::run(
low.clone(),
format,
dump_file,
pg_dump_env(&low),
Arc::new(JobLogger::new()),
)
.await;
assert!(res.is_err(), "preflight must reject an unprivileged role");
let a = crate::domain::postgres::connection::connect(&admin)
.await
.unwrap();
let n: i64 = a
.query_one(
"SELECT count(*) FROM information_schema.tables WHERE table_name = 'keep_me'",
&[],
)
.await
.unwrap()
.get(0);
assert_eq!(n, 1, "preflight must fail before dropping anything");
}
#[tokio::test]
async fn drop_database_preserves_encoding_and_owner() {
init_tracing_for_test();
let (_container, config) = create_config().await;
let client = crate::domain::postgres::connection::connect(&config)
.await
.unwrap();
client
.batch_execute("CREATE TABLE base_t(id int);")
.await
.unwrap();
let temp_dir = TempDir::new().unwrap();
let dump_file = crate::domain::postgres::backup::run(
config.clone(),
PostgresDumpFormat::Fc,
temp_dir.path().to_path_buf(),
pg_dump_env(&config),
Arc::new(JobLogger::new()),
)
.await
.unwrap();
let before = crate::domain::postgres::connection::connect(&config)
.await
.unwrap()
.query_one(
"SELECT pg_encoding_to_char(encoding), datcollate FROM pg_database WHERE datname = current_database()",
&[],
)
.await
.unwrap();
let enc0: String = before.get(0);
let coll0: String = before.get(1);
let mut cfg = config.clone();
cfg.options
.insert("clean_mode".into(), serde_json::json!("drop_database"));
let format = crate::domain::postgres::connection::detect_format_from_file(&dump_file);
let result = crate::domain::postgres::restore::run(
cfg.clone(),
format,
dump_file,
pg_dump_env(&cfg),
Arc::new(JobLogger::new()),
)
.await;
assert!(result.is_ok(), "restore::run failed: {:?}", result);
let after = crate::domain::postgres::connection::connect(&config)
.await
.unwrap()
.query_one(
"SELECT pg_encoding_to_char(encoding), datcollate FROM pg_database WHERE datname = current_database()",
&[],
)
.await
.unwrap();
let enc1: String = after.get(0);
let coll1: String = after.get(1);
assert_eq!(enc0, enc1);
assert_eq!(coll0, coll1);
}
#[tokio::test]
async fn drop_database_force_wins_over_open_connection() {
init_tracing_for_test();
let (_container, config) = create_config().await;
let temp_dir = TempDir::new().unwrap();
let dump_file = crate::domain::postgres::backup::run(
config.clone(),
PostgresDumpFormat::Fc,
temp_dir.path().to_path_buf(),
pg_dump_env(&config),
Arc::new(JobLogger::new()),
)
.await
.unwrap();
let mut cfg = config.clone();
cfg.options
.insert("clean_mode".into(), serde_json::json!("drop_database"));
let squatter = crate::domain::postgres::connection::connect(&config)
.await
.unwrap();
let _keep = tokio::spawn(async move {
let _ = squatter.query_one("SELECT pg_sleep(5)", &[]).await;
});
let format = crate::domain::postgres::connection::detect_format_from_file(&dump_file);
let result = crate::domain::postgres::restore::run(
cfg.clone(),
format,
dump_file,
pg_dump_env(&cfg),
Arc::new(JobLogger::new()),
)
.await;
assert!(result.is_ok(), "restore::run failed: {:?}", result);
}
#[test]
fn sniff_format_detects_custom_and_gzip() {
use crate::domain::postgres::connection::sniff_format;
let dir = TempDir::new().unwrap();
let fc = dir.path().join("a.dump");
std::fs::write(&fc, b"PGDMP\x01\x0e\x00").unwrap();
assert_eq!(sniff_format(&fc).unwrap(), PostgresDumpFormat::Fc);
let fd = dir.path().join("b.gz");
std::fs::write(&fd, [0x1f, 0x8b, 0x08, 0x00]).unwrap();
assert_eq!(sniff_format(&fd).unwrap(), PostgresDumpFormat::Fd);
let bad = dir.path().join("c.bin");
std::fs::write(&bad, b"not a dump").unwrap();
assert!(sniff_format(&bad).is_err());
}
#[tokio::test]
async fn corrupt_archive_leaves_database_untouched() {
init_tracing_for_test();
let (_container, config) = create_config().await;
let client = crate::domain::postgres::connection::connect(&config)
.await
.unwrap();
client
.batch_execute("CREATE TABLE must_survive(id int);")
.await
.unwrap();
let mut cfg = config.clone();
cfg.options
.insert("clean_mode".into(), serde_json::json!("drop_schemas"));
let dir = TempDir::new().unwrap();
let broken_file = dir.path().join("broken.tar.gz");
std::fs::write(&broken_file, [0x1f, 0x8b, 0x08, 0x00, 0xde, 0xad, 0xbe, 0xef]).unwrap();
let result = crate::domain::postgres::restore::run(
cfg.clone(),
PostgresDumpFormat::Fd,
broken_file,
pg_dump_env(&cfg),
Arc::new(JobLogger::new()),
)
.await;
assert!(result.is_err(), "corrupt archive must be rejected before any destructive step");
let client = crate::domain::postgres::connection::connect(&config)
.await
.unwrap();
let n: i64 = client
.query_one(
"SELECT count(*) FROM information_schema.tables WHERE table_name = 'must_survive'",
&[],
)
.await
.unwrap()
.get(0);
assert_eq!(n, 1, "corrupt archive must never trigger the schema drop");
}
mod select_pg_path_tests {
use crate::domain::postgres::connection::{
pg_dump_binary_name, pg_dump_exists_in, pg_dumpall_binary_name, pg_restore_binary_name,
psql_binary_name, select_pg_path_with,
};
#[test]
fn respects_pg_bin_dir_override() {
let custom = if cfg!(target_os = "windows") {
r"C:\custom\pg\bin"
} else {
"/custom/pg/bin"
};
let path = select_pg_path_with("16.4", custom);
assert_eq!(path, std::path::PathBuf::from(custom));
}
#[test]
fn pg_bin_dir_override_ignores_requested_version() {
let custom = if cfg!(target_os = "windows") {
r"C:\custom\pg\bin"
} else {
"/custom/pg/bin"
};
let path = select_pg_path_with("not-a-version", custom);
assert_eq!(path, std::path::PathBuf::from(custom));
}
#[test]
fn empty_pg_bin_dir_falls_through_to_detection() {
let path = select_pg_path_with("17", "");
assert_ne!(path, std::path::PathBuf::from(""));
}
#[test]
fn pg_dump_binary_name_is_platform_specific() {
let name = pg_dump_binary_name();
if cfg!(target_os = "windows") {
assert_eq!(name, "pg_dump.exe");
} else {
assert_eq!(name, "pg_dump");
}
}
#[test]
fn pg_dump_exists_in_is_false_for_nonexistent_dir() {
let dir = std::path::Path::new("this/path/almost-certainly/does-not-exist-12345");
assert!(!pg_dump_exists_in(dir));
}
#[test]
fn pg_dumpall_binary_name_is_platform_specific() {
let name = pg_dumpall_binary_name();
if cfg!(target_os = "windows") {
assert_eq!(name, "pg_dumpall.exe");
} else {
assert_eq!(name, "pg_dumpall");
}
}
#[test]
fn psql_binary_name_is_platform_specific() {
let name = psql_binary_name();
if cfg!(target_os = "windows") {
assert_eq!(name, "psql.exe");
} else {
assert_eq!(name, "psql");
}
}
#[test]
fn pg_restore_binary_name_is_platform_correct() {
let name = pg_restore_binary_name();
if cfg!(target_os = "windows") {
assert_eq!(name, "pg_restore.exe");
} else {
assert_eq!(name, "pg_restore");
}
}
}
mod quoting_tests {
use crate::domain::postgres::connection::{quote_ident, quote_literal};
#[test]
fn quote_ident_escapes_double_quotes() {
assert_eq!(quote_ident("devdb"), "\"devdb\"");
assert_eq!(quote_ident("a\"b"), "\"a\"\"b\"");
assert_eq!(quote_ident("drop\"; --"), "\"drop\"\"; --\"");
}
#[test]
fn quote_literal_escapes_single_quotes() {
assert_eq!(quote_literal("UTF8"), "'UTF8'");
assert_eq!(quote_literal("O'Brien"), "'O''Brien'");
}
}
mod clean_mode_tests {
use crate::domain::postgres::clean_mode::RestoreCleanMode as M;
use crate::services::config::{DatabaseConfig, DbType};
fn cfg_with(clean_mode: Option<&str>) -> DatabaseConfig {
let mut options = std::collections::HashMap::new();
if let Some(v) = clean_mode {
options.insert("clean_mode".to_string(), serde_json::json!(v));
}
DatabaseConfig {
name: "t".into(),
database: "testdb".into(),
db_type: DbType::Postgresql,
username: "testuser".into(),
password: "changeme".into(),
port: 5432,
host: "localhost".into(),
generated_id: "00000000-0000-0000-0000-000000000000".into(),
path: "".into(),
max_packet_size: "".into(),
volume_name: "".into(),
container_name: None,
options,
}
}
#[test]
fn clean_mode_parsing() {
assert_eq!(M::from_config(&cfg_with(None)), (M::Clean, None));
assert_eq!(M::from_config(&cfg_with(Some("clean"))), (M::Clean, None));
assert_eq!(M::from_config(&cfg_with(Some("none"))), (M::None, None));
assert_eq!(
M::from_config(&cfg_with(Some("drop_schemas"))),
(M::DropSchemas, None)
);
assert_eq!(
M::from_config(&cfg_with(Some("drop_database"))),
(M::DropDatabase, None)
);
assert_eq!(
M::from_config(&cfg_with(Some("bogus"))),
(M::Clean, Some("bogus".to_string()))
);
}
#[test]
fn uses_pg_restore_clean_behavior() {
assert!(M::Clean.uses_pg_restore_clean());
assert!(!M::DropSchemas.uses_pg_restore_clean());
assert!(!M::None.uses_pg_restore_clean());
assert!(!M::DropDatabase.uses_pg_restore_clean());
}
}
mod toc_tests {
use crate::domain::postgres::restore::toc_creates_public_schema;
#[test]
fn toc_public_schema_detection() {
let with = "215; 2615 2200 SCHEMA - public pg_database_owner";
let without_table = "200; 1259 12346 TABLE devschema users devuser";
let without_similar_schema = "216; 2615 2201 SCHEMA - publicish someowner";
assert!(toc_creates_public_schema(with));
assert!(!toc_creates_public_schema(without_table));
assert!(!toc_creates_public_schema(without_similar_schema));
}
}
+4
View File
@@ -28,6 +28,10 @@ async fn create_config() -> (ContainerAsync<Redis>, DatabaseConfig) {
host: host.to_string(), host: host.to_string(),
generated_id: "40875631-e3d2-4dfe-a26b-2a347ecc64fd".to_string(), generated_id: "40875631-e3d2-4dfe-a26b-2a347ecc64fd".to_string(),
path: "".to_string(), path: "".to_string(),
max_packet_size: "".to_string(),
volume_name: "".to_string(),
container_name: None,
options: std::collections::HashMap::new(),
}; };
(container, config) (container, config)
+4
View File
@@ -27,6 +27,10 @@ async fn create_config() -> (ContainerAsync<Valkey>, DatabaseConfig) {
host: host.to_string(), host: host.to_string(),
generated_id: "40875485-e3d2-4dfe-a26b-2a347ecc64fd".to_string(), generated_id: "40875485-e3d2-4dfe-a26b-2a347ecc64fd".to_string(),
path: "".to_string(), path: "".to_string(),
max_packet_size: "".to_string(),
volume_name: "".to_string(),
container_name: None,
options: std::collections::HashMap::new(),
}; };
(container, config) (container, config)
+1
View File
@@ -1,5 +1,6 @@
mod domain; mod domain;
mod services; mod services;
mod storage;
mod utils; mod utils;
use once_cell::sync::Lazy; use once_cell::sync::Lazy;
+134 -1
View File
@@ -2,7 +2,7 @@ use serde_json::json;
use crate::services::api::models::agent::backup::{BackupResponse, BackupUploadResponse}; use crate::services::api::models::agent::backup::{BackupResponse, BackupUploadResponse};
use crate::services::api::models::agent::restore::ResultRestoreResponse; use crate::services::api::models::agent::restore::ResultRestoreResponse;
use crate::services::api::models::agent::status::PingResult; use crate::services::api::models::agent::status::{DatabaseStatus, PingResult};
#[test] #[test]
fn backup_response_deserializes_nested_backup_id() { fn backup_response_deserializes_nested_backup_id() {
@@ -110,3 +110,136 @@ fn ping_result_deserializes_and_normalizes_storage_config_keys() {
assert!(result.databases[0].data.restore.file.is_none()); assert!(result.databases[0].data.restore.file.is_none());
assert!(result.databases[0].data.restore.meta_file.is_none()); assert!(result.databases[0].data.restore.meta_file.is_none());
} }
#[test]
fn database_status_legacy_plaintext_storages() {
let status: DatabaseStatus = serde_json::from_value(json!({
"dbms": "postgres",
"generatedId": "gen-1",
"storages": [ { "id": "s1", "config": { "bucket": "b" }, "provider": "s3" } ],
"encrypt": true,
"data": {
"backup": { "action": false, "cron": null },
"restore": { "action": false, "file": null, "metaFile": null, "size": null }
}
})).unwrap();
assert_eq!(status.storages.len(), 1);
assert_eq!(status.storages_encrypted, None);
assert!(status.storages_ciphertext.is_none());
}
#[test]
fn database_status_encrypted_envelope() {
let status: DatabaseStatus = serde_json::from_value(json!({
"dbms": "postgres",
"generatedId": "gen-1",
"storages": [],
"storages_encrypted": true,
"storages_ciphertext": "AQIDBA==",
"encrypt": true,
"data": {
"backup": { "action": true, "cron": null },
"restore": { "action": false, "file": null, "metaFile": null, "size": null }
}
})).unwrap();
assert!(status.storages.is_empty());
assert_eq!(status.storages_encrypted, Some(true));
assert_eq!(status.storages_ciphertext.as_deref(), Some("AQIDBA=="));
}
#[test]
fn database_status_defaults_config_fields_absent() {
let json = r#"{
"dbms": "postgresql",
"generatedId": "16678159-ff7e-4c97-8c83-0adeff214681",
"encrypt": false,
"data": { "backup": { "action": false, "cron": null },
"restore": { "action": false, "file": null, "metaFile": null, "size": null } }
}"#;
let status: crate::services::api::models::agent::status::DatabaseStatus =
serde_json::from_str(json).unwrap();
assert_eq!(status.config_encrypted, None);
assert!(status.config_ciphertext.is_none());
assert!(status.resolved_config.is_none());
}
#[test]
fn resolve_dashboard_config_decrypts_full_entry() {
use crate::services::status::resolve_dashboard_config;
use base64::{engine::general_purpose, Engine};
// 32-byte master key, base64 STANDARD (matches decrypt_json_gcm).
let master_key_b64 = general_purpose::STANDARD.encode([7u8; 32]);
// Full agent-entry shape the dashboard encrypts.
let entry = r#"{
"name": "Dashboard PG",
"type": "postgresql",
"database": "app",
"username": "postgres",
"password": "s3cret",
"port": 5432,
"host": "10.0.0.10",
"generated_id": "16678159-ff7e-4c97-8c83-0adeff214681"
}"#;
let ciphertext = encrypt_json_gcm(entry.as_bytes(), &master_key_b64);
let mut status: crate::services::api::models::agent::status::DatabaseStatus =
serde_json::from_str(
r#"{
"dbms": "postgresql",
"generatedId": "16678159-ff7e-4c97-8c83-0adeff214681",
"encrypt": false,
"config_encrypted": true,
"config_ciphertext": "PLACEHOLDER",
"data": { "backup": { "action": false, "cron": null },
"restore": { "action": false, "file": null, "metaFile": null, "size": null } }
}"#,
)
.unwrap();
status.config_ciphertext = Some(ciphertext);
resolve_dashboard_config(&mut status, &master_key_b64).unwrap();
let cfg = status.resolved_config.expect("resolved");
assert_eq!(cfg.name, "Dashboard PG");
assert_eq!(cfg.password, "s3cret");
assert_eq!(cfg.host, "10.0.0.10");
assert_eq!(cfg.db_type.as_str(), "postgresql");
}
#[test]
fn resolve_dashboard_config_noop_when_not_encrypted() {
use crate::services::status::resolve_dashboard_config;
let mut status: crate::services::api::models::agent::status::DatabaseStatus =
serde_json::from_str(
r#"{
"dbms": "postgresql",
"generatedId": "16678159-ff7e-4c97-8c83-0adeff214681",
"encrypt": false,
"data": { "backup": { "action": false, "cron": null },
"restore": { "action": false, "file": null, "metaFile": null, "size": null } }
}"#,
)
.unwrap();
resolve_dashboard_config(&mut status, "unused").unwrap();
assert!(status.resolved_config.is_none());
}
fn encrypt_json_gcm(plaintext: &[u8], master_key_b64: &str) -> String {
use aes_gcm::aead::{Aead, KeyInit};
use aes_gcm::{Aes256Gcm, Key, Nonce};
use base64::{engine::general_purpose, Engine};
let key_bytes = general_purpose::STANDARD.decode(master_key_b64).unwrap();
let key = Key::<Aes256Gcm>::try_from(key_bytes.as_slice()).unwrap();
let cipher = Aes256Gcm::new(&key);
let nonce_bytes = [0u8; 12];
let nonce = Nonce::try_from(&nonce_bytes[..]).unwrap();
let ct = cipher.encrypt(&nonce, plaintext).unwrap();
let mut data = nonce_bytes.to_vec();
data.extend_from_slice(&ct);
general_purpose::STANDARD.encode(data)
}
+68
View File
@@ -0,0 +1,68 @@
use crate::services::backup::BackupService;
use crate::services::backup::logger::JobLogger;
use crate::services::config::{DatabaseConfig, DbType};
use crate::tests::init_tracing_for_test;
use std::collections::HashMap;
use std::sync::Arc;
use tempfile::TempDir;
fn sqlite_config(path: &str) -> DatabaseConfig {
DatabaseConfig {
name: "retry-test".to_string(),
database: String::new(),
db_type: DbType::Sqlite,
username: String::new(),
password: String::new(),
port: 0,
host: String::new(),
generated_id: "retry-test-gen".to_string(),
path: path.to_string(),
max_packet_size: String::new(),
volume_name: String::new(),
container_name: None,
options: HashMap::new(),
}
}
#[tokio::test]
async fn a_failing_backup_is_retried_and_leaves_no_attempt_directory() {
init_tracing_for_test();
let temp_dir = TempDir::new().unwrap();
let tmp_path = temp_dir.path();
let logger = Arc::new(JobLogger::new());
let cfg = sqlite_config("/nonexistent/definitely-not-here.sqlite");
let result = BackupService::run(cfg, tmp_path, Arc::clone(&logger))
.await
.unwrap();
assert_eq!(result.status, "failed");
assert!(result.backup_file.is_none());
let entries = Arc::try_unwrap(logger).unwrap().into_entries();
assert_eq!(
entries.iter().filter(|e| e.level == "warn").count(),
2,
"expected one warn per non-final failed attempt"
);
assert!(
entries
.iter()
.any(|e| e.level == "error" && e.message.starts_with("Backup failed:")),
"expected a single terminal error from the runner"
);
let leftovers: Vec<_> = std::fs::read_dir(tmp_path)
.unwrap()
.filter_map(|e| e.ok())
.filter(|e| e.file_name().to_string_lossy().starts_with("attempt-"))
.collect();
assert!(
leftovers.is_empty(),
"failed attempt directories must be cleaned up, found {:?}",
leftovers.iter().map(|e| e.file_name()).collect::<Vec<_>>()
);
}
+205
View File
@@ -0,0 +1,205 @@
//! Regression test: a per-storage upload failure must be reported to the server via
//! `backup_upload_status("failed", ...)`. Previously the uploader early-returned on failure
//! and skipped the status call, so `backup_upload_init` opened a record that was never closed.
use crate::core::context::Context;
use crate::services::api::ApiClient;
use crate::services::api::models::agent::status::DatabaseStorage;
use crate::services::backup::BackupService;
use crate::services::backup::logger::JobLogger;
use crate::services::backup::models::BackupResult;
use crate::services::config::DbType;
use crate::tests::init_tracing_for_test;
use crate::utils::common::BackupMethod;
use crate::utils::edge_key::EdgeKey;
use serde_json::json;
use std::io::Write;
use std::sync::Arc;
use tempfile::NamedTempFile;
use wiremock::matchers::{body_partial_json, method, path};
use wiremock::{Mock, MockServer, ResponseTemplate};
fn ctx_pointing_at(base_url: String) -> Context {
Context {
edge_key: EdgeKey {
server_url: String::new(),
agent_id: "agent-1".to_string(),
master_key_b64: String::new(),
},
api: ApiClient::new(base_url),
}
}
#[tokio::test]
async fn failed_upload_reports_failed_status_to_server() {
init_tracing_for_test();
let server = MockServer::start().await;
// init opens the per-storage record and returns its id.
Mock::given(method("POST"))
.and(path("/agent/agent-1/backup/upload/init"))
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
"message": "ok",
"backupStorage": { "id": "bs-1" }
})))
.expect(1)
.mount(&server)
.await;
// The fix: on failure the uploader must PATCH the status as "failed".
Mock::given(method("PATCH"))
.and(path("/agent/agent-1/backup/upload/status"))
.and(body_partial_json(json!({ "status": "failed" })))
.respond_with(ResponseTemplate::new(200).set_body_json(json!({})))
.expect(1)
.mount(&server)
.await;
let service = BackupService::new(Arc::new(ctx_pointing_at(server.uri())));
// backup_file = None makes the provider fail immediately ("Missing backup file path"),
// exercising the failure path without any network/Azure dependency.
let result = BackupResult {
generated_id: "gen-1".to_string(),
db_type: DbType::Postgresql,
status: "success".to_string(),
backup_file: None,
code: None,
};
let storage: DatabaseStorage = serde_json::from_value(json!({
"id": "storage-1",
"provider": "blob",
"config": {}
}))
.unwrap();
let backup_id = "backup-1".to_string();
let logger = Arc::new(JobLogger::new());
let results = service
.upload(
result,
BackupMethod::Manual,
vec![storage],
false,
&backup_id,
logger,
)
.await
.unwrap();
assert_eq!(results.len(), 1);
assert!(!results[0].success);
// MockServer drop verifies both `.expect(1)` mounts were hit — including the "failed" PATCH.
}
#[tokio::test]
async fn a_failing_upload_is_retried_until_it_succeeds() {
init_tracing_for_test();
let server = MockServer::start().await;
Mock::given(method("POST"))
.and(path("/agent/agent-1/backup/upload/init"))
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
"message": "ok",
"backupStorage": { "id": "bs-1" }
})))
.expect(1)
.mount(&server)
.await;
Mock::given(method("POST"))
.and(path("/tus/files"))
.respond_with(ResponseTemplate::new(500))
.up_to_n_times(2)
.with_priority(1)
.expect(2)
.mount(&server)
.await;
Mock::given(method("POST"))
.and(path("/tus/files"))
.respond_with(
ResponseTemplate::new(201)
.insert_header("Location", format!("{}/tus/files/upload-1", server.uri()).as_str()),
)
.with_priority(2)
.expect(1)
.mount(&server)
.await;
Mock::given(method("PATCH"))
.and(path("/tus/files/upload-1"))
.respond_with(ResponseTemplate::new(204))
.mount(&server)
.await;
Mock::given(method("PATCH"))
.and(path("/agent/agent-1/backup/upload/status"))
.and(body_partial_json(json!({ "status": "success" })))
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
"message": "ok",
"backupStorage": { "id": "bs-1" }
})))
.expect(1)
.mount(&server)
.await;
let mut backup_file = NamedTempFile::new().unwrap();
backup_file.write_all(b"portabase-retry-test-payload").unwrap();
backup_file.flush().unwrap();
let ctx = Context {
edge_key: EdgeKey {
server_url: server.uri(),
agent_id: "agent-1".to_string(),
master_key_b64: String::new(),
},
api: ApiClient::new(server.uri()),
};
let service = BackupService::new(Arc::new(ctx));
let result = BackupResult {
generated_id: "gen-1".to_string(),
db_type: DbType::Postgresql,
status: "success".to_string(),
backup_file: Some(backup_file.path().to_path_buf()),
code: None,
};
let storage: DatabaseStorage = serde_json::from_value(json!({
"id": "storage-1",
"provider": "local",
"config": {}
}))
.unwrap();
let backup_id = "backup-1".to_string();
let logger = Arc::new(JobLogger::new());
let results = service
.upload(
result,
BackupMethod::Manual,
vec![storage],
false,
&backup_id,
Arc::clone(&logger),
)
.await
.unwrap();
assert_eq!(results.len(), 1);
assert!(results[0].success);
let entries = Arc::try_unwrap(logger).unwrap().into_entries();
assert_eq!(entries.iter().filter(|e| e.level == "warn").count(), 2);
assert!(
entries.iter().any(|e| e.message
== "Upload to storage storage-1 succeeded on attempt 3/3")
);
}
+333
View File
@@ -0,0 +1,333 @@
use crate::core::context::Context;
use crate::services::api::ApiClient;
use crate::services::config::ConfigService;
use crate::services::config::{build_config, DatabasesConfig, InputDatabaseConfig};
use crate::utils::edge_key::EdgeKey;
use std::io::Write;
use std::sync::Arc;
use tempfile::NamedTempFile;
fn test_context() -> Arc<Context> {
Arc::new(Context {
edge_key: EdgeKey {
server_url: String::new(),
agent_id: "agent-1".to_string(),
master_key_b64: String::new(),
},
api: ApiClient::new(String::new()),
})
}
fn write_json(contents: &str) -> NamedTempFile {
let mut file = NamedTempFile::with_suffix(".json").unwrap();
file.write_all(contents.as_bytes()).unwrap();
file
}
#[test]
fn parses_postgresql_cluster_type() {
let file = write_json(
r#"{
"databases": [
{
"name": "cluster1",
"type": "postgresql-cluster",
"username": "postgres",
"password": "p",
"port": 5432,
"host": "localhost",
"generated_id": "16678159-ff7e-4c97-8c83-0adeff214681"
}
]
}"#,
);
let service = ConfigService::new(test_context());
let cfg = service.load(Some(file.path().to_str().unwrap())).unwrap();
assert_eq!(cfg.databases[0].db_type.as_str(), "postgresql-cluster");
// `database` is optional for cluster entries and defaults to "postgres".
assert_eq!(cfg.databases[0].database, "postgres");
}
#[test]
fn postgresql_cluster_respects_explicit_database() {
let file = write_json(
r#"{
"databases": [
{
"name": "cluster1",
"type": "postgresql-cluster",
"database": "maintenance",
"username": "postgres",
"password": "p",
"port": 5432,
"host": "localhost",
"generated_id": "16678159-ff7e-4c97-8c83-0adeff214681"
}
]
}"#,
);
let service = ConfigService::new(test_context());
let cfg = service.load(Some(file.path().to_str().unwrap())).unwrap();
assert_eq!(cfg.databases[0].database, "maintenance");
}
#[test]
fn postgresql_options_keep_ownership_parses() {
let file = write_json(
r#"{
"databases": [
{
"name": "db1",
"type": "postgresql",
"username": "u",
"password": "p",
"port": 5432,
"host": "localhost",
"database": "mydb",
"generated_id": "16678159-ff7e-4c97-8c83-0adeff214681",
"options": {
"keep_ownership": true
}
}
]
}"#,
);
let service = ConfigService::new(test_context());
let cfg = service.load(Some(file.path().to_str().unwrap())).unwrap();
let keep = cfg.databases[0]
.options
.get("keep_ownership")
.and_then(|v| v.as_bool())
.unwrap_or(false);
assert!(keep);
}
#[test]
fn postgresql_options_absent_defaults_to_empty() {
let file = write_json(
r#"{
"databases": [
{
"name": "db1",
"type": "postgresql",
"username": "u",
"password": "p",
"port": 5432,
"host": "localhost",
"database": "mydb",
"generated_id": "16678159-ff7e-4c97-8c83-0adeff214681"
}
]
}"#,
);
let service = ConfigService::new(test_context());
let cfg = service.load(Some(file.path().to_str().unwrap())).unwrap();
assert!(cfg.databases[0].options.is_empty());
}
#[test]
fn postgresql_options_non_bool_keep_ownership_falls_back_to_false() {
let file = write_json(
r#"{
"databases": [
{
"name": "db1",
"type": "postgresql",
"username": "u",
"password": "p",
"port": 5432,
"host": "localhost",
"database": "mydb",
"generated_id": "16678159-ff7e-4c97-8c83-0adeff214681",
"options": {
"keep_ownership": "yes"
}
}
]
}"#,
);
let service = ConfigService::new(test_context());
let cfg = service.load(Some(file.path().to_str().unwrap())).unwrap();
let keep = cfg.databases[0]
.options
.get("keep_ownership")
.and_then(|v| v.as_bool())
.unwrap_or(false);
assert!(!keep);
}
#[test]
fn keep_ownership_extraction_logic() {
use serde_json::Value;
use std::collections::HashMap;
// true → keep ownership
let mut opts: HashMap<String, Value> = HashMap::new();
opts.insert("keep_ownership".to_string(), Value::Bool(true));
let keep = opts.get("keep_ownership").and_then(|v| v.as_bool()).unwrap_or(false);
assert!(keep, "should keep ownership when flag is true");
// false → strip
let mut opts2: HashMap<String, Value> = HashMap::new();
opts2.insert("keep_ownership".to_string(), Value::Bool(false));
let keep2 = opts2.get("keep_ownership").and_then(|v| v.as_bool()).unwrap_or(false);
assert!(!keep2, "should strip when flag is false");
// missing → strip
let opts3: HashMap<String, Value> = HashMap::new();
let keep3 = opts3.get("keep_ownership").and_then(|v| v.as_bool()).unwrap_or(false);
assert!(!keep3, "should strip when key absent");
// wrong type → strip
let mut opts4: HashMap<String, Value> = HashMap::new();
opts4.insert("keep_ownership".to_string(), Value::String("yes".to_string()));
let keep4 = opts4.get("keep_ownership").and_then(|v| v.as_bool()).unwrap_or(false);
assert!(!keep4, "should strip when value is not bool");
}
#[test]
fn parses_docker_volume_type() {
let file = write_json(
r#"{
"databases": [
{
"name": "uploads",
"type": "docker-volume",
"volume_name": "myapp_uploads",
"generated_id": "16678159-ff7e-4c97-8c83-0adeff214681",
"container_name": "myapp"
}
]
}"#,
);
let service = ConfigService::new(test_context());
let cfg = service.load(Some(file.path().to_str().unwrap())).unwrap();
assert_eq!(cfg.databases[0].db_type.as_str(), "docker-volume");
assert_eq!(cfg.databases[0].volume_name, "myapp_uploads");
assert_eq!(cfg.databases[0].container_name.as_deref(), Some("myapp"));
}
#[test]
fn docker_volume_container_name_optional() {
let file = write_json(
r#"{
"databases": [
{
"name": "uploads",
"type": "docker-volume",
"volume_name": "myapp_uploads",
"generated_id": "16678159-ff7e-4c97-8c83-0adeff214681"
}
]
}"#,
);
let service = ConfigService::new(test_context());
let cfg = service.load(Some(file.path().to_str().unwrap())).unwrap();
assert_eq!(cfg.databases[0].volume_name, "myapp_uploads");
assert!(cfg.databases[0].container_name.is_none());
}
#[test]
fn docker_volume_requires_volume_name() {
let file = write_json(
r#"{
"databases": [
{
"name": "uploads",
"type": "docker-volume",
"generated_id": "16678159-ff7e-4c97-8c83-0adeff214681"
}
]
}"#,
);
let service = ConfigService::new(test_context());
let err = service.load(Some(file.path().to_str().unwrap())).unwrap_err();
assert!(err.contains("volume_name"), "error was: {err}");
}
#[test]
fn build_config_applies_type_defaults() {
let input: InputDatabaseConfig = serde_json::from_str(
r#"{
"name": "cluster1",
"type": "postgresql-cluster",
"username": "postgres",
"password": "p",
"port": 5432,
"host": "localhost",
"generated_id": "16678159-ff7e-4c97-8c83-0adeff214681"
}"#,
)
.unwrap();
let cfg = build_config(input).unwrap();
assert_eq!(cfg.db_type.as_str(), "postgresql-cluster");
assert_eq!(cfg.database, "postgres"); // cluster default
}
#[test]
fn build_config_rejects_missing_required_field() {
let input: InputDatabaseConfig = serde_json::from_str(
r#"{
"name": "pg",
"type": "postgresql",
"username": "postgres",
"port": 5432,
"host": "localhost",
"generated_id": "16678159-ff7e-4c97-8c83-0adeff214681"
}"#,
)
.unwrap();
let err = build_config(input).unwrap_err();
assert!(err.contains("password"), "unexpected error: {err}");
}
#[test]
fn load_optional_returns_empty_when_file_missing() {
let service = ConfigService::new(test_context());
let cfg = service.load_optional(Some("/nonexistent/path/does-not-exist.json"));
assert!(cfg.databases.is_empty());
}
#[test]
fn databases_config_roundtrips_through_serde() {
let input: InputDatabaseConfig = serde_json::from_str(
r#"{
"name": "pg",
"type": "postgresql",
"database": "app",
"username": "postgres",
"password": "secret",
"port": 5432,
"host": "localhost",
"generated_id": "16678159-ff7e-4c97-8c83-0adeff214681"
}"#,
)
.unwrap();
let cfg = build_config(input).unwrap();
let wrapped = DatabasesConfig { databases: vec![cfg] };
let json = serde_json::to_string(&wrapped).unwrap();
let back: DatabasesConfig = serde_json::from_str(&json).unwrap();
assert_eq!(back.databases[0].name, "pg");
assert_eq!(back.databases[0].db_type.as_str(), "postgresql");
assert_eq!(back.databases[0].password, "secret");
}
@@ -0,0 +1,84 @@
use crate::services::config::{build_config, DatabaseConfig, InputDatabaseConfig};
use crate::services::dashboard_config::merge;
use crate::services::dashboard_config::{load_cache, persist_cache};
fn cfg(name: &str, gen_id: &str, host: &str) -> DatabaseConfig {
let json = format!(
r#"{{ "name": "{name}", "type": "postgresql", "database": "app",
"username": "u", "password": "p", "port": 5432,
"host": "{host}", "generated_id": "{gen_id}" }}"#
);
let input: InputDatabaseConfig = serde_json::from_str(&json).unwrap();
build_config(input).unwrap()
}
const ID_A: &str = "16678159-ff7e-4c97-8c83-0adeff214681";
const ID_B: &str = "16678124-ff7e-4c97-8c83-0adeff214681";
#[test]
fn merge_keeps_local_only_databases() {
let local = vec![cfg("local-a", ID_A, "local-host")];
let merged = merge(&local, &[]);
assert_eq!(merged.databases.len(), 1);
assert_eq!(merged.databases[0].host, "local-host");
}
#[test]
fn merge_appends_dashboard_only_databases() {
let local = vec![cfg("local-a", ID_A, "local-host")];
let dashboard = vec![cfg("dash-b", ID_B, "dash-host")];
let merged = merge(&local, &dashboard);
assert_eq!(merged.databases.len(), 2);
assert!(merged.databases.iter().any(|d| d.generated_id == ID_B));
}
#[test]
fn merge_dashboard_wins_on_id_collision() {
let local = vec![cfg("local-a", ID_A, "local-host")];
let dashboard = vec![cfg("dash-a", ID_A, "dash-host")];
let merged = merge(&local, &dashboard);
assert_eq!(merged.databases.len(), 1);
assert_eq!(merged.databases[0].host, "dash-host"); // dashboard wins
assert_eq!(merged.databases[0].name, "dash-a");
}
#[test]
fn cache_roundtrips() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("dashboard_databases.json");
let dbs = vec![cfg("dash-a", ID_A, "dash-host")];
persist_cache(&path, &dbs).unwrap();
let loaded = load_cache(&path);
assert_eq!(loaded.len(), 1);
assert_eq!(loaded[0].generated_id, ID_A);
assert_eq!(loaded[0].host, "dash-host");
}
#[test]
fn load_cache_missing_file_is_empty() {
let loaded = load_cache(std::path::Path::new("/nonexistent/dashboard_databases.json"));
assert!(loaded.is_empty());
}
#[test]
fn load_cache_corrupt_file_is_empty() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("dashboard_databases.json");
std::fs::write(&path, b"{ this is not valid json").unwrap();
let loaded = load_cache(&path);
assert!(loaded.is_empty());
}
#[test]
fn persist_cache_leaves_no_tmp_file() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("dashboard_databases.json");
persist_cache(&path, &[cfg("dash-a", ID_A, "h")]).unwrap();
let tmp = path.with_extension("json.tmp");
assert!(!tmp.exists(), "temp file should have been renamed away");
assert!(path.exists());
}

Some files were not shown because too many files have changed in this diff Show More