Compare commits

...

153 Commits

Author SHA1 Message Date
github-actions[bot] a2bd4d1572 chore: release 1.21.2 2026-09-22 12:10:46 +00:00
Charles Gauthereau 0e04cffc76 Merge pull request #109 from Portabase/fix/binaries
fix: binaries assets
2026-09-22 14:08:32 +02:00
charles-gauthereau 71e2473167 fix: scripts 2026-09-22 13:45:49 +02:00
charles-gauthereau 8a29f6b25d fix: Dockerfile 2026-09-22 13:43:53 +02:00
charles-gauthereau e05b28176c fix: binaries assets 2026-09-22 13:24:54 +02:00
charles-gauthereau 3c060fe150 fix: binaries assets 2026-09-22 12:37:13 +02:00
github-actions[bot] d42c31f3dc chore: release 1.21.1 2026-09-17 11:43:39 +00:00
Charles GTE 84673373a7 Merge pull request #107 from Portabase/fix/mongodb-options
fix: mongodb-options
2026-09-17 13:41:27 +02:00
Charles GTE 6f8a5c4228 fix: mongodb options 2026-09-17 11:20:16 +02:00
charles-gauthereau c5806a6ea4 fix: refactoring 2026-09-17 10:47:32 +02:00
Charles GTE c241020542 fix: mongodb options 2026-09-17 10:13:58 +02:00
github-actions[bot] e0bc6556aa chore: release 1.21.0 2026-09-14 13:11:26 +00:00
Charles GTE 947cb9bed7 Merge pull request #104 from Portabase/dev
Dev
2026-09-14 15:09:04 +02:00
Charles GTE b5d0035a3b Merge pull request #106 from Portabase/poc/hmac
fix: openssl windows
2026-09-14 14:52:26 +02:00
Charles GTE d52c328a99 fix: refactoring 2026-09-14 14:51:58 +02:00
Charles GTE 9b79ac4850 Merge branch 'refs/heads/dev' into poc/hmac
# Conflicts:
#	Cargo.lock
2026-09-14 14:51:22 +02:00
Charles GTE 6e50718b65 fix: tests 2026-09-14 14:30:11 +02:00
Charles GTE ee5ed81d48 fix: tests 2026-09-14 14:12:29 +02:00
Charles GTE df078b6202 Merge pull request #105 from Portabase/feat/sftp-storage
feat: sftp-storage
2026-09-14 10:55:40 +02:00
Charles GTE 24e3e3098d chore: update packages 2026-09-13 17:25:13 +02:00
Charles GTE f6e1a0df98 fix: refactoring 2026-09-12 21:15:04 +02:00
Charles GTE 49366ee24e refactor(storage): generic rclone config builder and shared obscure util 2026-09-12 21:10:50 +02:00
Charles GTE 012a984973 fix(storage): reject line breaks in sftp host/username; drop unused import 2026-09-12 21:02:28 +02:00
Charles GTE 6d1d7c6891 fix: refactoring 2026-09-12 18:15:45 +02:00
Charles GTE c70cde8a9c feat(storage): sftp provider over rclone rcat 2026-09-12 17:57:04 +02:00
Charles GTE c353443bdd feat(storage): sftp config model and rclone-config builder 2026-09-12 17:57:04 +02:00
Charles GTE 9f9dedfe87 feat: rclone-privider (#103)
* build: install rclone 1.75.1 in agent images

* build: exclude target/ and local artifacts from the docker context

* feat(storage): add rclone config model and validation helpers

* feat(storage): stream backups into rclone rcat

* fix(storage): abort truncated rclone uploads and strengthen stderr test

* feat(storage): add RcloneProvider upload path

* fix(storage): bound rclone timeouts, detect restore truncation, track sdd workspace

* fix(storage): block virtual and non-viable rclone backends

* fix: refactoring
2026-09-12 17:09:57 +02:00
Charles GTE 9bb830327e fix: refactoring 2026-09-10 22:55:47 +02:00
Charles GTE 31e55a7b4f fix(storage): block virtual and non-viable rclone backends 2026-09-10 22:38:08 +02:00
Charles GTE 576c055092 fix(storage): bound rclone timeouts, detect restore truncation, track sdd workspace 2026-09-10 22:38:08 +02:00
Charles GTE ba83b6b2b7 feat(storage): add RcloneProvider upload path 2026-09-10 22:38:07 +02:00
Charles GTE ea356ed54f fix(storage): abort truncated rclone uploads and strengthen stderr test 2026-09-10 22:37:37 +02:00
Charles GTE b222b13934 feat(storage): stream backups into rclone rcat 2026-09-10 22:37:09 +02:00
Charles GTE 9409f0ff25 feat(storage): add rclone config model and validation helpers 2026-09-10 22:37:00 +02:00
Charles GTE 59312215e9 build: exclude target/ and local artifacts from the docker context 2026-09-10 22:36:46 +02:00
Charles GTE 3ff360a971 build: install rclone 1.75.1 in agent images 2026-09-10 22:36:38 +02:00
Théo LAGACHE cc1703592e poc 2026-09-09 11:57:56 +02:00
github-actions[bot] 5f92297108 chore: release 1.20.1 2026-09-03 15:41:20 +00:00
Charles GTE d1821f7045 Merge pull request #102 from Portabase/fix/cron-crash
fix: cron crash for databases setup from dashboard
2026-09-03 17:38:53 +02:00
charles-gauthereau 0fe4d50cbd fix: docker-compose.yml 2026-09-03 17:38:35 +02:00
charles-gauthereau 1f29466a28 fix: cron crash for databases setup from dashboard 2026-09-03 17:22:03 +02:00
github-actions[bot] b74aaa0bbc chore: release 1.20.0 2026-08-28 15:42:27 +00:00
Charles GTE 7b5e5b2c78 Merge pull request #101 from Portabase/feat/retry-system
feat: retry-system
2026-08-28 17:40:20 +02:00
charles-gauthereau 99f1ef2081 fix: refactoring 2026-08-28 17:25:30 +02:00
charles-gauthereau 42c3c5945a fix: refactoring 2026-08-28 17:06:23 +02:00
charles-gauthereau db0f87c2e9 Merge branch 'main' into feat/retry-system 2026-08-28 16:54:51 +02:00
github-actions[bot] f95f0aa73a chore: release 1.19.2 2026-08-28 07:59:57 +00:00
Charles GTE 2177dfd44b Merge pull request #100 from Portabase/fix/kill-on-drop
fix: kill_on_drop on firebird, mariadb, mysql, redis, valkey
2026-08-28 09:57:51 +02:00
charles-gauthereau 0a53eec184 fix: kill_on_drop on firebird, mariadb, mysql, redis, valkey 2026-08-28 09:43:08 +02:00
charles-gauthereau 87f33af772 fix: wire retry env vars into helm configmap, dedupe terminal retry log
helm/templates/env-configmap.yaml never listed RETRY_ATTEMPTS and
RETRY_BACKOFF_MS even though values.yaml gained them, so --set
env.RETRY_ATTEMPTS=N was silently ignored by Kubernetes deployments.
Add both keys in the same explicit style as the existing entries.

src/utils/retry.rs logged its own "failed after N attempts" error on
exhaustion, on top of the terminal log each call site already writes,
producing two error entries per failure. Worse, it changed a log
level: FileLock::acquire's "backup_already_in_progress" bails through
the combinator, which now logged it as error before runner.rs got a
chance to reclassify it as the routine warn it always was. A manual
backup colliding with a scheduled one would show up as a hard error
on the dashboard instead of the harmless warn it used to be, breaking
the "fails exactly as it does today" guarantee for job records.

Drop the combinator's terminal error log and give download_backup its
own terminal error log so all three call sites (runner, uploader,
downloader) own their failure logging uniformly. Update the two tests
that asserted the removed message to assert the new behavior instead.
2026-08-27 22:46:49 +02:00
charles-gauthereau b6a120fcbf feat: retry the restore backup download
Extracts the download body to download_once and makes download_backup a
retry wrapper around it. This path had no retry at all before, so a
single dropped connection failed the whole restore job.

Retrying is safe because File::create truncates and the target filename
is derived from Content-Disposition or the URL, so it is stable across
attempts. There is no Range resume: a download that fails at 90% starts
over.
2026-08-27 19:01:10 +02:00
charles-gauthereau 5298d82576 feat: retry storage uploads
Wraps provider.upload in the retry combinator. No provider changes are
needed: each one builds its upload stream from the file on disk inside
upload(), so every attempt gets a fresh handle and a fresh nonce.

Result<UploadResult, UploadResult> is collapsed with an or-pattern so
the last attempt's error and metadata survive into the existing failure
branch. A missing backup file short-circuits into Err rather than
returning early, which skips the retry without skipping the
backup_upload_status(failed) call that closes the server-side record.

backup_upload_init and backup_upload_status are left unwrapped; they
are control-plane calls, not storage uploads.
2026-08-27 18:53:06 +02:00
charles-gauthereau b0da2e40a3 feat: retry the database backup
Each attempt now dumps into its own tmp_path/attempt-{n} directory,
which is removed when the attempt fails. Without the per-attempt
directory pg_dump -Fd would refuse every retry, because it will not
write into a directory a previous attempt left behind; removing it on
failure keeps peak disk at one attempt's artifacts rather than five.

A backup blocked by a concurrent job is retried before surfacing the
same backup_already_in_progress code, since FileLock reports it as an
ordinary error and the combinator cannot tell it apart.

Reshapes retry()'s bound from the native AsyncFnMut sugar to the
classic F: FnMut(u32) -> Fut, Fut: Future<Output = Result<T, E>> + Send
shape (the pattern tokio-retry and backoff both use). AsyncFnMut's
produced future is a lifetime-quantified associated type
(F::CallRefFuture<'_>) that cannot be named or bounded as Send on
stable Rust, so wiring a retried call through it into a future that
eventually gets polled inside tokio::spawn (dispatcher.rs, via
execute_backup) made rustc's opaque-type Send inference fail with
"implementation of Send is not general enough" at the spawn site,
several call layers away from the actual retry call. Naming Fut as its
own type parameter lets Send be asserted on it directly instead, which
resolves cleanly. The combinator's control flow, log messages, and
formats are unchanged; only the bound and the call sites' closure
shape (async |x| { } becomes |x| async { }, with shared references
bound outside a move closure so the inner async move block only moves
Copy references, not the originals) are affected.
2026-08-27 18:39:41 +02:00
charles-gauthereau 55e20d48e7 feat: configurable retry policy and combinator
Adds RETRY_ATTEMPTS (3..=5, default 3) and RETRY_BACKOFF_MS
(100..=30000, default 1000) to Settings, validated with the same
panic-on-invalid contract as POOLING and CHUNK_SIZE_MB.

The combinator logs every failed attempt and any late success through
the JobLogger it borrows, so retries reach the server on the existing
job-log path with no API change. It borrows rather than clones the Arc
so Arc::try_unwrap in the backup executor keeps working. Backoff is
exponential with equal jitter, because the uploader retries storages
concurrently and would otherwise retry them in lockstep.
2026-08-27 18:13:17 +02:00
github-actions[bot] f7f5f7e141 chore: release 1.19.1 2026-08-21 17:09:15 +00:00
Charles GTE 2170f96a72 Merge pull request #99 from Portabase/fix/config-file
fix: config directory creation
2026-08-21 19:06:48 +02:00
Charles GTE 298d46ba81 fix: config directory creation 2026-08-21 13:50:40 +02:00
github-actions[bot] 6537e9df53 chore: release 1.19.0 2026-08-20 11:56:59 +00:00
Charles GTE b8d869d5a6 Merge pull request #97 from Portabase/feat/dashboard-databases
feat: dashboard-databases
2026-08-20 13:54:46 +02:00
Charles GTE 90941ea67d fix 2026-08-20 13:41:14 +02:00
charles-gauthereau 046d593e2b fix 2026-08-18 18:20:21 +02:00
charles-gauthereau cf9a59a138 fix: config.rs 2026-08-18 13:07:22 +02:00
charles-gauthereau 2464f6dfb0 Merge branch 'main' into feat/dashboard-databases
# Conflicts:
#	docker-compose.yml
#	src/services/config.rs
2026-08-18 13:05:56 +02:00
github-actions[bot] 069067ca55 chore: release 1.18.6 2026-08-17 09:10:08 +00:00
Charles GTE 04b654d219 Merge pull request #96 from Portabase/fix/mongo-cloud
fix: mongodb cloud cluster issue
2026-08-17 11:07:35 +02:00
charles-gauthereau d83511cb64 fix: mongodb cloud cluster issue 2026-08-17 10:52:21 +02:00
charles-gauthereau ca294e968c fix 2026-08-13 22:02:17 +02:00
charles-gauthereau 1be88ffdb9 Merge branch 'main' into feat/dashboard-databases
# Conflicts:
#	docker-compose.yml
2026-08-13 21:56:38 +02:00
charles-gauthereau 9d393a96a4 fix 2026-08-13 21:55:59 +02:00
charles-gauthereau 044bf80633 feat(agent): ingest dashboard databases via merged cycle + cache 2026-08-13 18:47:18 +02:00
charles-gauthereau 0a6eb6db22 feat(dashboard_config): atomic cache load/persist 2026-08-13 18:40:56 +02:00
charles-gauthereau b26ff81889 feat(dashboard_config): merge (dashboard-wins) + collect_configs 2026-08-13 18:39:54 +02:00
charles-gauthereau 4e2f29f4ca feat(status): decrypt dashboard config_ciphertext into resolved_config 2026-08-13 18:38:15 +02:00
charles-gauthereau d1c8df4cac refactor(config): extract build_config, add load_optional + Serialize 2026-08-13 18:35:52 +02:00
github-actions[bot] 30f83bafcf chore: release 1.18.5 2026-07-26 09:06:42 +00:00
Charles GTE de106c835e Merge pull request #93 from Portabase/fix/s3-storage
fix: s3 logs
2026-07-26 11:04:35 +02:00
Charles GTE 23d6822ddc fix: s3 logs 2026-07-26 11:04:07 +02:00
github-actions[bot] fe1d74945f chore: release 1.18.4 2026-07-25 14:03:12 +00:00
Charles GTE 424a646385 Merge pull request #92 from Portabase/fix/windows-build
fix: windows-build
2026-07-25 16:00:57 +02:00
Charles GTE 0ef4bba5d7 fix: docker-compose.yml 2026-07-25 15:45:56 +02:00
Charles GTE 6548140eaf fix: windows build 2026-07-25 15:19:46 +02:00
github-actions[bot] c9725c381e chore: release 1.18.3 2026-07-24 16:46:51 +00:00
Charles GTE 03695b1897 Merge pull request #91 from Portabase/fix/firebird
fix: firebird
2026-07-24 18:44:20 +02:00
charles-gauthereau 80b1c0dac0 fix: firebird with -g for gbak 2026-07-24 18:29:04 +02:00
charles-gauthereau 5ededb3764 Merge branch 'main' into dev 2026-07-24 17:45:46 +02:00
github-actions[bot] 54cfd673e4 chore: release 1.18.2 2026-07-23 18:04:50 +00:00
Charles GTE 39dbeb5d30 Merge pull request #90 from Portabase/fix/ghcr-ci-cd
fix: ghcr.yml
2026-07-23 20:02:28 +02:00
charles-gauthereau 380cea79fb fix: ghcr.yml 2026-07-23 20:02:01 +02:00
github-actions[bot] 65e4538c2c chore: release 1.18.1 2026-07-23 17:47:38 +00:00
Charles GTE 86a706bca5 Merge pull request #89 from Portabase/fix/ghcr-ci-cd
fix: ghcr.yml
2026-07-23 19:45:23 +02:00
charles-gauthereau e8bda60f16 fix: ghcr.yml 2026-07-23 18:47:41 +02:00
charles-gauthereau 034912b176 Merge branch 'main' into dev 2026-07-23 18:24:18 +02:00
github-actions[bot] 4fd50ccda4 chore: release 1.18.0 2026-07-23 15:33:29 +00:00
Charles GTE 84c6b765c8 Merge pull request #88 from Portabase/fix/postgres-provider
fix: postgres-provider
2026-07-23 17:31:02 +02:00
charles-gauthereau 9790837200 fix: postgres 2026-07-23 17:14:27 +02:00
charles-gauthereau 8827234344 refactor(postgres): split restore into restore/ module (run, prepare, command, toc) 2026-07-23 16:40:56 +02:00
charles-gauthereau b3417304bb fix(postgres): match real pg_restore -l schema line in toc_creates_public_schema 2026-07-23 10:06:01 +02:00
charles-gauthereau 5f4006b1ca feat(postgres): sniff dump format by magic bytes 2026-07-23 09:56:46 +02:00
charles-gauthereau f24b1ec3e4 feat(postgres): wire clean_mode=drop_database with preflight gate 2026-07-23 09:51:49 +02:00
charles-gauthereau bb1b16d0f2 feat(postgres): add drop_and_recreate_database + preflight + FORCE/retry 2026-07-23 09:43:24 +02:00
charles-gauthereau 2aca036ee4 feat(postgres): wire clean_mode none/clean/drop_schemas into restore 2026-07-23 09:36:25 +02:00
charles-gauthereau 2a72a55e57 feat(postgres): detect CREATE SCHEMA public in archive TOC 2026-07-23 09:31:50 +02:00
charles-gauthereau ae63a0df50 feat(postgres): add drop_all_schemas + recreate_public_schema 2026-07-23 09:27:36 +02:00
charles-gauthereau 560c5e02f2 feat(postgres): add RestoreCleanMode option parsing 2026-07-23 09:24:21 +02:00
charles-gauthereau ca17bf9cb9 feat(postgres): add quote_ident/quote_literal for DDL 2026-07-23 09:20:15 +02:00
charles-gauthereau 9fbaab6feb refactor(postgres): unify FC/FD restore via run_pg_restore, validate-before-destroy 2026-07-23 09:15:32 +02:00
charles-gauthereau 1289497d98 refactor(postgres): extract prepare_archive, validate before restore 2026-07-23 09:09:19 +02:00
charles-gauthereau 27c55dbfd8 refactor(postgres): drive async restore work via captured Tokio Handle 2026-07-23 09:01:50 +02:00
charles-gauthereau 45e3acd709 fix(postgres): add pg_restore_binary_name() for Windows restore 2026-07-23 08:58:49 +02:00
charles-gauthereau a93054518a Merge branch 'main' into dev 2026-07-22 00:06:17 +02:00
Charles GTE baf2edc829 fix (#87)
Co-authored-by: charles-gauthereau <charles.gauthereau@soluce-technologies.com>
2026-07-21 23:58:11 +02:00
github-actions[bot] f050fa35e0 chore: release 1.17.1 2026-07-21 21:29:30 +00:00
Charles GTE 90d55f7630 fix: backup folder name (#86)
* fix: support configurable backup file prefix (#83)

* fix: backup folder name

* fix: ghcr publish agent image

---------

Co-authored-by: Antonin Jousson <18756890+Antoninj@users.noreply.github.com>
Co-authored-by: charles-gauthereau <charles.gauthereau@soluce-technologies.com>
2026-07-21 23:27:10 +02:00
charles-gauthereau c186d57105 fix: ghcr publish agent image 2026-07-21 23:10:40 +02:00
charles-gauthereau 37b4c1fe6d fix: backup folder name 2026-07-21 23:05:59 +02:00
Antonin Jousson 92a3324ca2 fix: support configurable backup file prefix (#83) 2026-07-21 21:50:11 +02:00
github-actions[bot] 82ffb71496 chore: release 1.17.0 2026-07-11 10:08:21 +00:00
Charles GTE 20b3b58024 Merge pull request #80 from Portabase/fix/storage-encryption
fix/storage-encryption
2026-07-11 12:05:59 +02:00
charles-gauthereau 6cddc73565 Merge branch 'main' into fix/storage-encryption
# Conflicts:
#	docker-compose.yml
2026-07-10 18:49:01 +02:00
github-actions[bot] fd3b0076c0 chore: release 1.16.3 2026-07-09 09:11:23 +00:00
Charles GTE 334ab51125 fix: cron (#79)
* fix: cron log

---------

Co-authored-by: charles-gauthereau <charles.gauthereau@soluce-technologies.com>
2026-07-09 11:03:55 +02:00
github-actions[bot] 4df3fe2d0c chore: release 1.16.2 2026-07-09 07:38:49 +00:00
Charles GTE ec7715ca26 fix: cron (#78)
* fix: docker-compose.yml

* fix: cron

---------

Co-authored-by: charles-gauthereau <charles.gauthereau@soluce-technologies.com>
2026-07-09 09:36:53 +02:00
charles-gauthereau 32d4e22196 fix: status.rs 2026-07-08 22:24:50 +02:00
charles-gauthereau f277c5485f feat: decrypt encrypted storages in status ping 2026-07-08 20:52:17 +02:00
charles-gauthereau 585e0bea72 feat: accept encrypted storages markers in DatabaseStatus 2026-07-08 20:47:20 +02:00
charles-gauthereau ee97c6056a feat: add decrypt_json_gcm for encrypted status storages 2026-07-08 20:43:01 +02:00
charles-gauthereau b37a4f2180 fix: docker-compose.yml 2026-07-07 18:43:30 +02:00
github-actions[bot] 1d15f40662 chore: release 1.16.1 2026-07-04 10:26:50 +00:00
Charles GTE 324a2ea3f2 fix: security (#75) 2026-07-04 12:24:37 +02:00
github-actions[bot] 9ec92af6c7 chore: release 1.16.0 2026-07-03 06:16:42 +00:00
Charles GTE 1dafdca2a0 Merge pull request #74 from Portabase/feat/docker-volume-backup-restore
feat: docker-volume-backup-restore
2026-07-03 08:14:30 +02:00
charles-gauthereau d12817a960 fix: codecov.yml 2026-07-02 22:42:02 +02:00
charles-gauthereau 608b82a254 fix: test docker_volume.rs 2026-07-02 22:08:55 +02:00
charles-gauthereau bbecca92f9 fix: databases.json 2026-07-02 21:51:17 +02:00
charles-gauthereau b2b2733c4d feat: docker-volume provider
refactor: move build_tar to utils::compress

refactor: move choose_restore_path to utils::common

docs: remove docker-volume README section

fix: keep archive path for multi-file non-docker-volume restores

docs: trim docker-volume README section to essentials

refactor: run docker-volume backup/restore inside spawn_blocking like other providers

docs: enable docker socket, add volume example and security notes

feat: sweep orphaned ephemeral helper containers on startup

feat: docker-volume clean-replace restore via upload_to_container

fix: serialize env-var access in docker-volume tests to avoid setenv/getenv UB

feat: docker-volume backup via download_from_container

feat: docker-volume ping via inspect_volume

fix: return extraction dir for multi-file restore archives

feat: gzip already-tar inputs directly instead of double-wrapping

feat: docker helper (client, self-image, container lifecycle, sweep)

feat: docker-volume provider
2026-07-02 21:45:11 +02:00
github-actions[bot] 2c257c5a58 chore: release 1.15.0 2026-06-29 17:14:03 +00:00
Charles GTE c24c0d7058 Merge pull request #73 from Portabase/fix/postgres-ownership
fix: postgres-ownership
2026-06-29 19:12:01 +02:00
charles-gauthereau 34f353b68f fix: postgres keep_ownership flag 2026-06-29 19:10:49 +02:00
charles-gauthereau ee51996401 chore: document keep_ownership option in example databases.json 2026-06-29 18:54:01 +02:00
charles-gauthereau 694b463787 feat: conditionally omit --no-owner/--no-privileges in pg_restore based on keep_ownership option 2026-06-29 18:52:37 +02:00
charles-gauthereau aed1e86956 feat: add generic options map to DatabaseConfig
Add `options: HashMap<String, serde_json::Value>` to DatabaseConfig and
`options: Option<HashMap<...>>` to InputDatabaseConfig so callers can
pass arbitrary per-database options (e.g. keep_ownership for pg_dumpall).
Update all DatabaseConfig struct literals in test files and add three
new config-service unit tests covering present, absent, and non-bool values.
2026-06-29 18:49:01 +02:00
charles-gauthereau 6388aff1e3 fix: postgres cluster dump command 2026-06-29 17:10:11 +02:00
github-actions[bot] 39a77b18a4 chore: release 1.14.1 2026-06-29 13:22:10 +00:00
Charles GTE e62167182e Merge pull request #72 from Portabase/fix/postgres-cluster
fix: postgres cluster dump command
2026-06-29 15:19:54 +02:00
charles-gauthereau 4be54a614e fix: postgres cluster dump command 2026-06-29 15:18:52 +02:00
github-actions[bot] c5eaa5086c chore: release 1.14.0 2026-06-27 19:16:46 +00:00
Charles GTE 610d443afc Dev (#69)
* fix: pgdump-cluster (#68)

* feat: add as_str/from_str to PostgresDumpFormat

* feat: resolve pg_dumpall/psql binary names

* feat: add include_globals field to database config

* feat: add pg_dumpall/psql globals dump and apply

* feat: add postgres backup bundle (manifest + build + resolve)

* feat: bundle globals into postgres backup when include_globals is set

* feat: replay globals before pg_restore when backup archive is a bundle

* refactor: bind FD restore tempdir guard once to clear unused warnings

* docs: demonstrate include_globals in sample databases.json

* chore: silence test-only re-export warning in non-test builds

* revert: remove include_globals feature, restore plain pg_dump/pg_restore

* feat: add pg_dumpall/psql binary names and is_superuser check

* feat: add postgresql-cluster db type and config parsing

* feat: pg_dumpall cluster backup and psql restore

* feat: route postgresql-cluster through PostgresClusterDatabase

* docs: add postgresql-cluster sample to databases.json

* refactor: split cluster mode into cluster/ module (backup, restore, database)

* test: mirror cluster tests into src/tests/domain/cluster/

* feat: add-gcs-storage-provider (#70)

* fix: refactoring

* chore: add google-cloud-storage and google-cloud-auth deps

* feat: add GCS provider config model

* feat: add GCS credential, client, and stream-source helpers

StreamSource bridges build_stream's Send-only byte stream into the SDK's
StreamingSource (which send_buffered requires to be Send+Sync+'static) via a
bounded mpsc channel, avoiding any change to the shared UploadStream type.

* feat: implement GCS StorageProvider upload

* feat: register google-cloud-storage provider in factory

* test: GCS upload roundtrip against fake-gcs-server

* fix: format GCS bucket as projects/_/buckets/<name> in upload_with_client

write_object rejects the bare bucket id with "malformed bucket name"; the
production provider passed config.bucket_name unformatted, so real uploads
would always fail. Format once in upload_with_client so prod and the
fake-gcs-server test share the corrected path.

* style: cargo fmt GCS provider files

* fix: gcs

* fix: azure (#71)
2026-06-27 21:14:21 +02:00
Charles GTE 16a37e033e fix: azure (#71) 2026-06-27 20:51:47 +02:00
Charles GTE 29ca5a317b feat: add-gcs-storage-provider (#70)
* fix: refactoring

* chore: add google-cloud-storage and google-cloud-auth deps

* feat: add GCS provider config model

* feat: add GCS credential, client, and stream-source helpers

StreamSource bridges build_stream's Send-only byte stream into the SDK's
StreamingSource (which send_buffered requires to be Send+Sync+'static) via a
bounded mpsc channel, avoiding any change to the shared UploadStream type.

* feat: implement GCS StorageProvider upload

* feat: register google-cloud-storage provider in factory

* test: GCS upload roundtrip against fake-gcs-server

* fix: format GCS bucket as projects/_/buckets/<name> in upload_with_client

write_object rejects the bare bucket id with "malformed bucket name"; the
production provider passed config.bucket_name unformatted, so real uploads
would always fail. Format once in upload_with_client so prod and the
fake-gcs-server test share the corrected path.

* style: cargo fmt GCS provider files

* fix: gcs
2026-06-27 20:37:22 +02:00
Charles GTE 01f5d34c96 fix: pgdump-cluster (#68)
* feat: add as_str/from_str to PostgresDumpFormat

* feat: resolve pg_dumpall/psql binary names

* feat: add include_globals field to database config

* feat: add pg_dumpall/psql globals dump and apply

* feat: add postgres backup bundle (manifest + build + resolve)

* feat: bundle globals into postgres backup when include_globals is set

* feat: replay globals before pg_restore when backup archive is a bundle

* refactor: bind FD restore tempdir guard once to clear unused warnings

* docs: demonstrate include_globals in sample databases.json

* chore: silence test-only re-export warning in non-test builds

* revert: remove include_globals feature, restore plain pg_dump/pg_restore

* feat: add pg_dumpall/psql binary names and is_superuser check

* feat: add postgresql-cluster db type and config parsing

* feat: pg_dumpall cluster backup and psql restore

* feat: route postgresql-cluster through PostgresClusterDatabase

* docs: add postgresql-cluster sample to databases.json

* refactor: split cluster mode into cluster/ module (backup, restore, database)

* test: mirror cluster tests into src/tests/domain/cluster/
2026-06-27 17:23:57 +02:00
github-actions[bot] 0b7b7ef59b chore: release 1.13.1 2026-06-25 12:58:08 +00:00
Charles GTE 038c5523b7 fix: download-crash-large-files (#67)
* fix(restore): add ProgressTracker for download progress logging

* fix(restore): stream backup download to disk instead of buffering in RAM

* chore(restore): add download diagnostics + exact byte logging

* refactor(restore): drop ProgressTracker, keep streaming download

* feat(restore): drive download progress from RestoreInfo.size

* fix(restore): accept integer or string for RestoreInfo.size

* fix(restore): show raw bytes in download progress log

* fix(restore): collapse progress to highest milestone per chunk

* fix: downloader.rs
2026-06-25 14:55:49 +02:00
407 changed files with 8262 additions and 1949 deletions
+3 -5
View File
@@ -1,11 +1,9 @@
# Git
.git
.gitignore
# MD files
CHANGELOG.md
README.md
RELEASE.md
#IDE configurations
.idea
target
dump.rdb
.superpowers
+1 -1
View File
@@ -36,7 +36,7 @@ jobs:
agent-test bash -c "
mkdir -p /app/coverage &&
rm -rf /app/target/* /app/coverage/* &&
cargo test --verbose &&
cargo test --verbose -- --test-threads=2 &&
sync
"
+134
View File
@@ -0,0 +1,134 @@
name: GHCR Publish
on:
workflow_call:
inputs:
version:
required: true
type: string
ref:
required: true
type: string
add_latest:
required: false
type: boolean
default: false
dockerfile:
required: false
type: string
default: "./docker/Dockerfile"
target:
required: false
type: string
default: "prod"
permissions:
contents: read
packages: write
jobs:
build:
name: Build ${{ matrix.platform }}
runs-on: ${{ matrix.platform == 'linux/amd64' && 'ubuntu-latest' || 'ubuntu-24.04-arm' }}
strategy:
fail-fast: false
matrix:
platform: [ linux/amd64, linux/arm64 ]
steps:
- name: Prepare vars
id: prep
run: |
ARCH="${{ matrix.platform == 'linux/amd64' && 'amd64' || 'arm64' }}"
echo "arch=$ARCH" >> "$GITHUB_OUTPUT"
echo "image=ghcr.io/${GITHUB_REPOSITORY_OWNER,,}/agent" >> "$GITHUB_OUTPUT"
- name: Checkout
uses: actions/checkout@v4
with:
ref: ${{ inputs.ref }}
fetch-depth: 1
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Log in to GHCR
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build and push by digest
id: build
uses: docker/build-push-action@v6
with:
context: .
file: ${{ inputs.dockerfile }}
platforms: ${{ matrix.platform }}
target: ${{ inputs.target }}
provenance: false
outputs: type=image,name=${{ steps.prep.outputs.image }},push-by-digest=true,name-canonical=true,push=true
cache-from: type=gha,scope=ghcr-${{ steps.prep.outputs.arch }}
cache-to: type=gha,mode=max,scope=ghcr-${{ steps.prep.outputs.arch }},ignore-error=true
- name: Export digest
env:
DIGEST: ${{ steps.build.outputs.digest }}
run: |
mkdir -p /tmp/digests
touch "/tmp/digests/${DIGEST#sha256:}"
- name: Upload digest
uses: actions/upload-artifact@v4
with:
name: digest-${{ steps.prep.outputs.arch }}
path: /tmp/digests/*
if-no-files-found: error
retention-days: 1
merge:
name: Create multi-arch manifest
runs-on: ubuntu-latest
needs: build
steps:
- name: Prepare vars
id: prep
run: echo "image=ghcr.io/${GITHUB_REPOSITORY_OWNER,,}/agent" >> "$GITHUB_OUTPUT"
- name: Download digests
uses: actions/download-artifact@v4
with:
path: /tmp/digests
pattern: digest-*
merge-multiple: true
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Log in to GHCR
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract Docker metadata
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ steps.prep.outputs.image }}
tags: |
type=semver,pattern={{version}},value=${{ inputs.version }}
type=semver,pattern={{major}}.{{minor}},value=${{ inputs.version }}
type=semver,pattern={{major}},value=${{ inputs.version }}
type=raw,value=latest,enable=${{ inputs.add_latest }}
- name: Create and push manifest list
working-directory: /tmp/digests
run: |
docker buildx imagetools create \
$(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \
$(printf '${{ steps.prep.outputs.image }}@sha256:%s ' *)
- name: Inspect
run: docker buildx imagetools inspect ${{ steps.prep.outputs.image }}:${{ inputs.version }}
+22
View File
@@ -100,6 +100,15 @@ jobs:
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
publish-docker-ghcr:
needs: create-release
if: ${{ needs.create-release.result == 'success' }}
uses: ./.github/workflows/ghcr.yml
with:
version: ${{ needs.create-release.outputs.version }}
ref: ${{ needs.create-release.outputs.version }}
add_latest: true
publish-helm:
needs: create-release
if: ${{ needs.create-release.result == 'success' }}
@@ -109,11 +118,24 @@ jobs:
secrets:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
build-windows:
needs: create-release
if: ${{ needs.create-release.result == 'success' }}
uses: ./.github/workflows/windows-release.yml
with:
version: ${{ needs.create-release.outputs.version }}
ref: ${{ needs.create-release.outputs.version }}
draft_tag: ${{ needs.create-release.outputs.draft_tag }}
secrets:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
finalize-release:
needs:
- create-release
- publish-docker
- publish-docker-ghcr
- publish-helm
- build-windows
runs-on: ubuntu-latest
outputs:
release_tag: ${{ steps.publish_release_step.outputs.release_tag }}
+41 -52
View File
@@ -1,14 +1,30 @@
name: Build Windows release
on:
workflow_call:
inputs:
version:
description: 'Release version (git tag), e.g. 1.18.4'
type: string
required: false
ref:
description: 'Git ref to check out and build'
type: string
required: false
draft_tag:
description: 'Draft GitHub release tag to attach the asset to (e.g. untagged-xxxx). Empty = skip upload.'
type: string
required: false
secrets:
GH_TOKEN:
required: false
workflow_dispatch:
push:
tags:
- '[0-9]+.[0-9]+.[0-9]+'
branches:
- main
- master
inputs:
ref:
description: 'Git ref to check out and build'
type: string
required: false
jobs:
build-windows:
@@ -17,80 +33,53 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
ref: ${{ inputs.ref || github.ref }}
- name: Set up Rust toolchain (MSVC)
uses: actions-rs/toolchain@v1
uses: dtolnay/rust-toolchain@stable
with:
toolchain: stable-x86_64-pc-windows-msvc
profile: minimal
override: true
targets: x86_64-pc-windows-msvc
- name: Install vcpkg and OpenSSL (x64)
shell: pwsh
run: |
# Install vcpkg and the prebuilt OpenSSL package
git clone https://github.com/microsoft/vcpkg C:\vcpkg
C:\vcpkg\bootstrap-vcpkg.bat
C:\vcpkg\vcpkg install openssl:x64-windows
# Export variables for subsequent steps
'VCPKG_ROOT=C:\vcpkg' | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append
'OPENSSL_DIR=C:\vcpkg\installed\x64-windows' | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append
- name: Cache cargo build
uses: Swatinem/rust-cache@v2
- name: Build (cargo release)
shell: pwsh
env:
# Cargo / openssl-sys will pick up OPENSSL_DIR from the environment
OPENSSL_DIR: ${{ env.OPENSSL_DIR }}
run: |
# Ensure the environment variable is present for this step
if (-Not $env:OPENSSL_DIR) { Write-Host "OPENSSL_DIR not set, printing env for debugging"; Get-ChildItem Env: | ForEach-Object { Write-Host $_ } }
# Build the declared bin target explicitly (Cargo.toml [[bin]] name = "app")
cargo build --release --bin app
run: cargo build --release --bin app
- name: Prepare artifact zip
id: prepare_artifact
shell: pwsh
env:
RELEASE_TAG: ${{ github.ref_name }}
RELEASE_VERSION: ${{ inputs.version }}
run: |
$tag = $env:RELEASE_TAG
$tag = $env:RELEASE_VERSION
if (-not $tag) { $tag = $env:GITHUB_SHA }
# Package the declared bin target deterministically (Cargo.toml [[bin]] name = "app")
$exe = "target\release\app.exe"
if (-not (Test-Path $exe)) { Write-Error "Built binary $exe not found in target/release"; exit 1 }
$outDir = "artifact"
New-Item -ItemType Directory -Path $outDir -Force | Out-Null
# Ship under the package name, not the internal bin name "app"
# Ship under the package name, not the internal bin name "app".
Copy-Item -Path $exe -Destination "$outDir\portabase-agent.exe"
$zipName = "windows-release-$tag.zip"
if (Test-Path $zipName) { Remove-Item $zipName }
Compress-Archive -Path "$outDir\*" -DestinationPath $zipName -Force
Write-Host "ZIP=$zipName"
Write-Output "zip=$zipName" | Out-File -FilePath $env:GITHUB_OUTPUT -Encoding utf8 -Append
"zip=$zipName" | Out-File -FilePath $env:GITHUB_OUTPUT -Encoding utf8 -Append
- name: Upload build artifact
uses: actions/upload-artifact@v4
with:
name: windows-release
path: windows-release-*.zip
path: ${{ steps.prepare_artifact.outputs.zip }}
- name: Create GitHub Release
if: startsWith(github.ref, 'refs/tags/')
id: create_release
uses: softprops/action-gh-release@v1
with:
tag_name: ${{ github.ref_name }}
- name: Attach asset to draft release
if: ${{ inputs.draft_tag != '' }}
shell: pwsh
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Upload release asset
if: startsWith(github.ref, 'refs/tags/')
uses: actions/upload-release-asset@v1
with:
upload_url: ${{ steps.create_release.outputs.upload_url }}
asset_path: windows-release-${{ github.ref_name }}.zip
asset_name: windows-release-${{ github.ref_name }}.zip
asset_content_type: application/zip
GH_TOKEN: ${{ secrets.GH_TOKEN }}
run: |
gh release upload "${{ inputs.draft_tag }}" "${{ steps.prepare_artifact.outputs.zip }}" --clobber
+2 -1
View File
@@ -6,5 +6,6 @@
.env
.claude
/docs
.superpowers
+1 -1
View File
@@ -27,5 +27,5 @@ keywords:
- self-hosted
- portabase
license: Apache-2.0
version: 1.13.0
version: 1.21.2
date-released: '2026-02-24'
Generated
+1541 -1310
View File
File diff suppressed because it is too large Load Diff
+9 -4
View File
@@ -1,6 +1,6 @@
[package]
name = "portabase-agent"
version = "1.13.0"
version = "1.21.2"
edition = "2024"
[dependencies]
@@ -19,10 +19,11 @@ log = "0.4.29"
toml = "0.9.10"
reqwest = { version = "0.13.1", features = ["json", "blocking", "multipart", "stream", "query"] }
anyhow = "1.0.100"
tokio = { version = "1.49.0", features = ["rt", "rt-multi-thread", "macros", "fs"] }
tokio = { version = "1.49.0", features = ["rt", "rt-multi-thread", "macros", "fs", "process", "io-util"] }
async-trait = "0.1.89"
tempfile = "3.24.0"
openssl = "0.10.75"
hmac = "0.12"
sha2 = "0.10"
hex = "0.4.3"
flate2 = "1.1.5"
tar = "0.4.44"
@@ -35,12 +36,14 @@ rand = "0.9.2"
bytes = "1.11.0"
async-stream = "0.3.6"
uuid = { version = "1.20.0", features = ["v4"] }
tokio-util = { version = "0.7.18", features = ["compat"] }
tokio-util = { version = "0.7.18", features = ["compat", "io"] }
tiberius = { version = "0.12", default-features = false, features = ["rustls", "chrono"] }
aws-config = "1.8.13"
aws-sdk-s3 = { version = "1.122.0", features = ["behavior-version-latest"] }
azure_core = "1.0.0"
azure_storage_blob = "1.0.0"
google-cloud-storage = "1.15"
google-cloud-auth = "1.13"
async-compression = { version = "0.4.37", features = ["tokio", "gzip"] }
tokio-tar = "0.3.1"
oauth2 = "5.0.0"
@@ -55,6 +58,8 @@ testcontainers = "0.27.1"
testcontainers-modules = { version = "0.15.0", features = ["postgres", "redis", "valkey", "mysql", "mariadb", "mongo"] }
postgres = "0.19.12"
url = "2.5.8"
percent-encoding = "2.3.2"
bollard = "0.20.0"
[dev-dependencies]
tokio = { version = "1", features = ["full"] }
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.

Some files were not shown because too many files have changed in this diff Show More