Commit Graph

65 Commits

Author SHA1 Message Date
charles-gauthereau d1c8df4cac refactor(config): extract build_config, add load_optional + Serialize 2026-08-13 18:35:52 +02:00
Charles GTE 90d55f7630 fix: backup folder name (#86)
* fix: support configurable backup file prefix (#83)

* fix: backup folder name

* fix: ghcr publish agent image

---------

Co-authored-by: Antonin Jousson <18756890+Antoninj@users.noreply.github.com>
Co-authored-by: charles-gauthereau <charles.gauthereau@soluce-technologies.com>
2026-07-21 23:27:10 +02:00
charles-gauthereau 32d4e22196 fix: status.rs 2026-07-08 22:24:50 +02:00
charles-gauthereau f277c5485f feat: decrypt encrypted storages in status ping 2026-07-08 20:52:17 +02:00
charles-gauthereau 585e0bea72 feat: accept encrypted storages markers in DatabaseStatus 2026-07-08 20:47:20 +02:00
Charles GTE 324a2ea3f2 fix: security (#75) 2026-07-04 12:24:37 +02:00
charles-gauthereau b2b2733c4d feat: docker-volume provider
refactor: move build_tar to utils::compress

refactor: move choose_restore_path to utils::common

docs: remove docker-volume README section

fix: keep archive path for multi-file non-docker-volume restores

docs: trim docker-volume README section to essentials

refactor: run docker-volume backup/restore inside spawn_blocking like other providers

docs: enable docker socket, add volume example and security notes

feat: sweep orphaned ephemeral helper containers on startup

feat: docker-volume clean-replace restore via upload_to_container

fix: serialize env-var access in docker-volume tests to avoid setenv/getenv UB

feat: docker-volume backup via download_from_container

feat: docker-volume ping via inspect_volume

fix: return extraction dir for multi-file restore archives

feat: gzip already-tar inputs directly instead of double-wrapping

feat: docker helper (client, self-image, container lifecycle, sweep)

feat: docker-volume provider
2026-07-02 21:45:11 +02:00
charles-gauthereau aed1e86956 feat: add generic options map to DatabaseConfig
Add `options: HashMap<String, serde_json::Value>` to DatabaseConfig and
`options: Option<HashMap<...>>` to InputDatabaseConfig so callers can
pass arbitrary per-database options (e.g. keep_ownership for pg_dumpall).
Update all DatabaseConfig struct literals in test files and add three
new config-service unit tests covering present, absent, and non-bool values.
2026-06-29 18:49:01 +02:00
Charles GTE 16a37e033e fix: azure (#71) 2026-06-27 20:51:47 +02:00
Charles GTE 29ca5a317b feat: add-gcs-storage-provider (#70)
* fix: refactoring

* chore: add google-cloud-storage and google-cloud-auth deps

* feat: add GCS provider config model

* feat: add GCS credential, client, and stream-source helpers

StreamSource bridges build_stream's Send-only byte stream into the SDK's
StreamingSource (which send_buffered requires to be Send+Sync+'static) via a
bounded mpsc channel, avoiding any change to the shared UploadStream type.

* feat: implement GCS StorageProvider upload

* feat: register google-cloud-storage provider in factory

* test: GCS upload roundtrip against fake-gcs-server

* fix: format GCS bucket as projects/_/buckets/<name> in upload_with_client

write_object rejects the bare bucket id with "malformed bucket name"; the
production provider passed config.bucket_name unformatted, so real uploads
would always fail. Format once in upload_with_client so prod and the
fake-gcs-server test share the corrected path.

* style: cargo fmt GCS provider files

* fix: gcs
2026-06-27 20:37:22 +02:00
Charles GTE 01f5d34c96 fix: pgdump-cluster (#68)
* feat: add as_str/from_str to PostgresDumpFormat

* feat: resolve pg_dumpall/psql binary names

* feat: add include_globals field to database config

* feat: add pg_dumpall/psql globals dump and apply

* feat: add postgres backup bundle (manifest + build + resolve)

* feat: bundle globals into postgres backup when include_globals is set

* feat: replay globals before pg_restore when backup archive is a bundle

* refactor: bind FD restore tempdir guard once to clear unused warnings

* docs: demonstrate include_globals in sample databases.json

* chore: silence test-only re-export warning in non-test builds

* revert: remove include_globals feature, restore plain pg_dump/pg_restore

* feat: add pg_dumpall/psql binary names and is_superuser check

* feat: add postgresql-cluster db type and config parsing

* feat: pg_dumpall cluster backup and psql restore

* feat: route postgresql-cluster through PostgresClusterDatabase

* docs: add postgresql-cluster sample to databases.json

* refactor: split cluster mode into cluster/ module (backup, restore, database)

* test: mirror cluster tests into src/tests/domain/cluster/
2026-06-27 17:23:57 +02:00
Charles GTE 038c5523b7 fix: download-crash-large-files (#67)
* fix(restore): add ProgressTracker for download progress logging

* fix(restore): stream backup download to disk instead of buffering in RAM

* chore(restore): add download diagnostics + exact byte logging

* refactor(restore): drop ProgressTracker, keep streaming download

* feat(restore): drive download progress from RestoreInfo.size

* fix(restore): accept integer or string for RestoreInfo.size

* fix(restore): show raw bytes in download progress log

* fix(restore): collapse progress to highest milestone per chunk

* fix: downloader.rs
2026-06-25 14:55:49 +02:00
Charles GTE 221ed4e7e1 feat: add azure blob storage support 2026-06-20 19:38:00 +02:00
Charles GTE 84735ac399 feat(storage): spike Azure Blob SAS auth + block round-trip against Azurite
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-20 17:49:54 +02:00
Charles GTE 1fece577cc fix: tests and config.rs and backup.rs args 2026-06-20 14:36:09 +02:00
Peter Crosthwaite 7328827435 Fix mysqldump timeout on large database backups
- Add mysql:8.4 client tools stage to Dockerfile so the real mysqldump
  binary (with --network-timeout) takes PATH precedence over MariaDB's
  mysqldump symlink, which lacks this flag
- Use --network-timeout in MySQL backup command; sets net_read_timeout and
  net_write_timeout to 86400s for the duration of the dump
- Simplify config: replace net_read_timeout/net_write_timeout/max_allowed_packet
  with a single max_packet_size field (optional, defaults to 512M)
- Log mysqldump/mariadb-dump client version as a debug message before each backup
2026-06-20 18:06:21 +10:00
Peter Crosthwaite 348eaac81b Added config options for setting timeouts on mysql/mariadb 2026-06-20 07:51:18 +10:00
charles-gauthereau 82c7122790 fix: refactoring 2026-06-07 17:32:16 +02:00
Charles GTE fba72f5c26 feat: add job logs for restoration process 2026-06-07 11:16:02 +02:00
Charles GTE 0261a07fc2 fix: test ci 2026-06-06 20:59:30 +02:00
Charles GTE 2d52eccbb2 fix: test ci 2026-06-06 19:43:54 +02:00
Charles GTE 3c853ee7ca fix: tests and refactoring 2026-06-06 19:15:22 +02:00
Charles GTE 45d791358c fix: tracing logs 2026-06-06 11:01:55 +02:00
Charles GTE 0b915d87c9 feat: job logs 2026-06-06 10:22:27 +02:00
Charles GTE eac2853b83 feat: add-mssql-support (#56)
* fix: tests and refactoring, add justfile instead of makefile

* chore: add tiberius dependency for MSSQL support

* feat: add Mssql variant to DbType enum and config validation

* feat: add MSSQL domain adapter with tiberius ping and sqlpackage backup/restore

* feat: install sqlpackage in Docker base and prod stages

* feat: add MSSQL service to docker-compose and config examples

* fix: log backup errors instead of silently discarding them

* fix: use dotnet-install.sh instead of Microsoft apt feed (SHA1 key rejected by Debian Trixie)

* fix: add /TrustServerCertificate:true to sqlpackage commands for self-signed certs

* fix: use connection string with TrustServerCertificate for sqlpackage

* fix: use user database in MSSQL config example (system databases cannot be exported)

* feat: add seed-mssql task and seed script

* fix: docker-compose.yml
2026-05-11 22:04:36 +02:00
Charles GTE 14068846b9 fix: migration-process (#55)
* fix: s3 port type mismatch, add string_or_number_to_string deserializer

* chore: docker-compose.databases.yml

* fix: tests and refactoring, add justfile instead of makefile

* fix: restoration process for postgres/mysql and mariadb

* fix: restoration process for postgres/mysql and mariadb

* fix: postgres restore

* fix: mariadb/mysql restore, added backticks to drop command

* fix: mongodb and migration test for firebird

* fix: docker-compose.yml

---------

Co-authored-by: charlesgauthereau <charles.gauthereau@soluce-technologies.com>
2026-04-29 22:40:02 +02:00
Charles GTE f37e4d2de9 feat: add firebird (#51)
* feat: add firebird

* feat: add firebird tests

* fix: tests and refactoring, add justfile instead of makefile
2026-03-31 15:10:05 +02:00
Charles GTE d4abcbcebc feat: separate mysql and mariadb (#49)
* feat: separate mysql and mariadb

* feat: separate mysql and mariadb
2026-03-29 09:28:30 +02:00
Charles GTE 517f3779cd feat: sending database ping result for each databases to portabase server 2026-03-27 23:07:06 +01:00
Charles GTE 725312e1d5 fix: tests (#47)
* fix: s3 port type mismatch, add string_or_number_to_string deserializer

* fix: adding some tests

* fix: adding some tests

* fix: working on pipeline

* fix: working on pipeline

* fix: working on pipeline

* fix: working on pipeline

* fix: working on pipeline

* fix: working on pipeline

* fix: working on pipeline

* fix: working on pipeline

* fix: working on pipeline

* fix: working on pipeline

* fix: working on pipeline

* fix: working on pipeline

* fix: working on pipeline

* fix: working on pipeline

* fix: working on pipeline

* fix: working on pipeline

* fix: working on pipeline

* fix: working on pipeline

* fix: working on pipeline

* fix: working on pipeline

* fix: working on pipeline

* fix: working on pipeline

---------

Co-authored-by: charlesgauthereau <charles.gauthereau@soluce-technologies.com>
2026-03-27 22:24:59 +01:00
Charles GTE 8489a6f6e3 fix: adding retry config for s3 provider (#46) 2026-03-21 08:51:15 +01:00
charles-gauthereau cf5de3115c feat: add valkey database engine for backup 2026-03-16 12:11:01 +01:00
Charles GTE e9147d6c60 fix: s3 port type mismatch, add string_or_number_to_string deserializer (#42)
Co-authored-by: charlesgauthereau <charles.gauthereau@soluce-technologies.com>
2026-03-16 08:57:00 +01:00
charlesgauthereau 1fdea31430 fix: postgres and redis tests 2026-03-14 17:59:44 +01:00
charlesgauthereau d1d633fd00 feat: add redis database support for backup. 2026-03-13 22:44:46 +01:00
Charles GTE f973bc1f52 fix: refactoring for backup and restore service
* fix: docker-compose.yml

* fix: refactoring for backup and restore service

* fix: cargo update

* fix: refactoring of the backup and restore services

---------

Co-authored-by: charlesgauthereau <charles.gauthereau@soluce-technologies.com>
2026-03-08 19:47:04 +01:00
charlesgauthereau 35933aae52 fix: refactoring config check on startup agent. 2026-02-23 21:47:18 +01:00
charlesgauthereau 178628cb1c feat: add sqlite support, first working version 2026-02-22 21:46:16 +01:00
charlesgauthereau 18fe211ad1 fix: s3 provider port config. 2026-02-19 21:14:35 +01:00
charlesgauthereau 8a05b93ff3 chore(release): 1.1.3-rc.1 2026-02-16 10:56:37 +01:00
charlesgauthereau b06f9d8186 fix: failed backup 2026-02-16 09:00:18 +01:00
charlesgauthereau d8b239a123 fix: s3 region to us-east-1 2026-02-16 08:18:14 +01:00
charlesgauthereau e678cf4b06 feat: Adding AES GCM encryption. 2026-02-12 19:49:17 +01:00
charlesgauthereau 3c599e94de feat: Adding AES GCM encryption. 2026-02-12 19:16:52 +01:00
charlesgauthereau 6bbc856f62 feat: Adding AES GCM encryption. 2026-02-12 18:45:38 +01:00
charlesgauthereau e2d7c4747e feat: Ready for Testing. 2026-02-10 20:08:00 +01:00
charlesgauthereau 08b5124a5a feat: added the is_locked check for database. 2026-02-10 19:59:13 +01:00
charlesgauthereau cc0de33a62 fix: working integrating google drive 2026-02-08 20:42:30 +01:00
charlesgauthereau ce00959feb fix: working on restore. 2026-02-08 16:54:45 +01:00
charlesgauthereau cfef9b047b fix: working on metadata update between server and agent. 2026-02-07 23:31:32 +01:00