Compare commits

...

124 Commits

Author SHA1 Message Date
github-actions[bot] 987c630c73 chore: release 1.5.1 2026-03-16 07:59:25 +00:00
Charles GTE e9147d6c60 fix: s3 port type mismatch, add string_or_number_to_string deserializer (#42)
Co-authored-by: charlesgauthereau <charles.gauthereau@soluce-technologies.com>
2026-03-16 08:57:00 +01:00
github-actions[bot] 174ead72a3 chore: release 1.5.0 2026-03-15 11:06:29 +00:00
Charles GTE f2a67b3490 feat: add-redis #39
feat: add-redis
2026-03-15 12:04:05 +01:00
charlesgauthereau d4afefbc1e fix: requirements.sh for tests tools 2026-03-14 20:08:26 +01:00
charlesgauthereau be6e147df2 fix: requirements.sh for tests tools 2026-03-14 19:54:15 +01:00
charlesgauthereau 91b6658852 fix: requirements.sh for tests tools 2026-03-14 19:40:01 +01:00
charlesgauthereau f7dcb6b2bb fix: requirements.sh for tests tools 2026-03-14 18:26:26 +01:00
charlesgauthereau ab14f0dcf0 fix: requirements.sh for tests tools 2026-03-14 18:17:43 +01:00
charlesgauthereau 58a0b92710 fix: pipeline codecov.yml 2026-03-14 18:05:29 +01:00
charlesgauthereau 2f24291c0e fix: pipeline codecov.yml 2026-03-14 18:02:06 +01:00
charlesgauthereau 1fdea31430 fix: postgres and redis tests 2026-03-14 17:59:44 +01:00
charlesgauthereau d1d633fd00 feat: add redis database support for backup. 2026-03-13 22:44:46 +01:00
Charles GTE b1c14f098b fix: codecov.yml (#38)
* feat: add codecov.yml

* fix: add codecov.yml

* fix: codecov.yml

---------

Co-authored-by: charles-gauthereau <charles.gauthereau@soluce-technologies.com>
2026-03-13 16:58:51 +01:00
github-actions[bot] 3abb955e24 chore: release 1.4.1 2026-03-13 15:19:42 +00:00
Charles GTE 6d22be9b50 fix: add codecov.yml (#37) [skip-release]
Co-authored-by: charles-gauthereau <charles.gauthereau@soluce-technologies.com>
2026-03-13 16:17:09 +01:00
Charles GTE c2df4ba71e fix: readme badge [skip-release] #36
fix: readme badge  [skip-release]
2026-03-13 15:57:16 +01:00
charles-gauthereau 46b3f2466a fix: readme badge 2026-03-13 13:50:30 +01:00
github-actions[bot] a2be03751f chore: release 1.4.0 2026-03-12 18:23:42 +00:00
Charles GTE 7df9df1605 feat : test coverage for utils functions #35
feat : test coverage for utils functions
2026-03-12 19:21:09 +01:00
charlesgauthereau df5b16a153 fix: docker-compose.yml 2026-03-12 19:16:12 +01:00
charlesgauthereau 824a5d52a3 feat: add test for util functions and start working on database providers tests procedures 2026-03-12 18:56:07 +01:00
charlesgauthereau 1496208db4 Merge branch 'main' into dev 2026-03-09 16:46:00 +01:00
charlesgauthereau 454e1b442f Merge remote-tracking branch 'origin/main' 2026-03-09 16:45:52 +01:00
github-actions[bot] 54bca7452a chore: release 1.3.0 2026-03-09 15:43:44 +00:00
Charles GTE a96cef6cb5 feat: helm chart (#34)
* fix: docker.yml

* feat: add helm chart config

* fix: helm config

* feat: add .github workflow for helm chart

---------

Co-authored-by: charlesgauthereau <charles.gauthereau@soluce-technologies.com>
2026-03-09 16:41:14 +01:00
charlesgauthereau fd9f183166 Merge branch 'main' into dev 2026-03-08 22:02:36 +01:00
charlesgauthereau 94fb2535ec Merge remote-tracking branch 'origin/main' 2026-03-08 21:59:02 +01:00
github-actions[bot] 189ad866de chore: release 1.2.5 2026-03-08 20:50:53 +00:00
Charles GTE 55bbd4e724 fix: new-workflow-ci (#32)
* feat: new workflow CI

* feat: new workflow CI

* feat: new workflow CI

* feat: new workflow CI

* feat: new workflow CI

* feat: new workflow CI

* fix: docker.yml

* fix: docker.yml

* fix: docker.yml

* fix: docker.yml

* fix: docker.yml

* fix: docker.yml

---------

Co-authored-by: charlesgauthereau <charles.gauthereau@soluce-technologies.com>
2026-03-08 21:48:13 +01:00
charlesgauthereau 7fabc9c98c Merge remote-tracking branch 'origin/main' 2026-03-08 21:47:15 +01:00
github-actions[bot] 4f67a57681 chore: release 1.2.4 2026-03-08 20:20:58 +00:00
Charles GTE 45a1118f64 fix: docker build (#31)
* feat: new workflow CI

* feat: new workflow CI

* feat: new workflow CI

* feat: new workflow CI

* feat: new workflow CI

* feat: new workflow CI

* fix: docker.yml

* fix: docker.yml

* fix: docker.yml

* fix: docker.yml

* fix: docker.yml

---------

Co-authored-by: charlesgauthereau <charles.gauthereau@soluce-technologies.com>
2026-03-08 21:18:27 +01:00
charlesgauthereau 063b2e1c2c Merge remote-tracking branch 'origin/main' 2026-03-08 21:12:09 +01:00
github-actions[bot] f2275d5aca chore: release 1.2.3 2026-03-08 20:00:47 +00:00
Charles GTE b136140b55 fix: new-workflow-ci (#30)
* feat: new workflow CI

* feat: new workflow CI

* feat: new workflow CI

* feat: new workflow CI

* feat: new workflow CI

* feat: new workflow CI

* fix: docker.yml

* fix: docker.yml

* fix: docker.yml

* fix: docker.yml

---------

Co-authored-by: charlesgauthereau <charles.gauthereau@soluce-technologies.com>
2026-03-08 20:58:19 +01:00
charlesgauthereau 924177a4fe Merge remote-tracking branch 'origin/main' 2026-03-08 20:52:36 +01:00
github-actions[bot] 773077f66b chore: release 1.2.2 2026-03-08 19:48:25 +00:00
Charles GTE 46ff086cf9 fix: new-workflow-ci (#29)
* feat: new workflow CI

* feat: new workflow CI

* feat: new workflow CI

* feat: new workflow CI

* feat: new workflow CI

* feat: new workflow CI

* fix: docker.yml

* fix: docker.yml

---------

Co-authored-by: charlesgauthereau <charles.gauthereau@soluce-technologies.com>
2026-03-08 20:47:41 +01:00
charlesgauthereau 678df8e2bc fix: docker.yml 2026-03-08 20:46:47 +01:00
github-actions[bot] f563c187db chore: release 1.2.1 2026-03-08 19:43:01 +00:00
Charles GTE df0dffa47b fix: new-workflow-ci (#28)
* feat: new workflow CI

* feat: new workflow CI

* feat: new workflow CI

* feat: new workflow CI

* feat: new workflow CI

* feat: new workflow CI

---------

Co-authored-by: charlesgauthereau <charles.gauthereau@soluce-technologies.com>
2026-03-08 20:42:16 +01:00
github-actions[bot] 864396422b chore: release 1.2.0 2026-03-08 19:38:10 +00:00
Charles GTE e280c1d2f8 fix: workflow (#27)
* feat: new workflow CI

* feat: new workflow CI

* feat: new workflow CI

* feat: new workflow CI

* feat: new workflow CI

---------

Co-authored-by: charlesgauthereau <charles.gauthereau@soluce-technologies.com>
2026-03-08 20:31:41 +01:00
Charles GTE 91a9d0654b feat: new workflow CI (#26)
* feat: new workflow CI

* feat: new workflow CI

* feat: new workflow CI

* feat: new workflow CI

---------

Co-authored-by: charlesgauthereau <charles.gauthereau@soluce-technologies.com>
2026-03-08 20:28:26 +01:00
Charles GTE bc857a4c18 feat: new workflow CI
* feat: new workflow CI

* feat: new workflow CI

---------

Co-authored-by: charlesgauthereau <charles.gauthereau@soluce-technologies.com>
2026-03-08 20:24:26 +01:00
Charles GTE 2454915866 feat: new workflow CI (#24)
Co-authored-by: charlesgauthereau <charles.gauthereau@soluce-technologies.com>
2026-03-08 20:22:45 +01:00
Charles GTE a4a7a9a2c4 feat: new workflow CI
feat: new workflow CI
2026-03-08 20:12:43 +01:00
charlesgauthereau 997ba2afa2 feat: new workflow CI 2026-03-08 20:12:00 +01:00
charlesgauthereau 7e0c7c0688 Merge branch 'main' into dev
# Conflicts:
#	docker-compose.yml
2026-03-08 19:48:00 +01:00
Charles GTE f973bc1f52 fix: refactoring for backup and restore service
* fix: docker-compose.yml

* fix: refactoring for backup and restore service

* fix: cargo update

* fix: refactoring of the backup and restore services

---------

Co-authored-by: charlesgauthereau <charles.gauthereau@soluce-technologies.com>
2026-03-08 19:47:04 +01:00
charlesgauthereau b9b9fcf6ee Merge branch 'main' into dev 2026-03-08 10:02:34 +01:00
charlesgauthereau 0224605c59 fix: docker-compose.yml 2026-03-08 10:02:17 +01:00
Charles GTE 6ea9fdf8bc Merge pull request #21 from Portabase/dev
chore: add coderabbitai config file
2026-03-05 11:21:45 +01:00
Charles GTE ad68be8a5a fix: coderrabbit.yaml 2026-03-05 11:19:47 +01:00
Charles GTE 434c131afb chore: add coderabbitai config file 2026-03-05 11:10:20 +01:00
Charles GTE 675ac5976d Merge pull request #20 from Portabase/dev
chore: add trailing newlines to README.md
2026-03-05 10:59:26 +01:00
Charles GTE 70543fe032 fix: README.md 2026-03-05 10:45:55 +01:00
Charles GTE 77f90308b4 Update README.md 2026-03-02 08:46:32 +01:00
charlesgauthereau 63cb254abd chore: Cargo.lock 2026-02-24 08:45:50 +01:00
charlesgauthereau 7738745def chore(release): 1.1.6 2026-02-24 08:45:26 +01:00
charlesgauthereau 82659f681c feat: add local test db for sqlite 2026-02-23 22:44:57 +01:00
charlesgauthereau eee3bebd50 feat: add local test db for sqlite 2026-02-23 22:43:44 +01:00
charlesgauthereau 58dd0259f9 chore(release): 1.1.6-rc.1 2026-02-23 21:55:47 +01:00
Charles GTE 5b7453208b Merge pull request #16 from Portabase/feat/sqlite
feat/sqlite
2026-02-23 21:54:04 +01:00
charlesgauthereau 35933aae52 fix: refactoring config check on startup agent. 2026-02-23 21:47:18 +01:00
charlesgauthereau 178628cb1c feat: add sqlite support, first working version 2026-02-22 21:46:16 +01:00
charlesgauthereau 3fc60e3f40 fix: s3 provider port config. 2026-02-19 21:16:13 +01:00
charlesgauthereau e62de95827 chore(release): 1.1.5 2026-02-19 21:15:55 +01:00
Charles GTE fc3612fa3f Merge pull request #14 from Portabase/dev
fix: s3 provider port config.
2026-02-19 21:15:19 +01:00
charlesgauthereau 18fe211ad1 fix: s3 provider port config. 2026-02-19 21:14:35 +01:00
charlesgauthereau 74ada01232 chore: Cargo.lock 2026-02-19 16:10:37 +01:00
charlesgauthereau 4fef360c06 chore(release): 1.1.4 2026-02-19 16:10:17 +01:00
charlesgauthereau 2647941008 fix: tus.rs 2026-02-19 16:09:43 +01:00
Charles GTE b4bdc89888 Merge pull request #13 from Portabase/fix/error-handling
fix/error-handling
2026-02-18 19:28:33 +01:00
charlesgauthereau 829e5bade3 fix: tus.rs 2026-02-18 17:28:59 +01:00
charlesgauthereau fc9a4ef8fb chore(release): 1.1.4-rc.1 2026-02-18 17:28:27 +01:00
charlesgauthereau 3c5734ed4e fix: tus.rs 2026-02-18 17:27:24 +01:00
charlesgauthereau 4b77896ecb fix: Cargo.lock 2026-02-16 11:30:38 +01:00
charlesgauthereau 40c4a4e2bb chore(release): 1.1.3 2026-02-16 11:30:16 +01:00
charlesgauthereau 8a05b93ff3 chore(release): 1.1.3-rc.1 2026-02-16 10:56:37 +01:00
charlesgauthereau c186932a79 fix: failed backup 2026-02-16 09:19:31 +01:00
charlesgauthereau bf74173cac chore(release): 1.1.2 2026-02-16 09:19:17 +01:00
charlesgauthereau 3dcebf4fcc fix: failed backup 2026-02-16 09:00:43 +01:00
charlesgauthereau 4de2b81b33 chore(release): 1.1.2-rc.2 2026-02-16 09:00:26 +01:00
charlesgauthereau b06f9d8186 fix: failed backup 2026-02-16 09:00:18 +01:00
charlesgauthereau f5a2182723 fix: blocking for mysql database 2026-02-16 08:43:27 +01:00
charlesgauthereau e8a96980f1 chore(release): 1.1.2-rc.1 2026-02-16 08:43:13 +01:00
charlesgauthereau a64792bcc6 fix: blocking for mysql database 2026-02-16 08:41:44 +01:00
charlesgauthereau 2d4542be5f chore: Cargo.lock 2026-02-16 08:21:32 +01:00
charlesgauthereau 4a00ed574d chore(release): 1.1.1 2026-02-16 08:21:10 +01:00
Charles GTE fd220598fc Merge pull request #12 from Portabase/fix/s3-region
fix/s3-region
2026-02-16 08:19:11 +01:00
charlesgauthereau d8b239a123 fix: s3 region to us-east-1 2026-02-16 08:18:14 +01:00
Charles GTE ebf0526366 Update README.md 2026-02-15 11:52:35 +01:00
Charles GTE 6461c1523b Merge pull request #11 from Portabase/dev
dev
2026-02-15 11:51:51 +01:00
charlesgauthereau 2809373de1 feat: README.md and docker-compose.yml 2026-02-15 11:50:16 +01:00
charlesgauthereau 5349e1fef9 chore(release): 1.1.0 2026-02-13 14:37:18 +01:00
charlesgauthereau 2bcb393d60 feat: Adding AES GCM encryption. 2026-02-13 14:36:54 +01:00
Charles GTE 406273796e Merge pull request #10 from Portabase/dev
dev
2026-02-13 14:36:19 +01:00
charlesgauthereau cb35347cb6 chore(release): 1.1.0-rc.2 2026-02-12 20:24:21 +01:00
Charles GTE b92138fd84 Merge pull request #9 from Portabase/fix/large-backup-v2
fix/large-backup-v2
2026-02-12 20:22:23 +01:00
charlesgauthereau f4c54836f1 feat: Adding AES GCM encryption. 2026-02-12 20:21:07 +01:00
charlesgauthereau e678cf4b06 feat: Adding AES GCM encryption. 2026-02-12 19:49:17 +01:00
charlesgauthereau 3c599e94de feat: Adding AES GCM encryption. 2026-02-12 19:16:52 +01:00
charlesgauthereau 6bbc856f62 feat: Adding AES GCM encryption. 2026-02-12 18:45:38 +01:00
charlesgauthereau 417212c9e8 chore(release): 1.1.0-rc.1 2026-02-10 20:12:05 +01:00
charlesgauthereau 552f1b8813 chore(release): 1.1.0-rc.1 2026-02-10 20:09:08 +01:00
charlesgauthereau e2d7c4747e feat: Ready for Testing. 2026-02-10 20:08:00 +01:00
charlesgauthereau 08b5124a5a feat: added the is_locked check for database. 2026-02-10 19:59:13 +01:00
charlesgauthereau cc0de33a62 fix: working integrating google drive 2026-02-08 20:42:30 +01:00
charlesgauthereau ce00959feb fix: working on restore. 2026-02-08 16:54:45 +01:00
charlesgauthereau cfef9b047b fix: working on metadata update between server and agent. 2026-02-07 23:31:32 +01:00
charlesgauthereau caf6127f7a fix: start working on s3 provider 2026-02-05 22:46:00 +01:00
charlesgauthereau b71f29eb21 fix: fix upload_futures 2026-02-05 08:57:15 +01:00
charlesgauthereau 2988cb297b fix: Working on multiple providers methods 2026-02-04 23:04:31 +01:00
charlesgauthereau 5019e9585b fix: adding the metadata into tasks and also updated them is cron of storages changed 2026-02-04 21:37:53 +01:00
charlesgauthereau 027751febb chore(release): 1.0.5-rc.2 2026-02-01 19:51:46 +01:00
charlesgauthereau f6f378d584 chore(release): 1.0.5-rc.1 2026-02-01 18:21:55 +01:00
charlesgauthereau 66c0b04759 fix: for testing 2026-02-01 18:20:49 +01:00
charlesgauthereau c348253015 fix: refactoring for big backup files. 2026-02-01 15:49:33 +01:00
charlesgauthereau e83e44a39e chore: Cargo.lock 2026-01-28 23:32:08 +01:00
charlesgauthereau 92d64c0b4b chore(release): 1.0.4 2026-01-28 23:31:34 +01:00
Charles GTE abd39e2db1 Merge pull request #8 from Portabase/fix/docker-image
Fix/docker image
2026-01-28 23:31:10 +01:00
charlesgauthereau 48a4ef6cc2 fix: Dockerfile 2026-01-28 23:30:37 +01:00
135 changed files with 7649 additions and 1613 deletions
+23
View File
@@ -0,0 +1,23 @@
reviews:
auto_review:
enabled: true
ignore_usernames: ["dependabot[bot]"]
labels: ["!wip", "!draft"]
auto_apply_labels: true
suggested_reviewers: true
auto_assign_reviewers: true
issue_enrichment:
auto_enrich:
enabled: true
labeling:
auto_apply_labels: true
labeling_instructions:
- label: bug
instructions: "Error, crash, or incorrect behavior reports"
- label: enhancement
instructions: "Feature requests or improvements"
- label: documentation
instructions: "Docs updates or missing documentation"
- label: good first issue
instructions: "Beginner-friendly tasks suitable for new contributors"
+58
View File
@@ -0,0 +1,58 @@
name: Codecov Rust
on:
push:
branches: ["main"]
pull_request:
branches: ["main"]
env:
CARGO_TERM_COLOR: always
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions-rs/toolchain@v1
with:
toolchain: stable
override: true
components: llvm-tools-preview
- name: Install grcov
run: cargo install grcov
- name: Install test requirements
run: bash scripts/tests/requirements.sh
- name: Build
run: cargo build --verbose
- name: Run tests
env:
CARGO_INCREMENTAL: 0
RUSTFLAGS: "-C instrument-coverage"
LLVM_PROFILE_FILE: "cargo-test-%p-%m.profraw"
run: cargo test --verbose
- name: Generate coverage
run: |
grcov . \
--binary-path ./target/debug/ \
-s . \
-t lcov \
--branch \
--ignore-not-existing \
-o lcov.info
- name: Upload to Codecov
uses: codecov/codecov-action@v5
with:
files: lcov.info
verbose: true
fail_ci_if_error: true
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
+73
View File
@@ -0,0 +1,73 @@
name: Discord Notification
on:
workflow_call:
inputs:
release_tag:
required: true
type: string
discord_title:
required: true
type: string
discord_color:
required: true
type: number
discord_footer:
required: true
type: string
secrets:
DISCORD_WEBHOOK:
required: true
GH_TOKEN:
required: true
jobs:
notify-discord:
runs-on: ubuntu-latest
steps:
- name: Send Discord Notification
env:
DISCORD_WEBHOOK: ${{ secrets.DISCORD_WEBHOOK }}
GH_TOKEN: ${{ secrets.GH_TOKEN }}
run: |
RELEASE_INFO=$(gh release view "${{ inputs.release_tag }}" -R ${{ github.repository }} --json name,url,body,author)
RELEASE_TITLE=$(echo "$RELEASE_INFO" | jq -r .name)
if [ -z "$RELEASE_TITLE" ] || [ "$RELEASE_TITLE" = "null" ]; then RELEASE_TITLE="${{ inputs.release_tag }}"; fi
RELEASE_URL=$(echo "$RELEASE_INFO" | jq -r .url)
RELEASE_BODY=$(echo "$RELEASE_INFO" | jq -r .body)
AUTHOR_NAME="Portabase"
AUTHOR_ICON="https://github.com/Portabase.png"
PAYLOAD=$(jq -n \
--arg title "$RELEASE_TITLE" \
--arg description "$RELEASE_BODY" \
--arg url "$RELEASE_URL" \
--arg author "$AUTHOR_NAME" \
--arg icon "$AUTHOR_ICON" \
--arg discord_title "${{ inputs.discord_title }}" \
--arg discord_footer "${{ inputs.discord_footer }}" \
--argjson discord_color ${{ inputs.discord_color }} \
'{
content: $discord_title,
embeds: [{
title: $title,
url: $url,
description: $description,
color: $discord_color,
author: {
name: $author,
icon_url: $icon
},
footer: {
text: $discord_footer
}
}]
}'
)
curl -H "Content-Type: application/json" \
-d "$PAYLOAD" \
"$DISCORD_WEBHOOK"
+66 -254
View File
@@ -1,207 +1,18 @@
#name: Docker Publish
#
#on:
# workflow_call:
# inputs:
# image_name:
# required: false
# type: string
# default: 'portabase/agent'
# add_latest:
# required: false
# type: boolean
# default: false
# target:
# required: false
# type: string
# default: 'prod'
# dockerfile:
# required: false
# type: string
# default: './docker/Dockerfile'
# secrets:
# DOCKER_USERNAME:
# required: true
# DOCKER_PASSWORD:
# required: true
#
#jobs:
# build-and-push:
# runs-on: ubuntu-latest
# permissions:
# packages: write
# contents: read
# steps:
# - name: Check out the repo
# uses: actions/checkout@v4
# with:
# fetch-depth: 0
#
# - name: Set up QEMU (enables multi-arch emulation)
# uses: docker/setup-qemu-action@v3
#
# - name: Set up Docker Buildx
# uses: docker/setup-buildx-action@v3
#
# - name: Log in to Docker Hub
# uses: docker/login-action@f4ef78c080cd8ba55a85445d5b36e214a81df20a
# with:
# username: ${{ secrets.DOCKER_USERNAME }}
# password: ${{ secrets.DOCKER_PASSWORD }}
#
# - name: Set tags
# id: set-tags
# run: |
# REF_NAME=${GITHUB_REF#refs/tags/}
# TAGS="${{ inputs.image_name }}:$REF_NAME"
# if [[ "${{ inputs.add_latest }}" == "true" ]]; then
# TAGS="$TAGS,${{ inputs.image_name }}:latest"
# fi
# echo "tags=$TAGS" >> $GITHUB_OUTPUT
#
# - name: Build and push Docker image
# uses: docker/build-push-action@v6
# with:
# context: .
# file: ${{ inputs.dockerfile }}
# platforms: linux/amd64,linux/arm64
# push: true
# tags: ${{ steps.set-tags.outputs.tags }}
# target: ${{ inputs.target }}
#
#name: Docker Publish
#
#on:
# workflow_call:
# inputs:
# image_name:
# required: false
# type: string
# default: 'portabase/agent'
# add_latest:
# required: false
# type: boolean
# default: false
# target:
# required: false
# type: string
# default: 'prod'
# dockerfile:
# required: false
# type: string
# default: './docker/Dockerfile'
# secrets:
# DOCKER_USERNAME:
# required: true
# DOCKER_PASSWORD:
# required: true
#
#jobs:
# build-amd64:
# name: Build AMD64 Image
# runs-on: ubuntu-latest
# outputs:
# image: ${{ steps.set-job-output.outputs.image }}
# steps:
# - uses: actions/checkout@v4
#
# - uses: docker/setup-buildx-action@v3
#
# - uses: docker/login-action@f4ef78c080cd8ba55a85445d5b36e214a81df20a
# with:
# username: ${{ secrets.DOCKER_USERNAME }}
# password: ${{ secrets.DOCKER_PASSWORD }}
#
# - name: Set AMD64 image tag
# id: set-tags
# run: |
# REF_NAME=${GITHUB_REF#refs/tags/}
# AMD64_IMAGE="${{ inputs.image_name }}:$REF_NAME-amd64"
# echo "amd64_image=$AMD64_IMAGE" >> $GITHUB_OUTPUT
#
# - name: Build and push AMD64
# uses: docker/build-push-action@v6
# with:
# context: .
# file: ${{ inputs.dockerfile }}
# platforms: linux/amd64
# push: true
# tags: ${{ steps.set-tags.outputs.amd64_image }}
# target: ${{ inputs.target }}
#
# build-arm64:
# name: Build ARM64 Image
# runs-on: ubuntu-24.04-arm
# outputs:
# image: ${{ steps.set-job-output.outputs.image }}
# steps:
# - uses: actions/checkout@v4
#
# - uses: docker/setup-buildx-action@v3
#
# - uses: docker/login-action@f4ef78c080cd8ba55a85445d5b36e214a81df20a
# with:
# username: ${{ secrets.DOCKER_USERNAME }}
# password: ${{ secrets.DOCKER_PASSWORD }}
#
# - name: Set ARM64 image tag
# id: set-tags
# run: |
# REF_NAME=${GITHUB_REF#refs/tags/}
# ARM64_IMAGE="${{ inputs.image_name }}:$REF_NAME-arm64"
# echo "arm64_image=$ARM64_IMAGE" >> $GITHUB_OUTPUT
#
# - name: Build and push ARM64
# uses: docker/build-push-action@v6
# with:
# context: .
# file: ${{ inputs.dockerfile }}
# platforms: linux/arm64
# push: true
# tags: ${{ steps.set-tags.outputs.arm64_image }}
# target: ${{ inputs.target }}
#
#
# create-manifest:
# name: Create Multi-Arch Manifest
# runs-on: ubuntu-latest
# needs:
# - build-amd64
# - build-arm64
# steps:
# - uses: docker/login-action@f4ef78c080cd8ba55a85445d5b36e214a81df20a
# with:
# username: ${{ secrets.DOCKER_USERNAME }}
# password: ${{ secrets.DOCKER_PASSWORD }}
#
# - name: Create Docker manifest
# run: |
# REF_NAME=${GITHUB_REF#refs/tags/}
#
# IMAGE="${{ inputs.image_name }}:$REF_NAME"
# AMD64_IMAGE="$IMAGE-amd64"
# ARM64_IMAGE="$IMAGE-arm64"
#
# echo "AMD64_IMAGE=$AMD64_IMAGE"
# echo "ARM64_IMAGE=$ARM64_IMAGE"
#
# docker manifest create $IMAGE $AMD64_IMAGE $ARM64_IMAGE
# docker manifest push $IMAGE
#
# if [ "${{ inputs.add_latest }}" = "true" ]; then
# docker manifest create ${{ inputs.image_name }}:latest $AMD64_IMAGE $ARM64_IMAGE
# docker manifest push ${{ inputs.image_name }}:latest
# fi
name: Docker Publish
on:
workflow_call:
inputs:
version:
required: true
type: string
ref:
required: true
type: string
image_name:
required: false
type: string
default: 'portabase/agent'
default: "portabase/agent"
add_latest:
required: false
type: boolean
@@ -209,11 +20,11 @@ on:
target:
required: false
type: string
default: 'prod'
default: "prod"
dockerfile:
required: false
type: string
default: './docker/Dockerfile'
default: "./docker/Dockerfile"
secrets:
DOCKER_USERNAME:
required: true
@@ -222,32 +33,34 @@ on:
jobs:
build:
name: Build and push Docker images
runs-on: ${{ matrix.platform == 'linux/amd64' && 'ubuntu-latest' || matrix.platform == 'linux/arm64' && 'ubuntu-24.04-arm' }}
name: Build architectures
runs-on: ${{ matrix.platform == 'linux/amd64' && 'ubuntu-latest' || 'ubuntu-24.04-arm' }}
strategy:
fail-fast: false
matrix:
platform: [linux/amd64, linux/arm64]
platform: [ linux/amd64, linux/arm64 ]
steps:
- name: Checkout
uses: actions/checkout@v4
- uses: actions/checkout@v4
with:
ref: ${{ inputs.ref }}
fetch-depth: 0
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Login to Docker
uses: docker/login-action@f4ef78c080cd8ba55a85445d5b36e214a81df20a
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Set image tag
id: set-tags
- name: Prepare Image Tags
id: prep
run: |
REF_NAME=${GITHUB_REF#refs/tags/}
if [ "${{ matrix.platform }}" = "linux/amd64" ]; then
IMAGE="${{ inputs.image_name }}:$REF_NAME-amd64"
else
IMAGE="${{ inputs.image_name }}:$REF_NAME-arm64"
fi
echo "image=$IMAGE" >> $GITHUB_OUTPUT
ARCH=${{ matrix.platform == 'linux/amd64' && 'amd64' || 'arm64' }}
echo "image=${{ inputs.image_name }}:${{inputs.version}}-$ARCH" >> $GITHUB_OUTPUT
echo "safe_platform=${{ matrix.platform == 'linux/amd64' && 'linux-amd64' || 'linux-arm64' }}" >> $GITHUB_OUTPUT
- name: Build and push image
uses: docker/build-push-action@v6
@@ -256,72 +69,71 @@ jobs:
file: ${{ inputs.dockerfile }}
platforms: ${{ matrix.platform }}
push: true
tags: ${{ steps.set-tags.outputs.image }}
tags: ${{ steps.prep.outputs.image }}
target: ${{ inputs.target }}
- name: Prepare artifact name
id: artifact
run: |
platform=${{ matrix.platform }}
echo "safe_platform=${platform//\//-}" >> $GITHUB_OUTPUT
echo "${{ steps.set-tags.outputs.image }}" > image.txt
- name: Save image name for manifest
run: echo "${{ steps.prep.outputs.image }}" > image.txt
- uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f
- uses: actions/upload-artifact@v4
with:
name: image-${{ steps.artifact.outputs.safe_platform }}
name: image-${{ steps.prep.outputs.safe_platform }}
path: image.txt
if-no-files-found: warn
compression-level: 6
overwrite: false
include-hidden-files: false
retention-days: 1
create-manifest:
name: Create multi-arch Docker manifest
name: Create multi-arch manifest
runs-on: ubuntu-latest
needs: build
steps:
- uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131
- uses: actions/download-artifact@v4
with:
name: image-linux-amd64
path: /tmp/digests/amd64
- uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131
- uses: actions/download-artifact@v4
with:
name: image-linux-arm64
path: /tmp/digests/arm64
- name: Login to Docker
uses: docker/login-action@f4ef78c080cd8ba55a85445d5b36e214a81df20a
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
# - name: Create and push manifest
# run: |
# AMD64_IMAGE=$(cat /tmp/digests/amd64/image.txt)
# ARM64_IMAGE=$(cat /tmp/digests/arm64/image.txt)
# REF_NAME=${GITHUB_REF#refs/tags/}
# MANIFEST_IMAGE="${{ inputs.image_name }}:$REF_NAME"
#
# docker manifest create $MANIFEST_IMAGE $AMD64_IMAGE $ARM64_IMAGE
# docker manifest push $MANIFEST_IMAGE
#
# if [ "${{ inputs.add_latest }}" = "true" ]; then
# docker manifest create ${{ inputs.image_name }}:latest $AMD64_IMAGE $ARM64_IMAGE
# docker manifest push ${{ inputs.image_name }}:latest
# fi
- name: Generate semantic tags
id: tags
run: |
VERSION="${{ inputs.version }}"
IFS='.' read -r MAJOR MINOR PATCH <<< "$VERSION"
echo "VERSION_TAG=$VERSION" >> $GITHUB_OUTPUT
if [ -n "$MINOR" ]; then
echo "MINOR_TAG=$MAJOR.$MINOR" >> $GITHUB_OUTPUT
fi
if [ -n "$MAJOR" ]; then
echo "MAJOR_TAG=$MAJOR" >> $GITHUB_OUTPUT
fi
if [ "${{ inputs.add_latest }}" = "true" ]; then
echo "LATEST_TAG=latest" >> $GITHUB_OUTPUT
fi
- name: Extract Docker metadata
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ inputs.image_name }}
tags: |
type=raw,value=${{ steps.tags.outputs.VERSION_TAG }}
type=raw,value=${{ steps.tags.outputs.MINOR_TAG }}
type=raw,value=${{ steps.tags.outputs.MAJOR_TAG }}
type=raw,value=${{ steps.tags.outputs.LATEST_TAG }}
- name: Create and push manifest list
working-directory: /tmp/digests
run: |
DOCKER_IMAGES="$(cat amd64/image.txt) $(cat arm64/image.txt)"
REF_NAME=${GITHUB_REF#refs/tags/}
MANIFEST_IMAGE="${{ inputs.image_name }}:$REF_NAME"
docker buildx imagetools create $DOCKER_IMAGES -t $MANIFEST_IMAGE
docker buildx imagetools inspect $MANIFEST_IMAGE
if [ "${{ inputs.add_latest }}" = "true" ]; then
docker buildx imagetools create $DOCKER_IMAGES -t ${{ inputs.image_name }}:latest
docker buildx imagetools inspect ${{ inputs.image_name }}:latest
fi
TAG_ARGS=$(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON")
echo $TAG_ARGS
docker buildx imagetools create $TAG_ARGS $DOCKER_IMAGES
-128
View File
@@ -1,128 +0,0 @@
name: GitHub Release
on:
workflow_call:
inputs:
prerelease:
required: false
type: boolean
default: false
make_latest:
required: false
type: boolean
default: false
discord_title:
required: true
type: string
discord_color:
required: true
type: number
discord_footer:
required: true
type: string
secrets:
DISCORD_WEBHOOK:
required: true
GH_TOKEN:
required: true
jobs:
create-release:
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- name: Check out the repo
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Build Changelog
id: build_changelog
uses: mikepenz/release-changelog-builder-action@v5
with:
mode: "COMMIT"
configurationJson: |
{
"template": "#{{CHANGELOG}}",
"categories": [
{
"title": "## Feature",
"labels": ["feat", "feature"]
},
{
"title": "## Fix",
"labels": ["fix", "bug"]
},
{
"title": "## Other",
"labels": []
}
],
"label_extractor": [
{
"pattern": "^(build|chore|ci|docs|feat|fix|perf|refactor|revert|style|test){1}(\([\w\-\.]+\))?(!)?: ([\w ])+([\s\S]*)",
"on_property": "title",
"target": "$1"
}
]
}
env:
GITHUB_TOKEN: ${{ secrets.GH_TOKEN }}
- name: Create GitHub Release
uses: softprops/action-gh-release@v2
with:
generate_release_notes: false
body: ${{ steps.build_changelog.outputs.changelog }}
prerelease: ${{ inputs.prerelease }}
make_latest: ${{ inputs.make_latest }}
env:
GITHUB_TOKEN: ${{ secrets.GH_TOKEN }}
- name: Send Discord Notification
env:
DISCORD_WEBHOOK: ${{ secrets.DISCORD_WEBHOOK }}
GH_TOKEN: ${{ secrets.GH_TOKEN }}
run: |
RELEASE_INFO=$(gh release view "${{ github.ref_name }}" -R ${{ github.repository }} --json name,url,body,author)
RELEASE_TITLE=$(echo "$RELEASE_INFO" | jq -r .name)
if [ -z "$RELEASE_TITLE" ] || [ "$RELEASE_TITLE" = "null" ]; then RELEASE_TITLE="${{ github.ref_name }}"; fi
RELEASE_URL=$(echo "$RELEASE_INFO" | jq -r .url)
RELEASE_BODY=$(echo "$RELEASE_INFO" | jq -r .body)
AUTHOR_NAME="Portabase"
AUTHOR_ICON="https://github.com/Portabase.png"
PAYLOAD=$(jq -n \
--arg title "$RELEASE_TITLE" \
--arg description "$RELEASE_BODY" \
--arg url "$RELEASE_URL" \
--arg author "$AUTHOR_NAME" \
--arg icon "$AUTHOR_ICON" \
--arg discord_title "${{ inputs.discord_title }}" \
--arg discord_footer "${{ inputs.discord_footer }}" \
--argjson discord_color ${{ inputs.discord_color }} \
'{
content: $discord_title,
embeds: [{
title: $title,
url: $url,
description: $description,
color: $discord_color,
author: {
name: $author,
icon_url: $icon
},
footer: {
text: $discord_footer
}
}]
}'
)
curl -H "Content-Type: application/json" \
-d "$PAYLOAD" \
"$DISCORD_WEBHOOK"
+37
View File
@@ -0,0 +1,37 @@
name: Publish Helm Chart
on:
workflow_call:
inputs:
version:
required: true
type: string
secrets:
GH_TOKEN:
required: true
jobs:
publish-helm:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Set up Helm
uses: azure/setup-helm@v4
- name: Package Helm chart
run: |
mkdir -p ./helm-packages
helm package helm \
--version ${{ inputs.version }} \
--app-version ${{ inputs.version }} \
--destination ./helm-packages
- name: Authenticate to GitHub Packages
run: |
echo "${{ secrets.GH_TOKEN }}" | helm registry login ghcr.io -u ${{ github.actor }} --password-stdin
- name: Push Helm chart to GitHub Packages (OCI)
run: |
helm push ./helm-packages/portabase-agent-${{ inputs.version }}.tgz oci://ghcr.io/portabase/charts
+128 -12
View File
@@ -1,32 +1,148 @@
name: Publish Docker image for release
name: Auto Release & Publish
on:
push:
tags:
- '*.*.*'
- '!*-*'
pull_request:
types: [ closed ]
branches:
- main
permissions:
contents: write
packages: write
jobs:
docker_publish:
check-skip:
runs-on: ubuntu-latest
outputs:
skip: ${{ steps.set-skip.outputs.skip }}
steps:
- name: Determine if release should be skipped
id: set-skip
run: |
TITLE="${{ github.event.pull_request.title }}"
echo "PR title: $TITLE"
if [[ "$TITLE" == *"[skip-release]"* ]]; then
echo "PR title contains [skip-release], skipping release jobs."
echo "skip=true" >> $GITHUB_OUTPUT
else
echo "skip=false" >> $GITHUB_OUTPUT
fi
create-release:
needs: check-skip
if: ${{ needs.check-skip.outputs.skip == 'false' }}
runs-on: ubuntu-latest
outputs:
draft_tag: ${{ steps.release_step.outputs.draft_tag }}
version: ${{ steps.release_step.outputs.version }}
steps:
- uses: actions/create-github-app-token@v1
id: app-token
with:
app-id: ${{ vars.APP_ID }}
private-key: ${{ secrets.APP_PRIVATE_KEY }}
- uses: actions/checkout@v4
with:
fetch-depth: 0
token: ${{ steps.app-token.outputs.token }}
ref: main
- name: Setup Rust
uses: dtolnay/rust-toolchain@stable
- name: Install cargo-edit
run: cargo install cargo-edit
- uses: actions/setup-node@v4
with:
node-version: "lts/*"
- name: Install release-it globally
run: |
npm install -g release-it
npm install -g @release-it/conventional-changelog
npm install -g @release-it/bumper
- run: |
git config --global user.name 'github-actions[bot]'
git config --global user.email 'github-actions[bot]@users.noreply.github.com'
- name: Run release-it
id: release_step
run: |
git pull origin main
VERSION=$(release-it --ci --release-version)
echo $VERSION
echo "version=$VERSION" >> $GITHUB_OUTPUT
OUTPUT=$(release-it --ci)
echo "$OUTPUT"
DRAFT_TAG=$(echo "$OUTPUT" | grep -oE 'untagged-[a-z0-9]+')
echo $DRAFT_TAG
echo "draft_tag=$DRAFT_TAG" >> $GITHUB_OUTPUT
env:
GITHUB_TOKEN: ${{ steps.app-token.outputs.token }}
publish-docker:
needs: create-release
if: ${{ needs.create-release.result == 'success' }}
uses: ./.github/workflows/docker.yml
with:
version: ${{ needs.create-release.outputs.version }}
ref: ${{ needs.create-release.outputs.version }}
add_latest: true
secrets:
DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
github_release:
needs: docker_publish
uses: ./.github/workflows/github.yml
publish-helm:
needs: create-release
if: ${{ needs.create-release.result == 'success' }}
uses: ./.github/workflows/helm.yml
with:
prerelease: false
make_latest: true
version: ${{ needs.create-release.outputs.version }}
secrets:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
finalize-release:
needs:
- create-release
- publish-docker
- publish-helm
runs-on: ubuntu-latest
outputs:
release_tag: ${{ steps.publish_release_step.outputs.release_tag }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Publish GitHub Release
id: publish_release_step
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
OUTPUT=$(gh release edit ${{ needs.create-release.outputs.draft_tag }} --draft=false)
echo "$OUTPUT"
RELEASE_TAG=$(echo "$OUTPUT" | sed -E 's|.*/releases/tag/||')
echo "release_tag=$RELEASE_TAG" >> $GITHUB_OUTPUT
notify-discord:
needs:
- publish-docker
- publish-helm
- create-release
- finalize-release
uses: ./.github/workflows/discord.yml
with:
release_tag: ${{ needs.create-release.outputs.version }}
discord_title: "||@release-agent|| New release published"
discord_color: 5814783
discord_color: 3066993
discord_footer: "Portabase"
secrets:
DISCORD_WEBHOOK: ${{ secrets.DISCORD_WEBHOOK }}
+62
View File
@@ -0,0 +1,62 @@
{
"github": {
"release": true,
"draft": true,
"tokenRef": "GITHUB_TOKEN"
},
"git": {
"commit": true,
"commitMessage": "chore: release ${version}",
"requireCleanWorkingDir": true,
"tag": true,
"tagName": "${version}",
"push": true
},
"hooks": {
"before:bump": "cargo set-version ${version}"
},
"plugins": {
"@release-it/conventional-changelog": {
"preset": {
"name": "conventionalcommits",
"types": [
{
"type": "feat",
"section": "✨ Features"
},
{
"type": "fix",
"section": "🐛 Bug Fixes"
},
{
"type": "perf",
"section": "⚡️ Performance Improvements"
},
{
"type": "revert",
"section": "⏪️ Reverts"
},
{
"type": "docs",
"section": "📝 Documentation",
"hidden": true
},
{
"type": "chore",
"section": "🔧 Chores",
"hidden": true
}
]
}
},
"@release-it/bumper": {
"out": [
{
"file": "CITATION.cff",
"path": "version",
"type": "text/yaml"
}
]
}
}
}
+9 -4
View File
@@ -1,6 +1,6 @@
cff-version: 1.2.0
title: Portabase Agent (Rust)
message: "If you use this software, please cite it as below."
message: If you use this software, please cite it as below.
type: software
authors:
- family-names: Gauthereau
@@ -9,7 +9,12 @@ authors:
given-names: Killian
repository-code: https://github.com/Portabase/agent-rust
url: https://portabase.io
abstract: "Portabase is a free, open-source, self-hosted solution for database administration, providing backup and restore capabilities, scheduling, retention policies, notifications, and support for multiple storage backends. Its headless agent architecture enables connection to multiple database instances securely and efficiently."
abstract: >-
Portabase is a free, open-source, self-hosted solution for database
administration, providing backup and restore capabilities, scheduling,
retention policies, notifications, and support for multiple storage backends.
Its headless agent architecture enables connection to multiple database
instances securely and efficiently.
keywords:
- database
- administration
@@ -22,5 +27,5 @@ keywords:
- self-hosted
- portabase
license: Apache-2.0
version: 1.0.4-rc.20
date-released: "2026-01-28"
version: 1.5.1
date-released: '2026-02-24'
Generated
+2505 -281
View File
File diff suppressed because it is too large Load Diff
+32 -4
View File
@@ -1,6 +1,6 @@
[package]
name = "portabase-agent"
version = "1.0.4-rc.20"
version = "1.5.1"
edition = "2024"
[dependencies]
@@ -17,7 +17,7 @@ base64 = "0.22.1"
thiserror = "2.0.17"
log = "0.4.29"
toml = "0.9.10"
reqwest = { version = "0.13.1", features = ["json", "blocking", "multipart"] }
reqwest = { version = "0.13.1", features = ["json", "blocking", "multipart", "stream", "query"] }
anyhow = "1.0.100"
tokio = { version = "1.49.0", features = ["rt", "rt-multi-thread", "macros", "fs"] }
async-trait = "0.1.89"
@@ -28,11 +28,39 @@ flate2 = "1.1.5"
tar = "0.4.44"
tokio-postgres = "0.7.15"
futures = "0.3.31"
tracing-log = "0.2.0"
tracing-appender = "0.2.4"
time = { version = "0.3.44", features = ["macros"] }
mongodb = "3.5.0"
rand = "0.9.2"
bytes = "1.11.0"
async-stream = "0.3.6"
uuid = { version = "1.20.0", features = ["v4"] }
tokio-util = "0.7.18"
aws-config = "1.8.13"
aws-sdk-s3 = { version = "1.122.0", features = ["behavior-version-latest"] }
async-compression = { version = "0.4.37", features = ["tokio", "gzip"] }
tokio-tar = "0.3.1"
oauth2 = "5.0.0"
hyper = "1.8.1"
aes-gcm = "0.11.0-rc.3"
generic-array = "0.14.7"
futures-util = "0.3.31"
tokio-stream = "0.1.18"
aes = "0.9.0-rc.4"
typenum = "1.19.0"
testcontainers = "0.27.1"
testcontainers-modules = { version = "0.15.0", features = ["postgres", "redis"] }
postgres = "0.19.12"
url = "2.5.8"
[dev-dependencies]
tokio = { version = "1", features = ["full"] }
mockall = "0.13"
testcontainers = "0.27.1"
testcontainers-modules = { version = "0.15.0", features = ["postgres", "redis"] }
wiremock = "0.6"
[[bin]]
name = "app"
path = "src/main.rs"
path = "src/main.rs"
+25
View File
@@ -1,8 +1,12 @@
include .env
export $(shell sed 's/=.*//' .env)
CLUSTER_SCRIPT=docker/entrypoints/app-dev-entrypoint.sh
.PHONY: seed-mongo seed-mysql seed-postgres
up:
@bash $(CLUSTER_SCRIPT)
seed-mongo:
@echo "Seeding MongoDB..."
bash ./scripts/mongo/seed-mongo.sh
@@ -15,6 +19,10 @@ seed-mysql:
@echo "Seeding MySQL..."
mysql -h 127.0.0.1 -P "$$MYSQL_PORT" -u "$$MYSQL_USER" -p"$$MYSQL_PASSWORD" "$$MYSQL_DB" < ./scripts/mysql/seed-mysql.sql
seed-mysql-1gb:
@echo "Seeding MySQL..."
mysql -h 127.0.0.1 -P "$$MYSQL_PORT" -u "$$MYSQL_USER" -p"$$MYSQL_PASSWORD" "$$MYSQL_DB" < ./scripts/mysql/seed-1gb.sql
seed-postgres:
@echo "Seeding Postgres..."
@@ -26,4 +34,21 @@ seed-postgres-1gb:
docker exec -i -e PGPASSWORD=$$PG_PASSWORD $$PG_CONTAINER \
psql -U $$PG_USER -d $$PG_DB < ./scripts/postgres/seed-1gb.sql
SQLITE_SEED_FILE := $(if $(filter big,$(SEED)),./scripts/sqlite/seed-big.sql,./scripts/sqlite/seed.sql)
seed-sqlite:
@echo "Seeding Sqlite..."
@echo "Run as root to fix permissions inside the volume"
docker exec -u 0 -it db-sqlite sh -c "chmod -R 777 /workspace/data"
@echo "Create the database file (if it doesnt exist)"
docker exec -u 0 -it db-sqlite sh -c "touch /workspace/data/app.db"
@echo "Seed the database"
docker exec -i db-sqlite sh -c "sqlite3 /workspace/data/app.db" < $(SQLITE_SEED_FILE)
@echo "Verify"
docker exec -it db-sqlite sqlite3 /workspace/data/app.db "SELECT name FROM sqlite_master WHERE type='table';"
@echo "Done"
seed-all: seed-mongo seed-mysql seed-postgres seed-postgres-1gb
+3 -1
View File
@@ -16,9 +16,9 @@
[![Platform](https://img.shields.io/badge/platform-linux%20%7C%20macos%20%7C%20windows-lightgrey)](https://github.com/Portabase/portabase)
[![Support Portabase](https://img.shields.io/badge/Support-Portabase-orange)](https://www.buymeacoffee.com/portabase)
[![PostgreSQL](https://img.shields.io/badge/python-3.13-blue.svg)](https://www.python.org/downloads/release/python-3120/)
[![PostgreSQL](https://img.shields.io/badge/PostgreSQL-336791?logo=postgresql&logoColor=white)](https://www.postgresql.org/)
[![MySQL](https://img.shields.io/badge/MySQL-4479A1?logo=mysql&logoColor=white)](https://www.mysql.com/)
[![SQLite](https://img.shields.io/badge/-SQLite-blue?logo=sqlite&logoColor=white)](https://sqlite.org/)
[![MariaDB](https://img.shields.io/badge/MariaDB-003545?logo=mariadb&logoColor=white)](https://mariadb.org/)
[![MongoDB](https://img.shields.io/badge/-MongoDB-13aa52?logo=mongodb&logoColor=white)](https://www.mongodb.com/)
[![Self Hosted](https://img.shields.io/badge/self--hosted-yes-brightgreen)](https://github.com/Portabase/portabase)
@@ -79,3 +79,5 @@ Distributed under the Apache License. See `LICENSE.txt` for more information.
[Docker-url]: https://www.docker.com/
+28 -2
View File
@@ -34,11 +34,37 @@
"name": "Test database 5 - MongoDB",
"database": "testdb",
"type": "mongodb",
"username": "",
"password": "",
"port": 27017,
"host": "db-mongodb",
"generated_id": "16678147-ff7e-4c97-8c83-0adeff214681"
},
{
"name": "Test database 6 - SQLite DB",
"type": "sqlite",
"path": "/sqlite-data/workspace/data/app.db",
"generated_id": "16678178-ff7e-4c97-8c83-0adeff214681"
},
{
"name": "Test database 7 - SQLite DB",
"type": "sqlite",
"path": "/sqlite-data-2/workspace/data/app.db",
"generated_id": "16678179-ff7e-4c97-8c83-0adeff214681"
},
{
"name": "Test database 8 - Redis",
"type": "redis",
"port": 6379,
"host": "db-redis",
"generated_id": "16678166-ff7e-4c97-8c83-0adeff214681"
},
{
"name": "Test database 9 - Redis Auth",
"type": "redis",
"password": "m6o2L0Ukw3Og5RJUKh",
"port": 6379,
"username": "default",
"host": "db-redis-auth",
"generated_id": "16678160-ff7e-4c97-8c83-0adeff214681"
}
]
}
+49 -7
View File
@@ -1,11 +1,11 @@
services:
rust-app:
build:
context: .
dockerfile: docker/Dockerfile
target: prod
# image: portabase/agent:latest
platform: linux/arm64
# build:
# context: .
# dockerfile: docker/Dockerfile
# target: prod
image: portabase/agent:latest
# platform: linux/arm64
container_name: rust-prod
volumes:
- ./databases.json:/config/config.json
@@ -14,7 +14,7 @@ services:
LOG: info
TZ: "Europe/Paris"
# DATABASES_CONFIG_FILE: "config.toml"
EDGE_KEY: "eyJzZXJ2ZXJVcmwiOiJodHRwOi8vbG9jYWxob3N0Ojg4ODciLCJhZ2VudElkIjoiY2VlZmNmNDQtOGE0YS00NjZlLTkwNDEtN2QzNDMzZjRjOTJkIiwicHVibGljS2V5IjoiLS0tLS1CRUdJTiBSU0EgUFVCTElDIEtFWS0tLS0tXG5NSUlCQ2dLQ0FRRUFuYUVKdnVBSExlNGF5d0RmNGplelVobm45VXRkdksyZ3pEMEg2cERJYXczYkJKRkpwVnVDXG5uVFV3MXA3Q2RnOXBzdjZhRnpyOXZPd0J2MjMzckxpdVpCT2lCb2p2Q0QrSlZid3hyTzBRRW5hN2dmaHV1ZGYwXG5VVlJOMkxmK1g1aTkvZzJTNm5xcExoTm1DaGFJNk8ybktYZUNlRmtubEErRUJrNnFoV1FCVGozb05TYTFTOFY1XG40UFRTT2I4NUo3a2k5YllEbXRiNWxrU3dCNXdXOTdtQjg0ZzI2WHAvU3FFcmhKc0NGK3YrN09vTWYzTzJqTTNoXG5XMUQ0MzBPRitWaklwUGdoV09rZy96NXZQUWFHRzhqQ0h4VDlJR0Q0bjhyS05LQ3FTOGNyN2diTGU0cWpNdmhvXG5BQVVvaHpHR2FRNkhlWlJ4S0UvM3J1a2JldnY5dnJ2TTNRSURBUUFCXG4tLS0tLUVORCBSU0EgUFVCTElDIEtFWS0tLS0tXG4ifQ"
EDGE_KEY: "eyJzZXJ2ZXJVcmwiOiJodHRwOi8vbG9jYWxob3N0Ojg4ODciLCJhZ2VudElkIjoiOGZmMDE4NTQtYjJhMS00ZTE0LTkwMjctZTJiOWIxZjQ1YzdlIiwicHVibGljS2V5IjoiLS0tLS1CRUdJTiBSU0EgUFVCTElDIEtFWS0tLS0tXG5NSUlCQ2dLQ0FRRUE5TWV4M2pmdnVLdFB5YU1ERnh2Ulp2dmd3YkRJQ2JzQi81Wll5NDNSVVRBaXZRYjJiSDdYXG5qRHBQd1lJeCs4UFBrbHlRbDVMQzV1UWZEaCs4SVd4OG1LZ3FvMXpWMkdiZXdGbEdEWFYxVEdyU1ZEU25aSWR4XG52bWdYc29EeXhVMlJvWUFUMS9YMWxuc2YxenZKdkFMTkhXdEhRdk42SjVDZTFSMmFsendVRGFEVXlJNzRmSldQXG5tNTh0SDMrYklXL0VVTXdjaWNxM0oySWw3Vm9KNkZNUHJQL1ZSOWEvdFF1SU1qa200MXpFY2NscExPa2luRkxuXG54NmVUWkFSZUpya2UrbnRvZ2t4TGEyRWV5a1lUNzB4V3hKNWp5ZExBVnRvNkkyQlVLVVJoTkowTUFaU29NYUtvXG5iMGJRcnY1UzExZWllMnMrT2I3aTYzSFpkVUx0UmV1MVJ3SURBUUFCXG4tLS0tLUVORCBSU0EgUFVCTElDIEtFWS0tLS0tXG4ifQ"
extra_hosts:
- "localhost:host-gateway"
networks:
@@ -22,6 +22,48 @@ services:
db-mongodb-auth:
container_name: db-mongodb-auth
image: mongo:latest
ports:
- "27082:27017"
environment:
MONGO_INITDB_ROOT_USERNAME: root
MONGO_INITDB_ROOT_PASSWORD: rootpassword
MONGO_INITDB_DATABASE: testdbauth
command: mongod --auth
networks:
- portabase
volumes:
- mongodb-data-auth:/data/db
healthcheck:
test: [ "CMD", "mongo", "--eval", "db.adminCommand('ping')" ]
interval: 5s
timeout: 5s
retries: 10
db-mongodb:
container_name: db-mongodb
image: mongo:latest
ports:
- "27083:27017"
volumes:
- mongodb-data:/data/db
healthcheck:
test: [ "CMD", "mongosh", "--eval", "db.adminCommand('ping')" ]
interval: 5s
timeout: 5s
retries: 10
environment:
MONGO_INITDB_DATABASE: testdb
networks:
- portabase
volumes:
mongodb-data:
mongodb-data-auth:
networks:
portabase:
name: portabase_network
+102 -65
View File
@@ -11,12 +11,14 @@ services:
# - ./databases.toml:/config/config.toml
- cargo-registry:/usr/local/cargo/registry
- cargo-git:/usr/local/cargo/git
- cargo-target:/app/target
# - cargo-target:/app/target
# - sqlite-data:/sqlite-data/workspace/data
# - ./scripts/sqlite/test-db:/sqlite-data-2/workspace/data
environment:
APP_ENV: development
LOG: debug
TZ: "Europe/Paris"
EDGE_KEY: "eyJzZXJ2ZXJVcmwiOiJodHRwOi8vbG9jYWxob3N0Ojg4ODciLCJhZ2VudElkIjoiOGZmMDE4NTQtYjJhMS00ZTE0LTkwMjctZTJiOWIxZjQ1YzdlIiwicHVibGljS2V5IjoiLS0tLS1CRUdJTiBSU0EgUFVCTElDIEtFWS0tLS0tXG5NSUlCQ2dLQ0FRRUE5TWV4M2pmdnVLdFB5YU1ERnh2Ulp2dmd3YkRJQ2JzQi81Wll5NDNSVVRBaXZRYjJiSDdYXG5qRHBQd1lJeCs4UFBrbHlRbDVMQzV1UWZEaCs4SVd4OG1LZ3FvMXpWMkdiZXdGbEdEWFYxVEdyU1ZEU25aSWR4XG52bWdYc29EeXhVMlJvWUFUMS9YMWxuc2YxenZKdkFMTkhXdEhRdk42SjVDZTFSMmFsendVRGFEVXlJNzRmSldQXG5tNTh0SDMrYklXL0VVTXdjaWNxM0oySWw3Vm9KNkZNUHJQL1ZSOWEvdFF1SU1qa200MXpFY2NscExPa2luRkxuXG54NmVUWkFSZUpya2UrbnRvZ2t4TGEyRWV5a1lUNzB4V3hKNWp5ZExBVnRvNkkyQlVLVVJoTkowTUFaU29NYUtvXG5iMGJRcnY1UzExZWllMnMrT2I3aTYzSFpkVUx0UmV1MVJ3SURBUUFCXG4tLS0tLUVORCBSU0EgUFVCTElDIEtFWS0tLS0tXG4ifQ"
EDGE_KEY: "eyJzZXJ2ZXJVcmwiOiJodHRwOi8vbG9jYWxob3N0Ojg4ODciLCJhZ2VudElkIjoiZWE5OWE5ZjMtNDhkYy00MjMyLTkzMTAtNTc5YWVkY2IyZjhmIiwibWFzdGVyS2V5QjY0IjoiQlhWM1hvbEM2NTZTVjdkTmdjV1BHUWxrKytycExJNmxHRGk3Q1BCNWllbz0ifQ=="
#POOLING: 1
#DATABASES_CONFIG_FILE: "config.toml"
extra_hosts:
@@ -24,87 +26,122 @@ services:
networks:
- portabase
# db-postgres:
# container_name: db-postgres
# image: postgres:17-alpine
# ports:
# - "5436:5432"
# volumes:
# - postgres-data:/var/lib/postgresql/data
# environment:
# - POSTGRES_DB=devdb
# - POSTGRES_USER=devuser
# - POSTGRES_PASSWORD=changeme
# networks:
# - portabase
#
# db-mariadb:
# container_name: db-mariadb
# image: mariadb:latest
# ports:
# - "3311:3306"
# environment:
# - MYSQL_DATABASE=mariadb
# - MYSQL_USER=mariadb
# - MYSQL_PASSWORD=changeme
# - MYSQL_RANDOM_ROOT_PASSWORD=yes
# volumes:
# - mariadb-data:/var/lib/mysql
# networks:
# - portabase
db-mongodb-auth:
container_name: db-mongodb-auth
image: mongo:latest
db-postgres:
container_name: db-postgres
image: postgres:17-alpine
ports:
- "27082:27017"
- "5436:5432"
volumes:
- postgres-data:/var/lib/postgresql/data
environment:
MONGO_INITDB_ROOT_USERNAME: root
MONGO_INITDB_ROOT_PASSWORD: rootpassword
MONGO_INITDB_DATABASE: testdbauth
command: mongod --auth
- POSTGRES_DB=devdb
- POSTGRES_USER=devuser
- POSTGRES_PASSWORD=changeme
networks:
- portabase
volumes:
- mongodb-data-auth:/data/db
healthcheck:
test: [ "CMD", "mongo", "--eval", "db.adminCommand('ping')" ]
interval: 5s
timeout: 5s
retries: 10
#
# db-mariadb:
# container_name: db-mariadb
# image: mariadb:latest
# ports:
# - "3311:3306"
# environment:
# - MYSQL_DATABASE=mariadb
# - MYSQL_USER=mariadb
# - MYSQL_PASSWORD=changeme
# - MYSQL_RANDOM_ROOT_PASSWORD=yes
# volumes:
# - mariadb-data:/var/lib/mysql
# networks:
# - portabase
#
#
# db-mongodb-auth:
# container_name: db-mongodb-auth
# image: mongo:latest
# ports:
# - "27082:27017"
# environment:
# MONGO_INITDB_ROOT_USERNAME: root
# MONGO_INITDB_ROOT_PASSWORD: rootpassword
# MONGO_INITDB_DATABASE: testdbauth
# command: mongod --auth
# networks:
# - portabase
# volumes:
# - mongodb-data-auth:/data/db
# healthcheck:
# test: [ "CMD", "mongo", "--eval", "db.adminCommand('ping')" ]
# interval: 5s
# timeout: 5s
# retries: 10
#
# db-mongodb:
# container_name: db-mongodb
# image: mongo:latest
# ports:
# - "27083:27017"
# volumes:
# - mongodb-data:/data/db
# healthcheck:
# test: [ "CMD", "mongosh", "--eval", "db.adminCommand('ping')" ]
# interval: 5s
# timeout: 5s
# retries: 10
# environment:
# MONGO_INITDB_DATABASE: testdb
# networks:
# - portabase
db-mongodb:
container_name: db-mongodb
image: mongo:latest
# sqlite:
# container_name: db-sqlite
# image: keinos/sqlite3
# volumes:
# - sqlite-data:/workspace/data
# working_dir: /workspace
# command: tail -f /dev/null
# stdin_open: true
# tty: true
db-redis:
image: redis:latest
container_name: db-redis
ports:
- "27083:27017"
- "6379:6379"
volumes:
- mongodb-data:/data/db
healthcheck:
test: [ "CMD", "mongosh", "--eval", "db.adminCommand('ping')" ]
interval: 5s
timeout: 5s
retries: 10
- redis-data:/data
command: [ "redis-server", "--appendonly", "yes" ]
networks:
- portabase
db-redis-auth:
image: redis:latest
container_name: db-redis-auth
ports:
- "6380:6379"
volumes:
- redis-data-auth:/data
environment:
MONGO_INITDB_DATABASE: testdb
- REDIS_PASSWORD=m6o2L0Ukw3Og5RJUKh
command: [ "redis-server", "--requirepass", "m6o2L0Ukw3Og5RJUKh", "--appendonly", "yes" ]
networks:
- portabase
volumes:
cargo-registry:
cargo-git:
cargo-target:
# cargo-target:
# postgres-data:
# mariadb-data:
mongodb-data:
mongodb-data-auth:
postgres-data:
# mariadb-data:
# mongodb-data:
# mongodb-data-auth:
# sqlite-data:
redis-data:
redis-data-auth:
networks:
portabase:
name: portabase_network
external: true
# docker network create portabase_network
+5 -125
View File
@@ -1,7 +1,7 @@
# =========================
# Base image (shared)
# =========================
FROM rust:1.92.0 AS base
FROM rust:1.94.0 AS base
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \
pkg-config \
@@ -15,6 +15,8 @@ RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \
zlib1g \
curl \
mariadb-client \
sqlite3 \
redis-tools \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
@@ -58,7 +60,6 @@ FROM base AS dev
RUN cargo install cargo-watch
# Pre-cache dependencies
COPY Cargo.toml Cargo.lock ./
RUN mkdir src && echo "fn main() {}" > src/main.rs
RUN cargo build
@@ -93,6 +94,8 @@ RUN apt-get update && apt-get install -y \
libncurses6 \
zlib1g \
mariadb-client \
sqlite3 \
redis-tools \
&& rm -rf /var/lib/apt/lists/*
@@ -110,126 +113,3 @@ COPY --from=base /usr/local/mongodb/bin/ /usr/local/mongodb/bin/
ENV APP_ENV=production
CMD ["/entrypoint.sh"]
# =========================
# Base image (shared)
# =========================
#FROM rust:1.92.0 AS base
#
#RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \
# pkg-config \
# libssl-dev \
# tzdata \
# redis-server \
# ca-certificates \
# libpq5 \
# libreadline8 \
# libncurses6 \
# zlib1g \
# curl \
# mariadb-client \
# && apt-get clean \
# && rm -rf /var/lib/apt/lists/*
#
#ARG TARGETARCH
#
## =========================
## PostgreSQL client binaries (versions 12-18)
## =========================
#RUN for v in 12 13 14 15 16 17 18; do mkdir -p /usr/lib/postgresql/$v/bin; done
#
#COPY assets/tools/amd64/postgresql/ /tmp/pg-x64/
#COPY assets/tools/arm64/postgresql/ /tmp/pg-arm/
#
#RUN if [ "$TARGETARCH" = "amd64" ]; then \
# for v in 12 13 14 15 16 17 18; do \
# cp -r /tmp/pg-x64/postgresql-$v/bin/* /usr/lib/postgresql/$v/bin/; \
# done; \
# else \
# for v in 12 13 14 15 16 17 18; do \
# cp -r /tmp/pg-arm/postgresql-$v/bin/* /usr/lib/postgresql/$v/bin/; \
# done; \
# fi && \
# rm -rf /tmp/pg-x64 /tmp/pg-arm && \
# chmod +x /usr/lib/postgresql/*/bin/*
#
## =========================
## MongoDB client binaries
## =========================
#COPY assets/tools/${TARGETARCH}/mongodb/ /usr/local/mongodb/
#RUN chmod +x /usr/local/mongodb/bin/*
#
#WORKDIR /app
#
## =========================
## Development image
## =========================
#FROM base AS dev
#
#RUN cargo install cargo-watch
#
#COPY Cargo.toml Cargo.lock ./
#RUN mkdir src && echo "fn main() {}" > src/main.rs
#RUN cargo build
#RUN rm -rf src
#
#COPY entrypoint.sh /entrypoint.sh
#RUN chmod +x /entrypoint.sh
#
#CMD ["/entrypoint.sh"]
#
## =========================
## Builder (production)
## =========================
#FROM base AS builder
#
#ARG TARGETARCH
#
#RUN rustup target add x86_64-unknown-linux-gnu aarch64-unknown-linux-gnu
#
#COPY . .
#
#RUN if [ "$TARGETARCH" = "amd64" ]; then \
# cargo build --release --target x86_64-unknown-linux-gnu && \
# mv /app/target/x86_64-unknown-linux-gnu/release /app/target/release-app; \
# else \
# cargo build --release --target aarch64-unknown-linux-gnu && \
# mv /app/target/aarch64-unknown-linux-gnu/release /app/target/release-app; \
# fi
#
#
#RUN echo "APP_VERSION=$(cargo pkgid | awk -F# '{print $2}')" > /app/version.env
#
## =========================
## Runtime (production)
## =========================
#FROM ubuntu:24.04 AS prod
#
#RUN apt-get update && apt-get install -y \
# redis-server \
# ca-certificates \
# tzdata \
# libpq5 \
# libreadline8 \
# libncurses6 \
# zlib1g \
# mariadb-client \
# && rm -rf /var/lib/apt/lists/*
#
#WORKDIR /app
#
#ARG TARGETARCH
#
#COPY --from=builder /app/target/release-app/app /usr/local/bin/app
#
#COPY --from=builder /app/version.env /app/version.env
#COPY entrypoint.sh /entrypoint.sh
#RUN chmod +x /entrypoint.sh
#
#COPY --from=base /usr/lib/postgresql/ /usr/lib/postgresql/
#COPY --from=base /usr/local/mongodb/bin/ /usr/local/mongodb/bin/
#
#ENV APP_ENV=production
#
#CMD ["/entrypoint.sh"]
+39
View File
@@ -0,0 +1,39 @@
#!/bin/bash
set -euo pipefail
check_docker() {
if ! docker info > /dev/null 2>&1; then
echo "Docker is not running. Attempting to start Docker..."
if [[ "$OSTYPE" == "darwin"* ]]; then
open -a Docker
echo "Waiting for Docker to start..."
until docker info > /dev/null 2>&1; do
sleep 2
done
elif command -v systemctl >/dev/null 2>&1; then
sudo systemctl start docker
else
echo "Cannot start Docker automatically. Please start Docker manually."
exit 1
fi
else
echo "Docker is running."
fi
}
check_network() {
local network_name="portabase_network"
if ! docker network ls --format '{{.Name}}' | grep -q "^${network_name}$"; then
echo "Docker network '${network_name}' not found. Creating..."
docker network create "${network_name}"
else
echo "Docker network '${network_name}' already exists."
fi
}
check_docker
check_network
echo "Starting docker-compose..."
docker-compose -f ./docker-compose.yml up
echo "Docker-compose started successfully."
+17
View File
@@ -0,0 +1,17 @@
.DS_Store
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
*.swp
*.bak
*.tmp
*.orig
*~
.project
.idea/
*.tmproj
.vscode/
+29
View File
@@ -0,0 +1,29 @@
apiVersion: v2
name: portabase-agent
description: Helm chart for Portabase Agent
type: application
version: 0.0.0
appVersion: "latest"
keywords:
- postgresql
- mariadb
- mongodb
- mysql
- sqlite
- backup
- database
- restore
- agent
home: https://github.com/Portabase/agent
sources:
- https://github.com/Portabase/agent
- https://github.com/Portabase/agent/tree/main/helm
maintainers:
- name: Charles Gauthereau
url: https://github.com/RambokDev
- name: Killian Larcher
url: https://github.com/KillianLarcher
icon: https://raw.githubusercontent.com/Portabase/agent/main/.github/assets/logo.png
+59
View File
@@ -0,0 +1,59 @@
# Development Notes
## Check that Kubernetes is reachable locally
```bash
kubectl get nodes
```
## Install the local Portabase Agent Helm chart
```bash
helm install portabase-agent . \
--set env.EDGE_KEY=<your-edge-key>
```
## Check the pods
```bash
kubectl get pods
```
## Check the services
```bash
kubectl get svc
```
## To update .env variables or JSON config:
```bash
kubectl rollout restart deployment portabase-agent
```
## Install or upgrade the Helm chart
```bash
helm upgrade portabase-agent . \
--reuse-values \
--set env.EDGE_KEY="NEW_EDGE_KEY"
```
## Rollout to restart
```bash
kubectl rollout restart deployment portabase-agent
```
## List pods to get the pod name
```bash
kubectl get pods -l app=portabase-agent
```
## Get logs for the pod
```bash
kubectl logs portabase-agent-6f7d4f5c6b-abc12
```
## Uninstall Agent
``` bash
helm uninstall portabase-agent
```
+7
View File
@@ -0,0 +1,7 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: portabase-agent-config
data:
config.json: |
{{ .Values.volume.configFile.content | nindent 4 }}
+62
View File
@@ -0,0 +1,62 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: portabase-agent
labels:
app: portabase-agent
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
app: portabase-agent
template:
metadata:
labels:
app: portabase-agent
spec:
hostAliases:
{{- range .Values.network.hostAliases }}
- ip: {{ .ip }}
hostnames:
{{- range .hostnames }}
- {{ . }}
{{- end }}
{{- end }}
containers:
- name: portabase-agent
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
envFrom:
- configMapRef:
name: portabase-agent-env
volumeMounts:
{{- if .Values.volume.configFile.enabled }}
{{- if .Values.volume.configFile.hostPath }}
- name: config
mountPath: /config/config.json
subPath: config.json
readOnly: true
# uses hostPath
{{- else }}
- name: config
mountPath: /config/config.json
subPath: config.json
# uses ConfigMap content
{{- end }}
{{- end }}
volumes:
{{- if .Values.volume.configFile.enabled }}
{{- if .Values.volume.configFile.hostPath }}
- name: config
hostPath:
path: {{ .Values.volume.configFile.hostPath }}
type: File
{{- else }}
- name: config
configMap:
name: portabase-agent-config
items:
- key: config.json
path: config.json
{{- end }}
{{- end }}
+10
View File
@@ -0,0 +1,10 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: portabase-agent-env
data:
EDGE_KEY: {{ .Values.env.EDGE_KEY | quote }}
TZ: {{ .Values.env.TZ | quote }}
POLLING: {{ .Values.env.POLLING | quote }}
APP_ENV: {{ .Values.env.APP_ENV | quote }}
LOG: {{ .Values.env.LOG | quote }}
+48
View File
@@ -0,0 +1,48 @@
replicaCount: 1
image:
repository: portabase/agent
tag: latest
pullPolicy: IfNotPresent
env:
EDGE_KEY: "your_edge_key_here"
TZ: "UTC"
POLLING: "5"
APP_ENV: "production"
LOG: "info"
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 250m
memory: 256Mi
volume:
configFile:
enabled: true
hostPath: "" # Use host file if set, otherwise use `content`
content: | # JSON content for config.json if no hostPath
{
"databases": [
{
"name": "my-site-prod (readable name)",
"database": "devdb",
"type": "postgresql",
"host": "localhost",
"port": 5432,
"username": "admin_prod",
"password": "super_secure_password",
"generated_id": "550e8400-e29b-41d4-a716-446655440000"
}
]
}
network:
hostAliases:
- ip: "127.0.0.1"
hostnames:
- "localhost"
-99
View File
@@ -1,99 +0,0 @@
#!/bin/bash
set -e
if [ -z "$1" ]; then
echo "Usage: ./release <version>"
echo "Example: ./release v1.0.0"
exit 1
fi
VERSION=$1
CURRENT_BRANCH=$(git rev-parse --abbrev-ref HEAD)
if [ "$CURRENT_BRANCH" = "main" ]; then
if [[ ! "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
echo "Error: On 'main' branch, only release tags (X.Y.Z) are allowed."
exit 1
fi
else
if [[ ! "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$ ]]; then
echo "Error: On branch '$CURRENT_BRANCH', only RC tags matching X.Y.Z-rc.W are allowed (e.g., 1.0.0-rc.1)."
exit 1
fi
fi
CLEAN_VERSION=${VERSION#v}
CURRENT_DATE=$(date +%Y-%m-%d)
echo "Preparing release $VERSION..."
# package.json
if [ -f package.json ]; then
echo "Updating package.json..."
if sed --version >/dev/null 2>&1; then
sed -i "s/\"version\": \".*\"/\"version\": \"$CLEAN_VERSION\"/" package.json
else
sed -i '' "s/\"version\": \".*\"/\"version\": \"$CLEAN_VERSION\"/" package.json
fi
fi
# pyproject.toml
if [ -f pyproject.toml ]; then
echo "Updating pyproject.toml..."
if sed --version >/dev/null 2>&1; then
sed -i "s/^version = \".*\"/version = \"$CLEAN_VERSION\"/" pyproject.toml
else
sed -i '' "s/^version = \".*\"/version = \"$CLEAN_VERSION\"/" pyproject.toml
fi
fi
# Cargo.toml
if [ -f Cargo.toml ]; then
echo "Updating Cargo.toml..."
if sed --version >/dev/null 2>&1; then
sed -i "s/^version = \".*\"/version = \"$CLEAN_VERSION\"/" Cargo.toml
else
sed -i '' "s/^version = \".*\"/version = \"$CLEAN_VERSION\"/" Cargo.toml
fi
fi
# CITATION.cff
if [ -f CITATION.cff ]; then
echo "Updating CITATION.cff..."
if sed --version >/dev/null 2>&1; then
sed -i "s/^version: .*/version: $CLEAN_VERSION/" CITATION.cff
sed -i "s/^date-released: .*/date-released: \"$CURRENT_DATE\"/" CITATION.cff
else
sed -i '' "s/^version: .*/version: $CLEAN_VERSION/" CITATION.cff
sed -i '' "s/^date-released: .*/date-released: \"$CURRENT_DATE\"/" CITATION.cff
fi
fi
git add .
if ! git diff-index --quiet HEAD --; then
echo "Committing changes..."
git commit -m "chore(release): $VERSION"
else
echo "No changes to commit. Proceeding to tag..."
fi
if git rev-parse "$VERSION" >/dev/null 2>&1; then
echo "Tag $VERSION already exists. Aborting."
exit 1
fi
echo "Creating tag $VERSION..."
git tag -a "$VERSION" -m "Release $VERSION"
echo "Pushing changes and tags to remote..."
git push
git push origin "$VERSION"
echo "Successfully released $VERSION!"
+16 -2
View File
@@ -1,12 +1,26 @@
# Seed instructions
## MongoDB
```bash
make seed-mongo
make seed-mongo-auth
make seed-mysql
make seed-mysql-1gb
make seed-postgres
make seed-postgres-1gb
make seed-all
make seed-all
make seed-sqlite
make seed-sqlite SEED=big
```
## Verify commands
### Sqlite
```bash
docker exec -it db-sqlite sqlite3 /workspace/data/app.db "SELECT * FROM users LIMIT 10;"
```
```bash
docker exec -it db-sqlite sqlite3 /workspace/data/app.db "SELECT name FROM sqlite_master WHERE type='table';"
```
+53
View File
@@ -0,0 +1,53 @@
CREATE DATABASE IF NOT EXISTS mariadb;
USE mariadb;
DROP TABLE IF EXISTS users;
DROP TABLE IF EXISTS products;
CREATE TABLE users (
id BIGINT AUTO_INCREMENT PRIMARY KEY,
username VARCHAR(50) NOT NULL UNIQUE,
email VARCHAR(100) NOT NULL UNIQUE,
password VARCHAR(255) NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
) ENGINE=InnoDB;
CREATE TABLE products (
id BIGINT AUTO_INCREMENT PRIMARY KEY,
name VARCHAR(100) NOT NULL,
description MEDIUMTEXT,
price DECIMAL(10,2) NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
) ENGINE=InnoDB;
DELIMITER $$
DROP PROCEDURE IF EXISTS seed_data$$
CREATE PROCEDURE seed_data()
BEGIN
DECLARE i BIGINT DEFAULT 1;
DECLARE j BIGINT;
DECLARE large_text TEXT;
SET large_text = REPEAT('Lorem ipsum dolor sit amet, consectetur adipiscing elit. ', 300);
WHILE i <= 200000 DO
INSERT INTO users (username, email, password)
VALUES (CONCAT('user', i), CONCAT('user', i, '@example.com'), 'changeme');
SET i = i + 1;
END WHILE;
SET i = 1;
WHILE i <= 200000 DO
INSERT INTO products (name, description, price)
VALUES (CONCAT('Product ', i), large_text, ROUND(RAND()*1000,2));
SET i = i + 1;
END WHILE;
END$$
DELIMITER ;
CALL seed_data();
DROP PROCEDURE IF EXISTS seed_data;
+1 -1
View File
@@ -60,7 +60,7 @@ SELECT
)
FROM users u
JOIN generate_series(1, 30) AS p(post_no)
ON u.id <= 300000;
ON u.id <= 500000;
-- ============================================================
-- OPTIONAL: FORCE DISK MATERIALIZATION
+74
View File
@@ -0,0 +1,74 @@
PRAGMA foreign_keys = ON;
BEGIN TRANSACTION;
-- Drop tables
DROP TABLE IF EXISTS users;
DROP TABLE IF EXISTS projects;
DROP TABLE IF EXISTS tasks;
-- Users
CREATE TABLE users (
id INTEGER PRIMARY KEY AUTOINCREMENT,
email TEXT NOT NULL UNIQUE,
full_name TEXT NOT NULL,
role TEXT NOT NULL CHECK (role IN ('admin','manager','user')),
created_at TEXT NOT NULL DEFAULT (datetime('now'))
);
-- Projects
CREATE TABLE projects (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT NOT NULL,
description TEXT,
owner_id INTEGER NOT NULL,
created_at TEXT NOT NULL DEFAULT (datetime('now')),
FOREIGN KEY (owner_id) REFERENCES users(id) ON DELETE CASCADE
);
-- Tasks
CREATE TABLE tasks (
id INTEGER PRIMARY KEY AUTOINCREMENT,
project_id INTEGER NOT NULL,
title TEXT NOT NULL,
status TEXT NOT NULL CHECK (status IN ('todo','in_progress','done')),
priority INTEGER NOT NULL DEFAULT 3,
due_date TEXT,
created_at TEXT NOT NULL DEFAULT (datetime('now')),
FOREIGN KEY (project_id) REFERENCES projects(id) ON DELETE CASCADE
);
-- Seed minimal users
INSERT INTO users (email, full_name, role) VALUES
('admin@example.com', 'System Admin', 'admin'),
('manager@example.com', 'Project Manager', 'manager'),
('user@example.com', 'Standard User', 'user');
-- Generate 10_000 projects
WITH RECURSIVE numbers(x) AS (
SELECT 1
UNION ALL
SELECT x+1 FROM numbers WHERE x<10000
)
INSERT INTO projects (name, description, owner_id)
SELECT
'Project #' || x,
'Auto-generated project description for project #' || x,
(1 + (x % 3)) -- cycle users 1..3
FROM numbers;
-- Generate 100_000 tasks
WITH RECURSIVE numbers(x) AS (
SELECT 1
UNION ALL
SELECT x+1 FROM numbers WHERE x<100000
)
INSERT INTO tasks (project_id, title, status, priority, due_date)
SELECT
(1 + (x % 10000)), -- project id cycle
'Task #' || x,
CASE (x % 3) WHEN 0 THEN 'todo' WHEN 1 THEN 'in_progress' ELSE 'done' END,
1 + (x % 5),
date('now', '+' || (x % 30) || ' days')
FROM numbers;
COMMIT;
+59
View File
@@ -0,0 +1,59 @@
PRAGMA foreign_keys = ON;
BEGIN TRANSACTION;
-- Drop existing tables (idempotent reset)
DROP TABLE IF EXISTS users;
DROP TABLE IF EXISTS projects;
DROP TABLE IF EXISTS tasks;
-- Users
CREATE TABLE users (
id INTEGER PRIMARY KEY AUTOINCREMENT,
email TEXT NOT NULL UNIQUE,
full_name TEXT NOT NULL,
role TEXT NOT NULL CHECK (role IN ('admin','manager','user')),
created_at TEXT NOT NULL DEFAULT (datetime('now'))
);
-- Projects
CREATE TABLE projects (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT NOT NULL,
description TEXT,
owner_id INTEGER NOT NULL,
created_at TEXT NOT NULL DEFAULT (datetime('now')),
FOREIGN KEY (owner_id) REFERENCES users(id) ON DELETE CASCADE
);
-- Tasks
CREATE TABLE tasks (
id INTEGER PRIMARY KEY AUTOINCREMENT,
project_id INTEGER NOT NULL,
title TEXT NOT NULL,
status TEXT NOT NULL CHECK (status IN ('todo','in_progress','done')),
priority INTEGER NOT NULL DEFAULT 3,
due_date TEXT,
created_at TEXT NOT NULL DEFAULT (datetime('now')),
FOREIGN KEY (project_id) REFERENCES projects(id) ON DELETE CASCADE
);
-- Seed users
INSERT INTO users (email, full_name, role) VALUES
('admin@example.com', 'System Admin', 'admin'),
('manager@example.com', 'Project Manager', 'manager'),
('user@example.com', 'Standard User', 'user');
-- Seed projects
INSERT INTO projects (name, description, owner_id) VALUES
('Internal Tooling', 'Backoffice automation platform', 2),
('Client Portal', 'Customer-facing SaaS interface', 2);
-- Seed tasks
INSERT INTO tasks (project_id, title, status, priority, due_date) VALUES
(1, 'Define architecture', 'done', 1, date('now', '+3 days')),
(1, 'Implement authentication', 'in_progress', 1, date('now', '+7 days')),
(2, 'Design landing page', 'todo', 2, date('now', '+5 days')),
(2, 'Setup CI/CD', 'todo', 2, date('now', '+10 days'));
COMMIT;
Binary file not shown.
+87
View File
@@ -0,0 +1,87 @@
#!/usr/bin/env bash
set -e
POSTGRES_BASE="/usr/local/postgresql"
echo "Detecting OS and architecture..."
OS_TYPE="$(uname -s)"
ARCH="$(uname -m)"
install_pg_binaries() {
echo "Installing PostgreSQL binaries for versions 12-18..."
for v in 12 13 14 15 16 17 18; do
TARGET_DIR="$POSTGRES_BASE/$v/bin"
sudo mkdir -p "$TARGET_DIR"
if [[ "$OS_TYPE" == "Linux" ]]; then
if [[ "$ARCH" == "x86_64" ]]; then
SRC_DIR="./assets/tools/amd64/postgresql/postgresql-$v/bin"
elif [[ "$ARCH" == "aarch64" ]]; then
SRC_DIR="./assets/tools/arm64/postgresql/postgresql-$v/bin"
else
echo "Unsupported architecture: $ARCH"
continue
fi
if [[ -d "$SRC_DIR" ]]; then
echo "Copying PostgreSQL $v binaries from $SRC_DIR to $TARGET_DIR"
sudo cp -r "$SRC_DIR"/* "$TARGET_DIR/"
else
echo "Binaries for PostgreSQL $v not found for Linux, skipping..."
continue
fi
elif [[ "$OS_TYPE" == "Darwin" ]]; then
PG_SRC="$(brew --prefix postgresql@$v)/bin" 2>/dev/null || true
if [[ ! -d "$PG_SRC" ]]; then
echo "PostgreSQL $v not installed via Homebrew. Trying to install..."
if ! brew install postgresql@$v; then
echo "PostgreSQL $v not available, skipping..."
continue
fi
PG_SRC="$(brew --prefix postgresql@$v)/bin"
fi
echo "Copying PostgreSQL $v binaries from $PG_SRC to $TARGET_DIR"
sudo cp -r "$PG_SRC"/* "$TARGET_DIR/"
fi
sudo chown -R "$(whoami)" "$TARGET_DIR"
chmod +x "$TARGET_DIR"/*
done
echo "PostgreSQL binaries installed under $POSTGRES_BASE"
}
if [[ "$OS_TYPE" == "Linux" ]]; then
if command -v apt >/dev/null 2>&1; then
echo "Linux detected with apt. Installing prerequisites..."
sudo apt update
sudo apt install -y wget gnupg lsb-release redis-tools
install_pg_binaries
else
echo "Unsupported Linux distribution. Only apt-based distros are supported."
exit 1
fi
elif [[ "$OS_TYPE" == "Darwin" ]]; then
if command -v brew >/dev/null 2>&1; then
echo "macOS detected. Installing prerequisites..."
brew install redis
sudo mkdir -p "$POSTGRES_BASE"
sudo chown -R "$(whoami)" "$POSTGRES_BASE"
install_pg_binaries
else
echo "Homebrew not found. Please install Homebrew first: https://brew.sh/"
exit 1
fi
else
echo "Unsupported OS: $OS_TYPE"
exit 1
fi
echo "Tools installation completed successfully."
+4 -3
View File
@@ -43,16 +43,17 @@ impl Agent {
let ping_result = self.status_service.ping(&config.databases).await?;
for db in ping_result.databases.iter() {
let database = config.databases.iter().find(|cfg_db|cfg_db.generated_id == db.generated_id).unwrap();
info!(
"Generated Id: {} | backup action: {} | restore action: {}",
db.generated_id, db.data.backup.action, db.data.restore.action
"Generated Id: {} | backup action: {} | restore action: {} | Database Name: {}",
db.generated_id, db.data.backup.action, db.data.restore.action, database.name,
);
let _ = self.cron_service.sync(db).await;
if db.data.backup.action {
let _ = self
.backup_service
.dispatch(&db.generated_id, &config, method.clone())
.dispatch(&db.generated_id, &config, method.clone(), &db.storages, db.encrypt)
.await;
} else if db.data.restore.action {
let _ = self
+9 -1
View File
@@ -1,3 +1,4 @@
use crate::services::api::ApiClient;
use crate::settings::CONFIG;
use crate::utils::edge_key::{EdgeKey, EdgeKeyError, decode_edge_key};
use tracing::{debug, error, info};
@@ -6,6 +7,7 @@ use tracing::{debug, error, info};
pub struct Context {
#[allow(dead_code)]
pub edge_key: EdgeKey,
pub api: ApiClient,
}
impl Context {
@@ -33,7 +35,13 @@ impl Context {
panic!("Cannot initialize AgentContext due to invalid EDGE_KEY");
}
};
let server_url = format!("{}/api", edge_key.server_url);
let api_client = ApiClient::new(server_url);
Context { edge_key }
Context {
edge_key: edge_key,
api: api_client,
}
}
}
+8 -2
View File
@@ -2,6 +2,8 @@ use crate::domain::mongodb::database::MongoDatabase;
use crate::domain::mysql::database::MySQLDatabase;
use crate::domain::postgres::database::PostgresDatabase;
use crate::domain::postgres::{detect_format_from_file, detect_format_from_size};
use crate::domain::redis::database::RedisDatabase;
use crate::domain::sqlite::database::SqliteDatabase;
use crate::services::config::{DatabaseConfig, DbType};
use anyhow::Result;
use std::path::{Path, PathBuf};
@@ -11,8 +13,8 @@ use std::sync::Arc;
pub trait Database: Send + Sync {
fn file_extension(&self) -> &'static str;
async fn ping(&self) -> Result<bool>;
async fn backup(&self, backup_dir: &Path) -> Result<PathBuf>;
async fn restore(&self, restore_file: &Path) -> Result<()>;
async fn backup(&self, backup_dir: &Path, is_test: Option<bool>) -> Result<PathBuf>;
async fn restore(&self, restore_file: &Path, is_test: Option<bool>) -> Result<()>;
}
pub struct DatabaseFactory;
@@ -27,6 +29,8 @@ impl DatabaseFactory {
DbType::Mysql => Arc::new(MySQLDatabase::new(cfg)),
DbType::Mariadb => Arc::new(MySQLDatabase::new(cfg)),
DbType::MongoDB => Arc::new(MongoDatabase::new(cfg)),
DbType::Sqlite => Arc::new(SqliteDatabase::new(cfg)),
DbType::Redis => Arc::new(RedisDatabase::new(cfg)),
}
}
@@ -39,6 +43,8 @@ impl DatabaseFactory {
DbType::Mysql => Arc::new(MySQLDatabase::new(cfg)),
DbType::Mariadb => Arc::new(MySQLDatabase::new(cfg)),
DbType::MongoDB => Arc::new(MongoDatabase::new(cfg)),
DbType::Sqlite => Arc::new(SqliteDatabase::new(cfg)),
DbType::Redis => Arc::new(RedisDatabase::new(cfg)),
}
}
}
+2
View File
@@ -2,4 +2,6 @@ pub mod factory;
pub mod postgres;
pub mod mysql;
mod mongodb;
mod sqlite;
mod redis;
+1 -1
View File
@@ -15,7 +15,7 @@ pub async fn run(
let file_path = backup_dir.join(format!("{}{}", cfg.generated_id, file_extension));
let mongodump = select_mongo_path().join("mongodump");
let uri = get_mongo_uri(cfg.clone());
let uri = get_mongo_uri(cfg.clone())?;
let output = Command::new(mongodump)
.arg(format!("--uri={}", uri))
+8 -6
View File
@@ -3,7 +3,7 @@ use anyhow::Result;
use mongodb::Client;
pub async fn connect(cfg: DatabaseConfig) -> Result<Client> {
let uri = get_mongo_uri(cfg);
let uri = get_mongo_uri(cfg)?;
let mut options = mongodb::options::ClientOptions::parse(&uri).await?;
options.server_selection_timeout = Some(std::time::Duration::from_secs(3));
options.connect_timeout = Some(std::time::Duration::from_secs(3));
@@ -15,14 +15,16 @@ pub fn select_mongo_path() -> std::path::PathBuf {
"/usr/local/mongodb/bin".to_string().into()
}
pub fn get_mongo_uri(cfg: DatabaseConfig) -> String {
if cfg.username.is_empty() {
format!("mongodb://{}:{}/{}", cfg.host, cfg.port, cfg.database)
pub fn get_mongo_uri(cfg: DatabaseConfig) -> Result<String> {
if cfg.username.is_empty() || cfg.password.is_empty() {
Ok(format!("mongodb://{}:{}/{}", cfg.host, cfg.port, cfg.database))
} else {
format!(
Ok(format!(
"mongodb://{}:{}@{}:{}/{}?authSource=admin",
cfg.username, cfg.password, cfg.host, cfg.port, cfg.database
)
))
}
}
+17 -12
View File
@@ -27,22 +27,27 @@ impl Database for MongoDatabase {
ping::run(self.cfg.clone()).await
}
async fn backup(&self, dir: &Path) -> Result<PathBuf> {
FileLock::acquire(&self.cfg.generated_id, DbOpLock::Backup.as_str()).await?;
let res = backup::run(
self.cfg.clone(),
dir.to_path_buf(),
self.file_extension(),
)
.await;
FileLock::release(&self.cfg.generated_id).await?;
async fn backup(&self, dir: &Path, is_test: Option<bool>) -> Result<PathBuf> {
let test_mode = is_test.unwrap_or(false);
if !test_mode {
FileLock::acquire(&self.cfg.generated_id, DbOpLock::Backup.as_str()).await?;
}
let res = backup::run(self.cfg.clone(), dir.to_path_buf(), self.file_extension()).await;
if !test_mode {
FileLock::release(&self.cfg.generated_id).await?;
}
res
}
async fn restore(&self, file: &Path) -> Result<()> {
FileLock::acquire(&self.cfg.generated_id, DbOpLock::Restore.as_str()).await?;
async fn restore(&self, file: &Path, is_test: Option<bool>) -> Result<()> {
let test_mode = is_test.unwrap_or(false);
if !test_mode {
FileLock::acquire(&self.cfg.generated_id, DbOpLock::Restore.as_str()).await?;
}
let res = restore::run(self.cfg.clone(), file.to_path_buf()).await;
FileLock::release(&self.cfg.generated_id).await?;
if !test_mode {
FileLock::release(&self.cfg.generated_id).await?;
}
res
}
}
+9 -3
View File
@@ -1,14 +1,20 @@
#![allow(dead_code)]
use crate::domain::mongodb::connection::connect;
use crate::services::config::DatabaseConfig;
use anyhow::Result;
use mongodb::bson::doc;
use tracing::{error};
use crate::domain::mongodb::connection::connect;
use tracing::error;
pub async fn run(cfg: DatabaseConfig) -> Result<bool> {
let client = connect(cfg.clone()).await?;
let db_name = if cfg.username.is_empty() { &cfg.database } else { "admin" };
let db_name = if cfg.username.is_empty() && cfg.password.is_empty() {
&cfg.database
} else {
"admin"
};
match client.database(db_name).run_command(doc! {"ping": 1}).await {
Ok(_) => Ok(true),
Err(e) => {
+1 -1
View File
@@ -10,7 +10,7 @@ pub async fn run(cfg: DatabaseConfig, restore_file: PathBuf) -> Result<()> {
debug!("Starting MongoDB restore for database {}", cfg.name);
let mongorestore = select_mongo_path().join("mongorestore");
let uri = get_mongo_uri(cfg.clone());
let uri = get_mongo_uri(cfg.clone())?;
let output = Command::new(mongorestore)
.arg(format!("--uri={}", uri))
+1 -1
View File
@@ -1,4 +1,4 @@
use crate::domain::mysql::connection::server_version;
use crate::domain::mysql::connection::{server_version};
use crate::services::config::DatabaseConfig;
use anyhow::{Context, Result};
use std::collections::HashMap;
+2
View File
@@ -3,6 +3,7 @@ use std::process::Command;
use anyhow::Result;
pub async fn server_version(cfg: &DatabaseConfig) -> Result<String> {
let output = Command::new("mysql")
.arg("--host").arg(&cfg.host)
.arg("--port").arg(cfg.port.to_string())
@@ -25,3 +26,4 @@ pub async fn server_version(cfg: &DatabaseConfig) -> Result<String> {
Ok(version)
}
+18 -7
View File
@@ -21,7 +21,7 @@ impl MySQLDatabase {
fn build_env(&self) -> HashMap<String, String> {
let mut envs = std::env::vars().collect::<HashMap<_, _>>();
envs.insert("MYSQL_PWD".to_string(), self.cfg.password.clone());
envs.insert("MYSQL_PWD".to_string(), self.cfg.password.to_string());
envs
}
}
@@ -36,17 +36,28 @@ impl Database for MySQLDatabase {
ping::run(self.cfg.clone(), self.build_env().clone()).await
}
async fn backup(&self, dir: &Path) -> Result<PathBuf> {
FileLock::acquire(&self.cfg.generated_id, DbOpLock::Backup.as_str()).await?;
async fn backup(&self, dir: &Path, is_test: Option<bool>) -> Result<PathBuf> {
let test_mode = is_test.unwrap_or(false);
if !test_mode {
FileLock::acquire(&self.cfg.generated_id, DbOpLock::Backup.as_str()).await?;
}
let res = backup::run(self.cfg.clone(), dir.to_path_buf(), self.build_env().clone(), self.file_extension()).await;
FileLock::release(&self.cfg.generated_id).await?;
if !test_mode {
FileLock::release(&self.cfg.generated_id).await?;
}
res
}
async fn restore(&self, file: &Path) -> Result<()> {
FileLock::acquire(&self.cfg.generated_id, DbOpLock::Restore.as_str()).await?;
async fn restore(&self, file: &Path, is_test: Option<bool>) -> Result<()> {
let test_mode = is_test.unwrap_or(false);
if !test_mode {
FileLock::acquire(&self.cfg.generated_id, DbOpLock::Restore.as_str()).await?;
}
let res = restore::run(self.cfg.clone(), file.to_path_buf()).await;
FileLock::release(&self.cfg.generated_id).await?;
if !test_mode {
FileLock::release(&self.cfg.generated_id).await?;
}
res
}
}
+16 -8
View File
@@ -1,19 +1,27 @@
use crate::services::config::DatabaseConfig;
use anyhow::Context;
use std::collections::HashMap;
use std::process::Command;
use tokio::process::Command;
use tokio::time::{Duration, timeout};
pub async fn run(cfg: DatabaseConfig, env: HashMap<String, String>) -> anyhow::Result<bool> {
let output = Command::new("mysqladmin")
.arg("--host")
let mut cmd = Command::new("mysqladmin");
cmd.arg("--host")
.arg(cfg.host)
.arg("--port")
.arg(cfg.port.to_string())
.arg("--user")
.arg(cfg.username)
.arg("ping")
.envs(env)
.output()
.with_context(|| format!("Failed to ping MySQL server {}", cfg.name))?;
Ok(output.status.success())
.envs(env);
let result = timeout(Duration::from_secs(10), cmd.output()).await;
match result {
Ok(output) => {
let output = output?;
Ok(output.status.success())
}
Err(_) => Ok(false),
}
}
+6 -7
View File
@@ -1,11 +1,10 @@
use crate::services::config::DatabaseConfig;
use anyhow::{Context, Result};
use std::fs::File;
use std::io::{Read, Write};
use anyhow::{Context, Result};
use tracing::{debug, error, info};
use std::path::PathBuf;
use std::process::Command;
use crate::services::config::DatabaseConfig;
use tracing::{debug, error, info};
pub async fn run(cfg: DatabaseConfig, restore_file: PathBuf) -> Result<()> {
let handle = tokio::task::spawn_blocking(move || -> Result<()> {
@@ -55,7 +54,8 @@ pub async fn run(cfg: DatabaseConfig, restore_file: PathBuf) -> Result<()> {
.with_context(|| format!("Failed to start mysql restore for {}", cfg.name))?;
let mut stdin = child.stdin.take().context("Failed to open child stdin")?;
stdin.write_all(sql_content.as_bytes())
stdin
.write_all(sql_content.as_bytes())
.context("Failed to write SQL content to mysql stdin")?;
stdin.flush()?;
drop(stdin);
@@ -74,8 +74,7 @@ pub async fn run(cfg: DatabaseConfig, restore_file: PathBuf) -> Result<()> {
Ok(())
});
handle
.await??;
handle.await??;
Ok(())
}
+4 -2
View File
@@ -1,7 +1,7 @@
use anyhow::Result;
use tracing::{debug, error, info};
use std::path::PathBuf;
use std::process::Command;
use tracing::{debug, error, info};
use super::connection::{select_pg_path, server_version};
use super::format::PostgresDumpFormat;
@@ -11,6 +11,7 @@ pub async fn run(
cfg: DatabaseConfig,
format: PostgresDumpFormat,
backup_dir: PathBuf,
is_test: Option<bool>
) -> Result<PathBuf> {
tokio::task::spawn_blocking(move || -> Result<PathBuf> {
debug!("Starting backup for database {}", cfg.name);
@@ -26,7 +27,8 @@ pub async fn run(
}
};
let pg_dump = select_pg_path(&version).join("pg_dump");
let pg_dump = select_pg_path(&version, is_test).join("pg_dump");
debug!("Using pg_dump at {:?}", pg_dump);
match format {
+12 -6
View File
@@ -1,11 +1,12 @@
use std::path::Path;
use crate::domain::postgres::format::PostgresDumpFormat;
use crate::services::config::DatabaseConfig;
use anyhow::Result;
use std::path::Path;
use tokio_postgres::{Client, NoTls};
use tracing::info;
use tracing::{error, info};
pub async fn connect(cfg: &DatabaseConfig) -> Result<Client> {
info!("Connecting to postgres database {}:{}", cfg.host, cfg.port);
let dsn = format!(
"host={} port={} user={} password={} dbname={}",
cfg.host, cfg.port, cfg.username, cfg.password, cfg.database
@@ -14,7 +15,7 @@ pub async fn connect(cfg: &DatabaseConfig) -> Result<Client> {
let (client, connection) = tokio_postgres::connect(&dsn, NoTls).await?;
tokio::spawn(async move {
if let Err(e) = connection.await {
tracing::error!("Postgres connection error: {}", e);
error!("Postgres connection error: {}", e);
}
});
Ok(client)
@@ -27,14 +28,19 @@ pub async fn server_version(cfg: &DatabaseConfig) -> Result<String> {
Ok(version)
}
pub fn select_pg_path(version: &str) -> std::path::PathBuf {
pub fn select_pg_path(version: &str, is_test: Option<bool>) -> std::path::PathBuf {
let major = version.split('.').next().unwrap_or("17");
format!("/usr/lib/postgresql/{}/bin", major).into()
if is_test.unwrap_or(false) {
format!("/usr/local/postgresql/{}/bin", major).into()
} else {
format!("/usr/lib/postgresql/{}/bin", major).into()
}
}
pub async fn terminate_connections(cfg: &DatabaseConfig) -> Result<()> {
let mut admin = cfg.clone();
admin.database = "postgres".into();
admin.database = "postgres".to_string().into();
let client = connect(&admin).await?;
+19 -15
View File
@@ -2,11 +2,7 @@ use anyhow::Result;
use async_trait::async_trait;
use std::path::{Path, PathBuf};
use super::{
backup,
format::PostgresDumpFormat,
ping, restore,
};
use super::{backup, format::PostgresDumpFormat, ping, restore};
use crate::domain::factory::Database;
use crate::services::config::DatabaseConfig;
use crate::utils::locks::{DbOpLock, FileLock};
@@ -28,7 +24,6 @@ impl Database for PostgresDatabase {
match self.format {
PostgresDumpFormat::Fc => ".dump",
PostgresDumpFormat::Fd => ".gz",
// PostgresDumpFormat::Fd => ".tar.gz",
}
}
@@ -36,18 +31,27 @@ impl Database for PostgresDatabase {
ping::run(self.cfg.clone()).await
}
async fn backup(&self, dir: &Path) -> Result<PathBuf> {
FileLock::acquire(&self.cfg.generated_id, DbOpLock::Backup.as_str()).await?;
let res = backup::run(self.cfg.clone(), self.format, dir.to_path_buf()).await;
FileLock::release(&self.cfg.generated_id).await?;
async fn backup(&self, dir: &Path, is_test: Option<bool>) -> Result<PathBuf> {
let test_mode = is_test.unwrap_or(false);
if !test_mode {
FileLock::acquire(&self.cfg.generated_id, DbOpLock::Backup.as_str()).await?;
}
let res = backup::run(self.cfg.clone(), self.format, dir.to_path_buf(), is_test).await;
if !test_mode {
FileLock::release(&self.cfg.generated_id).await?;
}
res
}
async fn restore(&self, file: &Path) -> Result<()> {
FileLock::acquire(&self.cfg.generated_id, DbOpLock::Restore.as_str()).await?;
let res = restore::run(self.cfg.clone(), self.format, file.to_path_buf()).await;
FileLock::release(&self.cfg.generated_id).await?;
async fn restore(&self, file: &Path, is_test: Option<bool>) -> Result<()> {
let test_mode = is_test.unwrap_or(false);
if !test_mode {
FileLock::acquire(&self.cfg.generated_id, DbOpLock::Restore.as_str()).await?;
}
let res = restore::run(self.cfg.clone(), self.format, file.to_path_buf(), is_test).await;
if !test_mode {
FileLock::release(&self.cfg.generated_id).await?;
}
res
}
}
+8 -2
View File
@@ -1,7 +1,7 @@
use anyhow::Result;
use tracing::{debug, error, info};
use std::path::PathBuf;
use std::process::Command;
use tracing::{debug, error, info};
use super::connection::{select_pg_path, server_version, terminate_connections};
use super::format::PostgresDumpFormat;
@@ -11,9 +11,11 @@ pub async fn run(
cfg: DatabaseConfig,
format: PostgresDumpFormat,
restore_file: PathBuf,
is_test: Option<bool>,
) -> Result<()> {
tokio::task::spawn_blocking(move || -> Result<()> {
debug!("Starting restore for database {}", cfg.name);
let version = match futures::executor::block_on(server_version(&cfg)) {
Ok(v) => {
debug!("Postgres version detected: {}", v);
@@ -25,7 +27,8 @@ pub async fn run(
}
};
let pg_restore = select_pg_path(&version).join("pg_restore");
let pg_restore = select_pg_path(&version, is_test).join("pg_restore");
debug!("Using pg_restore at {:?}", pg_restore);
if let Err(e) = futures::executor::block_on(terminate_connections(&cfg)) {
@@ -86,6 +89,8 @@ pub async fn run(
}
};
info!("tar_gz {:?}", tar_gz);
let dec = flate2::read::GzDecoder::new(tar_gz);
let mut archive = tar::Archive::new(dec);
@@ -106,6 +111,7 @@ pub async fn run(
}
debug!("Listing contents of temp dir: {}", tmp_dir.path().display());
for entry in std::fs::read_dir(tmp_dir.path())? {
if let Ok(entry) = entry {
let path = entry.path();
+65
View File
@@ -0,0 +1,65 @@
use crate::services::config::DatabaseConfig;
use anyhow::{Context, Result};
use std::path::PathBuf;
use std::process::Command;
use tracing::{debug, error, info};
pub async fn run(
cfg: DatabaseConfig,
backup_dir: PathBuf,
file_extension: &'static str,
) -> Result<PathBuf> {
tokio::task::spawn_blocking(move || -> Result<PathBuf> {
debug!("Starting Redis backup for database {}", cfg.name);
let file_path = backup_dir.join(format!("{}{}", cfg.generated_id, file_extension));
let mut cmd = Command::new("redis-cli");
cmd.arg("-h")
.arg(&cfg.host)
.arg("-p")
.arg(cfg.port.to_string());
if !cfg.username.is_empty() {
cmd.arg("--user").arg(&cfg.username);
}
if !cfg.password.is_empty() {
cmd.arg("-a").arg(&cfg.password);
}
cmd.arg("--rdb").arg(&file_path);
debug!("Command Backup: {:?}", cmd);
let output = cmd.output().context("Redis backup command failed")?;
let stderr = String::from_utf8_lossy(&output.stderr);
let stdout = String::from_utf8_lossy(&output.stdout);
if !output.status.success() {
if stderr.contains("NOAUTH") {
error!(
"Redis backup failed for {}: Authentication required (NOAUTH)",
cfg.name
);
anyhow::bail!(
"Redis backup failed for {}: Authentication required",
cfg.name
);
} else {
error!("Redis backup failed for {}: {}", cfg.name, stderr);
anyhow::bail!("Redis backup failed for {}: {}", cfg.name, stderr);
}
}
info!(
"Redis backup completed for {}. Output: {}",
cfg.name, stdout
);
Ok(file_path)
})
.await?
}
+45
View File
@@ -0,0 +1,45 @@
use anyhow::{Result, bail};
use async_trait::async_trait;
use std::path::{Path, PathBuf};
use crate::domain::factory::Database;
use crate::domain::redis::{backup, ping};
use crate::services::config::DatabaseConfig;
use crate::utils::locks::{DbOpLock, FileLock};
pub struct RedisDatabase {
cfg: DatabaseConfig,
}
impl RedisDatabase {
pub fn new(cfg: DatabaseConfig) -> Self {
Self { cfg }
}
}
#[async_trait]
impl Database for RedisDatabase {
fn file_extension(&self) -> &'static str {
".rdb"
}
async fn ping(&self) -> Result<bool> {
ping::run(self.cfg.clone()).await
}
async fn backup(&self, dir: &Path, is_test: Option<bool>) -> Result<PathBuf> {
let test_mode = is_test.unwrap_or(false);
if !test_mode {
FileLock::acquire(&self.cfg.generated_id, DbOpLock::Backup.as_str()).await?;
}
let res = backup::run(self.cfg.clone(), dir.to_path_buf(), self.file_extension()).await;
if !test_mode {
FileLock::release(&self.cfg.generated_id).await?;
}
res
}
async fn restore(&self, _file: &Path, _is_test: Option<bool>) -> Result<()> {
bail!("Restore not supported for Redis databases")
}
}
+3
View File
@@ -0,0 +1,3 @@
mod backup;
pub mod database;
mod ping;
+55
View File
@@ -0,0 +1,55 @@
use tracing::{debug, info};
use crate::services::config::DatabaseConfig;
use tokio::process::Command;
use tokio::time::{timeout, Duration};
use anyhow::{Result, Context};
pub async fn run(cfg: DatabaseConfig) -> Result<bool> {
let mut cmd = Command::new("redis-cli");
cmd.arg("-h")
.arg(&cfg.host)
.arg("-p")
.arg(cfg.port.to_string());
if !cfg.username.is_empty() {
cmd.arg("--user").arg(&cfg.username);
}
if !cfg.password.is_empty() {
cmd.arg("-a").arg(&cfg.password);
}
cmd.arg("PING");
debug!("Command Ping: {:?}", cmd);
let result = timeout(Duration::from_secs(10), cmd.output()).await;
match result {
Ok(output) => {
let output = output.context("Failed to execute redis-cli")?;
let stdout = String::from_utf8_lossy(&output.stdout);
let stderr = String::from_utf8_lossy(&output.stderr);
info!("Redis stdout: {}", stdout);
info!("Redis stderr: {}", stderr);
if stderr.contains("NOAUTH") {
info!("Redis authentication failed (NOAUTH required)");
return Ok(false);
}
if !output.status.success() {
info!("Redis command failed with status: {:?}", output.status);
return Ok(false);
}
Ok(stdout.contains("PONG"))
}
Err(_) => {
info!("Timeout connecting to Redis at {}:{}", cfg.host, cfg.port);
Ok(false)
}
}
}
+45
View File
@@ -0,0 +1,45 @@
use crate::services::config::DatabaseConfig;
use anyhow::{Context, Result};
use std::path::PathBuf;
use std::process::Command;
use tracing::{debug, error, info};
pub async fn run(
cfg: DatabaseConfig,
backup_dir: PathBuf,
file_extension: &'static str,
) -> Result<PathBuf> {
tokio::task::spawn_blocking(move || -> Result<PathBuf> {
debug!("Starting SQLite backup for database {}", cfg.name);
let db_path_str = if cfg.path.is_empty() {
anyhow::bail!("Database path not configured");
} else {
cfg.path.as_str().to_string()
};
let db_path = PathBuf::from(db_path_str);
if !db_path.exists() {
anyhow::bail!("SQLite database file not found: {}", db_path.display());
}
let file_path = backup_dir.join(format!("{}{}", cfg.generated_id, file_extension));
let output = Command::new("sqlite3")
.arg(db_path.as_os_str())
.arg(format!(".backup '{}'", file_path.display()))
.output()
.context("SQLite backup command failed to start")?;
if !output.status.success() {
let stderr = String::from_utf8_lossy(&output.stderr);
error!("SQLite backup failed for {}: {}", cfg.name, stderr);
anyhow::bail!("SQLite backup failed for {}: {}", cfg.name, stderr);
}
info!("SQLite backup completed for {}", cfg.name);
Ok(file_path)
})
.await?
}
+52
View File
@@ -0,0 +1,52 @@
use anyhow::Result;
use async_trait::async_trait;
use std::path::{Path, PathBuf};
use super::{backup, ping, restore};
use crate::domain::factory::Database;
use crate::services::config::DatabaseConfig;
use crate::utils::locks::{DbOpLock, FileLock};
pub struct SqliteDatabase {
cfg: DatabaseConfig,
}
impl SqliteDatabase {
pub fn new(cfg: DatabaseConfig) -> Self {
Self { cfg }
}
}
#[async_trait]
impl Database for SqliteDatabase {
fn file_extension(&self) -> &'static str {
".backup"
}
async fn ping(&self) -> Result<bool> {
ping::run(self.cfg.clone()).await
}
async fn backup(&self, dir: &Path, is_test: Option<bool>) -> Result<PathBuf> {
let test_mode = is_test.unwrap_or(false);
if !test_mode {
FileLock::acquire(&self.cfg.generated_id, DbOpLock::Backup.as_str()).await?;
}
let res = backup::run(self.cfg.clone(), dir.to_path_buf(), self.file_extension()).await;
if !test_mode {
FileLock::release(&self.cfg.generated_id).await?;
}
res
}
async fn restore(&self, file: &Path, is_test: Option<bool>) -> Result<()> {
let test_mode = is_test.unwrap_or(false);
if !test_mode {
FileLock::acquire(&self.cfg.generated_id, DbOpLock::Restore.as_str()).await?;
}
let res = restore::run(self.cfg.clone(), file.to_path_buf()).await;
if !test_mode {
FileLock::release(&self.cfg.generated_id).await?;
}
res
}
}
+4
View File
@@ -0,0 +1,4 @@
mod backup;
mod restore;
mod ping;
pub mod database;
+5
View File
@@ -0,0 +1,5 @@
use crate::services::config::DatabaseConfig;
pub async fn run(_cfg: DatabaseConfig) -> anyhow::Result<bool> {
Ok(true)
}
+44
View File
@@ -0,0 +1,44 @@
use crate::services::config::DatabaseConfig;
use anyhow::{Context, Result};
use std::path::PathBuf;
use std::process::Command;
use tracing::{debug, error, info};
pub async fn run(cfg: DatabaseConfig, restore_file: PathBuf) -> Result<()> {
tokio::task::spawn_blocking(move || -> Result<()> {
debug!("Starting SQLite restore for database {}", cfg.name);
let db_path_str = if cfg.path.is_empty() {
anyhow::bail!("Database path not configured");
} else {
cfg.path.as_str().to_string()
};
let db_path = PathBuf::from(db_path_str);
if !restore_file.exists() {
anyhow::bail!("Restore file not found: {}", restore_file.display());
}
if db_path.exists() {
std::fs::remove_file(&db_path)
.with_context(|| format!("Failed to remove existing DB {}", db_path.display()))?;
}
let output = Command::new("sqlite3")
.arg(db_path.as_os_str())
.arg(format!(".restore '{}'", restore_file.display()))
.output()
.with_context(|| format!("Failed to run sqlite3 restore for {}", cfg.name))?;
if !output.status.success() {
let stderr = String::from_utf8_lossy(&output.stderr);
error!("SQLite restore failed for {}: {}", cfg.name, stderr);
anyhow::bail!("SQLite restore failed for {}", cfg.name);
}
info!("SQLite restore completed for {}", cfg.name);
Ok(())
})
.await?
}
+2
View File
@@ -4,6 +4,8 @@ mod services;
mod settings;
mod tasks;
mod utils;
#[cfg(test)]
mod tests;
use crate::tasks::ping::ping_server;
use crate::utils::locks::FileLock;
+83
View File
@@ -0,0 +1,83 @@
#![allow(dead_code)]
use reqwest::{Client, Method};
use serde::de::DeserializeOwned;
use std::time::Duration;
use crate::services::api::ApiError;
#[derive(Clone, Debug)]
pub struct ApiClient {
base_url: String,
http: Client,
}
impl ApiClient {
pub fn new(base_url: impl Into<String>) -> Self {
let http = Client::builder()
.timeout(Duration::from_secs(30))
.build()
.expect("failed to build http client");
Self {
base_url: base_url.into(),
http,
}
}
pub async fn request<T: DeserializeOwned>(
&self,
method: Method,
path: &str,
) -> Result<Option<T>, ApiError> {
let url = format!("{}{}", self.base_url, path);
let res = self.http.request(method, &url).send().await?;
let status = res.status();
let body = res.text().await.unwrap_or_default();
if !status.is_success() {
return Err(ApiError::HttpResponse { status, body });
}
if body.trim().is_empty() {
Ok(None)
} else {
Ok(Some(serde_json::from_str::<T>(&body)?))
}
}
pub async fn request_with_body<T, B>(
&self,
method: Method,
path: &str,
body: &B,
) -> Result<Option<T>, ApiError>
where
T: DeserializeOwned,
B: serde::Serialize,
{
let url = format!("{}{}", self.base_url, path);
let res = self
.http
.request(method, &url)
.json(body)
.send()
.await?;
let status = res.status();
let body_text = res.text().await.unwrap_or_default();
if !status.is_success() {
return Err(ApiError::HttpResponse {
status,
body: body_text,
});
}
if body_text.trim().is_empty() {
Ok(None)
} else {
Ok(Some(serde_json::from_str::<T>(&body_text)?))
}
}
}
@@ -0,0 +1,67 @@
pub mod upload;
use crate::services::api::models::agent::backup::BackupResponse;
use crate::services::api::{ApiClient, ApiError};
use anyhow::Result;
use reqwest::Method;
use serde::Serialize;
#[derive(Serialize)]
pub struct BackupCreateRequest {
pub method: String,
#[serde(rename = "generatedId")]
pub generated_id: String,
}
#[derive(Serialize)]
pub struct BackupUpdateRequest {
#[serde(rename = "backupId")]
pub backup_id: String,
pub status: String,
pub size: Option<u64>,
#[serde(rename = "generatedId")]
pub generated_id: String,
}
impl ApiClient {
pub async fn backup_create(
&self,
method: impl Into<String>,
agent_id: impl Into<String>,
generated_id: impl Into<String>,
) -> Result<Option<BackupResponse>, ApiError> {
let body = BackupCreateRequest {
method: method.into(),
generated_id: generated_id.into(),
};
let agent_id = agent_id.into();
let path = format!("/agent/{}/backup", agent_id);
self.request_with_body(Method::POST, path.as_str(), &body)
.await
}
pub async fn backup_update(
&self,
agent_id: impl Into<String>,
backup_id: impl Into<String>,
status: impl Into<String>,
file_size: impl Into<Option<u64>>,
generated_id: impl Into<String>,
) -> Result<Option<BackupResponse>, ApiError> {
let body = BackupUpdateRequest {
backup_id: backup_id.into(),
status: status.into(),
size: file_size.into(),
generated_id: generated_id.into(),
};
let agent_id = agent_id.into();
let path = format!("/agent/{}/backup", agent_id);
self.request_with_body(Method::PATCH, path.as_str(), &body)
.await
}
}
@@ -0,0 +1,37 @@
use crate::services::api::{ApiClient, ApiError};
use anyhow::Result;
use reqwest::Method;
use serde::Serialize;
use crate::services::api::models::agent::backup::BackupUploadResponse;
#[derive(Serialize)]
pub struct InitUploadRequest {
#[serde(rename = "generatedId")]
pub generated_id: String,
#[serde(rename = "storageChannelId")]
pub storage_channel_id: String,
#[serde(rename = "backupId")]
pub backup_id: String,
}
impl ApiClient {
pub async fn backup_upload_init(
&self,
agent_id: impl Into<String>,
generated_id: impl Into<String>,
storage_channel_id: impl Into<String>,
backup_id: impl Into<String>,
) -> Result<Option<BackupUploadResponse>, ApiError> {
let body = InitUploadRequest {
generated_id: generated_id.into(),
storage_channel_id: storage_channel_id.into(),
backup_id: backup_id.into(),
};
let agent_id = agent_id.into();
let path = format!("/agent/{}/backup/upload/init", agent_id);
self.request_with_body(Method::POST, path.as_str(), &body)
.await
}
}
@@ -0,0 +1,2 @@
pub mod init;
pub mod status;
@@ -0,0 +1,46 @@
use crate::services::api::models::agent::backup::BackupUploadResponse;
use crate::services::api::{ApiClient, ApiError};
use anyhow::Result;
use reqwest::Method;
use serde::Serialize;
#[derive(Serialize)]
pub struct StatusUploadRequest {
#[serde(rename = "generatedId")]
pub generated_id: String,
#[serde(rename = "backupStorageId")]
pub backup_storage_id: String,
pub status: String,
pub path: String,
pub size: u64,
#[serde(rename = "backupId")]
pub backup_id: String,
}
impl ApiClient {
pub async fn backup_upload_status(
&self,
agent_id: impl Into<String>,
generated_id: impl Into<String>,
backup_storage_id: impl Into<String>,
status: impl Into<String>,
remote_path: impl Into<String>,
total_size: impl Into<u64>,
backup_id: impl Into<String>,
) -> Result<Option<BackupUploadResponse>, ApiError> {
let body = StatusUploadRequest {
generated_id: generated_id.into(),
backup_storage_id: backup_storage_id.into(),
status: status.into(),
path: remote_path.into(),
size: total_size.into(),
backup_id: backup_id.into(),
};
let agent_id = agent_id.into();
let path = format!("/agent/{}/backup/upload/status", agent_id);
self.request_with_body(Method::PATCH, path.as_str(), &body)
.await
}
}
+3
View File
@@ -0,0 +1,3 @@
pub mod status;
pub mod backup;
pub mod restore;
@@ -0,0 +1,33 @@
use crate::services::api::models::agent::restore::ResultRestoreResponse;
use crate::services::api::{ApiClient, ApiError};
use anyhow::Result;
use reqwest::Method;
use serde::Serialize;
#[derive(Serialize)]
pub struct ResultRestoreRequest {
#[serde(rename = "generatedId")]
pub generated_id: String,
pub status: String,
}
impl ApiClient {
pub async fn restore_result(
&self,
agent_id: impl Into<String>,
generated_id: impl Into<String>,
status: impl Into<String>,
) -> Result<Option<ResultRestoreResponse>, ApiError> {
let body = ResultRestoreRequest {
generated_id: generated_id.into(),
status: status.into(),
};
let agent_id = agent_id.into();
let path = format!("/agent/{}/restore", agent_id);
self.request_with_body(Method::POST, path.as_str(), &body)
.await
}
}
@@ -0,0 +1,36 @@
use crate::services::api::models::agent;
use crate::services::api::{ApiClient, ApiError};
use agent::status::PingResult;
use anyhow::Result;
use reqwest::Method;
use serde::Serialize;
#[derive(Serialize)]
pub struct DatabasePayload<'a> {
pub name: &'a str,
pub dbms: &'a str,
#[serde(rename = "generatedId")]
pub generated_id: &'a str,
}
#[derive(Serialize)]
pub struct StatusRequest<'a> {
pub version: &'a str,
pub databases: Vec<DatabasePayload<'a>>,
}
impl ApiClient {
pub async fn agent_status<'a>(
&self,
agent_id: impl Into<String>,
version: &'a str,
databases: Vec<DatabasePayload<'a>>,
) -> Result<Option<PingResult>, ApiError> {
let body = StatusRequest { version, databases };
let agent_id = agent_id.into();
let path = format!("/agent/{}/status", agent_id);
self.request_with_body(Method::POST, path.as_str(), &body).await
}
}
+3
View File
@@ -0,0 +1,3 @@
pub mod agent;
pub use agent::status;
+21
View File
@@ -0,0 +1,21 @@
use reqwest::StatusCode;
use thiserror::Error;
#[derive(Debug, Error)]
#[allow(dead_code)]
pub enum ApiError {
#[error("http client error: {0}")]
Http(#[from] reqwest::Error),
#[error("serialization error: {0}")]
Serialization(#[from] serde_json::Error),
#[error("api error: status={status}, body={body}")]
HttpResponse {
status: StatusCode,
body: String,
},
#[error("api returned unexpected response")]
UnexpectedResponse,
}
+8
View File
@@ -0,0 +1,8 @@
pub mod client;
pub mod error;
pub mod models;
pub mod endpoints;
pub use client::ApiClient;
pub use error::ApiError;
+24
View File
@@ -0,0 +1,24 @@
use serde::{Deserialize, Serialize};
#[derive(Debug, Deserialize, Serialize)]
pub struct BackupStorage {
pub id: String,
}
#[derive(Debug, Serialize, Deserialize)]
pub struct BackupUploadResponse {
pub message: String,
#[serde(rename = "backupStorage")]
pub backup_storage: BackupStorage,
}
#[derive(Debug, Deserialize, Serialize)]
pub struct Backup {
pub id: String,
}
#[derive(Debug, Serialize, Deserialize)]
pub struct BackupResponse {
pub message: String,
pub backup: Backup,
}
+3
View File
@@ -0,0 +1,3 @@
pub mod status;
pub mod backup;
pub mod restore;
+7
View File
@@ -0,0 +1,7 @@
use serde::{Deserialize, Serialize};
#[derive(Debug, Serialize, Deserialize)]
pub struct ResultRestoreResponse {
pub message: String,
pub status: bool,
}
+57
View File
@@ -0,0 +1,57 @@
#![allow(dead_code)]
use serde::{Deserialize, Serialize};
use toml::Value;
use crate::utils::deserializer::deserialize_snake_case;
#[derive(Debug, Deserialize)]
pub struct PingResult {
pub agent: AgentInfo,
pub databases: Vec<DatabaseStatus>,
}
#[derive(Debug, Deserialize)]
pub struct AgentInfo {
pub id: String,
#[serde(rename = "lastContact")]
pub last_contact: String,
}
#[derive(Debug, Deserialize, Serialize, Clone)]
#[serde(rename_all = "snake_case")]
pub struct DatabaseStorage {
pub id: String,
#[serde(deserialize_with = "deserialize_snake_case")]
pub config: Value,
pub provider: String,
}
#[derive(Debug, Deserialize)]
pub struct DatabaseStatus {
pub dbms: String,
#[serde(rename = "generatedId")]
pub generated_id: String,
pub storages: Vec<DatabaseStorage>,
pub encrypt: bool,
pub data: DatabaseData,
}
#[derive(Debug, Deserialize)]
pub struct DatabaseData {
pub backup: BackupInfo,
pub restore: RestoreInfo,
}
#[derive(Debug, Deserialize)]
pub struct BackupInfo {
pub action: bool,
pub cron: Option<String>,
}
#[derive(Debug, Deserialize)]
pub struct RestoreInfo {
pub action: bool,
pub file: Option<String>,
#[serde(rename = "metaFile")]
pub meta_file: Option<String>,
}
+2
View File
@@ -0,0 +1,2 @@
pub mod agent;
-219
View File
@@ -1,219 +0,0 @@
#![allow(dead_code)]
use crate::core::context::Context;
use crate::domain::factory::DatabaseFactory;
use crate::services::config::{DatabaseConfig, DatabasesConfig, DbType};
use crate::utils::common::BackupMethod;
use crate::utils::file::full_extension;
use anyhow::Result;
use hex;
use openssl::encrypt::Encrypter;
use openssl::hash::MessageDigest;
use openssl::pkey::PKey;
use openssl::rand::rand_bytes;
use openssl::rsa::Padding;
use openssl::symm::{Cipher, Crypter, Mode};
use reqwest::multipart::{Form, Part};
use std::path::{Path, PathBuf};
use std::sync::Arc;
use tempfile::TempDir;
use tokio::fs;
use tracing::{error, info};
#[derive(Debug)]
pub struct BackupResult {
pub generated_id: String,
pub db_type: DbType,
pub status: String,
pub backup_file: Option<PathBuf>,
pub code: Option<String>,
}
pub struct BackupService {
ctx: Arc<Context>,
}
impl BackupService {
pub fn new(ctx: Arc<Context>) -> Self {
Self { ctx }
}
pub async fn dispatch(
&self,
generated_id: &String,
config: &DatabasesConfig,
method: BackupMethod,
) {
if let Some(cfg) = config
.databases
.iter()
.find(|c| c.generated_id == generated_id.as_str())
{
let db_cfg = cfg.clone();
let ctx_clone = self.ctx.clone();
tokio::spawn(async move {
match TempDir::new() {
Ok(temp_dir) => {
let tmp_path = temp_dir.path().to_path_buf();
info!("Created temp directory {}", tmp_path.display());
match BackupService::run(db_cfg, &tmp_path).await {
Ok(result) => {
let service = BackupService { ctx: ctx_clone };
service.send_result(result, method).await;
}
Err(e) => error!("Backup error {}", e),
}
// TempDir is automatically deleted when dropped here
}
Err(e) => error!("Failed to create temp dir: {}", e),
}
});
}
}
pub async fn run(cfg: DatabaseConfig, tmp_path: &Path) -> Result<BackupResult> {
let db_instance = DatabaseFactory::create_for_backup(cfg.clone()).await;
let generated_id = cfg.generated_id.clone();
let db_type = cfg.db_type.clone();
let reachable = db_instance.ping().await.unwrap_or(false);
info!("Reachable: {}", reachable);
if !reachable {
return Ok(BackupResult {
generated_id,
db_type,
status: "failed".into(),
backup_file: None,
code: None,
});
}
match db_instance.backup(tmp_path).await {
Ok(file) => Ok(BackupResult {
generated_id,
db_type,
status: "success".into(),
backup_file: Some(file),
code: None,
}),
Err(e) => match e.to_string().as_str() {
"backup_already_in_progress" => Ok(BackupResult {
generated_id,
db_type,
status: "failed".into(),
backup_file: None,
code: Some(e.to_string()),
}),
_ => Ok(BackupResult {
generated_id,
db_type,
status: "failed".into(),
backup_file: None,
code: None,
}),
},
}
}
pub async fn send_result(&self, result: BackupResult, method: BackupMethod) {
if result.code.as_deref() == Some("backup_already_in_progress") {
info!(
"[BackupService] Skipping send for DB {}: backup already in progress",
result.generated_id
);
return;
}
info!(
"[BackupService] DB: {} Type: {} Status: {} File: {:?}",
result.generated_id,
result.db_type.as_str(),
result.status,
result.backup_file
);
let client = reqwest::Client::new();
let url = format!(
"{}/api/agent/{}/backup",
self.ctx.edge_key.server_url, self.ctx.edge_key.agent_id
);
let mut form = Form::new()
.text("generatedId", result.generated_id.clone())
.text("status", result.status.clone())
.text("method", method.to_string());
if let Some(file_path) = result.backup_file {
match fs::read(&file_path).await {
Ok(raw_data) => {
// AES key + IV
let mut aes_key = [0u8; 32];
rand_bytes(&mut aes_key).unwrap();
let mut iv = [0u8; 16];
rand_bytes(&mut iv).unwrap();
// AES CBC PKCS7 encryption
let cipher = Cipher::aes_256_cbc();
let mut encrypter =
Crypter::new(cipher, Mode::Encrypt, &aes_key, Some(&iv)).unwrap();
encrypter.pad(true);
let mut encrypted = vec![0u8; raw_data.len() + cipher.block_size()];
let count = encrypter.update(&raw_data, &mut encrypted).unwrap();
let rest = encrypter.finalize(&mut encrypted[count..]).unwrap();
encrypted.truncate(count + rest);
// Encrypt AES key with RSA public key
let pub_key_pem = self.ctx.edge_key.public_key.as_bytes();
let pkey = PKey::public_key_from_pem(pub_key_pem).unwrap();
let mut encrypter = Encrypter::new(&pkey).unwrap();
// Set OAEP padding (default OAEP uses SHA1, so override)
encrypter.set_rsa_padding(Padding::PKCS1_OAEP).unwrap();
// Set OAEP hash to SHA256
encrypter.set_rsa_oaep_md(MessageDigest::sha256()).unwrap();
encrypter.set_rsa_mgf1_md(MessageDigest::sha256()).unwrap();
let mut encrypted_key = vec![0u8; encrypter.encrypt_len(&aes_key).unwrap()];
let encrypted_len = encrypter.encrypt(&aes_key, &mut encrypted_key).unwrap();
encrypted_key.truncate(encrypted_len);
let extension = full_extension(&file_path);
// Attach file and AES info to multipart form
form = form
.part(
"file",
Part::bytes(encrypted)
.file_name(format!("{}.enc", result.generated_id)),
)
.text("aes_key", hex::encode(encrypted_key))
.text("iv", hex::encode(iv))
.text("extension", extension);
}
Err(e) => {
error!("Failed to read backup file: {}", e);
}
}
} else {
form = form.text("file", "");
}
match client.post(&url).multipart(form).send().await {
Ok(resp) => {
let status = resp.status();
if status.is_success() {
info!("Backup result sent successfully");
} else {
let text = resp.text().await.unwrap_or_default(); // consumes resp
error!("Backup result failed, status: {}, body: {}", status, text);
}
}
Err(e) => {
error!("Failed to send backup result: {}", e);
}
}
}
}
+20
View File
@@ -0,0 +1,20 @@
use super::service::BackupService;
use crate::utils::compress::compress_to_tar_gz_large;
use std::path::PathBuf;
use anyhow::Result;
impl BackupService {
pub async fn compress_backup(
&self,
backup_file: Option<PathBuf>,
) -> Result<PathBuf> {
let file = backup_file
.ok_or_else(|| anyhow::anyhow!("No backup file generated"))?;
let compression = compress_to_tar_gz_large(&file).await?;
Ok(compression.compressed_path)
}
}
+44
View File
@@ -0,0 +1,44 @@
use super::service::BackupService;
use crate::services::config::DatabasesConfig;
use crate::services::api::models::agent::status::DatabaseStorage;
use crate::utils::common::BackupMethod;
use tracing::error;
impl BackupService {
pub async fn dispatch(
&self,
generated_id: &String,
config: &DatabasesConfig,
method: BackupMethod,
storages: &Vec<DatabaseStorage>,
encrypt: bool,
) {
let Some(cfg) = config
.databases
.iter()
.find(|c| c.generated_id == generated_id.as_str())
else {
error!("Database config not found for {}", generated_id);
return;
};
let service = Self {
ctx: self.ctx.clone(),
};
let db_cfg = cfg.clone();
let storages = storages.clone();
let generated_id = generated_id.clone();
tokio::spawn(async move {
if let Err(e) = service
.execute_backup(generated_id, db_cfg, method, storages, encrypt)
.await
{
error!("Backup execution failed: {}", e);
}
});
}
}
+49
View File
@@ -0,0 +1,49 @@
use super::service::BackupService;
use crate::services::config::DatabaseConfig;
use crate::services::api::models::agent::status::DatabaseStorage;
use crate::utils::common::BackupMethod;
use crate::utils::locks::FileLock;
use tempfile::TempDir;
use anyhow::Result;
impl BackupService {
pub async fn execute_backup(
&self,
generated_id: String,
db_cfg: DatabaseConfig,
method: BackupMethod,
storages: Vec<DatabaseStorage>,
encrypt: bool,
) -> Result<()> {
if FileLock::is_locked(&generated_id).await? {
anyhow::bail!("backup already running");
}
let backup = self.create_backup_record(&generated_id, &method).await?;
let backup_id = backup.backup.id;
let temp_dir = TempDir::new()?;
let tmp_path = temp_dir.path();
let mut result = Self::run(db_cfg, tmp_path).await?;
if result.status == "failed" {
self.send_result(result, vec![], &backup_id).await?;
return Ok(());
}
let compressed = self.compress_backup(result.backup_file.take()).await?;
result.backup_file = Some(compressed);
let uploads = self
.upload(result.clone(), method, storages, encrypt, &backup_id)
.await?;
self.send_result(result, uploads, &backup_id).await?;
Ok(())
}
}
+26
View File
@@ -0,0 +1,26 @@
use super::service::BackupService;
use crate::utils::common::BackupMethod;
use anyhow::{Result, anyhow};
use crate::services::api::models::agent::backup::BackupResponse;
impl BackupService {
pub async fn create_backup_record(
&self,
generated_id: &str,
method: &BackupMethod,
) -> Result<BackupResponse> {
let response = self
.ctx
.api
.backup_create(
method.to_string(),
self.ctx.edge_key.agent_id.clone(),
generated_id,
)
.await?;
response.ok_or_else(|| anyhow!("backup_create returned empty response"))
}
}
+11
View File
@@ -0,0 +1,11 @@
pub mod service;
pub mod dispatcher;
pub mod executor;
pub mod compressor;
pub mod uploader;
pub mod result;
pub mod models;
pub mod helpers;
pub mod runner;
pub use service::BackupService;
+22
View File
@@ -0,0 +1,22 @@
#![allow(dead_code)]
use std::path::PathBuf;
use crate::services::config::DbType;
#[derive(Debug, Clone)]
pub struct BackupResult {
pub generated_id: String,
pub db_type: DbType,
pub status: String,
pub backup_file: Option<PathBuf>,
pub code: Option<String>,
}
#[derive(Debug)]
pub struct UploadResult {
pub storage_id: String,
pub success: bool,
pub error: Option<String>,
pub remote_file_path: Option<String>,
pub total_size: Option<u64>,
}
+45
View File
@@ -0,0 +1,45 @@
use super::models::{BackupResult, UploadResult};
use super::service::BackupService;
use crate::services::api::ApiError;
use crate::services::api::models::agent::backup::BackupResponse;
use anyhow::Result;
use tracing::error;
impl BackupService {
pub async fn send_result(
&self,
result: BackupResult,
upload_results: Vec<UploadResult>,
backup_id: &String,
) -> Result<Option<BackupResponse>, ApiError> {
let status = if upload_results.iter().any(|r| r.success) {
"success"
} else {
"failed"
};
let file_size = upload_results
.iter()
.filter_map(|r| r.total_size)
.reduce(|a, b| a + b)
.map(|sum| sum / upload_results.len() as u64);
self.ctx
.api
.backup_update(
self.ctx.edge_key.agent_id.clone(),
backup_id,
status,
file_size,
&result.generated_id,
)
.await
.map_err(|e| {
error!(
"backup_update failed (generated_id={}, backup_id={}): {}",
result.generated_id, backup_id, e
);
e.into()
})
}
}
+70
View File
@@ -0,0 +1,70 @@
use super::models::BackupResult;
use super::service::BackupService;
use crate::domain::factory::DatabaseFactory;
use crate::services::config::DatabaseConfig;
use anyhow::Result;
use std::path::Path;
use tracing::{error, info};
impl BackupService {
pub async fn run(
cfg: DatabaseConfig,
tmp_path: &Path,
) -> Result<BackupResult> {
let db = DatabaseFactory::create_for_backup(cfg.clone()).await;
let generated_id = cfg.generated_id.clone();
let db_type = cfg.db_type.clone();
let reachable = match db.ping().await {
Ok(v) => v,
Err(e) => {
error!("Ping failed: {}", e);
return Err(e.into());
}
};
info!("Reachable: {}", reachable);
if !reachable {
return Ok(BackupResult {
generated_id,
db_type,
status: "failed".into(),
backup_file: None,
code: None,
});
}
match db.backup(tmp_path, Some(false)).await {
Ok(file) => Ok(BackupResult {
generated_id,
db_type,
status: "success".into(),
backup_file: Some(file),
code: None,
}),
Err(e) if e.to_string() == "backup_already_in_progress" => Ok(BackupResult {
generated_id,
db_type,
status: "failed".into(),
backup_file: None,
code: Some("backup_already_in_progress".into()),
}),
Err(_) => Ok(BackupResult {
generated_id,
db_type,
status: "failed".into(),
backup_file: None,
code: None,
}),
}
}
}
+12
View File
@@ -0,0 +1,12 @@
use std::sync::Arc;
use crate::core::context::Context as CoreContext;
pub struct BackupService {
pub ctx: Arc<CoreContext>,
}
impl BackupService {
pub fn new(ctx: Arc<CoreContext>) -> Self {
Self { ctx }
}
}
+181
View File
@@ -0,0 +1,181 @@
use super::service::BackupService;
use super::models::{BackupResult, UploadResult};
use crate::services::storage;
use crate::services::api::models::agent::status::DatabaseStorage;
use crate::utils::common::BackupMethod;
use futures::future::join_all;
use anyhow::{Result, bail};
use tracing::{info, error};
impl BackupService {
pub async fn upload(
&self,
result: BackupResult,
method: BackupMethod,
storages: Vec<DatabaseStorage>,
encrypt: bool,
backup_id: &String,
) -> Result<Vec<UploadResult>> {
if result.code.as_deref() == Some("backup_already_in_progress") {
info!("Skipping send: backup already in progress");
bail!("backup_already_in_progress");
}
let ctx = self.ctx.clone();
let futures = storages.into_iter().map(|storage| {
let ctx_clone = ctx.clone();
let result_clone = result.clone();
let provider = storage::get_provider(&storage);
let storage_id = storage.id.clone();
let generated_id = result_clone.generated_id.clone();
async move {
info!("Uploading storage -> {:?} for {:?}", storage.provider, storage_id);
/*
INIT STEP
*/
let init = match ctx_clone.api
.backup_upload_init(
ctx_clone.edge_key.agent_id.clone(),
generated_id.clone(),
storage_id.clone(),
backup_id,
)
.await
{
Ok(v) => v,
Err(e) => {
error!("backup_upload_init failed: {}", e);
return UploadResult {
storage_id,
success: false,
error: Some("backup_upload_init failed".into()),
remote_file_path: None,
total_size: None,
};
}
};
let backup_storage_id = match init {
Some(v) => v.backup_storage.id,
None => {
return UploadResult {
storage_id,
success: false,
error: Some("backup_upload_init returned empty response".into()),
remote_file_path: None,
total_size: None,
};
}
};
/*
PROVIDER CHECK
*/
let Some(provider) = provider else {
error!("Skipping storage due to missing provider");
return UploadResult {
storage_id,
success: false,
error: Some("missing provider".into()),
remote_file_path: None,
total_size: None,
};
};
/*
STORAGE UPLOAD
*/
let upload_result = provider
.upload(
ctx_clone.clone(),
result_clone,
method,
&storage,
Some(encrypt),
)
.await;
let status = if upload_result.success { "success" } else { "failed" };
if status != "success" {
return upload_result;
}
info!(
"Storage {} uploaded to remote path {:?}",
storage_id,
upload_result.remote_file_path
);
/*
METADATA VALIDATION
*/
let (remote_path, total_size) = match (
&upload_result.remote_file_path,
upload_result.total_size,
) {
(Some(path), Some(size)) => (path.clone(), size),
_ => {
return UploadResult {
storage_id,
success: false,
error: Some("remote_file_path or total_size missing".into()),
remote_file_path: None,
total_size: None,
};
}
};
/*
STATUS UPDATE
*/
match ctx_clone.api.backup_upload_status(
ctx_clone.edge_key.agent_id.clone(),
generated_id,
backup_storage_id,
status,
remote_path,
total_size,
backup_id,
).await {
Ok(_) => upload_result,
Err(err) => {
error!(
"backup_upload_status failed (storage_id={}): {}",
storage_id,
err
);
UploadResult {
storage_id,
success: false,
error: Some(err.to_string()),
remote_file_path: None,
total_size: None,
}
}
}
}
});
let results: Vec<UploadResult> = join_all(futures).await;
info!("Upload results: {:#?}", results);
Ok(results)
}
}
+96 -6
View File
@@ -9,6 +9,7 @@ use std::path::Path;
use std::sync::Arc;
use toml;
use tracing::info;
use uuid::Uuid;
#[derive(Debug, Deserialize, Clone)]
#[serde(rename_all = "lowercase")]
@@ -16,8 +17,9 @@ pub enum DbType {
Mysql,
Mariadb,
Postgresql,
MongoDB
// Sqlite,
MongoDB,
Sqlite,
Redis
// Add other DB types if needed
}
@@ -28,7 +30,8 @@ impl DbType {
DbType::Mariadb => "mysql",
DbType::Postgresql => "postgresql",
DbType::MongoDB => "mongodb",
// DbType::Sqlite => "sqlite",
DbType::Sqlite => "sqlite",
DbType::Redis => "redis",
}
}
}
@@ -45,6 +48,7 @@ pub struct DatabaseConfig {
pub port: u16,
pub host: String,
pub generated_id: String,
pub path: String,
}
#[allow(dead_code)]
@@ -53,6 +57,29 @@ pub struct DatabasesConfig {
pub databases: Vec<DatabaseConfig>,
}
#[allow(dead_code)]
#[derive(Debug, Deserialize, Clone)]
pub struct InputDatabaseConfig {
pub name: String,
pub database: Option<String>,
#[serde(rename = "type")]
pub db_type: DbType,
pub username: Option<String>,
pub password: Option<String>,
pub port: Option<u16>,
pub host: Option<String>,
pub generated_id: String,
pub path: Option<String>,
}
#[allow(dead_code)]
#[derive(Debug, Deserialize, Clone)]
pub struct InputDatabasesConfig {
pub databases: Vec<InputDatabaseConfig>,
}
pub struct ConfigService {
ctx: Arc<Context>,
}
@@ -95,7 +122,7 @@ impl ConfigService {
file.read_to_string(&mut contents)
.map_err(|e| format!("Failed to read config file: {}", e))?;
let config: DatabasesConfig = match extension {
let input_config: InputDatabasesConfig = match extension {
"json" => {
serde_json::from_str(&contents).map_err(|e| format!("JSON parsing error: {}", e))?
}
@@ -105,8 +132,71 @@ impl ConfigService {
_ => return Err("Unsupported config file format. Use .json or .toml".to_string()),
};
info!("Databases : {:?} instances loaded", config.databases.len());
fn required<T: Clone>(opt: &Option<T>, db_name: &str, field_name: &str) -> Result<T, String> {
match opt {
Some(v) => Ok(v.clone()),
None => {
let msg = format!("Missing required field '{}' for database '{}'", field_name, db_name);
Err(msg)
}
}
}
Ok(config)
fn optional<T: Clone>(opt: &Option<T>) -> T where T: Default {
opt.clone().unwrap_or_default()
}
let mut databases = Vec::with_capacity(input_config.databases.len());
for db in input_config.databases {
if Uuid::parse_str(&db.generated_id).is_err() {
return Err(format!("Invalid UUID for database '{}'", db.name));
}
let username = match db.db_type {
DbType::Postgresql | DbType::Mysql | DbType::Mariadb => required(&db.username, &db.name, "username")?,
_ => optional(&db.username),
};
let password = match db.db_type {
DbType::Postgresql | DbType::Mysql | DbType::Mariadb => required(&db.password, &db.name, "password")?,
_ => optional(&db.password),
};
let host = match db.db_type {
DbType::Postgresql | DbType::Mysql | DbType::Mariadb | DbType::MongoDB | DbType::Redis => required(&db.host, &db.name, "host")?,
DbType::Sqlite => optional(&db.host),
};
let port = match db.db_type {
DbType::Postgresql | DbType::Mysql | DbType::Mariadb | DbType::MongoDB | DbType::Redis => required(&db.port, &db.name, "port")?,
DbType::Sqlite => db.port.unwrap_or(0),
};
let database_name = match db.db_type {
DbType::Sqlite | DbType::Redis => optional(&db.database),
_ => required(&db.database, &db.name, "database")?
};
let path_val = match db.db_type {
DbType::Sqlite => required(&db.path, &db.name, "path")?,
_ => optional(&db.path),
};
databases.push(DatabaseConfig {
name: db.name,
database: database_name,
db_type: db.db_type,
username,
password,
host,
port,
generated_id: db.generated_id,
path: path_val,
});
}
info!("Databases: {} instances loaded", databases.len());
Ok(DatabasesConfig { databases })
}
}
+13 -3
View File
@@ -1,11 +1,13 @@
#![allow(dead_code)]
use crate::core::context::Context;
use crate::services::status::DatabaseStatus;
use crate::utils::common::vec_to_option_json;
use crate::utils::redis_client;
use crate::utils::task_manager::cron::check_and_update_cron;
use std::sync::Arc;
use redis::aio::MultiplexedConnection;
use serde_json::{Value, json};
use std::sync::Arc;
use crate::services::api::models::agent::status::DatabaseStatus;
pub struct CronService {
ctx: Arc<Context>,
@@ -23,6 +25,12 @@ impl CronService {
let dbms = database.dbms.as_str();
let task_name = format!("periodic.backup_{}", generated_id);
let args = vec![generated_id.to_string(), dbms.to_string()];
let storages: Option<Value> = vec_to_option_json(database.storages.clone());
let encrypt: bool = database.encrypt;
let metadata = json!({
"storages": storages,
"encrypt": encrypt
});
check_and_update_cron(
&mut self.conn,
@@ -30,7 +38,9 @@ impl CronService {
args,
"tasks.database.periodic_backup",
task_name,
).await;
Option::from(metadata),
)
.await;
Ok(true)
}
+4 -2
View File
@@ -1,5 +1,7 @@
pub mod config;
pub mod status;
pub mod cron;
pub mod backup;
pub mod restore;
pub mod restore;
mod storage;
pub mod api;
pub mod status;
-162
View File
@@ -1,162 +0,0 @@
#![allow(dead_code)]
use crate::core::context::Context;
use crate::domain::factory::DatabaseFactory;
use crate::services::config::{DatabaseConfig, DatabasesConfig};
use crate::services::status::DatabaseStatus;
use anyhow::Result;
use tracing::{error, info};
use serde::Serialize;
use std::path::Path;
use std::sync::Arc;
use tempfile::TempDir;
#[derive(Debug, Serialize)]
pub struct RestoreResult {
#[serde(rename = "generatedId")]
pub generated_id: String,
pub status: String,
}
pub struct RestoreService {
ctx: Arc<Context>,
}
impl RestoreService {
pub fn new(ctx: Arc<Context>) -> Self {
Self { ctx }
}
pub async fn dispatch(&self, db: &DatabaseStatus, config: &DatabasesConfig) {
if let Some(cfg) = config
.databases
.iter()
.find(|c| c.generated_id == db.generated_id)
{
let db_cfg = cfg.clone();
let ctx_clone = self.ctx.clone();
let file_to_restore = db.data.restore.file.clone();
tokio::spawn(async move {
match TempDir::new() {
Ok(temp_dir) => {
let tmp_path = temp_dir.path().to_path_buf();
info!("Created temp directory {}", tmp_path.display());
match RestoreService::run(db_cfg, &tmp_path, &file_to_restore).await {
Ok(result) => {
let service = RestoreService { ctx: ctx_clone };
service.send_result(result).await;
}
Err(e) => error!("Restoration error {}", e),
}
// TempDir is automatically deleted when dropped here
}
Err(e) => error!("Failed to create temp dir: {}", e),
}
});
}
}
pub async fn run(
cfg: DatabaseConfig,
tmp_path: &Path,
file_url: &str,
) -> Result<RestoreResult> {
let generated_id = cfg.generated_id.clone();
info!("File url: {}", file_url);
let client = reqwest::Client::new();
let response = client.get(file_url).send().await?;
if !response.status().is_success() {
error!("Backup download failed with status {}", response.status());
return Ok(RestoreResult {
generated_id,
status: "failed".into(),
});
}
let bytes = response.bytes().await?;
let ext = if bytes.starts_with(b"PGDMP") {
// Postgres custom format
"dump"
} else if bytes.starts_with(&[0x1F, 0x8B]) {
// gzip compressed -> could be Postgres directory dump or MySQL gzipped SQL
"tar.gz"
} else if bytes.starts_with(b"--") || bytes.starts_with(b"/*") {
// Plain MySQL SQL dump
"sql"
} else {
// Fallback generic
"dump"
};
info!("Backup dump from {} to {}", tmp_path.display(), ext);
let backup_file_path = tmp_path.join(format!("backup_file_tmp.{}", ext));
tokio::fs::write(&backup_file_path, &bytes).await?;
info!("Backup downloaded to {}", backup_file_path.display());
let db_instance = DatabaseFactory::create_for_restore(cfg.clone(), &backup_file_path).await;
let reachable = db_instance.ping().await.unwrap_or(false);
info!("Reachable: {}", reachable);
if !reachable {
return Ok(RestoreResult {
generated_id,
status: "failed".into(),
});
}
match db_instance.restore(&backup_file_path).await {
Ok(_) => Ok(RestoreResult {
generated_id,
status: "success".into(),
}),
Err(e) => {
log::error!("Restore failed: {:?}", e);
Ok(RestoreResult {
generated_id,
status: "failed".into(),
})
}
}
}
pub async fn send_result(&self, result: RestoreResult) {
info!(
"[RestoreService] DB: {} | Status: {}",
result.generated_id, result.status,
);
let client = reqwest::Client::new();
let url = format!(
"{}/api/agent/{}/restore",
self.ctx.edge_key.server_url, self.ctx.edge_key.agent_id
);
let body = RestoreResult {
generated_id: result.generated_id,
status: result.status,
};
match client.post(&url).json(&body).send().await {
Ok(resp) => {
let status = resp.status();
if status.is_success() {
info!("Restoration result sent successfully");
} else {
let text = resp.text().await.unwrap_or_default(); // consumes resp
error!(
"Restoration result failed, status: {}, body: {}",
status, text
);
}
}
Err(e) => {
error!("Failed to send restoration result: {}", e);
}
}
}
}
+61
View File
@@ -0,0 +1,61 @@
use super::service::RestoreService;
use crate::utils::compress::decompress_large_tar_gz;
use crate::utils::file::decrypt_file_stream_gcm;
use anyhow::Result;
use std::path::{Path, PathBuf};
impl RestoreService {
pub async fn prepare_archive(
&self,
downloaded_file: PathBuf,
tmp_path: &Path,
) -> Result<PathBuf> {
let filename = downloaded_file
.file_name()
.unwrap()
.to_string_lossy()
.to_string();
let is_legacy = filename.ends_with(".sql") || filename.ends_with(".dump");
if is_legacy {
return Ok(downloaded_file);
}
let encrypted = filename.ends_with(".tar.gz.enc");
let mut archive = downloaded_file.clone();
if encrypted {
let new_name = filename.strip_suffix(".enc").unwrap();
let decrypted = tmp_path.join(new_name);
decrypt_file_stream_gcm(
downloaded_file,
decrypted.clone(),
self.ctx.edge_key.master_key_b64.clone(),
)
.await?;
archive = decrypted;
}
let files = decompress_large_tar_gz(archive.as_path(), tmp_path).await?;
if files.is_empty() {
anyhow::bail!("archive empty");
}
if files.len() == 1 {
Ok(files[0].clone())
} else {
Ok(archive)
}
}
}
+42
View File
@@ -0,0 +1,42 @@
use super::service::RestoreService;
use crate::services::config::DatabasesConfig;
use crate::services::api::models::agent::status::DatabaseStatus;
use tracing::error;
impl RestoreService {
pub async fn dispatch(&self, db: &DatabaseStatus, config: &DatabasesConfig) {
let Some(cfg) = config
.databases
.iter()
.find(|c| c.generated_id == db.generated_id)
else {
error!("Database config not found");
return;
};
let Some(file_to_restore) = db.data.restore.file.clone() else {
error!("restore file not found");
return;
};
let service = Self {
ctx: self.ctx.clone(),
};
let db_cfg = cfg.clone();
tokio::spawn(async move {
if let Err(e) = service
.execute_restore(db_cfg, file_to_restore)
.await
{
error!("Restore failed: {}", e);
}
});
}
}
+53
View File
@@ -0,0 +1,53 @@
use super::service::RestoreService;
use reqwest::{Client, Url};
use anyhow::Result;
use std::path::{Path, PathBuf};
use tracing::info;
impl RestoreService {
pub async fn download_backup(
&self,
file_url: &str,
tmp_path: &Path,
) -> Result<PathBuf> {
let client = Client::new();
let response = client.get(file_url).send().await?;
if !response.status().is_success() {
anyhow::bail!("download failed");
}
let filename_from_header = response
.headers()
.get(reqwest::header::CONTENT_DISPOSITION)
.and_then(|v| v.to_str().ok())
.and_then(|s| s.split("filename=").nth(1))
.map(|f| f.trim_matches('"').to_string());
let filename_from_url = Url::parse(file_url).ok().and_then(|u| {
u.path_segments()?
.last()
.filter(|s| !s.is_empty())
.map(|s| s.to_string())
});
let filename = filename_from_header
.or(filename_from_url)
.unwrap_or_else(|| "downloaded_file".to_string());
let path = tmp_path.join(&filename);
let bytes = response.bytes().await?;
tokio::fs::write(&path, &bytes).await?;
info!("Backup downloaded to {}", path.display());
Ok(path)
}
}
+30
View File
@@ -0,0 +1,30 @@
use super::service::RestoreService;
use crate::services::config::DatabaseConfig;
use tempfile::TempDir;
use anyhow::Result;
use tracing::info;
impl RestoreService {
pub async fn execute_restore(
&self,
cfg: DatabaseConfig,
file_url: String,
) -> Result<()> {
let temp_dir = TempDir::new()?;
let tmp_path = temp_dir.path();
info!("Created temp directory {}", tmp_path.display());
let downloaded = self.download_backup(&file_url, tmp_path).await?;
let backup_file = self.prepare_archive(downloaded, tmp_path).await?;
let result = self.run_restore(cfg, backup_file).await?;
self.send_result(result).await;
Ok(())
}
}
+10
View File
@@ -0,0 +1,10 @@
pub mod service;
pub mod dispatcher;
pub mod executor;
pub mod downloader;
pub mod archive;
pub mod runner;
pub mod result;
pub mod models;
pub use service::RestoreService;
+8
View File
@@ -0,0 +1,8 @@
use serde::Serialize;
#[derive(Debug, Serialize)]
pub struct RestoreResult {
#[serde(rename = "generatedId")]
pub generated_id: String,
pub status: String,
}
+30
View File
@@ -0,0 +1,30 @@
use super::service::RestoreService;
use super::models::RestoreResult;
use tracing::{info, error};
impl RestoreService {
pub async fn send_result(&self, result: RestoreResult) {
info!(
"[RestoreService] DB: {} | Status: {}",
result.generated_id, result.status
);
match self.ctx
.api
.restore_result(
self.ctx.edge_key.agent_id.clone(),
&result.generated_id,
&result.status,
)
.await {
Ok(_) => {
info!("Restoration result sent successfully");
}
Err(e) => {
error!("Failed to send restoration result: {}", e);
}
}
}
}
+51
View File
@@ -0,0 +1,51 @@
use super::service::RestoreService;
use super::models::RestoreResult;
use crate::domain::factory::DatabaseFactory;
use crate::services::config::DatabaseConfig;
use anyhow::Result;
use std::path::PathBuf;
use tracing::{info, error};
impl RestoreService {
pub async fn run_restore(
&self,
cfg: DatabaseConfig,
backup_file: PathBuf,
) -> Result<RestoreResult> {
let generated_id = cfg.generated_id.clone();
let db = DatabaseFactory::create_for_restore(cfg.clone(), &backup_file).await;
let reachable = db.ping().await.unwrap_or(false);
info!("Reachable: {}", reachable);
if !reachable {
return Ok(RestoreResult {
generated_id,
status: "failed".into(),
});
}
match db.restore(&backup_file, Some(false)).await {
Ok(_) => Ok(RestoreResult {
generated_id,
status: "success".into(),
}),
Err(e) => {
error!("Restore failed: {:?}", e);
Ok(RestoreResult {
generated_id,
status: "failed".into(),
})
}
}
}
}

Some files were not shown because too many files have changed in this diff Show More