Compare commits

..

6 Commits

Author SHA1 Message Date
trinity-1686a 45f44023a8 and again 2026-03-05 23:44:11 +01:00
trinity-1686a 173a54a83c identical but maybe valid 2026-03-05 23:41:20 +01:00
trinity-1686a 41c943a6b1 this schema is known-broken, but hey 2026-03-05 12:40:05 +01:00
trinity-1686a 84c1e189c4 this schema is known-broken, but hey 2026-03-05 11:20:42 +01:00
trinity-1686a dc8355d0f3 edit openapi by hand 😭 2026-03-05 11:03:58 +01:00
trinity-1686a 1c12ca4caf modify schema in a compatible way to maybe fix typescript sdk 2026-03-05 00:13:55 +01:00
281 changed files with 4893 additions and 10793 deletions
+7 -18
View File
@@ -2,14 +2,13 @@ labels:
nix: "enabled"
when:
- event:
- tag
- pull_request
- deployment
- cron
- manual
- event: push
branch: main-*
event:
- push
- tag
- pull_request
- deployment
- cron
- manual
steps:
- name: check formatting
@@ -17,16 +16,6 @@ steps:
commands:
- nix-build -j4 --attr flakePackages.fmt
- name: check typos
image: nixpkgs/nix:nixos-24.05
commands:
- nix-shell --attr ci --run typos
- name: check lints with clippy
image: nixpkgs/nix:nixos-24.05
commands:
- nix-build -j4 --attr flakePackages.clippy
- name: build
image: nixpkgs/nix:nixos-24.05
commands:
-231
View File
@@ -1,231 +0,0 @@
# Contributing to Garage
## Policy on AI
To ensure the quality of the codebase and documentation, the use of AI,
including LLMs and coding agents, is strictly restricted in the following way:
- AI **must not** be used to write documentation
- **Do not** use AI to write bug reports, commit descriptions and pull request
messages
- **Do not** use AI agents to make contributions to Garage, all contributions
must be led by a human that know what they are doing at all times
- AI **may** be used for some tedious code generation tasks, limited to very
mechanical translations from API docs or boilerplate writing. The code
generated must be so simple as to make it clear that it cannot be covered by
copyright.
You are free to make use of AI privately to explore the codebase and solve
conceptual problems, but please restrain from copying the output from an LLM
anywhere in your code or on the issue tracker, or from letting an agent edit
the codebase directly.
## Asking questions
Read the documentation before asking questions.
Do not use the issue tracker to ask questions about Garage.
Questions asked on the issue tracker will be closed.
Ask questions on the Matrix channel `#garage:deuxfleurs.fr` so that any
community member can see your question and help you out.
If you need in-depth support from the Garage developers specifically, write to
`garagehq@deuxfleurs.fr`. Even if you do so, we do not commit to giving you an
answer.
## Reporting bugs
When writing a bug report, use this checklist:
- For bugs that can be reproduced:
- confirm that you are using the latest version of Garage and that the bug still exists in this version
- set the log level to debug using the `RUST_LOG=garage=debug` environment variable and reproduce the bug to get more verbose logs
- Check whether there is already an open issue in the bug tracker. If so, your bug report is still valuable but please add it as a comment to the existing issue instead of opening a new one.
- Collect as much information as possible:
- logs of the Garage daemon at the time the issue happened, including logs that show what was happening before the issue occurred
- the output of `garage status`
- the output of `garage stats -a`
- the output of `garage layout history`
- Write a detailed bug report, including:
- a description of your cluster (number of nodes, hardware, operating system, networking, etc)
- a detailed description of what you did that led to the issue, including any code or command line that invoked a Garage API
- what you were expecting
- what actually happened, and how that's different from what you expected
- the information collected previously
- if possible, simple steps to help the developers reproduce the issue locally
Bug reports that are imprecise or otherwise unactionable will be closed.
## Suggesting new features
Garage can be improved in many ways, but just suggesting a new feature does not mean we will implement it.
Feature requests that may lead to an actual implementation are feature requests that:
- are precise and actionable, i.e. include a precise description of the expected behavior and any necessary architectural details required for the implementation
- are motivated by actual need from a variety of users
Moreover, a certain number of features are defined as out-of-scope for Garage, including but not limited to:
- extensions to the S3 API that are not present on AWS
- features that require the implementation of a consensus algorithm
- more generally, features that are incompatible with the architecture of Garage and its goal of staying simple
Only feature requests in one of the following category may stay open in the issue tracker:
- features that the Garage team wants to work on
- features that are being actively worked on by an external contributor which is clearly identified
- features that are easy to implement and could be an easy task for a new contributor that wants to get to know the codebase
All other feature requests will be closed after a few months of inactivity, so as to keep the number of open issues to a manageable level.
Feature requests that are clearly out of scope will be closed directly.
## Improving the documentation
An easy way to contribute to Garage which also adds a lot of value is to
improve the documentation. Make sure to write in clear technical English, and
write unambiguously. Documentation contributions are very appreciated if they
are well-written.
## For developers
We welcome code contributions to Garage that adhere to our standards for quality:
- Changes should be reviewed from a functional perspective to ensure that they work well with the existing codebase and do not introduce bugs or subtle issues.
- You must have tested your contribution to make sure that it does what it says. The amount of testing required is proportional to the complexity of the change introduced.
- Any new feature must be properly documented following existing practices (see below).
- Unit tests should be included when relevant.
- Contributions should pass basic lints for syntactic quality (`cargo fmt`, `cargo clippy`, `typos`).
- Contributions should pass our CI test suite.
- No user-facing breaking changes may be introduced between major releases.
- No internal data model change may be introduced between major releases, to
ensure that Garage daemons with different minor/patch versions numbers can
work together in a cluster. For major releases, a proper migration path
should be implemented and tested thoroughly.
Please follow up on your work when changes are requested, to avoid stale PRs.
Do not take it personally if a Garage developer pushes directly to your branch
to modify your contribution, as this might be necessary to get it merged
faster.
### Properly documenting your contribution
#### Configuration options
New configuration options should be documented in
`doc/book/reference-manual/configuration.md`. The documentation for a
configuration option should be exhaustive. For instance, for choice options all
choices should be listed explicitly with a precise description of their
meaning.
In terms of syntax, all configuration options should appear in three places:
- in the example at the top, with an example value
- in the index of all configuration options which is sorted by alphabetical order
- in its dedicated subsection with full reference text
#### CLI commands and command flags
CLI commands are self-documented using the doc commends in the codebase.
Make sure to write clear and precise comments for all options you are adding.
#### S3 features
If you implement new S3 features, make sure to update the compatibility matrix in `doc/book/reference-manual/s3-compatibility.md`.
#### Admin API
The admin API has an OpenAPI specification that is automatically generated
using Utoipa, from a description of each endpoint that is given in
`src/api/admin/openapi.rs` and a description of data structure schemas in
`src/api/admin/api.rs`. The code in `openapi.rs` is only used to generate the
OpenAPI specification document and not for the actual implementation in Garage,
whereas structures defined in `api.rs` are also used for the implementation of
API calls. Make sure to write good doc comments for all of these items so that
the OpenAPI specification will be precise and accurate.
An up-to-date version of the OpenAPI specification document should be kept in
the repository in `doc/api/garage-admin-v2.json`. When you are making changes
to the admin API, update this document with the following command:
```
cargo run -- admin-api-schema > doc/api/garage-admin-v2.json
```
## Garage team organization
Alex (handle `lx`) is the lead developer and is responsible of ensuring the
correctness of Garage and stability between version upgrades.
The other maintainers are Trinity (handle `trinity-1686a`), Quentin (handle `quentin`) and Maximilien (handle `halfa`).
Maximilien is responsible for coordinating effort on the Kubernetes integration / Helm chart.
## Pull request merging criteria
The following PRs should only be merged after review and approval from Alex:
- PRs that introduce architectural changes, such as changes in the data model
or change in the coordination protocols between nodes
- PRs that introduce changes on the format of data structures used for
persistent disk storage and internal cluster communication (RPC)
- PRs that are suspected of introducing some kind of breakage or unexpected
behavior due to their complexity
PRs that introduce breaking change for users but don't fall in one of the
previous category should be discussed between maintainers to evaluate the
impact on users when upgrading. Alex's approval is not required to merge them
as long as they are clearly identified as breaking in the PR title, and are
properly merged in the branch for the next major version and not in the current
main branch.
All other PRs can be merged by any maintainer on their own, once they are
confident that the quality standards defined in this document are respected
before merging.
## Merging strategy
When merging PRs, maintainers should ensure that a Git commit is created by
Forgejo that records the PR number, its title and its text in the commit
message. If a PR is fixing an issue, make sure that the issue number is
included in the PR title as well. This is to ensure that when releasing a new
version of Garage, the changelog in the release notes can be properly
constructed by reading the Git log since the last release.
We also want to keep the history "almost linear" to facilitate the use of `git
bisect` if it ever were necessary. This leaves the following two merging
strategies:
- For PRs that consist of many commits that should stay independent, the
"rebase and create merge commit" strategy should be used. The merge commit is
created automatically by Forgejo and saves the PR's number, title and text in
the commit message.
- For PRs that consist of only one commit, or a few number of commits that can
be merged, the "create squash commit" strategy should be used. This way a
single commit will be created by Forgejo which also saves the PR's number,
title and text in the commit message.
When cherry-picking commits from one branch to the other, a simple fast-forward
merging strategy can be used if the commit message already references a PR
number.
Generated
+920 -1478
View File
File diff suppressed because it is too large Load Diff
+67 -108
View File
@@ -16,7 +16,6 @@ members = [
"src/garage",
"src/k2v-client",
"src/format-table",
"fuzz",
]
default-members = ["src/garage"]
@@ -25,175 +24,135 @@ default-members = ["src/garage"]
# Internal Garage crates
format_table = { version = "0.1.1", path = "src/format-table" }
garage_api_common = { version = "2.3.0", path = "src/api/common" }
garage_api_admin = { version = "2.3.0", path = "src/api/admin" }
garage_api_s3 = { version = "2.3.0", path = "src/api/s3" }
garage_api_k2v = { version = "2.3.0", path = "src/api/k2v" }
garage_block = { version = "2.3.0", path = "src/block" }
garage_db = { version = "2.3.0", path = "src/db", default-features = false }
garage_model = { version = "2.3.0", path = "src/model", default-features = false }
garage_net = { version = "2.3.0", path = "src/net" }
garage_rpc = { version = "2.3.0", path = "src/rpc" }
garage_table = { version = "2.3.0", path = "src/table" }
garage_util = { version = "2.3.0", path = "src/util" }
garage_web = { version = "2.3.0", path = "src/web" }
garage_api_common = { version = "2.2.0", path = "src/api/common" }
garage_api_admin = { version = "2.2.0", path = "src/api/admin" }
garage_api_s3 = { version = "2.2.0", path = "src/api/s3" }
garage_api_k2v = { version = "2.2.0", path = "src/api/k2v" }
garage_block = { version = "2.2.0", path = "src/block" }
garage_db = { version = "2.2.0", path = "src/db", default-features = false }
garage_model = { version = "2.2.0", path = "src/model", default-features = false }
garage_net = { version = "2.2.0", path = "src/net" }
garage_rpc = { version = "2.2.0", path = "src/rpc" }
garage_table = { version = "2.2.0", path = "src/table" }
garage_util = { version = "2.2.0", path = "src/util" }
garage_web = { version = "2.2.0", path = "src/web" }
k2v-client = { version = "0.0.4", path = "src/k2v-client" }
# External crates from crates.io
arc-swap = "1.8"
arbitrary = { version = "1.4.2"}
arc-swap = "1.0"
argon2 = "0.5"
async-trait = "0.1"
async-trait = "0.1.7"
backtrace = "0.3"
base64 = "0.22"
base64 = "0.21"
blake2 = "0.10"
bytes = "1.11"
bytesize = "2.3"
bytes = "1.0"
bytesize = "1.1"
cfg-if = "1.0"
chrono = { version = "0.4", features = ["serde"] }
crc-fast = "1.9"
crc-fast = "1.6"
crypto-common = "0.1"
fundu = "2.0"
fundu-systemd = "0.3"
gethostname = "1.1"
git-version = "0.3"
gethostname = "0.4"
git-version = "0.3.4"
hex = "0.4"
hexdump = "0.1"
html-escape = "0.2.13"
hmac = "0.12"
itertools = "0.14"
ipnet = "2.11"
lazy_static = "1.5"
libfuzzer-sys = "0.4"
itertools = "0.12"
ipnet = "2.9.0"
lazy_static = "1.4"
md-5 = "0.10"
mktemp = "0.5"
nix = { version = "0.31", default-features = false, features = ["fs"] }
nom = "8.0"
nix = { version = "0.29", default-features = false, features = ["fs"] }
nom = "7.1"
parking_lot = "0.12"
parse_duration = "2.1"
paste = "1.0"
pin-project = "1.1"
pnet_datalink = "0.35"
rand = "0.9"
pin-project = "1.0.12"
pnet_datalink = "0.34"
rand = "0.8"
sha1 = "0.10"
sha2 = "0.10"
timeago = { version = "0.5", default-features = false }
timeago = { version = "0.4", default-features = false }
xxhash-rust = { version = "0.8", default-features = false, features = ["xxh3"] }
aes-gcm = { version = "0.10", features = ["aes", "stream"] }
sodiumoxide = { version = "0.2.5-0", package = "kuska-sodiumoxide" }
kuska-handshake = { version = "0.2.0", features = ["default", "async_std"] }
clap = { version = "4.5", features = ["derive", "env"] }
clap = { version = "4.1", features = ["derive", "env"] }
pretty_env_logger = "0.5"
structopt = { version = "0.3", default-features = false }
syslog-tracing = "0.3"
tracing = "0.1"
tracing-journald = "0.3"
tracing-journald = "0.3.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
heed = { version = "0.22", default-features = false, features = [] }
rusqlite = { version = "0.38", features = ["fallible_uint"] }
heed = { version = "0.11", default-features = false, features = ["lmdb"] }
rusqlite = "0.37"
r2d2 = "0.8"
r2d2_sqlite = "0.32"
fjall = "2.11"
r2d2_sqlite = "0.31"
fjall = "2.4"
async-compression = { version = "0.4", features = ["tokio", "zstd"] }
zstd = { version = "0.13", default-features = false }
quick-xml = { version = "0.39", features = ["serialize"] }
rmp-serde = "1.3"
quick-xml = { version = "0.26", features = [ "serialize" ] }
rmp-serde = "1.1.2"
serde = { version = "1.0", default-features = false, features = ["derive", "rc"] }
serde_bytes = "0.11"
serde_json = "1.0"
toml = { version = "0.9", default-features = false, features = ["parse", "serde"] }
utoipa = { version = "5.4", features = ["chrono"] }
toml = { version = "0.8", default-features = false, features = ["parse"] }
utoipa = { version = "5.3.1", features = ["chrono"] }
# newer version requires rust edition 2021
k8s-openapi = { version = "0.27", features = ["v1_35"] }
kube = { version = "3.0", default-features = false, features = [
"runtime",
"derive",
"client",
"rustls-tls",
] }
schemars = "1.2"
reqwest = { version = "0.13", default-features = false, features = [
"rustls-no-provider",
"json",
] }
k8s-openapi = { version = "0.21", features = ["v1_24"] }
kube = { version = "0.88", default-features = false, features = ["runtime", "derive", "client", "rustls-tls"] }
schemars = "0.8"
reqwest = { version = "0.11", default-features = false, features = ["rustls-tls-manual-roots", "json"] }
form_urlencoded = "1.2"
http = "1.4"
form_urlencoded = "1.0.0"
http = "1.0"
httpdate = "1.0"
http-range = "0.1"
http-body-util = "0.1"
hyper = { version = "1.8", default-features = false }
hyper-util = { version = "0.1", features = ["full"] }
multer = "3.1"
percent-encoding = "2.3"
roxmltree = "0.21"
url = "2.5"
hyper = { version = "1.0", default-features = false }
hyper-util = { version = "0.1", features = [ "full" ] }
multer = "3.0"
percent-encoding = "2.2"
roxmltree = "0.19"
url = "2.3"
futures = "0.3"
futures-util = "0.3"
tokio = { version = "1.49", default-features = false, features = [
"rt",
"rt-multi-thread",
"io-util",
"net",
"time",
"macros",
"sync",
"signal",
"fs",
] }
tokio = { version = "1.0", default-features = false, features = ["net", "rt", "rt-multi-thread", "io-util", "net", "time", "macros", "sync", "signal", "fs"] }
tokio-util = { version = "0.7", features = ["compat", "io"] }
tokio-stream = { version = "0.1", features = ["net"] }
socket2 = { version = "0.6", features = ["all"] }
opentelemetry = { version = "0.17", features = ["rt-tokio", "metrics", "trace"] }
opentelemetry = { version = "0.17", features = [ "rt-tokio", "metrics", "trace" ] }
opentelemetry-prometheus = "0.10"
opentelemetry-otlp = "0.10"
opentelemetry-contrib = "0.9"
prometheus = "0.13"
# used by the k2v-client crate only
aws-sigv4 = { version = "1.3", default-features = false }
hyper-rustls = { version = "0.27", default-features = false, features = [
"http1",
"http2",
"ring",
"rustls-native-certs",
] }
aws-sigv4 = { version = "1.1", default-features = false }
hyper-rustls = { version = "0.26", default-features = false, features = ["http1", "http2", "ring", "rustls-native-certs"] }
log = "0.4"
thiserror = "2.0"
# ---- used only as build / dev dependencies ----
assert-json-diff = "2.0"
rustc_version = "0.4"
rustc_version = "0.4.0"
static_init = "1.0"
aws-smithy-runtime = { version = "1.9", default-features = false, features = [
"tls-rustls",
] }
aws-sdk-config = { version = "1.99", default-features = false }
aws-sdk-s3 = { version = "1.121", default-features = false, features = [
"rt-tokio",
] }
aws-smithy-runtime = { version = "1.8", default-features = false, features = ["tls-rustls"] }
aws-sdk-config = { version = "1.62", default-features = false }
aws-sdk-s3 = { version = "1.79", default-features = false, features = ["rt-tokio"] }
[profile.dev]
#lto = "thin" # disabled for now, adds 2-4 min to each CI build
lto = "off"
[profile.release]
lto = "thin"
codegen-units = 16
lto = true
codegen-units = 1
opt-level = 3
strip = "debuginfo"
[workspace.lints.clippy]
# pedantic lints configuration
doc_markdown = "warn"
format_collect = "warn"
manual_midpoint = "warn"
semicolon_if_nothing_returned = "warn"
unnecessary_semicolon = "warn"
unnecessary_wraps = "warn"
# nursery lints configuration
# or_fun_call = "warn" # enable it to help detect non trivial code used in `_or` method
strip = true
-45
View File
@@ -1,45 +0,0 @@
# Governance of Gararge
This documents how the Garage project operates. It reflects the state of the project as of July 2026 and is not optimal. The team is interested to improve it in the future.
## Team organization
* **Contributors**: anyone can contribute by proposing changes in issues and pull requests.
* **Maintainers**: they are responsible for reviewing, merging pull requests, publishing releases and triaging issues.
The current maintainers are:
* Alex (handle `lx`)
* Trinity (handle `trinity-1686a`)
* Quentin (handle `quentin`)
* Maximilien (handle `halfa`), who is in particular responsible for coordinating effort on the Kubernetes integration / Helm chart.
They are added to a white-list of the branch protection rule of the repository to enable them to merge pull requests.
To become a maintainer, you need to be a long-term contributor and earn the personal trust of Alex.
There is no set process for leaving the maintainer role.
* **Lead developer**: Alex (handle `lx`) is the lead developer and is responsible of ensuring the
correctness of Garage and stability between version upgrades. He may transfer this role to someone else as he sees fit.
## Communication channels
The team coordinates in the following channels:
* The issue tracker and pull requests of the official repository.
* The `#garage:deuxfleurs.fr` matrix channel (in English), open to anyone.
On this channel, users may ask for support and discussions about development also happen.
* The `#garage-dev:deuxfleurs.fr` matrix channel (in French), not advertised to contributors but de facto accessible to anyone.
Discussions about development and project coordination happen there.
The moderators for those discussion channels are the Garage maintainers.
## Decision procedures
Decisions are taken by lazy consensus, with the lead developer settling discussions when a consensus cannot be reached.
## Governance changes
There is no set process for changing the governance of garage.
## See also
* [Project goals](https://garagehq.deuxfleurs.fr/documentation/design/goals/)
* [Contributing instructions](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main-v2/CONTRIBUTING.md)
-14
View File
@@ -1,14 +0,0 @@
# Security Reporting
If you wish to report responsibly a security vulnerability about Garage, we ask that you follow the following process.
Please report each security vulnerabilities by filling out the following template:
- PROJECT: A URL to the code repository containing the vulnerable version - be reminded that the source of truth is at https://git.deuxfleurs.fr/deuxfleurs/garage
- PUBLIC: Please let us know if this vulnerability has been made or discussed publicly already, and if so, please let us know where.
- DESCRIPTION: Please provide precise description of the security vulnerability you have found with as much information as you are able and willing to provide.
Please send the above info, along with any other information you feel is pertinent by emailing the core team at: garagehq@deuxfleurs.fr
The Garage Core Team will let you know within a few weeks whether or not your report has been accepted or rejected.
We ask that you please keep the report confidential until we have either responded or made a public announcement.
+4 -4
View File
@@ -3,10 +3,10 @@ info:
version: v0.8.0
title: Garage Administration API v0+garage-v0.8.0
description: |
Administrate your Garage cluster programmatically, including status, layout, keys, buckets, and maintenance tasks.
*Disclaimer: The API is not stable yet, hence its v0 tag. The API can change at any time, and changes can include breaking backward compatibility. Read the changelog and upgrade your scripts before upgrading. Additionally, this specification is very early stage and can contain bugs, especially on error return codes/types that are not tested yet. Do not expect a well finished and polished product!*
paths:
Administrate your Garage cluster programatically, including status, layout, keys, buckets, and maintainance tasks.
*Disclaimer: The API is not stable yet, hence its v0 tag. The API can change at any time, and changes can include breaking backward compatibility. Read the changelog and upgrade your scripts before upgrading. Additionnaly, this specification is very early stage and can contain bugs, especially on error return codes/types that are not tested yet. Do not expect a well finished and polished product!*
paths:
/status:
get:
tags:
+5 -5
View File
@@ -3,10 +3,10 @@ info:
version: v0.9.0
title: Garage Administration API v0+garage-v0.9.0
description: |
Administrate your Garage cluster programmatically, including status, layout, keys, buckets, and maintenance tasks.
*Disclaimer: The API is not stable yet, hence its v0 tag. The API can change at any time, and changes can include breaking backward compatibility. Read the changelog and upgrade your scripts before upgrading. Additionally, this specification is very early stage and can contain bugs, especially on error return codes/types that are not tested yet. Do not expect a well finished and polished product!*
paths:
Administrate your Garage cluster programatically, including status, layout, keys, buckets, and maintainance tasks.
*Disclaimer: The API is not stable yet, hence its v0 tag. The API can change at any time, and changes can include breaking backward compatibility. Read the changelog and upgrade your scripts before upgrading. Additionnaly, this specification is very early stage and can contain bugs, especially on error return codes/types that are not tested yet. Do not expect a well finished and polished product!*
paths:
/health:
get:
tags:
@@ -440,7 +440,7 @@ paths:
- "false"
example: "true"
required: false
description: "Whether or not the secret key should be returned in the response"
description: "Wether or not the secret key should be returned in the response"
responses:
'500':
description: "The server can not handle your request. Check your connectivity with the rest of the cluster."
File diff suppressed because it is too large Load Diff
File diff suppressed because one or more lines are too long
+1 -1
View File
@@ -51,4 +51,4 @@ We are currently building this SDK for [Python](@/documentation/build/python.md#
More information:
- [In the reference manual](@/documentation/reference-manual/admin-api.md)
- [Full specification](https://garagehq.deuxfleurs.fr/api/garage-admin-v0.html)
- [Full specifiction](https://garagehq.deuxfleurs.fr/api/garage-admin-v0.html)
+3 -3
View File
@@ -5,13 +5,13 @@ weight = 99
## S3
If you are developing a new application, you may want to use Garage to store your user's media.
If you are developping a new application, you may want to use Garage to store your user's media.
The S3 API that Garage uses is a standard REST API, so as long as you can make HTTP requests,
you can query it. You can check the [S3 REST API Reference](https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operations_Amazon_Simple_Storage_Service.html) from Amazon to learn more.
Developing your own wrapper around the REST API is time consuming and complicated.
Instead, there are some libraries already available.
Developping your own wrapper around the REST API is time consuming and complicated.
Instead, there are some libraries already avalaible.
Some of them are maintained by Amazon, some by Minio, others by the community.
+1 -1
View File
@@ -23,7 +23,7 @@ To configure S3-compatible software to interact with Garage,
you will need the following parameters:
- An **API endpoint**: this corresponds to the HTTP or HTTPS address
used to contact the Garage server. When running Garage locally this will usually
used to contact the Garage server. When runing Garage locally this will usually
be `http://127.0.0.1:3900`. In a real-world setting, you would usually have a reverse-proxy
that adds TLS support and makes your Garage server available under a public hostname
such as `https://garage.example.com`.
+8 -8
View File
@@ -54,7 +54,7 @@ garage bucket allow nextcloud --read --write --key nextcloud-key
Now edit your Nextcloud configuration file to enable object storage.
On my installation, the config. file is located at the following path: `/var/www/nextcloud/config/config.php`.
We will add a new root key to the `$CONFIG` dictionary named `objectstore`:
We will add a new root key to the `$CONFIG` dictionnary named `objectstore`:
```php
<?php
@@ -413,7 +413,7 @@ mc mirror --newer-than "3h" ./public/system/ garage/mastodon-data
## Matrix
Matrix is a chat communication protocol. Its main stable server implementation, [Synapse](https://matrix-org.github.io/synapse/latest/), provides a module to store media on a S3 backend. Additionally, a server independent media store supporting S3 has been developed by the community, it has been made possible thanks to how the matrix API has been designed and will work with implementations like Conduit, Dendrite, etc.
Matrix is a chat communication protocol. Its main stable server implementation, [Synapse](https://matrix-org.github.io/synapse/latest/), provides a module to store media on a S3 backend. Additionally, a server independent media store supporting S3 has been developped by the community, it has been made possible thanks to how the matrix API has been designed and will work with implementations like Conduit, Dendrite, etc.
### synapse-s3-storage-provider (synapse only)
@@ -450,7 +450,7 @@ media_storage_providers:
Note that uploaded media will also be stored locally and this behavior can not be deactivated, it is even required for
some operations like resizing images.
In fact, your local filesystem is considered as a cache but without any automated way to garbage collect it.
In fact, your local filesysem is considered as a cache but without any automated way to garbage collect it.
We can build our garbage collector with `s3_media_upload`, a tool provided with the module.
If you installed the module with the command provided before, you should be able to bring it in your path:
@@ -547,7 +547,7 @@ ejabberdctl module_install mod_s3_upload
Create the required key and bucket with:
```bash
garage key create ejabberd
garage key new --name ejabberd
garage bucket create objects.xmpp-server.fr
garage bucket allow objects.xmpp-server.fr --read --write --key ejabberd
garage bucket website --allow objects.xmpp-server.fr
@@ -646,7 +646,7 @@ s3:
b2-eu-cen: # Don't change this key, it is hardcoded
key: <keyID>
secret: <keySecret>
endpoint: garage:3900 # publicly accessible endpoint of your garage instance
endpoint: garage:3900 # publically accessible endpoint of your garage instance
region: garage
bucket: <yourbucketName>
use_path_style: true
@@ -678,7 +678,7 @@ For more information on deployment you can check the [ente documentation](https:
This is the usual Garage setup:
```bash
garage key create pleroma-key
garage key new --name pleroma-key
garage bucket create pleroma
garage bucket allow pleroma --read --write --owner --key pleroma-key
```
@@ -730,7 +730,7 @@ Pleroma have an internal migration tool that can encounter some fatal error
So, use [your best tool](https://garagehq.deuxfleurs.fr/documentation/connect/cli/) to sync `/var/lib/pleroma/uploads/` in your S3.
Then, to avoid some non existent problem (just in case of), run this command
Then, to avoid some non existant problem (just in case of), run this command
```bash
while true
@@ -759,7 +759,7 @@ This feature requires `pict-rs >= 4.0.0`.
This is the usual Garage setup:
```bash
garage key create pictrs-key
garage key new --name pictrs-key
garage bucket create pictrs-data
garage bucket allow pictrs-data --read --write --key pictrs-key
```
+1 -1
View File
@@ -22,7 +22,7 @@ Note that `git-annex` requires to be compiled with Haskell package version
`aws-0.24` to work with Garage.
```bash
garage key create my-key
garage key new --name my-key
garage bucket create my-git-annex
garage bucket allow my-git-annex --read --write --key my-key
```
+3 -3
View File
@@ -41,7 +41,7 @@ Some commands:
# list buckets
mc ls garage/
# list objects in a bucket
# list objets in a bucket
mc ls garage/my_files
# copy from your filesystem to garage
@@ -218,7 +218,7 @@ Within Cyberduck, a
available within the `Preferences -> Profiles` section. This can enabled and
then connections to Garage may be configured.
### Instructions for the CLI
### Instuctions for the CLI
To configure duck (Cyberduck's CLI tool), start by creating its folder hierarchy:
@@ -268,7 +268,7 @@ duck --delete garage:/my-files/an-object.txt
## WinSCP (libs3) {#winscp}
*You can find instructions on how to use the GUI in french [in our wiki](https://guide.deuxfleurs.fr/services/winscp/).*
*You can find instructions on how to use the GUI in french [in our wiki](https://guide.deuxfleurs.fr/prise_en_main/winscp/).*
How to use `winscp.com`, the CLI interface of WinSCP:
+1 -1
View File
@@ -27,7 +27,7 @@ which support storing metrics in an object store:
This can be configured with Garage with the following:
```bash
garage key create vector-system-logs
garage key new --name vector-system-logs
garage bucket create system-logs
garage bucket allow system-logs --read --write --key vector-system-logs
```
+3 -1
View File
@@ -201,9 +201,11 @@ on the binary cache, the client will download the result from the cache instead
### Channels
Channels additionally serve Nix definitions, ie. a `.nix` file referencing
Channels additionnaly serve Nix definitions, ie. a `.nix` file referencing
all the derivations you want to serve.
## Gitlab
*External link:* [Gitlab Documentation > Object storage](https://docs.gitlab.com/ee/administration/object_storage.html)
+1 -1
View File
@@ -13,7 +13,7 @@ have published Ansible roles. We list them and compare them below.
| **Runtime** | Systemd | Docker | Systemd |
| **Target OS** | Any Linux | Any Linux | Any Linux |
| **Architecture** | amd64, arm64, i686 | amd64, arm64 | arm64, arm, 386, amd64 |
| **Additional software** | None | Traefik | Nginx and Keepalived (optional) |
| **Additional software** | None | Traefik | Ngnix and Keepalived (optional) |
| **Automatic node connection** | ❌ | ✅ | ✅ |
| **Layout management** | ❌ | ✅ | ✅ |
| **Manage buckets & keys** | ❌ | ✅ (basic) | ✅ |
+4 -4
View File
@@ -33,7 +33,7 @@ by adding encryption at different levels.
We would be very curious to know your needs and thougs about ideas such as
encryption practices and things like key management, as we want Garage to be a
serious base platform for the development of secure, encrypted applications.
serious base platform for the developpment of secure, encrypted applications.
Do not hesitate to come talk to us if you have any thoughts or questions on the
subject.
@@ -59,7 +59,7 @@ For standard S3 API requests, Garage does not encrypt data at rest by itself.
For the most generic at rest encryption of data, we recommend setting up your
storage partitions on encrypted LUKS devices.
If you are developing your own client software that makes use of S3 storage,
If you are developping your own client software that makes use of S3 storage,
we recommend implementing data encryption directly on the client side and never
transmitting plaintext data to Garage. This makes it easy to use an external
untrusted storage provider if necessary.
@@ -108,14 +108,14 @@ Protects against the following threats:
- Stolen HDD
Crucially, does not protect against malicious sysadmins or remote attackers that
Crucially, does not protect againt malicious sysadmins or remote attackers that
might gain access to your servers.
Methods include full-disk encryption with tools such as LUKS.
## Encrypting data on the client side
Protects against the following threats:
Protects againt the following threats:
- A honest-but-curious administrator
- A malicious administrator that tries to corrupt your data
+1 -8
View File
@@ -9,7 +9,7 @@ There are three methods to expose buckets as website:
1. using the PutBucketWebsite S3 API call, which is allowed for access keys that have the owner permission bit set
2. from the Garage CLI, by an administrator of the cluster
2. from the Garage CLI, by an adminstrator of the cluster
3. using the Garage administration API
@@ -25,13 +25,6 @@ garage bucket website --allow my-website
Now it will be **publicly** exposed on the web endpoint (by default listening on port 3902).
> The bucket needs to have a *global alias* to be exposed as a website. If the
> bucket was created with `garage bucket create` it will have an alias;
> if created via the S3 API [you will have to manually add the alias
> ](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/742) first.
> Creating globally aliased buckets from the S3 API is [currently under
> discussion](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/649).
## How exposed websites work
Our website serving logic is as follow:
+2 -2
View File
@@ -25,7 +25,7 @@ sudo apt-get install build-essential
The primary location for Garage's source code is the
[Forgejo repository](https://git.deuxfleurs.fr/Deuxfleurs/garage),
which contains all of the released versions as well as the code
for the development of the next version.
for the developpement of the next version.
Clone the repository and enter it as follows:
@@ -41,7 +41,7 @@ git tag # List available tags
git checkout v0.8.0 # Change v0.8.0 with the version you wish to build
```
Otherwise you will be building a development build from the `main` branch
Otherwise you will be building a developpement build from the `main` branch
that includes all of the changes to be released in the next version.
Be careful that such a build might be unstable or contain bugs,
and could be incompatible with nodes that run stable versions of Garage.
+2 -2
View File
@@ -26,7 +26,7 @@ Or deploy with custom values:
helm install --create-namespace --namespace garage garage ./garage -f values.override.yaml
```
If you want to manage the CustomResourceDefinition used by garage for its `kubernetes_discovery` outside of the helm chart, add `garage.kubernetesSkipCrd: true` to your custom values and use the kustomization before deploying the helm chart:
If you want to manage the CustomRessourceDefinition used by garage for its `kubernetes_discovery` outside of the helm chart, add `garage.kubernetesSkipCrd: true` to your custom values and use the kustomization before deploying the helm chart:
```bash
kubectl apply -k ../k8s/crd
@@ -47,7 +47,7 @@ All possible configuration values can be found with:
helm show values ./garage
```
This is an example `values.override.yaml` for deploying in a microk8s cluster with a https s3 api ingress route:
This is an example `values.overrride.yaml` for deploying in a microk8s cluster with a https s3 api ingress route:
```yaml
garage:
+5 -5
View File
@@ -96,14 +96,14 @@ to store 2 TB of data in total.
## Get a Docker image
Our docker image is currently named `dxflrs/garage` and is stored on the [Docker Hub](https://hub.docker.com/r/dxflrs/garage/tags?page=1&ordering=last_updated).
We encourage you to use a fixed tag (eg. `v2.3.0`) and not the `latest` tag.
For this example, we will use the latest published version at the time of the writing which is `v2.3.0` but it's up to you
We encourage you to use a fixed tag (eg. `v2.2.0`) and not the `latest` tag.
For this example, we will use the latest published version at the time of the writing which is `v2.2.0` but it's up to you
to check [the most recent versions on the Docker Hub](https://hub.docker.com/r/dxflrs/garage/tags?page=1&ordering=last_updated).
For example:
```
docker pull dxflrs/garage:v2.3.0
sudo docker pull dxflrs/garage:v2.2.0
```
## Deploying and configuring Garage
@@ -171,7 +171,7 @@ docker run \
-v /etc/garage.toml:/etc/garage.toml \
-v /var/lib/garage/meta:/var/lib/garage/meta \
-v /var/lib/garage/data:/var/lib/garage/data \
dxflrs/garage:v2.3.0
dxflrs/garage:v2.2.0
```
With this command line, Garage should be started automatically at each boot.
@@ -185,7 +185,7 @@ If you want to use `docker-compose`, you may use the following `docker-compose.y
version: "3"
services:
garage:
image: dxflrs/garage:v2.3.0
image: dxflrs/garage:v2.2.0
network_mode: "host"
restart: unless-stopped
volumes:
+2 -69
View File
@@ -142,74 +142,7 @@ server {
## Apache httpd
The [Apache HTTP Server](https://httpd.apache.org/)
is a general purpose web server that includes
[reverse proxy](https://httpd.apache.org/docs/2.4/mod/mod_proxy.html)
capabilities.
### Exposing the S3 endpoints
Create a new [virtual host](https://httpd.apache.org/docs/2.4/vhosts/),
obtain a certificate using
[certbot](https://eff-certbot.readthedocs.io/en/stable/using.html#apache),
and add the
[`ProxyPass`](https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass)
and
[`ProxyPreserveHost`](https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypreservehost)
options:
```apache
<VirtualHost *:443>
ServerName garage.example.com
SSLCertificateFile /etc/letsencrypt/live/garage.example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/garage.example.com/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
Header always set Strict-Transport-Security "max-age=31536000"
Header always add Content-Security-Policy upgrade-insecure-requests
ProxyPass "/" "http://localhost:3900/" nocanon
ProxyPreserveHost on
</VirtualHost>
```
The `nocanon` keyword is important for
[presigned URLs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html);
otherwise,
> `mod_proxy` will canonicalise ProxyPassed URLs.
> But this may be incompatible with some backends,
> particularly those that make use of `PATH_INFO`.
> The optional `nocanon` keyword suppresses this
> and passes the URL path "raw" to the backend.
### Exposing the web endpoint
Adding static websites backed by Garage works very similarly,
with the only difference being the port selected in the `ProxyPass` directive.
```apache
ProxyPass "/" "http://localhost:3902/" nocanon
```
### Using Unix sockets
Apache can also proxy via Unix sockets instead of TCP ports,
if Garage is so configured.
`garage.toml`:
```toml
[s3_api]
api_bind_addr = "/run/garage/s3_api.socket"
```
Apache config:
```apache
ProxyPass "/" "unix:/run/garage/s3_api.socket|http://localhost/" nocanon
```
@TODO
## Traefik v2
@@ -339,7 +272,7 @@ Add the following configuration section [to compress response](https://doc.traef
### Add caching response
Traefik's caching middleware is only available on [enterprise version](https://doc.traefik.io/traefik-enterprise/middlewares/http-cache/), however the freely-available [Souin plugin](https://github.com/darkweak/souin#tr%C3%A6fik-container) can also do the job. (section to be completed)
Traefik's caching middleware is only available on [entreprise version](https://doc.traefik.io/traefik-enterprise/middlewares/http-cache/), however the freely-available [Souin plugin](https://github.com/darkweak/souin#tr%C3%A6fik-container) can also do the job. (section to be completed)
### Complete example
+1 -1
View File
@@ -38,7 +38,7 @@ WantedBy=multi-user.target
id is dynamically allocated by systemd (set with `DynamicUser=true`). It cannot
access (read or write) home folders (`/home`, `/root` and `/run/user`), the
rest of the filesystem can only be read but not written, only the path seen as
`/var/lib/garage` is writable as seen by the service. Additionally, the process
`/var/lib/garage` is writable as seen by the service. Additionnaly, the process
can not gain new privileges over time.
For this to work correctly, your `garage.toml` must be set with
+3 -1
View File
@@ -10,7 +10,7 @@ perspective. It will allow you to understand if Garage is a good fit for
you, how to better use it, how to contribute to it, what can Garage could
and could not do, etc.
- **[Goals and use cases](@/documentation/design/goals.md):** This page explains why Garage was conceived and what practical use cases it targets.
- **[Goals and use cases](@/documentation/design/goals.md):** This page explains why Garage was concieved and what practical use cases it targets.
- **[Related work](@/documentation/design/related-work.md):** This pages presents the theoretical background on which Garage is built, and describes other software storage solutions and why they didn't work for us.
@@ -31,3 +31,5 @@ We love to talk and hear about Garage, that's why we keep a log here:
- [(en, 2021-04-28) Distributed object storage is centralised](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/b1f60579a13d3c5eba7f74b1775c84639ea9b51a/doc/talks/2021-04-28_spirals-team/talk.pdf)
- [(fr, 2020-12-02) Garage : jouer dans la cour des grands quand on est un hébergeur associatif](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/b1f60579a13d3c5eba7f74b1775c84639ea9b51a/doc/talks/2020-12-02_wide-team/talk.pdf)
+5 -5
View File
@@ -15,14 +15,14 @@ The more a user request will require intra-cluster requests to complete, the mor
This is especially true for sequential requests: requests that must wait the result of another request to be sent.
We designed Garage without consensus algorithms (eg. Paxos or Raft) to minimize the number of sequential and parallel requests.
This series of benchmarks quantifies the impact of this design choice.
This serie of benchmarks quantifies the impact of this design choice.
### On a simple simulated network
We start with a controlled environment, all the instances are running on the same (powerful enough) machine.
To control the network latency, we simulate the network with [mknet](https://git.deuxfleurs.fr/trinity-1686a/mknet) (a tool we developed, based on `tc` and the linux network stack).
To measure S3 endpoints latency, we use our own tool [s3lat](https://git.deuxfleurs.fr/quentin/s3lat/) to observe only the intra-cluster latency and not some contention on the nodes (CPU, RAM, disk I/O, network bandwidth, etc.).
To control the network latency, we simulate the network with [mknet](https://git.deuxfleurs.fr/trinity-1686a/mknet) (a tool we developped, based on `tc` and the linux network stack).
To mesure S3 endpoints latency, we use our own tool [s3lat](https://git.deuxfleurs.fr/quentin/s3lat/) to observe only the intra-cluster latency and not some contention on the nodes (CPU, RAM, disk I/O, network bandwidth, etc.).
Compared to other benchmark tools, S3Lat sends only one (small) request at the same time and measures its latency.
We selected 5 standard endpoints that are often in the critical path: ListBuckets, ListObjects, GetObject, PutObject and RemoveObject.
@@ -32,7 +32,7 @@ In this first benchmark, we consider 5 instances that are located in a different
Compared to garage, minio latency drastically increases on 3 endpoints: GetObject, PutObject, RemoveObject.
We suppose that these requests on minio make transactions over Raft, involving 4 sequential requests: 1) sending the message to the leader, 2) having the leader dispatch it to the other nodes, 3) waiting for the confirmation of followers and finally 4) committing it. With our current configuration, one Raft transaction will take around 400 ms. GetObject seems to correlate to 1 transaction while PutObject and RemoveObject seems to correlate to 2 or 3. Reviewing minio code would be required to confirm this hypothesis.
We suppose that these requests on minio make transactions over Raft, involving 4 sequential requests: 1) sending the message to the leader, 2) having the leader dispatch it to the other nodes, 3) waiting for the confirmation of followers and finally 4) commiting it. With our current configuration, one Raft transaction will take around 400 ms. GetObject seems to correlate to 1 transaction while PutObject and RemoveObject seems to correlate to 2 or 3. Reviewing minio code would be required to confirm this hypothesis.
Conversely, garage uses an architecture similar to DynamoDB and never require global cluster coordination to answer a request.
Instead, garage can always contact the right node in charge of the requested data, and can answer in as low as one request in the case of GetObject and PutObject. We also observed that Garage latency, while often lower to minio, is more dispersed: garage is still in beta and has not received any performance optimization yet.
@@ -50,7 +50,7 @@ We plot a similar graph as before:
This new graph is very similar to the one before, neither minio or garage seems to benefit from this new topology, but they also do not suffer from it.
Considering garage, this is expected: nodes in the same DC are put in the same zone, and then data are spread on different zones for data resiliency and availability.
Considering garage, this is expected: nodes in the same DC are put in the same zone, and then data are spread on different zones for data resiliency and availaibility.
Then, in the default mode, requesting data requires to query at least 2 zones to be sure that we have the most up to date information.
These requests will involve at least one inter-DC communication.
In other words, we prioritize data availability and synchronization over raw performances.
+2 -1
View File
@@ -94,7 +94,7 @@ delete a tombstone, the following condition has to be met:
- All nodes responsible for storing this entry are aware of the existence of
the tombstone, i.e. they cannot hold another version of the entry that is
superseded by the tombstone. This ensures that deleting the tombstone is
superseeded by the tombstone. This ensures that deleting the tombstone is
safe and that no deleted value will come back in the system.
Garage uses atomic database operations (such as compare-and-swap and
@@ -141,3 +141,4 @@ rebalance of data, this would have led to the disk utilization to explode
during the rebalancing, only to shrink again after 24 hours. The 10-minute
delay is a compromise that gives good security while not having this problem of
disk space explosion on rebalance.
+2 -2
View File
@@ -37,7 +37,7 @@ However, Amazon S3 source code is not open but alternatives were proposed.
We identified Minio, Pithos, Swift and Ceph.
Minio/Ceph enforces a total order, so properties similar to a (relaxed) filesystem.
Swift and Pithos are probably the most similar to AWS S3 with their consistent hashing ring.
However Pithos is not maintained anymore. More precisely the company that published Pithos version 1 has developed a second version 2 but has not open sourced it.
However Pithos is not maintained anymore. More precisely the company that published Pithos version 1 has developped a second version 2 but has not open sourced it.
Some tests conducted by the [ACIDES project](https://acides.org/) have shown that Openstack Swift consumes way more resources (CPU+RAM) that we can afford. Furthermore, people developing Swift have not designed their software for geo-distribution.
There were many attempts in research too. I am only thinking to [LBFS](https://pdos.csail.mit.edu/papers/lbfs:sosp01/lbfs.pdf) that was used as a basis for Seafile. But none of them have been effectively implemented yet.
@@ -63,7 +63,7 @@ Due to its industry oriented design, Ceph is also far from being *Simple* to ope
In a certain way, Ceph and MinIO are closer together than they are from Garage or OpenStack Swift.
**[Pithos](https://github.com/exoscale/pithos):**
Pithos has been abandoned and should probably not used yet, in the following we explain why we did not pick their design.
Pithos has been abandonned and should probably not used yet, in the following we explain why we did not pick their design.
Pithos was relying as a S3 proxy in front of Cassandra (and was working with Scylla DB too).
From its designers' mouth, storing data in Cassandra has shown its limitations justifying the project abandonment.
They built a closed-source version 2 that does not store blobs in the database (only metadata) but did not communicate further on it.
+23
View File
@@ -82,6 +82,12 @@ nix-build \
*The result is located in `result/bin`. You can pass arguments to cross compile: check `.woodpecker/release.yml` for examples.*
If you modify a `Cargo.toml` or regenerate any `Cargo.lock`, you must run `cargo2nix`:
```
cargo2nix -f
```
Many tools like rclone, `mc` (minio-client), or `aws` (awscliv2) will be available in your environment and will be useful to test Garage.
**This is the recommended method.**
@@ -118,6 +124,23 @@ cargo fmt # format the project, run it before any commit!
cargo clippy # run the linter, run it before any commit!
```
This is specific to our project, but you will need one last tool, `cargo2nix`.
To install it, run:
```bash
cargo install --git https://github.com/superboum/cargo2nix --branch main cargo2nix
```
You must use it every time you modify a `Cargo.toml` or regenerate a `Cargo.lock` file as follow:
```bash
cargo build # Rebuild Cargo.lock if needed
cargo2nix -f
```
It will output a `Cargo.nix` file which is a specific `Cargo.lock` file dedicated to Nix that is required by our CI
which means you must include it in your commits.
Later, to use our scripts and integration tests, you might need additional tools.
These tools are listed at the end of the `shell.nix` package in the `nativeBuildInputs` part.
It is up to you to find a way to install the ones you need on your computer.
@@ -3,6 +3,15 @@ title = "Miscellaneous notes"
weight = 20
+++
## Quirks about cargo2nix/rust in Nix
If you use submodules in your crate (like `crdt` and `replication` in `garage_table`), you must list them in `default.nix`
The Windows target does not work. it might be solvable through [overrides](https://github.com/cargo2nix/cargo2nix/blob/master/overlay/overrides.nix). Indeed, we pass `x86_64-pc-windows-gnu` but mingw need `x86_64-w64-mingw32`
We have a simple [PR on cargo2nix](https://github.com/cargo2nix/cargo2nix/pull/201) that fixes critical bugs but the project does not seem very active currently. We must use [my patched version of cargo2nix](https://github.com/superboum/cargo2nix) to enable i686 and armv6l compilation. We might need to contribute to cargo2nix in the future.
## Nix
Nix has no armv7 + musl toolchains but armv7l is backward compatible with armv6l.
+5 -3
View File
@@ -23,7 +23,7 @@ This logic is defined in `nix/build_index.nix`.
For each commit, we first pass the code to a formatter (rustfmt) and a linter (clippy).
Then we try to build it in debug mode and run both unit tests and our integration tests.
Additionally, when releasing, our integration tests are run on the release build for amd64 and i686.
Additionnaly, when releasing, our integration tests are run on the release build for amd64 and i686.
## Generated Artifacts
@@ -32,7 +32,7 @@ We generate the following binary artifacts for now:
- **os**: linux
- **format**: static binary, docker container
Additionally we also build two web pages and one JSON document:
Additionnaly we also build two web pages and one JSON document:
- the documentation (this website)
- [the release page](https://garagehq.deuxfleurs.fr/_releases.html)
- [the release list in JSON format](https://garagehq.deuxfleurs.fr/_releases.json)
@@ -67,7 +67,7 @@ nix copy --to 's3://nix?endpoint=garage.deuxfleurs.fr&region=garage&secret-key=/
The previous command will only send the built package and not its dependencies.
In the case of our CI pipeline, we want to cache all intermediate build steps
as well. This can be done using this quite involved command (here as an example
for the `pkgs.amd64.release` package):
for the `pkgs.amd64.relase` package):
```bash
nix copy -j8 \
@@ -174,3 +174,5 @@ drone sign --save Deuxfleurs/garage
```
Looking at the file, you will see that most of the commands are `nix-shell` and `nix-build` commands with various parameters.
+1 -1
View File
@@ -91,7 +91,7 @@ is definitely lost, then there is no other choice than to declare your S3 object
as unrecoverable, and to delete them properly from the data store. This can be done
using the `garage block purge` command.
## Rebalancing data directories {#rebalance}
## Rebalancing data directories
In [multi-HDD setups](@/documentation/operations/multi-hdd.md), to ensure that
data blocks are well balanced between storage locations, you may run a
+2 -2
View File
@@ -242,7 +242,7 @@ dc3 Tags Partitions Capacity Usable capacity
TOTAL 256 (256 unique) 2.0 GB 1000.0 MB (50.0%)
```
As we can see, the node that was moved to `dc3` (node4) is only used at 25% (approximately),
As we can see, the node that was moved to `dc3` (node4) is only used at 25% (approximatively),
whereas the node that was already in `dc3` (node3) is used at 75%.
This can be explained by the following:
@@ -260,7 +260,7 @@ This can be explained by the following:
data can be removed to be moved to node1.
- Garage will move data in equal proportions from all possible sources, in this
case it means that it will transfer 25% of the entire data set from node3 to
case it means that it will tranfer 25% of the entire data set from node3 to
node1 and another 25% from node4 to node1.
This explains why node3 ends with 75% utilization (100% from before minus 25%
+5 -6
View File
@@ -40,7 +40,7 @@ First of all, Garage divides the set of all possible block hashes
in a fixed number of slices (currently 1024), and assigns
to each slice a primary storage location among the specified data directories.
The number of slices having their primary location in each data directory
is proportional to the capacity specified in the config file.
is proportionnal to the capacity specified in the config file.
When Garage receives a block to write, it will always write it in the primary
directory of the slice that contains its hash.
@@ -68,11 +68,10 @@ To rebalance data, two strategies can be used:
secondary directory. This might never end up rebalancing everything if there
are data blocks that are only read and never written.
- Active rebalancing: an operator of a Garage node can [explicitly launch a
repair procedure](@/documentation/operations/durability-repairs.md#rebalance)
that rebalances the data directories, moving all blocks to their primary
location. Once done, all secondary locations for all hash slices are removed
so that they won't be checked anymore when looking for a data block.
- Active rebalancing: an operator of a Garage node can explicitly launch a repair
procedure that rebalances the data directories, moving all blocks to their
primary location. Once done, all secondary locations for all hash slices are
removed so that they won't be checked anymore when looking for a data block.
## Read-only storage locations
+1 -1
View File
@@ -56,7 +56,7 @@ From a high level perspective, a major upgrade looks like this:
10. Enable API access (reverse step 1)
11. Monitor your cluster while load comes back, check that all your applications are happy with this new version
### Major upgrades with minimal downtime
### Major upgarades with minimal downtime
There is only one operation that has to be coordinated cluster-wide: the switch of one version of the internal RPC protocol to the next.
This means that an upgrade with very limited downtime can simply be performed from one major version to the next by restarting all nodes
+137 -210
View File
@@ -43,10 +43,12 @@ or if you want a build customized for your system,
you can [build Garage from source](@/documentation/cookbook/from-source.md).
If none of these option work for you, you can also run Garage in a Docker
container. For simplicity, a minimal command to launch Garage using Docker is
provided in this quick start guide. We recommend reading the tutorial on
[configuring a multi-node cluster](@/documentation/cookbook/real-world.md) to
learn about the full Docker workflow for Garage.
container. When using Docker, the commands used in this guide will not work
anymore. We recommend reading the tutorial on [configuring a
multi-node cluster](@/documentation/cookbook/real-world.md) to learn about
using Garage as a Docker container. For simplicity, a minimal command to launch
Garage using Docker is provided in this quick start guide as well.
## Configuring and starting Garage
@@ -80,6 +82,9 @@ bind_addr = "[::]:3902"
root_domain = ".web.garage.localhost"
index = "index.html"
[k2v_api]
api_bind_addr = "[::]:3904"
[admin]
api_bind_addr = "[::]:3903"
admin_token = "$(openssl rand -base64 32)"
@@ -90,13 +95,10 @@ EOF
See the [Configuration file format](https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/)
for complete options and values.
By default, Garage looks for its configuration file in **`/etc/garage.toml`.**
Since we have written our configuration file in the working directory, we will have to set
the following environment variable:
```bash
export GARAGE_CONFIG_FILE=$(pwd)/garage.toml
```
Now that your configuration file has been created, you may save it to the directory of your choice.
By default, Garage looks for **`/etc/garage.toml`.**
You can also store it somewhere else, but you will have to specify `-c path/to/garage.toml`
at each invocation of the `garage` binary (for example: `garage -c ./garage.toml server`, `garage -c ./garage.toml status`).
As you can see, the `rpc_secret` is a 32 bytes hexadecimal string.
You can regenerate it with `openssl rand -hex 32`.
@@ -109,36 +111,15 @@ Garage server will not be persistent. Change these to locations on your local di
your data to be persisted properly.
### Configuring initial access credentials
Since `v2.3.0`, Garage can automatically create a default access key and a default storage bucket,
based on values provided in environment variables.
To use this feature, export the following environment variables:
```bash
export GARAGE_DEFAULT_ACCESS_KEY="GK$(openssl rand -hex 16)"
export GARAGE_DEFAULT_SECRET_KEY="$(openssl rand -hex 32)"
export GARAGE_DEFAULT_BUCKET="default-bucket"
```
The example above creates a random access key ID and associated secret key.
You can also provide an access key ID and secret key of your own.
### Launching the Garage server
Use the following command to launch the Garage server:
```bash
garage server --single-node --default-bucket
```
garage -c path/to/garage.toml server
```
The `--single-node` flag instructs Garage to automatically configure a single-node cluster without data replication.
The `--default-bucket` flag instructs Garage to create a default access key and a default bucket using the environment variables we defined above.
Both flags are optional and can be omitted, in which case you will have to follow manual configuration steps described below.
**For older versions of Garage (before v2.3.0):** automatic configuration using `--single-node` and `--default-bucket` is not available,
you must follow the manual configuration steps.
If you have placed the `garage.toml` file in `/etc` (its default location), you can simply run `garage server`.
Alternatively, if you cannot or do not wish to run the Garage binary directly,
you may use Docker to run Garage in a container using the following command:
@@ -146,58 +127,21 @@ you may use Docker to run Garage in a container using the following command:
```bash
docker run \
-d \
--name garage-container \
--name garaged \
-p 3900:3900 -p 3901:3901 -p 3902:3902 -p 3903:3903 \
-v $(pwd)/garage.toml:/etc/garage.toml \
-e GARAGE_DEFAULT_ACCESS_KEY \
-e GARAGE_DEFAULT_SECRET_KEY \
-e GARAGE_DEFAULT_BUCKET \
dxflrs/garage:v2.3.0
/garage server --single-node --default-bucket
-v /path/to/garage.toml:/etc/garage.toml \
-v /path/to/garage/meta:/var/lib/garage/meta \
-v /path/to/garage/data:/var/lib/garage/data \
dxflrs/garage:v2.2.0
```
Note that this command will NOT create persistent volumes for Garage's data, so
your cluster will be wiped if the container terminates. To persist Garage's
data, you must manually add volumes for the `data` and `metadata` directories
and configure their correct paths in your `garage.toml` files (see [configuring
a multi-node cluster](@/documentation/cookbook/real-world.md)).
Under Linux, you can substitute `--network host` for `-p 3900:3900 -p 3901:3901 -p 3902:3902 -p 3903:3903`
Under Linux, you can substitute `--network host` for `-p 3900:3900 -p 3901:3901 -p 3902:3902 -p 3903:3903`.
### Checking that Garage runs correctly
The `garage` utility is also used as a CLI tool to administrate your Garage
deployment. It needs read access to your configuration file and to the metadata directory
to obtain connection parameters to contact the local Garage node.
Use the following command to show the status of your cluster:
```
garage status
```
If you are running Garage in a Docker container, you can use the following command instead:
```bash
docker exec garage-container /garage status
```
This should show something like this:
```
==== HEALTHY NODES ====
ID Hostname Address Tags Zone Capacity DataAvail Version
563e1ac825ee3323 linuxbox 127.0.0.1:3901 [default] dc1 19.9 GiB 19.5 GiB (97.6%) v2.3.0
```
### Troubleshooting
#### Troubleshooting
Ensure your configuration file, `metadata_dir` and `data_dir` are readable by the user running the `garage` server or Docker.
When running the `garage` CLI, ensure that the path to your configuration file is correctly specified (see below),
and that it can read it and read from your metadata directory.
You can tune Garage's verbosity by setting the `RUST_LOG=` environment variable.
You can tune Garage's verbosity by setting the `RUST_LOG=` environment variable. \
Available log levels are (from less verbose to more verbose): `error`, `warn`, `info` *(default)*, `debug` and `trace`.
```bash
@@ -210,135 +154,36 @@ Log level `info` is the default value and is recommended for most use cases.
Log level `debug` can help you check why your S3 API calls are not working.
### Checking that Garage runs correctly
## Uploading and downloading from Garage
The `garage` utility is also used as a CLI tool to configure your Garage deployment.
It uses values from the TOML configuration file to find the Garage daemon running on the
local node, therefore if your configuration file is not at `/etc/garage.toml` you will
again have to specify `-c path/to/garage.toml` at each invocation.
This section will show how to download and upload files on Garage using a third-party tool named `awscli`.
If you are running Garage in a Docker container, you can set `alias garage="docker exec -ti <container name> /garage"`
to use the Garage binary inside your container.
### Install and configure `awscli`
If you have python on your system, you can install it with:
```bash
python -m pip install --user awscli
```
Now that `awscli` is installed, you must configure it to talk to your Garage
instance using the credentials defined above. Here is a simple way to create
a configuration file in `~/.awsrc` using a single command that will save the
secrets from your environment:
```bash
cat > ~/.awsrc <<EOF
export AWS_ENDPOINT_URL='http://localhost:3900'
export AWS_DEFAULT_REGION='garage'
export AWS_ACCESS_KEY_ID='$GARAGE_DEFAULT_ACCESS_KEY'
export AWS_SECRET_ACCESS_KEY='$GARAGE_DEFAULT_SECRET_KEY'
aws --version
EOF
If the `garage` CLI is able to correctly detect the parameters of your local Garage node,
the following command should be enough to show the status of your cluster:
```
Note that you need to have at least `awscli` `>=1.29.0` or `>=2.13.0`, otherwise you
need to specify `--endpoint-url` explicitly on each `awscli` invocation.
Now, each time you want to use `awscli` on this target, run:
```bash
source ~/.awsrc
garage status
```
*You can create multiple files with different names if you
have multiple Garage clusters or different keys.
Switching from one cluster to another is as simple as
sourcing the right file.*
### Example usage of `awscli`
```bash
# list buckets
aws s3 ls
# list objects of a bucket
aws s3 ls s3://default-bucket
# copy from your filesystem to garage
aws s3 cp /proc/cpuinfo s3://default-bucket/cpuinfo.txt
# copy from garage to your filesystem
aws s3 cp s3://default-bucket/cpuinfo.txt /tmp/cpuinfo.txt
```
Note that you can use `awscli` for more advanced operations like
creating a bucket, pre-signing a request or managing your website.
[Read the full documentation to know more](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/index.html).
Some features are however not implemented like ACL or policy.
Check [our S3 compatibility list](@/documentation/reference-manual/s3-compatibility.md).
### Other tools for interacting with Garage
The following tools can also be used to send and receive files from/to Garage:
- [minio-client](@/documentation/connect/cli.md#minio-client)
- [s3cmd](@/documentation/connect/cli.md#s3cmd)
- [rclone](@/documentation/connect/cli.md#rclone)
- [Cyberduck](@/documentation/connect/cli.md#cyberduck)
- [WinSCP](@/documentation/connect/cli.md#winscp)
An exhaustive list is maintained in the ["Integrations" > "Browsing tools" section](@/documentation/connect/_index.md).
## Manual configuration
This section provides instructions that are equivalent to using the
`--single-node` and `--default-bucket` flags for automatic configuration. If
you are using an older version of Garage (before v2.3.0), you must follow
these instructions as automatic configuration is not available.
We will have to run quite a few `garage` administration commands to get started.
If you ever get lost, don't forget that the `help` command and the `--help` flags can help you anywhere,
the CLI tool is self-documented! Two examples:
```
garage help
garage bucket allow --help
```
### Configuring the `garage` CLI
Remember that the `garage` CLI needs to know the path of your `garage.toml` configuration file.
If it is not in the default location of `/etc/garage.toml`, you can specify it either:
- by setting the `GARAGE_CONFIG_FILE` environment variable;
- by adding the `-c` flag to each `garage` command, for example: `garage -c ./garage.toml status`.
If you are running Garage in a Docker container, you can set the following alias
to provide a fake `garage`command that uses the Garage binary inside your container:
```bash
alias garage="docker exec -ti <container name> /garage"
```
You can test that your `garage` CLI is configured correctly by running a basic command such as `garage status`.
### Creating a cluster layout
When you first start a cluster without automatic configuration, the output of `garage status` will look as follows:
This should show something like this:
```
==== HEALTHY NODES ====
ID Hostname Address Tags Zone Capacity DataAvail Version
563e1ac825ee3323 linuxbox 127.0.0.1:3901 NO ROLE ASSIGNED v2.3.0
ID Hostname Address Tag Zone Capacity
563e1ac825ee3323 linuxbox 127.0.0.1:3901 NO ROLE ASSIGNED
```
Creating a cluster layout for a Garage deployment means informing Garage of the
disk space available on each node of the cluster using the `-c` flag, as well
as the name of the zone (e.g. datacenter) each machine is located in using the
`-z` flag.
## Creating a cluster layout
Creating a cluster layout for a Garage deployment means informing Garage
of the disk space available on each node of the cluster, `-c`,
as well as the name of the zone (e.g. datacenter), `-z`, each machine is located in.
For our test deployment, we are have only one node with zone named `dc1` and a
capacity of `1G`, though the capacity is ignored for a single node deployment
@@ -359,29 +204,38 @@ garage layout apply --version 1
```
### Creating buckets and keys
## Creating buckets and keys
In this section, we will suppose that we want to create a bucket named `nextcloud-bucket`
that will be accessed through a key named `nextcloud-app-key`.
Don't forget that `help` command and `--help` subcommands can help you anywhere,
the CLI tool is self-documented! Two examples:
```
garage help
garage bucket allow --help
```
### Create a bucket
Let's take an example where we want to deploy NextCloud using Garage as the
main data storage. We will suppose that we want to create a bucket named
`nextcloud-bucket` that will be accessed through a key named
`nextcloud-app-key`.
main data storage.
#### Create a bucket
First, create the bucket with the following command:
First, create a bucket with the following command:
```
garage bucket create nextcloud-bucket
```
Check that the bucket was created properly:
Check that everything went well:
```
garage bucket list
garage bucket info nextcloud-bucket
```
#### Create an API key
### Create an API key
The `nextcloud-bucket` bucket now exists on the Garage server,
however it cannot be accessed until we add an API key with the proper access rights.
@@ -404,14 +258,14 @@ Secret key: 7d37d093435a41f2aab8f13c19ba067d9776c90215f56614adad6ece597dbb34
Authorized buckets:
```
Check that the key was created properly:
Check that everything works as intended:
```
garage key list
garage key info nextcloud-app-key
```
#### Allow a key to access a bucket
### Allow a key to access a bucket
Now that we have a bucket and a key, we need to give permissions to the key on the bucket:
@@ -430,5 +284,78 @@ You can check at any time the allowed keys on your bucket with:
garage bucket info nextcloud-bucket
```
You should now be able to read and write objects to the bucket using the
credentials created above.
## Uploading and downloading from Garage
To download and upload files on garage, we can use a third-party tool named `awscli`.
### Install and configure `awscli`
If you have python on your system, you can install it with:
```bash
python -m pip install --user awscli
```
Now that `awscli` is installed, you must configure it to talk to your Garage instance,
with your key. There are multiple ways to do that, the simplest one is to create a file
named `~/.awsrc` with this content:
```bash
export AWS_ACCESS_KEY_ID=xxxx # put your Key ID here
export AWS_SECRET_ACCESS_KEY=xxxx # put your Secret key here
export AWS_DEFAULT_REGION='garage'
export AWS_ENDPOINT_URL='http://localhost:3900'
aws --version
```
Note you need to have at least `awscli` `>=1.29.0` or `>=2.13.0`, otherwise you
need to specify `--endpoint-url` explicitly on each `awscli` invocation.
Now, each time you want to use `awscli` on this target, run:
```bash
source ~/.awsrc
```
*You can create multiple files with different names if you
have multiple Garage clusters or different keys.
Switching from one cluster to another is as simple as
sourcing the right file.*
### Example usage of `awscli`
```bash
# list buckets
aws s3 ls
# list objects of a bucket
aws s3 ls s3://nextcloud-bucket
# copy from your filesystem to garage
aws s3 cp /proc/cpuinfo s3://nextcloud-bucket/cpuinfo.txt
# copy from garage to your filesystem
aws s3 cp s3://nextcloud-bucket/cpuinfo.txt /tmp/cpuinfo.txt
```
Note that you can use `awscli` for more advanced operations like
creating a bucket, pre-signing a request or managing your website.
[Read the full documentation to know more](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/index.html).
Some features are however not implemented like ACL or policy.
Check [our s3 compatibility list](@/documentation/reference-manual/s3-compatibility.md).
### Other tools for interacting with Garage
The following tools can also be used to send and receive files from/to Garage:
- [minio-client](@/documentation/connect/cli.md#minio-client)
- [s3cmd](@/documentation/connect/cli.md#s3cmd)
- [rclone](@/documentation/connect/cli.md#rclone)
- [Cyberduck](@/documentation/connect/cli.md#cyberduck)
- [WinSCP](@/documentation/connect/cli.md#winscp)
An exhaustive list is maintained in the ["Integrations" > "Browsing tools" section](@/documentation/connect/_index.md).
+15 -16
View File
@@ -56,11 +56,10 @@ tls_skip_verify = false
service_name = "garage-daemon"
ca_cert = "/etc/consul/consul-ca.crt"
# for `agent` API mode, unset client_cert and client_key:
client_cert = "/etc/consul/consul-client.crt"
client_key = "/etc/consul/consul-key.crt"
# optionally enable `token` for authentication:
# for `agent` API mode, unset client_cert and client_key, and optionally enable `token`
# token = "abcdef-01234-56789"
tags = [ "dns-enabled" ]
@@ -373,7 +372,7 @@ Performance characteristics of the different DB engines are as follows:
not recommended.
- Keys in LMDB are limited to 511 bytes. This limit translates to limits on
object keys in S3 and sort keys in K2V that are limited to 479 bytes.
object keys in S3 and sort keys in K2V that are limted to 479 bytes.
- **Sqlite:** Garage supports Sqlite as an alternative storage backend for
metadata, which does not have the issues listed above for LMDB. Sqlite is
@@ -397,7 +396,7 @@ garage convert-db -a <input db engine> -i <input db path> \
```
Make sure to specify the full database path as presented in the table above
(third column), and not just the path to the metadata directory.
(third colummn), and not just the path to the metadata directory.
#### `metadata_fsync` {#metadata_fsync}
@@ -439,7 +438,7 @@ This might reduce the risk that a data block is lost in rare
situations such as simultaneous node losing power,
at the cost of a moderate drop in write performance.
Similarly to `metadata_fsync`, this is likely not necessary
Similarly to `metatada_fsync`, this is likely not necessary
if geographical replication is used.
#### `metadata_auto_snapshot_interval` (since `v0.9.4`) {#metadata_auto_snapshot_interval}
@@ -448,7 +447,7 @@ If this value is set, Garage will automatically take a snapshot of the metadata
DB file at a regular interval and save it in the metadata directory,
or in [`metadata_snapshots_dir`](#metadata_snapshots_dir) if it is set.
This parameter can take any duration string that can be parsed by
the [`fundu_systemd`](https://docs.rs/fundu-systemd) crate.
the [`parse_duration`](https://docs.rs/parse_duration/latest/parse_duration/#syntax) crate.
Snapshots can allow to recover from situations where the metadata DB file is
corrupted, for instance after an unclean shutdown. See [this
@@ -555,7 +554,7 @@ awaits for one of the `block_max_concurrent_reads` slots to be available
slot, it reads the entire block file to RAM and frees the slot as soon as the
block file is finished reading. Only after the slot is released will the
block's data start being transferred over the network. If the request fails to
acquire a reading slot within 15 seconds, it fails with a timeout error.
acquire a reading slot wihtin 15 seconds, it fails with a timeout error.
Timeout events can be monitored through the `block_read_semaphore_timeouts`
metric in Prometheus: a non-zero number of such events indicates an I/O
bottleneck on HDD read speed.
@@ -618,11 +617,11 @@ storing the secret as the `GARAGE_RPC_SECRET_FILE` environment variable.
#### `rpc_bind_addr` {#rpc_bind_addr}
The address and port on which to bind for inter-cluster communications
(referred to as RPC for remote procedure calls).
The address and port on which to bind for inter-cluster communcations
(reffered to as RPC for remote procedure calls).
The port specified here should be the same one that other nodes will used to contact
the node, even in the case of a NAT: the NAT should be configured to forward the external
port number to the same internal port number. This means that if you have several nodes running
port number to the same internal port nubmer. This means that if you have several nodes running
behind a NAT, they should each use a different RPC port number.
#### `rpc_bind_outgoing` (since `v0.9.2`) {#rpc_bind_outgoing}
@@ -785,14 +784,14 @@ manually.
#### `api_bind_addr` {#s3_api_bind_addr}
The IP and port on which to bind for accepting S3 API calls.
This endpoint does not support TLS: a reverse proxy should be used to provide it.
This endpoint does not suport TLS: a reverse proxy should be used to provide it.
Alternatively, since `v0.8.5`, a path can be used to create a unix socket with 0222 mode.
#### `s3_region` {#s3_region}
Garage will accept S3 API calls that are targeted to the S3 region defined here.
API calls targeted to other regions will fail with a AuthorizationHeaderMalformed error
Garage will accept S3 API calls that are targetted to the S3 region defined here.
API calls targetted to other regions will fail with a AuthorizationHeaderMalformed error
message that redirects the client to the correct region.
#### `root_domain` {#s3_root_domain}
@@ -800,7 +799,7 @@ message that redirects the client to the correct region.
The optional suffix to access bucket using vhost-style in addition to path-style request.
Note path-style requests are always enabled, whether or not vhost-style is configured.
Configuring vhost-style S3 required a wildcard DNS entry, and possibly a wildcard TLS certificate,
but might be required by software not supporting path-style requests.
but might be required by softwares not supporting path-style requests.
If `root_domain` is `s3.garage.eu`, a bucket called `my-bucket` can be interacted with
using the hostname `my-bucket.s3.garage.eu`.
@@ -816,7 +815,7 @@ behaviour of this module.
The IP and port on which to bind for accepting HTTP requests to buckets configured
for website access.
This endpoint does not support TLS: a reverse proxy should be used to provide it.
This endpoint does not suport TLS: a reverse proxy should be used to provide it.
Alternatively, since `v0.8.5`, a path can be used to create a unix socket with 0222 mode.
@@ -889,7 +888,7 @@ You can use any random string for this value. We recommend generating a random t
If this is set to `true`, accessing the metrics endpoint will always require
an access token. Valid tokens include the `metrics_token` if it is set,
and admin API token defined dynamically in Garage which have
and admin API token defined dynamicaly in Garage which have
the `Metrics` endpoint in their scope.
#### `trace_sink` {#admin_trace_sink}
+4 -4
View File
@@ -46,7 +46,7 @@ to select the replication mode best suited to your use case (hint: in most cases
### Compression and deduplication
All data stored in Garage is deduplicated, and optionally compressed using
All data stored in Garage is deduplicated, and optionnally compressed using
Zstd. Objects uploaded to Garage are chunked in blocks of constant sizes (see
[`block_size`](@/documentation/reference-manual/configuration.md#block_size)),
and the hashes of individual blocks are used to dispatch them to storage nodes
@@ -84,13 +84,13 @@ exposing the same content under different domain names.
Garage also supports bucket aliases which are local to a single user:
this allows different users to have different buckets with the same name, thus avoiding naming collisions.
This can be helpful for instance if you want to write an application that creates per-user buckets with always the same name.
This can be helpfull for instance if you want to write an application that creates per-user buckets with always the same name.
This feature is totally invisible to S3 clients and does not break compatibility with AWS.
### Cluster administration API
Garage provides a fully-fledged REST API to administer your cluster programmatically.
Garage provides a fully-fledged REST API to administer your cluster programatically.
Functionality included in the admin API include: setting up and monitoring
cluster nodes, managing access credentials, and managing storage buckets and bucket aliases.
A full reference of the administration API is available [here](@/documentation/reference-manual/admin-api.md).
@@ -100,7 +100,7 @@ A full reference of the administration API is available [here](@/documentation/r
Garage makes some internal metrics available in the Prometheus data format,
which allows you to build interactive dashboards to visualize the load and internal state of your storage cluster.
For developers and performance-savvy administrators,
For developpers and performance-savvy administrators,
Garage also supports exporting traces of what it does internally in OpenTelemetry format.
This allows to monitor the time spent at various steps of the processing of requests,
in order to detect potential performance bottlenecks.
+2 -1
View File
@@ -19,7 +19,7 @@ The specification of the K2V API can be found
[here](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/f8be15c37db857e177d543de7be863692628d567/doc/drafts/k2v-spec.md).
This document also includes a high-level overview of K2V's design.
The K2V API uses AWSv4 signatures for authentication, same as the S3 API.
The K2V API uses AWSv4 signatures for authentification, same as the S3 API.
The AWS region used for signature calculation is always the same as the one
defined for the S3 API in the config file.
@@ -55,3 +55,4 @@ cargo build --features cli --bin k2v-cli
The CLI utility is self-documented, run `k2v-cli --help` to learn how to use
it. There is also a short README.md in the `src/k2v-client` folder with some
instructions.
-188
View File
@@ -1,188 +0,0 @@
+++
title = "Known issues"
weight = 80
+++
Issues in each section are roughly sorted by order of decreasing impact, based on actual reports from users.
## Architectural limitations
Issues that are caused by design decisions of Garage internals, and that can't
be fixed without major architectural changes in the codebase.
### Metadata performance issues with many objects
**Related issues:**
- [#851 - Performances collapse with 10 millions pictures in a bucket](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/851)
- [#1222 - Cluster Setup Write Performance Degraded After Writing 10 Million Object (200-300Kb per object)](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1222)
### Very big objects cause performance degradation
For each object, there is a single metadata entry called a `Version` that
contains a list of all of the data blocks in the object. For very big objects,
this entry can contain thousands of block references. During the uploading of
an object, this metadata entry needs to be read, deserialized, reserialized and
written for each individual data block uploaded. This means that the
complexity of an upload is `O(n²)` in the number of blocks needed.
This manifests by excessive metadata I/O and CPU usage, and uploads eventually stalling.
**Mitigation:** Increase the `block_size` configuration parameter to reduce the
number of blocks. Make sure multipart uploads use chunks that are at least
`block_size` in size, and that are an exact multiple of `block_size` to avoid
the creation of smaller blocks.
**Long-term solution:** An architectural change in the metadata system would be
required to store block lists in many independent metadata entries instead of
one single big entry per object.
**Related issues:**
- [#662 - Large Files fail to upload](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/662)
- [#1366 - High CPU usage and performance degradation during long multipart uploads](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1366)
### No conditional writes / locking / WORM support (`if-none-match`, ...)
This is structurally impossible to implement in Garage due to the lack of a consensus algorithm,
which is one of Garage's core design choices which we cannot reconsider.
A semi-working, *unsafe* implementation of WORM and object locking could be
implemented, with the following constraint: only after the completion of the
first write (in case of WORM) or the setting of a lock (for object lock) can we
guarantee that the object cannot be overwritten. In case where an overwrite
requests arrives at the same time as the initial request to write or to lock
the object, we cannot implement a safe and consistent way to reject it. This
means that many practical use-cases for `if-none-match` cannot be supported
(e.g. using it to implement mutual exclusion between concurrent writers).
**Related issues:**
- [#1052 - Support conditional writes](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1052)
- [#1127 - Feature Request: WORM (Write Once Read Many) / Object Lock Support](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1127)
### `CreateBucket` race condition
Also due to the lack of a consensus algorithm, there is no mutual exclusion
between concurrent `CreateBucket` requests using the same bucket name.
**Related issues:**
- [#649 - Race condition in CreateBucket](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/649)
### Metadata and data have the same replication factor
There is a single `replication_factor` in the configuration file that applies both to data blocks and metadata entries.
This makes clusters with `replication_factor = 1` particularly vulnerable in cases of metadata corruption (see below), as there
is a single copy of the metadata for each object even in multi-node clusters.
**Mitigation:** Do not use `replication_factor = 1`.
**Long-term solution:** We want to allow scenarios such as replicating the
metadata on 2, 3 or more nodes and the data on only 1 or 2 nodes (for example),
so that the metadata can benefit from better redundancy without increasing the
storage costs for the entire dataset. This will require some important changes
in the codebase.
**Related issues:**
- [#720 - Separate replication modes for metadata/data](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/720)
### Node count limitation
Garage will have issues in clusters with too many nodes, it will not be able to
spread data uniformly among nodes and some nodes will fill up faster than
other. This starts to manifest when the number of nodes is bigger than `10 ×
replication_factor`. This is due to the fact that Garage uses only 256
partitions internally.
**Mitigation:** Build clusters with fewer, bigger nodes.
**Potential solution:** This can be fixed by increasing the number of
partitions in Garage. The code paths exist, there is [a `const`
somewhere](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/6fd9bba0cb55062cb1725ab961b7fa8acb9dcc61/src/rpc/layout/mod.rs#L35)
that theoretically allows to increase the number of partitions up to `2^16`,
but this has not been tested so there might be bugs.
### Buckets are not sharded
For each bucket, the first metadata layer that contains an index of all objects
is not sharded. This index, which includes the names and all metadata (size,
headers, ...) for each object, is stored on `$replication_factor` nodes.
For instance with `replication_factor = 3`, a given bucket will use only 3
specific nodes for this index (chosen at random when the bucket is created) to
store this index. In a multi-zone deployments, these nodes will be spread in
different zones. Each bucket uses a different set of 3 random nodes for its
index.
As a consequence, very large buckets might cause uneven load distribution
within a cluster. If all of the requests on a cluster are for objects in a
single bucket, then the `$replication_factor` nodes that store the index will
become a hotspot in the cluster, with more intensive metadata access patterns.
There is no way of choosing which nodes will have this role.
Currently, we have no report of this being an issue in practice.
**Mitigation:** This impacts in particular clusters that are used for a single
purpose with a single bucket. This can be solved by dividing your dataset among
many buckets, using a client-side sharding strategy that you will have to
design. Use at least as many buckets as you have nodes on your cluster.
## Bugs
Known bugs that are complex to diagnose and fix, and therefore have not been
fixed yet.
### LMDB metadata corruption
Many users have reported situations where the LMDB metadata db becomes
corrupted, sometimes after a forced shutdown of Garage or in case of power
loss. A corrupted database file is generally not recoverable.
**Mitigation:** Use a `replication_factor` of at least 2. Configure automatic
snapshotting using `metadata_auto_snapshot_interval` so that in case of
corruption you can rollback to a working database.
Note that taking filesystem-level snapshots of your `metadata_dir`, although it
is much faster and less I/O intensive than Garage's built-in snapshotting, does
not ensure that the snapshot will be consistent. If the snapshot is taking
during a metadata write, the snapshot itself might be corrupted and thus not
usable as a rollback point. Therefore, prefer using
`metadata_auto_snapshot_interval` in all cases.
### Layout updates might require manual intervention
In case of disconnected nodes, when changing the cluster layout to remove these
nodes and add other nodes instead, Garage might not be able to properly evict
the old nodes from the system. This is a built-in security measure to avoid any
inconsistent cluster states.
This manifests by several cluster layout versions staying active even after a
full resync. You can diagnose this situation with `garage layout history`,
which will give you instructions to fix it.
### Tag assignment
In the `garage layout assign` command, the `-t` argument has to be repeated
multiple times to set multiple tags on a node. Writing multiple tags separated
by commas will result in a single string.
## General footguns
Choices made by the developers that users must be aware of if they don't want
to run into potential issues.
### Resync tranquility is conservative by default
By default, the worker parameters `resync-tranquility` and `resync-worker-count` are set to very conservative values, to avoid overloading nodes with I/O when data needs to be resynchronized between nodes.
This can cause issues where the resync queue grows faster than it can be cleared, which in turn causes performance issues in the rest of Garage.
This situation is indicated by a big resync queue with few resync errors (the queue is not caused by a disconnected/malfunctionning node).
To fix it, increase the number of resync workers and reduce the resync tranquility. For instance, if you want to resync as fast as possible:
```
garage worker set -a resync-worker-count 8
garage worker set -a resync-tranquility 0
```
@@ -45,7 +45,7 @@ we suppose that OpenIO supports presigned URLs.
All endpoints that are missing on Garage will return a 501 Not Implemented.
Some `x-amz-` headers are not implemented.
### Core endpoints
### Core endoints
| Endpoint | Garage | [Openstack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) | [Ceph Object Gateway](https://docs.ceph.com/en/latest/radosgw/s3/) | [Riak CS](https://docs.riak.com/riak/cs/2.1.1/references/apis/storage/s3/index.html) | [OpenIO](https://docs.openio.io/latest/source/arch-design/s3_compliancy.html) |
|------------------------------|----------------------------------|-----------------|---------------|---------|-----|
@@ -135,12 +135,12 @@ If you need this feature, please [share your use case in our dedicated issue](ht
**PutBucketLifecycleConfiguration:** The only actions supported are
`AbortIncompleteMultipartUpload` and `Expiration` (without the
`ExpiredObjectDeleteMarker` field). All other operations are dependent on
either bucket versioning or storage classes which Garage currently does not
either bucket versionning or storage classes which Garage currently does not
implement. The deprecated `Prefix` member directly in the the `Rule`
structure/XML tag is not supported, specified prefixes must be inside the
`Filter` structure/XML tag.
**GetBucketVersioning:** Stub implementation which always returns "versioning not enabled", since Garage does not yet support bucket versioning.
**GetBucketVersioning:** Stub implementation which always returns "versionning not enabled", since Garage does not yet support bucket versionning.
### Replication endpoints
@@ -155,7 +155,7 @@ Please open an issue if you have a use case for replication.
*Note: Ceph documentation briefly says that Ceph supports
[replication through the S3 API](https://docs.ceph.com/en/latest/radosgw/multisite-sync-policy/#s3-replication-api)
but with some limitations.
Additionally, replication endpoints are not documented in the S3 compatibility page so I don't know what kind of support we can expect.*
Additionaly, replication endpoints are not documented in the S3 compatibility page so I don't know what kind of support we can expect.*
### Locking objects
@@ -197,7 +197,7 @@ Please open an issue if you have a use case.
### Vendor specific endpoints
<details><summary>Display Amazon specific endpoints</summary>
<details><summary>Display Amazon specifc endpoints</summary>
| Endpoint | Garage | [Openstack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) | [Ceph Object Gateway](https://docs.ceph.com/en/latest/radosgw/s3/) | [Riak CS](https://docs.riak.com/riak/cs/2.1.1/references/apis/storage/s3/index.html) | [OpenIO](https://docs.openio.io/latest/source/arch-design/s3_compliancy.html) |
@@ -234,3 +234,4 @@ Please open an issue if you have a use case.
| [SelectObjectContent](https://docs.aws.amazon.com/AmazonS3/latest/API/API_SelectObjectContent.html) | ❌ Missing | ❌| ❌| ❌| ❌|
</details>
@@ -3,7 +3,7 @@ title = "S3 compatibility target"
weight = 5
+++
If there is a specific S3 functionality you have a need for, feel free to open
If there is a specific S3 functionnality you have a need for, feel free to open
a PR to put the corresponding endpoints higher in the list. Please explain
your motivations for doing so in the PR message.
+2 -2
View File
@@ -68,7 +68,7 @@ Workflow for DELETE:
1. Check write permission (LDAP)
2. Get current version (or versions) in object table
3. Do the deletion of those versions NOT IN A BACKGROUND JOB THIS TIME
4. Return success to the user if we were able to delete blocks from the blocks table and entries from the object table
4. Return succes to the user if we were able to delete blocks from the blocks table and entries from the object table
To delete a version:
@@ -92,7 +92,7 @@ Known issue: if someone is reading from a version that we want to delete and the
- file path = /meta/(first 3 hex digits of hash)/(rest of hash)
- map block hash -> set of version UUIDs where it is referenced
Useful metadata:
Usefull metadata:
- list of versions that reference this block in the Casandra table, so that we can do GC by checking in Cassandra that the lines still exist
- list of other nodes that we know have acknowledged a write of this block, useful in the rebalancing algorithm
+3 -3
View File
@@ -49,12 +49,12 @@ The ring construction that selects `n_token` random positions for each nodes giv
is not well-balanced: the space between the tokens varies a lot, and some partitions are thus bigger than others.
This problem was demonstrated in the original Dynamo DB paper.
To solve this, we want to apply a better second method for partitioning our dataset:
To solve this, we want to apply a better second method for partitionning our dataset:
1. fix an initially large number of partitions (say 1024) with evenly-spaced delimiters,
2. attribute each partition randomly to a node, with a probability
proportional to its capacity (which `n_tokens` represented in the first
proportionnal to its capacity (which `n_tokens` represented in the first
method)
For now we continue using the multi-DC ring walking described above.
@@ -66,7 +66,7 @@ I have studied two ways to do the attribution of partitions to nodes, in a way t
MagLev provided significantly better balancing, as it guarantees that the exact
same number of partitions is attributed to all nodes that have the same
capacity (and that this number is proportional to the node's capacity, except
capacity (and that this number is proportionnal to the node's capacity, except
for large values), however in both cases:
- the distribution is still bad, because we use the naive multi-DC ring walking
+1 -1
View File
@@ -19,7 +19,7 @@ The migration steps are as follows:
2. Disable API and web access. Garage does not support disabling
these endpoints but you can change the port number or stop your reverse
proxy for instance.
3. Check once again that your cluster is healthy. Run again `garage repair --all-nodes --yes tables` which is quick.
3. Check once again that your cluster is healty. Run again `garage repair --all-nodes --yes tables` which is quick.
Also check your queues are empty, run `garage stats` to query them.
4. Turn off Garage v0.6
5. Backup the metadata folder of all your nodes: `cd /var/lib/garage ; tar -acf meta-v0.6.tar.zst meta/`
@@ -28,11 +28,11 @@ We should try to test in least invasive ways, i.e. minimize the impact of the te
- Not making `garage` a shared library (launch using `execve`, it's perfectly fine)
Instead, we should focus on building a clean outer interface for the `garage` binary,
for example loading configuration using environment variables instead of the configuration file if that's helpful for writing the tests.
for example loading configuration using environnement variables instead of the configuration file if that's helpfull for writing the tests.
There are two reasons for this:
- Keep the source code clean and focused
- Keep the soure code clean and focused
- Test something that is as close as possible as the true garage that will actually be running
Reminder: rules of simplicity, concerning changes to Garage's source code.
@@ -71,3 +71,5 @@ Interesting blog posts on the blog of the Sled database:
Misc:
- [mutagen](https://github.com/llogiq/mutagen) - mutation testing is a way to assert our test quality by mutating the code and see if the mutation makes the tests fail
- [fuzzing](https://rust-fuzz.github.io/book/) - cargo supports fuzzing, it could be a way to test our software reliability in presence of garbage data.
+5 -5
View File
@@ -176,7 +176,7 @@ Returns the cluster's current health in JSON format, with the following variable
- degraded: Garage node is not connected to all storage nodes, but a quorum of write nodes is available for all partitions
- unavailable: a quorum of write nodes is not available for some partitions
- `knownNodes`: the number of nodes this Garage node has had a TCP connection to since the daemon started
- `connectedNodes`: the number of nodes this Garage node currently has an open connection to
- `connectedNodes`: the nubmer of nodes this Garage node currently has an open connection to
- `storageNodes`: the number of storage nodes currently registered in the cluster layout
- `storageNodesOk`: the number of storage nodes to which a connection is currently open
- `partitions`: the total number of partitions of the data (currently always 256)
@@ -379,7 +379,7 @@ Example response:
]
```
#### GetKeyInfo `GET /v2/GetKeyInfo?id=<access key id>`
#### GetKeyInfo `GET /v2/GetKeyInfo?id=<acces key id>`
#### GetKeyInfo `GET /v2/GetKeyInfo?search=<pattern>`
Returns information about the requested API access key.
@@ -388,7 +388,7 @@ If `id` is set, the key is looked up using its exact identifier (faster).
If `search` is set, the key is looked up using its name or prefix
of identifier (slower, all keys are enumerated to do this).
Optionally, the query parameter `showSecretKey=true` can be set to reveal the
Optionnally, the query parameter `showSecretKey=true` can be set to reveal the
associated secret access key.
Example response:
@@ -487,7 +487,7 @@ Request body format:
This returns the key info in the same format as the result of GetKeyInfo.
#### UpdateKey `POST /v2/UpdateKey?id=<access key id>`
#### UpdateKey `POST /v2/UpdateKey?id=<acces key id>`
Updates information about the specified API access key.
@@ -509,7 +509,7 @@ The possible flags in `allow` and `deny` are: `createBucket`.
This returns the key info in the same format as the result of GetKeyInfo.
#### DeleteKey `POST /v2/DeleteKey?id=<access key id>`
#### DeleteKey `POST /v2/DeleteKey?id=<acces key id>`
Deletes an API access key.
+6 -53
View File
@@ -35,7 +35,7 @@ Triples in K2V are constituted of three fields:
partition key in which the client wants to read/delete lists of items
- a sort key (`sk`), an utf8 string that defines the index of the triplet inside its
partition; triplets are uniquely identified by their partition key + sort key
partition; triplets are uniquely idendified by their partition key + sort key
- a value (`v`), an opaque binary blob associated to the partition key + sort key;
they are transmitted as binary when possible but in most case in the JSON API
@@ -74,7 +74,7 @@ are obsoleted by the new write.
**Basic insertion.** To insert a new value `v4` with context `[(node1, t2), (node2, t3)]`, in a
simple case where there was no insertion in-between reading the value
mentioned above and writing `v4`, and supposing that node2 receives the
mentionned above and writing `v4`, and supposing that node2 receives the
InsertItem query:
- `node2` generates a timestamp `t4` such that `t4 > t3`.
@@ -166,25 +166,6 @@ that map to zeroes. Note that we need to filter out values from nodes that are
no longer part of the cluster layout, as when nodes are removed they won't
necessarily have had the time to set their counters to zero.
### Consistency guarantees
K2V provides the following consistency guarantees:
**Read after Write**. After a write has been acknowledged (the request returned
successfully), a subsequent read is guaranteed to contain the value that was
written.
**Monotonic Reads**. Two sequential reads will return values in an order that is
consistent with the order in which they are written (e.g. by concurrent writes).
For example, consider a scenario where a value is set initially set to 0 and a
request writing 1 is performed. Doing two subsequent reads concurrently with the
write is guaranteed to return either `0`, `0` or `0`,`1` or `1`,`1`, but not
`1`,`0`.
It is also possible to perform non-monotonic reads (allowing this last
behavior), which are slightly faster than monotonic reads. This is done by
passing a dedicated flag to read operations (see the endpoints documentation).
## Important details
**THIS SECTION CONTAINS A FEW WARNINGS ON THE K2V API WHICH ARE IMPORTANT
@@ -229,12 +210,6 @@ Query parameters:
|------------|---------------|----------------------------------|
| `sort_key` | **mandatory** | The sort key of the item to read |
Headers:
| name | default value | meaning |
|-------------------------------|---------------|------------------------------------------|
| `X-Garage-Non-Monotonic-Read` | `false` | Whether to allow for non-monotonic reads |
Returns the item with specified partition key and sort key. Values can be
returned in either of two ways:
@@ -350,12 +325,6 @@ Query parameters:
The timeout can be set to any number of seconds, with a maximum of 600 seconds (10 minutes).
Headers:
| name | default value | meaning |
|-------------------------------|---------------|------------------------------------------|
| `X-Garage-Non-Monotonic-Read` | `false` | Whether to allow for non-monotonic reads |
**InsertItem: `PUT /<bucket>/<partition key>?sort_key=<sort_key>`**
@@ -363,7 +332,7 @@ Inserts a single item. This request does not use JSON, the body is sent directly
To supersede previous values, the HTTP header `X-Garage-Causality-Token` should
be set to the causality token returned by a previous read on this key. This
header can be omitted for the first writes to the key.
header can be ommitted for the first writes to the key.
Example query:
@@ -428,7 +397,7 @@ smallest partition key that exists. It returns partition keys in increasing
order, or decreasing order if `reverse` is set to `true`,
and stops when either of the following conditions is met:
1. if `end` is specified, the partition key `end` is reached or surpassed (if it
1. if `end` is specfied, the partition key `end` is reached or surpassed (if it
is reached exactly, it is not included in the result)
2. if `limit` is specified, `limit` partition keys have been listed
@@ -522,7 +491,7 @@ the triplet is inserted for the first time, the causality token should be set to
The value is expected to be a base64-encoded binary blob. The value `null` can
also be used to delete the triplet while preserving causality information: this
allows to know if a delete has happened concurrently with an insert, in which
allows to know if a delete has happenned concurrently with an insert, in which
case both are preserved and returned on reads (see below).
Partition keys and sort keys are utf8 strings which are stored sorted by
@@ -552,14 +521,6 @@ HTTP/1.1 204 NO CONTENT
Batch read of triplets in a bucket.
Headers:
| name | default value | meaning |
|-------------------------------|---------------|------------------------------------------|
| `X-Garage-Non-Monotonic-Read` | `false` | Whether to allow for non-monotonic reads |
Body:
The request body is a JSON list of searches, that each specify a range of
items to get (to get single items, set `singleItem` to `true`). A search is a
JSON struct with the following fields:
@@ -579,7 +540,7 @@ JSON struct with the following fields:
For each of the searches, triplets are listed and returned separately. The
semantics of `prefix`, `start`, `end`, `limit` and `reverse` are the same as for ReadIndex. The
additional parameter `singleItem` allows to get a single item, whose sort key
additionnal parameter `singleItem` allows to get a single item, whose sort key
is the one given in `start`. Parameters `conflictsOnly` and `tombstones`
control additional filters on the items that are returned.
@@ -750,14 +711,6 @@ HTTP/1.1 200 OK
Polls a range of items for changes.
Headers:
| name | default value | meaning |
|-------------------------------|---------------|------------------------------------------|
| `X-Garage-Non-Monotonic-Read` | `false` | Whether to allow for non-monotonic reads |
Body:
The query body is a JSON object consisting of the following fields:
| name | default value | meaning |
+1 -1
View File
@@ -59,7 +59,7 @@ To link the effective storage capacity of the cluster to partition assignment, w
\end{equation}
This assumption is justified by the dispersion of the hashing function, when the number of partitions is small relative to the number of stored blocks.
Every node $n$ will store some number $p_n$ of partitions (it is the number of partitions $p$ such that $n$ appears in the $\alpha_p$). Hence the partitions stored by $n$ (and hence all partitions by our assumption) have there size bounded by $c_n/p_n$. This remark leads us to define the optimal size that we will want to maximize:
Every node $n$ wille store some number $p_n$ of partitions (it is the number of partitions $p$ such that $n$ appears in the $\alpha_p$). Hence the partitions stored by $n$ (and hence all partitions by our assumption) have there size bounded by $c_n/p_n$. This remark leads us to define the optimal size that we will want to maximize:
\begin{equation}
\label{eq:optimal}
+10 -10
View File
@@ -38,7 +38,7 @@ We would like to compute an assignment of nodes to partitions. We will impose so
\end{equation}
This assumption is justified by the dispersion of the hashing function, when the number of partitions is small relative to the number of stored large objects.
Every node $n$ will store some number $k_n$ of partitions. Hence the partitions stored by $n$ (and hence all partitions by our assumption) have there size bounded by $c_n/k_n$. This remark leads us to define the optimal size that we will want to maximize:
Every node $n$ wille store some number $k_n$ of partitions. Hence the partitions stored by $n$ (and hence all partitions by our assumption) have there size bounded by $c_n/k_n$. This remark leads us to define the optimal size that we will want to maximize:
\begin{equation}
\label{eq:optimal}
@@ -62,7 +62,7 @@ For now, in the following, we ask the following redundancy constraint:
\textbf{Mode 3:} every partition needs to be assignated to three nodes. We try to spread the three nodes over different zones as much as possible.
\textbf{Warning:} This is a working document written incrementally. The last version of the algorithm is the \textbf{parametric assignment} described in the next section.
\textbf{Warning:} This is a working document written incrementaly. The last version of the algorithm is the \textbf{parametric assignment} described in the next section.
\section{Computation of a parametric assignment}
@@ -318,7 +318,7 @@ $$
$$
which is the universal upper bound on $s^*$. Hence any optimal utilization $(n_v)$ can be modified to another optimal utilization such that $n_v\ge \hat{n}_v$
Because $z_0$ cannot store more than $N$ partition occurrences, in any assignment, at least $2N$ partitions must be assignated to the zones $Z\setminus\{z_0\}$. Let $C_0 = C-c_{z_0}$. Suppose that there exists a zone $z_1\neq z_0$ such that $c_{z_1}/C_0 \ge 1/2$. Then, with the same argument as for $z_0$, we can define
Because $z_0$ cannot store more than $N$ partition occurences, in any assignment, at least $2N$ partitions must be assignated to the zones $Z\setminus\{z_0\}$. Let $C_0 = C-c_{z_0}$. Suppose that there exists a zone $z_1\neq z_0$ such that $c_{z_1}/C_0 \ge 1/2$. Then, with the same argument as for $z_0$, we can define
$$\hat{n}_v = \left\lfloor\frac{c_v}{c_{z_1}}N\right\rfloor$$
for every $v\in z_1$.
@@ -351,7 +351,7 @@ Define $3N$ tokens $t_1,\ldots, t_{3N}\in V$ as follows:
Then for $1\le i \le N$, define the triplet $T_i$ to be
$(t_i, t_{i+N}, t_{i+2N})$. Since the same nodes of a zone appear contiguously, the three nodes of a triplet must belong to three distinct zones.
However simple, this solution to go from an utilization to an assignment has the drawback of not spreading the triplets: a node will tend to be associated to the same two other nodes for many partitions. Hence, during data transfer, it will tend to use only two link, instead of spreading the bandwidth use over many other links to other nodes. To achieve this goal, we will reframe the search of an assignment as a flow problem. and in the flow algorithm, we will introduce randomness in the order of exploration. This will be sufficient to obtain a good dispersion of the triplets.
However simple, this solution to go from an utilization to an assignment has the drawback of not spreading the triplets: a node will tend to be associated to the same two other nodes for many partitions. Hence, during data transfer, it will tend to use only two link, instead of spreading the bandwith use over many other links to other nodes. To achieve this goal, we will reframe the search of an assignment as a flow problem. and in the flow algorithm, we will introduce randomness in the order of exploration. This will be sufficient to obtain a good dispersion of the triplets.
\begin{figure}
\centering
@@ -436,7 +436,7 @@ T_3=(b,c,d').
$$
One can check that in this case, it is impossible to minimize both the number of zone and node changes.
Because of the redundancy constraint, we cannot use a greedy algorithm to just replace nodes in the triplets to try to get the new utilization rate: this could lead to blocking situation where there is still a hole to fill in a triplet but no available node satisfies the zone separation constraint. To circumvent this issue, we propose an algorithm based on finding cycles in a graph encoding of the assignment. As in section \ref{sec:opt_assign}, we can explore the neighbours in a random order in the graph algorithms, to spread the triplets distribution.
Because of the redundancy constraint, we cannot use a greedy algorithm to just replace nodes in the triplets to try to get the new utilization rate: this could lead to blocking situation where there is still a hole to fill in a triplet but no available node satisfies the zone separation constraint. To circumvent this issue, we propose an algorithm based on finding cycles in a graph encoding of the assignment. As in section \ref{sec:opt_assign}, we can explore the neigbours in a random order in the graph algorithms, to spread the triplets distribution.
\subsubsection{Minimizing the zone discrepancy}
@@ -550,8 +550,8 @@ We give some considerations of worst case complexity for these algorithms. In th
Algorithm \ref{alg:util} can be implemented with complexity $O(\#V^2)$. The complexity of the function call at line \ref{lin:subutil} is $O(\#V)$. The difference between the sum of the subutilizations and $3N$ is at most the sum of the rounding errors when computing the $\hat{n}_v$. Hence it is bounded by $\#V$ and the loop at line \ref{lin:loopsub} is iterated at most $\#V$ times. Finding the minimizing $v$ at line \ref{lin:findmin} takes $O(\#V)$ operations (naively, we could also use a heap).
Algorithm \ref{alg:opt} can be implemented with complexity $O(N^3\times \#Z)$. The flow graph has $O(N+\#Z)$ vertices and $O(N\times \#Z)$ edges. Dinic's algorithm has complexity $O(\#\mathrm{Vertices}^2\#\mathrm{Edges})$ hence in our case it is $O(N^3\times \#Z)$.
Algorithm \ref{alg:mini} can be implemented with complexity $O(N^3\# Z)$ under \eqref{hyp:A} and $O(N^3 \#Z \#V)$ under \eqref{hyp:B}.
Algorithm \ref{alg:mini} can be implented with complexity $O(N^3\# Z)$ under \eqref{hyp:A} and $O(N^3 \#Z \#V)$ under \eqref{hyp:B}.
The graph $G_T$ has $O(N)$ vertices and $O(N\times \#Z)$ edges under assumption \eqref{hyp:A} and respectively $O(N\times \#Z)$ vertices and $O(N\times \#V)$ edges under assumption \eqref{hyp:B}. The loop at line \ref{lin:repeat} is iterated at most $N$ times since the distance between $T$ and $T'$ decreases at every iteration. Bellman-Ford algorithm has complexity $O(\#\mathrm{Vertices}\#\mathrm{Edges})$, which in our case amounts to $O(N^2\# Z)$ under \eqref{hyp:A} and $O(N^2 \#Z \#V)$ under \eqref{hyp:B}.
\begin{algorithm}
@@ -637,7 +637,7 @@ We try to maximize $s^*$ defined in \eqref{eq:optimal}. So we can compute the op
\subsection{Computation of a candidate assignment}
To compute a candidate assignment (that does not optimize zone spreading nor distance to a previous assignment yet), we can use the following flow problem.
To compute a candidate assignment (that does not optimize zone spreading nor distance to a previous assignment yet), we can use the folowing flow problem.
Define the oriented weighted graph $(X,E)$. The set of vertices $X$ contains the source $\mathbf{s}$, the sink $\mathbf{t}$, vertices
$\mathbf{x}_p, \mathbf{u}^+_p, \mathbf{u}^-_p$ for every partition $p$, vertices $\mathbf{y}_{p,z}$ for every partition $p$ and zone $z$, and vertices $\mathbf{z}_v$ for every node $v$.
@@ -680,14 +680,14 @@ Given the flow $f$, let $G_f=(X',E_f)$ be the multi-graph where $X' = X\setminus
\end{itemize}
To summarize, arcs are oriented left to right if they correspond to a presence of flow in $f$, and right to left if they correspond to an absence of flow. They are positively weighted if we want them to stay at their current state, and negatively if we want them to switch. Let us compute the weight of such graph.
\begin{multiline*}
\begin{multline*}
w(G_f) = \sum_{e\in E_f} w(e_f) \\
=
(\alpha - \beta -\gamma) N_1 + (\alpha +\beta - \gamma) N_2 + (\alpha+\beta+\gamma) N_3
\\ +
\#V\times N - 4 \sum_p 3-\#(T_p\cap T'_p) \\
=(\#V-12+\alpha-\beta-\gamma)\times N + 4Q_V + 2\beta N_2 + 2(\beta+\gamma) N_3 \\
\end{multiline*}
\end{multline*}
As for the mode 3-strict, one can check that the difference of two such graphs corresponding to the same $(n_v)$ is always eulerian. Hence we can navigate in this class with the same greedy algorithm that discovers positive cycles and flips them.
-18
View File
@@ -1,18 +0,0 @@
*
!*.txt
!*.md
!assets
!.gitignore
!*.svg
!*.png
!*.jpg
!*.tex
!Makefile
!.gitignore
!assets/*.drawio.pdf
talk.{nav,out,snm,toc,aux,log}
!talk.pdf
-3
View File
@@ -1,3 +0,0 @@
talk.pdf: talk.tex
pdflatex talk.tex
Binary file not shown.

Before

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 297 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 240 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 43 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 82 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 129 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 233 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 52 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 58 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 145 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 13 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 14 KiB

-330
View File
@@ -1,330 +0,0 @@
%\nonstopmode
\documentclass[aspectratio=169]{beamer}
\usepackage[utf8]{inputenc}
% \usepackage[frenchb]{babel}
\usepackage{amsmath}
\usepackage{mathtools}
\usepackage{breqn}
\usepackage{multirow}
\usetheme{boxes}
\usepackage{graphicx}
%\useoutertheme[footline=authortitle,subsection=false]{miniframes}
\beamertemplatenavigationsymbolsempty
\definecolor{TitleOrange}{RGB}{255,137,0}
\setbeamercolor{title}{fg=TitleOrange}
\setbeamercolor{frametitle}{fg=TitleOrange}
\definecolor{ListOrange}{RGB}{255,145,5}
\setbeamertemplate{itemize item}{\color{ListOrange}$\blacktriangleright$}
\definecolor{verygrey}{RGB}{70,70,70}
\setbeamercolor{normal text}{fg=verygrey}
\usepackage{tabu}
\usepackage{multicol}
\usepackage{vwcol}
\usepackage{stmaryrd}
\usepackage{graphicx}
\usepackage[normalem]{ulem}
\title{Garage Object Storage: 2.0 update and best practices}
\subtitle{a new storage platform for self-hosted geo-distributed clusters}
\author{Maximilien Richer, Deuxfleurs}
\date{FOSDEM '26}
\begin{document}
\begin{frame}
\centering
\includegraphics[width=.3\linewidth]{../../sticker/Garage.pdf}
\vspace{1em}
{\large\bf Maximilien Richer, Deuxfleurs}
\vspace{1em}
\url{https://garagehq.deuxfleurs.fr/}
Matrix channel: \texttt{\#garage:deuxfleurs.fr}
\end{frame}
\begin{frame}
\frametitle{Our objective at Deuxfleurs}
\begin{center}
French association promoting digital sovereignty and privacy\\
through self-hosting hosting \textbf{as an alternative to large cloud providers}
\end{center}
\vspace{2em}
\vspace{2em}
\begin{center}
\textbf{This requires \underline{resilience}}\\
{\footnotesize (we want good uptime/availability with low supervision)}
\end{center}
\end{frame}
\begin{frame}
\frametitle{But what is Garage, exactly?}
\textbf{Garage is a self-hosted drop-in replacement for the Amazon S3 object store}\\
\vspace{.5em}
that implements resilience through geographical redundancy on commodity hardware
\begin{center}
\includegraphics[width=.8\linewidth]{assets/garageuses.png}
\end{center}
\end{frame}
\begin{frame}
\frametitle{What makes Garage different?}
\textbf{Coordination-free:}
\vspace{2em}
\begin{itemize}
\item No Raft or Paxos
\vspace{1em}
\item Internal data types are CRDTs
\vspace{1em}
\item All nodes are equivalent (no master/leader/index node)
\end{itemize}
\vspace{2em}
$\to$ less sensitive to higher latencies between nodes
\end{frame}
\begin{frame}
\frametitle{What makes Garage different?}
\begin{center}
TODO update with latest garage and minio versions
\includegraphics[width=.9\linewidth]{assets/endpoint-latency-dc.png}
\end{center}
\end{frame}
\begin{frame}
\frametitle{What makes Garage different?}
\textbf{Consistency model:}
\vspace{2em}
\begin{itemize}
\item Not ACID (not required by S3 spec) / not linearizable
\vspace{1em}
\item \textbf{Read-after-write consistency}\\
{\footnotesize (stronger than eventual consistency)}
\end{itemize}
\end{frame}
\begin{frame}
\frametitle{What makes Garage different?}
\textbf{Location-aware:}
\vspace{2em}
\begin{center}
\includegraphics[width=\linewidth]{assets/location-aware.png}
\end{center}
\vspace{2em}
Garage replicates data on different zones when possible
\end{frame}
\begin{frame}
\frametitle{What makes Garage different?}
\begin{center}
\includegraphics[width=.8\linewidth]{assets/map.png}
\end{center}
\end{frame}
\begin{frame}
\frametitle{An ever-increasing compatibility list}
\begin{center}
\includegraphics[width=.7\linewidth]{assets/compatibility.png}
\end{center}
\end{frame}
\begin{frame}
\frametitle{Version history and roadmap}
\begin{itemize}
\item v0.3: initial beta release (2021)
\item v0.7: first released version (2022)
\item v1.0: stable release (2024), will be deprecated in summer 2026 1y after v2.0 was released
\item v2.0: stable release (2025)
\begin{itemize}
\item new HTTP admin API
\item reworded replication configuration: \texttt{replication\_mode} changed to \texttt{replication\_factor} \& \texttt{consistency\_policy}
\end{itemize}
\item
\end{itemize}
\begin{center}
v3.0: TBA may include versionning support, tag on buckets and objets, retention policies...
\end{center}
\end{frame}
\begin{frame}
\centering
{\large\bf Best practices for Garage deployments}
\end{frame}
\begin{frame}
\frametitle{Things you should know}
\begin{itemize}
\item no TLS support, use your own proxy
\item no anonymous access (use website endpoint)
\item you need to assign roles to nodes manually
\item the replication factor cannot be changed easily
\item the default region is \texttt{garage} and not \texttt{us-east-1}
\item only use the \texttt{degraded} consistency policy for data recovery!
\end{itemize}
\end{frame}
\begin{frame}
\frametitle{What hardware should I use?}
\begin{itemize}
\item do NOT use network file storage (NFS, SMB, etc.) for \texttt{\/metadata}
\item get a \textbf{write-intensive flash disk} for the \texttt{\/metadata} folder
\item set \texttt{metadata} on a RAID1 if possible, with a COW filesystem (e.g. Btrfs or ZFS)
\item get large HDDs for the \texttt{\/data} folder
\item use XFS and garage multi-hdd mode for best performance
\item you can use a RAID for data but you'll leave a lot of performance on the table
\end{itemize}
\center\textit{Garage doesn't require a powerful CPUs nor much RAM, but your performance will depend on your disks!}
\end{frame}
\begin{frame}
\frametitle{Picking a metadata engine}
All files-to-block mappings are stored in the metadata engine, including bucket and object metadata. Files below 3KB are stored directly in the metadata engine.
\vspace{1em}
\begin{itemize}
\item Sled: removed in 1.x, move to SQLite or LMDB
\item \textbf{SQLite}: safer, \textbf{recommended for small clusters and single-node}
\item LMDB: faster, recommended for large clusters with metadata redundancy
\begin{itemize}
\item Warning: limited to 480 bytes per key with LMDB (not an issue in practice)
\end{itemize}
\item Fjall: experimental but promising rust-native engine, test it and let us know!
\end{itemize}
\center{Metadata engine can be set node per node, and changed later with a migration tool}
\end{frame}
\begin{frame}
\frametitle{Single-node deployment}
\begin{itemize}
\item garage was initially designed for multi-node deployments
\item single-node deployments are possible, but you will lose resilience
\item \textbf{If you do please ensure you have backups} (especially for metadata)
\begin{itemize}
\item set up \texttt{metadata\_auto\_snapshot\_interval}
\end{itemize}
\item use sqlite to minimize data loss risks on powercuts
\item or use a UPS!
\end{itemize}
\vspace{1em}
Use \texttt{github.com/bikeshedder/garage-single-node} for an easy single-node setup!
\end{frame}
\begin{frame}
\frametitle{Multi-node deployment}
\begin{itemize}
\item try to have geo-distributed zones
\item multiple nodes per zone to add more capacity
\item at least 3 zones for best resilience
\item keep in mind your available network and IO bandwidth
\item \textbf{Rebalancing a cluster can take multiple weeks with large HDDs and slow network links}
\item monitor your nodes with Prometheus + Grafana
\end{itemize}
\center{Deuxfleurs has been running a 9TB (3TB usable) 8-nodes cluster (3+3+2) over retail fiber (10ms site-to-site latency) for close to 5 years now. We heard there are petabyte clusters out there!}
\end{frame}
\begin{frame}
\frametitle{Deploying and administering garage at scale}
\begin{itemize}
\item deploy with your favorite tool (eg. Ansible) and system manager (eg. systemd)
\item or use Docker, docker-compose, Kubernetes or Nomad
\item Kubernetes and Consul are supported for node-to-node discovery
\begin{itemize}
\item you'll still have to manage the layout manually!
\end{itemize}
\item use gateway nodes to optimize network usage
\item ajust \texttt{resync-tranquility} and \texttt{scrub-tranquility} to your ressources
\end{itemize}
\center{Kubernetes storage controller: \texttt{github.com/bmarinov/garage-storage-controller}}
\end{frame}
\begin{frame}
\frametitle{Community UI available!}
\begin{center}
\includegraphics[width=0.9\linewidth]{assets/community-ui.png}\\
\vspace{-1em}
\url{https://github.com/khairul169/garage-webui}
\end{center}
\end{frame}
\begin{frame}
\frametitle{Official Embedded UI comming later this year!}
\begin{center}
\includegraphics[width=0.9\linewidth]{assets/Garage Web Admin - Dashboard@2x.png}\\
\vspace{-1em}
\end{center}
\end{frame}
\begin{frame}
\frametitle{Official Embedded UI comming this year!}
\begin{center}
\includegraphics[width=0.9\linewidth]{assets/Garage Web Admin - Bucket details page@2x.png}\\
\vspace{-1em}
\end{center}
\end{frame}
\begin{frame}
\frametitle{How to make sense of garage metrics?}
\begin{center}
\includegraphics[width=0.7\linewidth]{assets/garage-stats.png}\\
\vspace{-1em}
\end{center}
\end{frame}
\begin{frame}
\frametitle{What if things go wrong?}
\begin{itemize}
\item set logs to debug with \texttt{RUST_LOG=garage_api_common=debug,garage_api_s3=debug,garage=debug}
\item auth issues: check your reverse proxy configuration
\item slow resync: check your network and disk IO usage, and \texttt{resync-tranquility} worker configuration
\item big LMDB database: stop garage and compact with \texttt{mdb\_copy -c}
\item ask us on matrix \texttt{\#garage:deuxfleurs.fr} or open an issue on git.deuxfleurs.fr!
\begin{itemize}
\item provide the output of \texttt{garage status}, \texttt{garage stats} and relevant metrics and logs
\end{itemize}
\end{itemize}
\end{frame}
\begin{frame}
\frametitle{Moving from Minio}
\begin{itemize}
\item list your buckets and your keys
\item create buckets and keys on the garage cluster
\begin{itemize}
\item you cannot import non-garage keys yet, patch to come soon!
\end{itemize}
\item loop over buckets, copy with rclone
\begin{itemize}
\item see doc \url{https://garagehq.deuxfleurs.fr/documentation/connect/cli/}
\end{itemize}
\item blog post coming soon!
\end{itemize}
\end{frame}
\begin{frame}
\frametitle{Demo time!}
\end{frame}
\begin{frame}
\frametitle{Get Garage now!}
\begin{center}
\includegraphics[width=.3\linewidth]{../../logo/garage_hires.png}\\
\vspace{-1em}
\url{https://garagehq.deuxfleurs.fr/}\\
Matrix channel: \texttt{\#garage:deuxfleurs.fr}
\vspace{2em}
\includegraphics[width=.09\linewidth]{assets/rust_logo.png}
\includegraphics[width=.2\linewidth]{assets/AGPLv3_Logo.png}
\end{center}
\end{frame}
\end{document}
%% vim: set ts=4 sw=4 tw=0 noet spelllang=fr :
Generated
+4 -4
View File
@@ -81,17 +81,17 @@
]
},
"locked": {
"lastModified": 1776914043,
"narHash": "sha256-qug5r56yW1qOsjSI99l3Jm15JNT9CvS2otkXNRNtrPI=",
"lastModified": 1763952169,
"narHash": "sha256-+PeDBD8P+NKauH+w7eO/QWCIp8Cx4mCfWnh9sJmy9CM=",
"owner": "oxalica",
"repo": "rust-overlay",
"rev": "2d35c4358d7de3a0e606a6e8b27925d981c01cc3",
"rev": "ab726555a9a72e6dc80649809147823a813fa95b",
"type": "github"
},
"original": {
"owner": "oxalica",
"repo": "rust-overlay",
"rev": "2d35c4358d7de3a0e606a6e8b27925d981c01cc3",
"rev": "ab726555a9a72e6dc80649809147823a813fa95b",
"type": "github"
}
},
+3 -11
View File
@@ -6,9 +6,9 @@
inputs.nixpkgs.url =
"github:NixOS/nixpkgs/cfe2c7d5b5d3032862254e68c37a6576b633d632";
# Rust overlay as of 2026-04-23
# Rust overlay as of 2025-11-24
inputs.rust-overlay.url =
"github:oxalica/rust-overlay/2d35c4358d7de3a0e606a6e8b27925d981c01cc3";
"github:oxalica/rust-overlay/ab726555a9a72e6dc80649809147823a813fa95b";
inputs.rust-overlay.inputs.nixpkgs.follows = "nixpkgs";
# Crane as of 2025-01-24
@@ -67,7 +67,7 @@
clippy = lints.garage-cargo-clippy;
};
# ---- development shell, for making native builds only ----
# ---- developpment shell, for making native builds only ----
devShells =
let
targets = compile {
@@ -95,14 +95,6 @@
killall
];
};
# dev shell for fuzzing
fuzz = pkgs.mkShell {
buildInputs = with pkgs; [
targets.toolchainNightly
cargo-fuzz
];
};
};
});
}
-4
View File
@@ -1,4 +0,0 @@
target
corpus
artifacts
coverage
-73
View File
@@ -1,73 +0,0 @@
[package]
name = "garage-fuzz"
version = "0.0.0"
publish = false
edition = "2018"
[package.metadata]
cargo-fuzz = true
[dependencies]
arbitrary = { workspace = true, features = ["derive"]}
libfuzzer-sys = { workspace = true }
garage_db.workspace = true
garage_table.workspace = true
garage_util.workspace = true
garage_model = { workspace = true, default-features = false, features = ["arbitrary", "k2v"] }
[[bin]]
name = "version_crdt"
path = "fuzz_targets/version_crdt.rs"
test = false
doc = false
bench = false
[[bin]]
name = "mpu_crdt"
path = "fuzz_targets/mpu_crdt.rs"
test = false
doc = false
bench = false
[[bin]]
name = "bucket_crdt"
path = "fuzz_targets/bucket_crdt.rs"
test = false
doc = false
bench = false
[[bin]]
name = "block_ref_crdt"
path = "fuzz_targets/block_ref_crdt.rs"
test = false
doc = false
bench = false
[[bin]]
name = "admin_api_token_crdt"
path = "fuzz_targets/admin_api_token_crdt.rs"
test = false
doc = false
bench = false
[[bin]]
name = "key_crdt"
path = "fuzz_targets/key_crdt.rs"
test = false
doc = false
bench = false
[[bin]]
name = "bucket_alias_crdt"
path = "fuzz_targets/bucket_alias_crdt.rs"
test = false
doc = false
bench = false
[[bin]]
name = "k2v_item_crdt"
path = "fuzz_targets/k2v_item_crdt.rs"
test = false
doc = false
bench = false
-11
View File
@@ -1,11 +0,0 @@
# Fuzzing
## Setup
Install cargo fuzz: `cargo install cargo-fuzz`
## Launch
Run `cargo fuzz run <fuzz_target>` where `<fuzz_target>` is the name (without extension) of one of the `.rs` files in the `fuzz_targets` directory.
If you launch the command outside of the fuzz directory, you need to force the nightly toolchain with `cargo +nightly`.
-38
View File
@@ -1,38 +0,0 @@
#![no_main]
use garage_fuzz::check_crdt_laws;
use garage_model::admin_token_table::{AdminApiToken, AdminApiTokenParams, AdminApiTokenScope};
use garage_model::permission::ExpirationTime;
use garage_util::crdt;
use libfuzzer_sys::fuzz_target;
type Input = (
bool,
crdt::Lww<String>,
crdt::Lww<crdt::MergingOption<ExpirationTime>>,
crdt::Lww<AdminApiTokenScope>,
);
fn make(input: Input) -> AdminApiToken {
let (deleted, name, expiration, scope) = input;
let state = if deleted {
crdt::Deletable::Deleted
} else {
crdt::Deletable::present(AdminApiTokenParams {
created: 0,
token_hash: String::new(),
name,
expiration,
scope,
})
};
AdminApiToken {
prefix: String::new(),
state,
}
}
fuzz_target!(|inputs: (Input, Input, Input)| {
let (a, b, c) = inputs;
check_crdt_laws(make(a), make(b), make(c));
});
-20
View File
@@ -1,20 +0,0 @@
#![no_main]
use garage_fuzz::check_crdt_laws;
use garage_model::s3::block_ref_table::BlockRef;
use libfuzzer_sys::fuzz_target;
/// Build a BlockRef with a fixed block hash and version UUID so that CRDT state
/// can be compared across merge results. Only the deleted flag varies.
fn make_block_ref(deleted: bool) -> BlockRef {
BlockRef {
block: [0u8; 32].into(),
version: [0u8; 32].into(),
deleted: deleted.into(),
}
}
fuzz_target!(|inputs: (bool, bool, bool)| {
let (d1, d2, d3) = inputs;
check_crdt_laws(make_block_ref(d1), make_block_ref(d2), make_block_ref(d3));
});
-25
View File
@@ -1,25 +0,0 @@
#![no_main]
use garage_fuzz::check_crdt_laws;
use garage_model::bucket_alias_table::BucketAlias;
use garage_util::data::Uuid;
use libfuzzer_sys::fuzz_target;
/// Build a BucketAlias with a fixed name so that CRDT state can be compared
/// across merge results. The timestamp and optional bucket ID are the CRDT state.
fn make_bucket_alias(ts: u64, bucket_id: Option<[u8; 32]>) -> BucketAlias {
BucketAlias::new(String::new(), ts, bucket_id.map(Uuid::from))
}
fuzz_target!(|inputs: (
(u64, Option<[u8; 32]>),
(u64, Option<[u8; 32]>),
(u64, Option<[u8; 32]>)
)| {
let ((ts1, b1), (ts2, b2), (ts3, b3)) = inputs;
check_crdt_laws(
make_bucket_alias(ts1, b1),
make_bucket_alias(ts2, b2),
make_bucket_alias(ts3, b3),
);
});
-22
View File
@@ -1,22 +0,0 @@
#![no_main]
use garage_fuzz::check_crdt_laws;
use garage_model::bucket_table::{Bucket, BucketParams};
use garage_util::crdt::{self, Deletable};
use libfuzzer_sys::fuzz_target;
fn make(state: Deletable<BucketParams>) -> Bucket {
Bucket {
id: [0u8; 32].into(),
state,
}
}
fuzz_target!(|inputs: (
crdt::Deletable<BucketParams>,
crdt::Deletable<BucketParams>,
crdt::Deletable<BucketParams>
)| {
let (a, b, c) = inputs;
check_crdt_laws(make(a), make(b), make(c));
});
-36
View File
@@ -1,36 +0,0 @@
#![no_main]
use std::collections::BTreeMap;
use garage_fuzz::check_crdt_laws;
use garage_model::k2v::item_table::{DvvsEntry, DvvsValue, K2VItem};
use libfuzzer_sys::fuzz_target;
// Timestamps are encoded as `(ts << 32) | shift` so that items built with different
// shifts (0, 1, 2) have disjoint timestamp spaces that still interleave in the sorted merge.
fn make(raw: BTreeMap<u64, (u32, BTreeMap<u32, DvvsValue>)>, shift: u32) -> K2VItem {
let shift = shift as u64;
let items = raw
.into_iter()
.map(|(node, (t_discard, values))| {
let entry = DvvsEntry::from_raw(
(t_discard as u64) << 32 | shift,
values
.into_iter()
.map(|(ts, v)| ((ts as u64) << 32 | shift, v))
.collect(),
);
(node, entry)
})
.collect();
K2VItem::with_raw_items(items)
}
fuzz_target!(|inputs: (
BTreeMap<u64, (u32, BTreeMap<u32, DvvsValue>)>,
BTreeMap<u64, (u32, BTreeMap<u32, DvvsValue>)>,
BTreeMap<u64, (u32, BTreeMap<u32, DvvsValue>)>,
)| {
let (a, b, c) = inputs;
check_crdt_laws(make(a, 0), make(b, 1), make(c, 2));
});
-43
View File
@@ -1,43 +0,0 @@
#![no_main]
use garage_fuzz::check_crdt_laws;
use garage_model::key_table::{Key, KeyParams};
use garage_model::permission::{BucketKeyPerm, ExpirationTime};
use garage_util::crdt;
use garage_util::data::Uuid;
use libfuzzer_sys::fuzz_target;
type Input = (
bool,
crdt::Lww<String>,
crdt::Lww<crdt::MergingOption<ExpirationTime>>,
crdt::Lww<bool>,
crdt::Map<Uuid, BucketKeyPerm>,
crdt::LwwMap<String, crdt::CancelingOption<Uuid>>,
);
fn make(input: Input) -> Key {
let (deleted, name, expiration, allow_create_bucket, authorized_buckets, local_aliases) = input;
let state = if deleted {
crdt::Deletable::Deleted
} else {
crdt::Deletable::present(KeyParams {
created: None,
secret_key: String::new(),
name,
expiration,
allow_create_bucket,
authorized_buckets,
local_aliases,
})
};
Key {
key_id: String::new(),
state,
}
}
fuzz_target!(|inputs: (Input, Input, Input)| {
let (a, b, c) = inputs;
check_crdt_laws(make(a), make(b), make(c));
});
-37
View File
@@ -1,37 +0,0 @@
#![no_main]
use garage_fuzz::check_crdt_laws;
use garage_model::s3::mpu_table::{MpuPart, MpuPartKey, MultipartUpload};
use libfuzzer_sys::fuzz_target;
/// Build a MultipartUpload from an arbitrary deleted flag and parts list, using a fixed
/// upload_id/bucket_id/key so that CRDT state can be compared across merge results.
/// `MpuPart.version` is fixed to a constant since it is identity data, not CRDT state:
/// two replicas of the same part (same MpuPartKey) always share the same version UUID.
/// If deleted, parts are cleared to ensure a valid initial CRDT state.
fn make_mpu(deleted: bool, parts: Vec<(MpuPartKey, MpuPart)>) -> MultipartUpload {
let mut mpu = MultipartUpload::new(
[0u8; 32].into(),
0,
[0u8; 32].into(),
String::new(),
deleted,
);
for (key, mut part) in parts {
part.version = [0u8; 32].into();
mpu.parts.put(key, part);
}
if mpu.deleted.get() {
mpu.parts.clear();
}
mpu
}
fuzz_target!(|inputs: (
(bool, Vec<(MpuPartKey, MpuPart)>),
(bool, Vec<(MpuPartKey, MpuPart)>),
(bool, Vec<(MpuPartKey, MpuPart)>)
)| {
let ((d1, p1), (d2, p2), (d3, p3)) = inputs;
check_crdt_laws(make_mpu(d1, p1), make_mpu(d2, p2), make_mpu(d3, p3));
});
-42
View File
@@ -1,42 +0,0 @@
#![no_main]
use garage_fuzz::check_crdt_laws;
use garage_model::s3::version_table::{Version, VersionBacklink, VersionBlock, VersionBlockKey};
use libfuzzer_sys::fuzz_target;
/// Build a Version from an arbitrary deleted flag and block list, using a fixed uuid/backlink
/// so that CRDT state can be compared across merge results.
/// Duplicate block keys are dropped before construction.
/// If deleted, blocks are cleared to ensure a valid initial CRDT state.
fn make_version(deleted: bool, mut blocks: Vec<(VersionBlockKey, VersionBlock)>) -> Version {
blocks.sort_by_key(|(k, _)| *k);
blocks.dedup_by_key(|(k, _)| *k);
let mut v = Version::new(
[0u8; 32].into(),
VersionBacklink::Object {
bucket_id: [0u8; 32].into(),
key: String::new(),
},
deleted,
);
for (key, block) in blocks {
v.blocks.put(key, block);
}
if v.deleted.get() {
v.blocks.clear();
}
v
}
fuzz_target!(|inputs: (
(bool, Vec<(VersionBlockKey, VersionBlock)>),
(bool, Vec<(VersionBlockKey, VersionBlock)>),
(bool, Vec<(VersionBlockKey, VersionBlock)>)
)| {
let ((d1, b1), (d2, b2), (d3, b3)) = inputs;
check_crdt_laws(
make_version(d1, b1),
make_version(d2, b2),
make_version(d3, b3),
);
});
-2
View File
@@ -1,2 +0,0 @@
[toolchain]
channel = "nightly"
-56
View File
@@ -1,56 +0,0 @@
use garage_table::crdt::Crdt;
use std::fmt::Debug;
pub fn check_crdt_laws<T>(a: T, b: T, c: T)
where
T: Crdt + PartialEq + Clone + Debug,
{
// Idempotency: merge(a, a) == a
{
let mut a2 = a.clone();
a2.merge(&a);
assert_eq!(a2, a, "merge is not idempotent: {a2:#?} != {a:#?}");
}
// Commutativity: merge(a, b) == merge(b, a)
let ab = {
let mut t = a.clone();
t.merge(&b);
t
};
let ba = {
let mut t = b.clone();
t.merge(&a);
t
};
assert_eq!(ab, ba, "merge is not commutative: {ab:#?} != {ba:#?}");
// LX's corrolary: merge(merge(a,b),b) = merge(a,b)
let ab_b = {
let mut t = ab.clone();
t.merge(&b);
t
};
assert_eq!(ab, ab_b);
// Associativity: merge(merge(a, b), c) == merge(a, merge(b, c))
let ab_c = {
let mut t = ab;
t.merge(&c);
t
};
let bc = {
let mut t = b;
t.merge(&c);
t
};
let a_bc = {
let mut t = a;
t.merge(&bc);
t
};
assert_eq!(
ab_c, a_bc,
"merge is not associative: {ab_c:#?} != {a_bc:#?}"
);
}
+1 -9
View File
@@ -48,7 +48,7 @@ let
inherit (pkgs) lib stdenv;
toolchainFn = (p: p.rust-bin.stable."1.95.0".default.override {
toolchainFn = (p: p.rust-bin.stable."1.91.0".default.override {
targets = lib.optionals (target != null) [ rustTarget ];
extensions = [
"rust-src"
@@ -148,14 +148,6 @@ let
in rec {
toolchain = toolchainFn pkgs;
toolchainNightly = pkgs.rust-bin.selectLatestNightlyWith (toolchain: toolchain.default.override {
targets = lib.optionals (target != null) [ rustTarget ];
extensions = [
"rust-src"
"rustfmt"
];
});
devShell = pkgs.mkShell {
buildInputs = [
toolchain
+1 -1
View File
@@ -18,7 +18,7 @@ fi
$GARAGE_BIN -c /tmp/config.1.toml bucket create eprouvette
if [ "$GARAGE_OLDVER" = "v08" ]; then
KEY_INFO=$($GARAGE_BIN -c /tmp/config.1.toml key create opérateur)
KEY_INFO=$($GARAGE_BIN -c /tmp/config.1.toml key new --name opérateur)
ACCESS_KEY=`echo $KEY_INFO|grep -Po 'GK[a-f0-9]+'`
SECRET_KEY=`echo $KEY_INFO|grep -Po 'Secret key: [a-f0-9]+'|grep -Po '[a-f0-9]+$'`
elif [ "$GARAGE_OLDVER" = "v1" ]; then
+2 -2
View File
@@ -2,8 +2,8 @@ apiVersion: v2
name: garage
description: S3-compatible object store for small self-hosted geo-distributed deployments
type: application
version: 0.9.3
appVersion: "v2.3.0"
version: 0.9.2
appVersion: "v2.2.0"
home: https://garagehq.deuxfleurs.fr/
icon: https://garagehq.deuxfleurs.fr/images/garage-logo.svg
+3 -6
View File
@@ -1,6 +1,6 @@
# garage
![Version: 0.9.3](https://img.shields.io/badge/Version-0.9.3-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: v2.3.0](https://img.shields.io/badge/AppVersion-v2.3.0-informational?style=flat-square)
![Version: 0.9.2](https://img.shields.io/badge/Version-0.9.2-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: v2.2.0](https://img.shields.io/badge/AppVersion-v2.2.0-informational?style=flat-square)
S3-compatible object store for small self-hosted geo-distributed deployments
@@ -29,18 +29,15 @@ S3-compatible object store for small self-hosted geo-distributed deployments
| garage.dbEngine | string | `"lmdb"` | Can be changed for better performance on certain systems https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/#db_engine |
| garage.existingConfigMap | string | `""` | if not empty string, allow using an existing ConfigMap for the garage.toml, if set, ignores garage.toml |
| garage.garageTomlString | string | `""` | String Template for the garage configuration if set, ignores above values. Values can be templated, see https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/ |
| garage.kubernetesSkipCrd | bool | `false` | Set to true if you want to use k8s discovery but install the CRDs manually outside of the helm chart, for example if you operate at namespace level without cluster resources |
| garage.kubernetesSkipCrd | bool | `false` | Set to true if you want to use k8s discovery but install the CRDs manually outside of the helm chart, for example if you operate at namespace level without cluster ressources |
| garage.replicationFactor | string | `"3"` | Default to 3 replicas, see the replication_factor section at https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/#replication_factor |
| garage.consistencyMode | string | `"consistent"` | Default to read-after-write consistency, see the consistency_mode section at https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/#consistency_mode |
| garage.metadataAutoSnapshotInterval | string | `""` | If this value is set, Garage will automatically take a snapshot of the metadata DB file at a regular interval and save it in the metadata directory. https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/#metadata_auto_snapshot_interval |
| garage.admin.apiBindAddr | string | `"[::]:3903"` | |
| garage.rpcBindAddr | string | `"[::]:3901"` | |
| garage.rpcSecret | string | `""` | If not given, a random secret will be generated and stored in a Secret object |
| garage.s3.api.bindAddr | string | `"[::]:3900"` | |
| garage.s3.api.region | string | `"garage"` | |
| garage.s3.api.rootDomain | string | `".s3.garage.tld"` | |
| garage.s3.web.index | string | `"index.html"` | |
| garage.s3.web.bindAddr | string | `"[::]:3902"` | |
| garage.s3.web.rootDomain | string | `".web.garage.tld"` | |
| image.pullPolicy | string | `"IfNotPresent"` | |
| image.repository | string | `"dxflrs/amd64_garage"` | default to amd64 docker image |
@@ -79,7 +76,7 @@ S3-compatible object store for small self-hosted geo-distributed deployments
| persistence.enabled | bool | `true` | |
| persistence.meta.hostPath | string | `"/var/lib/garage/meta"` | |
| persistence.meta.size | string | `"100Mi"` | |
| podAnnotations | object | `{}` | additional pod annotations |
| podAnnotations | object | `{}` | additonal pod annotations |
| podSecurityContext.fsGroup | int | `1000` | |
| podSecurityContext.runAsGroup | int | `1000` | |
| podSecurityContext.runAsNonRoot | bool | `true` | |
@@ -71,13 +71,6 @@ Create the name of the service account to use
{{- end }}
{{- end }}
{{/*
Extract the trailing port number from a bind address like [::]:3900 or 0.0.0.0:3900.
*/}}
{{- define "garage.portFromBindAddr" -}}
{{- regexFind "[0-9]+$" . -}}
{{- end }}
{{/*
Returns given number of random Hex characters.
In practice, it generates up to 100 randAlphaNum strings
@@ -5,11 +5,9 @@ metadata:
labels:
{{- include "garage.labels" . | nindent 4 }}
rules:
{{- if eq .Values.garage.kubernetesSkipCrd false }}
- apiGroups: ["apiextensions.k8s.io"]
resources: ["customresourcedefinitions"]
verbs: ["get", "list", "watch", "create", "patch"]
{{ end }}
- apiGroups: ["deuxfleurs.fr"]
resources: ["garagenodes"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
@@ -27,4 +25,4 @@ subjects:
roleRef:
kind: ClusterRole
name: manage-crds-{{ .Release.Namespace }}-{{ .Release.Name }}
apiGroup: rbac.authorization.k8s.io
apiGroup: rbac.authorization.k8s.io
+4 -4
View File
@@ -13,7 +13,7 @@ data:
db_engine = "{{ .Values.garage.dbEngine }}"
block_size = "{{ .Values.garage.blockSize }}"
block_size = {{ .Values.garage.blockSize }}
replication_factor = {{ .Values.garage.replicationFactor }}
consistency_mode = "{{ .Values.garage.consistencyMode }}"
@@ -45,16 +45,16 @@ data:
[s3_api]
s3_region = "{{ .Values.garage.s3.api.region }}"
api_bind_addr = "{{ .Values.garage.s3.api.bindAddr }}"
api_bind_addr = "[::]:3900"
root_domain = "{{ .Values.garage.s3.api.rootDomain }}"
[s3_web]
bind_addr = "{{ .Values.garage.s3.web.bindAddr }}"
bind_addr = "[::]:3902"
root_domain = "{{ .Values.garage.s3.web.rootDomain }}"
index = "{{ .Values.garage.s3.web.index }}"
[admin]
api_bind_addr = "{{ .Values.garage.admin.apiBindAddr }}"
api_bind_addr = "[::]:3903"
{{- if .Values.monitoring.tracing.sink }}
trace_sink = "{{ .Values.monitoring.tracing.sink }}"
{{- end }}
@@ -10,11 +10,11 @@ spec:
clusterIP: None
ports:
- port: {{ .Values.service.s3.api.port }}
targetPort: {{ include "garage.portFromBindAddr" .Values.garage.s3.api.bindAddr | int }}
targetPort: 3900
protocol: TCP
name: s3-api
- port: {{ .Values.service.s3.web.port }}
targetPort: {{ include "garage.portFromBindAddr" .Values.garage.s3.web.bindAddr | int }}
targetPort: 3902
protocol: TCP
name: s3-web
selector:
+4 -4
View File
@@ -12,11 +12,11 @@ spec:
type: {{ .Values.service.type }}
ports:
- port: {{ .Values.service.s3.api.port }}
targetPort: {{ include "garage.portFromBindAddr" .Values.garage.s3.api.bindAddr | int }}
targetPort: 3900
protocol: TCP
name: s3-api
- port: {{ .Values.service.s3.web.port }}
targetPort: {{ include "garage.portFromBindAddr" .Values.garage.s3.web.bindAddr | int }}
targetPort: 3902
protocol: TCP
name: s3-web
selector:
@@ -35,8 +35,8 @@ spec:
type: ClusterIP
clusterIP: None
ports:
- port: {{ include "garage.portFromBindAddr" .Values.garage.admin.apiBindAddr | int }}
targetPort: {{ include "garage.portFromBindAddr" .Values.garage.admin.apiBindAddr | int }}
- port: 3903
targetPort: 3903
protocol: TCP
name: metrics
selector:
+4 -7
View File
@@ -28,9 +28,6 @@ spec:
{{- toYaml . | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "garage.serviceAccountName" . }}
{{- with .Values.priorityClassName }}
priorityClassName: {{ . }}
{{- end }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
initContainers:
@@ -60,11 +57,11 @@ spec:
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- containerPort: {{ include "garage.portFromBindAddr" .Values.garage.s3.api.bindAddr | int }}
- containerPort: 3900
name: s3-api
- containerPort: {{ include "garage.portFromBindAddr" .Values.garage.s3.web.bindAddr | int }}
- containerPort: 3902
name: web-api
- containerPort: {{ include "garage.portFromBindAddr" .Values.garage.admin.apiBindAddr | int }}
- containerPort: 3903
name: admin
{{- with .Values.environment }}
env:
@@ -94,7 +91,7 @@ spec:
volumes:
- name: configmap
configMap:
name: {{ if .Values.garage.existingConfigMap }}{{ .Values.garage.existingConfigMap }}{{ else }}{{ include "garage.fullname" . }}-config{{ end }}
name: {{ include "garage.fullname" . }}-config
- name: etc
emptyDir: {}
{{- if .Values.persistence.enabled }}
+5 -13
View File
@@ -44,19 +44,15 @@ garage:
# -- This is not required if you use the integrated kubernetes discovery
bootstrapPeers: []
# -- Set to true if you want to use k8s discovery but install the CRDs manually outside
# of the helm chart, for example if you operate at namespace level without cluster resources
# of the helm chart, for example if you operate at namespace level without cluster ressources
kubernetesSkipCrd: false
s3:
api:
bindAddr: "[::]:3900"
region: "garage"
rootDomain: ".s3.garage.tld"
web:
bindAddr: "[::]:3902"
rootDomain: ".web.garage.tld"
index: "index.html"
admin:
apiBindAddr: "[::]:3903"
# -- Additional configuration to append to garage.toml. Use a multi-line string for custom config.
# Example:
@@ -124,7 +120,7 @@ serviceAccount:
# If not set and create is true, a name is generated using the fullname template
name: ""
# -- additional pod annotations
# -- additonal pod annotations
podAnnotations: {}
podSecurityContext:
@@ -213,7 +209,7 @@ ingress:
# - kubernetes.docker.internal
resources: {}
# The following are indicative for a small-size deployment, for anything serious double them.
# The following are indicative for a small-size deployement, for anything serious double them.
# limits:
# cpu: 100m
# memory: 1024Mi
@@ -225,14 +221,14 @@ resources: {}
livenessProbe: {}
#httpGet:
# path: /health
# port: 3903 # or the port from garage.admin.apiBindAddr
# port: 3903
#initialDelaySeconds: 5
#periodSeconds: 30
# -- Specifies a readinessProbe
readinessProbe: {}
#httpGet:
# path: /health
# port: 3903 # or the port from garage.admin.apiBindAddr
# port: 3903
#initialDelaySeconds: 5
#periodSeconds: 30
@@ -242,10 +238,6 @@ tolerations: []
affinity: {}
# -- Optional priority class name to assign to the pods.
# See https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/
priorityClassName: ""
environment: {}
extraVolumes: {}
+3 -3
View File
@@ -127,7 +127,7 @@ They are due to the download being interrupted in the middle (^C during first la
Add `:force?` to the `cached-wget!` call in `daemon.clj` to re-download the binary,
or restar the VMs to clear temporary files.
### In `jepsen.garage`: prefix weirdness
### In `jepsen.garage`: prefix wierdness
In `store/garage set1/20231019T163358.615+0200`:
@@ -146,12 +146,12 @@ and passing all values that were previously in the context (creds and prefix) as
The reg2 test is our custom checker for CRDT read-after-write on individual object keys, acting as registers which can be updated.
The test fails without the timestamp fix, which is expected as the clock scrambler will prevent nodes from having a correct ordering of objects.
With the timestamp fix (`--patch tsfix1`), the happened-before relationship should at least be respected, meaning that when a PutObject call starts
With the timestamp fix (`--patch tsfix1`), the happenned-before relationship should at least be respected, meaning that when a PutObject call starts
after another PutObject call has ended, the second call should overwrite the value of the first call, and that value should not be
readable by future GetObject calls.
However, we observed inconsistencies even with the timestamp fix.
The inconsistencies seemed to always happened after writing a nil value, which translates to a DeleteObject call
The inconsistencies seemed to always happenned after writing a nil value, which translates to a DeleteObject call
instead of a PutObject. By removing the possibility of writing nil values, therefore only doing
PutObject calls, the issue disappears. There is therefore an issue to fix in DeleteObject.
+2 -2
View File
@@ -2,7 +2,7 @@
: '
This script tests whether uploaded parts can be skipped in a
CompleteMultipartUpload
CompleteMultipartUpoad
On Minio: yes, parts can be skipped
@@ -52,7 +52,7 @@
Conclusions:
- Skipping a part in a CompleteMultipartUpload call is OK
- Skipping a part in a CompleteMultipartUpoad call is OK
- The part is simply not included in the stored object
- Sequential part renumbering counts only non-skipped parts
'
+1 -2
View File
@@ -34,7 +34,6 @@ in
openssl
curl
jq
typos
];
shellHook = ''
export AWS_REQUEST_CHECKSUM_CALCULATION='when_required'
@@ -52,7 +51,7 @@ in
function to_docker {
executor \
--force \
--custom-platform="$(echo "''${DOCKER_PLATFORM}" | sed 's/i386/386/')" \
--customPlatform="$(echo "''${DOCKER_PLATFORM}" | sed 's/i386/386/')" \
--destination "$(echo "''${CONTAINER_NAME}" | sed 's/i386/386/'):''${CONTAINER_TAG}" \
--context dir://`pwd` \
--verbosity=debug
+3 -6
View File
@@ -1,6 +1,6 @@
[package]
name = "garage_api_admin"
version = "2.3.0"
version = "2.2.0"
authors = ["Alex Auvolat <alex@adnab.me>"]
edition = "2018"
license = "AGPL-3.0"
@@ -46,8 +46,5 @@ opentelemetry-prometheus = { workspace = true, optional = true }
prometheus = { workspace = true, optional = true }
[features]
metrics = ["opentelemetry-prometheus", "prometheus"]
k2v = ["garage_model/k2v"]
[lints]
workspace = true
metrics = [ "opentelemetry-prometheus", "prometheus" ]
k2v = [ "garage_model/k2v" ]
+7 -8
View File
@@ -7,7 +7,6 @@ use garage_util::time::now_msec;
use garage_model::admin_token_table::*;
use garage_model::garage::Garage;
use garage_model::permission::ExpirationTime;
use crate::api::*;
use crate::error::*;
@@ -144,7 +143,7 @@ impl RequestHandler for UpdateAdminTokenRequest {
garage: &Arc<Garage>,
_admin: &Admin,
) -> Result<UpdateAdminTokenResponse, Error> {
let mut token = get_existing_admin_token(garage, &self.id).await?;
let mut token = get_existing_admin_token(&garage, &self.id).await?;
apply_token_updates(&mut token, self.body)?;
@@ -165,7 +164,7 @@ impl RequestHandler for DeleteAdminTokenRequest {
garage: &Arc<Garage>,
_admin: &Admin,
) -> Result<DeleteAdminTokenResponse, Error> {
let token = get_existing_admin_token(garage, &self.id).await?;
let token = get_existing_admin_token(&garage, &self.id).await?;
garage
.admin_token_table
@@ -225,7 +224,7 @@ impl RequestHandler for GetCurrentAdminTokenInfoRequest {
}
let (prefix, _) = self.admin_token.split_once('.').unwrap();
let token = get_existing_admin_token(garage, &prefix.to_string()).await?;
let token = get_existing_admin_token(&garage, &prefix.to_string()).await?;
Ok(GetCurrentAdminTokenInfoResponse(admin_token_info_results(
&token, now,
@@ -245,8 +244,8 @@ fn admin_token_info_results(token: &AdminApiToken, now: u64) -> GetAdminTokenInf
.expect("invalid timestamp stored in db"),
),
name: params.name.get().to_string(),
expiration: params.expiration.get().inner().map(|x| {
DateTime::from_timestamp_millis(x.0 as i64).expect("invalid timestamp stored in db")
expiration: params.expiration.get().map(|x| {
DateTime::from_timestamp_millis(x as i64).expect("invalid timestamp stored in db")
}),
expired: params.is_expired(now),
scope: params.scope.get().0.clone(),
@@ -280,10 +279,10 @@ fn apply_token_updates(
if let Some(expiration) = updates.expiration {
params
.expiration
.update(Some(ExpirationTime(expiration.timestamp_millis() as u64)).into());
.update(Some(expiration.timestamp_millis() as u64));
}
if updates.never_expires {
params.expiration.update(None.into());
params.expiration.update(None);
}
if let Some(scope) = updates.scope {
params.scope.update(AdminApiTokenScope(scope));
+11 -163
View File
@@ -12,7 +12,7 @@ use garage_rpc::*;
use garage_model::garage::Garage;
use garage_api_common::{common_error::CommonError, helpers::is_default, xml};
use garage_api_common::{common_error::CommonError, helpers::is_default};
use crate::api_server::{find_matching_nodes, AdminRpc, AdminRpcResponse};
use crate::error::Error;
@@ -262,7 +262,7 @@ pub struct GetClusterHealthResponse {
pub status: String,
/// the number of nodes this Garage node has had a TCP connection to since the daemon started
pub known_nodes: usize,
/// the number of nodes this Garage node currently has an open connection to
/// the nubmer of nodes this Garage node currently has an open connection to
pub connected_nodes: usize,
/// the number of storage nodes currently registered in the cluster layout
pub storage_nodes: usize,
@@ -282,34 +282,8 @@ pub struct GetClusterHealthResponse {
pub struct GetClusterStatisticsRequest;
#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)]
#[serde(rename_all = "camelCase")]
pub struct GetClusterStatisticsResponse {
// FIXME for v3: remove freeform field and move display logic to garage crate
/// cluster statistics as a free-form string, kept for compatibility with nodes
/// running older v2.x versions of garage
pub freeform: String,
// FIXME for v3: remove Option<> and serde(default) for all fields below
/// available storage space for object data in the entire cluster, in bytes
#[serde(default, skip_serializing_if = "Option::is_none")]
pub data_avail: Option<u64>,
/// available storage space for object metadata in the entire cluster, in bytes
#[serde(default, skip_serializing_if = "Option::is_none")]
pub metadata_avail: Option<u64>,
/// true if the available storage space statistics are imprecise due to missing
/// information of disconnected nodes. When this is the case, the actual
/// space available in the cluster might be lower than the reported values.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub incomplete_avail_info: Option<bool>,
/// number of buckets in the cluster
#[serde(default, skip_serializing_if = "Option::is_none")]
pub bucket_count: Option<u64>,
/// total number of objects stored in all buckets
#[serde(default, skip_serializing_if = "Option::is_none")]
pub total_object_count: Option<u64>,
/// total size of objects stored in all buckets, before compression, deduplication and
/// replication (this is NOT equivalent to actual disk usage in the cluster)
#[serde(default, skip_serializing_if = "Option::is_none")]
pub total_object_bytes: Option<u64>,
}
// ---- ConnectClusterNodes ----
@@ -413,7 +387,7 @@ pub struct UpdateAdminTokenRequestBody {
/// `GetClusterStatus`, etc), or the special value `*` to allow all
/// admin endpoints. **WARNING:** Granting a scope of `CreateAdminToken` or
/// `UpdateAdminToken` trivially allows for privilege escalation, and is thus
/// functionally equivalent to granting a scope of `*`.
/// functionnally equivalent to granting a scope of `*`.
pub scope: Option<Vec<String>>,
}
@@ -618,10 +592,6 @@ pub enum PreviewClusterLayoutChangesResponse {
/// Plain-text information about the layout computation
/// (do not try to parse this)
message: Vec<String>,
/// Structured statistics about the layout computation
// FIXME for v3: remove default and skip_serializing_if
#[serde(default, skip_serializing_if = "Option::is_none")]
statistics: Option<Box<garage_rpc::layout::ComputationStat>>,
/// Details about the new cluster layout
new_layout: GetClusterLayoutResponse,
},
@@ -643,10 +613,6 @@ pub struct ApplyClusterLayoutResponse {
/// Plain-text information about the layout computation
/// (do not try to parse this)
pub message: Vec<String>,
/// Structured statistics about the layout computation
// FIXME for v3: remove default and skip_serializing_if
#[serde(default, skip_serializing_if = "Option::is_none")]
pub statistics: Option<garage_rpc::layout::ComputationStat>,
/// Details about the new cluster layout
pub layout: GetClusterLayoutResponse,
}
@@ -688,26 +654,11 @@ pub struct ClusterLayoutSkipDeadNodesResponse {
// ---- ListKeys ----
#[derive(Debug, Clone, Serialize, Deserialize, Default, IntoParams)]
#[into_params(parameter_in = Query)]
pub struct ListKeysRequest {
/// Returned detailed informations in the same format as GetKeyInfo for each bucket
#[serde(default)]
pub details: bool,
/// Key ID of the first key to return
#[serde(default)]
pub offset: Option<String>,
/// Maximum number of keys to return in a single call
#[serde(default)]
pub limit: Option<usize>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ListKeysRequest;
#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)]
#[serde(untagged)]
pub enum ListKeysResponse {
WithoutDetails(Vec<ListKeysResponseItem>),
WithDetails(Vec<GetKeyInfoResponse>),
}
pub struct ListKeysResponse(pub Vec<ListKeysResponseItem>);
#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)]
#[serde(rename_all = "camelCase")]
@@ -845,26 +796,11 @@ pub struct DeleteKeyResponse;
// ---- ListBuckets ----
#[derive(Debug, Clone, Serialize, Deserialize, Default, IntoParams)]
#[into_params(parameter_in = Query)]
pub struct ListBucketsRequest {
/// Returned detailed informations in the same format as GetBucketInfo for each bucket
#[serde(default)]
pub details: bool,
/// Bucket ID of the first bucket to return
#[serde(default)]
pub offset: Option<String>,
/// Maximum number of buckets to return in a single call
#[serde(default)]
pub limit: Option<usize>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ListBucketsRequest;
#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)]
#[serde(untagged)]
pub enum ListBucketsResponse {
WithoutDetails(Vec<ListBucketsResponseItem>),
WithDetails(Vec<GetBucketInfoResponse>),
}
pub struct ListBucketsResponse(pub Vec<ListBucketsResponseItem>);
#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)]
#[serde(rename_all = "camelCase")]
@@ -905,18 +841,11 @@ pub struct GetBucketInfoResponse {
pub created: DateTime<Utc>,
/// List of global aliases for this bucket
pub global_aliases: Vec<String>,
/// Whether website access is enabled for this bucket
/// Whether website acces is enabled for this bucket
pub website_access: bool,
#[serde(default)]
/// Website configuration for this bucket
#[serde(default, skip_serializing_if = "Option::is_none")]
pub website_config: Option<GetBucketInfoWebsiteResponse>,
// FIXME for v3: remove serde(default) for the two fields below
/// CORS rules for this bucket
#[serde(default, skip_serializing_if = "Option::is_none")]
pub cors_rules: Option<Vec<xml::cors::CorsRule>>,
/// Object lifecycle rules for this bucket
#[serde(default, skip_serializing_if = "Option::is_none")]
pub lifecycle_rules: Option<Vec<xml::lifecycle::LifecycleRule>>,
/// List of access keys that have permissions granted on this bucket
pub keys: Vec<GetBucketInfoKey>,
/// Number of objects in this bucket
@@ -940,9 +869,6 @@ pub struct GetBucketInfoResponse {
pub struct GetBucketInfoWebsiteResponse {
pub index_document: String,
pub error_document: Option<String>,
// FIXME for v3: remove serde(default) for field below
#[serde(default, skip_serializing_if = "Option::is_none")]
pub routing_rules: Option<Vec<xml::website::RoutingRule>>,
}
#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)]
@@ -1001,11 +927,6 @@ pub struct UpdateBucketResponse(pub GetBucketInfoResponse);
pub struct UpdateBucketRequestBody {
pub website_access: Option<UpdateBucketWebsiteAccess>,
pub quotas: Option<ApiBucketQuotas>,
// FIXME for v3: remove serde(default) for the two fields below
#[serde(default)]
pub cors_rules: Option<Vec<xml::cors::CorsRule>>,
#[serde(default)]
pub lifecycle_rules: Option<Vec<xml::lifecycle::LifecycleRule>>,
}
#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)]
@@ -1014,9 +935,6 @@ pub struct UpdateBucketWebsiteAccess {
pub enabled: bool,
pub index_document: Option<String>,
pub error_document: Option<String>,
// FIXME for v3: remove serde(default) for field below
#[serde(default)]
pub routing_rules: Option<Vec<xml::website::RoutingRule>>,
}
// ---- DeleteBucket ----
@@ -1191,41 +1109,10 @@ pub struct LocalGetNodeInfoRequest;
#[serde(rename_all = "camelCase")]
pub struct LocalGetNodeInfoResponse {
pub node_id: String,
// FIXME for v3: remove Option<> and serde(default) for field below
/// hostname of this node
#[serde(default, skip_serializing_if = "Option::is_none")]
pub hostname: Option<String>,
/// garage version running on this node
pub garage_version: String,
/// build-time features enabled for this garage release
pub garage_features: Option<Vec<String>>,
/// rustc version with which this garage release was compiled
pub rust_version: String,
/// database engine used for metadata
pub db_engine: String,
// FIXME for v3: remove Option<> and serde(default) for field below
// FIXME for v3: merge LocalGetNodeInfoResponse and NodeResp
/// Socket address used by other nodes to connect to this node for RPC
#[serde(default, skip_serializing_if = "Option::is_none")]
#[schema(value_type = Option<String>)]
pub addr: Option<SocketAddr>,
/// Whether this node is connected in the cluster
#[serde(default, skip_serializing_if = "Option::is_none")]
pub is_up: Option<bool>,
/// Role assigned to this node in the current cluster layout
#[serde(default, skip_serializing_if = "Option::is_none")]
pub role: Option<NodeAssignedRole>,
/// Whether this node is part of an older layout version and is draining data.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub draining: Option<bool>,
/// Total and available space on the disk partition(s) containing the data
/// directory(ies)
#[serde(default, skip_serializing_if = "Option::is_none")]
pub data_partition: Option<FreeSpaceResp>,
/// Total and available space on the disk partition containing the
/// metadata directory
#[serde(default, skip_serializing_if = "Option::is_none")]
pub metadata_partition: Option<FreeSpaceResp>,
}
// ---- GetNodeStatistics ----
@@ -1234,47 +1121,8 @@ pub struct LocalGetNodeInfoResponse {
pub struct LocalGetNodeStatisticsRequest;
#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)]
#[serde(rename_all = "camelCase")]
pub struct LocalGetNodeStatisticsResponse {
// FIXME for v3: remove freeform field and move display logic to garage crate
/// node statistics as a free-form string, kept for compatibility with nodes
/// running older v2.x versions of garage
pub freeform: String,
// FIXME for v3: remove serde(default) for fields below
/// metadata table statistics
#[serde(default, skip_serializing_if = "Option::is_none")]
pub table_stats: Option<Vec<NodeTableStats>>,
/// block manager statistics
#[serde(default, skip_serializing_if = "Option::is_none")]
pub block_manager_stats: Option<NodeBlockManagerStats>,
}
#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)]
#[serde(rename_all = "camelCase")]
pub struct NodeTableStats {
/// name of metadata table
pub table_name: String,
/// number of items stored in metadata table
pub items: u64,
/// size of the merkle tree representing all items in the table
pub merkle_items: u64,
/// number of items in the merkle tree update queue
pub merkle_queue_len: u64,
/// number of items in the remote insert queue
pub insert_queue_len: u64,
/// number of items in the garbage collection queue
pub gc_queue_len: u64,
}
#[derive(Debug, Clone, Serialize, Deserialize, ToSchema, Default)]
#[serde(rename_all = "camelCase")]
pub struct NodeBlockManagerStats {
/// number of reference counter entries
pub rc_entries: u64,
/// number of blocks in the resync queue
pub resync_queue_len: u64,
/// number of blocks with resync errors
pub resync_errors: u64,
}
// ---- CreateMetadataSnapshot ----
+7 -7
View File
@@ -64,7 +64,7 @@ impl EndpointHandler<AdminRpc> for AdminApiServer {
match message {
AdminRpc::Proxy(req) => {
info!("Proxied admin API request: {}", req.name());
let res = req.clone().handle(&self.garage, self).await;
let res = req.clone().handle(&self.garage, &self).await;
match res {
Ok(res) => Ok(AdminRpcResponse::ProxyApiOkResponse(res.tagged())),
Err(e) => Ok(AdminRpcResponse::ApiErrorResponse {
@@ -76,7 +76,7 @@ impl EndpointHandler<AdminRpc> for AdminApiServer {
}
AdminRpc::Internal(req) => {
info!("Internal admin API request: {}", req.name());
let res = req.clone().handle(&self.garage, self).await;
let res = req.clone().handle(&self.garage, &self).await;
match res {
Ok(res) => Ok(AdminRpcResponse::InternalApiOkResponse(res)),
Err(e) => Ok(AdminRpcResponse::ApiErrorResponse {
@@ -173,12 +173,12 @@ impl AdminApiServer {
}
match request {
AdminApiRequest::Options(req) => req.handle(&self.garage, self).await,
AdminApiRequest::CheckDomain(req) => req.handle(&self.garage, self).await,
AdminApiRequest::Health(req) => req.handle(&self.garage, self).await,
AdminApiRequest::Metrics(req) => req.handle(&self.garage, self).await,
AdminApiRequest::Options(req) => req.handle(&self.garage, &self).await,
AdminApiRequest::CheckDomain(req) => req.handle(&self.garage, &self).await,
AdminApiRequest::Health(req) => req.handle(&self.garage, &self).await,
AdminApiRequest::Metrics(req) => req.handle(&self.garage, &self).await,
req => {
let res = req.handle(&self.garage, self).await?;
let res = req.handle(&self.garage, &self).await?;
let mut res = json_ok_response(&res)?;
res.headers_mut()
.insert(ACCESS_CONTROL_ALLOW_ORIGIN, HeaderValue::from_static("*"));

Some files were not shown because too many files have changed in this diff Show More