mirror of
https://github.com/deuxfleurs-org/garage.git
synced 2026-08-06 04:47:42 +00:00
add final pdf
This commit is contained in:
Alex Auvolat
File diff suppressed because one or more lines are too long
@@ -61,7 +61,7 @@
|
||||
],
|
||||
[
|
||||
*Part of a degrowth initiative*\
|
||||
Garage has been created at Deuxfleurs where we experiment running Internet services without datacenter on commodity and refurbished hardware.
|
||||
Garage has been created at Deuxfleurs, where we experiment running Internet services without datacenter on commodity and refurbished hardware.
|
||||
],
|
||||
[#v(2em)],[],
|
||||
[
|
||||
@@ -76,8 +76,8 @@
|
||||
#image("../assets/logos/AGPLv3_Logo.png", width: 50%)
|
||||
],
|
||||
[
|
||||
*Owned by nobody, open-core is impossible, zero VC money*\
|
||||
AGPL + no Contributor License Agreement = Garage ownership spreads among hundredth of contributors.
|
||||
*Owned by nobody*\
|
||||
AGPL + no Contributor License Agreement = Garage ownership spreads among dozens of contributors.
|
||||
]
|
||||
)
|
||||
|
||||
@@ -145,7 +145,21 @@ MinIO: not suited for geo-distributed deployments, becoming closed source
|
||||
|
||||
#v(2em)
|
||||
|
||||
== Principle 1: based on CRDTs
|
||||
== Principle 1: geo-distributed data model
|
||||
|
||||
#imgcenter("../assets/map.png", width: 90%)
|
||||
|
||||
Garage stores replicas on different zones when possible
|
||||
|
||||
== Zone-aware cluster configuration
|
||||
|
||||
#imgcenter("../assets/screenshots/garage_status_0.9_prod_zonehl.png", width: 100%)
|
||||
|
||||
Trust model: full trust between zones
|
||||
|
||||
#v(5em)
|
||||
|
||||
== Principle 2: based on CRDTs
|
||||
|
||||
#v(1cm)
|
||||
|
||||
@@ -195,16 +209,6 @@ Why not Raft, Paxos, ...? Issues of consensus algorithms:
|
||||
#text(size: 0.8em)[(stronger than eventual consistency)]
|
||||
]
|
||||
|
||||
== Principle 2: geo-distributed data model
|
||||
|
||||
#imgcenter("../assets/map.png", width: 90%)
|
||||
|
||||
Garage stores replicas on different zones when possible
|
||||
|
||||
== Zone-aware cluster configuration
|
||||
|
||||
#imgcenter("../assets/screenshots/garage_status_0.9_prod_zonehl.png", width: 100%)
|
||||
|
||||
|
||||
== Performance evaluation
|
||||
|
||||
@@ -217,8 +221,6 @@ Garage stores replicas on different zones when possible
|
||||
#imgcenter("../assets/perf/io-0.7-0.8-minio.png", width: 100%)
|
||||
|
||||
|
||||
= Deploying Garage
|
||||
|
||||
== Garage in the wild
|
||||
|
||||
#imgcenter("../assets/cluster_kind.png", width: 100%)
|
||||
@@ -231,24 +233,7 @@ _"Petabyte storage setup for a video site. Nginx as CDN in-front using garage-s3
|
||||
|
||||
_"We currently manage 7 Garage nodes, 28TB total storage, 6M blocks for 3M objects and 4TB of object data. We have been running Garage in production for 2.5 years."_
|
||||
|
||||
== Setting up data and metadata storage
|
||||
|
||||
#vhcenter[
|
||||
#mytable(
|
||||
columns: (0.7fr, 1fr, 1fr),
|
||||
inset: 0.8em,
|
||||
align: center + horizon,
|
||||
table.header[][*Metadata storage*][*Data storage*],
|
||||
[*Content*],[access keys, buckets\ index of objects],[raw data blocks],
|
||||
[*Size*],[\< 10\% of data\ rarely over 100GB],[replication × dataset size\ *no erasure-coding*],
|
||||
[*Constraints*],[latency sensitive\ write-intensive under load],[big\ many files],
|
||||
[*Ideal hardware*],[entreprise-grade SSD],[HDD],
|
||||
[*Recommended redundancy*],[RAID1],[none, use disks directly\ *avoid RAID if possible*],
|
||||
[*Recommended filesystem*],[ZFS, Btrfs],[XFS on invidual disks],
|
||||
[*Tunables in Garage*],[database engine\
|
||||
automatic snapshots],[block size\ compression],
|
||||
)
|
||||
]
|
||||
= Deploying Garage
|
||||
|
||||
== Chosing a replication factor
|
||||
|
||||
@@ -270,6 +255,25 @@ _"We currently manage 7 Garage nodes, 28TB total storage, 6M blocks for 3M objec
|
||||
Choose well, this cannot be changed easily!
|
||||
]
|
||||
|
||||
== Setting up data and metadata storage
|
||||
|
||||
#vhcenter[
|
||||
#mytable(
|
||||
columns: (0.7fr, 1fr, 1fr),
|
||||
inset: 0.8em,
|
||||
align: center + horizon,
|
||||
table.header[][*Metadata storage*][*Data storage*],
|
||||
[*Content*],[access keys, buckets\ index of objects],[raw data blocks],
|
||||
[*Size*],[\< 10\% of data\ rarely over 100GB],[replication × dataset size\ *no erasure-coding*],
|
||||
[*Constraints*],[latency sensitive\ write-intensive under load],[big\ many files],
|
||||
[*Ideal hardware*],[entreprise-grade SSD],[HDD],
|
||||
[*Recommended redundancy*],[RAID1],[none, use disks directly\ *avoid RAID if possible*],
|
||||
[*Recommended filesystem*],[ZFS, Btrfs],[XFS on invidual disks],
|
||||
[*Tunables in Garage*],[database engine\
|
||||
automatic snapshots],[block size\ compression],
|
||||
)
|
||||
]
|
||||
|
||||
== Picking a metadata engine
|
||||
|
||||
|
||||
|
||||
@@ -1,403 +0,0 @@
|
||||
#import "@preview/slydst:0.1.5": *
|
||||
|
||||
#show: slides
|
||||
|
||||
#title-slide[
|
||||
#image("../../sticker/Garage.png", width: 30%)
|
||||
#v(1em)
|
||||
#text(1.2em, weight: "bold")[Garage, an S3 backend as reliable as possible]
|
||||
#v(1em)
|
||||
Garage Authors
|
||||
|
||||
#v(1em)
|
||||
#link("https://garagehq.deuxfleurs.fr/")
|
||||
#v(0.5em)
|
||||
#link("mailto:garagehq@deuxfleurs.fr")
|
||||
#v(0.5em)
|
||||
Matrix channel: `#garage:deuxfleurs.fr`
|
||||
]
|
||||
|
||||
== Meet Garage
|
||||
|
||||
== A non-profit initiative
|
||||
|
||||
#grid(
|
||||
columns: (2fr, 8fr),
|
||||
[
|
||||
#image("../assets/logos/deuxfleurs.svg", width: 50%)
|
||||
],
|
||||
[
|
||||
*Part of a degrowth initiative*\
|
||||
Garage has been created at Deuxfleurs where we experiment running Internet services without datacenter on commodity and refurbished hardware.
|
||||
]
|
||||
)
|
||||
|
||||
#v(2em)
|
||||
|
||||
#grid(
|
||||
columns: (2fr, 8fr),
|
||||
[
|
||||
#image("../assets/community.png", width: 50%)
|
||||
],
|
||||
[
|
||||
*Developed by a community*\
|
||||
#text(size: 0.8em)[Some recent contributors: Arthur C, Charles H, dongdigua, Etienne L, Jonah A, Julien K, Lapineige, MagicRR, Milas B, Niklas M, RockWolf, Schwitzd, trinity-1686a, Xavier S, babykart, Baptiste J, eddster2309, James O'C, Joker9944, Maximilien R, Renjaya RZ, Yureka...]
|
||||
]
|
||||
)
|
||||
|
||||
#v(2em)
|
||||
|
||||
#grid(
|
||||
columns: (2fr, 8fr),
|
||||
[
|
||||
#image("../assets/logos/AGPLv3_Logo.png", width: 50%)
|
||||
],
|
||||
[
|
||||
*Owned by nobody, open-core is impossible, zero VC money*\
|
||||
AGPL + no Contributor License Agreement = Garage ownership spreads among hundredth of contributors.
|
||||
]
|
||||
)
|
||||
|
||||
== Getting support for Garage
|
||||
|
||||
#grid(
|
||||
columns: (2fr, 4fr, 3fr, 1fr),
|
||||
[
|
||||
#image("../assets/alex.jpg", width: 40%)
|
||||
],
|
||||
[
|
||||
*Alex Auvolat*\
|
||||
PhD; co-founder of Deuxfleurs\
|
||||
Garage maintainer, Freelance
|
||||
],
|
||||
[
|
||||
#image("../assets/support.png", width: 40%)
|
||||
],
|
||||
[
|
||||
""
|
||||
]
|
||||
)
|
||||
|
||||
#v(2em)
|
||||
|
||||
#grid(
|
||||
columns: (2fr, 4fr, 4fr),
|
||||
[
|
||||
#image("../assets/quentin.jpg", width: 40%)
|
||||
],
|
||||
[
|
||||
*Quentin Dufour*\
|
||||
PhD; co-founder of Deuxfleurs\
|
||||
Garage contributor, Freelance
|
||||
],
|
||||
[
|
||||
For support requests, write at:\
|
||||
#link("mailto:garagehq@deuxfleurs.fr")
|
||||
]
|
||||
)
|
||||
|
||||
#v(2em)
|
||||
|
||||
#grid(
|
||||
columns: (2fr, 4fr, 4fr),
|
||||
[
|
||||
#image("../assets/armael.jpg", width: 40%)
|
||||
],
|
||||
[
|
||||
*Armaël Guéneau*\
|
||||
PhD; member of Deuxfleurs\
|
||||
Garage contributor, Freelance
|
||||
],
|
||||
[
|
||||
Eligible: email support, architecture design, specific feature development, etc.
|
||||
]
|
||||
)
|
||||
|
||||
== Our initial goal
|
||||
|
||||
#align(center)[
|
||||
#text(size: 1.2em)[Being a self-sovereign community to be free of our degrowth choice]
|
||||
#v(1em)
|
||||
$arrow.b.double$
|
||||
#v(1em)
|
||||
As web citizens, datacenters are big black boxes.\
|
||||
We want to leave them to autonomously manage our servers.
|
||||
#v(1em)
|
||||
$arrow.b.double$
|
||||
#v(1em)
|
||||
We want reliable services without relying on dedicated hardware or places.
|
||||
]
|
||||
|
||||
== Building a resilient system with cheap stuff
|
||||
|
||||
- Commodity hardware (e.g. old desktop PCs)\
|
||||
#text(size: 0.8em)[(can die at any time)]
|
||||
- Regular Internet (e.g. FTTB, FTTH) and power grid connections\
|
||||
#text(size: 0.8em)[(can be unavailable randomly)]
|
||||
- *Geographical redundancy* (multi-site replication)
|
||||
|
||||
#v(1em)
|
||||
#image("../assets/neptune.jpg", width: 80%)
|
||||
#pagebreak()
|
||||
#image("../assets/atuin.jpg", width: 80%)
|
||||
#pagebreak()
|
||||
#image("../assets/inframap_jdll2023.pdf", width: 80%)
|
||||
|
||||
== Object storage: a crucial component
|
||||
|
||||
#grid(
|
||||
columns: (3fr, 3fr, 3fr),
|
||||
[#image("../assets/logos/Amazon-S3.jpg", height: 6em)],
|
||||
[#image("../assets/logos/minio.png", height: 5em)],
|
||||
[#image("../../logo/garage_hires_crop.png", height: 6em)]
|
||||
)
|
||||
|
||||
S3: a de-facto standard, many compatible applications
|
||||
|
||||
MinIO is self-hostable but not suited for geo-distributed deployments
|
||||
|
||||
*Garage is a self-hosted drop-in replacement for the Amazon S3 object store*
|
||||
|
||||
== CRDTs / weak consistency instead of consensus
|
||||
|
||||
#underline[Internally, Garage uses only CRDTs] (conflict-free replicated data types)
|
||||
|
||||
Why not Raft, Paxos, ...? Issues of consensus algorithms:
|
||||
|
||||
- *Software complexity*
|
||||
- *Performance issues:*
|
||||
- The leader is a *bottleneck* for all requests
|
||||
- *Sensitive to higher latency* between nodes
|
||||
- *Takes time to reconverge* when disrupted (e.g. node going down)
|
||||
|
||||
== The data model of object storage
|
||||
|
||||
Object storage is basically a *key-value store*:
|
||||
|
||||
#table(
|
||||
columns: (2fr, 5fr),
|
||||
align: left,
|
||||
[*Key: file path + name*], [*Value: file data + metadata*],
|
||||
[`index.html`], [
|
||||
Content-Type: text/html; charset=utf-8\
|
||||
Content-Length: 24929\
|
||||
\<binary blob\>
|
||||
],
|
||||
[`img/logo.svg`], [
|
||||
Content-Type: text/svg+xml\
|
||||
Content-Length: 13429\
|
||||
\<binary blob\>
|
||||
],
|
||||
[`download/index.html`], [
|
||||
Content-Type: text/html; charset=utf-8\
|
||||
Content-Length: 26563\
|
||||
\<binary blob\>
|
||||
]
|
||||
)
|
||||
|
||||
- Maps well to CRDT data types
|
||||
|
||||
== Performance gains in practice
|
||||
|
||||
#image("../assets/perf/endpoint_latency_0.7_0.8_minio.png", width: 80%)
|
||||
|
||||
== Production clusters
|
||||
|
||||
== Deployment kinds
|
||||
|
||||
#image("../assets/cluster_kind.png", width: 90%)
|
||||
|
||||
== How big they are?
|
||||
|
||||
#image("../assets/cluster_size.png", width: 90%)
|
||||
|
||||
_"Petabyte storage setup for a video site. Nginx as CDN in-front using garage-s3-website feature. Each storage node has ~64TB storage with raid10, no replication within garage. 25gbit nic. haproxy to loadbalance across 5 nodes. mostly reads with very few writes."_
|
||||
|
||||
#v(1em)
|
||||
|
||||
_"We currently manage 7 Garage nodes, 28TB total storage, 6M blocks for 3M objects and 4TB of object data. We have been running Garage in production for 2.5 years."_
|
||||
|
||||
== Operating Garage
|
||||
|
||||
#image("../assets/screenshots/garage_status_0.10.png", width: 90%)
|
||||
#image("../assets/screenshots/garage_status_unhealthy_0.10.png", width: 90%)
|
||||
|
||||
== Garage's architecture
|
||||
|
||||
#image("../assets/garage.drawio.pdf", width: 45%)
|
||||
#image("../assets/garage_sync.drawio.pdf", width: 60%)
|
||||
|
||||
== Digging deeper
|
||||
|
||||
#image("../assets/screenshots/garage_stats_0.10.png", width: 90%)
|
||||
#image("../assets/screenshots/garage_worker_list_0.10.png", width: 50%)
|
||||
#image("../assets/screenshots/garage_worker_param_0.10.png", width: 60%)
|
||||
|
||||
== Potential limitations and bottlenecks
|
||||
|
||||
- Global:
|
||||
- Max. ~100 nodes per cluster (excluding gateways)
|
||||
- Metadata:
|
||||
- One big bucket = bottleneck, object list on 3 nodes only
|
||||
- Block manager:
|
||||
- Lots of small files on disk
|
||||
- Processing the resync queue can be slow
|
||||
|
||||
== Deployment advice for very large clusters
|
||||
|
||||
- Metadata storage:
|
||||
- ZFS mirror (x2) on fast NVMe
|
||||
- Use LMDB storage engine
|
||||
- Data block storage:
|
||||
- Use Garage's native multi-HDD support
|
||||
- XFS on individual drives
|
||||
- Increase block size (1MB → 10MB, requires more RAM and good networking)
|
||||
- Tune `resync-tranquility` and `resync-worker-count` dynamically
|
||||
- Other:
|
||||
- Split data over several buckets
|
||||
- Use less than 100 storage nodes
|
||||
- Use gateway nodes
|
||||
|
||||
Our deployments: < 10 TB. Some people have done more!
|
||||
|
||||
== Focus on Deuxfleurs
|
||||
|
||||
Host institutional websites, partnership with a web agency.\
|
||||
Matrix media backend.
|
||||
|
||||
Plan to use it as an email backend for an internally developed email server.
|
||||
|
||||
== Recent developments
|
||||
|
||||
#image("../assets/tl.drawio.png", width: 80%)
|
||||
|
||||
== April 2022 - Garage v0.7.0
|
||||
|
||||
Focus on #underline[observability and ecosystem integration]
|
||||
|
||||
- *Monitoring:* metrics and traces, using OpenTelemetry
|
||||
- Replication modes with 1 or 2 copies / weaker consistency
|
||||
- Kubernetes integration for node discovery
|
||||
- Admin API (v0.7.2)
|
||||
|
||||
== Metrics (Prometheus + Grafana)
|
||||
|
||||
#image("../assets/screenshots/grafana_dashboard.png", width: 90%)
|
||||
|
||||
== Traces (Jaeger)
|
||||
|
||||
#image("../assets/screenshots/jaeger_listobjects.png", width: 80%)
|
||||
|
||||
== November 2022 - Garage v0.8.0
|
||||
|
||||
Focus on #underline[performance]
|
||||
|
||||
- *Alternative metadata DB engines* (LMDB, Sqlite)
|
||||
- *Performance improvements:* block streaming, various optimizations...
|
||||
- Bucket quotas (max size, max \#objects)
|
||||
- Quality of life improvements, observability, etc.
|
||||
|
||||
== About metadata DB engines
|
||||
|
||||
*Issues with Sled:*
|
||||
|
||||
- Huge files on disk
|
||||
- Unpredictable performance, especially on HDD
|
||||
- API limitations
|
||||
- Not actively maintained
|
||||
|
||||
#v(2em)
|
||||
*LMDB:* very stable, good performance, file size is reasonable\
|
||||
*Sqlite* also available as a second choice
|
||||
|
||||
#v(1em)
|
||||
Sled will be removed in Garage v1.0
|
||||
|
||||
== DB engine performance comparison
|
||||
|
||||
#image("../assets/perf/db_engine.png", width: 60%)
|
||||
#text(size: 0.8em)[NB: Sqlite was slow due to synchronous mode, now configurable]
|
||||
|
||||
== Block streaming
|
||||
|
||||
#image("../assets/schema-streaming-1.png", width: 80%)
|
||||
#image("../assets/schema-streaming-2.png", width: 80%)
|
||||
|
||||
== TTFB benchmark
|
||||
|
||||
#image("../assets/perf/ttfb.png", width: 80%)
|
||||
|
||||
== Throughput benchmark
|
||||
|
||||
#image("../assets/perf/io-0.7-0.8-minio.png", width: 70%)
|
||||
|
||||
== October 2023 - Garage v0.9.0
|
||||
|
||||
Focus on #underline[streamlining & usability]
|
||||
|
||||
- Support multiple HDDs per node
|
||||
- S3 compatibility:
|
||||
- support basic lifecycle configurations
|
||||
- allow for multipart upload part retries
|
||||
- LMDB by default, deprecation of Sled
|
||||
- New layout computation algorithm
|
||||
|
||||
== Layout computation
|
||||
|
||||
#image("../assets/screenshots/garage_status_0.9_prod_zonehl.png", width: 100%)
|
||||
#image("../assets/map.png", width: 70%)
|
||||
|
||||
Garage stores replicas on different zones when possible
|
||||
|
||||
== What a "layout" is
|
||||
|
||||
*A layout is a precomputed index table:*
|
||||
|
||||
#table(
|
||||
columns: (2fr, 2fr, 2fr, 2fr),
|
||||
align: left,
|
||||
[*Partition*], [*Node 1*], [*Node 2*], [*Node 3*],
|
||||
[Partition 0], [df-ymk (bespin)], [Abricot (scorpio)], [Courgette (neptune)],
|
||||
[Partition 1], [Ananas (scorpio)], [Courgette (neptune)], [df-ykl (bespin)],
|
||||
[Partition 2], [df-ymf (bespin)], [Celeri (neptune)], [Abricot (scorpio)],
|
||||
[⋮], [⋮], [⋮], [⋮],
|
||||
[Partition 255], [Concombre (neptune)], [df-ykl (bespin)], [Abricot (scorpio)]
|
||||
)
|
||||
|
||||
The index table is built centrally using an optimal algorithm, then propagated to all nodes
|
||||
|
||||
#text(size: 0.8em)[
|
||||
Oulamara, M., & Auvolat, A. (2023). _An algorithm for geo-distributed and redundant storage in Garage_. arXiv preprint arXiv:2302.13798.
|
||||
]
|
||||
|
||||
== April 2024 - Garage v1.0.0
|
||||
|
||||
Focus on #underline[consistency, security & stability]
|
||||
|
||||
- Fix consistency issues when reshuffling data (Jepsen testing)
|
||||
- *Security audit* by Radically Open Security
|
||||
- Misc. S3 features (SSE-C, checksums, ...) and compatibility fixes
|
||||
|
||||
== Garage v2.0.0
|
||||
|
||||
Focus on #underline[]
|
||||
- TODO
|
||||
|
||||
== Currently funding...
|
||||
|
||||
_..._
|
||||
|
||||
== We run community surveys
|
||||
|
||||
#image("../assets/survey_requested_features.png", width: 60%)
|
||||
|
||||
== Where to find us
|
||||
|
||||
#image("../../logo/garage_hires.png", width: 25%)
|
||||
#link("https://garagehq.deuxfleurs.fr/")
|
||||
#link("mailto:garagehq@deuxfleurs.fr")
|
||||
`#garage:deuxfleurs.fr` on Matrix
|
||||
|
||||
#v(1.5em)
|
||||
#image("../assets/logos/rust_logo.png", width: 6%)
|
||||
#image("../assets/logos/AGPLv3_Logo.png", width: 13%)
|
||||
@@ -1,236 +0,0 @@
|
||||
#import "@preview/slydst:0.1.5": *
|
||||
|
||||
#show: slides
|
||||
|
||||
#title-slide[
|
||||
#image("../../sticker/Garage.pdf", width: 30%)
|
||||
#v(1em)
|
||||
#text(1.2em, weight: "bold")[Garage Object Storage: 2.0 update and best practices]
|
||||
#v(0.5em)
|
||||
_a new storage platform for self-hosted geo-distributed clusters_
|
||||
#v(1em)
|
||||
Maximilien Richer, Deuxfleurs
|
||||
|
||||
#v(1em)
|
||||
#link("https://garagehq.deuxfleurs.fr/")
|
||||
#v(0.5em)
|
||||
Matrix channel: `\#garage:deuxfleurs.fr`
|
||||
]
|
||||
|
||||
== Our objective at Deuxfleurs
|
||||
|
||||
#align(center)[
|
||||
French association promoting digital sovereignty and privacy\
|
||||
through self-hosting hosting *as an alternative to large cloud providers*
|
||||
]
|
||||
#v(2em)
|
||||
#align(center)[
|
||||
*This requires #underline[resilience]*\
|
||||
#text(size: 0.8em)[(we want good uptime/availability with low supervision)]
|
||||
]
|
||||
|
||||
== But what is Garage, exactly?
|
||||
|
||||
*Garage is a self-hosted drop-in replacement for the Amazon S3 object store*\
|
||||
that implements resilience through geographical redundancy on commodity hardware
|
||||
|
||||
#v(1em)
|
||||
#image("../2026-01-31-fosdem/assets/garageuses.png", width: 80%)
|
||||
|
||||
== What makes Garage different?
|
||||
|
||||
*Coordination-free:*
|
||||
#v(2em)
|
||||
- No Raft or Paxos
|
||||
- Internal data types are CRDTs
|
||||
- All nodes are equivalent (no master/leader/index node)
|
||||
|
||||
#v(2em)
|
||||
$\to$ less sensitive to higher latencies between nodes
|
||||
|
||||
== What makes Garage different?
|
||||
|
||||
#align(center)[
|
||||
_TODO update with latest garage and minio versions_
|
||||
#image("../2026-01-31-fosdem/assets/endpoint-latency-dc.png", width: 90%)
|
||||
]
|
||||
|
||||
== What makes Garage different?
|
||||
|
||||
*Consistency model:*
|
||||
#v(2em)
|
||||
- Not ACID (not required by S3 spec) / not linearizable
|
||||
- *Read-after-write consistency*\
|
||||
#text(size: 0.8em)[(stronger than eventual consistency)]
|
||||
|
||||
== What makes Garage different?
|
||||
|
||||
*Location-aware:*
|
||||
#v(2em)
|
||||
#image("../2026-01-31-fosdem/assets/location-aware.png", width: 100%)
|
||||
#v(2em)
|
||||
Garage replicates data on different zones when possible
|
||||
|
||||
== What makes Garage different?
|
||||
|
||||
#image("../2026-01-31-fosdem/assets/map.png", width: 80%)
|
||||
|
||||
== An ever-increasing compatibility list
|
||||
|
||||
#image("../2026-01-31-fosdem/assets/compatibility.png", width: 70%)
|
||||
|
||||
== Version history and roadmap
|
||||
|
||||
- v0.3: initial beta release (2021)
|
||||
- v0.7: first released version (2022)
|
||||
- v1.0: stable release (2024), will be deprecated in summer 2026 1y after v2.0 was released
|
||||
- v2.0: stable release (2025)
|
||||
- new HTTP admin API
|
||||
- reworded replication configuration: `replication_mode` changed to `replication_factor` \& `consistency_policy`
|
||||
-
|
||||
|
||||
#align(center)[
|
||||
v3.0: TBA may include versionning support, tag on buckets and objects, retention policies...
|
||||
]
|
||||
|
||||
== Best practices for Garage deployments
|
||||
|
||||
#align(center)[
|
||||
#text(1.2em, weight: "bold")[Best practices for Garage deployments]
|
||||
]
|
||||
|
||||
== Things you should know
|
||||
|
||||
- no TLS support, use your own proxy
|
||||
- no anonymous access (use website endpoint)
|
||||
- you need to assign roles to nodes manually
|
||||
- the replication factor cannot be changed easily
|
||||
- the default region is `garage` and not `us-east-1`
|
||||
- only use the `degraded` consistency policy for data recovery!
|
||||
|
||||
== What hardware should I use?
|
||||
|
||||
- do NOT use network file storage (NFS, SMB, etc.) for `metadata`
|
||||
- get a *write-intensive flash disk* for the `metadata` folder
|
||||
- set `metadata` on a RAID1 if possible, with a COW filesystem (e.g. Btrfs or ZFS)
|
||||
- get large HDDs for the `data` folder
|
||||
- use XFS and garage multi-hdd mode for best performance
|
||||
- you can use a RAID for data but you'll leave a lot of performance on the table
|
||||
|
||||
#align(center)[
|
||||
_Garage doesn't require a powerful CPUs nor much RAM, but your performance will depend on your disks!_
|
||||
]
|
||||
|
||||
== Picking a metadata engine
|
||||
|
||||
All files-to-block mappings are stored in the metadata engine, including bucket and object metadata. Files below 3KB are stored directly in the metadata engine.
|
||||
|
||||
#v(1em)
|
||||
- Sled: removed in 1.x, move to SQLite or LMDB
|
||||
- *SQLite*: safer, _recommended for small clusters and single-node_
|
||||
- LMDB: faster, recommended for large clusters with metadata redundancy
|
||||
- Warning: limited to 480 bytes per key with LMDB (not an issue in practice)
|
||||
- Fjall: experimental but promising rust-native engine, test it and let us know!
|
||||
|
||||
#align(center)[
|
||||
Metadata engine can be set node per node, and changed later with a migration tool
|
||||
]
|
||||
|
||||
== Single-node deployment
|
||||
|
||||
- garage was initially designed for multi-node deployments
|
||||
- single-node deployments are possible, but you will lose resilience
|
||||
- *If you do please ensure you have backups* (especially for metadata)
|
||||
- set up `metadata_auto_snapshot_interval`
|
||||
- use sqlite to minimize data loss risks on powercuts
|
||||
- or use a UPS!
|
||||
|
||||
#v(1em)
|
||||
Use `github.com/bikeshedder/garage-single-node` for an easy single-node setup!
|
||||
|
||||
== Multi-node deployment
|
||||
|
||||
- try to have geo-distributed zones
|
||||
- multiple nodes per zone to add more capacity
|
||||
- at least 3 zones for best resilience
|
||||
- keep in mind your available network and IO bandwidth
|
||||
- *Rebalancing a cluster can take multiple weeks with large HDDs and slow network links*
|
||||
- monitor your nodes with Prometheus + Grafana
|
||||
|
||||
#align(center)[
|
||||
Deuxfleurs has been running a 9TB (3TB usable) 8-nodes cluster (3+3+2) over retail fiber (10ms site-to-site latency) for close to 5 years now. We heard there are petabyte clusters out there!
|
||||
]
|
||||
|
||||
== Deploying and administering garage at scale
|
||||
|
||||
- deploy with your favorite tool (eg. Ansible) and system manager (eg. systemd)
|
||||
- or use Docker, docker-compose, Kubernetes or Nomad
|
||||
- Kubernetes and Consul are supported for node-to-node discovery
|
||||
- you'll still have to manage the layout manually!
|
||||
- use gateway nodes to optimize network usage
|
||||
- ajust `resync-tranquility` and `scrub-tranquility` to your ressources
|
||||
|
||||
#align(center)[
|
||||
Kubernetes storage controller: `github.com/bmarinov/garage-storage-controller`
|
||||
]
|
||||
|
||||
== Community UI available!
|
||||
|
||||
#align(center)[
|
||||
#image("../2026-01-31-fosdem/assets/community-ui.png", width: 90%)
|
||||
#v(-1em)
|
||||
#link("https://github.com/khairul169/garage-webui")
|
||||
]
|
||||
|
||||
== Official Embedded UI coming later this year!
|
||||
|
||||
#align(center)[
|
||||
#image("../2026-01-31-fosdem/assets/Garage Web Admin - Dashboard@2x.png", width: 90%)
|
||||
#v(-1em)
|
||||
]
|
||||
|
||||
== Official Embedded UI coming this year!
|
||||
|
||||
#align(center)[
|
||||
#image("../2026-01-31-fosdem/assets/Garage Web Admin - Bucket details page@2x.png", width: 90%)
|
||||
#v(-1em)
|
||||
]
|
||||
|
||||
== How to make sense of garage metrics?
|
||||
|
||||
#align(center)[
|
||||
#image("../2026-01-31-fosdem/assets/garage-stats.png", width: 70%)
|
||||
#v(-1em)
|
||||
]
|
||||
|
||||
== What if things go wrong?
|
||||
|
||||
- set logs to debug with `RUST_LOG=garage_api_common=debug,garage_api_s3=debug,garage=debug`
|
||||
- auth issues: check your reverse proxy configuration
|
||||
- slow resync: check your network and disk IO usage, and `resync-tranquility` worker configuration
|
||||
- big LMDB database: stop garage and compact with `mdb_copy -c`
|
||||
- ask us on matrix `\#garage:deuxfleurs.fr` or open an issue on git.deuxfleurs.fr!
|
||||
- provide the output of `garage status`, `garage stats` and relevant metrics and logs
|
||||
|
||||
== Moving from Minio
|
||||
|
||||
- list your buckets and your keys
|
||||
- create buckets and keys on the garage cluster
|
||||
- you cannot import non-garage keys yet, patch to come soon!
|
||||
- loop over buckets, copy with rclone
|
||||
- see doc #link("https://garagehq.deuxfleurs.fr/documentation/connect/cli/")
|
||||
- blog post coming soon!
|
||||
|
||||
== Demo time!
|
||||
|
||||
== Get Garage now!
|
||||
|
||||
#align(center)[
|
||||
#image("../../logo/garage_hires.png", width: 30%)
|
||||
#v(-1em)
|
||||
#link("https://garagehq.deuxfleurs.fr/")
|
||||
Matrix channel: `\#garage:deuxfleurs.fr`
|
||||
#v(2em)
|
||||
#image("../2026-01-31-fosdem/assets/rust_logo.png", width: 9%)
|
||||
#image("../2026-01-31-fosdem/assets/AGPLv3_Logo.png", width: 20%)
|
||||
]
|
||||
Reference in New Issue
Block a user