mirror of
https://github.com/deuxfleurs-org/garage.git
synced 2026-07-26 16:08:13 +00:00
add translation of second talk to typst, based on 2025-10-06-josy
This commit is contained in:
Alex Auvolat
@@ -0,0 +1,403 @@
|
||||
#import "@preview/slydst:0.1.5": *
|
||||
|
||||
#show: slides
|
||||
|
||||
#title-slide[
|
||||
#image("../../sticker/Garage.png", width: 30%)
|
||||
#v(1em)
|
||||
#text(1.2em, weight: "bold")[Garage, an S3 backend as reliable as possible]
|
||||
#v(1em)
|
||||
Garage Authors
|
||||
|
||||
#v(1em)
|
||||
#link("https://garagehq.deuxfleurs.fr/")
|
||||
#v(0.5em)
|
||||
#link("mailto:garagehq@deuxfleurs.fr")
|
||||
#v(0.5em)
|
||||
Matrix channel: `#garage:deuxfleurs.fr`
|
||||
]
|
||||
|
||||
== Meet Garage
|
||||
|
||||
== A non-profit initiative
|
||||
|
||||
#grid(
|
||||
columns: (2fr, 8fr),
|
||||
[
|
||||
#image("../assets/logos/deuxfleurs.svg", width: 50%)
|
||||
],
|
||||
[
|
||||
*Part of a degrowth initiative*\
|
||||
Garage has been created at Deuxfleurs where we experiment running Internet services without datacenter on commodity and refurbished hardware.
|
||||
]
|
||||
)
|
||||
|
||||
#v(2em)
|
||||
|
||||
#grid(
|
||||
columns: (2fr, 8fr),
|
||||
[
|
||||
#image("../assets/community.png", width: 50%)
|
||||
],
|
||||
[
|
||||
*Developed by a community*\
|
||||
#text(size: 0.8em)[Some recent contributors: Arthur C, Charles H, dongdigua, Etienne L, Jonah A, Julien K, Lapineige, MagicRR, Milas B, Niklas M, RockWolf, Schwitzd, trinity-1686a, Xavier S, babykart, Baptiste J, eddster2309, James O'C, Joker9944, Maximilien R, Renjaya RZ, Yureka...]
|
||||
]
|
||||
)
|
||||
|
||||
#v(2em)
|
||||
|
||||
#grid(
|
||||
columns: (2fr, 8fr),
|
||||
[
|
||||
#image("../assets/logos/AGPLv3_Logo.png", width: 50%)
|
||||
],
|
||||
[
|
||||
*Owned by nobody, open-core is impossible, zero VC money*\
|
||||
AGPL + no Contributor License Agreement = Garage ownership spreads among hundredth of contributors.
|
||||
]
|
||||
)
|
||||
|
||||
== Getting support for Garage
|
||||
|
||||
#grid(
|
||||
columns: (2fr, 4fr, 3fr, 1fr),
|
||||
[
|
||||
#image("../assets/alex.jpg", width: 40%)
|
||||
],
|
||||
[
|
||||
*Alex Auvolat*\
|
||||
PhD; co-founder of Deuxfleurs\
|
||||
Garage maintainer, Freelance
|
||||
],
|
||||
[
|
||||
#image("../assets/support.png", width: 40%)
|
||||
],
|
||||
[
|
||||
""
|
||||
]
|
||||
)
|
||||
|
||||
#v(2em)
|
||||
|
||||
#grid(
|
||||
columns: (2fr, 4fr, 4fr),
|
||||
[
|
||||
#image("../assets/quentin.jpg", width: 40%)
|
||||
],
|
||||
[
|
||||
*Quentin Dufour*\
|
||||
PhD; co-founder of Deuxfleurs\
|
||||
Garage contributor, Freelance
|
||||
],
|
||||
[
|
||||
For support requests, write at:\
|
||||
#link("mailto:garagehq@deuxfleurs.fr")
|
||||
]
|
||||
)
|
||||
|
||||
#v(2em)
|
||||
|
||||
#grid(
|
||||
columns: (2fr, 4fr, 4fr),
|
||||
[
|
||||
#image("../assets/armael.jpg", width: 40%)
|
||||
],
|
||||
[
|
||||
*Armaël Guéneau*\
|
||||
PhD; member of Deuxfleurs\
|
||||
Garage contributor, Freelance
|
||||
],
|
||||
[
|
||||
Eligible: email support, architecture design, specific feature development, etc.
|
||||
]
|
||||
)
|
||||
|
||||
== Our initial goal
|
||||
|
||||
#align(center)[
|
||||
#text(size: 1.2em)[Being a self-sovereign community to be free of our degrowth choice]
|
||||
#v(1em)
|
||||
$arrow.b.double$
|
||||
#v(1em)
|
||||
As web citizens, datacenters are big black boxes.\
|
||||
We want to leave them to autonomously manage our servers.
|
||||
#v(1em)
|
||||
$arrow.b.double$
|
||||
#v(1em)
|
||||
We want reliable services without relying on dedicated hardware or places.
|
||||
]
|
||||
|
||||
== Building a resilient system with cheap stuff
|
||||
|
||||
- Commodity hardware (e.g. old desktop PCs)\
|
||||
#text(size: 0.8em)[(can die at any time)]
|
||||
- Regular Internet (e.g. FTTB, FTTH) and power grid connections\
|
||||
#text(size: 0.8em)[(can be unavailable randomly)]
|
||||
- *Geographical redundancy* (multi-site replication)
|
||||
|
||||
#v(1em)
|
||||
#image("../assets/neptune.jpg", width: 80%)
|
||||
#pagebreak()
|
||||
#image("../assets/atuin.jpg", width: 80%)
|
||||
#pagebreak()
|
||||
#image("../assets/inframap_jdll2023.pdf", width: 80%)
|
||||
|
||||
== Object storage: a crucial component
|
||||
|
||||
#grid(
|
||||
columns: (3fr, 3fr, 3fr),
|
||||
[#image("../assets/logos/Amazon-S3.jpg", height: 6em)],
|
||||
[#image("../assets/logos/minio.png", height: 5em)],
|
||||
[#image("../../logo/garage_hires_crop.png", height: 6em)]
|
||||
)
|
||||
|
||||
S3: a de-facto standard, many compatible applications
|
||||
|
||||
MinIO is self-hostable but not suited for geo-distributed deployments
|
||||
|
||||
*Garage is a self-hosted drop-in replacement for the Amazon S3 object store*
|
||||
|
||||
== CRDTs / weak consistency instead of consensus
|
||||
|
||||
#underline[Internally, Garage uses only CRDTs] (conflict-free replicated data types)
|
||||
|
||||
Why not Raft, Paxos, ...? Issues of consensus algorithms:
|
||||
|
||||
- *Software complexity*
|
||||
- *Performance issues:*
|
||||
- The leader is a *bottleneck* for all requests
|
||||
- *Sensitive to higher latency* between nodes
|
||||
- *Takes time to reconverge* when disrupted (e.g. node going down)
|
||||
|
||||
== The data model of object storage
|
||||
|
||||
Object storage is basically a *key-value store*:
|
||||
|
||||
#table(
|
||||
columns: (2fr, 5fr),
|
||||
align: left,
|
||||
[*Key: file path + name*], [*Value: file data + metadata*],
|
||||
[`index.html`], [
|
||||
Content-Type: text/html; charset=utf-8\
|
||||
Content-Length: 24929\
|
||||
\<binary blob\>
|
||||
],
|
||||
[`img/logo.svg`], [
|
||||
Content-Type: text/svg+xml\
|
||||
Content-Length: 13429\
|
||||
\<binary blob\>
|
||||
],
|
||||
[`download/index.html`], [
|
||||
Content-Type: text/html; charset=utf-8\
|
||||
Content-Length: 26563\
|
||||
\<binary blob\>
|
||||
]
|
||||
)
|
||||
|
||||
- Maps well to CRDT data types
|
||||
|
||||
== Performance gains in practice
|
||||
|
||||
#image("../assets/perf/endpoint_latency_0.7_0.8_minio.png", width: 80%)
|
||||
|
||||
== Production clusters
|
||||
|
||||
== Deployment kinds
|
||||
|
||||
#image("../assets/cluster_kind.png", width: 90%)
|
||||
|
||||
== How big they are?
|
||||
|
||||
#image("../assets/cluster_size.png", width: 90%)
|
||||
|
||||
_"Petabyte storage setup for a video site. Nginx as CDN in-front using garage-s3-website feature. Each storage node has ~64TB storage with raid10, no replication within garage. 25gbit nic. haproxy to loadbalance across 5 nodes. mostly reads with very few writes."_
|
||||
|
||||
#v(1em)
|
||||
|
||||
_"We currently manage 7 Garage nodes, 28TB total storage, 6M blocks for 3M objects and 4TB of object data. We have been running Garage in production for 2.5 years."_
|
||||
|
||||
== Operating Garage
|
||||
|
||||
#image("../assets/screenshots/garage_status_0.10.png", width: 90%)
|
||||
#image("../assets/screenshots/garage_status_unhealthy_0.10.png", width: 90%)
|
||||
|
||||
== Garage's architecture
|
||||
|
||||
#image("../assets/garage.drawio.pdf", width: 45%)
|
||||
#image("../assets/garage_sync.drawio.pdf", width: 60%)
|
||||
|
||||
== Digging deeper
|
||||
|
||||
#image("../assets/screenshots/garage_stats_0.10.png", width: 90%)
|
||||
#image("../assets/screenshots/garage_worker_list_0.10.png", width: 50%)
|
||||
#image("../assets/screenshots/garage_worker_param_0.10.png", width: 60%)
|
||||
|
||||
== Potential limitations and bottlenecks
|
||||
|
||||
- Global:
|
||||
- Max. ~100 nodes per cluster (excluding gateways)
|
||||
- Metadata:
|
||||
- One big bucket = bottleneck, object list on 3 nodes only
|
||||
- Block manager:
|
||||
- Lots of small files on disk
|
||||
- Processing the resync queue can be slow
|
||||
|
||||
== Deployment advice for very large clusters
|
||||
|
||||
- Metadata storage:
|
||||
- ZFS mirror (x2) on fast NVMe
|
||||
- Use LMDB storage engine
|
||||
- Data block storage:
|
||||
- Use Garage's native multi-HDD support
|
||||
- XFS on individual drives
|
||||
- Increase block size (1MB → 10MB, requires more RAM and good networking)
|
||||
- Tune `resync-tranquility` and `resync-worker-count` dynamically
|
||||
- Other:
|
||||
- Split data over several buckets
|
||||
- Use less than 100 storage nodes
|
||||
- Use gateway nodes
|
||||
|
||||
Our deployments: < 10 TB. Some people have done more!
|
||||
|
||||
== Focus on Deuxfleurs
|
||||
|
||||
Host institutional websites, partnership with a web agency.\
|
||||
Matrix media backend.
|
||||
|
||||
Plan to use it as an email backend for an internally developed email server.
|
||||
|
||||
== Recent developments
|
||||
|
||||
#image("../assets/tl.drawio.png", width: 80%)
|
||||
|
||||
== April 2022 - Garage v0.7.0
|
||||
|
||||
Focus on #underline[observability and ecosystem integration]
|
||||
|
||||
- *Monitoring:* metrics and traces, using OpenTelemetry
|
||||
- Replication modes with 1 or 2 copies / weaker consistency
|
||||
- Kubernetes integration for node discovery
|
||||
- Admin API (v0.7.2)
|
||||
|
||||
== Metrics (Prometheus + Grafana)
|
||||
|
||||
#image("../assets/screenshots/grafana_dashboard.png", width: 90%)
|
||||
|
||||
== Traces (Jaeger)
|
||||
|
||||
#image("../assets/screenshots/jaeger_listobjects.png", width: 80%)
|
||||
|
||||
== November 2022 - Garage v0.8.0
|
||||
|
||||
Focus on #underline[performance]
|
||||
|
||||
- *Alternative metadata DB engines* (LMDB, Sqlite)
|
||||
- *Performance improvements:* block streaming, various optimizations...
|
||||
- Bucket quotas (max size, max \#objects)
|
||||
- Quality of life improvements, observability, etc.
|
||||
|
||||
== About metadata DB engines
|
||||
|
||||
*Issues with Sled:*
|
||||
|
||||
- Huge files on disk
|
||||
- Unpredictable performance, especially on HDD
|
||||
- API limitations
|
||||
- Not actively maintained
|
||||
|
||||
#v(2em)
|
||||
*LMDB:* very stable, good performance, file size is reasonable\
|
||||
*Sqlite* also available as a second choice
|
||||
|
||||
#v(1em)
|
||||
Sled will be removed in Garage v1.0
|
||||
|
||||
== DB engine performance comparison
|
||||
|
||||
#image("../assets/perf/db_engine.png", width: 60%)
|
||||
#text(size: 0.8em)[NB: Sqlite was slow due to synchronous mode, now configurable]
|
||||
|
||||
== Block streaming
|
||||
|
||||
#image("../assets/schema-streaming-1.png", width: 80%)
|
||||
#image("../assets/schema-streaming-2.png", width: 80%)
|
||||
|
||||
== TTFB benchmark
|
||||
|
||||
#image("../assets/perf/ttfb.png", width: 80%)
|
||||
|
||||
== Throughput benchmark
|
||||
|
||||
#image("../assets/perf/io-0.7-0.8-minio.png", width: 70%)
|
||||
|
||||
== October 2023 - Garage v0.9.0
|
||||
|
||||
Focus on #underline[streamlining & usability]
|
||||
|
||||
- Support multiple HDDs per node
|
||||
- S3 compatibility:
|
||||
- support basic lifecycle configurations
|
||||
- allow for multipart upload part retries
|
||||
- LMDB by default, deprecation of Sled
|
||||
- New layout computation algorithm
|
||||
|
||||
== Layout computation
|
||||
|
||||
#image("../assets/screenshots/garage_status_0.9_prod_zonehl.png", width: 100%)
|
||||
#image("../assets/map.png", width: 70%)
|
||||
|
||||
Garage stores replicas on different zones when possible
|
||||
|
||||
== What a "layout" is
|
||||
|
||||
*A layout is a precomputed index table:*
|
||||
|
||||
#table(
|
||||
columns: (2fr, 2fr, 2fr, 2fr),
|
||||
align: left,
|
||||
[*Partition*], [*Node 1*], [*Node 2*], [*Node 3*],
|
||||
[Partition 0], [df-ymk (bespin)], [Abricot (scorpio)], [Courgette (neptune)],
|
||||
[Partition 1], [Ananas (scorpio)], [Courgette (neptune)], [df-ykl (bespin)],
|
||||
[Partition 2], [df-ymf (bespin)], [Celeri (neptune)], [Abricot (scorpio)],
|
||||
[⋮], [⋮], [⋮], [⋮],
|
||||
[Partition 255], [Concombre (neptune)], [df-ykl (bespin)], [Abricot (scorpio)]
|
||||
)
|
||||
|
||||
The index table is built centrally using an optimal algorithm, then propagated to all nodes
|
||||
|
||||
#text(size: 0.8em)[
|
||||
Oulamara, M., & Auvolat, A. (2023). _An algorithm for geo-distributed and redundant storage in Garage_. arXiv preprint arXiv:2302.13798.
|
||||
]
|
||||
|
||||
== April 2024 - Garage v1.0.0
|
||||
|
||||
Focus on #underline[consistency, security & stability]
|
||||
|
||||
- Fix consistency issues when reshuffling data (Jepsen testing)
|
||||
- *Security audit* by Radically Open Security
|
||||
- Misc. S3 features (SSE-C, checksums, ...) and compatibility fixes
|
||||
|
||||
== Garage v2.0.0
|
||||
|
||||
Focus on #underline[]
|
||||
- TODO
|
||||
|
||||
== Currently funding...
|
||||
|
||||
_..._
|
||||
|
||||
== We run community surveys
|
||||
|
||||
#image("../assets/survey_requested_features.png", width: 60%)
|
||||
|
||||
== Where to find us
|
||||
|
||||
#image("../../logo/garage_hires.png", width: 25%)
|
||||
#link("https://garagehq.deuxfleurs.fr/")
|
||||
#link("mailto:garagehq@deuxfleurs.fr")
|
||||
`#garage:deuxfleurs.fr` on Matrix
|
||||
|
||||
#v(1.5em)
|
||||
#image("../assets/logos/rust_logo.png", width: 6%)
|
||||
#image("../assets/logos/AGPLv3_Logo.png", width: 13%)
|
||||
Reference in New Issue
Block a user