mirror of
https://github.com/deuxfleurs-org/garage.git
synced 2026-08-09 14:19:22 +00:00
add translation of second talk to typst, based on 2025-10-06-josy
This commit is contained in:
Alex Auvolat
@@ -0,0 +1,403 @@
|
|||||||
|
#import "@preview/slydst:0.1.5": *
|
||||||
|
|
||||||
|
#show: slides
|
||||||
|
|
||||||
|
#title-slide[
|
||||||
|
#image("../../sticker/Garage.png", width: 30%)
|
||||||
|
#v(1em)
|
||||||
|
#text(1.2em, weight: "bold")[Garage, an S3 backend as reliable as possible]
|
||||||
|
#v(1em)
|
||||||
|
Garage Authors
|
||||||
|
|
||||||
|
#v(1em)
|
||||||
|
#link("https://garagehq.deuxfleurs.fr/")
|
||||||
|
#v(0.5em)
|
||||||
|
#link("mailto:garagehq@deuxfleurs.fr")
|
||||||
|
#v(0.5em)
|
||||||
|
Matrix channel: `#garage:deuxfleurs.fr`
|
||||||
|
]
|
||||||
|
|
||||||
|
== Meet Garage
|
||||||
|
|
||||||
|
== A non-profit initiative
|
||||||
|
|
||||||
|
#grid(
|
||||||
|
columns: (2fr, 8fr),
|
||||||
|
[
|
||||||
|
#image("../assets/logos/deuxfleurs.svg", width: 50%)
|
||||||
|
],
|
||||||
|
[
|
||||||
|
*Part of a degrowth initiative*\
|
||||||
|
Garage has been created at Deuxfleurs where we experiment running Internet services without datacenter on commodity and refurbished hardware.
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
#v(2em)
|
||||||
|
|
||||||
|
#grid(
|
||||||
|
columns: (2fr, 8fr),
|
||||||
|
[
|
||||||
|
#image("../assets/community.png", width: 50%)
|
||||||
|
],
|
||||||
|
[
|
||||||
|
*Developed by a community*\
|
||||||
|
#text(size: 0.8em)[Some recent contributors: Arthur C, Charles H, dongdigua, Etienne L, Jonah A, Julien K, Lapineige, MagicRR, Milas B, Niklas M, RockWolf, Schwitzd, trinity-1686a, Xavier S, babykart, Baptiste J, eddster2309, James O'C, Joker9944, Maximilien R, Renjaya RZ, Yureka...]
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
#v(2em)
|
||||||
|
|
||||||
|
#grid(
|
||||||
|
columns: (2fr, 8fr),
|
||||||
|
[
|
||||||
|
#image("../assets/logos/AGPLv3_Logo.png", width: 50%)
|
||||||
|
],
|
||||||
|
[
|
||||||
|
*Owned by nobody, open-core is impossible, zero VC money*\
|
||||||
|
AGPL + no Contributor License Agreement = Garage ownership spreads among hundredth of contributors.
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
== Getting support for Garage
|
||||||
|
|
||||||
|
#grid(
|
||||||
|
columns: (2fr, 4fr, 3fr, 1fr),
|
||||||
|
[
|
||||||
|
#image("../assets/alex.jpg", width: 40%)
|
||||||
|
],
|
||||||
|
[
|
||||||
|
*Alex Auvolat*\
|
||||||
|
PhD; co-founder of Deuxfleurs\
|
||||||
|
Garage maintainer, Freelance
|
||||||
|
],
|
||||||
|
[
|
||||||
|
#image("../assets/support.png", width: 40%)
|
||||||
|
],
|
||||||
|
[
|
||||||
|
""
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
#v(2em)
|
||||||
|
|
||||||
|
#grid(
|
||||||
|
columns: (2fr, 4fr, 4fr),
|
||||||
|
[
|
||||||
|
#image("../assets/quentin.jpg", width: 40%)
|
||||||
|
],
|
||||||
|
[
|
||||||
|
*Quentin Dufour*\
|
||||||
|
PhD; co-founder of Deuxfleurs\
|
||||||
|
Garage contributor, Freelance
|
||||||
|
],
|
||||||
|
[
|
||||||
|
For support requests, write at:\
|
||||||
|
#link("mailto:garagehq@deuxfleurs.fr")
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
#v(2em)
|
||||||
|
|
||||||
|
#grid(
|
||||||
|
columns: (2fr, 4fr, 4fr),
|
||||||
|
[
|
||||||
|
#image("../assets/armael.jpg", width: 40%)
|
||||||
|
],
|
||||||
|
[
|
||||||
|
*Armaël Guéneau*\
|
||||||
|
PhD; member of Deuxfleurs\
|
||||||
|
Garage contributor, Freelance
|
||||||
|
],
|
||||||
|
[
|
||||||
|
Eligible: email support, architecture design, specific feature development, etc.
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
== Our initial goal
|
||||||
|
|
||||||
|
#align(center)[
|
||||||
|
#text(size: 1.2em)[Being a self-sovereign community to be free of our degrowth choice]
|
||||||
|
#v(1em)
|
||||||
|
$arrow.b.double$
|
||||||
|
#v(1em)
|
||||||
|
As web citizens, datacenters are big black boxes.\
|
||||||
|
We want to leave them to autonomously manage our servers.
|
||||||
|
#v(1em)
|
||||||
|
$arrow.b.double$
|
||||||
|
#v(1em)
|
||||||
|
We want reliable services without relying on dedicated hardware or places.
|
||||||
|
]
|
||||||
|
|
||||||
|
== Building a resilient system with cheap stuff
|
||||||
|
|
||||||
|
- Commodity hardware (e.g. old desktop PCs)\
|
||||||
|
#text(size: 0.8em)[(can die at any time)]
|
||||||
|
- Regular Internet (e.g. FTTB, FTTH) and power grid connections\
|
||||||
|
#text(size: 0.8em)[(can be unavailable randomly)]
|
||||||
|
- *Geographical redundancy* (multi-site replication)
|
||||||
|
|
||||||
|
#v(1em)
|
||||||
|
#image("../assets/neptune.jpg", width: 80%)
|
||||||
|
#pagebreak()
|
||||||
|
#image("../assets/atuin.jpg", width: 80%)
|
||||||
|
#pagebreak()
|
||||||
|
#image("../assets/inframap_jdll2023.pdf", width: 80%)
|
||||||
|
|
||||||
|
== Object storage: a crucial component
|
||||||
|
|
||||||
|
#grid(
|
||||||
|
columns: (3fr, 3fr, 3fr),
|
||||||
|
[#image("../assets/logos/Amazon-S3.jpg", height: 6em)],
|
||||||
|
[#image("../assets/logos/minio.png", height: 5em)],
|
||||||
|
[#image("../../logo/garage_hires_crop.png", height: 6em)]
|
||||||
|
)
|
||||||
|
|
||||||
|
S3: a de-facto standard, many compatible applications
|
||||||
|
|
||||||
|
MinIO is self-hostable but not suited for geo-distributed deployments
|
||||||
|
|
||||||
|
*Garage is a self-hosted drop-in replacement for the Amazon S3 object store*
|
||||||
|
|
||||||
|
== CRDTs / weak consistency instead of consensus
|
||||||
|
|
||||||
|
#underline[Internally, Garage uses only CRDTs] (conflict-free replicated data types)
|
||||||
|
|
||||||
|
Why not Raft, Paxos, ...? Issues of consensus algorithms:
|
||||||
|
|
||||||
|
- *Software complexity*
|
||||||
|
- *Performance issues:*
|
||||||
|
- The leader is a *bottleneck* for all requests
|
||||||
|
- *Sensitive to higher latency* between nodes
|
||||||
|
- *Takes time to reconverge* when disrupted (e.g. node going down)
|
||||||
|
|
||||||
|
== The data model of object storage
|
||||||
|
|
||||||
|
Object storage is basically a *key-value store*:
|
||||||
|
|
||||||
|
#table(
|
||||||
|
columns: (2fr, 5fr),
|
||||||
|
align: left,
|
||||||
|
[*Key: file path + name*], [*Value: file data + metadata*],
|
||||||
|
[`index.html`], [
|
||||||
|
Content-Type: text/html; charset=utf-8\
|
||||||
|
Content-Length: 24929\
|
||||||
|
\<binary blob\>
|
||||||
|
],
|
||||||
|
[`img/logo.svg`], [
|
||||||
|
Content-Type: text/svg+xml\
|
||||||
|
Content-Length: 13429\
|
||||||
|
\<binary blob\>
|
||||||
|
],
|
||||||
|
[`download/index.html`], [
|
||||||
|
Content-Type: text/html; charset=utf-8\
|
||||||
|
Content-Length: 26563\
|
||||||
|
\<binary blob\>
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
- Maps well to CRDT data types
|
||||||
|
|
||||||
|
== Performance gains in practice
|
||||||
|
|
||||||
|
#image("../assets/perf/endpoint_latency_0.7_0.8_minio.png", width: 80%)
|
||||||
|
|
||||||
|
== Production clusters
|
||||||
|
|
||||||
|
== Deployment kinds
|
||||||
|
|
||||||
|
#image("../assets/cluster_kind.png", width: 90%)
|
||||||
|
|
||||||
|
== How big they are?
|
||||||
|
|
||||||
|
#image("../assets/cluster_size.png", width: 90%)
|
||||||
|
|
||||||
|
_"Petabyte storage setup for a video site. Nginx as CDN in-front using garage-s3-website feature. Each storage node has ~64TB storage with raid10, no replication within garage. 25gbit nic. haproxy to loadbalance across 5 nodes. mostly reads with very few writes."_
|
||||||
|
|
||||||
|
#v(1em)
|
||||||
|
|
||||||
|
_"We currently manage 7 Garage nodes, 28TB total storage, 6M blocks for 3M objects and 4TB of object data. We have been running Garage in production for 2.5 years."_
|
||||||
|
|
||||||
|
== Operating Garage
|
||||||
|
|
||||||
|
#image("../assets/screenshots/garage_status_0.10.png", width: 90%)
|
||||||
|
#image("../assets/screenshots/garage_status_unhealthy_0.10.png", width: 90%)
|
||||||
|
|
||||||
|
== Garage's architecture
|
||||||
|
|
||||||
|
#image("../assets/garage.drawio.pdf", width: 45%)
|
||||||
|
#image("../assets/garage_sync.drawio.pdf", width: 60%)
|
||||||
|
|
||||||
|
== Digging deeper
|
||||||
|
|
||||||
|
#image("../assets/screenshots/garage_stats_0.10.png", width: 90%)
|
||||||
|
#image("../assets/screenshots/garage_worker_list_0.10.png", width: 50%)
|
||||||
|
#image("../assets/screenshots/garage_worker_param_0.10.png", width: 60%)
|
||||||
|
|
||||||
|
== Potential limitations and bottlenecks
|
||||||
|
|
||||||
|
- Global:
|
||||||
|
- Max. ~100 nodes per cluster (excluding gateways)
|
||||||
|
- Metadata:
|
||||||
|
- One big bucket = bottleneck, object list on 3 nodes only
|
||||||
|
- Block manager:
|
||||||
|
- Lots of small files on disk
|
||||||
|
- Processing the resync queue can be slow
|
||||||
|
|
||||||
|
== Deployment advice for very large clusters
|
||||||
|
|
||||||
|
- Metadata storage:
|
||||||
|
- ZFS mirror (x2) on fast NVMe
|
||||||
|
- Use LMDB storage engine
|
||||||
|
- Data block storage:
|
||||||
|
- Use Garage's native multi-HDD support
|
||||||
|
- XFS on individual drives
|
||||||
|
- Increase block size (1MB → 10MB, requires more RAM and good networking)
|
||||||
|
- Tune `resync-tranquility` and `resync-worker-count` dynamically
|
||||||
|
- Other:
|
||||||
|
- Split data over several buckets
|
||||||
|
- Use less than 100 storage nodes
|
||||||
|
- Use gateway nodes
|
||||||
|
|
||||||
|
Our deployments: < 10 TB. Some people have done more!
|
||||||
|
|
||||||
|
== Focus on Deuxfleurs
|
||||||
|
|
||||||
|
Host institutional websites, partnership with a web agency.\
|
||||||
|
Matrix media backend.
|
||||||
|
|
||||||
|
Plan to use it as an email backend for an internally developed email server.
|
||||||
|
|
||||||
|
== Recent developments
|
||||||
|
|
||||||
|
#image("../assets/tl.drawio.png", width: 80%)
|
||||||
|
|
||||||
|
== April 2022 - Garage v0.7.0
|
||||||
|
|
||||||
|
Focus on #underline[observability and ecosystem integration]
|
||||||
|
|
||||||
|
- *Monitoring:* metrics and traces, using OpenTelemetry
|
||||||
|
- Replication modes with 1 or 2 copies / weaker consistency
|
||||||
|
- Kubernetes integration for node discovery
|
||||||
|
- Admin API (v0.7.2)
|
||||||
|
|
||||||
|
== Metrics (Prometheus + Grafana)
|
||||||
|
|
||||||
|
#image("../assets/screenshots/grafana_dashboard.png", width: 90%)
|
||||||
|
|
||||||
|
== Traces (Jaeger)
|
||||||
|
|
||||||
|
#image("../assets/screenshots/jaeger_listobjects.png", width: 80%)
|
||||||
|
|
||||||
|
== November 2022 - Garage v0.8.0
|
||||||
|
|
||||||
|
Focus on #underline[performance]
|
||||||
|
|
||||||
|
- *Alternative metadata DB engines* (LMDB, Sqlite)
|
||||||
|
- *Performance improvements:* block streaming, various optimizations...
|
||||||
|
- Bucket quotas (max size, max \#objects)
|
||||||
|
- Quality of life improvements, observability, etc.
|
||||||
|
|
||||||
|
== About metadata DB engines
|
||||||
|
|
||||||
|
*Issues with Sled:*
|
||||||
|
|
||||||
|
- Huge files on disk
|
||||||
|
- Unpredictable performance, especially on HDD
|
||||||
|
- API limitations
|
||||||
|
- Not actively maintained
|
||||||
|
|
||||||
|
#v(2em)
|
||||||
|
*LMDB:* very stable, good performance, file size is reasonable\
|
||||||
|
*Sqlite* also available as a second choice
|
||||||
|
|
||||||
|
#v(1em)
|
||||||
|
Sled will be removed in Garage v1.0
|
||||||
|
|
||||||
|
== DB engine performance comparison
|
||||||
|
|
||||||
|
#image("../assets/perf/db_engine.png", width: 60%)
|
||||||
|
#text(size: 0.8em)[NB: Sqlite was slow due to synchronous mode, now configurable]
|
||||||
|
|
||||||
|
== Block streaming
|
||||||
|
|
||||||
|
#image("../assets/schema-streaming-1.png", width: 80%)
|
||||||
|
#image("../assets/schema-streaming-2.png", width: 80%)
|
||||||
|
|
||||||
|
== TTFB benchmark
|
||||||
|
|
||||||
|
#image("../assets/perf/ttfb.png", width: 80%)
|
||||||
|
|
||||||
|
== Throughput benchmark
|
||||||
|
|
||||||
|
#image("../assets/perf/io-0.7-0.8-minio.png", width: 70%)
|
||||||
|
|
||||||
|
== October 2023 - Garage v0.9.0
|
||||||
|
|
||||||
|
Focus on #underline[streamlining & usability]
|
||||||
|
|
||||||
|
- Support multiple HDDs per node
|
||||||
|
- S3 compatibility:
|
||||||
|
- support basic lifecycle configurations
|
||||||
|
- allow for multipart upload part retries
|
||||||
|
- LMDB by default, deprecation of Sled
|
||||||
|
- New layout computation algorithm
|
||||||
|
|
||||||
|
== Layout computation
|
||||||
|
|
||||||
|
#image("../assets/screenshots/garage_status_0.9_prod_zonehl.png", width: 100%)
|
||||||
|
#image("../assets/map.png", width: 70%)
|
||||||
|
|
||||||
|
Garage stores replicas on different zones when possible
|
||||||
|
|
||||||
|
== What a "layout" is
|
||||||
|
|
||||||
|
*A layout is a precomputed index table:*
|
||||||
|
|
||||||
|
#table(
|
||||||
|
columns: (2fr, 2fr, 2fr, 2fr),
|
||||||
|
align: left,
|
||||||
|
[*Partition*], [*Node 1*], [*Node 2*], [*Node 3*],
|
||||||
|
[Partition 0], [df-ymk (bespin)], [Abricot (scorpio)], [Courgette (neptune)],
|
||||||
|
[Partition 1], [Ananas (scorpio)], [Courgette (neptune)], [df-ykl (bespin)],
|
||||||
|
[Partition 2], [df-ymf (bespin)], [Celeri (neptune)], [Abricot (scorpio)],
|
||||||
|
[⋮], [⋮], [⋮], [⋮],
|
||||||
|
[Partition 255], [Concombre (neptune)], [df-ykl (bespin)], [Abricot (scorpio)]
|
||||||
|
)
|
||||||
|
|
||||||
|
The index table is built centrally using an optimal algorithm, then propagated to all nodes
|
||||||
|
|
||||||
|
#text(size: 0.8em)[
|
||||||
|
Oulamara, M., & Auvolat, A. (2023). _An algorithm for geo-distributed and redundant storage in Garage_. arXiv preprint arXiv:2302.13798.
|
||||||
|
]
|
||||||
|
|
||||||
|
== April 2024 - Garage v1.0.0
|
||||||
|
|
||||||
|
Focus on #underline[consistency, security & stability]
|
||||||
|
|
||||||
|
- Fix consistency issues when reshuffling data (Jepsen testing)
|
||||||
|
- *Security audit* by Radically Open Security
|
||||||
|
- Misc. S3 features (SSE-C, checksums, ...) and compatibility fixes
|
||||||
|
|
||||||
|
== Garage v2.0.0
|
||||||
|
|
||||||
|
Focus on #underline[]
|
||||||
|
- TODO
|
||||||
|
|
||||||
|
== Currently funding...
|
||||||
|
|
||||||
|
_..._
|
||||||
|
|
||||||
|
== We run community surveys
|
||||||
|
|
||||||
|
#image("../assets/survey_requested_features.png", width: 60%)
|
||||||
|
|
||||||
|
== Where to find us
|
||||||
|
|
||||||
|
#image("../../logo/garage_hires.png", width: 25%)
|
||||||
|
#link("https://garagehq.deuxfleurs.fr/")
|
||||||
|
#link("mailto:garagehq@deuxfleurs.fr")
|
||||||
|
`#garage:deuxfleurs.fr` on Matrix
|
||||||
|
|
||||||
|
#v(1.5em)
|
||||||
|
#image("../assets/logos/rust_logo.png", width: 6%)
|
||||||
|
#image("../assets/logos/AGPLv3_Logo.png", width: 13%)
|
||||||
Reference in New Issue
Block a user