|
|
|
@@ -2,11 +2,46 @@
|
|
|
|
|
|
|
|
|
|
To run Garage in cluster mode, we recommend having at least 3 nodes.
|
|
|
|
|
This will allow you to setup Garage for three-way replication of your data,
|
|
|
|
|
the safest and most available mode avaialble.
|
|
|
|
|
the safest and most available mode proposed by Garage.
|
|
|
|
|
|
|
|
|
|
We recommend first following the [quick start guide](../quick_start/index.md) in order
|
|
|
|
|
to get familiar with Garage's command line and usage patterns.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Prerequisites
|
|
|
|
|
|
|
|
|
|
To run a real-world deployment, make sure you the following conditions are met:
|
|
|
|
|
|
|
|
|
|
- You have at least three machines with sufficient storage space available.
|
|
|
|
|
|
|
|
|
|
- Each machine has a public IP address which is reachable by other machines.
|
|
|
|
|
Running behind a NAT is possible, but having several Garage nodes behind a single NAT
|
|
|
|
|
is slightly more involved as each will have to have a different RPC port number
|
|
|
|
|
(the local port number of a node must be the same as the port number exposed publicly
|
|
|
|
|
by the NAT).
|
|
|
|
|
|
|
|
|
|
- Ideally, each machine should have a SSD available in addition to the HDD you are dedicating
|
|
|
|
|
to Garage. This will allow for faster access to metadata and has the potential
|
|
|
|
|
to drastically reduce Garage's response times.
|
|
|
|
|
|
|
|
|
|
- This guide will assume you are using Docker containers to deploy Garage on each node.
|
|
|
|
|
Garage can also be run independently, for instance as a [Systemd service](systemd.md).
|
|
|
|
|
You can also use an orchestrator such as Nomad or Kubernetes to automatically manage
|
|
|
|
|
Docker containers on a fleet of nodes.
|
|
|
|
|
|
|
|
|
|
Before deploying Garage on your infrastructure, you must inventory your machines.
|
|
|
|
|
For our example, we will suppose the following infrastructure with IPv6 connectivity:
|
|
|
|
|
|
|
|
|
|
| Location | Name | IP Address | Disk Space |
|
|
|
|
|
|----------|---------|------------|------------|
|
|
|
|
|
| Paris | Mercury | fc00:1::1 | 1 To |
|
|
|
|
|
| Paris | Venus | fc00:1::2 | 2 To |
|
|
|
|
|
| London | Earth | fc00:B::1 | 2 To |
|
|
|
|
|
| Brussels | Mars | fc00:F::1 | 1.5 To |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Get a Docker image
|
|
|
|
|
|
|
|
|
|
Our docker image is currently named `lxpz/garage_amd64` and is stored on the [Docker Hub](https://hub.docker.com/r/lxpz/garage_amd64/tags?page=1&ordering=last_updated).
|
|
|
|
@@ -35,43 +70,26 @@ chmod +x genkeys.sh
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
It will creates a folder named `pki/` containing the keys that you will used for the cluster.
|
|
|
|
|
These files will have to be copied to all of your cluster nodes, as explained below.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Deploying and configuring Garage
|
|
|
|
|
|
|
|
|
|
To run a real-world deployment, make sure you the following conditions are met:
|
|
|
|
|
|
|
|
|
|
- You have at least three machines with sufficient storage space available
|
|
|
|
|
|
|
|
|
|
- Each machine has a public IP address which is reachable by other machines.
|
|
|
|
|
Running behind a NAT is possible, but having several Garage nodes behind a single NAT
|
|
|
|
|
is slightly more involved as each will have to have a different RPC port number
|
|
|
|
|
(the local port number of a node must be the same as the port number exposed publicly
|
|
|
|
|
by the NAT).
|
|
|
|
|
|
|
|
|
|
- Ideally, each machine should have a SSD available in addition to the HDD you are dedicating
|
|
|
|
|
to Garage. This will allow for faster access to metadata and has the potential
|
|
|
|
|
to drastically reduce Garage's response times.
|
|
|
|
|
|
|
|
|
|
Before deploying garage on your infrastructure, you must inventory your machines.
|
|
|
|
|
For our example, we will suppose the following infrastructure with IPv6 connectivity:
|
|
|
|
|
|
|
|
|
|
| Location | Name | IP Address | Disk Space |
|
|
|
|
|
|----------|---------|------------|------------|
|
|
|
|
|
| Paris | Mercury | fc00:1::1 | 1 To |
|
|
|
|
|
| Paris | Venus | fc00:1::2 | 2 To |
|
|
|
|
|
| London | Earth | fc00:B::1 | 2 To |
|
|
|
|
|
| Brussels | Mars | fc00:F::1 | 1.5 To |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
On each machine, we will have a similar setup,
|
|
|
|
|
especially you must consider the following folders/files:
|
|
|
|
|
|
|
|
|
|
- `/etc/garage/garage.toml`: Garage daemon's configuration (see below)
|
|
|
|
|
- `/etc/garage/pki/`: Folder containing Garage certificates, must be generated on your computer and copied on the servers
|
|
|
|
|
- `/var/lib/garage/meta/`: Folder containing Garage's metadata, put this folder on a SSD if possible
|
|
|
|
|
- `/var/lib/garage/data/`: Folder containing Garage's data, this folder will grows and must be on a large storage, possibly big HDDs.
|
|
|
|
|
- `/etc/systemd/system/garage.service`: Service file to start garage at boot automatically (defined below, not required if you use docker)
|
|
|
|
|
- `/etc/garage/garage.toml`: Garage daemon's configuration (see below)
|
|
|
|
|
|
|
|
|
|
- `/etc/garage/pki/`: Folder containing Garage certificates,
|
|
|
|
|
must be generated on your computer and copied on the servers.
|
|
|
|
|
Only the files `garage-ca.crt`, `garage.crt` and `garage.key` are necessary.
|
|
|
|
|
|
|
|
|
|
- `/var/lib/garage/meta/`: Folder containing Garage's metadata,
|
|
|
|
|
put this folder on a SSD if possible
|
|
|
|
|
|
|
|
|
|
- `/var/lib/garage/data/`: Folder containing Garage's data,
|
|
|
|
|
this folder will be your main data storage and must be on a large storage (e.g. large HDD)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
A valid `/etc/garage/garage.toml` for our cluster would be:
|
|
|
|
|
|
|
|
|
@@ -128,14 +146,14 @@ docker run \
|
|
|
|
|
lxpz/garage_amd64:v0.3.0
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
It should be restart automatically at each reboot.
|
|
|
|
|
It should be restarted automatically at each reboot.
|
|
|
|
|
Please note that we use host networking as otherwise Docker containers
|
|
|
|
|
can not communicate with IPv6.
|
|
|
|
|
|
|
|
|
|
Upgrading between Garage versions should be supported transparently,
|
|
|
|
|
but please check the relase notes before doing so!
|
|
|
|
|
To upgrade, simply stop and remove this container and
|
|
|
|
|
start again the command with a new version of garage.
|
|
|
|
|
start again the command with a new version of Garage.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Controling the daemon
|
|
|
|
@@ -166,7 +184,7 @@ You will now have a shell where the Garage binary is available as `/garage/garag
|
|
|
|
|
-h, --rpc-host <rpc-host>
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The 3 first ones are certificates and keys needed by TLS, the last one is simply the address of garage's RPC endpoint.
|
|
|
|
|
The 3 first ones are certificates and keys needed by TLS, the last one is simply the address of Garage's RPC endpoint.
|
|
|
|
|
|
|
|
|
|
If you are invoking `garage` from a server node directly, you do not need to set `--rpc-host`
|
|
|
|
|
as the default value `127.0.0.1:3901` will allow it to contact Garage correctly.
|
|
|
|
@@ -196,18 +214,19 @@ You should get something like that as result:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
Healthy nodes:
|
|
|
|
|
2a638ed6c775b69a… 37f0ba978d27 [::ffff:172.20.0.101]:3901 UNCONFIGURED/REMOVED
|
|
|
|
|
68143d720f20c89d… 9795a2f7abb5 [::ffff:172.20.0.103]:3901 UNCONFIGURED/REMOVED
|
|
|
|
|
8781c50c410a41b3… 758338dde686 [::ffff:172.20.0.102]:3901 UNCONFIGURED/REMOVED
|
|
|
|
|
8781c50c410a41b3… Mercury [fc00:1::1]:3901 UNCONFIGURED/REMOVED
|
|
|
|
|
2a638ed6c775b69a… Venus [fc00:1::2]:3901 UNCONFIGURED/REMOVED
|
|
|
|
|
68143d720f20c89d… Earth [fc00:B::1]:3901 UNCONFIGURED/REMOVED
|
|
|
|
|
212f7572f0c89da9… Mars [fc00:F::1]:3901 UNCONFIGURED/REMOVED
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Configuring a cluster
|
|
|
|
|
|
|
|
|
|
We will now inform garage of the disk space available on each node of the cluster
|
|
|
|
|
We will now inform Garage of the disk space available on each node of the cluster
|
|
|
|
|
as well as the zone (e.g. datacenter) in which each machine is located.
|
|
|
|
|
|
|
|
|
|
For our example, we will suppose we have the following infrastructure (Capacity, Identifier and Datacenter are specific values to garage described in the following):
|
|
|
|
|
For our example, we will suppose we have the following infrastructure (Capacity, Identifier and Datacenter are specific values to Garage described in the following):
|
|
|
|
|
|
|
|
|
|
| Location | Name | Disk Space | `Capacity` | `Identifier` | `Zone` |
|
|
|
|
|
|----------|---------|------------|------------|--------------|--------------|
|
|
|
|
@@ -218,7 +237,7 @@ For our example, we will suppose we have the following infrastructure (Capacity,
|
|
|
|
|
|
|
|
|
|
#### Node identifiers
|
|
|
|
|
|
|
|
|
|
After its first launch, garage generates a random and unique identifier for each nodes, such as:
|
|
|
|
|
After its first launch, Garage generates a random and unique identifier for each nodes, such as:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
8781c50c410a41b363167e9d49cc468b6b9e4449b6577b64f15a249a149bdcbc
|
|
|
|
@@ -233,12 +252,13 @@ The most simple way to match an identifier to a node is to run:
|
|
|
|
|
garagectl status
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
It will display the IP address associated with each node; from the IP address you will be able to recognize the node.
|
|
|
|
|
It will display the IP address associated with each node;
|
|
|
|
|
from the IP address you will be able to recognize the node.
|
|
|
|
|
|
|
|
|
|
#### Zones
|
|
|
|
|
|
|
|
|
|
Zones are simply a user-chosen identifier that identify a group of server that are grouped together logically.
|
|
|
|
|
It is up to the system administrator deploying garage to identify what does "grouped together" means.
|
|
|
|
|
It is up to the system administrator deploying Garage to identify what does "grouped together" means.
|
|
|
|
|
|
|
|
|
|
In most cases, a zone will correspond to a geographical location (i.e. a datacenter).
|
|
|
|
|
Behind the scene, Garage will use zone definition to try to store the same data on different zones,
|
|
|
|
@@ -246,13 +266,15 @@ in order to provide high availability despite failure of a zone.
|
|
|
|
|
|
|
|
|
|
#### Capacity
|
|
|
|
|
|
|
|
|
|
Garage reasons on an arbitrary metric about disk storage that is named the *capacity* of a node.
|
|
|
|
|
Garage reasons on an abstract metric about disk storage that is named the *capacity* of a node.
|
|
|
|
|
The capacity configured in Garage must be proportional to the disk space dedicated to the node.
|
|
|
|
|
Additionaly, the capacity values used in Garage should be as small as possible, with
|
|
|
|
|
1 ideally representing the size of your smallest server.
|
|
|
|
|
Due to the way the Garage allocation algorithm works, capacity values must
|
|
|
|
|
be **integers**, and must be **as small as possible**, for instance with
|
|
|
|
|
1 representing the size of your smallest server.
|
|
|
|
|
|
|
|
|
|
Here we chose that 1 unit of capacity = 0.5 To, so that we can express servers of size
|
|
|
|
|
1 To and 2 To, as wel as the intermediate size 1.5 To.
|
|
|
|
|
1 To and 2 To, as wel as the intermediate size 1.5 To, with the integer values 2, 4 and
|
|
|
|
|
3 respectively (see table above).
|
|
|
|
|
|
|
|
|
|
Note that the amount of data stored by Garage on each server may not be strictly proportional to
|
|
|
|
|
its capacity value, as Garage will priorize having 3 copies of data in different zones,
|
|
|
|
@@ -270,3 +292,14 @@ garagectl node configure -z par1 -c 4 -t venus 2a638e
|
|
|
|
|
garagectl node configure -z lon1 -c 4 -t earth 68143d
|
|
|
|
|
garagectl node configure -z bru1 -c 3 -t mars 212f75
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Using your Garage cluster
|
|
|
|
|
|
|
|
|
|
Creating buckets and managing keys is done using the `garagectl` CLI,
|
|
|
|
|
and is covered in the [quick start guide](../quick_start/index.md).
|
|
|
|
|
Remember also that the CLI is self-documented thanks to the `--help` flag and
|
|
|
|
|
the `help` subcommand (e.g. `garage help`, `garage key --help`).
|
|
|
|
|
|
|
|
|
|
Configuring an S3 client to interact with Garage is covered
|
|
|
|
|
[in the next section](clients.md).
|
|
|
|
|
Alex Auvolat