gsadmin/ziti
1
0
Fork 0
mirror of https://github.com/openziti/ziti.git synced 2026-10-12 15:28:05 +00:00
Code Issues Packages Projects Releases Wiki Activity

Tweaks ha docs based on feedback.

Browse Source
This commit is contained in:
Paul Lorenz
2023-11-20 16:41:15 -05:00
parent 2571929eb4
commit 67478fcbd8
3 changed files with 171 additions and 142 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/ha/dev-setup-ha-pki.md
+69
View File
@@ -0,0 +1,69 @@
# Overview
This guide walks you through creating the certificates necessary to run a three node HA cluster.
**NOTE**: This folder contains a script `create-pki.sh` which will perform the steps outlined in
this guide for you.
## Create a CA
Create a self-signed certificate authority (CA) for the trust-root of your cluster
```
ziti pki create ca --trust-domain ha.test --pki-root ./pki --ca-file ca --ca-name 'HA Example Trust Root'
```
## Create Controller Certs
We are going to create an intermediate CA for each controller. We'll use this intermediate CA for
the following purposes:
1. To create a cert which will represent the controller. It will be used
1. On the client side when dialing other controllers in the cluster
2. On the server side when receiving connections from other controllers
3. On the server side when receiving connections from routers
4. On the server when handling REST API requests
2. To create identity certs as part of the identity enrollment process
3. To create router certs as part of the router enrollment process
### Notes
#### Client vs Server Certs
You may use separate certs and keys for client and server connections, but it's not necessary. When
you use a server cert on the client side it exposes information about what IPs and DNS entries the
cert is valid for, but since we're only connecting to other controllers, this should not be a
concern. However, the option to use separate certs is available, should you wish to use it.
#### REST Endpoint Certs
You may also use a different set of certs for the REST endpoint.
#### Sharing Signing Certs
You could use the same signing cert for all controllers in a cluster. However, if a signing cert is
ever compromised, all certs signed by the signing cert would need to be revoked. By using a separate
cert for each controller we limit the fallout from an individual controller or cert being
compromised.
### Create the Controller 1 signing and server certs
```shell
# Create the controller 1 intermediate/signing cert
ziti pki create intermediate --pki-root ./pki --ca-name ca --intermediate-file ctrl1 --intermediate-name 'Controller One Signing Cert'
# Create the controller 1 server cert
ziti pki create server --pki-root ./pki --ca-name ctrl1 --dns localhost --ip 127.0.0.1 --server-name ctrl1 --spiffe-id 'controller/ctrl1'
# Create the controller 2 intermediate/signing cert
ziti pki create intermediate --pki-root ./pki --ca-name ca --intermediate-file ctrl2 --intermediate-name 'Controller Two Signing Cert'
# Create the controller 2 server cert
ziti pki create server --pki-root ./pki --ca-name ctrl2 --dns localhost --ip 127.0.0.1 --server-name ctrl2 --spiffe-id 'controller/ctrl2'
# Create the controller 3 intermediate/signing cert
ziti pki create intermediate --pki-root ./pki --ca-name ca --intermediate-file ctrl3 --intermediate-name 'Controller Three Signing Cert'
# Create the controller 3 server cert
ziti pki create server --pki-root ./pki --ca-name ctrl3 --dns localhost --ip 127.0.0.1 --server-name ctrl3 --spiffe-id 'controller/ctrl3'
```
doc/ha/dev-setup.md
+18 -65
View File
@@ -1,81 +1,26 @@
# HA Setup for Development
**NOTE: HA is a work in progress and not yet usable for anything other than developing HA**
**NOTE: HA is in alpha. Expect bugs. Bug reports are appreciated**
To set up a local three node HA cluster, do the following.
## Create a CA
## Create The Necessary PKI
Create a self-signed certificate authority (CA) for the trust-root of your cluster
```
ziti pki create ca --trust-domain ha.test --pki-root ./pki --ca-file ca --ca-name 'HA Example Trust Root'
```
## Create Controller Certs
We are going to create an intermediate CA for each controller. We'll use this intermediate CA
for the following purposes:
1. To create a cert which will represent the controller. It will be used
1. On the client side when dialing other controllers in the cluster
2. On the server side when receiving connections from other controllers
3. On the server side when receiving connections from routers
4. On the server when handling REST API requests
2. To create identity certs as part of the identity enrollment process
3. To create router certs as part of the router enrollment process
### Notes
#### Client vs Server Certs
You may use separate certs and keys for client and server connections, but it's not necessary.
When you use a server cert on the client side it exposes information about what IPs and DNS entries
the cert is valid for, but since we're only connecting to other controllers, this should not be
a concern. However, the option to use separate certs is available, should you wish to use it.
#### REST Endpoint Certs
You may also use a different set of certs for the REST endpoint.
#### Sharing Signing Certs
You could use the same signing cert for all controllers in a cluster. However, if a signing
cert is ever compromised, all certs signed by the signing cert would need to be revoked. By
using a separate cert for each controller we limit the fallout from an individual controller
or cert being compromised.
### Create the Controller 1 signing and server certs
```shell
# Create the controller 1 intermediate/signing cert
ziti pki create intermediate --pki-root ./pki --ca-name ca --intermediate-file ctrl1 --intermediate-name 'Controller One Signing Cert'
# Create the controller 1 server cert
ziti pki create server --pki-root ./pki --ca-name ctrl1 --dns localhost --ip 127.0.0.1 --server-name ctrl1 --spiffe-id 'controller/ctrl1'
# Create the controller 2 intermediate/signing cert
ziti pki create intermediate --pki-root ./pki --ca-name ca --intermediate-file ctrl2 --intermediate-name 'Controller Two Signing Cert'
# Create the controller 2 server cert
ziti pki create server --pki-root ./pki --ca-name ctrl2 --dns localhost --ip 127.0.0.1 --server-name ctrl2 --spiffe-id 'controller/ctrl2'
# Create the controller 3 intermediate/signing cert
ziti pki create intermediate --pki-root ./pki --ca-name ca --intermediate-file ctrl3 --intermediate-name 'Controller Three Signing Cert'
# Create the controller 3 server cert
ziti pki create server --pki-root ./pki --ca-name ctrl3 --dns localhost --ip 127.0.0.1 --server-name ctrl3 --spiffe-id 'controller/ctrl3'
```
Either run the `create-pki.sh` script found in the folder, or follow the steps in
the [HA PKI Guide](./dev-setup-ha-pki.md)
## Running the Controllers
1. The controller configuration files have relative paths, so make sure you're running things from this directory.
1. The controller configuration files have relative paths, so make sure you're running things from
this directory.
2. Start all three controllers
1. `ziti controller run ctrl1.yml`
2. `ziti controller run ctrl2.yml`
3. `ziti controller run ctrl3.yml`
4. All three are configured with `minClusterSize` of 3, so they will wait to be joined to a raft cluster
5. The ctrl1.yml config file has the other two controllers as bootstrap members, so when it starts the first controller will start trying form the raft cluster.
4. All three are configured with `minClusterSize` of 3, so they will wait to be joined to a raft
cluster
5. The ctrl1.yml config file has the other two controllers as bootstrap members, so when it
starts the first controller will start trying form the raft cluster.
3. Initialize the edge using the agent
1. `ziti agent controller init <pid of any controller> admin admin 'Default Admin'`
2. You can of course use different values if you desire
@@ -89,14 +34,22 @@ You should now have a three node cluster running. You can log into each controll
You could then create some model data on any controller:
```
# This will create the client side identity and policies
ziti demo setup echo client
# This will create the server side identity and policies
ziti demo setup echo single-sdk-hosted
```
Any view the results on any controller
```
ziti edge login localhost:1280
ziti edge ls services
ziti edge login -i ctrl2 localhost:1380
ziti edge -i ctrl2 ls services
ziti edge login -i ctrl3 localhost:1480
ziti edge -i ctrl3 ls services
```
doc/ha/overview.md
+84 -77
View File
@@ -3,13 +3,15 @@
This document gives a brief overview of how OpenZiti HA works and how it differs from running
OpenZiti in non-HA mode.
To set up a developer HA network see the [HA Developer Setup Guide](./dev-setup.md).
## Operational Considerations
### System of Record
In controller that's not configured for HA, the bolt database is the system of record. In
an HA setup, the raft journal is the system of record. The raft journal is stored in two places,
a snapshot directory and a bolt database of raft journal entries.
In controller that's not configured for HA, the bolt database is the system of record. In an HA
setup, the raft journal is the system of record. The raft journal is stored in two places, a
snapshot directory and a bolt database of raft journal entries.
So a non-HA setup will have:
@@ -18,7 +20,8 @@ So a non-HA setup will have:
An HA setup will have:
* raft.db - the bolt database containing raft journal entries
* snapshots/ - a directory containing raft snapshots. Each snapshot is snapshot of the controller bolt db
* snapshots/ - a directory containing raft snapshots. Each snapshot is snapshot of the controller
bolt db
* ctrl.db - the controller bolt db, with the current state of the model
The location of all three is controlled by the raft/dataDir config property.
@@ -28,9 +31,9 @@ raft:
dataDir: /var/ziti/data/
```
When an HA controller starts up, it will first apply the newest snapshot, then any newer journal entries
that aren't yet contained in a snapshot. This means that an HA controller should start with a
blank DB that can be overwritten by snapshot and/or have journal entries applied to it. So an HA
When an HA controller starts up, it will first apply the newest snapshot, then any newer journal
entries that aren't yet contained in a snapshot. This means that an HA controller should start with
a blank DB that can be overwritten by snapshot and/or have journal entries applied to it. So an HA
controller will delete or rename the existing controller database and start with a fresh bolt db.
### Bootstrapping
@@ -39,9 +42,9 @@ controller will delete or rename the existing controller database and start with
Initial cluster setup can be configured either in the config file or via agent commands.
The controller will not fully start until the raft cluster has bootstrapped. The minimum
number of cluster members is set in the config file. Recommended cluster sizes are
3, 5 or 7. A cluster size of 1 is mostly useful for testing and development.
The controller will not fully start until the raft cluster has bootstrapped. The minimum number of
cluster members is set in the config file. Recommended cluster sizes are 3, 5 or 7. A cluster size
of 1 is mostly useful for testing and development.
**Config File Example**
@@ -54,9 +57,9 @@ raft:
- tls:192.168.1.101
```
Note that `bootstrapMembers` can only be used when setting the cluster up for the first time
and should only be used on a single node. It cannot be used later to add additional nodes
to an existing cluster.
Note that `bootstrapMembers` can only be used when setting the cluster up for the first time and
should only be used on a single node. It cannot be used later to add additional nodes to an existing
cluster.
**Agent Comands**
@@ -80,11 +83,10 @@ ziti agent cluster transfer-leadership [new leader id]
#### Edge Admin Initialization
Because RAFT is now the system of record, the previous pattern for configuring the default
admin won't work. In a non-HA system you initialized the controller raft DB with an admin
user directly. If you do this with an HA system, the changes you made directly to the DB
will be lost and replaced by whatever is in raft. To initialize an HA controller cluster
use the new agent command.
Because RAFT is now the system of record, the previous pattern for configuring the default admin
won't work. In a non-HA system you initialized the controller raft DB with an admin user directly.
If you do this with an HA system, the changes you made directly to the DB will be lost and replaced
by whatever is in raft. To initialize an HA controller cluster use the new agent command.
```shell
ziti agent controller init <admin username> <admin password> <admin name>
@@ -94,36 +96,36 @@ The controller will not fully start until the edge admin has been initialized.
### Snapshot Application and Restarts
If a controller receives a snapshot to apply after starting up, it will apply the snapshot and
then terminate. This assumes that there is a restart script which will bring the controller
back up after it terminates.
If a controller receives a snapshot to apply after starting up, it will apply the snapshot and then
terminate. This assumes that there is a restart script which will bring the controller back up after
it terminates.
This should only happen if a controller is connected to the cluster and then gets disconnected
for long enough that a snapshot is created while it's disconnected. Because applying a snapshot
requires replacing the underlying controller bolt DB, the easiest way to do that is restart.
That way we don't have to worry about replacing the bolt DB underneath a running system.
This should only happen if a controller is connected to the cluster and then gets disconnected for
long enough that a snapshot is created while it's disconnected. Because applying a snapshot requires
replacing the underlying controller bolt DB, the easiest way to do that is restart. That way we
don't have to worry about replacing the bolt DB underneath a running system.
### Metrics
In an HA system, routers will send metrics to all controllers to which they are connected.
There is a new `doNotPropagate` flag in the metrics message, which will be set to false
until the router has successfully delivered the metrics message to a controller. The flag
will then be set to true. So the first controller to get the metrics message is expected
to deliver the metrics message to the events system for external integrators. The other
controllers will have `doNotPropage` set to true, and will only use the metrics message
internally, to update routing data.
In an HA system, routers will send metrics to all controllers to which they are connected. There is
a new `doNotPropagate` flag in the metrics message, which will be set to false until the router has
successfully delivered the metrics message to a controller. The flag will then be set to true. So
the first controller to get the metrics message is expected to deliver the metrics message to the
events system for external integrators. The other controllers will have `doNotPropage` set to true,
and will only use the metrics message internally, to update routing data.
### Certificates
There are many ways to set up certificates, so this will just cover a recommended configuration.
The primary thing to ensure is that controllers have a shared root of trust. A configuration
that works would be as follows:
The primary thing to ensure is that controllers have a shared root of trust. A configuration that
works would be as follows:
1. Create a self-signed root CA
2. Create an intermediate signing cert for each controller
3. Create a server cert using the signing cert for each controller
4. Make sure that the CA bundle for each server includes both the root CA and the intermediate CA for that server
4. Make sure that the CA bundle for each server includes both the root CA and the intermediate CA
for that server
Note that controller server certs must contain a SPIFFE id of the form
@@ -140,7 +142,8 @@ spiffe://example.com/controller/ctrl1
**SPIFFE ID Notes:**
* This ID must be set as the only URI in the `X509v3 Subject Alternative Name` field in the certificate.
* This ID must be set as the only URI in the `X509v3 Subject Alternative Name` field in the
certificate.
* These IDs are used to allow the controllers to identify each during the mTLS negotiation.
* The OpenZiti CLI supports creating SPIFFE IDs in your certs
* Use the `--trust-domain` flag when creating CAs
@@ -158,19 +161,18 @@ Controllers now establish connections with each other, for two purposes.
Both kinds of traffic flow over the same connection.
These connections do not require any extra open ports as we are using the control
channel listener to listen to both router and controller connections. As part of
the connection process the connection type is provided and the appropriate
authentication and connection setup happens based on the connection type. If no
connection type is provided, it's assumed to be a router.
These connections do not require any extra open ports as we are using the control channel listener
to listen to both router and controller connections. As part of the connection process the
connection type is provided and the appropriate authentication and connection setup happens based on
the connection type. If no connection type is provided, it's assumed to be a router.
## Distributed Model
When looking at how to make the OpenZiti controller model distributed, we first looked at
what characteristics we needed for the model data.
When looking at how to make the OpenZiti controller model distributed, we first looked at what
characteristics we needed for the model data.
Model data is the information the controller needs to figure out what it can do. This
includes things like:
Model data is the information the controller needs to figure out what it can do. This includes
things like:
* Services
* Routers
@@ -186,12 +188,14 @@ includes things like:
* Reads happen all the time, from every client and as well as admins
* Speed is very important. They affect how every client perceives the system.
* Availability is very important. Without reading definitions, can’t create new connections
* Can be against stale data, if we get consistency within a reasonable timeframe (seconds to minutes)
* Can be against stale data, if we get consistency within a reasonable timeframe (seconds to
minutes)
* Write characteristics
* Writes only happen from administrators
* Speed needs to be reasonable, but doesn't need to be blazing fast
* Write availability can be interrupted, since it primarily affects management operations
* Must be consistent. Write validation can’t happen with stale data. Don’t want to have to deal with reconciling concurrent, contradictory write operations.
* Must be consistent. Write validation can’t happen with stale data. Don’t want to have to deal
with reconciling concurrent, contradictory write operations.
* Generally involves controller to controller coordination
Of the distribution mechanisms we looked at, RAFT had the best fit.
@@ -217,23 +221,27 @@ So the OpenZiti controller uses RAFT to distribute the data model. Specifically
The basic flow for model updates is as follows:
1. A client requests a model update via the REST API.
2. The controller checks if it is the raft cluster leader. If it is not, it forwards the request to the leader.
3. Once the request is on the leader, it applies the model update to the raft log. This involves getting a quorum of the controllers to accept the update.
4. One the update has been accepted, it will be executed on each node of the cluster. This will generate create one or more changes to the bolt database.
5. The results of the operation (success or failure) are returned to the controller which received the original REST request.
2. The controller checks if it is the raft cluster leader. If it is not, it forwards the request to
the leader.
3. Once the request is on the leader, it applies the model update to the raft log. This involves
getting a quorum of the controllers to accept the update.
4. One the update has been accepted, it will be executed on each node of the cluster. This will
generate create one or more changes to the bolt database.
5. The results of the operation (success or failure) are returned to the controller which received
the original REST request.
6. The controller waits until the operation has been applied locally.
7. The result is returned to the REST client.
### Reads
Reads are always done to the local bolt database for performance. The assumption is that if something
like a policy change is delayed, it may temporarily allow a circuit to be created, but as soon as
the policy update is applied, it will make changes to circuits as necessary.
Reads are always done to the local bolt database for performance. The assumption is that if
something like a policy change is delayed, it may temporarily allow a circuit to be created, but as
soon as the policy update is applied, it will make changes to circuits as necessary.
## Runtime Data
In addition to model data, the controller also manages some amount of runtime data. This data is
for running OpenZiti's core functions, i.e. managing the flow of data across the mesh, along with
In addition to model data, the controller also manages some amount of runtime data. This data is for
running OpenZiti's core functions, i.e. managing the flow of data across the mesh, along with
related authentication data. So this includes things like:
* Links
@@ -250,37 +258,36 @@ Runtime data has different characteristics than the model data does.
* Reads **and** writes must be very fast
* Generally involves sdk to controller or controller to router coordination
Because writes must also be fast, RAFT is not a good candidate for storing this data.
Good performance is critical for these components, so they are each evaluated individually.
Because writes must also be fast, RAFT is not a good candidate for storing this data. Good
performance is critical for these components, so they are each evaluated individually.
### Links
Each controller currently needs to know about links so that it can make routing decisions.
However, links exist on routers. So, routers are the source of record for links.
When a router connects to a controller, the router will tell the controller about any
links that it already has. The controller will ask to fill in any missing links and
the controller will ensure that it doesn't create duplicate links if multiple
controllers request the same link be created. If there are duplicates, the router
will inform the controller of the existing link.
Each controller currently needs to know about links so that it can make routing decisions. However,
links exist on routers. So, routers are the source of record for links. When a router connects to a
controller, the router will tell the controller about any links that it already has. The controller
will ask to fill in any missing links and the controller will ensure that it doesn't create
duplicate links if multiple controllers request the same link be created. If there are duplicates,
the router will inform the controller of the existing link.
The allows the routers to properly handle link dials from multiple routers and keep
controllers up to date with the current known links.
The allows the routers to properly handle link dials from multiple routers and keep controllers up
to date with the current known links.
### Circuits
Circuits were and continue to be stored in memory for both standalone and HA mode
controllers.Circuits are not distributed. Rather, each controller remains responsible
for any circuits that it created.
controllers.Circuits are not distributed. Rather, each controller remains responsible for any
circuits that it created.
When a router needs to initiate circuit creation it will pick the one with the lowest
response time and send a circuit creation request to that router. The controller
will establish a route. Route tables as well as the xgress endpoints now track
which controller is responsible for the associated circuit. This way when failures
or other notifications need to be sent, the router knows which controller to talk to.
When a router needs to initiate circuit creation it will pick the one with the lowest response time
and send a circuit creation request to that router. The controller will establish a route. Route
tables as well as the xgress endpoints now track which controller is responsible for the associated
circuit. This way when failures or other notifications need to be sent, the router knows which
controller to talk to.
This gets routing working with multiple controllers without a major refactor. Future
work will likely delegate more routing control to the routers, so routing should
get more robust and distributed over time.
This gets routing working with multiple controllers without a major refactor. Future work will
likely delegate more routing control to the routers, so routing should get more robust and
distributed over time.
### Api Sessions, Sessions, Posture Data