Merge branch 'release-0.15.0' into edge.api.updates

This commit is contained in:
Andrew M
2020-06-19 15:08:18 -04:00
12 changed files with 140 additions and 26 deletions
+26 -5
View File
@@ -1,11 +1,19 @@
# Release 0.15.0
Ziti 0.15.0 includes the following:
* [Create fabric mgmt API to request database snapshot/backup be created](https://github.com/openziti/fabric/issues/99)
* [Remove postgres store code including migrations](https://github.com/openziti/edge/issues/195)
* Remove deprecated AppWan and Clusters - These have been replaced by service policies and service edge router policies respectively
* The ability to invoke a database snapshot/backup
* [Create fabric mgmt API to request database snapshot/backup be created](https://github.com/openziti/fabric/issues/99)
* [Add snapshot db REST API](https://github.com/openziti/edge/issues/206)
* Removal of deprecated code/migrations
* [Remove postgres store code including migrations](https://github.com/openziti/edge/issues/195)
* Remove deprecated AppWan and Clusters - These have been replaced by service policies and service edge router policies respectively
* Edge Routers are now a subtype of Fabric Routers
* see [Unverified Edge Routers Cannot Be Used For Terminators](https://github.com/openziti/edge/issues/144)
* Fabric services and routers now have names
* see [Add name to service and router](https://github.com/openziti/fabric/issues/101)
* cosmetic changes to the ziti-enroller binary
* cosmetic changes to the ziti-tunnel binary when running the enroll subcommand
* Memory leak remediation in the `PayloadBuffer` subsystem. Corrects unbounded memory growth in `ziti-router`.
* Edge REST API Enhancements
* [OpenApi 2.0/Swagger](https://github.com/openziti/edge/issues/108)
* [Changes to support Fabric REST API](https://github.com/openziti/edge/issues/101)
@@ -14,11 +22,24 @@ Ziti 0.15.0 includes the following:
The code to migrate a Ziti instance from pre-0.9 releases has been removed. If you want to migrate from a pre-0.9 version you should first update to 0.14.12, then to new versions.
## Database Snapshots
The ziti-fabric tool can now be used to trigger the creation of a dabase backup/snapshot.
Database snapshots can now be triggered in a variety of ways to cause the creation of a dabase backup/snapshot. This can be done from the ziti-fabric CLI, the ziti CLI and the REST API
$ ziti-fabric snapshot-db
$ ziti edge snapshot-db
This will create a copy of the database file within a bolt transaction. The file name will have the data and time appended to it. Snapshotting can be done at most once per minute. Support to trigger this from the Edge REST API will be comming in a follow-up release.
The REST API is available by POSTing to `/edge/v1/database/snapshot`. This ability is only available to administrators.
The snapshot will be a copy of the database file created within a bolt transaction. The file name will have the data and time appended to it. Snapshotting can be done at most once per minute.
## Edge Routers/Fabric Router subtyping
Previously edge routers and fabric routers were closely related, but weren't actually the same entity. When an edge router was created, there was no corresponding fabric router until the edge router had been succesfully enrolled.
Now, edge routers are a type of fabric router. When an edge router is created, it will be visible as a fabric router with no fingerprint. This means that the corresponding router application won't be able to connect until enrollment is complete.
This simplifies some things, including allowing adding terminators to an edge router before enrollment is complete.
## Fabric Router and Service Names
Previously fabric routers and services only had ids, which were assumed to be something user friendly. Now they also have a name. If no name is provided, the id will be used for the name as well. This was done primarily so that we have consistency between the fabric and the edge. Now when viewing a service or router you can be sure to find the label in the same place.
### Edge REST API Enhancements
+6 -1
View File
@@ -37,7 +37,7 @@ listeners:
address: tcp:0.0.0.0:1122
service: ssh
options:
mtu: 768
retransmission: true
# for iperf_tcp (iperf3)
- binding: proxy
@@ -65,6 +65,11 @@ listeners:
randomDrops: true
drop1InN: 5000
dialers:
- binding: transport
options:
retransmission: true
# xctrl_example
#
#example:
+4 -4
View File
@@ -26,9 +26,9 @@ require (
github.com/keybase/go-ps v0.0.0-20190827175125-91aafc93ba19
github.com/michaelquigley/pfxlog v0.0.0-20190813191113-2be43bd0dccc
github.com/openziti/edge v0.14.45
github.com/openziti/fabric v0.11.43
github.com/openziti/foundation v0.11.7
github.com/openziti/sdk-golang v0.13.10
github.com/openziti/fabric v0.11.48
github.com/openziti/foundation v0.11.8
github.com/openziti/sdk-golang v0.13.11
github.com/pborman/uuid v1.2.0
github.com/pkg/errors v0.9.1
github.com/rcrowley/go-metrics v0.0.0-20200313005456-10cdbea86bc0
@@ -42,7 +42,7 @@ require (
github.com/spf13/viper v1.5.0
github.com/stretchr/testify v1.6.1
github.com/urfave/negroni v1.0.0
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b
golang.org/x/net v0.0.0-20200602114024-627f9648deb9
google.golang.org/grpc v1.25.1
gopkg.in/AlecAivazis/survey.v1 v1.8.7
gopkg.in/resty.v1 v1.12.0
+10
View File
@@ -325,11 +325,18 @@ github.com/openziti/edge v0.14.45 h1:P/bUFsDVMj3Eh9Rbqk1O9EjxJz51iWAW51LFr/jGnxI
github.com/openziti/edge v0.14.45/go.mod h1:BMfLnzTjmdsZCudteTDEuZ2zO0Jv4oh5qBCwTssNo2o=
github.com/openziti/fabric v0.11.43 h1:+EEo1xYossvXf7LDLcnw3cJxaZwh+k8ez/Ew9qBUd9s=
github.com/openziti/fabric v0.11.43/go.mod h1:va9H0K20ugtCsMj8/1vdRAWsWlFdy1T+OEdXm0e1LW4=
github.com/openziti/fabric v0.11.48 h1:1mFFeelzAKj+ZAMS4cX64z4WTHULZCyz+h6QpePLteo=
github.com/openziti/fabric v0.11.48/go.mod h1:DHTpBAV096+0d3T65IDNFwd/a+ht3IxqSbMNZbsd+2Q=
github.com/openziti/foundation v0.11.5 h1:RmhFzKgH9KdztsF1T/70s+ftPpMGu1X3A6RhmPlOFiY=
github.com/openziti/foundation v0.11.5/go.mod h1:jfnQ8WWFJGrDD2COPtQ/yX+gRlaiMdwUuXbvEBOpnLc=
github.com/openziti/foundation v0.11.7 h1:Eb7jHhRsEo9rorwqUQce5SR/NCSqTmlMyVHd0JiGEb8=
github.com/openziti/foundation v0.11.7/go.mod h1:Qdd7eCuTrpRwEeDbvYy89sFbEY4+J7OuX8UygxNBsvY=
github.com/openziti/foundation v0.11.8 h1:fcJzxdtU1URLwPB6DHRLlod+by4TMXYmMPWxOKu2vEw=
github.com/openziti/foundation v0.11.8/go.mod h1:Qdd7eCuTrpRwEeDbvYy89sFbEY4+J7OuX8UygxNBsvY=
github.com/openziti/sdk-golang v0.13.10 h1:iLYvWVNvU2rMUITRXLiriwRnSNngioROVrjewv7QQZQ=
github.com/openziti/sdk-golang v0.13.10/go.mod h1:jXBekQvK4B9+w/JwN0uE9AAorFuqbph36uFGZHSmwkw=
github.com/openziti/sdk-golang v0.13.11 h1:VlaW8H2dOeq4LwZtc5VFObjcI9YJW+rNNVA43LYKAec=
github.com/openziti/sdk-golang v0.13.11/go.mod h1:CQFpRGd/+x/i5uHdnXaY7V1Eusb6siCmZcF7vVhUXKo=
github.com/orcaman/concurrent-map v0.0.0-20190826125027-8c72a8bb44f6 h1:lNCW6THrCKBiJBpz8kbVGjC7MgdCGKwuvBgc7LoD6sw=
github.com/orcaman/concurrent-map v0.0.0-20190826125027-8c72a8bb44f6/go.mod h1:Lu3tH6HLW3feq74c2GC+jIMS/K2CFcDWnWD9XkenwhI=
github.com/pborman/uuid v1.2.0 h1:J7Q5mO4ysT1dv8hyrUGHb9+ooztCXu1D8MY8DZYsu3g=
@@ -489,6 +496,8 @@ golang.org/x/net v0.0.0-20191007182048-72f939374954/go.mod h1:z5CRVTTTmAJ677TzLL
golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b h1:0mm1VjtFUOIlE1SbDlwjYaDxZVDP2S5ou6y0gSgXHu8=
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200602114024-627f9648deb9 h1:pNX+40auqi2JqRfOP1akLGtYcn15TUbkhwuCO3foqqM=
golang.org/x/net v0.0.0-20200602114024-627f9648deb9/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -522,6 +531,7 @@ golang.org/x/sys v0.0.0-20190924154521-2837fb4f24fe/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20191008105621-543471e840be/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5 h1:LfCXLvNmTYH9kEmVgqbnsWfruoXZIrh4YBgqVHtDvw0=
golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200615200032-f1bc736245b1 h1:ogLJMz+qpzav7lGMh10LMvAkM/fAoGlaiiHYiFYdm80=
golang.org/x/sys v0.0.0-20200615200032-f1bc736245b1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg=
+7 -3
View File
@@ -17,18 +17,21 @@
package subcmd
import (
"github.com/openziti/foundation/channel2"
"github.com/openziti/fabric/pb/mgmt_pb"
"github.com/openziti/foundation/identity/certtools"
"crypto/sha1"
"errors"
"fmt"
"github.com/golang/protobuf/proto"
"github.com/openziti/fabric/pb/mgmt_pb"
"github.com/openziti/foundation/channel2"
"github.com/openziti/foundation/identity/certtools"
"github.com/spf13/cobra"
"time"
)
var createRouterName string
func init() {
createRouter.Flags().StringVar(&createServiceName, "name", "", "Router name. If not provided defaults to the router ID")
createRouterClient = NewMgmtClient(createRouter)
createCmd.AddCommand(createRouter)
}
@@ -43,6 +46,7 @@ var createRouter = &cobra.Command{
request := &mgmt_pb.CreateRouterRequest{
Router: &mgmt_pb.Router{
Id: cert[0].Subject.CommonName,
Name: createRouterName,
Fingerprint: fmt.Sprintf("%x", sha1.Sum(cert[0].Raw)),
},
}
+3
View File
@@ -27,8 +27,10 @@ import (
var createServiceClient *mgmtClient
var createServiceTerminatorStrategy string
var createServiceName string
func init() {
createService.Flags().StringVar(&createServiceName, "name", "", "Service name. If not provided defaults to the ID")
createService.Flags().StringVar(&createServiceTerminatorStrategy, "terminator-strategy", "", "Terminator strategy for service")
createServiceClient = NewMgmtClient(createService)
createCmd.AddCommand(createService)
@@ -43,6 +45,7 @@ var createService = &cobra.Command{
request := &mgmt_pb.CreateServiceRequest{
Service: &mgmt_pb.Service{
Id: args[0],
Name: createServiceName,
TerminatorStrategy: createServiceTerminatorStrategy,
},
}
+3 -3
View File
@@ -52,10 +52,10 @@ var getService = &cobra.Command{
response := &mgmt_pb.GetServiceResponse{}
err := proto.Unmarshal(responseMsg.Body, response)
if err == nil {
fmt.Printf("\n%10s | %30s\n", "Id", "Terminator Strategy")
fmt.Printf("-----------+--------------------------------+----------\n")
fmt.Printf("\n%10s | %30s | %30s\n", "Id", "Name", "Terminator Strategy")
fmt.Printf("-----------+--------------------------------+------------------------------\n")
svc := response.Service
fmt.Printf("%10s | %30s\n\n", svc.Id, svc.TerminatorStrategy)
fmt.Printf("%10s | %30s | %30s\n\n", svc.Id, svc.Name, svc.TerminatorStrategy)
fmt.Printf("Terminators (%v)\n", len(svc.Terminators))
fmt.Printf("\n%10s | %-12s| %v\n", "Id", "Binding", "Destination")
for _, terminator := range svc.Terminators {
+4 -4
View File
@@ -17,11 +17,11 @@
package subcmd
import (
"github.com/openziti/foundation/channel2"
"github.com/openziti/fabric/pb/mgmt_pb"
"errors"
"fmt"
"github.com/golang/protobuf/proto"
"github.com/openziti/fabric/pb/mgmt_pb"
"github.com/openziti/foundation/channel2"
"github.com/spf13/cobra"
"time"
)
@@ -53,7 +53,7 @@ var listRouters = &cobra.Command{
err := proto.Unmarshal(responseMsg.Body, response)
if err == nil {
out := fmt.Sprintf("\nRouters: (%d)\n\n", len(response.Routers))
out += fmt.Sprintf("%-12s | %-40s | %s\n", "Id", "Fingerprint", "Status")
out += fmt.Sprintf("%-12s | %-30s | %-40s | %s\n", "Id", "Name", "Fingerprint", "Status")
for _, r := range response.Routers {
status := ""
if r.Connected {
@@ -62,7 +62,7 @@ var listRouters = &cobra.Command{
if r.ListenerAddress != "" {
status += " (" + r.ListenerAddress + ")"
}
out += fmt.Sprintf("%-12s | %-40s | %s\n", r.Id, r.Fingerprint, status)
out += fmt.Sprintf("%-12s | %-30s | %-40s | %s\n", r.Id, r.Name, r.Fingerprint, status)
}
out += "\n"
fmt.Print(out)
+3 -3
View File
@@ -50,17 +50,17 @@ var listServices = &cobra.Command{
response := &mgmt_pb.ListServicesResponse{}
if err := proto.Unmarshal(responseMsg.Body, response); err == nil {
out := fmt.Sprintf("\nServices: (%d)\n\n", len(response.Services))
out += fmt.Sprintf("%-12s | %-12s | %s\n", "Id", "Terminator Strategy", "Destination(s)")
out += fmt.Sprintf("%-12s | %-12s | %-12s | %s\n", "Id", "Name", "Terminator Strategy", "Destination(s)")
for _, svc := range response.Services {
if len(svc.Terminators) > 0 {
out += fmt.Sprintf("%-12s | %-12s | %s\n", svc.Id, svc.TerminatorStrategy,
out += fmt.Sprintf("%-12s | %-12s | %-12s | %s\n", svc.Id, svc.Name, svc.TerminatorStrategy,
fmt.Sprintf("%-12s -> %s", svc.Terminators[0].RouterId, svc.Terminators[0].Address))
for _, terminator := range svc.Terminators[1:] {
out += fmt.Sprintf("%-12s | %-12s | %s\n", "", "",
fmt.Sprintf("%-12s -> %s", terminator.RouterId, terminator.Address))
}
} else {
out += fmt.Sprintf("%-12s | %-12s | \n", svc.Id, svc.TerminatorStrategy)
out += fmt.Sprintf("%-12s | %-12s | %-12s \n", svc.Id, svc.Name, svc.TerminatorStrategy)
}
}
out += "\n"
@@ -0,0 +1,65 @@
/*
Copyright NetFoundry, Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package edge_controller
import (
"github.com/openziti/ziti/ziti/cmd/ziti/cmd/common"
cmdutil "github.com/openziti/ziti/ziti/cmd/ziti/cmd/factory"
cmdhelper "github.com/openziti/ziti/ziti/cmd/ziti/cmd/helpers"
"github.com/openziti/ziti/ziti/cmd/ziti/util"
"github.com/spf13/cobra"
"io"
"net/http"
)
type snapshotDbOptions struct {
commonOptions
}
// newSnapshotDbCmd creates the 'edge controller create service-policy' command
func newSnapshotDbCmd(f cmdutil.Factory, out io.Writer, errOut io.Writer) *cobra.Command {
options := &snapshotDbOptions{
commonOptions: commonOptions{
CommonOptions: common.CommonOptions{Factory: f, Out: out, Err: errOut},
},
}
cmd := &cobra.Command{
Use: "snapshot-db",
Short: "creates a database snapshot",
Args: cobra.ExactArgs(0),
Run: func(cmd *cobra.Command, args []string) {
options.Cmd = cmd
options.Args = args
err := runSnapshotDb(options)
cmdhelper.CheckErr(err)
},
SuggestFor: []string{},
}
// allow interspersing positional args and flags
cmd.Flags().SetInterspersed(true)
cmd.Flags().BoolVarP(&options.OutputJSONResponse, "output-json", "j", false, "Output the full JSON response from the Ziti Edge Controller")
return cmd
}
// runSnapshotDb create a new config on the Ziti Edge Controller
func runSnapshotDb(o *snapshotDbOptions) error {
_, err := util.EdgeControllerUpdate("database/snapshot", "", o.Out, http.MethodPost, false)
return err
}
@@ -57,6 +57,7 @@ func populateEdgeCommands(f cmdutil.Factory, out io.Writer, errOut io.Writer, cm
cmd.AddCommand(newVersionCmd(f, out, errOut))
cmd.AddCommand(newPolicyAdivsorCmd(f, out, errOut))
cmd.AddCommand(newVerifyCmd(f, out, errOut))
cmd.AddCommand(newSnapshotDbCmd(f, out, errOut))
return cmd
}
+8 -3
View File
@@ -609,6 +609,10 @@ func EdgeControllerUpdate(entityType string, body string, out io.Writer, method
outputJson(out, resp.Body())
}
if len(resp.Body()) == 0 {
return nil, nil
}
jsonParsed, err := gabs.ParseJSON(resp.Body())
if err != nil {
@@ -685,6 +689,10 @@ func EdgeControllerRequest(entityType string, out io.Writer, logJSON bool, doReq
outputJson(out, resp.Body())
}
if resp.Body() == nil {
return nil, nil
}
jsonParsed, err := gabs.ParseJSON(resp.Body())
if err != nil {
@@ -692,7 +700,4 @@ func EdgeControllerRequest(entityType string, out io.Writer, logJSON bool, doReq
}
return jsonParsed, nil
}
//util.EdgeControllerRequest(session.Host, session.Cert, session.Token, entityType, options.Out, options.OutputJSONResponse, func(request *resty.Request) error {