mirror of
https://github.com/mulgadc/spinifex.git
synced 2026-10-23 04:36:40 +00:00
TODO updated for next ec2 implementation
This commit is contained in:
benduncan
@@ -4,26 +4,26 @@
|
||||
|
||||
Big rocks:
|
||||
|
||||
* Add multi-node support for predastore (S3) using reed solomon encoding
|
||||
* Implement KV store for object lookup, WAL files, offset. Use hash-ring to determine which nodes an object belongs to.
|
||||
* Implement VPC support using Open vSwitch across multiple nodes, core VPC functionality included
|
||||
* Add support for NVIDIA Bluefield DPU with Open vSwitch
|
||||
* Implement basic IAM using NATS Jetstream as KV store, vs IAM/access-keys in local config/TOML files for beta.
|
||||
* Move `daemon.go` instances.json state to Jetstream KV
|
||||
* Add support using the `hive` CLI tool to provision a new user with AWS access-keys/IAM.
|
||||
* Support multi-tenant operations and isolation
|
||||
* Add support to include capabilities when adding a new hardware node to MulgaOS (e.g EC2 target, S3, EBS, NATs, etc) - Features can be turned on/off depending on hardware scope.
|
||||
* Add simple Web UI console, using the AWS JS SDK, communicating to local AWS gateway.
|
||||
* Implement ShadCNblocks for UI framework
|
||||
* Simple Go webserver, static files, easy build process.
|
||||
- Add multi-node support for predastore (S3) using reed solomon encoding
|
||||
- Implement KV store for object lookup, WAL files, offset. Use hash-ring to determine which nodes an object belongs to.
|
||||
- Implement VPC support using Open vSwitch across multiple nodes, core VPC functionality included
|
||||
- Add support for NVIDIA Bluefield DPU with Open vSwitch
|
||||
- Implement basic IAM using NATS Jetstream as KV store, vs IAM/access-keys in local config/TOML files for beta.
|
||||
- Move `daemon.go` instances.json state to Jetstream KV
|
||||
- Add support using the `hive` CLI tool to provision a new user with AWS access-keys/IAM.
|
||||
- Support multi-tenant operations and isolation
|
||||
- Add support to include capabilities when adding a new hardware node to MulgaOS (e.g EC2 target, S3, EBS, NATs, etc) - Features can be turned on/off depending on hardware scope.
|
||||
- Add simple Web UI console, using the AWS JS SDK, communicating to local AWS gateway.
|
||||
- Implement ShadCNblocks for UI framework
|
||||
- Simple Go webserver, static files, easy build process.
|
||||
|
||||
Implementation gaps:
|
||||
|
||||
* EC2 - Support extended features for `run-instance`
|
||||
* Volume resize of AMI
|
||||
* Confirm cloud-init will resize volume, configured correctly.
|
||||
* Attach additional volumes
|
||||
* Attach to VPC / Security group (required Open vSwitch implementation)
|
||||
- EC2 - Support extended features for `run-instance`
|
||||
- Volume resize of AMI
|
||||
- Confirm cloud-init will resize volume, configured correctly.
|
||||
- Attach additional volumes
|
||||
- Attach to VPC / Security group (required Open vSwitch implementation)
|
||||
|
||||
### EC2
|
||||
|
||||
@@ -36,6 +36,32 @@ Implementation gaps:
|
||||
- [DONE] - stop-instances
|
||||
- [DONE] - terminate-instances
|
||||
|
||||
### To implement
|
||||
|
||||
Easier methods to implement
|
||||
|
||||
- attach-volume
|
||||
- copy-image
|
||||
- copy-snapshot
|
||||
- copy-volumes
|
||||
- create-image
|
||||
- create-snapshot
|
||||
- create-snapshots
|
||||
- create-tags
|
||||
- create-volume
|
||||
- delete-snapshot
|
||||
- describe-regions
|
||||
- describe-snapshots
|
||||
- describe-subnets
|
||||
- describe-tags
|
||||
- describe-volume-attribute
|
||||
- describe-volume-status
|
||||
- describe-volumes
|
||||
- detach-network-interface
|
||||
- detach-volume
|
||||
- get-console-output
|
||||
- monitor-instances
|
||||
|
||||
TODO:
|
||||
|
||||
- allocate-address
|
||||
@@ -50,18 +76,13 @@ TODO:
|
||||
- associate-subnet-cidr-block
|
||||
- attach-internet-gateway
|
||||
- attach-network-interface
|
||||
- attach-volume
|
||||
- authorize-security-group-egress
|
||||
- authorize-security-group-ingress
|
||||
- copy-image
|
||||
- copy-snapshot
|
||||
- copy-volumes
|
||||
- create-customer-gateway
|
||||
- create-default-subnet
|
||||
- create-default-vpc
|
||||
- create-dhcp-options
|
||||
- create-egress-only-internet-gateway
|
||||
- create-image
|
||||
- create-internet-gateway
|
||||
- create-launch-template
|
||||
- create-launch-template-version
|
||||
@@ -80,12 +101,8 @@ TODO:
|
||||
- create-route-server-peer
|
||||
- create-route-table
|
||||
- create-security-group
|
||||
- create-snapshot
|
||||
- create-snapshots
|
||||
- create-subnet
|
||||
- create-subnet-cidr-reservation
|
||||
- create-tags
|
||||
- create-volume
|
||||
- delete-coip-cidr
|
||||
- delete-customer-gateway
|
||||
- delete-dhcp-options
|
||||
@@ -107,7 +124,6 @@ TODO:
|
||||
- delete-route-server-peer
|
||||
- delete-route-table
|
||||
- delete-security-group
|
||||
- delete-snapshot
|
||||
- delete-subnet
|
||||
- delete-subnet-cidr-reservation
|
||||
- delete-tags
|
||||
@@ -123,7 +139,6 @@ TODO:
|
||||
- describe-managed-prefix-lists
|
||||
- describe-nat-gateways
|
||||
- describe-network-acls
|
||||
- describe-regions
|
||||
- describe-route-server-endpoints
|
||||
- describe-route-server-peers
|
||||
- describe-route-servers
|
||||
@@ -131,16 +146,8 @@ TODO:
|
||||
- describe-security-group-references
|
||||
- describe-security-group-rules
|
||||
- describe-security-groups
|
||||
- describe-snapshots
|
||||
- describe-subnets
|
||||
- describe-tags
|
||||
- describe-volume-attribute
|
||||
- describe-volume-status
|
||||
- describe-volumes
|
||||
- describe-volumes-modifications
|
||||
- detach-internet-gateway
|
||||
- detach-network-interface
|
||||
- detach-volume
|
||||
- detach-vpn-gateway
|
||||
- disable-image
|
||||
- disable-route-server-propagation
|
||||
@@ -151,7 +158,6 @@ TODO:
|
||||
- enable-image
|
||||
- enable-image-block-public-access
|
||||
- export-image
|
||||
- get-console-output
|
||||
- get-console-screenshot
|
||||
- get-instance-metadata-defaults
|
||||
- get-instance-tpm-ek-pub
|
||||
@@ -180,7 +186,6 @@ TODO:
|
||||
- modify-volume-attribute
|
||||
- modify-vpn-connection
|
||||
- modify-vpn-connection-options
|
||||
- monitor-instances
|
||||
- reboot-instances
|
||||
- register-image
|
||||
- release-address
|
||||
@@ -245,22 +250,23 @@ TODO:
|
||||
|
||||
### S3
|
||||
|
||||
* Consider moving S3 control/data plane, from predastore, to Hive format.
|
||||
- Consider moving S3 control/data plane, from predastore, to Hive format.
|
||||
|
||||
## Update Nov 2025
|
||||
|
||||
[PARTIAL] * Implement multi-tenant support
|
||||
[PARTIAL] * Move config settings from ~/hive/*.toml, to using Nats Jetstream for core config which can be synced between nodes.
|
||||
* Implement a lightweight IAM using Nats Jetstream, vs current config files for auth settings.
|
||||
[ONGOING] * Implement Reed Solomon Encoding for Predastore (S3) objects and a WAL implementation, for storing multiple objects in a single shard (e.g 4MB), with the WAL referencing the location of each object (e.g 4kb min)
|
||||
* Implement basic KY lookup, object key sha512 (bucket/key), location to shard on S3 (obj.0000124.bin), read WAL (e.g first 4096 bytes) to determine location of the object. e.g key-1234 => obj.0000124.bin, wal header (4096 bytes) > key-1234 == offset location (seek) 16384, len 32768.
|
||||
* Read multiple nodes (e.g 5 predastore instances, k = 3 (data shards), n = 5 (total shards), n - k = 2 (parity shards) )
|
||||
[ONGOING] * Complete core scaffolding AWS SDK/API requirements (ec2 describe-instances, run-instances, etc)
|
||||
* Implement UEFI support for image downloads and `qemu` exec in `vm.go`
|
||||
* Confirm Alpine Linux, fails import image AMI > (run-instance) ""Failed to read block from AMI source" err="request out of range" - Block size correct?
|
||||
* Improve shutdown gracefully, `./scripts/stop-dev.sh` waits 60 seconds, while qemu/nbd could still be shutting down.
|
||||
* Add delete-volume support via EBS (s3 vol-*) for terminated instance
|
||||
* Add default LRU cache support for viperblock, depending on the instance type / volume size and system memory available.
|
||||
[PARTIAL] _ Implement multi-tenant support
|
||||
[PARTIAL] _ Move config settings from ~/hive/\*.toml, to using Nats Jetstream for core config which can be synced between nodes.
|
||||
|
||||
- Implement a lightweight IAM using Nats Jetstream, vs current config files for auth settings.
|
||||
[ONGOING] _ Implement Reed Solomon Encoding for Predastore (S3) objects and a WAL implementation, for storing multiple objects in a single shard (e.g 4MB), with the WAL referencing the location of each object (e.g 4kb min)
|
||||
_ Implement basic KY lookup, object key sha512 (bucket/key), location to shard on S3 (obj.0000124.bin), read WAL (e.g first 4096 bytes) to determine location of the object. e.g key-1234 => obj.0000124.bin, wal header (4096 bytes) > key-1234 == offset location (seek) 16384, len 32768.
|
||||
_ Read multiple nodes (e.g 5 predastore instances, k = 3 (data shards), n = 5 (total shards), n - k = 2 (parity shards) )
|
||||
[ONGOING] _ Complete core scaffolding AWS SDK/API requirements (ec2 describe-instances, run-instances, etc)
|
||||
- Implement UEFI support for image downloads and `qemu` exec in `vm.go`
|
||||
- Confirm Alpine Linux, fails import image AMI > (run-instance) ""Failed to read block from AMI source" err="request out of range" - Block size correct?
|
||||
- Improve shutdown gracefully, `./scripts/stop-dev.sh` waits 60 seconds, while qemu/nbd could still be shutting down.
|
||||
- Add delete-volume support via EBS (s3 vol-\*) for terminated instance
|
||||
- Add default LRU cache support for viperblock, depending on the instance type / volume size and system memory available.
|
||||
|
||||
## Multi-node setup
|
||||
|
||||
@@ -288,17 +294,17 @@ For production, it is recommended to run Hive on at least three physical nodes.
|
||||
|
||||
### Overview:
|
||||
|
||||
* init creates a cluster-id, node-id, and a short join token, starts a tiny control server on :8443, and writes DNS hint files if available.
|
||||
- init creates a cluster-id, node-id, and a short join token, starts a tiny control server on :8443, and writes DNS hint files if available.
|
||||
|
||||
* init sets target size to 3 and waits until all 3 nodes join and ack.
|
||||
- init sets target size to 3 and waits until all 3 nodes join and ack.
|
||||
|
||||
* join nodes contact node1, present the token or cluster-id, and advertise capabilities.
|
||||
- join nodes contact node1, present the token or cluster-id, and advertise capabilities.
|
||||
|
||||
* node1 appends them to the member set and immediately pushes the current settings bundle to them.
|
||||
- node1 appends them to the member set and immediately pushes the current settings bundle to them.
|
||||
|
||||
* When member count reaches 3, node1 broadcasts the final cluster settings to all nodes.
|
||||
- When member count reaches 3, node1 broadcasts the final cluster settings to all nodes.
|
||||
|
||||
* Every node writes the same cluster.json and starts the services that match its capabilities.
|
||||
- Every node writes the same cluster.json and starts the services that match its capabilities.
|
||||
|
||||
Node exchange payload
|
||||
|
||||
@@ -308,7 +314,7 @@ Node exchange payload
|
||||
"node_id": "n-rpi2",
|
||||
"addr": "rpi2.local:8443",
|
||||
"version": "0.1.0",
|
||||
"caps": ["ec2","s3"],
|
||||
"caps": ["ec2", "s3"],
|
||||
"ts": 1731388800
|
||||
}
|
||||
```
|
||||
@@ -320,18 +326,22 @@ Settings bundle (identical on all 3 once committed)
|
||||
"cluster_id": "c-82d5",
|
||||
"target_size": 3,
|
||||
"members": [
|
||||
{"node_id":"n-rpi1","addr":"rpi1.local:8443","caps":["ec2","s3","nats","ebs"]},
|
||||
{"node_id":"n-rpi2","addr":"rpi2.local:8443","caps":["ec2","s3"]},
|
||||
{"node_id":"n-rpi3","addr":"rpi3.local:8443","caps":["nats","ebs"]}
|
||||
{
|
||||
"node_id": "n-rpi1",
|
||||
"addr": "rpi1.local:8443",
|
||||
"caps": ["ec2", "s3", "nats", "ebs"]
|
||||
},
|
||||
{ "node_id": "n-rpi2", "addr": "rpi2.local:8443", "caps": ["ec2", "s3"] },
|
||||
{ "node_id": "n-rpi3", "addr": "rpi3.local:8443", "caps": ["nats", "ebs"] }
|
||||
],
|
||||
"services": {
|
||||
"ec2": {"api_bind":":9001"},
|
||||
"s3": {"api_bind":":9002","replicas":2},
|
||||
"nats":{"cluster":"c-82d5","quorum":2},
|
||||
"ebs": {"replicas":3,"block_size":4096}
|
||||
"ec2": { "api_bind": ":9001" },
|
||||
"s3": { "api_bind": ":9002", "replicas": 2 },
|
||||
"nats": { "cluster": "c-82d5", "quorum": 2 },
|
||||
"ebs": { "replicas": 3, "block_size": 4096 }
|
||||
},
|
||||
"epoch": 1,
|
||||
"sig": "ed25519:..." // signed by rpi1 during init
|
||||
"sig": "ed25519:..." // signed by rpi1 during init
|
||||
}
|
||||
```
|
||||
|
||||
@@ -341,46 +351,54 @@ Settings bundle (identical on all 3 once committed)
|
||||
|
||||
All original TODO items have been incorporated into the structured development phases:
|
||||
|
||||
| Original TODO | Status | Integrated Into | Phase |
|
||||
|---------------|--------|-----------------|-------|
|
||||
| #1: Binary compile and install.sh | ✅ Planned | Phase 7: Task 7.1 (Binary Compilation) | Phase 7 |
|
||||
| #2: Move daemon.go to services/hived/ | ✅ Planned | Phase 3: Task 3.2 (Specialized Services) | Phase 3 |
|
||||
| #3: VPC with openvs-switch as `vpcd` | ✅ Planned | Phase 5: Task 5.1 (VPC with Open vSwitch) | Phase 5 |
|
||||
| #4: AWS HTTP gateway (`awsd`) | ✅ Planned | Phase 3: Task 3.1 (AWS Gateway Service) | Phase 3 |
|
||||
| #5: AWS SDK v2 input/output | ✅ Planned | Phase 2: Task 2.1 & Phase 3: Task 3.1 | Phase 2-3 |
|
||||
| #6: `hive.service` for systemd | ✅ Planned | Phase 7: Task 7.2 (System Service Integration) | Phase 7 |
|
||||
| #7: Gossip and RAFT protocols | ✅ Planned | Phase 0: Task 0.1 (Service Registry) | Phase 0 |
|
||||
| #8: etcd/KV for configuration sync | ✅ Planned | Phase 0: Task 0.1 (Distributed Config) | Phase 0 |
|
||||
| #9: Smithy model code generation | ✅ Planned | Phase 2: Task 2.1 (Smithy Code Generation) | Phase 2 |
|
||||
| Original TODO | Status | Integrated Into | Phase |
|
||||
| ------------------------------------- | ---------- | ---------------------------------------------- | --------- |
|
||||
| #1: Binary compile and install.sh | ✅ Planned | Phase 7: Task 7.1 (Binary Compilation) | Phase 7 |
|
||||
| #2: Move daemon.go to services/hived/ | ✅ Planned | Phase 3: Task 3.2 (Specialized Services) | Phase 3 |
|
||||
| #3: VPC with openvs-switch as `vpcd` | ✅ Planned | Phase 5: Task 5.1 (VPC with Open vSwitch) | Phase 5 |
|
||||
| #4: AWS HTTP gateway (`awsd`) | ✅ Planned | Phase 3: Task 3.1 (AWS Gateway Service) | Phase 3 |
|
||||
| #5: AWS SDK v2 input/output | ✅ Planned | Phase 2: Task 2.1 & Phase 3: Task 3.1 | Phase 2-3 |
|
||||
| #6: `hive.service` for systemd | ✅ Planned | Phase 7: Task 7.2 (System Service Integration) | Phase 7 |
|
||||
| #7: Gossip and RAFT protocols | ✅ Planned | Phase 0: Task 0.1 (Service Registry) | Phase 0 |
|
||||
| #8: etcd/KV for configuration sync | ✅ Planned | Phase 0: Task 0.1 (Distributed Config) | Phase 0 |
|
||||
| #9: Smithy model code generation | ✅ Planned | Phase 2: Task 2.1 (Smithy Code Generation) | Phase 2 |
|
||||
|
||||
## Development Phase Overview
|
||||
|
||||
**Phase 0**: Distributed Systems Foundation (1-2 weeks)
|
||||
|
||||
- Gossip, RAFT, and distributed configuration (#7, #8)
|
||||
|
||||
**Phase 1**: Development Environment Automation (2-3 weeks)
|
||||
|
||||
- Multi-service orchestration and hot reloading
|
||||
|
||||
**Phase 2**: AWS API Model Implementation (3-4 weeks)
|
||||
|
||||
- Smithy-based code generation (#9)
|
||||
- AWS SDK v2 integration (#5)
|
||||
|
||||
**Phase 3**: Scalable Gateway and Daemon Architecture (2-3 weeks)
|
||||
|
||||
- AWS gateway service `awsd` (#4)
|
||||
- Service refactoring to `hived` (#2)
|
||||
- VPC daemon `vpcd` foundation (#3)
|
||||
|
||||
**Phase 4**: Service Integration and Orchestration (3-4 weeks)
|
||||
|
||||
- NATS clustering and cross-service coordination
|
||||
|
||||
**Phase 5**: Infrastructure Services (4-5 weeks)
|
||||
|
||||
- VPC networking with Open vSwitch (#3)
|
||||
- Advanced VM features
|
||||
|
||||
**Phase 6**: Testing and Validation Framework (2-3 weeks)
|
||||
|
||||
- AWS CLI compatibility testing
|
||||
|
||||
**Phase 7**: Production Deployment and Packaging (2-3 weeks)
|
||||
|
||||
- Binary compilation and distribution (#1)
|
||||
- Systemd service integration (#6)
|
||||
- Production configuration management
|
||||
@@ -398,9 +416,10 @@ The development plan implements these services:
|
||||
## Getting Started
|
||||
|
||||
For development setup:
|
||||
|
||||
```bash
|
||||
./scripts/dev-setup.sh # Setup complete development environment
|
||||
./scripts/start-dev.sh # Start all services
|
||||
```
|
||||
|
||||
For detailed implementation guidance, see [HIVE_DEVELOPMENT_PLAN.md](HIVE_DEVELOPMENT_PLAN.md).
|
||||
For detailed implementation guidance, see [HIVE_DEVELOPMENT_PLAN.md](HIVE_DEVELOPMENT_PLAN.md).
|
||||
|
||||
Reference in New Issue
Block a user