flowfish/docs/architecture/DATA_ARCHITECTURE.md

Flowfish Data Architecture

Overview

Flowfish uses a polyglot persistence architecture with three specialized databases, each optimized for its specific use case.

┌─────────────────────────────────────────────────────────────────────────────┐
│                         Flowfish Data Architecture                           │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │                    PostgreSQL (Metadata & Config)                    │    │
│  │                         Port: 5432                                   │    │
│  │                                                                      │    │
│  │  • Cluster configurations (connection details, credentials)          │    │
│  │  • Analysis definitions (scope, gadgets, time settings)              │    │
│  │  • User management and RBAC                                          │    │
│  │  • Workload metadata (synced from ClickHouse)                        │    │
│  │  • System settings and notification hooks                            │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                                                              │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │                    ClickHouse (Event Lake)                           │    │
│  │                         Port: 9000                                   │    │
│  │                                                                      │    │
│  │  • Network flows (millions/day)                                      │    │
│  │  • DNS queries (millions/day)                                        │    │
│  │  • Process events (millions/day)                                     │    │
│  │  • Change events (infrastructure changes)                            │    │
│  │  • Workload metadata (pod discovery)                                 │    │
│  │  • Communication edges (aggregated)                                  │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                                                              │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │                    Neo4j (Dependency Graph)                          │    │
│  │                         Port: 7687                                   │    │
│  │                                                                      │    │
│  │  • Workload nodes (pods, deployments, services)                      │    │
│  │  • Communication relationships (COMMUNICATES_WITH)                   │    │
│  │  • Path finding and impact analysis                                  │    │
│  │  • Topology visualization                                            │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                                                              │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │                    Supporting Services                               │    │
│  │                                                                      │    │
│  │  Redis (6379): Session cache, leader election, rate limiting         │    │
│  │  RabbitMQ (5672): Event streaming, async processing                  │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

---

PostgreSQL Tables

Core Metadata

Table	Purpose	Key Fields
clusters	Cluster connection config	api_url, kubeconfig_encrypted, gadget_namespace
analyses	Analysis definitions	scope_config, gadget_config, change_detection_enabled
analysis_runs	Execution history	started_at, completed_at, events_collected
users	User accounts	username, email, password_hash, role
roles	RBAC roles	name, permissions

Supporting Tables

Table	Purpose
namespaces	Namespace metadata
workloads	Workload metadata (synced from ClickHouse)
communications	Communication relationships
notification_hooks	Alert configurations
system_settings	Global settings
oauth_providers	SSO configurations

Analysis Configuration

-- Key analysis table structure
CREATE TABLE analyses (
    id SERIAL PRIMARY KEY,
    name VARCHAR(255) NOT NULL,
    cluster_id INTEGER REFERENCES clusters(id),
    cluster_ids JSONB DEFAULT '[]',           -- Multi-cluster support
    is_multi_cluster BOOLEAN DEFAULT FALSE,
    scope_type VARCHAR(50),                   -- cluster, namespace, pod
    scope_config JSONB,                       -- Scope details
    gadget_config JSONB,                      -- eBPF modules
    time_config JSONB,                        -- Duration settings
    output_config JSONB,                      -- Dashboard, alerts
    change_detection_enabled BOOLEAN DEFAULT TRUE,  -- Feature toggle
    change_detection_strategy VARCHAR(50) DEFAULT 'baseline',  -- baseline, rolling_window, run_comparison
    change_detection_types JSONB DEFAULT '["all"]',  -- Types to track
    status VARCHAR(50) DEFAULT 'draft',
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

---

ClickHouse Tables

Event Tables

Table	Source	Description	Volume
network_flows	trace_tcp	Network connections	Millions/day
dns_queries	trace_dns	DNS lookups	Millions/day
tcp_lifecycle	trace_tcp	TCP state transitions	High
process_events	trace_exec	Process execution	High
file_operations	trace_open	File I/O	Medium
capability_checks	trace_capabilities	Linux caps	Low
oom_kills	trace_oomkill	OOM events	Low
bind_events	trace_bind	Socket binds	Medium
sni_events	trace_sni	TLS SNI	Medium
mount_events	trace_mount	Mount operations	Low

Change Events Table

CREATE TABLE change_events (
    event_id UUID,
    analysis_id Int32,
    run_id Int32,                    -- Analysis run reference
    cluster_id Int32,
    change_type String,              -- workload_added, workload_removed, etc.
    entity_type String,              -- workload, connection, namespace
    target_name String,
    namespace String,
    risk_level String,               -- low, medium, high, critical
    affected_services Int32,
    details String,
    changed_by String,
    detected_at DateTime,
    created_at DateTime DEFAULT now()
) ENGINE = ReplacingMergeTree()
PARTITION BY toYYYYMMDD(detected_at)
ORDER BY (analysis_id, cluster_id, detected_at, event_id);

Derived Tables

Table	Purpose
workload_metadata	Pod discovery (ReplacingMergeTree)
communication_edges	Aggregated communications (SummingMergeTree)
change_events_*_mv	Materialized views for stats

---

Neo4j Graph Model

Nodes

// Workload Node (Pod, Deployment, Service)
(:Workload {
    name: "pod-name",
    namespace: "default",
    workload_type: "pod",
    cluster_id: 1,
    analysis_id: 123,
    labels: '{"app": "frontend"}',
    created_at: datetime(),
    last_seen: datetime()
})

// Namespace Node
(:Namespace {
    name: "production",
    cluster_id: 1
})

Relationships

// Communication between workloads
(:Workload)-[:COMMUNICATES_WITH {
    port: 8080,
    protocol: "TCP",
    request_count: 12450,
    avg_latency_ms: 12.5,
    first_seen: datetime(),
    last_seen: datetime()
}]->(:Workload)

// Namespace containment
(:Namespace)-[:CONTAINS]->(:Workload)

---

Data Flow Patterns

Write Path (Ingestion)

Inspector Gadget (eBPF)
        │
        ▼
Ingestion Service
        │
        ├──► RabbitMQ (flowfish.network_flows)
        │           │
        │           └──► Timeseries Writer ──► ClickHouse
        │
        ├──► RabbitMQ (flowfish.workload_metadata)
        │           │
        │           └──► Timeseries Writer ──► ClickHouse
        │                                  └──► PostgreSQL (workloads sync)
        │
        └──► RabbitMQ (flowfish.change_events)  [If enabled]
                    │
                    └──► Timeseries Writer ──► ClickHouse

Change Detection Data Flow

┌─────────────────────────────────────────────────────────────────────────────┐
│                         Change Detection Worker                              │
│                                                                             │
│  ┌─────────────────────────┐        ┌─────────────────────────┐            │
│  │    K8s API Detector     │        │    eBPF Event Detector   │            │
│  │                         │        │                          │            │
│  │ Source: K8s API         │        │ Source: ClickHouse       │            │
│  │ Detects:                │        │ Detects:                 │            │
│  │  • replica_changed      │        │  • connection_added      │            │
│  │  • config_changed       │        │  • connection_removed    │            │
│  │  • image_changed        │        │  • port_changed          │            │
│  │  • label_changed        │        │  • traffic_anomaly       │            │
│  └───────────┬─────────────┘        └───────────┬──────────────┘            │
│              │                                  │                            │
│              └──────────────┬───────────────────┘                            │
│                             │                                                │
│                             ▼                                                │
│                    ┌────────────────┐                                       │
│                    │  Merge Changes │                                       │
│                    └────────┬───────┘                                       │
│                             │                                                │
└─────────────────────────────┼────────────────────────────────────────────────┘
                              │
                              ▼
                    ┌─────────────────┐
                    │    RabbitMQ     │
                    │ (change_events) │
                    └────────┬────────┘
                             │
                             ▼
                    ┌─────────────────┐
                    │Timeseries Writer│
                    └────────┬────────┘
                             │
                             ▼
                    ┌─────────────────┐
                    │   ClickHouse    │
                    │ (change_events) │
                    └─────────────────┘

Detection Sources

Source	Data Type	Storage	Update Frequency
K8s API	Infrastructure state	PostgreSQL (workloads)	60s polling
ClickHouse	Behavioral events	ClickHouse (network_flows)	Real-time

Detection Strategies

Strategy	Description	Data Windows
baseline	Compare against initial state	First 10min vs Now
rolling_window	Compare recent periods	Last 10min vs Last 5min
run_comparison	Compare between runs	Run N-1 vs Run N

Read Path (Query)

Frontend
    │
    ├── Events/Analytics ──► Backend ──► ClickHouse
    │
    ├── Dependency Graph ──► Backend ──► Neo4j
    │
    └── Metadata/Config ──► Backend ──► PostgreSQL

---

Data Lifecycle

Analysis-Scoped Data

All event data is scoped to an analysis. When an analysis is deleted:

Layer	Cleanup Action
PostgreSQL	CASCADE delete (analyses → analysis_runs)
ClickHouse	DELETE WHERE analysis_id = X (all tables)
Neo4j	DELETE nodes/edges WHERE analysis_id = X
Redis	DELETE analysis:* cache keys

Retention Policy

• ClickHouse: Time-partitioned with configurable TTL (default: 90 days)
• Neo4j: No TTL, cleaned with analysis deletion
• PostgreSQL: No TTL, cleaned with analysis deletion
• Redis: Session TTL (1 hour), cache TTL (5 minutes)

---

Feature Toggles

Change Detection

The change detection settings on analyses table control:

change_detection_enabled = TRUE (default)
├── Change Detection Worker monitors for changes
├── Both K8s API and eBPF detectors run
├── Changes published to RabbitMQ → ClickHouse
└── Change Detection page shows data

change_detection_enabled = FALSE
├── No change tracking
├── No data written to change_events
└── Lower storage/processing overhead

Detection Strategy

The change_detection_strategy controls how changes are detected:

strategy = 'baseline' (default)
├── First N minutes captured as baseline
├── Current state compared against baseline
└── Best for long-running analyses

strategy = 'rolling_window'
├── Compares recent window vs previous window
├── Continuous real-time detection
└── Best for monitoring/alerting

strategy = 'run_comparison'
├── Compares current run vs previous run
├── Requires multiple runs
└── Best for deployment validation

Change Types Filter

The change_detection_types array controls which changes to track:

types = ['all'] (default)
└── All change types tracked

types = ['replica_changed', 'connection_added']
├── Only specified types tracked
├── K8s types: replica_changed, config_changed, image_changed, label_changed
└── eBPF types: connection_added, connection_removed, port_changed

---

Data Distribution Summary

Data Type	PostgreSQL	ClickHouse	Neo4j
Cluster config	✅	❌	❌
Analysis config	✅	❌	❌
User/RBAC	✅	❌	❌
Workload metadata	✅ (synced)	✅ (source)	✅ (nodes)
Network events	❌	✅	❌
Change events	❌	✅	❌
Dependencies	❌	❌	✅
Aggregations	❌	✅	❌
Path finding	❌	❌	✅

---

Best Practices

Query Optimization

1. Time-series queries: Always include time range filter
2. Graph queries: Limit traversal depth (default: 1 hop, max: 5 hops)
3. Metadata queries: Use indexed columns (cluster_id, analysis_id)

Data Consistency

1. Analysis ID propagates across all layers
2. Deletion is atomic across all databases
3. Workload sync ensures PostgreSQL reflects ClickHouse state

Scaling

Database	Scaling Strategy
PostgreSQL	Vertical (single master)
ClickHouse	Horizontal (sharding by analysis_id)
Neo4j	Causal cluster (read replicas)

---

Last Updated: January 2026 Architecture Version: 3.0