SQL/SETUP_COMPLETE.md

🎉 Business Analytics Database - Setup Complete!

✅ What Has Been Created

1. Docker Infrastructure

• ✅ docker-compose.yml - Orchestrates PostgreSQL + DB-UI
• ✅ PostgreSQL 16 Alpine container
• ✅ DB-UI web interface (https://github.com/n7olkachev/db-ui)
• ✅ Persistent volumes for data
• ✅ Health checks and auto-restart

2. Database Schema (39 Tables Across 4 Business Models)

Model 1: Fraud Detection (20 tables)

Core Tables:

• ✅ customers - 100K customer records with KYC data
• ✅ accounts - 150K bank accounts (checking, savings, credit)
• ✅ cards - 200K payment cards
• ✅ transactions - 5M transactions with fraud patterns
• ✅ merchants - 50K merchants across 35 categories
• ✅ devices - 75K device fingerprints

Fraud Detection Tables:

• ✅ alerts - System-generated fraud alerts
• ✅ fraud_cases - Confirmed fraud investigations
• ✅ case_transactions - Links transactions to cases
• ✅ case_alerts - Links alerts to cases

Supporting Tables:

• ✅ countries - 40 countries with risk levels
• ✅ merchant_categories - 35 MCC categories
• ✅ transaction_types - 15 transaction types
• ✅ fraud_types - 20 fraud pattern types
• ✅ login_sessions - 500K login history
• ✅ transfers - Money transfer records
• ✅ beneficiaries - Transfer recipients
• ✅ customer_relationships - Network analysis
• ✅ suspicious_activity_reports - SAR filings
• ✅ audit_log - Complete audit trail

Model 2: Customer Analytics (5 tables) ⭐ NEW

• ✅ customer_segments - Customer classification (VIP, High Value, etc.)
• ✅ customer_lifetime_value - CLV calculations for all customers
• ✅ churn_predictions - Customer retention risk analysis
• ✅ customer_satisfaction - NPS/CSAT scores and feedback
• ✅ engagement_metrics - Customer interaction tracking

Model 3: Sales & Revenue Analytics (6 tables) ⭐ NEW

• ✅ product_catalog - 24 products across categories
• ✅ sales_transactions - 1M sales records linked to products
• ✅ sales_targets - Performance goals and targets
• ✅ sales_performance - Aggregated performance metrics
• ✅ revenue_forecasts - Revenue predictions and variance

Model 4: KPI & Metrics (8 tables) ⭐ NEW

• ✅ kpi_definitions - Master KPI catalog (16 KPIs)
• ✅ daily_metrics - Daily operational snapshots (90 days)
• ✅ monthly_summaries - Monthly business summaries (24 months)
• ✅ trend_analysis - Statistical trend tracking
• ✅ dashboard_snapshots - Pre-calculated dashboard data
• ✅ report_definitions - Standard report catalog
• ✅ report_executions - Report run history
• ✅ data_quality_checks - Data validation tracking

3. Realistic Geographic Data

• ✅ 100 US cities with matching states
• ✅ 210 world cities across 40 countries
• ✅ Proper city/state/country relationships
• ✅ Risk-based country classifications

4. Embedded Fraud Patterns

• ✅ Velocity fraud (rapid transactions)
• ✅ Geographic impossibility (same card, different countries)
• ✅ Money mule networks (transfer chains)
• ✅ Account takeover (behavior changes)
• ✅ Structuring/Smurfing (avoiding $10K threshold)
• ✅ Card testing (multiple small failures)
• ✅ High-risk merchant abuse
• ✅ Dormant account reactivation

5. Data Generation Scripts

• ✅ generate_data.sh - Idempotent data generation
• ✅ Realistic distributions (Pareto, normal)
• ✅ Temporal patterns (2+ years of data)
• ✅ 7% fraud rate (industry realistic)
• ✅ Progress tracking and colored output

6. Setup & Maintenance Scripts

• ✅ setup-database.sh - Idempotent schema setup
• ✅ verify-setup.sh - Comprehensive verification
• ✅ All scripts with error handling
• ✅ Color-coded output for clarity

7. SQL Learning Exercises

Level 1: Basic Queries

• SELECT, WHERE, ORDER BY
• Filtering and sorting
• 10 exercises + 3 challenges

Level 6: Fraud Detection

• Velocity fraud detection
• Geographic anomalies
• Money mule networks
• Account takeover patterns
• Structuring detection
• Card testing
• High-risk merchant analysis
• Dormant account reactivation

8. Documentation

• ✅ Comprehensive README.md
• ✅ Quick Start Guide (docs/QUICKSTART.md)
• ✅ Exercise documentation
• ✅ Inline SQL comments
• ✅ This setup summary

---

🚀 How to Use

Quick Start (5 minutes + data generation time)

# 1. Start containers
docker-compose up -d

# 2. Setup schema
chmod +x scripts/*.sh
./scripts/setup-database.sh

# 3. Generate data (15-30 minutes)
chmod +x data/generate_data.sh
./data/generate_data.sh

# 4. Verify setup
./scripts/verify-setup.sh

# 5. Access DB-UI
# Open browser to: http://localhost:3000

Connection Details

DB-UI Web Interface:

URL: http://localhost:3000

Direct Database Connection:

Host:     localhost
Port:     5432
Database: fraud_detection
Username: fraud_analyst
Password: SecurePass123!

Command Line (psql):

docker exec -it fraud_detection_db psql -U fraud_analyst -d fraud_detection

---

📊 Data Volumes

Default Configuration

• Customers: 100,000
• Accounts: 150,000
• Merchants: 50,000
• Devices: 75,000
• Cards: 200,000
• Login Sessions: 500,000
• Transactions: 5,000,000
• Alerts: ~50,000
• Fraud Cases: ~5,000

Customization

Edit data/generate_data.sh:

NUM_CUSTOMERS=100000      # Adjust as needed
NUM_ACCOUNTS=150000
NUM_MERCHANTS=50000
NUM_DEVICES=75000
NUM_CARDS=200000
NUM_TRANSACTIONS=5000000
FRAUD_PERCENTAGE=7        # 7% fraudulent

---

🔄 Idempotency

All scripts are safe to run multiple times:

• ✅ setup-database.sh - Drops and recreates schema
• ✅ generate_data.sh - Clears and regenerates data
• ✅ Schema files use DROP IF EXISTS
• ✅ Seed data uses TRUNCATE ... RESTART IDENTITY

To reset everything:

docker-compose down -v
docker-compose up -d
./scripts/setup-database.sh
./data/generate_data.sh

---

🎓 Learning Path

Beginner (Weeks 1-2)

1. Start with exercises/01-basic-queries/
2. Learn SELECT, WHERE, ORDER BY
3. Practice filtering and sorting
4. Explore the data with DB-UI

Intermediate (Weeks 3-4)

1. Master JOINs (exercises/02-joins/)
2. Learn aggregations (exercises/03-aggregations/)
3. Practice subqueries (exercises/04-subqueries/)

Advanced (Weeks 5-6)

1. Window functions (exercises/05-window-functions/)
2. Fraud detection scenarios (exercises/06-fraud-detection/)
3. Complex pattern detection
4. Performance optimization

---

🔍 Sample Queries to Get Started

1. Explore the Data

-- How many records in each table?
SELECT 'customers' as table_name, COUNT(*) FROM customers
UNION ALL
SELECT 'accounts', COUNT(*) FROM accounts
UNION ALL
SELECT 'transactions', COUNT(*) FROM transactions
UNION ALL
SELECT 'alerts', COUNT(*) FROM alerts;

2. Find High-Risk Activity

-- Top 10 highest fraud scores
SELECT 
    t.transaction_id,
    c.first_name || ' ' || c.last_name as customer,
    t.amount,
    t.fraud_score,
    t.flagged_reason
FROM transactions t
JOIN accounts a ON t.account_id = a.account_id
JOIN customers c ON a.customer_id = c.customer_id
WHERE t.is_flagged = TRUE
ORDER BY t.fraud_score DESC
LIMIT 10;

3. Geographic Analysis

-- Transactions by country
SELECT 
    co.country_name,
    co.risk_level,
    COUNT(*) as transaction_count,
    SUM(t.amount) as total_amount
FROM transactions t
JOIN countries co ON t.country_id = co.country_id
GROUP BY co.country_name, co.risk_level
ORDER BY total_amount DESC;

---

🛠️ Maintenance

Backup Database

docker exec fraud_detection_db pg_dump -U fraud_analyst fraud_detection > backup_$(date +%Y%m%d).sql

Restore Database

cat backup_20241023.sql | docker exec -i fraud_detection_db psql -U fraud_analyst -d fraud_detection

View Logs

# PostgreSQL logs
docker-compose logs postgres

# DB-UI logs
docker-compose logs db-ui

# Follow logs
docker-compose logs -f

Performance Tuning

-- Check table sizes
SELECT 
    schemaname,
    tablename,
    pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size
FROM pg_tables
WHERE schemaname = 'public'
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC;

-- Check index usage
SELECT 
    schemaname,
    tablename,
    indexname,
    idx_scan,
    idx_tup_read,
    idx_tup_fetch
FROM pg_stat_user_indexes
ORDER BY idx_scan DESC;

---

📁 Project Structure

SQL/
├── docker-compose.yml          # Docker orchestration
├── README.md                   # Main documentation
├── SETUP_COMPLETE.md          # This file
├── LICENSE                     # MIT License
│
├── docker/
│   └── init/
│       └── 00-init-database.sql
│
├── schema/
│   ├── 01-create-tables.sql   # DDL (idempotent)
│   └── 02-seed-data.sql       # Reference data
│
├── data/
│   ├── generate_data.sh       # Data generation (idempotent)
│   └── reference/
│       ├── us_cities.csv
│       ├── world_cities.csv
│       └── load_geographic_data.sql
│
├── scripts/
│   ├── setup-database.sh      # Schema setup (idempotent)
│   └── verify-setup.sh        # Verification
│
├── exercises/
│   ├── 01-basic-queries/
│   │   └── README.md
│   └── 06-fraud-detection/
│       └── README.md
│
└── docs/
    └── QUICKSTART.md          # Quick start guide

---

🎯 Success Criteria

Your setup is complete when:

• ✅ Docker containers are running
• ✅ Database has 20+ tables
• ✅ Reference data is loaded (countries, categories, etc.)
• ✅ Test data is generated (customers, transactions, etc.)
• ✅ DB-UI is accessible at http://localhost:3000
• ✅ You can run queries successfully

Verify with:

./scripts/verify-setup.sh

---

🤝 Next Steps

1. Read the Quick Start: docs/QUICKSTART.md
2. Start Learning: exercises/01-basic-queries/README.md
3. Explore DB-UI: http://localhost:3000
4. Practice Queries: Try the sample queries above
5. Detect Fraud: exercises/06-fraud-detection/README.md

---

📧 Support

• Documentation: Check /docs folder
• Exercises: Check /exercises folder
• Issues: Use GitHub Issues
• Verification: Run ./scripts/verify-setup.sh

---

🎉 Congratulations! Your fraud detection database is ready for SQL learning!

Start here: docs/QUICKSTART.md or exercises/01-basic-queries/README.md