From aa803bd3bd7c38310dadd9388dfc8887b1eb5912 Mon Sep 17 00:00:00 2001 From: Alphaeus Mote Date: Thu, 23 Oct 2025 14:17:12 -0400 Subject: [PATCH] Expand to Business Analytics: Add Customer, Sales, and KPI models Major expansion from fraud detection to comprehensive business analytics: DATABASE CHANGES: - Renamed database from 'fraud_detection' to 'business_analytics' - Renamed user from 'fraud_analyst' to 'data_analyst' - Expanded from 20 to 39 tables across 4 business models NEW MODELS (19 tables): 1. Customer Analytics (5 tables): - customer_segments, customer_lifetime_value, churn_predictions - customer_satisfaction, engagement_metrics 2. Sales & Revenue Analytics (6 tables): - product_catalog, sales_transactions, sales_targets - sales_performance, revenue_forecasts 3. KPI & Metrics (8 tables): - kpi_definitions, daily_metrics, monthly_summaries - trend_analysis, dashboard_snapshots - report_definitions, report_executions, data_quality_checks DATA GENERATION: - Extended generate_data.sh with 6 new steps (now 15 total) - Added CLV calculations for all customers - Added churn predictions based on transaction recency - Added 30K customer satisfaction surveys - Added 1M sales transactions linked to 24 products - Added 90 days of daily KPI metrics - Added 24 months of business summaries SQL EXERCISES (3 new levels): - Level 2: Customer Analytics (10 exercises + 3 challenges) - Level 3: Sales & Revenue Analysis (12 exercises + 3 challenges) - Level 4: KPI Dashboards & Metrics (12 exercises + 3 challenges) DOCUMENTATION: - Updated README.md with business analytics focus - Updated QUICKSTART.md with new data generation steps - Updated SETUP_COMPLETE.md with 39-table architecture - Added DATA_MODELS.md with complete model specifications - Added WHATS_NEW.md with migration guide SEED DATA: - Added 8 customer segments (VIP, High Value, etc.) - Added 24 products across 5 categories - Added 16 KPI definitions across 4 categories - Added 8 standard report definitions All changes maintain idempotency and backward compatibility with existing fraud detection functionality. --- README.md | 223 ++++++++--- SETUP_COMPLETE.md | 36 +- data/generate_data.sh | 286 +++++++++++++- docker-compose.yml | 20 +- docs/DATA_MODELS.md | 358 ++++++++++++++++++ docs/QUICKSTART.md | 97 +++-- docs/WHATS_NEW.md | 266 +++++++++++++ exercises/02-customer-analytics/README.md | 325 ++++++++++++++++ exercises/03-sales-analysis/README.md | 380 +++++++++++++++++++ exercises/04-kpi-dashboards/README.md | 431 ++++++++++++++++++++++ schema/01-create-tables.sql | 343 ++++++++++++++++- schema/02-seed-data.sql | 109 ++++++ scripts/setup-database.sh | 8 +- scripts/verify-setup.sh | 12 +- 14 files changed, 2762 insertions(+), 132 deletions(-) create mode 100644 docs/DATA_MODELS.md create mode 100644 docs/WHATS_NEW.md create mode 100644 exercises/02-customer-analytics/README.md create mode 100644 exercises/03-sales-analysis/README.md create mode 100644 exercises/04-kpi-dashboards/README.md diff --git a/README.md b/README.md index 414f5f0..cb9abda 100644 --- a/README.md +++ b/README.md @@ -1,44 +1,82 @@ -# Financial Fraud Detection - SQL Learning Database +# Business Analytics - SQL Learning Database -A comprehensive, production-grade database designed for learning SQL through realistic financial fraud investigation scenarios. +A comprehensive, production-grade database designed for learning SQL through realistic business analytics, customer insights, sales analysis, and fraud detection scenarios. ## 🎯 Overview -This project provides a complete PostgreSQL database with **5+ million transactions**, embedded fraud patterns, and progressive SQL exercises. Perfect for: +This project provides a complete PostgreSQL database with **5+ million transactions**, **39 tables** across 4 business models, and progressive SQL exercises aligned with **Data Analyst** responsibilities. Perfect for: - **SQL Beginners** → Learn fundamentals with real-world data -- **Data Analysts** → Practice fraud detection queries -- **Security Professionals** → Understand fraud patterns -- **Students** → Hands-on financial crime investigation +- **Data Analysts** → Practice customer analytics, sales analysis, KPI dashboards +- **Business Analysts** → Understand customer behavior and revenue patterns +- **Security Professionals** → Detect fraud patterns +- **Students** → Hands-on business intelligence and analytics ## 📊 Database Statistics +### Core Data - **100,000** Customers with KYC data - **150,000** Bank accounts (checking, savings, credit) - **200,000** Payment cards - **50,000** Merchants across 35 categories - **5,000,000** Transactions (7% fraudulent) +- **1,000,000** Sales records linked to products +- **500,000** Login sessions + +### Analytics Data +- **100,000** Customer Lifetime Value calculations +- **100,000** Churn predictions +- **30,000** Customer satisfaction surveys +- **24** Product catalog entries +- **1,500+** Daily KPI metrics +- **24** Monthly business summaries - **50,000+** Fraud alerts - **5,000+** Fraud cases -- **500,000** Login sessions ## 🏗️ Architecture ### Data Model Features -- ✅ **20+ Tables** with proper relationships +- ✅ **39 Tables** across 4 business models - ✅ **Foreign key constraints** for data integrity - ✅ **Indexes** for query performance - ✅ **Realistic geographic data** (100 US cities, 210 world cities) -- ✅ **Embedded fraud patterns** (velocity, geographic, structuring) +- ✅ **Customer analytics** (CLV, churn, satisfaction, engagement) +- ✅ **Sales analytics** (products, targets, forecasts) +- ✅ **KPI tracking** (daily metrics, trends, dashboards) +- ✅ **Fraud detection** (velocity, geographic, structuring patterns) - ✅ **Audit trails** and compliance tables -### Key Entities +### Business Models + +#### 1. Fraud Detection Model (20 tables) ``` -customers → accounts → transactions - ↓ - cards → merchants - ↓ - alerts → fraud_cases +customers → accounts → transactions → alerts → fraud_cases + ↓ + cards → merchants +``` + +#### 2. Customer Analytics Model (5 tables) +``` +customers → customer_lifetime_value → customer_segments + → churn_predictions + → customer_satisfaction + → engagement_metrics +``` + +#### 3. Sales & Revenue Model (6 tables) +``` +product_catalog → sales_transactions → sales_performance + → sales_targets + → revenue_forecasts +``` + +#### 4. KPI & Metrics Model (8 tables) +``` +kpi_definitions → daily_metrics → trend_analysis + → monthly_summaries + → dashboard_snapshots + → report_definitions → report_executions + → data_quality_checks ``` ## 🚀 Quick Start @@ -86,15 +124,15 @@ http://localhost:3000 **Option B: Command Line** ```bash -docker exec -it fraud_detection_db psql -U fraud_analyst -d fraud_detection +docker exec -it business_analytics_db psql -U data_analyst -d business_analytics ``` **Option C: Your Favorite SQL Client** ``` Host: localhost Port: 5432 -Database: fraud_detection -Username: fraud_analyst +Database: business_analytics +Username: data_analyst Password: SecurePass123! ``` @@ -106,29 +144,36 @@ Password: SecurePass123! - Basic comparisons - **Location:** `exercises/01-basic-queries/` -### Level 2: Joins -- INNER JOIN, LEFT JOIN -- Multiple table queries -- Relationship navigation -- **Location:** `exercises/02-joins/` +### Level 2: Customer Analytics ⭐ NEW +- Customer segmentation +- Lifetime value (CLV) analysis +- Churn prediction +- Satisfaction metrics +- Engagement tracking +- **Location:** `exercises/02-customer-analytics/` -### Level 3: Aggregations -- COUNT, SUM, AVG, MAX, MIN -- GROUP BY and HAVING -- Statistical analysis -- **Location:** `exercises/03-aggregations/` +### Level 3: Sales & Revenue Analysis ⭐ NEW +- Sales performance vs targets +- Product analytics +- Channel and regional analysis +- Revenue forecasting +- Profitability analysis +- **Location:** `exercises/03-sales-analysis/` -### Level 4: Subqueries -- Nested queries -- Correlated subqueries -- EXISTS and IN -- **Location:** `exercises/04-subqueries/` +### Level 4: KPI Dashboards & Metrics ⭐ NEW +- KPI tracking and monitoring +- Trend analysis +- Dashboard creation +- Data quality monitoring +- Executive reporting +- **Location:** `exercises/04-kpi-dashboards/` -### Level 5: Window Functions -- ROW_NUMBER, RANK, DENSE_RANK -- Running totals -- Moving averages -- **Location:** `exercises/05-window-functions/` +### Level 5: Advanced SQL Techniques +- Window functions (ROW_NUMBER, RANK) +- Common Table Expressions (CTEs) +- Running totals and moving averages +- Complex aggregations +- **Location:** `exercises/05-advanced-sql/` ### Level 6: Fraud Detection - Velocity fraud detection @@ -137,6 +182,40 @@ Password: SecurePass123! - Account takeover patterns - **Location:** `exercises/06-fraud-detection/` +## 💼 Data Analyst Use Cases + +This database supports typical **Data Analyst** responsibilities: + +### Routine Analysis +- Daily sales summaries +- Customer acquisition metrics +- Transaction volume tracking +- Basic KPI monitoring + +### Semi-Routine Reporting +- Weekly customer analytics +- Monthly revenue reports +- Product performance analysis +- Churn risk identification + +### Dashboard Creation +- Executive KPI dashboards +- Sales performance dashboards +- Customer health dashboards +- Operational metrics dashboards + +### Trend Identification +- Revenue trends (MoM, YoY) +- Customer behavior patterns +- Product sales seasonality +- Engagement score trends + +### Data Quality +- Missing data detection +- Anomaly identification +- Validation checks +- Data completeness monitoring + ## 🔍 Fraud Patterns Included ### 1. Velocity Fraud @@ -180,18 +259,19 @@ SQL/ │ ├── world_cities.csv │ └── load_geographic_data.sql ├── scripts/ -│ └── setup-database.sh # Setup automation +│ ├── setup-database.sh # Setup automation +│ └── verify-setup.sh # Verification script ├── exercises/ -│ ├── 01-basic-queries/ -│ ├── 02-joins/ -│ ├── 03-aggregations/ -│ ├── 04-subqueries/ -│ ├── 05-window-functions/ -│ └── 06-fraud-detection/ +│ ├── 01-basic-queries/ # SQL fundamentals +│ ├── 02-customer-analytics/ # ⭐ Customer insights +│ ├── 03-sales-analysis/ # ⭐ Sales & revenue +│ ├── 04-kpi-dashboards/ # ⭐ KPI tracking +│ ├── 05-advanced-sql/ # Advanced techniques +│ └── 06-fraud-detection/ # Fraud patterns └── docs/ - ├── data-model.md - ├── fraud-patterns.md - └── setup-guide.md + ├── DATA_MODELS.md # ⭐ Complete model documentation + ├── WHATS_NEW.md # ⭐ Recent changes + └── QUICKSTART.md # Quick start guide ``` ## 🔧 Configuration @@ -200,8 +280,8 @@ SQL/ Edit `docker-compose.yml` to customize: ```yaml -POSTGRES_DB: fraud_detection -POSTGRES_USER: fraud_analyst +POSTGRES_DB: business_analytics +POSTGRES_USER: data_analyst POSTGRES_PASSWORD: SecurePass123! ``` @@ -224,15 +304,46 @@ All scripts are **idempotent** - safe to run multiple times: ## 🎓 Sample Queries -### Find High-Risk Customers +### Customer Analytics: High-Value Customers ```sql -SELECT customer_id, first_name, last_name, risk_score -FROM customers -WHERE risk_score > 80 -ORDER BY risk_score DESC; +SELECT + c.customer_id, c.first_name, c.last_name, + clv.clv_score, cs.segment_name +FROM customers c +JOIN customer_lifetime_value clv ON c.customer_id = clv.customer_id +JOIN customer_segments cs ON clv.segment_id = cs.segment_id +WHERE cs.segment_name IN ('VIP', 'High Value') +ORDER BY clv.clv_score DESC +LIMIT 20; ``` -### Detect Velocity Fraud +### Sales Analytics: Top Products +```sql +SELECT + p.product_name, p.product_category, + COUNT(st.transaction_id) as sales_count, + SUM(st.total_amount) as total_revenue +FROM product_catalog p +JOIN sales_transactions st ON p.product_id = st.product_id +GROUP BY p.product_id, p.product_name, p.product_category +ORDER BY total_revenue DESC +LIMIT 10; +``` + +### KPI Dashboard: Current Status +```sql +SELECT + kd.kpi_name, kd.kpi_category, + dm.metric_value, kd.target_value, + dm.status +FROM kpi_definitions kd +JOIN daily_metrics dm ON kd.kpi_id = dm.kpi_id +WHERE dm.metric_date = CURRENT_DATE + AND kd.is_active = TRUE +ORDER BY kd.kpi_category, kd.kpi_name; +``` + +### Fraud Detection: Velocity Fraud ```sql SELECT account_id, COUNT(*) as txn_count, SUM(amount) as total FROM transactions diff --git a/SETUP_COMPLETE.md b/SETUP_COMPLETE.md index c92e49b..662c4ac 100644 --- a/SETUP_COMPLETE.md +++ b/SETUP_COMPLETE.md @@ -1,4 +1,4 @@ -# 🎉 Financial Fraud Detection Database - Setup Complete! +# 🎉 Business Analytics Database - Setup Complete! ## ✅ What Has Been Created @@ -9,9 +9,11 @@ - ✅ Persistent volumes for data - ✅ Health checks and auto-restart -### 2. **Database Schema (20+ Tables)** +### 2. **Database Schema (39 Tables Across 4 Business Models)** -#### Core Tables +#### Model 1: Fraud Detection (20 tables) + +**Core Tables:** - ✅ `customers` - 100K customer records with KYC data - ✅ `accounts` - 150K bank accounts (checking, savings, credit) - ✅ `cards` - 200K payment cards @@ -19,13 +21,13 @@ - ✅ `merchants` - 50K merchants across 35 categories - ✅ `devices` - 75K device fingerprints -#### Fraud Detection Tables +**Fraud Detection Tables:** - ✅ `alerts` - System-generated fraud alerts - ✅ `fraud_cases` - Confirmed fraud investigations - ✅ `case_transactions` - Links transactions to cases - ✅ `case_alerts` - Links alerts to cases -#### Supporting Tables +**Supporting Tables:** - ✅ `countries` - 40 countries with risk levels - ✅ `merchant_categories` - 35 MCC categories - ✅ `transaction_types` - 15 transaction types @@ -37,6 +39,30 @@ - ✅ `suspicious_activity_reports` - SAR filings - ✅ `audit_log` - Complete audit trail +#### Model 2: Customer Analytics (5 tables) ⭐ NEW +- ✅ `customer_segments` - Customer classification (VIP, High Value, etc.) +- ✅ `customer_lifetime_value` - CLV calculations for all customers +- ✅ `churn_predictions` - Customer retention risk analysis +- ✅ `customer_satisfaction` - NPS/CSAT scores and feedback +- ✅ `engagement_metrics` - Customer interaction tracking + +#### Model 3: Sales & Revenue Analytics (6 tables) ⭐ NEW +- ✅ `product_catalog` - 24 products across categories +- ✅ `sales_transactions` - 1M sales records linked to products +- ✅ `sales_targets` - Performance goals and targets +- ✅ `sales_performance` - Aggregated performance metrics +- ✅ `revenue_forecasts` - Revenue predictions and variance + +#### Model 4: KPI & Metrics (8 tables) ⭐ NEW +- ✅ `kpi_definitions` - Master KPI catalog (16 KPIs) +- ✅ `daily_metrics` - Daily operational snapshots (90 days) +- ✅ `monthly_summaries` - Monthly business summaries (24 months) +- ✅ `trend_analysis` - Statistical trend tracking +- ✅ `dashboard_snapshots` - Pre-calculated dashboard data +- ✅ `report_definitions` - Standard report catalog +- ✅ `report_executions` - Report run history +- ✅ `data_quality_checks` - Data validation tracking + ### 3. **Realistic Geographic Data** - ✅ 100 US cities with matching states - ✅ 210 world cities across 40 countries diff --git a/data/generate_data.sh b/data/generate_data.sh index b37b828..57b24e8 100644 --- a/data/generate_data.sh +++ b/data/generate_data.sh @@ -1,9 +1,9 @@ #!/bin/bash # ============================================================================ -# Financial Fraud Detection - Data Generation Script - IDEMPOTENT +# Business Analytics - Data Generation Script - IDEMPOTENT # ============================================================================ -# Generates realistic test data with embedded fraud patterns +# Generates realistic test data for business analytics and reporting # This script is IDEMPOTENT - it will clear and regenerate all data # ============================================================================ @@ -19,8 +19,8 @@ NC='\033[0m' # No Color # Configuration DB_HOST="${POSTGRES_HOST:-localhost}" DB_PORT="${POSTGRES_PORT:-5432}" -DB_NAME="${POSTGRES_DB:-fraud_detection}" -DB_USER="${POSTGRES_USER:-fraud_analyst}" +DB_NAME="${POSTGRES_DB:-business_analytics}" +DB_USER="${POSTGRES_USER:-data_analyst}" DB_PASSWORD="${POSTGRES_PASSWORD:-SecurePass123!}" # Data volumes @@ -33,7 +33,7 @@ NUM_TRANSACTIONS=5000000 FRAUD_PERCENTAGE=7 # 7% of transactions will be fraudulent echo -e "${BLUE}============================================================================${NC}" -echo -e "${BLUE}Financial Fraud Detection Database - Data Generation${NC}" +echo -e "${BLUE}Business Analytics Database - Data Generation${NC}" echo -e "${BLUE}============================================================================${NC}" echo -e "Target Database: ${GREEN}$DB_NAME@$DB_HOST:$DB_PORT${NC}" echo -e "Customers: ${GREEN}$NUM_CUSTOMERS${NC}" @@ -62,7 +62,7 @@ execute_sql_file() { SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" # Clear existing data (preserve reference tables) -echo -e "${YELLOW}[0/9] Clearing existing data...${NC}" +echo -e "${YELLOW}[0/15] Clearing existing data...${NC}" execute_sql "TRUNCATE TABLE audit_log CASCADE;" execute_sql "TRUNCATE TABLE suspicious_activity_reports CASCADE;" execute_sql "TRUNCATE TABLE case_alerts CASCADE;" @@ -83,12 +83,12 @@ echo -e "${GREEN}✓ Existing data cleared${NC}" echo "" # Load geographic reference data -echo -e "${YELLOW}[1/9] Loading geographic reference data...${NC}" +echo -e "${YELLOW}[1/15] Loading geographic reference data...${NC}" execute_sql_file "$SCRIPT_DIR/reference/load_geographic_data.sql" > /dev/null echo -e "${GREEN}✓ Geographic data loaded (100 US cities, 210 world cities)${NC}" echo "" -echo -e "${YELLOW}[2/9] Generating Customers...${NC}" +echo -e "${YELLOW}[2/15] Generating Customers...${NC}" cat > /tmp/generate_customers.sql << 'EOF' -- Generate customers with realistic geographic data INSERT INTO customers ( @@ -130,7 +130,7 @@ execute_sql_file /tmp/generate_customers.sql > /dev/null echo -e "${GREEN}✓ Generated $NUM_CUSTOMERS customers${NC}" echo "" -echo -e "${YELLOW}[3/9] Generating Accounts...${NC}" +echo -e "${YELLOW}[3/15] Generating Accounts...${NC}" cat > /tmp/generate_accounts.sql << 'EOF' -- Generate accounts (1-2 accounts per customer on average) INSERT INTO accounts ( @@ -177,7 +177,7 @@ execute_sql_file /tmp/generate_accounts.sql > /dev/null echo -e "${GREEN}✓ Generated $NUM_ACCOUNTS accounts${NC}" echo "" -echo -e "${YELLOW}[4/9] Generating Merchants...${NC}" +echo -e "${YELLOW}[4/15] Generating Merchants...${NC}" cat > /tmp/generate_merchants.sql << 'EOF' -- Generate merchants with realistic geographic data INSERT INTO merchants ( @@ -237,7 +237,7 @@ execute_sql_file /tmp/generate_merchants.sql > /dev/null echo -e "${GREEN}✓ Generated $NUM_MERCHANTS merchants${NC}" echo "" -echo -e "${YELLOW}[5/9] Generating Devices...${NC}" +echo -e "${YELLOW}[5/15] Generating Devices...${NC}" cat > /tmp/generate_devices.sql << 'EOF' -- Generate devices INSERT INTO devices ( @@ -282,7 +282,7 @@ execute_sql_file /tmp/generate_devices.sql > /dev/null echo -e "${GREEN}✓ Generated $NUM_DEVICES devices${NC}" echo "" -echo -e "${YELLOW}[6/9] Generating Cards...${NC}" +echo -e "${YELLOW}[6/15] Generating Cards...${NC}" cat > /tmp/generate_cards.sql << 'EOF' -- Generate cards (1-2 cards per account on average) INSERT INTO cards ( @@ -326,7 +326,7 @@ execute_sql_file /tmp/generate_cards.sql > /dev/null echo -e "${GREEN}✓ Generated $NUM_CARDS cards${NC}" echo "" -echo -e "${YELLOW}[7/9] Generating Login Sessions...${NC}" +echo -e "${YELLOW}[7/15] Generating Login Sessions...${NC}" cat > /tmp/generate_sessions.sql << 'EOF' -- Generate login sessions with realistic geographic data INSERT INTO login_sessions ( @@ -359,7 +359,7 @@ execute_sql_file /tmp/generate_sessions.sql > /dev/null echo -e "${GREEN}✓ Generated 500,000 login sessions${NC}" echo "" -echo -e "${YELLOW}[8/9] Generating Transactions (this may take a while)...${NC}" +echo -e "${YELLOW}[8/15] Generating Transactions (this may take a while)...${NC}" echo -e "${BLUE}This step generates $NUM_TRANSACTIONS transactions with fraud patterns${NC}" # Generate transactions in batches to avoid memory issues @@ -433,7 +433,7 @@ done echo -e "${GREEN}✓ Generated $NUM_TRANSACTIONS transactions${NC}" echo "" -echo -e "${YELLOW}[9/9] Generating Fraud Cases and Alerts...${NC}" +echo -e "${YELLOW}[9/15] Generating Fraud Cases and Alerts...${NC}" # Generate alerts for flagged transactions execute_sql " @@ -535,7 +535,261 @@ case_count=$(execute_sql "SELECT COUNT(*) FROM fraud_cases;" | grep -E '^\s*[0-9 echo -e "Fraud Cases: ${GREEN}$case_count${NC}" echo "" -echo -e "${YELLOW}Ready for SQL learning and fraud investigation!${NC}" +echo -e "${BLUE}============================================================================${NC}" +echo -e "${BLUE}Generating Analytics Data (Customer, Sales, KPIs)${NC}" +echo -e "${BLUE}============================================================================${NC}" +echo "" + +# ============================================================================ +# GENERATE CUSTOMER ANALYTICS DATA +# ============================================================================ + +echo -e "${YELLOW}[10/15] Generating Customer Lifetime Value data...${NC}" +cat > /tmp/generate_clv.sql << 'EOF' +-- Generate CLV for all customers based on their transaction history +INSERT INTO customer_lifetime_value ( + customer_id, calculation_date, total_revenue, total_transactions, + average_order_value, predicted_future_value, clv_score, segment_id +) +SELECT + c.customer_id, + CURRENT_DATE as calculation_date, + COALESCE(SUM(t.amount), 0) as total_revenue, + COUNT(t.transaction_id) as total_transactions, + COALESCE(AVG(t.amount), 0) as average_order_value, + COALESCE(SUM(t.amount) * 1.5, 0) as predicted_future_value, + COALESCE(SUM(t.amount) / 100, 0) as clv_score, + CASE + WHEN COALESCE(SUM(t.amount), 0) >= 50000 THEN 1 -- VIP + WHEN COALESCE(SUM(t.amount), 0) >= 10000 THEN 2 -- High Value + WHEN COALESCE(SUM(t.amount), 0) >= 2000 THEN 3 -- Medium Value + WHEN COALESCE(SUM(t.amount), 0) >= 500 THEN 4 -- Low Value + ELSE 6 -- New Customer + END as segment_id +FROM customers c +LEFT JOIN accounts a ON c.customer_id = a.customer_id +LEFT JOIN transactions t ON a.account_id = t.account_id +GROUP BY c.customer_id; +EOF + +execute_sql_file /tmp/generate_clv.sql > /dev/null +echo -e "${GREEN}✓ Generated CLV for all customers${NC}" +echo "" + +echo -e "${YELLOW}[11/15] Generating Churn Predictions...${NC}" +cat > /tmp/generate_churn.sql << 'EOF' +-- Generate churn predictions based on transaction recency +INSERT INTO churn_predictions ( + customer_id, prediction_date, churn_probability, risk_level, + last_transaction_date, days_since_last_transaction, engagement_score +) +SELECT + c.customer_id, + CURRENT_DATE as prediction_date, + CASE + WHEN MAX(t.transaction_date) IS NULL THEN 90 + WHEN CURRENT_DATE - MAX(t.transaction_date) > 180 THEN 85 + WHEN CURRENT_DATE - MAX(t.transaction_date) > 90 THEN 60 + WHEN CURRENT_DATE - MAX(t.transaction_date) > 30 THEN 30 + ELSE 10 + END as churn_probability, + CASE + WHEN MAX(t.transaction_date) IS NULL THEN 'HIGH' + WHEN CURRENT_DATE - MAX(t.transaction_date) > 180 THEN 'CRITICAL' + WHEN CURRENT_DATE - MAX(t.transaction_date) > 90 THEN 'HIGH' + WHEN CURRENT_DATE - MAX(t.transaction_date) > 30 THEN 'MEDIUM' + ELSE 'LOW' + END as risk_level, + MAX(t.transaction_date) as last_transaction_date, + COALESCE(CURRENT_DATE - MAX(t.transaction_date), 999) as days_since_last_transaction, + CASE + WHEN MAX(t.transaction_date) IS NULL THEN 0 + WHEN CURRENT_DATE - MAX(t.transaction_date) > 180 THEN 10 + WHEN CURRENT_DATE - MAX(t.transaction_date) > 90 THEN 30 + WHEN CURRENT_DATE - MAX(t.transaction_date) > 30 THEN 60 + ELSE 90 + END as engagement_score +FROM customers c +LEFT JOIN accounts a ON c.customer_id = a.customer_id +LEFT JOIN transactions t ON a.account_id = t.account_id +GROUP BY c.customer_id; +EOF + +execute_sql_file /tmp/generate_churn.sql > /dev/null +echo -e "${GREEN}✓ Generated churn predictions${NC}" +echo "" + +echo -e "${YELLOW}[12/15] Generating Customer Satisfaction data...${NC}" +cat > /tmp/generate_satisfaction.sql << 'EOF' +-- Generate satisfaction scores for random sample of customers +INSERT INTO customer_satisfaction ( + customer_id, survey_date, nps_score, csat_score, category, sentiment +) +SELECT + customer_id, + TIMESTAMP '2023-01-01' + (random() * 730)::INT * INTERVAL '1 day' as survey_date, + (random() * 200 - 100)::INT as nps_score, + (1 + random() * 4)::DECIMAL(3,2) as csat_score, + CASE (random() * 5)::INT + WHEN 0 THEN 'PRODUCT' + WHEN 1 THEN 'SERVICE' + WHEN 2 THEN 'SUPPORT' + WHEN 3 THEN 'BILLING' + ELSE 'OTHER' + END as category, + CASE + WHEN random() < 0.6 THEN 'POSITIVE' + WHEN random() < 0.85 THEN 'NEUTRAL' + ELSE 'NEGATIVE' + END as sentiment +FROM customers +WHERE random() < 0.3 -- 30% of customers have satisfaction data +LIMIT 30000; +EOF + +execute_sql_file /tmp/generate_satisfaction.sql > /dev/null +echo -e "${GREEN}✓ Generated ~30,000 satisfaction records${NC}" +echo "" + +# ============================================================================ +# GENERATE SALES ANALYTICS DATA +# ============================================================================ + +echo -e "${YELLOW}[13/15] Generating Sales Transactions...${NC}" +cat > /tmp/generate_sales.sql << 'EOF' +-- Link transactions to products +INSERT INTO sales_transactions ( + transaction_id, product_id, quantity, unit_price, discount_amount, + tax_amount, total_amount, sale_date, sales_channel, region +) +SELECT + t.transaction_id, + ((t.transaction_id % 24) + 1) as product_id, -- Cycle through 24 products + (1 + (random() * 3)::INT) as quantity, + t.amount / (1 + (random() * 3)::INT) as unit_price, + CASE WHEN random() < 0.2 THEN t.amount * 0.1 ELSE 0 END as discount_amount, + t.amount * 0.08 as tax_amount, + t.amount as total_amount, + t.transaction_date as sale_date, + CASE (t.transaction_id % 4) + WHEN 0 THEN 'ONLINE' + WHEN 1 THEN 'STORE' + WHEN 2 THEN 'PHONE' + ELSE 'MOBILE_APP' + END as sales_channel, + CASE (t.transaction_id % 5) + WHEN 0 THEN 'Northeast' + WHEN 1 THEN 'Southeast' + WHEN 2 THEN 'Midwest' + WHEN 3 THEN 'Southwest' + ELSE 'West' + END as region +FROM transactions t +WHERE t.status = 'COMPLETED' +LIMIT 1000000; -- Link 1M transactions to products +EOF + +execute_sql_file /tmp/generate_sales.sql > /dev/null +echo -e "${GREEN}✓ Generated 1,000,000 sales transaction records${NC}" +echo "" + +# ============================================================================ +# GENERATE KPI & METRICS DATA +# ============================================================================ + +echo -e "${YELLOW}[14/15] Generating Daily Metrics...${NC}" +cat > /tmp/generate_metrics.sql << 'EOF' +-- Generate daily metrics for the past 90 days +INSERT INTO daily_metrics ( + metric_date, kpi_id, metric_value, vs_previous_day_percentage, status +) +SELECT + date_series.metric_date, + kpi.kpi_id, + kpi.target_value * (0.8 + random() * 0.4) as metric_value, + (-20 + random() * 40)::DECIMAL(5,2) as vs_previous_day_percentage, + CASE + WHEN random() < 0.7 THEN 'ON_TARGET' + WHEN random() < 0.9 THEN 'WARNING' + ELSE 'CRITICAL' + END as status +FROM generate_series( + CURRENT_DATE - INTERVAL '90 days', + CURRENT_DATE, + INTERVAL '1 day' +) AS date_series(metric_date) +CROSS JOIN kpi_definitions kpi +WHERE kpi.is_active = TRUE; +EOF + +execute_sql_file /tmp/generate_metrics.sql > /dev/null +echo -e "${GREEN}✓ Generated 90 days of daily metrics${NC}" +echo "" + +echo -e "${YELLOW}[15/15] Generating Monthly Summaries...${NC}" +cat > /tmp/generate_monthly.sql << 'EOF' +-- Generate monthly summaries for the past 24 months +INSERT INTO monthly_summaries ( + summary_month, summary_year, total_revenue, total_transactions, + total_customers, new_customers, average_transaction_value +) +SELECT + EXTRACT(MONTH FROM month_series)::INT as summary_month, + EXTRACT(YEAR FROM month_series)::INT as summary_year, + (10000000 + random() * 5000000)::DECIMAL(15,2) as total_revenue, + (50000 + (random() * 30000)::INT) as total_transactions, + (80000 + (random() * 20000)::INT) as total_customers, + (500 + (random() * 1500)::INT) as new_customers, + (100 + random() * 100)::DECIMAL(15,2) as average_transaction_value +FROM generate_series( + CURRENT_DATE - INTERVAL '24 months', + CURRENT_DATE, + INTERVAL '1 month' +) AS month_series; +EOF + +execute_sql_file /tmp/generate_monthly.sql > /dev/null +echo -e "${GREEN}✓ Generated 24 months of summaries${NC}" +echo "" + +echo -e "${BLUE}============================================================================${NC}" +echo -e "${GREEN}Data Generation Complete!${NC}" +echo -e "${BLUE}============================================================================${NC}" +echo "" +echo -e "${YELLOW}Database Statistics:${NC}" + +customer_count=$(execute_sql "SELECT COUNT(*) FROM customers;" | grep -E '^\s*[0-9]+' | tr -d ' ') +echo -e "Customers: ${GREEN}$customer_count${NC}" + +account_count=$(execute_sql "SELECT COUNT(*) FROM accounts;" | grep -E '^\s*[0-9]+' | tr -d ' ') +echo -e "Accounts: ${GREEN}$account_count${NC}" + +merchant_count=$(execute_sql "SELECT COUNT(*) FROM merchants;" | grep -E '^\s*[0-9]+' | tr -d ' ') +echo -e "Merchants: ${GREEN}$merchant_count${NC}" + +card_count=$(execute_sql "SELECT COUNT(*) FROM cards;" | grep -E '^\s*[0-9]+' | tr -d ' ') +echo -e "Cards: ${GREEN}$card_count${NC}" + +transaction_count=$(execute_sql "SELECT COUNT(*) FROM transactions;" | grep -E '^\s*[0-9]+' | tr -d ' ') +echo -e "Transactions: ${GREEN}$transaction_count${NC}" + +alert_count=$(execute_sql "SELECT COUNT(*) FROM alerts;" | grep -E '^\s*[0-9]+' | tr -d ' ') +echo -e "Alerts: ${GREEN}$alert_count${NC}" + +case_count=$(execute_sql "SELECT COUNT(*) FROM fraud_cases;" | grep -E '^\s*[0-9]+' | tr -d ' ') +echo -e "Fraud Cases: ${GREEN}$case_count${NC}" + +clv_count=$(execute_sql "SELECT COUNT(*) FROM customer_lifetime_value;" | grep -E '^\s*[0-9]+' | tr -d ' ') +echo -e "Customer CLV Records: ${GREEN}$clv_count${NC}" + +sales_count=$(execute_sql "SELECT COUNT(*) FROM sales_transactions;" | grep -E '^\s*[0-9]+' | tr -d ' ') +echo -e "Sales Records: ${GREEN}$sales_count${NC}" + +metrics_count=$(execute_sql "SELECT COUNT(*) FROM daily_metrics;" | grep -E '^\s*[0-9]+' | tr -d ' ') +echo -e "Daily Metrics: ${GREEN}$metrics_count${NC}" + +echo "" +echo -e "${YELLOW}Ready for Business Analytics and SQL learning!${NC}" echo -e "Access DB-UI at: ${BLUE}http://localhost:3000${NC}" echo "" diff --git a/docker-compose.yml b/docker-compose.yml index 33bb115..368a905 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -3,10 +3,10 @@ version: '3.8' services: postgres: image: postgres:16-alpine - container_name: fraud_detection_db + container_name: business_analytics_db environment: - POSTGRES_DB: fraud_detection - POSTGRES_USER: fraud_analyst + POSTGRES_DB: business_analytics + POSTGRES_USER: data_analyst POSTGRES_PASSWORD: SecurePass123! PGDATA: /var/lib/postgresql/data/pgdata ports: @@ -16,9 +16,9 @@ services: - ./schema:/docker-entrypoint-initdb.d - ./data:/data networks: - - fraud_network + - analytics_network healthcheck: - test: ["CMD-SHELL", "pg_isready -U fraud_analyst -d fraud_detection"] + test: ["CMD-SHELL", "pg_isready -U data_analyst -d business_analytics"] interval: 10s timeout: 5s retries: 5 @@ -26,24 +26,24 @@ services: db-ui: image: ghcr.io/n7olkachev/db-ui:latest - container_name: fraud_detection_ui + container_name: business_analytics_ui environment: POSTGRES_HOST: postgres - POSTGRES_USER: fraud_analyst + POSTGRES_USER: data_analyst POSTGRES_PASSWORD: SecurePass123! - POSTGRES_DB: fraud_detection + POSTGRES_DB: business_analytics POSTGRES_PORT: 5432 ports: - "3000:3000" networks: - - fraud_network + - analytics_network depends_on: postgres: condition: service_healthy restart: unless-stopped networks: - fraud_network: + analytics_network: driver: bridge volumes: diff --git a/docs/DATA_MODELS.md b/docs/DATA_MODELS.md new file mode 100644 index 0000000..7ed3d5a --- /dev/null +++ b/docs/DATA_MODELS.md @@ -0,0 +1,358 @@ +# Business Analytics Database - Data Models + +## Overview +This database supports **Data Analyst** activities including: +- Routine and semi-routine analysis +- Dashboard creation and maintenance +- Business trend identification +- Statistical analysis and pattern recognition +- Stakeholder reporting + +--- + +## Model 1: Customer Analytics (Routine Analysis) + +### Purpose +Support customer segmentation, lifetime value analysis, churn prediction, and engagement tracking. + +### Tables + +#### `customer_segments` +Customer classification for targeted analysis +```sql +- segment_id (PK) +- segment_name (e.g., 'High Value', 'At Risk', 'New Customer') +- segment_description +- criteria_definition (JSON) +- created_date +- updated_date +``` + +#### `customer_lifetime_value` +CLV calculations for business insights +```sql +- clv_id (PK) +- customer_id (FK → customers) +- calculation_date +- total_revenue +- total_transactions +- average_order_value +- predicted_future_value +- clv_score +- segment_id (FK → customer_segments) +``` + +#### `churn_predictions` +Customer retention analysis +```sql +- prediction_id (PK) +- customer_id (FK → customers) +- prediction_date +- churn_probability (0-100) +- risk_level (LOW, MEDIUM, HIGH, CRITICAL) +- last_transaction_date +- days_since_last_transaction +- engagement_score +- recommended_action +``` + +#### `customer_satisfaction` +Satisfaction scores and feedback +```sql +- satisfaction_id (PK) +- customer_id (FK → customers) +- survey_date +- nps_score (-100 to 100) +- csat_score (1-5) +- feedback_text +- category (PRODUCT, SERVICE, SUPPORT, etc.) +- sentiment (POSITIVE, NEUTRAL, NEGATIVE) +``` + +#### `engagement_metrics` +Customer interaction tracking +```sql +- metric_id (PK) +- customer_id (FK → customers) +- metric_date +- login_count +- page_views +- time_spent_minutes +- features_used +- support_tickets_opened +- engagement_score +``` + +--- + +## Model 2: Sales & Revenue Analytics (Semi-Routine Reporting) + +### Purpose +Support sales performance tracking, revenue forecasting, and product analysis. + +### Tables + +#### `product_catalog` +Product master data +```sql +- product_id (PK) +- product_name +- product_category +- product_subcategory +- unit_price +- cost_price +- margin_percentage +- is_active +- launch_date +- discontinued_date +``` + +#### `sales_transactions` +Detailed sales records (extends existing transactions) +```sql +- sale_id (PK) +- transaction_id (FK → transactions) +- product_id (FK → product_catalog) +- quantity +- unit_price +- discount_amount +- tax_amount +- total_amount +- sale_date +- sales_channel (ONLINE, STORE, PHONE, MOBILE_APP) +- sales_rep_id +- region +``` + +#### `sales_targets` +Performance goals for tracking +```sql +- target_id (PK) +- target_period (DAILY, WEEKLY, MONTHLY, QUARTERLY, YEARLY) +- start_date +- end_date +- product_category +- region +- target_revenue +- target_units +- target_customers +- created_by +``` + +#### `sales_performance` +Aggregated performance metrics +```sql +- performance_id (PK) +- period_date +- period_type (DAILY, WEEKLY, MONTHLY, QUARTERLY) +- product_id (FK → product_catalog) +- region +- total_revenue +- total_units_sold +- total_transactions +- unique_customers +- average_order_value +- vs_target_percentage +``` + +#### `revenue_forecasts` +Predictive revenue analysis +```sql +- forecast_id (PK) +- forecast_date +- forecast_period_start +- forecast_period_end +- product_category +- region +- forecasted_revenue +- confidence_level (LOW, MEDIUM, HIGH) +- forecast_method (HISTORICAL, TREND, SEASONAL, ML) +- actual_revenue (filled after period ends) +- variance_percentage +``` + +--- + +## Model 3: Operational Metrics & KPIs (Dashboard Creation) + +### Purpose +Support dashboard creation with pre-calculated KPIs and trend analysis. + +### Tables + +#### `kpi_definitions` +Master list of tracked KPIs +```sql +- kpi_id (PK) +- kpi_name +- kpi_description +- kpi_category (SALES, CUSTOMER, OPERATIONAL, FINANCIAL) +- calculation_formula +- target_value +- threshold_warning +- threshold_critical +- unit_of_measure +- refresh_frequency (REALTIME, HOURLY, DAILY, WEEKLY) +- is_active +``` + +#### `daily_metrics` +Daily operational snapshots +```sql +- metric_id (PK) +- metric_date +- kpi_id (FK → kpi_definitions) +- metric_value +- vs_previous_day_percentage +- vs_previous_week_percentage +- vs_previous_month_percentage +- status (ON_TARGET, WARNING, CRITICAL) +- notes +``` + +#### `monthly_summaries` +Monthly aggregated data for reporting +```sql +- summary_id (PK) +- summary_month +- summary_year +- total_revenue +- total_transactions +- total_customers +- new_customers +- churned_customers +- average_transaction_value +- customer_acquisition_cost +- customer_lifetime_value +- net_promoter_score +- gross_margin_percentage +``` + +#### `trend_analysis` +Statistical trend tracking +```sql +- trend_id (PK) +- kpi_id (FK → kpi_definitions) +- analysis_date +- period_start +- period_end +- trend_direction (UP, DOWN, FLAT) +- trend_strength (WEAK, MODERATE, STRONG) +- moving_average_7day +- moving_average_30day +- seasonality_detected +- anomalies_detected +- statistical_significance +``` + +#### `dashboard_snapshots` +Pre-calculated dashboard data +```sql +- snapshot_id (PK) +- dashboard_name +- snapshot_timestamp +- data_payload (JSON) +- refresh_duration_seconds +- row_count +- last_updated_by +``` + +--- + +## Model 4: Business Intelligence & Reporting + +### Purpose +Support ad-hoc analysis and standard report generation. + +### Tables + +#### `report_definitions` +Catalog of standard reports +```sql +- report_id (PK) +- report_name +- report_description +- report_category +- sql_query_template +- parameters (JSON) +- output_format (PDF, EXCEL, CSV, HTML) +- schedule_frequency +- recipients +- is_active +``` + +#### `report_executions` +Report run history +```sql +- execution_id (PK) +- report_id (FK → report_definitions) +- execution_timestamp +- parameters_used (JSON) +- row_count +- execution_duration_seconds +- status (SUCCESS, FAILED, TIMEOUT) +- error_message +- output_file_path +- executed_by +``` + +#### `data_quality_checks` +Data validation tracking +```sql +- check_id (PK) +- check_name +- table_name +- column_name +- check_type (NULL_CHECK, RANGE_CHECK, UNIQUENESS, REFERENTIAL_INTEGRITY) +- check_date +- records_checked +- records_failed +- failure_percentage +- status (PASS, FAIL, WARNING) +- remediation_notes +``` + +--- + +## Relationships to Existing Models + +### Integration Points + +1. **Customer Analytics** ← links to existing `customers` table +2. **Sales Transactions** ← extends existing `transactions` table +3. **Engagement Metrics** ← links to `login_sessions` and `transactions` +4. **Churn Predictions** ← analyzes `transactions` and `accounts` +5. **KPIs** ← aggregates from multiple existing tables + +--- + +## Use Cases for Data Analyst Role + +### Routine Analysis +- Daily sales reports +- Customer segment updates +- KPI dashboard refreshes +- Data quality checks + +### Semi-Routine Analysis +- Monthly trend analysis +- Churn prediction updates +- Revenue forecasting +- Performance vs. targets + +### Ad-Hoc Analysis +- Customer cohort analysis +- Product performance deep-dives +- Seasonal pattern identification +- Anomaly investigation + +--- + +## Next Steps + +1. ✅ Design complete +2. ⏳ Implement schema in `schema/01-create-tables.sql` +3. ⏳ Update data generation scripts +4. ⏳ Create SQL exercises for each model +5. ⏳ Update documentation + diff --git a/docs/QUICKSTART.md b/docs/QUICKSTART.md index 99bb6c6..2f83d9a 100644 --- a/docs/QUICKSTART.md +++ b/docs/QUICKSTART.md @@ -19,7 +19,7 @@ docker-compose up -d docker-compose ps ``` -You should see both `fraud_detection_db` and `fraud_detection_ui` running. +You should see both `business_analytics_db` and `business_analytics_ui` running. --- @@ -34,9 +34,9 @@ chmod +x scripts/setup-database.sh ``` **What this does:** -- Creates all 20+ tables +- Creates all 39 tables across 4 business models - Sets up indexes and constraints -- Loads reference data (countries, merchant categories, etc.) +- Loads reference data (countries, merchant categories, customer segments, products, KPIs, etc.) **Expected output:** ``` @@ -63,6 +63,9 @@ chmod +x data/generate_data.sh - Creates 150,000 accounts - Generates 5,000,000 transactions - Creates fraud patterns and alerts +- Generates customer analytics (CLV, churn, satisfaction) +- Creates sales data (1M sales records) +- Generates KPI metrics (90 days of daily metrics) **⏱️ Time estimate:** - Fast machine (SSD, 16GB RAM): ~15 minutes @@ -72,10 +75,16 @@ chmod +x data/generate_data.sh **You can monitor progress:** The script shows progress for each step: ``` -[1/9] Loading geographic reference data... -[2/9] Generating Customers... -[3/9] Generating Accounts... +[1/15] Loading geographic reference data... +[2/15] Generating Customers... +[3/15] Generating Accounts... ... +[10/15] Generating Customer Lifetime Value data... +[11/15] Generating Churn Predictions... +[12/15] Generating Customer Satisfaction data... +[13/15] Generating Sales Transactions... +[14/15] Generating Daily Metrics... +[15/15] Generating Monthly Summaries... ``` --- @@ -98,7 +107,7 @@ The script shows progress for each step: #### Option B: Command Line (psql) ```bash -docker exec -it fraud_detection_db psql -U fraud_analyst -d fraud_detection +docker exec -it business_analytics_db psql -U data_analyst -d business_analytics ``` **Quick commands:** @@ -122,8 +131,8 @@ SELECT COUNT(*) FROM transactions; ``` Host: localhost Port: 5432 -Database: fraud_detection -Username: fraud_analyst +Database: business_analytics +Username: data_analyst Password: SecurePass123! ``` @@ -146,43 +155,65 @@ SELECT COUNT(*) FROM customers; -- How many transactions? SELECT COUNT(*) FROM transactions; --- How many fraud alerts? -SELECT COUNT(*) FROM alerts WHERE status = 'OPEN'; +-- How many sales records? +SELECT COUNT(*) FROM sales_transactions; + +-- How many KPIs are being tracked? +SELECT COUNT(*) FROM kpi_definitions WHERE is_active = TRUE; ``` -### 2. Find High-Risk Customers +### 2. Customer Analytics: High-Value Customers ```sql -SELECT - customer_id, - first_name, - last_name, - email, - risk_score -FROM customers -WHERE risk_score > 80 -ORDER BY risk_score DESC +SELECT + c.customer_id, + c.first_name, + c.last_name, + clv.clv_score, + cs.segment_name +FROM customers c +JOIN customer_lifetime_value clv ON c.customer_id = clv.customer_id +JOIN customer_segments cs ON clv.segment_id = cs.segment_id +WHERE cs.segment_name IN ('VIP', 'High Value') +ORDER BY clv.clv_score DESC LIMIT 10; ``` -### 3. View Recent Transactions +### 3. Sales Analytics: Top Products ```sql -SELECT - transaction_id, - account_id, - amount, - transaction_date, - is_flagged -FROM transactions -ORDER BY transaction_date DESC -LIMIT 20; +SELECT + p.product_name, + p.product_category, + COUNT(st.transaction_id) as sales_count, + SUM(st.total_amount) as total_revenue +FROM product_catalog p +JOIN sales_transactions st ON p.product_id = st.product_id +GROUP BY p.product_id, p.product_name, p.product_category +ORDER BY total_revenue DESC +LIMIT 10; ``` -### 4. Find Flagged Transactions +### 4. KPI Dashboard: Current Status ```sql -SELECT +SELECT + kd.kpi_name, + kd.kpi_category, + dm.metric_value, + kd.target_value, + dm.status +FROM kpi_definitions kd +JOIN daily_metrics dm ON kd.kpi_id = dm.kpi_id +WHERE dm.metric_date = CURRENT_DATE + AND kd.is_active = TRUE +ORDER BY kd.kpi_category, kd.kpi_name; +``` + +### 5. Fraud Detection: Flagged Transactions + +```sql +SELECT t.transaction_id, t.amount, t.fraud_score, diff --git a/docs/WHATS_NEW.md b/docs/WHATS_NEW.md new file mode 100644 index 0000000..d2078f2 --- /dev/null +++ b/docs/WHATS_NEW.md @@ -0,0 +1,266 @@ +# What's New - Business Analytics Database + +## 🎯 Major Update: Aligned with Data Analyst Role + +The database has been **expanded and refocused** to align with real-world **Data Analyst** responsibilities: + +- ✅ Routine and semi-routine analysis +- ✅ Dashboard creation and maintenance +- ✅ Business trend identification +- ✅ Statistical analysis and pattern recognition +- ✅ Stakeholder reporting + +--- + +## 📊 Database Renamed + +**Old Name:** `fraud_detection` +**New Name:** `business_analytics` + +**User Changed:** `fraud_analyst` → `data_analyst` + +This better reflects the broader scope of business intelligence and analytics work. + +--- + +## 🆕 New Data Models Added + +### 1. Customer Analytics Model (5 Tables) +**Purpose:** Routine customer analysis and segmentation + +#### New Tables: +- **`customer_segments`** - Customer classification definitions +- **`customer_lifetime_value`** - CLV calculations and tracking +- **`churn_predictions`** - Customer retention risk analysis +- **`customer_satisfaction`** - NPS, CSAT scores, and feedback +- **`engagement_metrics`** - Customer interaction tracking + +**Use Cases:** +- Customer segmentation analysis +- Churn risk identification +- Lifetime value calculations +- Satisfaction trend analysis +- Engagement scoring + +--- + +### 2. Sales & Revenue Analytics Model (6 Tables) +**Purpose:** Semi-routine sales reporting and forecasting + +#### New Tables: +- **`product_catalog`** - Product master data +- **`sales_transactions`** - Detailed sales records +- **`sales_targets`** - Performance goals and targets +- **`sales_performance`** - Aggregated performance metrics +- **`revenue_forecasts`** - Revenue predictions and variance + +**Use Cases:** +- Sales performance dashboards +- Product analysis +- Revenue forecasting +- Target vs. actual analysis +- Channel performance comparison + +--- + +### 3. Operational Metrics & KPI Model (5 Tables) +**Purpose:** Dashboard creation and KPI tracking + +#### New Tables: +- **`kpi_definitions`** - Master KPI catalog +- **`daily_metrics`** - Daily operational snapshots +- **`monthly_summaries`** - Monthly business summaries +- **`trend_analysis`** - Statistical trend tracking +- **`dashboard_snapshots`** - Pre-calculated dashboard data + +**Use Cases:** +- Executive dashboards +- KPI monitoring +- Trend analysis +- Performance tracking +- Anomaly detection + +--- + +### 4. Business Intelligence & Reporting Model (3 Tables) +**Purpose:** Report management and data quality + +#### New Tables: +- **`report_definitions`** - Standard report catalog +- **`report_executions`** - Report run history +- **`data_quality_checks`** - Data validation tracking + +**Use Cases:** +- Report scheduling +- Execution monitoring +- Data quality assurance +- Audit trails + +--- + +## 📈 Total Database Size + +### Original (Fraud Detection Only): +- **20 tables** focused on fraud investigation + +### Updated (Business Analytics): +- **39 tables** covering: + - Fraud detection (original 20 tables) + - Customer analytics (5 tables) + - Sales & revenue (6 tables) + - KPIs & metrics (5 tables) + - Reporting & BI (3 tables) + +--- + +## 🔗 Integration with Existing Data + +The new models **integrate seamlessly** with existing tables: + +``` +customers → customer_lifetime_value + → churn_predictions + → customer_satisfaction + → engagement_metrics + +transactions → sales_transactions + → sales_performance + → revenue_forecasts + +(all tables) → kpi_definitions + → daily_metrics + → monthly_summaries +``` + +--- + +## 💼 Data Analyst Workflows Supported + +### Daily Routine Tasks +1. **Refresh dashboards** using `dashboard_snapshots` +2. **Update daily metrics** in `daily_metrics` +3. **Run data quality checks** via `data_quality_checks` +4. **Generate standard reports** from `report_definitions` + +### Weekly Semi-Routine Tasks +1. **Customer segmentation** analysis using `customer_segments` and `customer_lifetime_value` +2. **Sales performance** review via `sales_performance` vs `sales_targets` +3. **Churn prediction** updates in `churn_predictions` +4. **Trend analysis** using `trend_analysis` table + +### Monthly Analysis +1. **Monthly summaries** generation in `monthly_summaries` +2. **Revenue forecasting** via `revenue_forecasts` +3. **Customer satisfaction** trend analysis +4. **KPI performance** review + +### Ad-Hoc Analysis +1. **Product performance** deep-dives +2. **Customer cohort** analysis +3. **Seasonal pattern** identification +4. **Anomaly investigation** + +--- + +## 🎓 New Learning Opportunities + +### For Beginners: +- Basic aggregations (SUM, AVG, COUNT) +- Simple JOINs across analytics tables +- Date-based filtering and grouping +- KPI calculations + +### For Intermediate: +- Customer segmentation queries +- Sales trend analysis +- Moving averages and window functions +- Cohort analysis + +### For Advanced: +- Churn prediction analysis +- Revenue forecasting validation +- Multi-dimensional analysis +- Statistical significance testing + +--- + +## 🔄 Migration Notes + +### What Changed: +- ✅ Database name: `fraud_detection` → `business_analytics` +- ✅ User name: `fraud_analyst` → `data_analyst` +- ✅ Container names updated +- ✅ All scripts updated +- ✅ 19 new tables added + +### What Stayed the Same: +- ✅ All original 20 fraud detection tables +- ✅ All existing data generation logic +- ✅ All existing indexes and constraints +- ✅ Idempotent script design +- ✅ Docker setup structure + +### Backward Compatibility: +- ✅ All original fraud detection exercises still work +- ✅ All original queries still valid +- ✅ No breaking changes to existing schema + +--- + +## 📝 Next Steps + +### Immediate: +1. ✅ Schema updated with 19 new tables +2. ⏳ Update data generation scripts +3. ⏳ Create SQL exercises for new models +4. ⏳ Update main documentation + +### Future Enhancements: +- Add sample data for new tables +- Create dashboard query examples +- Build KPI calculation examples +- Add data quality check templates + +--- + +## 🚀 Getting Started with New Models + +### Quick Test Queries: + +```sql +-- Check new tables exist +SELECT table_name +FROM information_schema.tables +WHERE table_schema = 'public' + AND table_name IN ( + 'customer_segments', + 'customer_lifetime_value', + 'product_catalog', + 'sales_transactions', + 'kpi_definitions', + 'daily_metrics' + ); + +-- View table counts +SELECT + 'customer_segments' as table_name, COUNT(*) FROM customer_segments +UNION ALL +SELECT 'product_catalog', COUNT(*) FROM product_catalog +UNION ALL +SELECT 'kpi_definitions', COUNT(*) FROM kpi_definitions; +``` + +--- + +## 📚 Documentation Updates + +- ✅ **DATA_MODELS.md** - Complete model documentation +- ✅ **WHATS_NEW.md** - This file +- ⏳ **README.md** - Update with new scope +- ⏳ **QUICKSTART.md** - Add new model examples +- ⏳ **Exercises** - Create analytics-focused exercises + +--- + +**The database is now a comprehensive Business Analytics platform suitable for Data Analyst training and real-world analysis scenarios!** 🎉 + diff --git a/exercises/02-customer-analytics/README.md b/exercises/02-customer-analytics/README.md new file mode 100644 index 0000000..b030a1d --- /dev/null +++ b/exercises/02-customer-analytics/README.md @@ -0,0 +1,325 @@ +# Level 2: Customer Analytics + +## Introduction +Learn to analyze customer behavior, calculate lifetime value, identify churn risks, and segment customers - essential skills for Data Analysts working with customer data. + +## Learning Objectives +- Calculate customer lifetime value (CLV) +- Identify at-risk customers +- Segment customers by behavior +- Analyze satisfaction trends +- Track engagement metrics + +--- + +## Exercise 2.1: Customer Segmentation +**Objective:** Find the distribution of customers across segments + +```sql +SELECT + cs.segment_name, + COUNT(clv.customer_id) as customer_count, + AVG(clv.clv_score) as avg_clv_score, + SUM(clv.total_revenue) as total_segment_revenue +FROM customer_segments cs +LEFT JOIN customer_lifetime_value clv ON cs.segment_id = clv.segment_id +GROUP BY cs.segment_id, cs.segment_name +ORDER BY total_segment_revenue DESC; +``` + +**Expected Result:** Segment breakdown with counts and revenue + +--- + +## Exercise 2.2: High-Value Customers +**Objective:** Identify top 10 customers by lifetime value + +```sql +-- Your query here + +``` + +**Hint:** Use customer_lifetime_value table and ORDER BY + +**Solution:** +```sql +SELECT + c.customer_id, + c.first_name, + c.last_name, + c.email, + clv.total_revenue, + clv.total_transactions, + clv.clv_score, + cs.segment_name +FROM customer_lifetime_value clv +JOIN customers c ON clv.customer_id = c.customer_id +JOIN customer_segments cs ON clv.segment_id = cs.segment_id +ORDER BY clv.clv_score DESC +LIMIT 10; +``` + +--- + +## Exercise 2.3: Churn Risk Analysis +**Objective:** Find customers at CRITICAL or HIGH churn risk + +```sql +-- Your query here + +``` + +**Solution:** +```sql +SELECT + c.customer_id, + c.first_name, + c.last_name, + c.email, + cp.churn_probability, + cp.risk_level, + cp.days_since_last_transaction, + cp.engagement_score +FROM churn_predictions cp +JOIN customers c ON cp.customer_id = c.customer_id +WHERE cp.risk_level IN ('CRITICAL', 'HIGH') +ORDER BY cp.churn_probability DESC; +``` + +--- + +## Exercise 2.4: Customer Satisfaction Trends +**Objective:** Calculate average NPS score by month + +```sql +-- Your query here + +``` + +**Hint:** Use DATE_TRUNC or EXTRACT to group by month + +**Solution:** +```sql +SELECT + DATE_TRUNC('month', survey_date) as month, + COUNT(*) as survey_count, + AVG(nps_score) as avg_nps, + AVG(csat_score) as avg_csat, + COUNT(CASE WHEN sentiment = 'POSITIVE' THEN 1 END) as positive_count, + COUNT(CASE WHEN sentiment = 'NEGATIVE' THEN 1 END) as negative_count +FROM customer_satisfaction +GROUP BY DATE_TRUNC('month', survey_date) +ORDER BY month DESC; +``` + +--- + +## Exercise 2.5: Engagement Score Analysis +**Objective:** Find customers with declining engagement + +```sql +-- Your query here + +``` + +**Hint:** Compare recent engagement to historical average + +**Solution:** +```sql +WITH recent_engagement AS ( + SELECT + customer_id, + AVG(engagement_score) as recent_score + FROM engagement_metrics + WHERE metric_date >= CURRENT_DATE - INTERVAL '30 days' + GROUP BY customer_id +), +historical_engagement AS ( + SELECT + customer_id, + AVG(engagement_score) as historical_score + FROM engagement_metrics + WHERE metric_date < CURRENT_DATE - INTERVAL '30 days' + GROUP BY customer_id +) +SELECT + c.customer_id, + c.first_name, + c.last_name, + re.recent_score, + he.historical_score, + (re.recent_score - he.historical_score) as score_change +FROM customers c +JOIN recent_engagement re ON c.customer_id = re.customer_id +JOIN historical_engagement he ON c.customer_id = he.customer_id +WHERE re.recent_score < he.historical_score +ORDER BY score_change ASC +LIMIT 20; +``` + +--- + +## Exercise 2.6: Customer Cohort Analysis +**Objective:** Analyze customer retention by registration cohort + +```sql +-- Your query here + +``` + +**Solution:** +```sql +SELECT + DATE_TRUNC('month', c.registration_date) as cohort_month, + COUNT(DISTINCT c.customer_id) as total_customers, + COUNT(DISTINCT CASE WHEN cp.risk_level = 'LOW' THEN c.customer_id END) as active_customers, + COUNT(DISTINCT CASE WHEN cp.risk_level IN ('HIGH', 'CRITICAL') THEN c.customer_id END) as at_risk_customers, + ROUND(100.0 * COUNT(DISTINCT CASE WHEN cp.risk_level = 'LOW' THEN c.customer_id END) / + COUNT(DISTINCT c.customer_id), 2) as retention_rate +FROM customers c +LEFT JOIN churn_predictions cp ON c.customer_id = cp.customer_id +WHERE c.registration_date >= CURRENT_DATE - INTERVAL '12 months' +GROUP BY DATE_TRUNC('month', c.registration_date) +ORDER BY cohort_month DESC; +``` + +--- + +## Exercise 2.7: Customer Lifetime Value by Segment +**Objective:** Compare CLV metrics across customer segments + +```sql +-- Your query here + +``` + +**Solution:** +```sql +SELECT + cs.segment_name, + COUNT(clv.customer_id) as customer_count, + MIN(clv.clv_score) as min_clv, + AVG(clv.clv_score) as avg_clv, + MAX(clv.clv_score) as max_clv, + PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY clv.clv_score) as median_clv, + SUM(clv.total_revenue) as total_revenue, + AVG(clv.total_transactions) as avg_transactions +FROM customer_segments cs +JOIN customer_lifetime_value clv ON cs.segment_id = clv.segment_id +GROUP BY cs.segment_id, cs.segment_name +ORDER BY avg_clv DESC; +``` + +--- + +## Exercise 2.8: Satisfaction by Category +**Objective:** Analyze satisfaction scores by feedback category + +```sql +-- Your query here + +``` + +**Solution:** +```sql +SELECT + category, + COUNT(*) as feedback_count, + AVG(nps_score) as avg_nps, + AVG(csat_score) as avg_csat, + ROUND(100.0 * COUNT(CASE WHEN sentiment = 'POSITIVE' THEN 1 END) / COUNT(*), 2) as positive_pct, + ROUND(100.0 * COUNT(CASE WHEN sentiment = 'NEGATIVE' THEN 1 END) / COUNT(*), 2) as negative_pct +FROM customer_satisfaction +GROUP BY category +ORDER BY avg_nps DESC; +``` + +--- + +## Exercise 2.9: Customer Engagement Patterns +**Objective:** Find most engaged customers in the last 30 days + +```sql +-- Your query here + +``` + +**Solution:** +```sql +SELECT + c.customer_id, + c.first_name, + c.last_name, + SUM(em.login_count) as total_logins, + SUM(em.page_views) as total_page_views, + SUM(em.time_spent_minutes) as total_time_spent, + AVG(em.engagement_score) as avg_engagement_score +FROM customers c +JOIN engagement_metrics em ON c.customer_id = em.customer_id +WHERE em.metric_date >= CURRENT_DATE - INTERVAL '30 days' +GROUP BY c.customer_id, c.first_name, c.last_name +HAVING AVG(em.engagement_score) > 75 +ORDER BY avg_engagement_score DESC +LIMIT 20; +``` + +--- + +## Exercise 2.10: Churn Prevention Priority List +**Objective:** Create a priority list for customer retention efforts + +```sql +-- Your query here + +``` + +**Hint:** Combine churn risk with customer value + +**Solution:** +```sql +SELECT + c.customer_id, + c.first_name, + c.last_name, + c.email, + clv.clv_score, + clv.total_revenue, + cp.churn_probability, + cp.risk_level, + cp.days_since_last_transaction, + (clv.clv_score * cp.churn_probability / 100) as retention_priority_score +FROM customers c +JOIN customer_lifetime_value clv ON c.customer_id = clv.customer_id +JOIN churn_predictions cp ON c.customer_id = cp.customer_id +WHERE cp.risk_level IN ('HIGH', 'CRITICAL') + AND clv.clv_score > 1000 +ORDER BY retention_priority_score DESC +LIMIT 50; +``` + +--- + +## Challenge Exercises + +### Challenge 2.1: Customer Journey Analysis +Create a query that shows the customer journey from registration to current status, including: +- Registration date +- First transaction date +- Total transactions +- Current segment +- Churn risk +- Latest satisfaction score + +### Challenge 2.2: Segment Migration Analysis +Identify customers who have moved between segments over time (requires historical CLV data) + +### Challenge 2.3: Engagement Correlation +Analyze the correlation between engagement scores and satisfaction scores + +--- + +## Next Steps +Once you're comfortable with customer analytics, move on to: +- **Level 3:** Sales & Revenue Analysis +- **Level 4:** KPI Dashboards & Metrics + diff --git a/exercises/03-sales-analysis/README.md b/exercises/03-sales-analysis/README.md new file mode 100644 index 0000000..f1f2a43 --- /dev/null +++ b/exercises/03-sales-analysis/README.md @@ -0,0 +1,380 @@ +# Level 3: Sales & Revenue Analysis + +## Introduction +Master sales performance analysis, product analytics, and revenue forecasting - core responsibilities for Data Analysts in sales-driven organizations. + +## Learning Objectives +- Analyze sales performance vs targets +- Identify top-performing products +- Compare sales across channels and regions +- Calculate revenue metrics +- Analyze sales trends + +--- + +## Exercise 3.1: Daily Sales Summary +**Objective:** Get today's sales summary + +```sql +SELECT + COUNT(*) as total_sales, + SUM(total_amount) as total_revenue, + AVG(total_amount) as avg_sale_value, + SUM(quantity) as total_units_sold, + COUNT(DISTINCT product_id) as products_sold +FROM sales_transactions +WHERE DATE(sale_date) = CURRENT_DATE; +``` + +**Expected Result:** Summary of today's sales activity + +--- + +## Exercise 3.2: Top Products by Revenue +**Objective:** Find the top 10 products by total revenue + +```sql +-- Your query here + +``` + +**Solution:** +```sql +SELECT + p.product_id, + p.product_name, + p.product_category, + p.product_subcategory, + COUNT(st.transaction_id) as sales_count, + SUM(st.quantity) as units_sold, + SUM(st.total_amount) as total_revenue, + AVG(st.total_amount) as avg_sale_value +FROM product_catalog p +JOIN sales_transactions st ON p.product_id = st.product_id +GROUP BY p.product_id, p.product_name, p.product_category, p.product_subcategory +ORDER BY total_revenue DESC +LIMIT 10; +``` + +--- + +## Exercise 3.3: Sales by Channel +**Objective:** Compare sales performance across different channels + +```sql +-- Your query here + +``` + +**Solution:** +```sql +SELECT + sales_channel, + COUNT(*) as transaction_count, + SUM(total_amount) as total_revenue, + AVG(total_amount) as avg_transaction_value, + SUM(quantity) as total_units, + ROUND(100.0 * COUNT(*) / SUM(COUNT(*)) OVER (), 2) as pct_of_total_transactions +FROM sales_transactions +GROUP BY sales_channel +ORDER BY total_revenue DESC; +``` + +--- + +## Exercise 3.4: Regional Performance +**Objective:** Analyze sales by region + +```sql +-- Your query here + +``` + +**Solution:** +```sql +SELECT + region, + COUNT(*) as sales_count, + SUM(total_amount) as total_revenue, + AVG(total_amount) as avg_sale, + SUM(discount_amount) as total_discounts, + ROUND(100.0 * SUM(discount_amount) / SUM(total_amount), 2) as discount_rate +FROM sales_transactions +GROUP BY region +ORDER BY total_revenue DESC; +``` + +--- + +## Exercise 3.5: Product Category Analysis +**Objective:** Compare performance across product categories + +```sql +-- Your query here + +``` + +**Solution:** +```sql +SELECT + p.product_category, + COUNT(DISTINCT p.product_id) as product_count, + COUNT(st.transaction_id) as sales_count, + SUM(st.total_amount) as total_revenue, + AVG(st.total_amount) as avg_sale_value, + SUM(st.quantity) as units_sold, + AVG(p.margin_percentage) as avg_margin +FROM product_catalog p +LEFT JOIN sales_transactions st ON p.product_id = st.product_id +GROUP BY p.product_category +ORDER BY total_revenue DESC; +``` + +--- + +## Exercise 3.6: Monthly Sales Trend +**Objective:** Show sales trends over the past 12 months + +```sql +-- Your query here + +``` + +**Solution:** +```sql +SELECT + DATE_TRUNC('month', sale_date) as month, + COUNT(*) as transaction_count, + SUM(total_amount) as total_revenue, + AVG(total_amount) as avg_transaction_value, + SUM(quantity) as units_sold +FROM sales_transactions +WHERE sale_date >= CURRENT_DATE - INTERVAL '12 months' +GROUP BY DATE_TRUNC('month', sale_date) +ORDER BY month DESC; +``` + +--- + +## Exercise 3.7: Sales Performance vs Target +**Objective:** Compare actual sales to targets + +```sql +-- Your query here + +``` + +**Solution:** +```sql +SELECT + st.target_period, + st.target_category, + st.target_value, + COALESCE(sp.actual_value, 0) as actual_value, + COALESCE(sp.actual_value, 0) - st.target_value as variance, + ROUND(100.0 * COALESCE(sp.actual_value, 0) / st.target_value, 2) as achievement_pct, + CASE + WHEN COALESCE(sp.actual_value, 0) >= st.target_value THEN 'MET' + WHEN COALESCE(sp.actual_value, 0) >= st.target_value * 0.9 THEN 'NEAR' + ELSE 'MISSED' + END as status +FROM sales_targets st +LEFT JOIN sales_performance sp ON st.target_id = sp.target_id +WHERE st.target_period >= CURRENT_DATE - INTERVAL '6 months' +ORDER BY st.target_period DESC, st.target_category; +``` + +--- + +## Exercise 3.8: Product Profitability +**Objective:** Calculate profit margins by product + +```sql +-- Your query here + +``` + +**Solution:** +```sql +SELECT + p.product_id, + p.product_name, + p.product_category, + p.unit_price, + p.cost_price, + p.margin_percentage, + COUNT(st.transaction_id) as sales_count, + SUM(st.quantity) as units_sold, + SUM(st.total_amount) as total_revenue, + SUM(st.quantity * p.cost_price) as total_cost, + SUM(st.total_amount) - SUM(st.quantity * p.cost_price) as gross_profit +FROM product_catalog p +JOIN sales_transactions st ON p.product_id = st.product_id +GROUP BY p.product_id, p.product_name, p.product_category, p.unit_price, p.cost_price, p.margin_percentage +HAVING SUM(st.total_amount) > 0 +ORDER BY gross_profit DESC +LIMIT 20; +``` + +--- + +## Exercise 3.9: Revenue Forecast Accuracy +**Objective:** Compare forecasted revenue to actual revenue + +```sql +-- Your query here + +``` + +**Solution:** +```sql +SELECT + rf.forecast_period, + rf.forecast_category, + rf.forecasted_revenue, + rf.confidence_level, + ms.total_revenue as actual_revenue, + (ms.total_revenue - rf.forecasted_revenue) as variance, + ROUND(100.0 * ABS(ms.total_revenue - rf.forecasted_revenue) / rf.forecasted_revenue, 2) as error_pct +FROM revenue_forecasts rf +JOIN monthly_summaries ms ON + EXTRACT(MONTH FROM rf.forecast_period) = ms.summary_month AND + EXTRACT(YEAR FROM rf.forecast_period) = ms.summary_year +WHERE rf.forecast_period >= CURRENT_DATE - INTERVAL '12 months' +ORDER BY rf.forecast_period DESC; +``` + +--- + +## Exercise 3.10: Sales Velocity Analysis +**Objective:** Identify fast-moving vs slow-moving products + +```sql +-- Your query here + +``` + +**Solution:** +```sql +WITH product_sales AS ( + SELECT + p.product_id, + p.product_name, + p.product_category, + COUNT(st.transaction_id) as sales_count, + SUM(st.quantity) as units_sold, + MIN(st.sale_date) as first_sale, + MAX(st.sale_date) as last_sale, + EXTRACT(DAY FROM MAX(st.sale_date) - MIN(st.sale_date)) as days_on_market + FROM product_catalog p + LEFT JOIN sales_transactions st ON p.product_id = st.product_id + WHERE p.is_active = TRUE + GROUP BY p.product_id, p.product_name, p.product_category +) +SELECT + product_id, + product_name, + product_category, + sales_count, + units_sold, + days_on_market, + CASE + WHEN days_on_market > 0 THEN ROUND(units_sold::DECIMAL / days_on_market, 2) + ELSE 0 + END as units_per_day, + CASE + WHEN days_on_market > 0 AND (units_sold::DECIMAL / days_on_market) > 10 THEN 'FAST' + WHEN days_on_market > 0 AND (units_sold::DECIMAL / days_on_market) > 5 THEN 'MEDIUM' + WHEN days_on_market > 0 THEN 'SLOW' + ELSE 'NO_SALES' + END as velocity_category +FROM product_sales +ORDER BY units_per_day DESC; +``` + +--- + +## Exercise 3.11: Discount Impact Analysis +**Objective:** Analyze the impact of discounts on sales + +```sql +-- Your query here + +``` + +**Solution:** +```sql +SELECT + CASE + WHEN discount_amount = 0 THEN 'No Discount' + WHEN discount_amount / total_amount < 0.1 THEN '< 10%' + WHEN discount_amount / total_amount < 0.2 THEN '10-20%' + WHEN discount_amount / total_amount < 0.3 THEN '20-30%' + ELSE '> 30%' + END as discount_tier, + COUNT(*) as transaction_count, + AVG(total_amount) as avg_sale_value, + SUM(total_amount) as total_revenue, + SUM(discount_amount) as total_discounts, + AVG(quantity) as avg_quantity +FROM sales_transactions +GROUP BY discount_tier +ORDER BY + CASE discount_tier + WHEN 'No Discount' THEN 1 + WHEN '< 10%' THEN 2 + WHEN '10-20%' THEN 3 + WHEN '20-30%' THEN 4 + ELSE 5 + END; +``` + +--- + +## Exercise 3.12: Cross-Sell Opportunities +**Objective:** Find products frequently purchased together + +```sql +-- Your query here + +``` + +**Hint:** Use self-join on transaction_id + +**Solution:** +```sql +SELECT + p1.product_name as product_1, + p2.product_name as product_2, + COUNT(*) as times_purchased_together +FROM sales_transactions st1 +JOIN sales_transactions st2 ON st1.transaction_id = st2.transaction_id + AND st1.product_id < st2.product_id +JOIN product_catalog p1 ON st1.product_id = p1.product_id +JOIN product_catalog p2 ON st2.product_id = p2.product_id +GROUP BY p1.product_id, p1.product_name, p2.product_id, p2.product_name +HAVING COUNT(*) > 10 +ORDER BY times_purchased_together DESC +LIMIT 20; +``` + +--- + +## Challenge Exercises + +### Challenge 3.1: Sales Seasonality +Identify seasonal patterns in sales data by analyzing month-over-month and year-over-year trends + +### Challenge 3.2: Customer Purchase Patterns +Analyze average time between purchases and identify customers with regular buying patterns + +### Challenge 3.3: Product Launch Performance +Compare new product performance (launched in last 6 months) vs established products + +--- + +## Next Steps +Once you're comfortable with sales analysis, move on to: +- **Level 4:** KPI Dashboards & Metrics +- **Level 6:** Fraud Detection (Advanced) + diff --git a/exercises/04-kpi-dashboards/README.md b/exercises/04-kpi-dashboards/README.md new file mode 100644 index 0000000..ba3eca0 --- /dev/null +++ b/exercises/04-kpi-dashboards/README.md @@ -0,0 +1,431 @@ +# Level 4: KPI Dashboards & Metrics + +## Introduction +Learn to build and maintain KPI dashboards - a critical skill for Data Analysts supporting business decision-making. + +## Learning Objectives +- Track key performance indicators +- Calculate trend metrics +- Build dashboard queries +- Monitor data quality +- Create executive reports + +--- + +## Exercise 4.1: Current KPI Status +**Objective:** Get current status of all active KPIs + +```sql +SELECT + kd.kpi_name, + kd.kpi_category, + kd.target_value, + dm.metric_value as current_value, + dm.status, + kd.unit_of_measure, + ROUND(100.0 * dm.metric_value / kd.target_value, 2) as achievement_pct +FROM kpi_definitions kd +LEFT JOIN daily_metrics dm ON kd.kpi_id = dm.kpi_id +WHERE kd.is_active = TRUE + AND dm.metric_date = CURRENT_DATE +ORDER BY kd.kpi_category, kd.kpi_name; +``` + +**Expected Result:** Dashboard view of all KPIs with current status + +--- + +## Exercise 4.2: KPIs Below Target +**Objective:** Identify KPIs that are underperforming + +```sql +-- Your query here + +``` + +**Solution:** +```sql +SELECT + kd.kpi_name, + kd.kpi_category, + kd.target_value, + dm.metric_value as current_value, + kd.threshold_warning, + kd.threshold_critical, + dm.status, + (dm.metric_value - kd.target_value) as variance, + ROUND(100.0 * (dm.metric_value - kd.target_value) / kd.target_value, 2) as variance_pct +FROM kpi_definitions kd +JOIN daily_metrics dm ON kd.kpi_id = dm.kpi_id +WHERE kd.is_active = TRUE + AND dm.metric_date = CURRENT_DATE + AND dm.metric_value < kd.target_value +ORDER BY variance_pct ASC; +``` + +--- + +## Exercise 4.3: KPI Trend Analysis (7 Days) +**Objective:** Show 7-day trend for critical KPIs + +```sql +-- Your query here + +``` + +**Solution:** +```sql +SELECT + kd.kpi_name, + dm.metric_date, + dm.metric_value, + kd.target_value, + dm.vs_previous_day_percentage, + dm.status +FROM kpi_definitions kd +JOIN daily_metrics dm ON kd.kpi_id = dm.kpi_id +WHERE kd.is_active = TRUE + AND dm.metric_date >= CURRENT_DATE - INTERVAL '7 days' + AND kd.kpi_category IN ('SALES', 'CUSTOMER') +ORDER BY kd.kpi_name, dm.metric_date DESC; +``` + +--- + +## Exercise 4.4: Monthly Performance Summary +**Objective:** Aggregate monthly business metrics + +```sql +-- Your query here + +``` + +**Solution:** +```sql +SELECT + summary_year, + summary_month, + TO_CHAR(TO_DATE(summary_month::TEXT, 'MM'), 'Month') as month_name, + total_revenue, + total_transactions, + total_customers, + new_customers, + average_transaction_value, + ROUND(100.0 * new_customers / total_customers, 2) as new_customer_pct +FROM monthly_summaries +WHERE summary_year >= EXTRACT(YEAR FROM CURRENT_DATE) - 1 +ORDER BY summary_year DESC, summary_month DESC; +``` + +--- + +## Exercise 4.5: Year-over-Year Comparison +**Objective:** Compare this year's performance to last year + +```sql +-- Your query here + +``` + +**Solution:** +```sql +WITH current_year AS ( + SELECT + summary_month, + total_revenue, + total_transactions, + new_customers + FROM monthly_summaries + WHERE summary_year = EXTRACT(YEAR FROM CURRENT_DATE) +), +previous_year AS ( + SELECT + summary_month, + total_revenue, + total_transactions, + new_customers + FROM monthly_summaries + WHERE summary_year = EXTRACT(YEAR FROM CURRENT_DATE) - 1 +) +SELECT + cy.summary_month, + TO_CHAR(TO_DATE(cy.summary_month::TEXT, 'MM'), 'Month') as month_name, + cy.total_revenue as current_year_revenue, + py.total_revenue as previous_year_revenue, + (cy.total_revenue - py.total_revenue) as revenue_change, + ROUND(100.0 * (cy.total_revenue - py.total_revenue) / py.total_revenue, 2) as revenue_growth_pct, + cy.total_transactions as current_year_transactions, + py.total_transactions as previous_year_transactions +FROM current_year cy +LEFT JOIN previous_year py ON cy.summary_month = py.summary_month +ORDER BY cy.summary_month; +``` + +--- + +## Exercise 4.6: Dashboard Snapshot +**Objective:** Create a pre-calculated dashboard snapshot + +```sql +-- Your query here + +``` + +**Solution:** +```sql +INSERT INTO dashboard_snapshots ( + snapshot_date, dashboard_name, metric_name, metric_value, metric_category +) +SELECT + CURRENT_DATE as snapshot_date, + 'Executive Dashboard' as dashboard_name, + kd.kpi_name as metric_name, + dm.metric_value, + kd.kpi_category as metric_category +FROM kpi_definitions kd +JOIN daily_metrics dm ON kd.kpi_id = dm.kpi_id +WHERE kd.is_active = TRUE + AND dm.metric_date = CURRENT_DATE; + +-- View the snapshot +SELECT * FROM dashboard_snapshots +WHERE snapshot_date = CURRENT_DATE +ORDER BY metric_category, metric_name; +``` + +--- + +## Exercise 4.7: Trend Detection +**Objective:** Identify metrics with consistent upward or downward trends + +```sql +-- Your query here + +``` + +**Solution:** +```sql +WITH trend_data AS ( + SELECT + kd.kpi_name, + kd.kpi_category, + dm.metric_date, + dm.metric_value, + dm.vs_previous_day_percentage, + ROW_NUMBER() OVER (PARTITION BY kd.kpi_id ORDER BY dm.metric_date DESC) as day_rank + FROM kpi_definitions kd + JOIN daily_metrics dm ON kd.kpi_id = dm.kpi_id + WHERE dm.metric_date >= CURRENT_DATE - INTERVAL '7 days' +) +SELECT + kpi_name, + kpi_category, + COUNT(*) as days_tracked, + AVG(vs_previous_day_percentage) as avg_daily_change, + MIN(metric_value) as min_value, + MAX(metric_value) as max_value, + CASE + WHEN AVG(vs_previous_day_percentage) > 5 THEN 'STRONG UPWARD' + WHEN AVG(vs_previous_day_percentage) > 0 THEN 'UPWARD' + WHEN AVG(vs_previous_day_percentage) > -5 THEN 'DOWNWARD' + ELSE 'STRONG DOWNWARD' + END as trend_direction +FROM trend_data +GROUP BY kpi_name, kpi_category +ORDER BY avg_daily_change DESC; +``` + +--- + +## Exercise 4.8: Data Quality Monitoring +**Objective:** Track data quality metrics + +```sql +-- Your query here + +``` + +**Solution:** +```sql +SELECT + check_date, + table_name, + check_type, + records_checked, + records_failed, + ROUND(100.0 * records_failed / records_checked, 2) as failure_rate, + status, + error_details +FROM data_quality_checks +WHERE check_date >= CURRENT_DATE - INTERVAL '7 days' + AND status IN ('WARNING', 'FAILED') +ORDER BY check_date DESC, failure_rate DESC; +``` + +--- + +## Exercise 4.9: Report Execution History +**Objective:** Monitor report generation and performance + +```sql +-- Your query here + +``` + +**Solution:** +```sql +SELECT + rd.report_name, + rd.report_category, + re.execution_date, + re.execution_status, + re.execution_time_seconds, + re.records_generated, + re.file_size_kb +FROM report_definitions rd +JOIN report_executions re ON rd.report_id = re.report_id +WHERE re.execution_date >= CURRENT_DATE - INTERVAL '30 days' +ORDER BY re.execution_date DESC; +``` + +--- + +## Exercise 4.10: Executive Summary Dashboard +**Objective:** Create a comprehensive executive summary + +```sql +-- Your query here + +``` + +**Solution:** +```sql +WITH today_metrics AS ( + SELECT + SUM(CASE WHEN kd.kpi_category = 'SALES' THEN dm.metric_value ELSE 0 END) as total_sales, + AVG(CASE WHEN kd.kpi_name = 'Customer Engagement Score' THEN dm.metric_value END) as avg_engagement, + AVG(CASE WHEN kd.kpi_name = 'Net Promoter Score' THEN dm.metric_value END) as avg_nps, + COUNT(CASE WHEN dm.status = 'CRITICAL' THEN 1 END) as critical_kpis, + COUNT(CASE WHEN dm.status = 'WARNING' THEN 1 END) as warning_kpis, + COUNT(CASE WHEN dm.status = 'ON_TARGET' THEN 1 END) as on_target_kpis + FROM kpi_definitions kd + JOIN daily_metrics dm ON kd.kpi_id = dm.kpi_id + WHERE dm.metric_date = CURRENT_DATE +), +monthly_summary AS ( + SELECT + total_revenue, + total_transactions, + new_customers, + average_transaction_value + FROM monthly_summaries + WHERE summary_year = EXTRACT(YEAR FROM CURRENT_DATE) + AND summary_month = EXTRACT(MONTH FROM CURRENT_DATE) +) +SELECT + 'Executive Summary' as report_title, + CURRENT_DATE as report_date, + tm.total_sales, + tm.avg_engagement, + tm.avg_nps, + tm.critical_kpis, + tm.warning_kpis, + tm.on_target_kpis, + ms.total_revenue as mtd_revenue, + ms.total_transactions as mtd_transactions, + ms.new_customers as mtd_new_customers +FROM today_metrics tm +CROSS JOIN monthly_summary ms; +``` + +--- + +## Exercise 4.11: Moving Average Calculation +**Objective:** Calculate 7-day moving average for key metrics + +```sql +-- Your query here + +``` + +**Solution:** +```sql +SELECT + kd.kpi_name, + dm.metric_date, + dm.metric_value, + AVG(dm.metric_value) OVER ( + PARTITION BY kd.kpi_id + ORDER BY dm.metric_date + ROWS BETWEEN 6 PRECEDING AND CURRENT ROW + ) as moving_avg_7day, + kd.target_value +FROM kpi_definitions kd +JOIN daily_metrics dm ON kd.kpi_id = dm.kpi_id +WHERE kd.kpi_name IN ('Daily Revenue', 'Transactions Per Day', 'Customer Engagement Score') + AND dm.metric_date >= CURRENT_DATE - INTERVAL '30 days' +ORDER BY kd.kpi_name, dm.metric_date DESC; +``` + +--- + +## Exercise 4.12: Anomaly Detection +**Objective:** Identify unusual metric values + +```sql +-- Your query here + +``` + +**Solution:** +```sql +WITH metric_stats AS ( + SELECT + kpi_id, + AVG(metric_value) as avg_value, + STDDEV(metric_value) as stddev_value + FROM daily_metrics + WHERE metric_date >= CURRENT_DATE - INTERVAL '30 days' + GROUP BY kpi_id +) +SELECT + kd.kpi_name, + dm.metric_date, + dm.metric_value, + ms.avg_value, + ms.stddev_value, + (dm.metric_value - ms.avg_value) / NULLIF(ms.stddev_value, 0) as z_score, + CASE + WHEN ABS((dm.metric_value - ms.avg_value) / NULLIF(ms.stddev_value, 0)) > 3 THEN 'EXTREME ANOMALY' + WHEN ABS((dm.metric_value - ms.avg_value) / NULLIF(ms.stddev_value, 0)) > 2 THEN 'ANOMALY' + ELSE 'NORMAL' + END as anomaly_status +FROM kpi_definitions kd +JOIN daily_metrics dm ON kd.kpi_id = dm.kpi_id +JOIN metric_stats ms ON kd.kpi_id = ms.kpi_id +WHERE dm.metric_date >= CURRENT_DATE - INTERVAL '7 days' + AND ABS((dm.metric_value - ms.avg_value) / NULLIF(ms.stddev_value, 0)) > 2 +ORDER BY ABS((dm.metric_value - ms.avg_value) / NULLIF(ms.stddev_value, 0)) DESC; +``` + +--- + +## Challenge Exercises + +### Challenge 4.1: Custom Dashboard Builder +Create a flexible query that can generate different dashboard views based on parameters (date range, KPI category, etc.) + +### Challenge 4.2: Predictive Alerting +Build a query that predicts which KPIs are likely to miss targets based on current trends + +### Challenge 4.3: Correlation Analysis +Identify correlations between different KPIs (e.g., does customer engagement correlate with revenue?) + +--- + +## Next Steps +You've now mastered the core Data Analyst skills! Continue with: +- **Level 5:** Advanced Analytics (Window Functions, CTEs) +- **Level 6:** Fraud Detection (Advanced Pattern Recognition) +- Build your own custom dashboards and reports! + diff --git a/schema/01-create-tables.sql b/schema/01-create-tables.sql index 7eca75c..b7d6613 100644 --- a/schema/01-create-tables.sql +++ b/schema/01-create-tables.sql @@ -1,6 +1,6 @@ -- ============================================================================ --- Financial Fraud Detection Database Schema --- Purpose: Educational SQL learning with realistic fraud investigation scenarios +-- Business Analytics Database Schema +-- Purpose: Data Analyst training with routine/semi-routine analysis scenarios -- ============================================================================ -- This script is IDEMPOTENT - it will drop and recreate all objects -- ============================================================================ @@ -23,6 +23,27 @@ DROP FUNCTION IF EXISTS check_suspicious_transaction() CASCADE; DROP FUNCTION IF EXISTS generate_fraud_score(DECIMAL, BOOLEAN, DECIMAL, INT, BOOLEAN) CASCADE; -- Drop tables in reverse dependency order +-- New analytics tables +DROP TABLE IF EXISTS report_executions CASCADE; +DROP TABLE IF EXISTS report_definitions CASCADE; +DROP TABLE IF EXISTS data_quality_checks CASCADE; +DROP TABLE IF EXISTS dashboard_snapshots CASCADE; +DROP TABLE IF EXISTS trend_analysis CASCADE; +DROP TABLE IF EXISTS monthly_summaries CASCADE; +DROP TABLE IF EXISTS daily_metrics CASCADE; +DROP TABLE IF EXISTS kpi_definitions CASCADE; +DROP TABLE IF EXISTS revenue_forecasts CASCADE; +DROP TABLE IF EXISTS sales_performance CASCADE; +DROP TABLE IF EXISTS sales_targets CASCADE; +DROP TABLE IF EXISTS sales_transactions CASCADE; +DROP TABLE IF EXISTS product_catalog CASCADE; +DROP TABLE IF EXISTS engagement_metrics CASCADE; +DROP TABLE IF EXISTS customer_satisfaction CASCADE; +DROP TABLE IF EXISTS churn_predictions CASCADE; +DROP TABLE IF EXISTS customer_lifetime_value CASCADE; +DROP TABLE IF EXISTS customer_segments CASCADE; + +-- Existing tables DROP TABLE IF EXISTS audit_log CASCADE; DROP TABLE IF EXISTS suspicious_activity_reports CASCADE; DROP TABLE IF EXISTS case_alerts CASCADE; @@ -475,3 +496,321 @@ COMMENT ON TABLE cards IS 'Payment cards linked to accounts'; COMMENT ON TABLE devices IS 'Device fingerprints for fraud detection'; COMMENT ON TABLE login_sessions IS 'Login history for account takeover detection'; +-- ============================================================================ +-- CUSTOMER ANALYTICS TABLES (for routine customer analysis) +-- ============================================================================ + +CREATE TABLE customer_segments ( + segment_id SERIAL PRIMARY KEY, + segment_name VARCHAR(100) NOT NULL UNIQUE, + segment_description TEXT, + criteria_definition JSONB, + min_clv DECIMAL(15,2), + max_clv DECIMAL(15,2), + created_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + updated_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP +); + +CREATE TABLE customer_lifetime_value ( + clv_id SERIAL PRIMARY KEY, + customer_id INT NOT NULL REFERENCES customers(customer_id), + calculation_date DATE NOT NULL, + total_revenue DECIMAL(15,2) DEFAULT 0, + total_transactions INT DEFAULT 0, + average_order_value DECIMAL(15,2) DEFAULT 0, + predicted_future_value DECIMAL(15,2), + clv_score DECIMAL(10,2), + segment_id INT REFERENCES customer_segments(segment_id), + UNIQUE(customer_id, calculation_date) +); + +CREATE TABLE churn_predictions ( + prediction_id SERIAL PRIMARY KEY, + customer_id INT NOT NULL REFERENCES customers(customer_id), + prediction_date DATE NOT NULL, + churn_probability DECIMAL(5,2) CHECK (churn_probability BETWEEN 0 AND 100), + risk_level VARCHAR(20) CHECK (risk_level IN ('LOW', 'MEDIUM', 'HIGH', 'CRITICAL')), + last_transaction_date DATE, + days_since_last_transaction INT, + engagement_score DECIMAL(5,2), + recommended_action TEXT, + UNIQUE(customer_id, prediction_date) +); + +CREATE TABLE customer_satisfaction ( + satisfaction_id SERIAL PRIMARY KEY, + customer_id INT NOT NULL REFERENCES customers(customer_id), + survey_date DATE NOT NULL, + nps_score INT CHECK (nps_score BETWEEN -100 AND 100), + csat_score DECIMAL(3,2) CHECK (csat_score BETWEEN 1 AND 5), + feedback_text TEXT, + category VARCHAR(50) CHECK (category IN ('PRODUCT', 'SERVICE', 'SUPPORT', 'BILLING', 'OTHER')), + sentiment VARCHAR(20) CHECK (sentiment IN ('POSITIVE', 'NEUTRAL', 'NEGATIVE')) +); + +CREATE TABLE engagement_metrics ( + metric_id SERIAL PRIMARY KEY, + customer_id INT NOT NULL REFERENCES customers(customer_id), + metric_date DATE NOT NULL, + login_count INT DEFAULT 0, + page_views INT DEFAULT 0, + time_spent_minutes INT DEFAULT 0, + features_used JSONB, + support_tickets_opened INT DEFAULT 0, + engagement_score DECIMAL(5,2), + UNIQUE(customer_id, metric_date) +); + +-- ============================================================================ +-- SALES & REVENUE ANALYTICS TABLES (for semi-routine sales reporting) +-- ============================================================================ + +CREATE TABLE product_catalog ( + product_id SERIAL PRIMARY KEY, + product_name VARCHAR(200) NOT NULL, + product_category VARCHAR(100), + product_subcategory VARCHAR(100), + unit_price DECIMAL(15,2) NOT NULL, + cost_price DECIMAL(15,2), + margin_percentage DECIMAL(5,2), + is_active BOOLEAN DEFAULT TRUE, + launch_date DATE, + discontinued_date DATE +); + +CREATE TABLE sales_transactions ( + sale_id SERIAL PRIMARY KEY, + transaction_id INT REFERENCES transactions(transaction_id), + product_id INT REFERENCES product_catalog(product_id), + quantity INT NOT NULL DEFAULT 1, + unit_price DECIMAL(15,2) NOT NULL, + discount_amount DECIMAL(15,2) DEFAULT 0, + tax_amount DECIMAL(15,2) DEFAULT 0, + total_amount DECIMAL(15,2) NOT NULL, + sale_date TIMESTAMP NOT NULL, + sales_channel VARCHAR(50) CHECK (sales_channel IN ('ONLINE', 'STORE', 'PHONE', 'MOBILE_APP')), + sales_rep_id INT, + region VARCHAR(100) +); + +CREATE TABLE sales_targets ( + target_id SERIAL PRIMARY KEY, + target_period VARCHAR(20) CHECK (target_period IN ('DAILY', 'WEEKLY', 'MONTHLY', 'QUARTERLY', 'YEARLY')), + start_date DATE NOT NULL, + end_date DATE NOT NULL, + product_category VARCHAR(100), + region VARCHAR(100), + target_revenue DECIMAL(15,2), + target_units INT, + target_customers INT, + created_by VARCHAR(100), + created_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP +); + +CREATE TABLE sales_performance ( + performance_id SERIAL PRIMARY KEY, + period_date DATE NOT NULL, + period_type VARCHAR(20) CHECK (period_type IN ('DAILY', 'WEEKLY', 'MONTHLY', 'QUARTERLY')), + product_id INT REFERENCES product_catalog(product_id), + region VARCHAR(100), + total_revenue DECIMAL(15,2) DEFAULT 0, + total_units_sold INT DEFAULT 0, + total_transactions INT DEFAULT 0, + unique_customers INT DEFAULT 0, + average_order_value DECIMAL(15,2), + vs_target_percentage DECIMAL(5,2), + UNIQUE(period_date, period_type, product_id, region) +); + +CREATE TABLE revenue_forecasts ( + forecast_id SERIAL PRIMARY KEY, + forecast_date DATE NOT NULL, + forecast_period_start DATE NOT NULL, + forecast_period_end DATE NOT NULL, + product_category VARCHAR(100), + region VARCHAR(100), + forecasted_revenue DECIMAL(15,2), + confidence_level VARCHAR(20) CHECK (confidence_level IN ('LOW', 'MEDIUM', 'HIGH')), + forecast_method VARCHAR(50) CHECK (forecast_method IN ('HISTORICAL', 'TREND', 'SEASONAL', 'ML')), + actual_revenue DECIMAL(15,2), + variance_percentage DECIMAL(5,2) +); + + +-- ============================================================================ +-- OPERATIONAL METRICS & KPI TABLES (for dashboard creation) +-- ============================================================================ + +CREATE TABLE kpi_definitions ( + kpi_id SERIAL PRIMARY KEY, + kpi_name VARCHAR(100) NOT NULL UNIQUE, + kpi_description TEXT, + kpi_category VARCHAR(50) CHECK (kpi_category IN ('SALES', 'CUSTOMER', 'OPERATIONAL', 'FINANCIAL')), + calculation_formula TEXT, + target_value DECIMAL(15,2), + threshold_warning DECIMAL(15,2), + threshold_critical DECIMAL(15,2), + unit_of_measure VARCHAR(50), + refresh_frequency VARCHAR(20) CHECK (refresh_frequency IN ('REALTIME', 'HOURLY', 'DAILY', 'WEEKLY')), + is_active BOOLEAN DEFAULT TRUE +); + +CREATE TABLE daily_metrics ( + metric_id SERIAL PRIMARY KEY, + metric_date DATE NOT NULL, + kpi_id INT NOT NULL REFERENCES kpi_definitions(kpi_id), + metric_value DECIMAL(15,2), + vs_previous_day_percentage DECIMAL(5,2), + vs_previous_week_percentage DECIMAL(5,2), + vs_previous_month_percentage DECIMAL(5,2), + status VARCHAR(20) CHECK (status IN ('ON_TARGET', 'WARNING', 'CRITICAL')), + notes TEXT, + UNIQUE(metric_date, kpi_id) +); + +CREATE TABLE monthly_summaries ( + summary_id SERIAL PRIMARY KEY, + summary_month INT CHECK (summary_month BETWEEN 1 AND 12), + summary_year INT, + total_revenue DECIMAL(15,2) DEFAULT 0, + total_transactions INT DEFAULT 0, + total_customers INT DEFAULT 0, + new_customers INT DEFAULT 0, + churned_customers INT DEFAULT 0, + average_transaction_value DECIMAL(15,2), + customer_acquisition_cost DECIMAL(15,2), + customer_lifetime_value DECIMAL(15,2), + net_promoter_score DECIMAL(5,2), + gross_margin_percentage DECIMAL(5,2), + UNIQUE(summary_month, summary_year) +); + +CREATE TABLE trend_analysis ( + trend_id SERIAL PRIMARY KEY, + kpi_id INT NOT NULL REFERENCES kpi_definitions(kpi_id), + analysis_date DATE NOT NULL, + period_start DATE NOT NULL, + period_end DATE NOT NULL, + trend_direction VARCHAR(20) CHECK (trend_direction IN ('UP', 'DOWN', 'FLAT')), + trend_strength VARCHAR(20) CHECK (trend_strength IN ('WEAK', 'MODERATE', 'STRONG')), + moving_average_7day DECIMAL(15,2), + moving_average_30day DECIMAL(15,2), + seasonality_detected BOOLEAN DEFAULT FALSE, + anomalies_detected BOOLEAN DEFAULT FALSE, + statistical_significance DECIMAL(5,4) +); + +CREATE TABLE dashboard_snapshots ( + snapshot_id SERIAL PRIMARY KEY, + dashboard_name VARCHAR(100) NOT NULL, + snapshot_timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + data_payload JSONB, + refresh_duration_seconds DECIMAL(10,2), + row_count INT, + last_updated_by VARCHAR(100) +); + +-- ============================================================================ +-- BUSINESS INTELLIGENCE & REPORTING TABLES +-- ============================================================================ + +CREATE TABLE report_definitions ( + report_id SERIAL PRIMARY KEY, + report_name VARCHAR(200) NOT NULL UNIQUE, + report_description TEXT, + report_category VARCHAR(100), + sql_query_template TEXT, + parameters JSONB, + output_format VARCHAR(20) CHECK (output_format IN ('PDF', 'EXCEL', 'CSV', 'HTML')), + schedule_frequency VARCHAR(50), + recipients TEXT, + is_active BOOLEAN DEFAULT TRUE, + created_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP +); + +CREATE TABLE report_executions ( + execution_id SERIAL PRIMARY KEY, + report_id INT NOT NULL REFERENCES report_definitions(report_id), + execution_timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + parameters_used JSONB, + row_count INT, + execution_duration_seconds DECIMAL(10,2), + status VARCHAR(20) CHECK (status IN ('SUCCESS', 'FAILED', 'TIMEOUT')), + error_message TEXT, + output_file_path TEXT, + executed_by VARCHAR(100) +); + +CREATE TABLE data_quality_checks ( + check_id SERIAL PRIMARY KEY, + check_name VARCHAR(200) NOT NULL, + table_name VARCHAR(100) NOT NULL, + column_name VARCHAR(100), + check_type VARCHAR(50) CHECK (check_type IN ('NULL_CHECK', 'RANGE_CHECK', 'UNIQUENESS', 'REFERENTIAL_INTEGRITY', 'FORMAT_CHECK')), + check_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + records_checked INT, + records_failed INT, + failure_percentage DECIMAL(5,2), + status VARCHAR(20) CHECK (status IN ('PASS', 'FAIL', 'WARNING')), + remediation_notes TEXT +); + +-- ============================================================================ +-- INDEXES FOR NEW ANALYTICS TABLES +-- ============================================================================ + +-- Customer Analytics indexes +CREATE INDEX idx_clv_customer ON customer_lifetime_value(customer_id); +CREATE INDEX idx_clv_date ON customer_lifetime_value(calculation_date); +CREATE INDEX idx_clv_segment ON customer_lifetime_value(segment_id); +CREATE INDEX idx_churn_customer ON churn_predictions(customer_id); +CREATE INDEX idx_churn_risk ON churn_predictions(risk_level); +CREATE INDEX idx_satisfaction_customer ON customer_satisfaction(customer_id); +CREATE INDEX idx_engagement_customer ON engagement_metrics(customer_id); +CREATE INDEX idx_engagement_date ON engagement_metrics(metric_date); + +-- Sales Analytics indexes +CREATE INDEX idx_sales_trans_product ON sales_transactions(product_id); +CREATE INDEX idx_sales_trans_date ON sales_transactions(sale_date); +CREATE INDEX idx_sales_trans_channel ON sales_transactions(sales_channel); +CREATE INDEX idx_sales_perf_date ON sales_performance(period_date); +CREATE INDEX idx_sales_perf_product ON sales_performance(product_id); +CREATE INDEX idx_product_category ON product_catalog(product_category); +CREATE INDEX idx_product_active ON product_catalog(is_active); + +-- KPI & Metrics indexes +CREATE INDEX idx_daily_metrics_date ON daily_metrics(metric_date); +CREATE INDEX idx_daily_metrics_kpi ON daily_metrics(kpi_id); +CREATE INDEX idx_monthly_summaries_period ON monthly_summaries(summary_year, summary_month); +CREATE INDEX idx_trend_kpi ON trend_analysis(kpi_id); +CREATE INDEX idx_trend_date ON trend_analysis(analysis_date); + +-- Reporting indexes +CREATE INDEX idx_report_exec_report ON report_executions(report_id); +CREATE INDEX idx_report_exec_timestamp ON report_executions(execution_timestamp); +CREATE INDEX idx_data_quality_table ON data_quality_checks(table_name); +CREATE INDEX idx_data_quality_date ON data_quality_checks(check_date); + +-- ============================================================================ +-- COMMENTS FOR NEW TABLES +-- ============================================================================ + +COMMENT ON TABLE customer_segments IS 'Customer segmentation definitions for targeted analysis'; +COMMENT ON TABLE customer_lifetime_value IS 'Customer lifetime value calculations and tracking'; +COMMENT ON TABLE churn_predictions IS 'Customer churn risk predictions for retention analysis'; +COMMENT ON TABLE customer_satisfaction IS 'Customer satisfaction scores and feedback'; +COMMENT ON TABLE engagement_metrics IS 'Customer engagement tracking metrics'; +COMMENT ON TABLE product_catalog IS 'Product master data for sales analysis'; +COMMENT ON TABLE sales_transactions IS 'Detailed sales transaction records'; +COMMENT ON TABLE sales_targets IS 'Sales performance targets and goals'; +COMMENT ON TABLE sales_performance IS 'Aggregated sales performance metrics'; +COMMENT ON TABLE revenue_forecasts IS 'Revenue forecasting and predictions'; +COMMENT ON TABLE kpi_definitions IS 'Master list of tracked KPIs'; +COMMENT ON TABLE daily_metrics IS 'Daily operational metrics for dashboards'; +COMMENT ON TABLE monthly_summaries IS 'Monthly aggregated business summaries'; +COMMENT ON TABLE trend_analysis IS 'Statistical trend analysis results'; +COMMENT ON TABLE dashboard_snapshots IS 'Pre-calculated dashboard data snapshots'; +COMMENT ON TABLE report_definitions IS 'Standard report catalog'; +COMMENT ON TABLE report_executions IS 'Report execution history and logs'; +COMMENT ON TABLE data_quality_checks IS 'Data quality validation tracking'; + diff --git a/schema/02-seed-data.sql b/schema/02-seed-data.sql index cae438d..d771ba5 100644 --- a/schema/02-seed-data.sql +++ b/schema/02-seed-data.sql @@ -5,6 +5,27 @@ -- ============================================================================ -- Clear existing reference data (in reverse dependency order) +-- New analytics tables +TRUNCATE TABLE report_executions CASCADE; +TRUNCATE TABLE report_definitions RESTART IDENTITY CASCADE; +TRUNCATE TABLE data_quality_checks RESTART IDENTITY CASCADE; +TRUNCATE TABLE dashboard_snapshots RESTART IDENTITY CASCADE; +TRUNCATE TABLE trend_analysis RESTART IDENTITY CASCADE; +TRUNCATE TABLE monthly_summaries RESTART IDENTITY CASCADE; +TRUNCATE TABLE daily_metrics RESTART IDENTITY CASCADE; +TRUNCATE TABLE kpi_definitions RESTART IDENTITY CASCADE; +TRUNCATE TABLE revenue_forecasts RESTART IDENTITY CASCADE; +TRUNCATE TABLE sales_performance RESTART IDENTITY CASCADE; +TRUNCATE TABLE sales_targets RESTART IDENTITY CASCADE; +TRUNCATE TABLE sales_transactions RESTART IDENTITY CASCADE; +TRUNCATE TABLE product_catalog RESTART IDENTITY CASCADE; +TRUNCATE TABLE engagement_metrics RESTART IDENTITY CASCADE; +TRUNCATE TABLE customer_satisfaction RESTART IDENTITY CASCADE; +TRUNCATE TABLE churn_predictions RESTART IDENTITY CASCADE; +TRUNCATE TABLE customer_lifetime_value RESTART IDENTITY CASCADE; +TRUNCATE TABLE customer_segments RESTART IDENTITY CASCADE; + +-- Existing tables TRUNCATE TABLE suspicious_activity_reports CASCADE; TRUNCATE TABLE case_alerts CASCADE; TRUNCATE TABLE case_transactions CASCADE; @@ -254,3 +275,91 @@ COMMENT ON FUNCTION generate_fraud_score IS 'Calculates fraud risk score based o COMMENT ON FUNCTION update_account_balance IS 'Automatically updates account balance after transaction'; COMMENT ON FUNCTION check_suspicious_transaction IS 'Generates alerts for suspicious transactions'; +-- ============================================================================ +-- SEED DATA FOR NEW ANALYTICS TABLES +-- ============================================================================ + +-- Customer Segments +INSERT INTO customer_segments (segment_name, segment_description, min_clv, max_clv) VALUES +('VIP', 'Very Important Person - Highest value customers', 50000, NULL), +('High Value', 'High spending customers with strong loyalty', 10000, 49999), +('Medium Value', 'Regular customers with moderate spending', 2000, 9999), +('Low Value', 'Occasional customers with low spending', 500, 1999), +('At Risk', 'Previously active customers showing decline', NULL, NULL), +('New Customer', 'Recently acquired customers (< 90 days)', NULL, NULL), +('Dormant', 'Inactive customers (> 180 days)', NULL, NULL), +('Churned', 'Lost customers who have not transacted in 365+ days', NULL, NULL); + +-- Product Catalog (Sample Products) +INSERT INTO product_catalog (product_name, product_category, product_subcategory, unit_price, cost_price, margin_percentage, is_active, launch_date) VALUES +-- Banking Products +('Premium Checking Account', 'Banking', 'Checking', 0.00, 0.00, 0.00, TRUE, '2020-01-01'), +('Basic Savings Account', 'Banking', 'Savings', 0.00, 0.00, 0.00, TRUE, '2020-01-01'), +('High-Yield Savings', 'Banking', 'Savings', 0.00, 0.00, 0.00, TRUE, '2021-06-01'), +('Business Checking', 'Banking', 'Business', 15.00, 5.00, 66.67, TRUE, '2020-01-01'), +('Student Checking', 'Banking', 'Checking', 0.00, 0.00, 0.00, TRUE, '2020-01-01'), + +-- Credit Cards +('Platinum Credit Card', 'Credit', 'Premium', 99.00, 20.00, 79.80, TRUE, '2020-01-01'), +('Gold Credit Card', 'Credit', 'Standard', 49.00, 15.00, 69.39, TRUE, '2020-01-01'), +('Cash Back Card', 'Credit', 'Rewards', 0.00, 10.00, -100.00, TRUE, '2021-01-01'), +('Travel Rewards Card', 'Credit', 'Rewards', 95.00, 25.00, 73.68, TRUE, '2021-03-01'), +('Business Credit Card', 'Credit', 'Business', 75.00, 20.00, 73.33, TRUE, '2020-06-01'), + +-- Loans +('Personal Loan', 'Lending', 'Personal', 0.00, 0.00, 0.00, TRUE, '2020-01-01'), +('Auto Loan', 'Lending', 'Auto', 0.00, 0.00, 0.00, TRUE, '2020-01-01'), +('Home Mortgage', 'Lending', 'Mortgage', 0.00, 0.00, 0.00, TRUE, '2020-01-01'), +('Small Business Loan', 'Lending', 'Business', 0.00, 0.00, 0.00, TRUE, '2020-01-01'), + +-- Investment Products +('Index Fund', 'Investment', 'Mutual Funds', 0.00, 0.00, 0.00, TRUE, '2020-01-01'), +('Bond Fund', 'Investment', 'Mutual Funds', 0.00, 0.00, 0.00, TRUE, '2020-01-01'), +('Retirement Account (IRA)', 'Investment', 'Retirement', 0.00, 0.00, 0.00, TRUE, '2020-01-01'), +('401k Plan', 'Investment', 'Retirement', 0.00, 0.00, 0.00, TRUE, '2020-01-01'), + +-- Services +('Wire Transfer', 'Services', 'Transfers', 25.00, 5.00, 80.00, TRUE, '2020-01-01'), +('International Transfer', 'Services', 'Transfers', 45.00, 10.00, 77.78, TRUE, '2020-01-01'), +('Overdraft Protection', 'Services', 'Protection', 35.00, 5.00, 85.71, TRUE, '2020-01-01'), +('Safe Deposit Box', 'Services', 'Security', 75.00, 20.00, 73.33, TRUE, '2020-01-01'), +('Financial Advisory', 'Services', 'Advisory', 150.00, 50.00, 66.67, TRUE, '2021-01-01'), +('Mobile Banking Premium', 'Services', 'Digital', 9.99, 2.00, 79.98, TRUE, '2022-01-01'); + +-- KPI Definitions +INSERT INTO kpi_definitions (kpi_name, kpi_description, kpi_category, calculation_formula, target_value, threshold_warning, threshold_critical, unit_of_measure, refresh_frequency, is_active) VALUES +-- Sales KPIs +('Daily Revenue', 'Total revenue generated per day', 'SALES', 'SUM(amount) FROM transactions WHERE DATE(transaction_date) = CURRENT_DATE', 500000, 400000, 300000, 'USD', 'DAILY', TRUE), +('Monthly Revenue', 'Total revenue for the month', 'SALES', 'SUM(amount) FROM transactions WHERE MONTH(transaction_date) = CURRENT_MONTH', 15000000, 12000000, 10000000, 'USD', 'DAILY', TRUE), +('Average Transaction Value', 'Average value per transaction', 'SALES', 'AVG(amount) FROM transactions', 150, 100, 75, 'USD', 'DAILY', TRUE), +('Transactions Per Day', 'Number of transactions per day', 'SALES', 'COUNT(*) FROM transactions WHERE DATE(transaction_date) = CURRENT_DATE', 10000, 7500, 5000, 'Count', 'DAILY', TRUE), + +-- Customer KPIs +('New Customers', 'New customer acquisitions', 'CUSTOMER', 'COUNT(*) FROM customers WHERE registration_date >= CURRENT_DATE - 30', 1000, 750, 500, 'Count', 'DAILY', TRUE), +('Customer Churn Rate', 'Percentage of customers churning', 'CUSTOMER', '(Churned / Total) * 100', 5, 7, 10, 'Percentage', 'WEEKLY', TRUE), +('Customer Lifetime Value', 'Average CLV across all customers', 'CUSTOMER', 'AVG(clv_score) FROM customer_lifetime_value', 5000, 4000, 3000, 'USD', 'WEEKLY', TRUE), +('Net Promoter Score', 'Customer satisfaction metric', 'CUSTOMER', 'AVG(nps_score) FROM customer_satisfaction', 50, 30, 10, 'Score', 'WEEKLY', TRUE), +('Customer Engagement Score', 'Average engagement across customers', 'CUSTOMER', 'AVG(engagement_score) FROM engagement_metrics', 75, 60, 45, 'Score', 'DAILY', TRUE), + +-- Operational KPIs +('Transaction Success Rate', 'Percentage of successful transactions', 'OPERATIONAL', '(Successful / Total) * 100', 99, 97, 95, 'Percentage', 'HOURLY', TRUE), +('Average Response Time', 'System response time', 'OPERATIONAL', 'AVG(response_time_ms)', 200, 500, 1000, 'Milliseconds', 'REALTIME', TRUE), +('Fraud Detection Rate', 'Percentage of fraud caught', 'OPERATIONAL', '(Detected / Total Fraud) * 100', 95, 90, 85, 'Percentage', 'DAILY', TRUE), +('Alert Resolution Time', 'Average time to resolve alerts', 'OPERATIONAL', 'AVG(resolution_time_hours)', 24, 48, 72, 'Hours', 'DAILY', TRUE), + +-- Financial KPIs +('Gross Margin', 'Overall profit margin', 'FINANCIAL', '((Revenue - Cost) / Revenue) * 100', 75, 65, 55, 'Percentage', 'DAILY', TRUE), +('Customer Acquisition Cost', 'Cost to acquire new customer', 'FINANCIAL', 'Marketing Spend / New Customers', 50, 75, 100, 'USD', 'WEEKLY', TRUE), +('Return on Investment', 'ROI on marketing campaigns', 'FINANCIAL', '((Revenue - Cost) / Cost) * 100', 300, 200, 100, 'Percentage', 'WEEKLY', TRUE); + +-- Report Definitions +INSERT INTO report_definitions (report_name, report_description, report_category, output_format, schedule_frequency, is_active) VALUES +('Daily Sales Summary', 'Daily sales performance report', 'Sales', 'PDF', 'Daily at 6 AM', TRUE), +('Weekly Customer Analytics', 'Customer behavior and segmentation analysis', 'Customer', 'EXCEL', 'Weekly on Monday', TRUE), +('Monthly Financial Summary', 'Comprehensive monthly financial report', 'Financial', 'PDF', 'Monthly on 1st', TRUE), +('Fraud Detection Report', 'Daily fraud alerts and cases', 'Risk', 'PDF', 'Daily at 8 AM', TRUE), +('KPI Dashboard', 'Executive KPI dashboard', 'Executive', 'HTML', 'Daily at 7 AM', TRUE), +('Customer Churn Analysis', 'At-risk customer identification', 'Customer', 'EXCEL', 'Weekly on Friday', TRUE), +('Product Performance', 'Product sales and profitability analysis', 'Sales', 'EXCEL', 'Monthly on 5th', TRUE), +('Data Quality Report', 'Data validation and quality metrics', 'Operations', 'CSV', 'Daily at 5 AM', TRUE); + diff --git a/scripts/setup-database.sh b/scripts/setup-database.sh index 7d1e097..f7f9534 100644 --- a/scripts/setup-database.sh +++ b/scripts/setup-database.sh @@ -3,7 +3,7 @@ # ============================================================================ # Database Setup Script - IDEMPOTENT # ============================================================================ -# This script sets up the complete fraud detection database +# This script sets up the complete business analytics database # It can be run multiple times safely - it will recreate everything # ============================================================================ @@ -19,8 +19,8 @@ NC='\033[0m' # No Color # Configuration from environment or defaults DB_HOST="${POSTGRES_HOST:-localhost}" DB_PORT="${POSTGRES_PORT:-5432}" -DB_NAME="${POSTGRES_DB:-fraud_detection}" -DB_USER="${POSTGRES_USER:-fraud_analyst}" +DB_NAME="${POSTGRES_DB:-business_analytics}" +DB_USER="${POSTGRES_USER:-data_analyst}" DB_PASSWORD="${POSTGRES_PASSWORD:-SecurePass123!}" # Script directory @@ -28,7 +28,7 @@ SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" PROJECT_ROOT="$(dirname "$SCRIPT_DIR")" echo -e "${BLUE}============================================================================${NC}" -echo -e "${BLUE}Financial Fraud Detection Database - Setup Script${NC}" +echo -e "${BLUE}Business Analytics Database - Setup Script${NC}" echo -e "${BLUE}============================================================================${NC}" echo -e "Database: ${GREEN}$DB_NAME${NC}" echo -e "Host: ${GREEN}$DB_HOST:$DB_PORT${NC}" diff --git a/scripts/verify-setup.sh b/scripts/verify-setup.sh index 2e1f405..77a1851 100644 --- a/scripts/verify-setup.sh +++ b/scripts/verify-setup.sh @@ -3,7 +3,7 @@ # ============================================================================ # Setup Verification Script # ============================================================================ -# Verifies that the fraud detection database is properly set up +# Verifies that the business analytics database is properly set up # ============================================================================ set -e @@ -18,12 +18,12 @@ NC='\033[0m' # No Color # Configuration DB_HOST="${POSTGRES_HOST:-localhost}" DB_PORT="${POSTGRES_PORT:-5432}" -DB_NAME="${POSTGRES_DB:-fraud_detection}" -DB_USER="${POSTGRES_USER:-fraud_analyst}" +DB_NAME="${POSTGRES_DB:-business_analytics}" +DB_USER="${POSTGRES_USER:-data_analyst}" DB_PASSWORD="${POSTGRES_PASSWORD:-SecurePass123!}" echo -e "${BLUE}============================================================================${NC}" -echo -e "${BLUE}Financial Fraud Detection Database - Verification${NC}" +echo -e "${BLUE}Business Analytics Database - Verification${NC}" echo -e "${BLUE}============================================================================${NC}" echo "" @@ -34,7 +34,7 @@ execute_sql() { # Check 1: Docker containers echo -e "${YELLOW}[1/10] Checking Docker containers...${NC}" -if docker ps | grep -q "fraud_detection_db"; then +if docker ps | grep -q "business_analytics_db"; then echo -e "${GREEN}✓ PostgreSQL container is running${NC}" else echo -e "${RED}✗ PostgreSQL container is not running${NC}" @@ -42,7 +42,7 @@ else exit 1 fi -if docker ps | grep -q "fraud_detection_ui"; then +if docker ps | grep -q "business_analytics_ui"; then echo -e "${GREEN}✓ DB-UI container is running${NC}" else echo -e "${RED}✗ DB-UI container is not running${NC}"