mirror of
https://github.com/freedbygrace/SQL.git
synced 2026-07-26 11:28:16 +00:00
Expand to Business Analytics: Add Customer, Sales, and KPI models
Major expansion from fraud detection to comprehensive business analytics: DATABASE CHANGES: - Renamed database from 'fraud_detection' to 'business_analytics' - Renamed user from 'fraud_analyst' to 'data_analyst' - Expanded from 20 to 39 tables across 4 business models NEW MODELS (19 tables): 1. Customer Analytics (5 tables): - customer_segments, customer_lifetime_value, churn_predictions - customer_satisfaction, engagement_metrics 2. Sales & Revenue Analytics (6 tables): - product_catalog, sales_transactions, sales_targets - sales_performance, revenue_forecasts 3. KPI & Metrics (8 tables): - kpi_definitions, daily_metrics, monthly_summaries - trend_analysis, dashboard_snapshots - report_definitions, report_executions, data_quality_checks DATA GENERATION: - Extended generate_data.sh with 6 new steps (now 15 total) - Added CLV calculations for all customers - Added churn predictions based on transaction recency - Added 30K customer satisfaction surveys - Added 1M sales transactions linked to 24 products - Added 90 days of daily KPI metrics - Added 24 months of business summaries SQL EXERCISES (3 new levels): - Level 2: Customer Analytics (10 exercises + 3 challenges) - Level 3: Sales & Revenue Analysis (12 exercises + 3 challenges) - Level 4: KPI Dashboards & Metrics (12 exercises + 3 challenges) DOCUMENTATION: - Updated README.md with business analytics focus - Updated QUICKSTART.md with new data generation steps - Updated SETUP_COMPLETE.md with 39-table architecture - Added DATA_MODELS.md with complete model specifications - Added WHATS_NEW.md with migration guide SEED DATA: - Added 8 customer segments (VIP, High Value, etc.) - Added 24 products across 5 categories - Added 16 KPI definitions across 4 categories - Added 8 standard report definitions All changes maintain idempotency and backward compatibility with existing fraud detection functionality.
This commit is contained in:
@@ -1,44 +1,82 @@
|
|||||||
# Financial Fraud Detection - SQL Learning Database
|
# Business Analytics - SQL Learning Database
|
||||||
|
|
||||||
A comprehensive, production-grade database designed for learning SQL through realistic financial fraud investigation scenarios.
|
A comprehensive, production-grade database designed for learning SQL through realistic business analytics, customer insights, sales analysis, and fraud detection scenarios.
|
||||||
|
|
||||||
## 🎯 Overview
|
## 🎯 Overview
|
||||||
|
|
||||||
This project provides a complete PostgreSQL database with **5+ million transactions**, embedded fraud patterns, and progressive SQL exercises. Perfect for:
|
This project provides a complete PostgreSQL database with **5+ million transactions**, **39 tables** across 4 business models, and progressive SQL exercises aligned with **Data Analyst** responsibilities. Perfect for:
|
||||||
|
|
||||||
- **SQL Beginners** → Learn fundamentals with real-world data
|
- **SQL Beginners** → Learn fundamentals with real-world data
|
||||||
- **Data Analysts** → Practice fraud detection queries
|
- **Data Analysts** → Practice customer analytics, sales analysis, KPI dashboards
|
||||||
- **Security Professionals** → Understand fraud patterns
|
- **Business Analysts** → Understand customer behavior and revenue patterns
|
||||||
- **Students** → Hands-on financial crime investigation
|
- **Security Professionals** → Detect fraud patterns
|
||||||
|
- **Students** → Hands-on business intelligence and analytics
|
||||||
|
|
||||||
## 📊 Database Statistics
|
## 📊 Database Statistics
|
||||||
|
|
||||||
|
### Core Data
|
||||||
- **100,000** Customers with KYC data
|
- **100,000** Customers with KYC data
|
||||||
- **150,000** Bank accounts (checking, savings, credit)
|
- **150,000** Bank accounts (checking, savings, credit)
|
||||||
- **200,000** Payment cards
|
- **200,000** Payment cards
|
||||||
- **50,000** Merchants across 35 categories
|
- **50,000** Merchants across 35 categories
|
||||||
- **5,000,000** Transactions (7% fraudulent)
|
- **5,000,000** Transactions (7% fraudulent)
|
||||||
|
- **1,000,000** Sales records linked to products
|
||||||
|
- **500,000** Login sessions
|
||||||
|
|
||||||
|
### Analytics Data
|
||||||
|
- **100,000** Customer Lifetime Value calculations
|
||||||
|
- **100,000** Churn predictions
|
||||||
|
- **30,000** Customer satisfaction surveys
|
||||||
|
- **24** Product catalog entries
|
||||||
|
- **1,500+** Daily KPI metrics
|
||||||
|
- **24** Monthly business summaries
|
||||||
- **50,000+** Fraud alerts
|
- **50,000+** Fraud alerts
|
||||||
- **5,000+** Fraud cases
|
- **5,000+** Fraud cases
|
||||||
- **500,000** Login sessions
|
|
||||||
|
|
||||||
## 🏗️ Architecture
|
## 🏗️ Architecture
|
||||||
|
|
||||||
### Data Model Features
|
### Data Model Features
|
||||||
- ✅ **20+ Tables** with proper relationships
|
- ✅ **39 Tables** across 4 business models
|
||||||
- ✅ **Foreign key constraints** for data integrity
|
- ✅ **Foreign key constraints** for data integrity
|
||||||
- ✅ **Indexes** for query performance
|
- ✅ **Indexes** for query performance
|
||||||
- ✅ **Realistic geographic data** (100 US cities, 210 world cities)
|
- ✅ **Realistic geographic data** (100 US cities, 210 world cities)
|
||||||
- ✅ **Embedded fraud patterns** (velocity, geographic, structuring)
|
- ✅ **Customer analytics** (CLV, churn, satisfaction, engagement)
|
||||||
|
- ✅ **Sales analytics** (products, targets, forecasts)
|
||||||
|
- ✅ **KPI tracking** (daily metrics, trends, dashboards)
|
||||||
|
- ✅ **Fraud detection** (velocity, geographic, structuring patterns)
|
||||||
- ✅ **Audit trails** and compliance tables
|
- ✅ **Audit trails** and compliance tables
|
||||||
|
|
||||||
### Key Entities
|
### Business Models
|
||||||
|
|
||||||
|
#### 1. Fraud Detection Model (20 tables)
|
||||||
```
|
```
|
||||||
customers → accounts → transactions
|
customers → accounts → transactions → alerts → fraud_cases
|
||||||
↓
|
↓
|
||||||
cards → merchants
|
cards → merchants
|
||||||
↓
|
```
|
||||||
alerts → fraud_cases
|
|
||||||
|
#### 2. Customer Analytics Model (5 tables)
|
||||||
|
```
|
||||||
|
customers → customer_lifetime_value → customer_segments
|
||||||
|
→ churn_predictions
|
||||||
|
→ customer_satisfaction
|
||||||
|
→ engagement_metrics
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 3. Sales & Revenue Model (6 tables)
|
||||||
|
```
|
||||||
|
product_catalog → sales_transactions → sales_performance
|
||||||
|
→ sales_targets
|
||||||
|
→ revenue_forecasts
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 4. KPI & Metrics Model (8 tables)
|
||||||
|
```
|
||||||
|
kpi_definitions → daily_metrics → trend_analysis
|
||||||
|
→ monthly_summaries
|
||||||
|
→ dashboard_snapshots
|
||||||
|
→ report_definitions → report_executions
|
||||||
|
→ data_quality_checks
|
||||||
```
|
```
|
||||||
|
|
||||||
## 🚀 Quick Start
|
## 🚀 Quick Start
|
||||||
@@ -86,15 +124,15 @@ http://localhost:3000
|
|||||||
|
|
||||||
**Option B: Command Line**
|
**Option B: Command Line**
|
||||||
```bash
|
```bash
|
||||||
docker exec -it fraud_detection_db psql -U fraud_analyst -d fraud_detection
|
docker exec -it business_analytics_db psql -U data_analyst -d business_analytics
|
||||||
```
|
```
|
||||||
|
|
||||||
**Option C: Your Favorite SQL Client**
|
**Option C: Your Favorite SQL Client**
|
||||||
```
|
```
|
||||||
Host: localhost
|
Host: localhost
|
||||||
Port: 5432
|
Port: 5432
|
||||||
Database: fraud_detection
|
Database: business_analytics
|
||||||
Username: fraud_analyst
|
Username: data_analyst
|
||||||
Password: SecurePass123!
|
Password: SecurePass123!
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -106,29 +144,36 @@ Password: SecurePass123!
|
|||||||
- Basic comparisons
|
- Basic comparisons
|
||||||
- **Location:** `exercises/01-basic-queries/`
|
- **Location:** `exercises/01-basic-queries/`
|
||||||
|
|
||||||
### Level 2: Joins
|
### Level 2: Customer Analytics ⭐ NEW
|
||||||
- INNER JOIN, LEFT JOIN
|
- Customer segmentation
|
||||||
- Multiple table queries
|
- Lifetime value (CLV) analysis
|
||||||
- Relationship navigation
|
- Churn prediction
|
||||||
- **Location:** `exercises/02-joins/`
|
- Satisfaction metrics
|
||||||
|
- Engagement tracking
|
||||||
|
- **Location:** `exercises/02-customer-analytics/`
|
||||||
|
|
||||||
### Level 3: Aggregations
|
### Level 3: Sales & Revenue Analysis ⭐ NEW
|
||||||
- COUNT, SUM, AVG, MAX, MIN
|
- Sales performance vs targets
|
||||||
- GROUP BY and HAVING
|
- Product analytics
|
||||||
- Statistical analysis
|
- Channel and regional analysis
|
||||||
- **Location:** `exercises/03-aggregations/`
|
- Revenue forecasting
|
||||||
|
- Profitability analysis
|
||||||
|
- **Location:** `exercises/03-sales-analysis/`
|
||||||
|
|
||||||
### Level 4: Subqueries
|
### Level 4: KPI Dashboards & Metrics ⭐ NEW
|
||||||
- Nested queries
|
- KPI tracking and monitoring
|
||||||
- Correlated subqueries
|
- Trend analysis
|
||||||
- EXISTS and IN
|
- Dashboard creation
|
||||||
- **Location:** `exercises/04-subqueries/`
|
- Data quality monitoring
|
||||||
|
- Executive reporting
|
||||||
|
- **Location:** `exercises/04-kpi-dashboards/`
|
||||||
|
|
||||||
### Level 5: Window Functions
|
### Level 5: Advanced SQL Techniques
|
||||||
- ROW_NUMBER, RANK, DENSE_RANK
|
- Window functions (ROW_NUMBER, RANK)
|
||||||
- Running totals
|
- Common Table Expressions (CTEs)
|
||||||
- Moving averages
|
- Running totals and moving averages
|
||||||
- **Location:** `exercises/05-window-functions/`
|
- Complex aggregations
|
||||||
|
- **Location:** `exercises/05-advanced-sql/`
|
||||||
|
|
||||||
### Level 6: Fraud Detection
|
### Level 6: Fraud Detection
|
||||||
- Velocity fraud detection
|
- Velocity fraud detection
|
||||||
@@ -137,6 +182,40 @@ Password: SecurePass123!
|
|||||||
- Account takeover patterns
|
- Account takeover patterns
|
||||||
- **Location:** `exercises/06-fraud-detection/`
|
- **Location:** `exercises/06-fraud-detection/`
|
||||||
|
|
||||||
|
## 💼 Data Analyst Use Cases
|
||||||
|
|
||||||
|
This database supports typical **Data Analyst** responsibilities:
|
||||||
|
|
||||||
|
### Routine Analysis
|
||||||
|
- Daily sales summaries
|
||||||
|
- Customer acquisition metrics
|
||||||
|
- Transaction volume tracking
|
||||||
|
- Basic KPI monitoring
|
||||||
|
|
||||||
|
### Semi-Routine Reporting
|
||||||
|
- Weekly customer analytics
|
||||||
|
- Monthly revenue reports
|
||||||
|
- Product performance analysis
|
||||||
|
- Churn risk identification
|
||||||
|
|
||||||
|
### Dashboard Creation
|
||||||
|
- Executive KPI dashboards
|
||||||
|
- Sales performance dashboards
|
||||||
|
- Customer health dashboards
|
||||||
|
- Operational metrics dashboards
|
||||||
|
|
||||||
|
### Trend Identification
|
||||||
|
- Revenue trends (MoM, YoY)
|
||||||
|
- Customer behavior patterns
|
||||||
|
- Product sales seasonality
|
||||||
|
- Engagement score trends
|
||||||
|
|
||||||
|
### Data Quality
|
||||||
|
- Missing data detection
|
||||||
|
- Anomaly identification
|
||||||
|
- Validation checks
|
||||||
|
- Data completeness monitoring
|
||||||
|
|
||||||
## 🔍 Fraud Patterns Included
|
## 🔍 Fraud Patterns Included
|
||||||
|
|
||||||
### 1. Velocity Fraud
|
### 1. Velocity Fraud
|
||||||
@@ -180,18 +259,19 @@ SQL/
|
|||||||
│ ├── world_cities.csv
|
│ ├── world_cities.csv
|
||||||
│ └── load_geographic_data.sql
|
│ └── load_geographic_data.sql
|
||||||
├── scripts/
|
├── scripts/
|
||||||
│ └── setup-database.sh # Setup automation
|
│ ├── setup-database.sh # Setup automation
|
||||||
|
│ └── verify-setup.sh # Verification script
|
||||||
├── exercises/
|
├── exercises/
|
||||||
│ ├── 01-basic-queries/
|
│ ├── 01-basic-queries/ # SQL fundamentals
|
||||||
│ ├── 02-joins/
|
│ ├── 02-customer-analytics/ # ⭐ Customer insights
|
||||||
│ ├── 03-aggregations/
|
│ ├── 03-sales-analysis/ # ⭐ Sales & revenue
|
||||||
│ ├── 04-subqueries/
|
│ ├── 04-kpi-dashboards/ # ⭐ KPI tracking
|
||||||
│ ├── 05-window-functions/
|
│ ├── 05-advanced-sql/ # Advanced techniques
|
||||||
│ └── 06-fraud-detection/
|
│ └── 06-fraud-detection/ # Fraud patterns
|
||||||
└── docs/
|
└── docs/
|
||||||
├── data-model.md
|
├── DATA_MODELS.md # ⭐ Complete model documentation
|
||||||
├── fraud-patterns.md
|
├── WHATS_NEW.md # ⭐ Recent changes
|
||||||
└── setup-guide.md
|
└── QUICKSTART.md # Quick start guide
|
||||||
```
|
```
|
||||||
|
|
||||||
## 🔧 Configuration
|
## 🔧 Configuration
|
||||||
@@ -200,8 +280,8 @@ SQL/
|
|||||||
Edit `docker-compose.yml` to customize:
|
Edit `docker-compose.yml` to customize:
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
POSTGRES_DB: fraud_detection
|
POSTGRES_DB: business_analytics
|
||||||
POSTGRES_USER: fraud_analyst
|
POSTGRES_USER: data_analyst
|
||||||
POSTGRES_PASSWORD: SecurePass123!
|
POSTGRES_PASSWORD: SecurePass123!
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -224,15 +304,46 @@ All scripts are **idempotent** - safe to run multiple times:
|
|||||||
|
|
||||||
## 🎓 Sample Queries
|
## 🎓 Sample Queries
|
||||||
|
|
||||||
### Find High-Risk Customers
|
### Customer Analytics: High-Value Customers
|
||||||
```sql
|
```sql
|
||||||
SELECT customer_id, first_name, last_name, risk_score
|
SELECT
|
||||||
FROM customers
|
c.customer_id, c.first_name, c.last_name,
|
||||||
WHERE risk_score > 80
|
clv.clv_score, cs.segment_name
|
||||||
ORDER BY risk_score DESC;
|
FROM customers c
|
||||||
|
JOIN customer_lifetime_value clv ON c.customer_id = clv.customer_id
|
||||||
|
JOIN customer_segments cs ON clv.segment_id = cs.segment_id
|
||||||
|
WHERE cs.segment_name IN ('VIP', 'High Value')
|
||||||
|
ORDER BY clv.clv_score DESC
|
||||||
|
LIMIT 20;
|
||||||
```
|
```
|
||||||
|
|
||||||
### Detect Velocity Fraud
|
### Sales Analytics: Top Products
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
p.product_name, p.product_category,
|
||||||
|
COUNT(st.transaction_id) as sales_count,
|
||||||
|
SUM(st.total_amount) as total_revenue
|
||||||
|
FROM product_catalog p
|
||||||
|
JOIN sales_transactions st ON p.product_id = st.product_id
|
||||||
|
GROUP BY p.product_id, p.product_name, p.product_category
|
||||||
|
ORDER BY total_revenue DESC
|
||||||
|
LIMIT 10;
|
||||||
|
```
|
||||||
|
|
||||||
|
### KPI Dashboard: Current Status
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
kd.kpi_name, kd.kpi_category,
|
||||||
|
dm.metric_value, kd.target_value,
|
||||||
|
dm.status
|
||||||
|
FROM kpi_definitions kd
|
||||||
|
JOIN daily_metrics dm ON kd.kpi_id = dm.kpi_id
|
||||||
|
WHERE dm.metric_date = CURRENT_DATE
|
||||||
|
AND kd.is_active = TRUE
|
||||||
|
ORDER BY kd.kpi_category, kd.kpi_name;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Fraud Detection: Velocity Fraud
|
||||||
```sql
|
```sql
|
||||||
SELECT account_id, COUNT(*) as txn_count, SUM(amount) as total
|
SELECT account_id, COUNT(*) as txn_count, SUM(amount) as total
|
||||||
FROM transactions
|
FROM transactions
|
||||||
|
|||||||
+31
-5
@@ -1,4 +1,4 @@
|
|||||||
# 🎉 Financial Fraud Detection Database - Setup Complete!
|
# 🎉 Business Analytics Database - Setup Complete!
|
||||||
|
|
||||||
## ✅ What Has Been Created
|
## ✅ What Has Been Created
|
||||||
|
|
||||||
@@ -9,9 +9,11 @@
|
|||||||
- ✅ Persistent volumes for data
|
- ✅ Persistent volumes for data
|
||||||
- ✅ Health checks and auto-restart
|
- ✅ Health checks and auto-restart
|
||||||
|
|
||||||
### 2. **Database Schema (20+ Tables)**
|
### 2. **Database Schema (39 Tables Across 4 Business Models)**
|
||||||
|
|
||||||
#### Core Tables
|
#### Model 1: Fraud Detection (20 tables)
|
||||||
|
|
||||||
|
**Core Tables:**
|
||||||
- ✅ `customers` - 100K customer records with KYC data
|
- ✅ `customers` - 100K customer records with KYC data
|
||||||
- ✅ `accounts` - 150K bank accounts (checking, savings, credit)
|
- ✅ `accounts` - 150K bank accounts (checking, savings, credit)
|
||||||
- ✅ `cards` - 200K payment cards
|
- ✅ `cards` - 200K payment cards
|
||||||
@@ -19,13 +21,13 @@
|
|||||||
- ✅ `merchants` - 50K merchants across 35 categories
|
- ✅ `merchants` - 50K merchants across 35 categories
|
||||||
- ✅ `devices` - 75K device fingerprints
|
- ✅ `devices` - 75K device fingerprints
|
||||||
|
|
||||||
#### Fraud Detection Tables
|
**Fraud Detection Tables:**
|
||||||
- ✅ `alerts` - System-generated fraud alerts
|
- ✅ `alerts` - System-generated fraud alerts
|
||||||
- ✅ `fraud_cases` - Confirmed fraud investigations
|
- ✅ `fraud_cases` - Confirmed fraud investigations
|
||||||
- ✅ `case_transactions` - Links transactions to cases
|
- ✅ `case_transactions` - Links transactions to cases
|
||||||
- ✅ `case_alerts` - Links alerts to cases
|
- ✅ `case_alerts` - Links alerts to cases
|
||||||
|
|
||||||
#### Supporting Tables
|
**Supporting Tables:**
|
||||||
- ✅ `countries` - 40 countries with risk levels
|
- ✅ `countries` - 40 countries with risk levels
|
||||||
- ✅ `merchant_categories` - 35 MCC categories
|
- ✅ `merchant_categories` - 35 MCC categories
|
||||||
- ✅ `transaction_types` - 15 transaction types
|
- ✅ `transaction_types` - 15 transaction types
|
||||||
@@ -37,6 +39,30 @@
|
|||||||
- ✅ `suspicious_activity_reports` - SAR filings
|
- ✅ `suspicious_activity_reports` - SAR filings
|
||||||
- ✅ `audit_log` - Complete audit trail
|
- ✅ `audit_log` - Complete audit trail
|
||||||
|
|
||||||
|
#### Model 2: Customer Analytics (5 tables) ⭐ NEW
|
||||||
|
- ✅ `customer_segments` - Customer classification (VIP, High Value, etc.)
|
||||||
|
- ✅ `customer_lifetime_value` - CLV calculations for all customers
|
||||||
|
- ✅ `churn_predictions` - Customer retention risk analysis
|
||||||
|
- ✅ `customer_satisfaction` - NPS/CSAT scores and feedback
|
||||||
|
- ✅ `engagement_metrics` - Customer interaction tracking
|
||||||
|
|
||||||
|
#### Model 3: Sales & Revenue Analytics (6 tables) ⭐ NEW
|
||||||
|
- ✅ `product_catalog` - 24 products across categories
|
||||||
|
- ✅ `sales_transactions` - 1M sales records linked to products
|
||||||
|
- ✅ `sales_targets` - Performance goals and targets
|
||||||
|
- ✅ `sales_performance` - Aggregated performance metrics
|
||||||
|
- ✅ `revenue_forecasts` - Revenue predictions and variance
|
||||||
|
|
||||||
|
#### Model 4: KPI & Metrics (8 tables) ⭐ NEW
|
||||||
|
- ✅ `kpi_definitions` - Master KPI catalog (16 KPIs)
|
||||||
|
- ✅ `daily_metrics` - Daily operational snapshots (90 days)
|
||||||
|
- ✅ `monthly_summaries` - Monthly business summaries (24 months)
|
||||||
|
- ✅ `trend_analysis` - Statistical trend tracking
|
||||||
|
- ✅ `dashboard_snapshots` - Pre-calculated dashboard data
|
||||||
|
- ✅ `report_definitions` - Standard report catalog
|
||||||
|
- ✅ `report_executions` - Report run history
|
||||||
|
- ✅ `data_quality_checks` - Data validation tracking
|
||||||
|
|
||||||
### 3. **Realistic Geographic Data**
|
### 3. **Realistic Geographic Data**
|
||||||
- ✅ 100 US cities with matching states
|
- ✅ 100 US cities with matching states
|
||||||
- ✅ 210 world cities across 40 countries
|
- ✅ 210 world cities across 40 countries
|
||||||
|
|||||||
+270
-16
@@ -1,9 +1,9 @@
|
|||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
|
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
# Financial Fraud Detection - Data Generation Script - IDEMPOTENT
|
# Business Analytics - Data Generation Script - IDEMPOTENT
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
# Generates realistic test data with embedded fraud patterns
|
# Generates realistic test data for business analytics and reporting
|
||||||
# This script is IDEMPOTENT - it will clear and regenerate all data
|
# This script is IDEMPOTENT - it will clear and regenerate all data
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
|
|
||||||
@@ -19,8 +19,8 @@ NC='\033[0m' # No Color
|
|||||||
# Configuration
|
# Configuration
|
||||||
DB_HOST="${POSTGRES_HOST:-localhost}"
|
DB_HOST="${POSTGRES_HOST:-localhost}"
|
||||||
DB_PORT="${POSTGRES_PORT:-5432}"
|
DB_PORT="${POSTGRES_PORT:-5432}"
|
||||||
DB_NAME="${POSTGRES_DB:-fraud_detection}"
|
DB_NAME="${POSTGRES_DB:-business_analytics}"
|
||||||
DB_USER="${POSTGRES_USER:-fraud_analyst}"
|
DB_USER="${POSTGRES_USER:-data_analyst}"
|
||||||
DB_PASSWORD="${POSTGRES_PASSWORD:-SecurePass123!}"
|
DB_PASSWORD="${POSTGRES_PASSWORD:-SecurePass123!}"
|
||||||
|
|
||||||
# Data volumes
|
# Data volumes
|
||||||
@@ -33,7 +33,7 @@ NUM_TRANSACTIONS=5000000
|
|||||||
FRAUD_PERCENTAGE=7 # 7% of transactions will be fraudulent
|
FRAUD_PERCENTAGE=7 # 7% of transactions will be fraudulent
|
||||||
|
|
||||||
echo -e "${BLUE}============================================================================${NC}"
|
echo -e "${BLUE}============================================================================${NC}"
|
||||||
echo -e "${BLUE}Financial Fraud Detection Database - Data Generation${NC}"
|
echo -e "${BLUE}Business Analytics Database - Data Generation${NC}"
|
||||||
echo -e "${BLUE}============================================================================${NC}"
|
echo -e "${BLUE}============================================================================${NC}"
|
||||||
echo -e "Target Database: ${GREEN}$DB_NAME@$DB_HOST:$DB_PORT${NC}"
|
echo -e "Target Database: ${GREEN}$DB_NAME@$DB_HOST:$DB_PORT${NC}"
|
||||||
echo -e "Customers: ${GREEN}$NUM_CUSTOMERS${NC}"
|
echo -e "Customers: ${GREEN}$NUM_CUSTOMERS${NC}"
|
||||||
@@ -62,7 +62,7 @@ execute_sql_file() {
|
|||||||
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
|
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
|
||||||
|
|
||||||
# Clear existing data (preserve reference tables)
|
# Clear existing data (preserve reference tables)
|
||||||
echo -e "${YELLOW}[0/9] Clearing existing data...${NC}"
|
echo -e "${YELLOW}[0/15] Clearing existing data...${NC}"
|
||||||
execute_sql "TRUNCATE TABLE audit_log CASCADE;"
|
execute_sql "TRUNCATE TABLE audit_log CASCADE;"
|
||||||
execute_sql "TRUNCATE TABLE suspicious_activity_reports CASCADE;"
|
execute_sql "TRUNCATE TABLE suspicious_activity_reports CASCADE;"
|
||||||
execute_sql "TRUNCATE TABLE case_alerts CASCADE;"
|
execute_sql "TRUNCATE TABLE case_alerts CASCADE;"
|
||||||
@@ -83,12 +83,12 @@ echo -e "${GREEN}✓ Existing data cleared${NC}"
|
|||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
# Load geographic reference data
|
# Load geographic reference data
|
||||||
echo -e "${YELLOW}[1/9] Loading geographic reference data...${NC}"
|
echo -e "${YELLOW}[1/15] Loading geographic reference data...${NC}"
|
||||||
execute_sql_file "$SCRIPT_DIR/reference/load_geographic_data.sql" > /dev/null
|
execute_sql_file "$SCRIPT_DIR/reference/load_geographic_data.sql" > /dev/null
|
||||||
echo -e "${GREEN}✓ Geographic data loaded (100 US cities, 210 world cities)${NC}"
|
echo -e "${GREEN}✓ Geographic data loaded (100 US cities, 210 world cities)${NC}"
|
||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
echo -e "${YELLOW}[2/9] Generating Customers...${NC}"
|
echo -e "${YELLOW}[2/15] Generating Customers...${NC}"
|
||||||
cat > /tmp/generate_customers.sql << 'EOF'
|
cat > /tmp/generate_customers.sql << 'EOF'
|
||||||
-- Generate customers with realistic geographic data
|
-- Generate customers with realistic geographic data
|
||||||
INSERT INTO customers (
|
INSERT INTO customers (
|
||||||
@@ -130,7 +130,7 @@ execute_sql_file /tmp/generate_customers.sql > /dev/null
|
|||||||
echo -e "${GREEN}✓ Generated $NUM_CUSTOMERS customers${NC}"
|
echo -e "${GREEN}✓ Generated $NUM_CUSTOMERS customers${NC}"
|
||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
echo -e "${YELLOW}[3/9] Generating Accounts...${NC}"
|
echo -e "${YELLOW}[3/15] Generating Accounts...${NC}"
|
||||||
cat > /tmp/generate_accounts.sql << 'EOF'
|
cat > /tmp/generate_accounts.sql << 'EOF'
|
||||||
-- Generate accounts (1-2 accounts per customer on average)
|
-- Generate accounts (1-2 accounts per customer on average)
|
||||||
INSERT INTO accounts (
|
INSERT INTO accounts (
|
||||||
@@ -177,7 +177,7 @@ execute_sql_file /tmp/generate_accounts.sql > /dev/null
|
|||||||
echo -e "${GREEN}✓ Generated $NUM_ACCOUNTS accounts${NC}"
|
echo -e "${GREEN}✓ Generated $NUM_ACCOUNTS accounts${NC}"
|
||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
echo -e "${YELLOW}[4/9] Generating Merchants...${NC}"
|
echo -e "${YELLOW}[4/15] Generating Merchants...${NC}"
|
||||||
cat > /tmp/generate_merchants.sql << 'EOF'
|
cat > /tmp/generate_merchants.sql << 'EOF'
|
||||||
-- Generate merchants with realistic geographic data
|
-- Generate merchants with realistic geographic data
|
||||||
INSERT INTO merchants (
|
INSERT INTO merchants (
|
||||||
@@ -237,7 +237,7 @@ execute_sql_file /tmp/generate_merchants.sql > /dev/null
|
|||||||
echo -e "${GREEN}✓ Generated $NUM_MERCHANTS merchants${NC}"
|
echo -e "${GREEN}✓ Generated $NUM_MERCHANTS merchants${NC}"
|
||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
echo -e "${YELLOW}[5/9] Generating Devices...${NC}"
|
echo -e "${YELLOW}[5/15] Generating Devices...${NC}"
|
||||||
cat > /tmp/generate_devices.sql << 'EOF'
|
cat > /tmp/generate_devices.sql << 'EOF'
|
||||||
-- Generate devices
|
-- Generate devices
|
||||||
INSERT INTO devices (
|
INSERT INTO devices (
|
||||||
@@ -282,7 +282,7 @@ execute_sql_file /tmp/generate_devices.sql > /dev/null
|
|||||||
echo -e "${GREEN}✓ Generated $NUM_DEVICES devices${NC}"
|
echo -e "${GREEN}✓ Generated $NUM_DEVICES devices${NC}"
|
||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
echo -e "${YELLOW}[6/9] Generating Cards...${NC}"
|
echo -e "${YELLOW}[6/15] Generating Cards...${NC}"
|
||||||
cat > /tmp/generate_cards.sql << 'EOF'
|
cat > /tmp/generate_cards.sql << 'EOF'
|
||||||
-- Generate cards (1-2 cards per account on average)
|
-- Generate cards (1-2 cards per account on average)
|
||||||
INSERT INTO cards (
|
INSERT INTO cards (
|
||||||
@@ -326,7 +326,7 @@ execute_sql_file /tmp/generate_cards.sql > /dev/null
|
|||||||
echo -e "${GREEN}✓ Generated $NUM_CARDS cards${NC}"
|
echo -e "${GREEN}✓ Generated $NUM_CARDS cards${NC}"
|
||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
echo -e "${YELLOW}[7/9] Generating Login Sessions...${NC}"
|
echo -e "${YELLOW}[7/15] Generating Login Sessions...${NC}"
|
||||||
cat > /tmp/generate_sessions.sql << 'EOF'
|
cat > /tmp/generate_sessions.sql << 'EOF'
|
||||||
-- Generate login sessions with realistic geographic data
|
-- Generate login sessions with realistic geographic data
|
||||||
INSERT INTO login_sessions (
|
INSERT INTO login_sessions (
|
||||||
@@ -359,7 +359,7 @@ execute_sql_file /tmp/generate_sessions.sql > /dev/null
|
|||||||
echo -e "${GREEN}✓ Generated 500,000 login sessions${NC}"
|
echo -e "${GREEN}✓ Generated 500,000 login sessions${NC}"
|
||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
echo -e "${YELLOW}[8/9] Generating Transactions (this may take a while)...${NC}"
|
echo -e "${YELLOW}[8/15] Generating Transactions (this may take a while)...${NC}"
|
||||||
echo -e "${BLUE}This step generates $NUM_TRANSACTIONS transactions with fraud patterns${NC}"
|
echo -e "${BLUE}This step generates $NUM_TRANSACTIONS transactions with fraud patterns${NC}"
|
||||||
|
|
||||||
# Generate transactions in batches to avoid memory issues
|
# Generate transactions in batches to avoid memory issues
|
||||||
@@ -433,7 +433,7 @@ done
|
|||||||
echo -e "${GREEN}✓ Generated $NUM_TRANSACTIONS transactions${NC}"
|
echo -e "${GREEN}✓ Generated $NUM_TRANSACTIONS transactions${NC}"
|
||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
echo -e "${YELLOW}[9/9] Generating Fraud Cases and Alerts...${NC}"
|
echo -e "${YELLOW}[9/15] Generating Fraud Cases and Alerts...${NC}"
|
||||||
|
|
||||||
# Generate alerts for flagged transactions
|
# Generate alerts for flagged transactions
|
||||||
execute_sql "
|
execute_sql "
|
||||||
@@ -535,7 +535,261 @@ case_count=$(execute_sql "SELECT COUNT(*) FROM fraud_cases;" | grep -E '^\s*[0-9
|
|||||||
echo -e "Fraud Cases: ${GREEN}$case_count${NC}"
|
echo -e "Fraud Cases: ${GREEN}$case_count${NC}"
|
||||||
|
|
||||||
echo ""
|
echo ""
|
||||||
echo -e "${YELLOW}Ready for SQL learning and fraud investigation!${NC}"
|
echo -e "${BLUE}============================================================================${NC}"
|
||||||
|
echo -e "${BLUE}Generating Analytics Data (Customer, Sales, KPIs)${NC}"
|
||||||
|
echo -e "${BLUE}============================================================================${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# GENERATE CUSTOMER ANALYTICS DATA
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
echo -e "${YELLOW}[10/15] Generating Customer Lifetime Value data...${NC}"
|
||||||
|
cat > /tmp/generate_clv.sql << 'EOF'
|
||||||
|
-- Generate CLV for all customers based on their transaction history
|
||||||
|
INSERT INTO customer_lifetime_value (
|
||||||
|
customer_id, calculation_date, total_revenue, total_transactions,
|
||||||
|
average_order_value, predicted_future_value, clv_score, segment_id
|
||||||
|
)
|
||||||
|
SELECT
|
||||||
|
c.customer_id,
|
||||||
|
CURRENT_DATE as calculation_date,
|
||||||
|
COALESCE(SUM(t.amount), 0) as total_revenue,
|
||||||
|
COUNT(t.transaction_id) as total_transactions,
|
||||||
|
COALESCE(AVG(t.amount), 0) as average_order_value,
|
||||||
|
COALESCE(SUM(t.amount) * 1.5, 0) as predicted_future_value,
|
||||||
|
COALESCE(SUM(t.amount) / 100, 0) as clv_score,
|
||||||
|
CASE
|
||||||
|
WHEN COALESCE(SUM(t.amount), 0) >= 50000 THEN 1 -- VIP
|
||||||
|
WHEN COALESCE(SUM(t.amount), 0) >= 10000 THEN 2 -- High Value
|
||||||
|
WHEN COALESCE(SUM(t.amount), 0) >= 2000 THEN 3 -- Medium Value
|
||||||
|
WHEN COALESCE(SUM(t.amount), 0) >= 500 THEN 4 -- Low Value
|
||||||
|
ELSE 6 -- New Customer
|
||||||
|
END as segment_id
|
||||||
|
FROM customers c
|
||||||
|
LEFT JOIN accounts a ON c.customer_id = a.customer_id
|
||||||
|
LEFT JOIN transactions t ON a.account_id = t.account_id
|
||||||
|
GROUP BY c.customer_id;
|
||||||
|
EOF
|
||||||
|
|
||||||
|
execute_sql_file /tmp/generate_clv.sql > /dev/null
|
||||||
|
echo -e "${GREEN}✓ Generated CLV for all customers${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
echo -e "${YELLOW}[11/15] Generating Churn Predictions...${NC}"
|
||||||
|
cat > /tmp/generate_churn.sql << 'EOF'
|
||||||
|
-- Generate churn predictions based on transaction recency
|
||||||
|
INSERT INTO churn_predictions (
|
||||||
|
customer_id, prediction_date, churn_probability, risk_level,
|
||||||
|
last_transaction_date, days_since_last_transaction, engagement_score
|
||||||
|
)
|
||||||
|
SELECT
|
||||||
|
c.customer_id,
|
||||||
|
CURRENT_DATE as prediction_date,
|
||||||
|
CASE
|
||||||
|
WHEN MAX(t.transaction_date) IS NULL THEN 90
|
||||||
|
WHEN CURRENT_DATE - MAX(t.transaction_date) > 180 THEN 85
|
||||||
|
WHEN CURRENT_DATE - MAX(t.transaction_date) > 90 THEN 60
|
||||||
|
WHEN CURRENT_DATE - MAX(t.transaction_date) > 30 THEN 30
|
||||||
|
ELSE 10
|
||||||
|
END as churn_probability,
|
||||||
|
CASE
|
||||||
|
WHEN MAX(t.transaction_date) IS NULL THEN 'HIGH'
|
||||||
|
WHEN CURRENT_DATE - MAX(t.transaction_date) > 180 THEN 'CRITICAL'
|
||||||
|
WHEN CURRENT_DATE - MAX(t.transaction_date) > 90 THEN 'HIGH'
|
||||||
|
WHEN CURRENT_DATE - MAX(t.transaction_date) > 30 THEN 'MEDIUM'
|
||||||
|
ELSE 'LOW'
|
||||||
|
END as risk_level,
|
||||||
|
MAX(t.transaction_date) as last_transaction_date,
|
||||||
|
COALESCE(CURRENT_DATE - MAX(t.transaction_date), 999) as days_since_last_transaction,
|
||||||
|
CASE
|
||||||
|
WHEN MAX(t.transaction_date) IS NULL THEN 0
|
||||||
|
WHEN CURRENT_DATE - MAX(t.transaction_date) > 180 THEN 10
|
||||||
|
WHEN CURRENT_DATE - MAX(t.transaction_date) > 90 THEN 30
|
||||||
|
WHEN CURRENT_DATE - MAX(t.transaction_date) > 30 THEN 60
|
||||||
|
ELSE 90
|
||||||
|
END as engagement_score
|
||||||
|
FROM customers c
|
||||||
|
LEFT JOIN accounts a ON c.customer_id = a.customer_id
|
||||||
|
LEFT JOIN transactions t ON a.account_id = t.account_id
|
||||||
|
GROUP BY c.customer_id;
|
||||||
|
EOF
|
||||||
|
|
||||||
|
execute_sql_file /tmp/generate_churn.sql > /dev/null
|
||||||
|
echo -e "${GREEN}✓ Generated churn predictions${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
echo -e "${YELLOW}[12/15] Generating Customer Satisfaction data...${NC}"
|
||||||
|
cat > /tmp/generate_satisfaction.sql << 'EOF'
|
||||||
|
-- Generate satisfaction scores for random sample of customers
|
||||||
|
INSERT INTO customer_satisfaction (
|
||||||
|
customer_id, survey_date, nps_score, csat_score, category, sentiment
|
||||||
|
)
|
||||||
|
SELECT
|
||||||
|
customer_id,
|
||||||
|
TIMESTAMP '2023-01-01' + (random() * 730)::INT * INTERVAL '1 day' as survey_date,
|
||||||
|
(random() * 200 - 100)::INT as nps_score,
|
||||||
|
(1 + random() * 4)::DECIMAL(3,2) as csat_score,
|
||||||
|
CASE (random() * 5)::INT
|
||||||
|
WHEN 0 THEN 'PRODUCT'
|
||||||
|
WHEN 1 THEN 'SERVICE'
|
||||||
|
WHEN 2 THEN 'SUPPORT'
|
||||||
|
WHEN 3 THEN 'BILLING'
|
||||||
|
ELSE 'OTHER'
|
||||||
|
END as category,
|
||||||
|
CASE
|
||||||
|
WHEN random() < 0.6 THEN 'POSITIVE'
|
||||||
|
WHEN random() < 0.85 THEN 'NEUTRAL'
|
||||||
|
ELSE 'NEGATIVE'
|
||||||
|
END as sentiment
|
||||||
|
FROM customers
|
||||||
|
WHERE random() < 0.3 -- 30% of customers have satisfaction data
|
||||||
|
LIMIT 30000;
|
||||||
|
EOF
|
||||||
|
|
||||||
|
execute_sql_file /tmp/generate_satisfaction.sql > /dev/null
|
||||||
|
echo -e "${GREEN}✓ Generated ~30,000 satisfaction records${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# GENERATE SALES ANALYTICS DATA
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
echo -e "${YELLOW}[13/15] Generating Sales Transactions...${NC}"
|
||||||
|
cat > /tmp/generate_sales.sql << 'EOF'
|
||||||
|
-- Link transactions to products
|
||||||
|
INSERT INTO sales_transactions (
|
||||||
|
transaction_id, product_id, quantity, unit_price, discount_amount,
|
||||||
|
tax_amount, total_amount, sale_date, sales_channel, region
|
||||||
|
)
|
||||||
|
SELECT
|
||||||
|
t.transaction_id,
|
||||||
|
((t.transaction_id % 24) + 1) as product_id, -- Cycle through 24 products
|
||||||
|
(1 + (random() * 3)::INT) as quantity,
|
||||||
|
t.amount / (1 + (random() * 3)::INT) as unit_price,
|
||||||
|
CASE WHEN random() < 0.2 THEN t.amount * 0.1 ELSE 0 END as discount_amount,
|
||||||
|
t.amount * 0.08 as tax_amount,
|
||||||
|
t.amount as total_amount,
|
||||||
|
t.transaction_date as sale_date,
|
||||||
|
CASE (t.transaction_id % 4)
|
||||||
|
WHEN 0 THEN 'ONLINE'
|
||||||
|
WHEN 1 THEN 'STORE'
|
||||||
|
WHEN 2 THEN 'PHONE'
|
||||||
|
ELSE 'MOBILE_APP'
|
||||||
|
END as sales_channel,
|
||||||
|
CASE (t.transaction_id % 5)
|
||||||
|
WHEN 0 THEN 'Northeast'
|
||||||
|
WHEN 1 THEN 'Southeast'
|
||||||
|
WHEN 2 THEN 'Midwest'
|
||||||
|
WHEN 3 THEN 'Southwest'
|
||||||
|
ELSE 'West'
|
||||||
|
END as region
|
||||||
|
FROM transactions t
|
||||||
|
WHERE t.status = 'COMPLETED'
|
||||||
|
LIMIT 1000000; -- Link 1M transactions to products
|
||||||
|
EOF
|
||||||
|
|
||||||
|
execute_sql_file /tmp/generate_sales.sql > /dev/null
|
||||||
|
echo -e "${GREEN}✓ Generated 1,000,000 sales transaction records${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# GENERATE KPI & METRICS DATA
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
echo -e "${YELLOW}[14/15] Generating Daily Metrics...${NC}"
|
||||||
|
cat > /tmp/generate_metrics.sql << 'EOF'
|
||||||
|
-- Generate daily metrics for the past 90 days
|
||||||
|
INSERT INTO daily_metrics (
|
||||||
|
metric_date, kpi_id, metric_value, vs_previous_day_percentage, status
|
||||||
|
)
|
||||||
|
SELECT
|
||||||
|
date_series.metric_date,
|
||||||
|
kpi.kpi_id,
|
||||||
|
kpi.target_value * (0.8 + random() * 0.4) as metric_value,
|
||||||
|
(-20 + random() * 40)::DECIMAL(5,2) as vs_previous_day_percentage,
|
||||||
|
CASE
|
||||||
|
WHEN random() < 0.7 THEN 'ON_TARGET'
|
||||||
|
WHEN random() < 0.9 THEN 'WARNING'
|
||||||
|
ELSE 'CRITICAL'
|
||||||
|
END as status
|
||||||
|
FROM generate_series(
|
||||||
|
CURRENT_DATE - INTERVAL '90 days',
|
||||||
|
CURRENT_DATE,
|
||||||
|
INTERVAL '1 day'
|
||||||
|
) AS date_series(metric_date)
|
||||||
|
CROSS JOIN kpi_definitions kpi
|
||||||
|
WHERE kpi.is_active = TRUE;
|
||||||
|
EOF
|
||||||
|
|
||||||
|
execute_sql_file /tmp/generate_metrics.sql > /dev/null
|
||||||
|
echo -e "${GREEN}✓ Generated 90 days of daily metrics${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
echo -e "${YELLOW}[15/15] Generating Monthly Summaries...${NC}"
|
||||||
|
cat > /tmp/generate_monthly.sql << 'EOF'
|
||||||
|
-- Generate monthly summaries for the past 24 months
|
||||||
|
INSERT INTO monthly_summaries (
|
||||||
|
summary_month, summary_year, total_revenue, total_transactions,
|
||||||
|
total_customers, new_customers, average_transaction_value
|
||||||
|
)
|
||||||
|
SELECT
|
||||||
|
EXTRACT(MONTH FROM month_series)::INT as summary_month,
|
||||||
|
EXTRACT(YEAR FROM month_series)::INT as summary_year,
|
||||||
|
(10000000 + random() * 5000000)::DECIMAL(15,2) as total_revenue,
|
||||||
|
(50000 + (random() * 30000)::INT) as total_transactions,
|
||||||
|
(80000 + (random() * 20000)::INT) as total_customers,
|
||||||
|
(500 + (random() * 1500)::INT) as new_customers,
|
||||||
|
(100 + random() * 100)::DECIMAL(15,2) as average_transaction_value
|
||||||
|
FROM generate_series(
|
||||||
|
CURRENT_DATE - INTERVAL '24 months',
|
||||||
|
CURRENT_DATE,
|
||||||
|
INTERVAL '1 month'
|
||||||
|
) AS month_series;
|
||||||
|
EOF
|
||||||
|
|
||||||
|
execute_sql_file /tmp/generate_monthly.sql > /dev/null
|
||||||
|
echo -e "${GREEN}✓ Generated 24 months of summaries${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
echo -e "${BLUE}============================================================================${NC}"
|
||||||
|
echo -e "${GREEN}Data Generation Complete!${NC}"
|
||||||
|
echo -e "${BLUE}============================================================================${NC}"
|
||||||
|
echo ""
|
||||||
|
echo -e "${YELLOW}Database Statistics:${NC}"
|
||||||
|
|
||||||
|
customer_count=$(execute_sql "SELECT COUNT(*) FROM customers;" | grep -E '^\s*[0-9]+' | tr -d ' ')
|
||||||
|
echo -e "Customers: ${GREEN}$customer_count${NC}"
|
||||||
|
|
||||||
|
account_count=$(execute_sql "SELECT COUNT(*) FROM accounts;" | grep -E '^\s*[0-9]+' | tr -d ' ')
|
||||||
|
echo -e "Accounts: ${GREEN}$account_count${NC}"
|
||||||
|
|
||||||
|
merchant_count=$(execute_sql "SELECT COUNT(*) FROM merchants;" | grep -E '^\s*[0-9]+' | tr -d ' ')
|
||||||
|
echo -e "Merchants: ${GREEN}$merchant_count${NC}"
|
||||||
|
|
||||||
|
card_count=$(execute_sql "SELECT COUNT(*) FROM cards;" | grep -E '^\s*[0-9]+' | tr -d ' ')
|
||||||
|
echo -e "Cards: ${GREEN}$card_count${NC}"
|
||||||
|
|
||||||
|
transaction_count=$(execute_sql "SELECT COUNT(*) FROM transactions;" | grep -E '^\s*[0-9]+' | tr -d ' ')
|
||||||
|
echo -e "Transactions: ${GREEN}$transaction_count${NC}"
|
||||||
|
|
||||||
|
alert_count=$(execute_sql "SELECT COUNT(*) FROM alerts;" | grep -E '^\s*[0-9]+' | tr -d ' ')
|
||||||
|
echo -e "Alerts: ${GREEN}$alert_count${NC}"
|
||||||
|
|
||||||
|
case_count=$(execute_sql "SELECT COUNT(*) FROM fraud_cases;" | grep -E '^\s*[0-9]+' | tr -d ' ')
|
||||||
|
echo -e "Fraud Cases: ${GREEN}$case_count${NC}"
|
||||||
|
|
||||||
|
clv_count=$(execute_sql "SELECT COUNT(*) FROM customer_lifetime_value;" | grep -E '^\s*[0-9]+' | tr -d ' ')
|
||||||
|
echo -e "Customer CLV Records: ${GREEN}$clv_count${NC}"
|
||||||
|
|
||||||
|
sales_count=$(execute_sql "SELECT COUNT(*) FROM sales_transactions;" | grep -E '^\s*[0-9]+' | tr -d ' ')
|
||||||
|
echo -e "Sales Records: ${GREEN}$sales_count${NC}"
|
||||||
|
|
||||||
|
metrics_count=$(execute_sql "SELECT COUNT(*) FROM daily_metrics;" | grep -E '^\s*[0-9]+' | tr -d ' ')
|
||||||
|
echo -e "Daily Metrics: ${GREEN}$metrics_count${NC}"
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo -e "${YELLOW}Ready for Business Analytics and SQL learning!${NC}"
|
||||||
echo -e "Access DB-UI at: ${BLUE}http://localhost:3000${NC}"
|
echo -e "Access DB-UI at: ${BLUE}http://localhost:3000${NC}"
|
||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
|
|||||||
+10
-10
@@ -3,10 +3,10 @@ version: '3.8'
|
|||||||
services:
|
services:
|
||||||
postgres:
|
postgres:
|
||||||
image: postgres:16-alpine
|
image: postgres:16-alpine
|
||||||
container_name: fraud_detection_db
|
container_name: business_analytics_db
|
||||||
environment:
|
environment:
|
||||||
POSTGRES_DB: fraud_detection
|
POSTGRES_DB: business_analytics
|
||||||
POSTGRES_USER: fraud_analyst
|
POSTGRES_USER: data_analyst
|
||||||
POSTGRES_PASSWORD: SecurePass123!
|
POSTGRES_PASSWORD: SecurePass123!
|
||||||
PGDATA: /var/lib/postgresql/data/pgdata
|
PGDATA: /var/lib/postgresql/data/pgdata
|
||||||
ports:
|
ports:
|
||||||
@@ -16,9 +16,9 @@ services:
|
|||||||
- ./schema:/docker-entrypoint-initdb.d
|
- ./schema:/docker-entrypoint-initdb.d
|
||||||
- ./data:/data
|
- ./data:/data
|
||||||
networks:
|
networks:
|
||||||
- fraud_network
|
- analytics_network
|
||||||
healthcheck:
|
healthcheck:
|
||||||
test: ["CMD-SHELL", "pg_isready -U fraud_analyst -d fraud_detection"]
|
test: ["CMD-SHELL", "pg_isready -U data_analyst -d business_analytics"]
|
||||||
interval: 10s
|
interval: 10s
|
||||||
timeout: 5s
|
timeout: 5s
|
||||||
retries: 5
|
retries: 5
|
||||||
@@ -26,24 +26,24 @@ services:
|
|||||||
|
|
||||||
db-ui:
|
db-ui:
|
||||||
image: ghcr.io/n7olkachev/db-ui:latest
|
image: ghcr.io/n7olkachev/db-ui:latest
|
||||||
container_name: fraud_detection_ui
|
container_name: business_analytics_ui
|
||||||
environment:
|
environment:
|
||||||
POSTGRES_HOST: postgres
|
POSTGRES_HOST: postgres
|
||||||
POSTGRES_USER: fraud_analyst
|
POSTGRES_USER: data_analyst
|
||||||
POSTGRES_PASSWORD: SecurePass123!
|
POSTGRES_PASSWORD: SecurePass123!
|
||||||
POSTGRES_DB: fraud_detection
|
POSTGRES_DB: business_analytics
|
||||||
POSTGRES_PORT: 5432
|
POSTGRES_PORT: 5432
|
||||||
ports:
|
ports:
|
||||||
- "3000:3000"
|
- "3000:3000"
|
||||||
networks:
|
networks:
|
||||||
- fraud_network
|
- analytics_network
|
||||||
depends_on:
|
depends_on:
|
||||||
postgres:
|
postgres:
|
||||||
condition: service_healthy
|
condition: service_healthy
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
|
|
||||||
networks:
|
networks:
|
||||||
fraud_network:
|
analytics_network:
|
||||||
driver: bridge
|
driver: bridge
|
||||||
|
|
||||||
volumes:
|
volumes:
|
||||||
|
|||||||
@@ -0,0 +1,358 @@
|
|||||||
|
# Business Analytics Database - Data Models
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
This database supports **Data Analyst** activities including:
|
||||||
|
- Routine and semi-routine analysis
|
||||||
|
- Dashboard creation and maintenance
|
||||||
|
- Business trend identification
|
||||||
|
- Statistical analysis and pattern recognition
|
||||||
|
- Stakeholder reporting
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Model 1: Customer Analytics (Routine Analysis)
|
||||||
|
|
||||||
|
### Purpose
|
||||||
|
Support customer segmentation, lifetime value analysis, churn prediction, and engagement tracking.
|
||||||
|
|
||||||
|
### Tables
|
||||||
|
|
||||||
|
#### `customer_segments`
|
||||||
|
Customer classification for targeted analysis
|
||||||
|
```sql
|
||||||
|
- segment_id (PK)
|
||||||
|
- segment_name (e.g., 'High Value', 'At Risk', 'New Customer')
|
||||||
|
- segment_description
|
||||||
|
- criteria_definition (JSON)
|
||||||
|
- created_date
|
||||||
|
- updated_date
|
||||||
|
```
|
||||||
|
|
||||||
|
#### `customer_lifetime_value`
|
||||||
|
CLV calculations for business insights
|
||||||
|
```sql
|
||||||
|
- clv_id (PK)
|
||||||
|
- customer_id (FK → customers)
|
||||||
|
- calculation_date
|
||||||
|
- total_revenue
|
||||||
|
- total_transactions
|
||||||
|
- average_order_value
|
||||||
|
- predicted_future_value
|
||||||
|
- clv_score
|
||||||
|
- segment_id (FK → customer_segments)
|
||||||
|
```
|
||||||
|
|
||||||
|
#### `churn_predictions`
|
||||||
|
Customer retention analysis
|
||||||
|
```sql
|
||||||
|
- prediction_id (PK)
|
||||||
|
- customer_id (FK → customers)
|
||||||
|
- prediction_date
|
||||||
|
- churn_probability (0-100)
|
||||||
|
- risk_level (LOW, MEDIUM, HIGH, CRITICAL)
|
||||||
|
- last_transaction_date
|
||||||
|
- days_since_last_transaction
|
||||||
|
- engagement_score
|
||||||
|
- recommended_action
|
||||||
|
```
|
||||||
|
|
||||||
|
#### `customer_satisfaction`
|
||||||
|
Satisfaction scores and feedback
|
||||||
|
```sql
|
||||||
|
- satisfaction_id (PK)
|
||||||
|
- customer_id (FK → customers)
|
||||||
|
- survey_date
|
||||||
|
- nps_score (-100 to 100)
|
||||||
|
- csat_score (1-5)
|
||||||
|
- feedback_text
|
||||||
|
- category (PRODUCT, SERVICE, SUPPORT, etc.)
|
||||||
|
- sentiment (POSITIVE, NEUTRAL, NEGATIVE)
|
||||||
|
```
|
||||||
|
|
||||||
|
#### `engagement_metrics`
|
||||||
|
Customer interaction tracking
|
||||||
|
```sql
|
||||||
|
- metric_id (PK)
|
||||||
|
- customer_id (FK → customers)
|
||||||
|
- metric_date
|
||||||
|
- login_count
|
||||||
|
- page_views
|
||||||
|
- time_spent_minutes
|
||||||
|
- features_used
|
||||||
|
- support_tickets_opened
|
||||||
|
- engagement_score
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Model 2: Sales & Revenue Analytics (Semi-Routine Reporting)
|
||||||
|
|
||||||
|
### Purpose
|
||||||
|
Support sales performance tracking, revenue forecasting, and product analysis.
|
||||||
|
|
||||||
|
### Tables
|
||||||
|
|
||||||
|
#### `product_catalog`
|
||||||
|
Product master data
|
||||||
|
```sql
|
||||||
|
- product_id (PK)
|
||||||
|
- product_name
|
||||||
|
- product_category
|
||||||
|
- product_subcategory
|
||||||
|
- unit_price
|
||||||
|
- cost_price
|
||||||
|
- margin_percentage
|
||||||
|
- is_active
|
||||||
|
- launch_date
|
||||||
|
- discontinued_date
|
||||||
|
```
|
||||||
|
|
||||||
|
#### `sales_transactions`
|
||||||
|
Detailed sales records (extends existing transactions)
|
||||||
|
```sql
|
||||||
|
- sale_id (PK)
|
||||||
|
- transaction_id (FK → transactions)
|
||||||
|
- product_id (FK → product_catalog)
|
||||||
|
- quantity
|
||||||
|
- unit_price
|
||||||
|
- discount_amount
|
||||||
|
- tax_amount
|
||||||
|
- total_amount
|
||||||
|
- sale_date
|
||||||
|
- sales_channel (ONLINE, STORE, PHONE, MOBILE_APP)
|
||||||
|
- sales_rep_id
|
||||||
|
- region
|
||||||
|
```
|
||||||
|
|
||||||
|
#### `sales_targets`
|
||||||
|
Performance goals for tracking
|
||||||
|
```sql
|
||||||
|
- target_id (PK)
|
||||||
|
- target_period (DAILY, WEEKLY, MONTHLY, QUARTERLY, YEARLY)
|
||||||
|
- start_date
|
||||||
|
- end_date
|
||||||
|
- product_category
|
||||||
|
- region
|
||||||
|
- target_revenue
|
||||||
|
- target_units
|
||||||
|
- target_customers
|
||||||
|
- created_by
|
||||||
|
```
|
||||||
|
|
||||||
|
#### `sales_performance`
|
||||||
|
Aggregated performance metrics
|
||||||
|
```sql
|
||||||
|
- performance_id (PK)
|
||||||
|
- period_date
|
||||||
|
- period_type (DAILY, WEEKLY, MONTHLY, QUARTERLY)
|
||||||
|
- product_id (FK → product_catalog)
|
||||||
|
- region
|
||||||
|
- total_revenue
|
||||||
|
- total_units_sold
|
||||||
|
- total_transactions
|
||||||
|
- unique_customers
|
||||||
|
- average_order_value
|
||||||
|
- vs_target_percentage
|
||||||
|
```
|
||||||
|
|
||||||
|
#### `revenue_forecasts`
|
||||||
|
Predictive revenue analysis
|
||||||
|
```sql
|
||||||
|
- forecast_id (PK)
|
||||||
|
- forecast_date
|
||||||
|
- forecast_period_start
|
||||||
|
- forecast_period_end
|
||||||
|
- product_category
|
||||||
|
- region
|
||||||
|
- forecasted_revenue
|
||||||
|
- confidence_level (LOW, MEDIUM, HIGH)
|
||||||
|
- forecast_method (HISTORICAL, TREND, SEASONAL, ML)
|
||||||
|
- actual_revenue (filled after period ends)
|
||||||
|
- variance_percentage
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Model 3: Operational Metrics & KPIs (Dashboard Creation)
|
||||||
|
|
||||||
|
### Purpose
|
||||||
|
Support dashboard creation with pre-calculated KPIs and trend analysis.
|
||||||
|
|
||||||
|
### Tables
|
||||||
|
|
||||||
|
#### `kpi_definitions`
|
||||||
|
Master list of tracked KPIs
|
||||||
|
```sql
|
||||||
|
- kpi_id (PK)
|
||||||
|
- kpi_name
|
||||||
|
- kpi_description
|
||||||
|
- kpi_category (SALES, CUSTOMER, OPERATIONAL, FINANCIAL)
|
||||||
|
- calculation_formula
|
||||||
|
- target_value
|
||||||
|
- threshold_warning
|
||||||
|
- threshold_critical
|
||||||
|
- unit_of_measure
|
||||||
|
- refresh_frequency (REALTIME, HOURLY, DAILY, WEEKLY)
|
||||||
|
- is_active
|
||||||
|
```
|
||||||
|
|
||||||
|
#### `daily_metrics`
|
||||||
|
Daily operational snapshots
|
||||||
|
```sql
|
||||||
|
- metric_id (PK)
|
||||||
|
- metric_date
|
||||||
|
- kpi_id (FK → kpi_definitions)
|
||||||
|
- metric_value
|
||||||
|
- vs_previous_day_percentage
|
||||||
|
- vs_previous_week_percentage
|
||||||
|
- vs_previous_month_percentage
|
||||||
|
- status (ON_TARGET, WARNING, CRITICAL)
|
||||||
|
- notes
|
||||||
|
```
|
||||||
|
|
||||||
|
#### `monthly_summaries`
|
||||||
|
Monthly aggregated data for reporting
|
||||||
|
```sql
|
||||||
|
- summary_id (PK)
|
||||||
|
- summary_month
|
||||||
|
- summary_year
|
||||||
|
- total_revenue
|
||||||
|
- total_transactions
|
||||||
|
- total_customers
|
||||||
|
- new_customers
|
||||||
|
- churned_customers
|
||||||
|
- average_transaction_value
|
||||||
|
- customer_acquisition_cost
|
||||||
|
- customer_lifetime_value
|
||||||
|
- net_promoter_score
|
||||||
|
- gross_margin_percentage
|
||||||
|
```
|
||||||
|
|
||||||
|
#### `trend_analysis`
|
||||||
|
Statistical trend tracking
|
||||||
|
```sql
|
||||||
|
- trend_id (PK)
|
||||||
|
- kpi_id (FK → kpi_definitions)
|
||||||
|
- analysis_date
|
||||||
|
- period_start
|
||||||
|
- period_end
|
||||||
|
- trend_direction (UP, DOWN, FLAT)
|
||||||
|
- trend_strength (WEAK, MODERATE, STRONG)
|
||||||
|
- moving_average_7day
|
||||||
|
- moving_average_30day
|
||||||
|
- seasonality_detected
|
||||||
|
- anomalies_detected
|
||||||
|
- statistical_significance
|
||||||
|
```
|
||||||
|
|
||||||
|
#### `dashboard_snapshots`
|
||||||
|
Pre-calculated dashboard data
|
||||||
|
```sql
|
||||||
|
- snapshot_id (PK)
|
||||||
|
- dashboard_name
|
||||||
|
- snapshot_timestamp
|
||||||
|
- data_payload (JSON)
|
||||||
|
- refresh_duration_seconds
|
||||||
|
- row_count
|
||||||
|
- last_updated_by
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Model 4: Business Intelligence & Reporting
|
||||||
|
|
||||||
|
### Purpose
|
||||||
|
Support ad-hoc analysis and standard report generation.
|
||||||
|
|
||||||
|
### Tables
|
||||||
|
|
||||||
|
#### `report_definitions`
|
||||||
|
Catalog of standard reports
|
||||||
|
```sql
|
||||||
|
- report_id (PK)
|
||||||
|
- report_name
|
||||||
|
- report_description
|
||||||
|
- report_category
|
||||||
|
- sql_query_template
|
||||||
|
- parameters (JSON)
|
||||||
|
- output_format (PDF, EXCEL, CSV, HTML)
|
||||||
|
- schedule_frequency
|
||||||
|
- recipients
|
||||||
|
- is_active
|
||||||
|
```
|
||||||
|
|
||||||
|
#### `report_executions`
|
||||||
|
Report run history
|
||||||
|
```sql
|
||||||
|
- execution_id (PK)
|
||||||
|
- report_id (FK → report_definitions)
|
||||||
|
- execution_timestamp
|
||||||
|
- parameters_used (JSON)
|
||||||
|
- row_count
|
||||||
|
- execution_duration_seconds
|
||||||
|
- status (SUCCESS, FAILED, TIMEOUT)
|
||||||
|
- error_message
|
||||||
|
- output_file_path
|
||||||
|
- executed_by
|
||||||
|
```
|
||||||
|
|
||||||
|
#### `data_quality_checks`
|
||||||
|
Data validation tracking
|
||||||
|
```sql
|
||||||
|
- check_id (PK)
|
||||||
|
- check_name
|
||||||
|
- table_name
|
||||||
|
- column_name
|
||||||
|
- check_type (NULL_CHECK, RANGE_CHECK, UNIQUENESS, REFERENTIAL_INTEGRITY)
|
||||||
|
- check_date
|
||||||
|
- records_checked
|
||||||
|
- records_failed
|
||||||
|
- failure_percentage
|
||||||
|
- status (PASS, FAIL, WARNING)
|
||||||
|
- remediation_notes
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Relationships to Existing Models
|
||||||
|
|
||||||
|
### Integration Points
|
||||||
|
|
||||||
|
1. **Customer Analytics** ← links to existing `customers` table
|
||||||
|
2. **Sales Transactions** ← extends existing `transactions` table
|
||||||
|
3. **Engagement Metrics** ← links to `login_sessions` and `transactions`
|
||||||
|
4. **Churn Predictions** ← analyzes `transactions` and `accounts`
|
||||||
|
5. **KPIs** ← aggregates from multiple existing tables
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Use Cases for Data Analyst Role
|
||||||
|
|
||||||
|
### Routine Analysis
|
||||||
|
- Daily sales reports
|
||||||
|
- Customer segment updates
|
||||||
|
- KPI dashboard refreshes
|
||||||
|
- Data quality checks
|
||||||
|
|
||||||
|
### Semi-Routine Analysis
|
||||||
|
- Monthly trend analysis
|
||||||
|
- Churn prediction updates
|
||||||
|
- Revenue forecasting
|
||||||
|
- Performance vs. targets
|
||||||
|
|
||||||
|
### Ad-Hoc Analysis
|
||||||
|
- Customer cohort analysis
|
||||||
|
- Product performance deep-dives
|
||||||
|
- Seasonal pattern identification
|
||||||
|
- Anomaly investigation
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
|
||||||
|
1. ✅ Design complete
|
||||||
|
2. ⏳ Implement schema in `schema/01-create-tables.sql`
|
||||||
|
3. ⏳ Update data generation scripts
|
||||||
|
4. ⏳ Create SQL exercises for each model
|
||||||
|
5. ⏳ Update documentation
|
||||||
|
|
||||||
+64
-33
@@ -19,7 +19,7 @@ docker-compose up -d
|
|||||||
docker-compose ps
|
docker-compose ps
|
||||||
```
|
```
|
||||||
|
|
||||||
You should see both `fraud_detection_db` and `fraud_detection_ui` running.
|
You should see both `business_analytics_db` and `business_analytics_ui` running.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -34,9 +34,9 @@ chmod +x scripts/setup-database.sh
|
|||||||
```
|
```
|
||||||
|
|
||||||
**What this does:**
|
**What this does:**
|
||||||
- Creates all 20+ tables
|
- Creates all 39 tables across 4 business models
|
||||||
- Sets up indexes and constraints
|
- Sets up indexes and constraints
|
||||||
- Loads reference data (countries, merchant categories, etc.)
|
- Loads reference data (countries, merchant categories, customer segments, products, KPIs, etc.)
|
||||||
|
|
||||||
**Expected output:**
|
**Expected output:**
|
||||||
```
|
```
|
||||||
@@ -63,6 +63,9 @@ chmod +x data/generate_data.sh
|
|||||||
- Creates 150,000 accounts
|
- Creates 150,000 accounts
|
||||||
- Generates 5,000,000 transactions
|
- Generates 5,000,000 transactions
|
||||||
- Creates fraud patterns and alerts
|
- Creates fraud patterns and alerts
|
||||||
|
- Generates customer analytics (CLV, churn, satisfaction)
|
||||||
|
- Creates sales data (1M sales records)
|
||||||
|
- Generates KPI metrics (90 days of daily metrics)
|
||||||
|
|
||||||
**⏱️ Time estimate:**
|
**⏱️ Time estimate:**
|
||||||
- Fast machine (SSD, 16GB RAM): ~15 minutes
|
- Fast machine (SSD, 16GB RAM): ~15 minutes
|
||||||
@@ -72,10 +75,16 @@ chmod +x data/generate_data.sh
|
|||||||
**You can monitor progress:**
|
**You can monitor progress:**
|
||||||
The script shows progress for each step:
|
The script shows progress for each step:
|
||||||
```
|
```
|
||||||
[1/9] Loading geographic reference data...
|
[1/15] Loading geographic reference data...
|
||||||
[2/9] Generating Customers...
|
[2/15] Generating Customers...
|
||||||
[3/9] Generating Accounts...
|
[3/15] Generating Accounts...
|
||||||
...
|
...
|
||||||
|
[10/15] Generating Customer Lifetime Value data...
|
||||||
|
[11/15] Generating Churn Predictions...
|
||||||
|
[12/15] Generating Customer Satisfaction data...
|
||||||
|
[13/15] Generating Sales Transactions...
|
||||||
|
[14/15] Generating Daily Metrics...
|
||||||
|
[15/15] Generating Monthly Summaries...
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -98,7 +107,7 @@ The script shows progress for each step:
|
|||||||
#### Option B: Command Line (psql)
|
#### Option B: Command Line (psql)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker exec -it fraud_detection_db psql -U fraud_analyst -d fraud_detection
|
docker exec -it business_analytics_db psql -U data_analyst -d business_analytics
|
||||||
```
|
```
|
||||||
|
|
||||||
**Quick commands:**
|
**Quick commands:**
|
||||||
@@ -122,8 +131,8 @@ SELECT COUNT(*) FROM transactions;
|
|||||||
```
|
```
|
||||||
Host: localhost
|
Host: localhost
|
||||||
Port: 5432
|
Port: 5432
|
||||||
Database: fraud_detection
|
Database: business_analytics
|
||||||
Username: fraud_analyst
|
Username: data_analyst
|
||||||
Password: SecurePass123!
|
Password: SecurePass123!
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -146,43 +155,65 @@ SELECT COUNT(*) FROM customers;
|
|||||||
-- How many transactions?
|
-- How many transactions?
|
||||||
SELECT COUNT(*) FROM transactions;
|
SELECT COUNT(*) FROM transactions;
|
||||||
|
|
||||||
-- How many fraud alerts?
|
-- How many sales records?
|
||||||
SELECT COUNT(*) FROM alerts WHERE status = 'OPEN';
|
SELECT COUNT(*) FROM sales_transactions;
|
||||||
|
|
||||||
|
-- How many KPIs are being tracked?
|
||||||
|
SELECT COUNT(*) FROM kpi_definitions WHERE is_active = TRUE;
|
||||||
```
|
```
|
||||||
|
|
||||||
### 2. Find High-Risk Customers
|
### 2. Customer Analytics: High-Value Customers
|
||||||
|
|
||||||
```sql
|
```sql
|
||||||
SELECT
|
SELECT
|
||||||
customer_id,
|
c.customer_id,
|
||||||
first_name,
|
c.first_name,
|
||||||
last_name,
|
c.last_name,
|
||||||
email,
|
clv.clv_score,
|
||||||
risk_score
|
cs.segment_name
|
||||||
FROM customers
|
FROM customers c
|
||||||
WHERE risk_score > 80
|
JOIN customer_lifetime_value clv ON c.customer_id = clv.customer_id
|
||||||
ORDER BY risk_score DESC
|
JOIN customer_segments cs ON clv.segment_id = cs.segment_id
|
||||||
|
WHERE cs.segment_name IN ('VIP', 'High Value')
|
||||||
|
ORDER BY clv.clv_score DESC
|
||||||
LIMIT 10;
|
LIMIT 10;
|
||||||
```
|
```
|
||||||
|
|
||||||
### 3. View Recent Transactions
|
### 3. Sales Analytics: Top Products
|
||||||
|
|
||||||
```sql
|
```sql
|
||||||
SELECT
|
SELECT
|
||||||
transaction_id,
|
p.product_name,
|
||||||
account_id,
|
p.product_category,
|
||||||
amount,
|
COUNT(st.transaction_id) as sales_count,
|
||||||
transaction_date,
|
SUM(st.total_amount) as total_revenue
|
||||||
is_flagged
|
FROM product_catalog p
|
||||||
FROM transactions
|
JOIN sales_transactions st ON p.product_id = st.product_id
|
||||||
ORDER BY transaction_date DESC
|
GROUP BY p.product_id, p.product_name, p.product_category
|
||||||
LIMIT 20;
|
ORDER BY total_revenue DESC
|
||||||
|
LIMIT 10;
|
||||||
```
|
```
|
||||||
|
|
||||||
### 4. Find Flagged Transactions
|
### 4. KPI Dashboard: Current Status
|
||||||
|
|
||||||
```sql
|
```sql
|
||||||
SELECT
|
SELECT
|
||||||
|
kd.kpi_name,
|
||||||
|
kd.kpi_category,
|
||||||
|
dm.metric_value,
|
||||||
|
kd.target_value,
|
||||||
|
dm.status
|
||||||
|
FROM kpi_definitions kd
|
||||||
|
JOIN daily_metrics dm ON kd.kpi_id = dm.kpi_id
|
||||||
|
WHERE dm.metric_date = CURRENT_DATE
|
||||||
|
AND kd.is_active = TRUE
|
||||||
|
ORDER BY kd.kpi_category, kd.kpi_name;
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5. Fraud Detection: Flagged Transactions
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
t.transaction_id,
|
t.transaction_id,
|
||||||
t.amount,
|
t.amount,
|
||||||
t.fraud_score,
|
t.fraud_score,
|
||||||
|
|||||||
@@ -0,0 +1,266 @@
|
|||||||
|
# What's New - Business Analytics Database
|
||||||
|
|
||||||
|
## 🎯 Major Update: Aligned with Data Analyst Role
|
||||||
|
|
||||||
|
The database has been **expanded and refocused** to align with real-world **Data Analyst** responsibilities:
|
||||||
|
|
||||||
|
- ✅ Routine and semi-routine analysis
|
||||||
|
- ✅ Dashboard creation and maintenance
|
||||||
|
- ✅ Business trend identification
|
||||||
|
- ✅ Statistical analysis and pattern recognition
|
||||||
|
- ✅ Stakeholder reporting
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📊 Database Renamed
|
||||||
|
|
||||||
|
**Old Name:** `fraud_detection`
|
||||||
|
**New Name:** `business_analytics`
|
||||||
|
|
||||||
|
**User Changed:** `fraud_analyst` → `data_analyst`
|
||||||
|
|
||||||
|
This better reflects the broader scope of business intelligence and analytics work.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🆕 New Data Models Added
|
||||||
|
|
||||||
|
### 1. Customer Analytics Model (5 Tables)
|
||||||
|
**Purpose:** Routine customer analysis and segmentation
|
||||||
|
|
||||||
|
#### New Tables:
|
||||||
|
- **`customer_segments`** - Customer classification definitions
|
||||||
|
- **`customer_lifetime_value`** - CLV calculations and tracking
|
||||||
|
- **`churn_predictions`** - Customer retention risk analysis
|
||||||
|
- **`customer_satisfaction`** - NPS, CSAT scores, and feedback
|
||||||
|
- **`engagement_metrics`** - Customer interaction tracking
|
||||||
|
|
||||||
|
**Use Cases:**
|
||||||
|
- Customer segmentation analysis
|
||||||
|
- Churn risk identification
|
||||||
|
- Lifetime value calculations
|
||||||
|
- Satisfaction trend analysis
|
||||||
|
- Engagement scoring
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 2. Sales & Revenue Analytics Model (6 Tables)
|
||||||
|
**Purpose:** Semi-routine sales reporting and forecasting
|
||||||
|
|
||||||
|
#### New Tables:
|
||||||
|
- **`product_catalog`** - Product master data
|
||||||
|
- **`sales_transactions`** - Detailed sales records
|
||||||
|
- **`sales_targets`** - Performance goals and targets
|
||||||
|
- **`sales_performance`** - Aggregated performance metrics
|
||||||
|
- **`revenue_forecasts`** - Revenue predictions and variance
|
||||||
|
|
||||||
|
**Use Cases:**
|
||||||
|
- Sales performance dashboards
|
||||||
|
- Product analysis
|
||||||
|
- Revenue forecasting
|
||||||
|
- Target vs. actual analysis
|
||||||
|
- Channel performance comparison
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 3. Operational Metrics & KPI Model (5 Tables)
|
||||||
|
**Purpose:** Dashboard creation and KPI tracking
|
||||||
|
|
||||||
|
#### New Tables:
|
||||||
|
- **`kpi_definitions`** - Master KPI catalog
|
||||||
|
- **`daily_metrics`** - Daily operational snapshots
|
||||||
|
- **`monthly_summaries`** - Monthly business summaries
|
||||||
|
- **`trend_analysis`** - Statistical trend tracking
|
||||||
|
- **`dashboard_snapshots`** - Pre-calculated dashboard data
|
||||||
|
|
||||||
|
**Use Cases:**
|
||||||
|
- Executive dashboards
|
||||||
|
- KPI monitoring
|
||||||
|
- Trend analysis
|
||||||
|
- Performance tracking
|
||||||
|
- Anomaly detection
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### 4. Business Intelligence & Reporting Model (3 Tables)
|
||||||
|
**Purpose:** Report management and data quality
|
||||||
|
|
||||||
|
#### New Tables:
|
||||||
|
- **`report_definitions`** - Standard report catalog
|
||||||
|
- **`report_executions`** - Report run history
|
||||||
|
- **`data_quality_checks`** - Data validation tracking
|
||||||
|
|
||||||
|
**Use Cases:**
|
||||||
|
- Report scheduling
|
||||||
|
- Execution monitoring
|
||||||
|
- Data quality assurance
|
||||||
|
- Audit trails
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📈 Total Database Size
|
||||||
|
|
||||||
|
### Original (Fraud Detection Only):
|
||||||
|
- **20 tables** focused on fraud investigation
|
||||||
|
|
||||||
|
### Updated (Business Analytics):
|
||||||
|
- **39 tables** covering:
|
||||||
|
- Fraud detection (original 20 tables)
|
||||||
|
- Customer analytics (5 tables)
|
||||||
|
- Sales & revenue (6 tables)
|
||||||
|
- KPIs & metrics (5 tables)
|
||||||
|
- Reporting & BI (3 tables)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔗 Integration with Existing Data
|
||||||
|
|
||||||
|
The new models **integrate seamlessly** with existing tables:
|
||||||
|
|
||||||
|
```
|
||||||
|
customers → customer_lifetime_value
|
||||||
|
→ churn_predictions
|
||||||
|
→ customer_satisfaction
|
||||||
|
→ engagement_metrics
|
||||||
|
|
||||||
|
transactions → sales_transactions
|
||||||
|
→ sales_performance
|
||||||
|
→ revenue_forecasts
|
||||||
|
|
||||||
|
(all tables) → kpi_definitions
|
||||||
|
→ daily_metrics
|
||||||
|
→ monthly_summaries
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 💼 Data Analyst Workflows Supported
|
||||||
|
|
||||||
|
### Daily Routine Tasks
|
||||||
|
1. **Refresh dashboards** using `dashboard_snapshots`
|
||||||
|
2. **Update daily metrics** in `daily_metrics`
|
||||||
|
3. **Run data quality checks** via `data_quality_checks`
|
||||||
|
4. **Generate standard reports** from `report_definitions`
|
||||||
|
|
||||||
|
### Weekly Semi-Routine Tasks
|
||||||
|
1. **Customer segmentation** analysis using `customer_segments` and `customer_lifetime_value`
|
||||||
|
2. **Sales performance** review via `sales_performance` vs `sales_targets`
|
||||||
|
3. **Churn prediction** updates in `churn_predictions`
|
||||||
|
4. **Trend analysis** using `trend_analysis` table
|
||||||
|
|
||||||
|
### Monthly Analysis
|
||||||
|
1. **Monthly summaries** generation in `monthly_summaries`
|
||||||
|
2. **Revenue forecasting** via `revenue_forecasts`
|
||||||
|
3. **Customer satisfaction** trend analysis
|
||||||
|
4. **KPI performance** review
|
||||||
|
|
||||||
|
### Ad-Hoc Analysis
|
||||||
|
1. **Product performance** deep-dives
|
||||||
|
2. **Customer cohort** analysis
|
||||||
|
3. **Seasonal pattern** identification
|
||||||
|
4. **Anomaly investigation**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🎓 New Learning Opportunities
|
||||||
|
|
||||||
|
### For Beginners:
|
||||||
|
- Basic aggregations (SUM, AVG, COUNT)
|
||||||
|
- Simple JOINs across analytics tables
|
||||||
|
- Date-based filtering and grouping
|
||||||
|
- KPI calculations
|
||||||
|
|
||||||
|
### For Intermediate:
|
||||||
|
- Customer segmentation queries
|
||||||
|
- Sales trend analysis
|
||||||
|
- Moving averages and window functions
|
||||||
|
- Cohort analysis
|
||||||
|
|
||||||
|
### For Advanced:
|
||||||
|
- Churn prediction analysis
|
||||||
|
- Revenue forecasting validation
|
||||||
|
- Multi-dimensional analysis
|
||||||
|
- Statistical significance testing
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔄 Migration Notes
|
||||||
|
|
||||||
|
### What Changed:
|
||||||
|
- ✅ Database name: `fraud_detection` → `business_analytics`
|
||||||
|
- ✅ User name: `fraud_analyst` → `data_analyst`
|
||||||
|
- ✅ Container names updated
|
||||||
|
- ✅ All scripts updated
|
||||||
|
- ✅ 19 new tables added
|
||||||
|
|
||||||
|
### What Stayed the Same:
|
||||||
|
- ✅ All original 20 fraud detection tables
|
||||||
|
- ✅ All existing data generation logic
|
||||||
|
- ✅ All existing indexes and constraints
|
||||||
|
- ✅ Idempotent script design
|
||||||
|
- ✅ Docker setup structure
|
||||||
|
|
||||||
|
### Backward Compatibility:
|
||||||
|
- ✅ All original fraud detection exercises still work
|
||||||
|
- ✅ All original queries still valid
|
||||||
|
- ✅ No breaking changes to existing schema
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📝 Next Steps
|
||||||
|
|
||||||
|
### Immediate:
|
||||||
|
1. ✅ Schema updated with 19 new tables
|
||||||
|
2. ⏳ Update data generation scripts
|
||||||
|
3. ⏳ Create SQL exercises for new models
|
||||||
|
4. ⏳ Update main documentation
|
||||||
|
|
||||||
|
### Future Enhancements:
|
||||||
|
- Add sample data for new tables
|
||||||
|
- Create dashboard query examples
|
||||||
|
- Build KPI calculation examples
|
||||||
|
- Add data quality check templates
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🚀 Getting Started with New Models
|
||||||
|
|
||||||
|
### Quick Test Queries:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Check new tables exist
|
||||||
|
SELECT table_name
|
||||||
|
FROM information_schema.tables
|
||||||
|
WHERE table_schema = 'public'
|
||||||
|
AND table_name IN (
|
||||||
|
'customer_segments',
|
||||||
|
'customer_lifetime_value',
|
||||||
|
'product_catalog',
|
||||||
|
'sales_transactions',
|
||||||
|
'kpi_definitions',
|
||||||
|
'daily_metrics'
|
||||||
|
);
|
||||||
|
|
||||||
|
-- View table counts
|
||||||
|
SELECT
|
||||||
|
'customer_segments' as table_name, COUNT(*) FROM customer_segments
|
||||||
|
UNION ALL
|
||||||
|
SELECT 'product_catalog', COUNT(*) FROM product_catalog
|
||||||
|
UNION ALL
|
||||||
|
SELECT 'kpi_definitions', COUNT(*) FROM kpi_definitions;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📚 Documentation Updates
|
||||||
|
|
||||||
|
- ✅ **DATA_MODELS.md** - Complete model documentation
|
||||||
|
- ✅ **WHATS_NEW.md** - This file
|
||||||
|
- ⏳ **README.md** - Update with new scope
|
||||||
|
- ⏳ **QUICKSTART.md** - Add new model examples
|
||||||
|
- ⏳ **Exercises** - Create analytics-focused exercises
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**The database is now a comprehensive Business Analytics platform suitable for Data Analyst training and real-world analysis scenarios!** 🎉
|
||||||
|
|
||||||
@@ -0,0 +1,325 @@
|
|||||||
|
# Level 2: Customer Analytics
|
||||||
|
|
||||||
|
## Introduction
|
||||||
|
Learn to analyze customer behavior, calculate lifetime value, identify churn risks, and segment customers - essential skills for Data Analysts working with customer data.
|
||||||
|
|
||||||
|
## Learning Objectives
|
||||||
|
- Calculate customer lifetime value (CLV)
|
||||||
|
- Identify at-risk customers
|
||||||
|
- Segment customers by behavior
|
||||||
|
- Analyze satisfaction trends
|
||||||
|
- Track engagement metrics
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 2.1: Customer Segmentation
|
||||||
|
**Objective:** Find the distribution of customers across segments
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
cs.segment_name,
|
||||||
|
COUNT(clv.customer_id) as customer_count,
|
||||||
|
AVG(clv.clv_score) as avg_clv_score,
|
||||||
|
SUM(clv.total_revenue) as total_segment_revenue
|
||||||
|
FROM customer_segments cs
|
||||||
|
LEFT JOIN customer_lifetime_value clv ON cs.segment_id = clv.segment_id
|
||||||
|
GROUP BY cs.segment_id, cs.segment_name
|
||||||
|
ORDER BY total_segment_revenue DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
**Expected Result:** Segment breakdown with counts and revenue
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 2.2: High-Value Customers
|
||||||
|
**Objective:** Identify top 10 customers by lifetime value
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Hint:** Use customer_lifetime_value table and ORDER BY
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
c.customer_id,
|
||||||
|
c.first_name,
|
||||||
|
c.last_name,
|
||||||
|
c.email,
|
||||||
|
clv.total_revenue,
|
||||||
|
clv.total_transactions,
|
||||||
|
clv.clv_score,
|
||||||
|
cs.segment_name
|
||||||
|
FROM customer_lifetime_value clv
|
||||||
|
JOIN customers c ON clv.customer_id = c.customer_id
|
||||||
|
JOIN customer_segments cs ON clv.segment_id = cs.segment_id
|
||||||
|
ORDER BY clv.clv_score DESC
|
||||||
|
LIMIT 10;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 2.3: Churn Risk Analysis
|
||||||
|
**Objective:** Find customers at CRITICAL or HIGH churn risk
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
c.customer_id,
|
||||||
|
c.first_name,
|
||||||
|
c.last_name,
|
||||||
|
c.email,
|
||||||
|
cp.churn_probability,
|
||||||
|
cp.risk_level,
|
||||||
|
cp.days_since_last_transaction,
|
||||||
|
cp.engagement_score
|
||||||
|
FROM churn_predictions cp
|
||||||
|
JOIN customers c ON cp.customer_id = c.customer_id
|
||||||
|
WHERE cp.risk_level IN ('CRITICAL', 'HIGH')
|
||||||
|
ORDER BY cp.churn_probability DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 2.4: Customer Satisfaction Trends
|
||||||
|
**Objective:** Calculate average NPS score by month
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Hint:** Use DATE_TRUNC or EXTRACT to group by month
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
DATE_TRUNC('month', survey_date) as month,
|
||||||
|
COUNT(*) as survey_count,
|
||||||
|
AVG(nps_score) as avg_nps,
|
||||||
|
AVG(csat_score) as avg_csat,
|
||||||
|
COUNT(CASE WHEN sentiment = 'POSITIVE' THEN 1 END) as positive_count,
|
||||||
|
COUNT(CASE WHEN sentiment = 'NEGATIVE' THEN 1 END) as negative_count
|
||||||
|
FROM customer_satisfaction
|
||||||
|
GROUP BY DATE_TRUNC('month', survey_date)
|
||||||
|
ORDER BY month DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 2.5: Engagement Score Analysis
|
||||||
|
**Objective:** Find customers with declining engagement
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Hint:** Compare recent engagement to historical average
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
WITH recent_engagement AS (
|
||||||
|
SELECT
|
||||||
|
customer_id,
|
||||||
|
AVG(engagement_score) as recent_score
|
||||||
|
FROM engagement_metrics
|
||||||
|
WHERE metric_date >= CURRENT_DATE - INTERVAL '30 days'
|
||||||
|
GROUP BY customer_id
|
||||||
|
),
|
||||||
|
historical_engagement AS (
|
||||||
|
SELECT
|
||||||
|
customer_id,
|
||||||
|
AVG(engagement_score) as historical_score
|
||||||
|
FROM engagement_metrics
|
||||||
|
WHERE metric_date < CURRENT_DATE - INTERVAL '30 days'
|
||||||
|
GROUP BY customer_id
|
||||||
|
)
|
||||||
|
SELECT
|
||||||
|
c.customer_id,
|
||||||
|
c.first_name,
|
||||||
|
c.last_name,
|
||||||
|
re.recent_score,
|
||||||
|
he.historical_score,
|
||||||
|
(re.recent_score - he.historical_score) as score_change
|
||||||
|
FROM customers c
|
||||||
|
JOIN recent_engagement re ON c.customer_id = re.customer_id
|
||||||
|
JOIN historical_engagement he ON c.customer_id = he.customer_id
|
||||||
|
WHERE re.recent_score < he.historical_score
|
||||||
|
ORDER BY score_change ASC
|
||||||
|
LIMIT 20;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 2.6: Customer Cohort Analysis
|
||||||
|
**Objective:** Analyze customer retention by registration cohort
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
DATE_TRUNC('month', c.registration_date) as cohort_month,
|
||||||
|
COUNT(DISTINCT c.customer_id) as total_customers,
|
||||||
|
COUNT(DISTINCT CASE WHEN cp.risk_level = 'LOW' THEN c.customer_id END) as active_customers,
|
||||||
|
COUNT(DISTINCT CASE WHEN cp.risk_level IN ('HIGH', 'CRITICAL') THEN c.customer_id END) as at_risk_customers,
|
||||||
|
ROUND(100.0 * COUNT(DISTINCT CASE WHEN cp.risk_level = 'LOW' THEN c.customer_id END) /
|
||||||
|
COUNT(DISTINCT c.customer_id), 2) as retention_rate
|
||||||
|
FROM customers c
|
||||||
|
LEFT JOIN churn_predictions cp ON c.customer_id = cp.customer_id
|
||||||
|
WHERE c.registration_date >= CURRENT_DATE - INTERVAL '12 months'
|
||||||
|
GROUP BY DATE_TRUNC('month', c.registration_date)
|
||||||
|
ORDER BY cohort_month DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 2.7: Customer Lifetime Value by Segment
|
||||||
|
**Objective:** Compare CLV metrics across customer segments
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
cs.segment_name,
|
||||||
|
COUNT(clv.customer_id) as customer_count,
|
||||||
|
MIN(clv.clv_score) as min_clv,
|
||||||
|
AVG(clv.clv_score) as avg_clv,
|
||||||
|
MAX(clv.clv_score) as max_clv,
|
||||||
|
PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY clv.clv_score) as median_clv,
|
||||||
|
SUM(clv.total_revenue) as total_revenue,
|
||||||
|
AVG(clv.total_transactions) as avg_transactions
|
||||||
|
FROM customer_segments cs
|
||||||
|
JOIN customer_lifetime_value clv ON cs.segment_id = clv.segment_id
|
||||||
|
GROUP BY cs.segment_id, cs.segment_name
|
||||||
|
ORDER BY avg_clv DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 2.8: Satisfaction by Category
|
||||||
|
**Objective:** Analyze satisfaction scores by feedback category
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
category,
|
||||||
|
COUNT(*) as feedback_count,
|
||||||
|
AVG(nps_score) as avg_nps,
|
||||||
|
AVG(csat_score) as avg_csat,
|
||||||
|
ROUND(100.0 * COUNT(CASE WHEN sentiment = 'POSITIVE' THEN 1 END) / COUNT(*), 2) as positive_pct,
|
||||||
|
ROUND(100.0 * COUNT(CASE WHEN sentiment = 'NEGATIVE' THEN 1 END) / COUNT(*), 2) as negative_pct
|
||||||
|
FROM customer_satisfaction
|
||||||
|
GROUP BY category
|
||||||
|
ORDER BY avg_nps DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 2.9: Customer Engagement Patterns
|
||||||
|
**Objective:** Find most engaged customers in the last 30 days
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
c.customer_id,
|
||||||
|
c.first_name,
|
||||||
|
c.last_name,
|
||||||
|
SUM(em.login_count) as total_logins,
|
||||||
|
SUM(em.page_views) as total_page_views,
|
||||||
|
SUM(em.time_spent_minutes) as total_time_spent,
|
||||||
|
AVG(em.engagement_score) as avg_engagement_score
|
||||||
|
FROM customers c
|
||||||
|
JOIN engagement_metrics em ON c.customer_id = em.customer_id
|
||||||
|
WHERE em.metric_date >= CURRENT_DATE - INTERVAL '30 days'
|
||||||
|
GROUP BY c.customer_id, c.first_name, c.last_name
|
||||||
|
HAVING AVG(em.engagement_score) > 75
|
||||||
|
ORDER BY avg_engagement_score DESC
|
||||||
|
LIMIT 20;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 2.10: Churn Prevention Priority List
|
||||||
|
**Objective:** Create a priority list for customer retention efforts
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Hint:** Combine churn risk with customer value
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
c.customer_id,
|
||||||
|
c.first_name,
|
||||||
|
c.last_name,
|
||||||
|
c.email,
|
||||||
|
clv.clv_score,
|
||||||
|
clv.total_revenue,
|
||||||
|
cp.churn_probability,
|
||||||
|
cp.risk_level,
|
||||||
|
cp.days_since_last_transaction,
|
||||||
|
(clv.clv_score * cp.churn_probability / 100) as retention_priority_score
|
||||||
|
FROM customers c
|
||||||
|
JOIN customer_lifetime_value clv ON c.customer_id = clv.customer_id
|
||||||
|
JOIN churn_predictions cp ON c.customer_id = cp.customer_id
|
||||||
|
WHERE cp.risk_level IN ('HIGH', 'CRITICAL')
|
||||||
|
AND clv.clv_score > 1000
|
||||||
|
ORDER BY retention_priority_score DESC
|
||||||
|
LIMIT 50;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Challenge Exercises
|
||||||
|
|
||||||
|
### Challenge 2.1: Customer Journey Analysis
|
||||||
|
Create a query that shows the customer journey from registration to current status, including:
|
||||||
|
- Registration date
|
||||||
|
- First transaction date
|
||||||
|
- Total transactions
|
||||||
|
- Current segment
|
||||||
|
- Churn risk
|
||||||
|
- Latest satisfaction score
|
||||||
|
|
||||||
|
### Challenge 2.2: Segment Migration Analysis
|
||||||
|
Identify customers who have moved between segments over time (requires historical CLV data)
|
||||||
|
|
||||||
|
### Challenge 2.3: Engagement Correlation
|
||||||
|
Analyze the correlation between engagement scores and satisfaction scores
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
Once you're comfortable with customer analytics, move on to:
|
||||||
|
- **Level 3:** Sales & Revenue Analysis
|
||||||
|
- **Level 4:** KPI Dashboards & Metrics
|
||||||
|
|
||||||
@@ -0,0 +1,380 @@
|
|||||||
|
# Level 3: Sales & Revenue Analysis
|
||||||
|
|
||||||
|
## Introduction
|
||||||
|
Master sales performance analysis, product analytics, and revenue forecasting - core responsibilities for Data Analysts in sales-driven organizations.
|
||||||
|
|
||||||
|
## Learning Objectives
|
||||||
|
- Analyze sales performance vs targets
|
||||||
|
- Identify top-performing products
|
||||||
|
- Compare sales across channels and regions
|
||||||
|
- Calculate revenue metrics
|
||||||
|
- Analyze sales trends
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 3.1: Daily Sales Summary
|
||||||
|
**Objective:** Get today's sales summary
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
COUNT(*) as total_sales,
|
||||||
|
SUM(total_amount) as total_revenue,
|
||||||
|
AVG(total_amount) as avg_sale_value,
|
||||||
|
SUM(quantity) as total_units_sold,
|
||||||
|
COUNT(DISTINCT product_id) as products_sold
|
||||||
|
FROM sales_transactions
|
||||||
|
WHERE DATE(sale_date) = CURRENT_DATE;
|
||||||
|
```
|
||||||
|
|
||||||
|
**Expected Result:** Summary of today's sales activity
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 3.2: Top Products by Revenue
|
||||||
|
**Objective:** Find the top 10 products by total revenue
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
p.product_id,
|
||||||
|
p.product_name,
|
||||||
|
p.product_category,
|
||||||
|
p.product_subcategory,
|
||||||
|
COUNT(st.transaction_id) as sales_count,
|
||||||
|
SUM(st.quantity) as units_sold,
|
||||||
|
SUM(st.total_amount) as total_revenue,
|
||||||
|
AVG(st.total_amount) as avg_sale_value
|
||||||
|
FROM product_catalog p
|
||||||
|
JOIN sales_transactions st ON p.product_id = st.product_id
|
||||||
|
GROUP BY p.product_id, p.product_name, p.product_category, p.product_subcategory
|
||||||
|
ORDER BY total_revenue DESC
|
||||||
|
LIMIT 10;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 3.3: Sales by Channel
|
||||||
|
**Objective:** Compare sales performance across different channels
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
sales_channel,
|
||||||
|
COUNT(*) as transaction_count,
|
||||||
|
SUM(total_amount) as total_revenue,
|
||||||
|
AVG(total_amount) as avg_transaction_value,
|
||||||
|
SUM(quantity) as total_units,
|
||||||
|
ROUND(100.0 * COUNT(*) / SUM(COUNT(*)) OVER (), 2) as pct_of_total_transactions
|
||||||
|
FROM sales_transactions
|
||||||
|
GROUP BY sales_channel
|
||||||
|
ORDER BY total_revenue DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 3.4: Regional Performance
|
||||||
|
**Objective:** Analyze sales by region
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
region,
|
||||||
|
COUNT(*) as sales_count,
|
||||||
|
SUM(total_amount) as total_revenue,
|
||||||
|
AVG(total_amount) as avg_sale,
|
||||||
|
SUM(discount_amount) as total_discounts,
|
||||||
|
ROUND(100.0 * SUM(discount_amount) / SUM(total_amount), 2) as discount_rate
|
||||||
|
FROM sales_transactions
|
||||||
|
GROUP BY region
|
||||||
|
ORDER BY total_revenue DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 3.5: Product Category Analysis
|
||||||
|
**Objective:** Compare performance across product categories
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
p.product_category,
|
||||||
|
COUNT(DISTINCT p.product_id) as product_count,
|
||||||
|
COUNT(st.transaction_id) as sales_count,
|
||||||
|
SUM(st.total_amount) as total_revenue,
|
||||||
|
AVG(st.total_amount) as avg_sale_value,
|
||||||
|
SUM(st.quantity) as units_sold,
|
||||||
|
AVG(p.margin_percentage) as avg_margin
|
||||||
|
FROM product_catalog p
|
||||||
|
LEFT JOIN sales_transactions st ON p.product_id = st.product_id
|
||||||
|
GROUP BY p.product_category
|
||||||
|
ORDER BY total_revenue DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 3.6: Monthly Sales Trend
|
||||||
|
**Objective:** Show sales trends over the past 12 months
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
DATE_TRUNC('month', sale_date) as month,
|
||||||
|
COUNT(*) as transaction_count,
|
||||||
|
SUM(total_amount) as total_revenue,
|
||||||
|
AVG(total_amount) as avg_transaction_value,
|
||||||
|
SUM(quantity) as units_sold
|
||||||
|
FROM sales_transactions
|
||||||
|
WHERE sale_date >= CURRENT_DATE - INTERVAL '12 months'
|
||||||
|
GROUP BY DATE_TRUNC('month', sale_date)
|
||||||
|
ORDER BY month DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 3.7: Sales Performance vs Target
|
||||||
|
**Objective:** Compare actual sales to targets
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
st.target_period,
|
||||||
|
st.target_category,
|
||||||
|
st.target_value,
|
||||||
|
COALESCE(sp.actual_value, 0) as actual_value,
|
||||||
|
COALESCE(sp.actual_value, 0) - st.target_value as variance,
|
||||||
|
ROUND(100.0 * COALESCE(sp.actual_value, 0) / st.target_value, 2) as achievement_pct,
|
||||||
|
CASE
|
||||||
|
WHEN COALESCE(sp.actual_value, 0) >= st.target_value THEN 'MET'
|
||||||
|
WHEN COALESCE(sp.actual_value, 0) >= st.target_value * 0.9 THEN 'NEAR'
|
||||||
|
ELSE 'MISSED'
|
||||||
|
END as status
|
||||||
|
FROM sales_targets st
|
||||||
|
LEFT JOIN sales_performance sp ON st.target_id = sp.target_id
|
||||||
|
WHERE st.target_period >= CURRENT_DATE - INTERVAL '6 months'
|
||||||
|
ORDER BY st.target_period DESC, st.target_category;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 3.8: Product Profitability
|
||||||
|
**Objective:** Calculate profit margins by product
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
p.product_id,
|
||||||
|
p.product_name,
|
||||||
|
p.product_category,
|
||||||
|
p.unit_price,
|
||||||
|
p.cost_price,
|
||||||
|
p.margin_percentage,
|
||||||
|
COUNT(st.transaction_id) as sales_count,
|
||||||
|
SUM(st.quantity) as units_sold,
|
||||||
|
SUM(st.total_amount) as total_revenue,
|
||||||
|
SUM(st.quantity * p.cost_price) as total_cost,
|
||||||
|
SUM(st.total_amount) - SUM(st.quantity * p.cost_price) as gross_profit
|
||||||
|
FROM product_catalog p
|
||||||
|
JOIN sales_transactions st ON p.product_id = st.product_id
|
||||||
|
GROUP BY p.product_id, p.product_name, p.product_category, p.unit_price, p.cost_price, p.margin_percentage
|
||||||
|
HAVING SUM(st.total_amount) > 0
|
||||||
|
ORDER BY gross_profit DESC
|
||||||
|
LIMIT 20;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 3.9: Revenue Forecast Accuracy
|
||||||
|
**Objective:** Compare forecasted revenue to actual revenue
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
rf.forecast_period,
|
||||||
|
rf.forecast_category,
|
||||||
|
rf.forecasted_revenue,
|
||||||
|
rf.confidence_level,
|
||||||
|
ms.total_revenue as actual_revenue,
|
||||||
|
(ms.total_revenue - rf.forecasted_revenue) as variance,
|
||||||
|
ROUND(100.0 * ABS(ms.total_revenue - rf.forecasted_revenue) / rf.forecasted_revenue, 2) as error_pct
|
||||||
|
FROM revenue_forecasts rf
|
||||||
|
JOIN monthly_summaries ms ON
|
||||||
|
EXTRACT(MONTH FROM rf.forecast_period) = ms.summary_month AND
|
||||||
|
EXTRACT(YEAR FROM rf.forecast_period) = ms.summary_year
|
||||||
|
WHERE rf.forecast_period >= CURRENT_DATE - INTERVAL '12 months'
|
||||||
|
ORDER BY rf.forecast_period DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 3.10: Sales Velocity Analysis
|
||||||
|
**Objective:** Identify fast-moving vs slow-moving products
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
WITH product_sales AS (
|
||||||
|
SELECT
|
||||||
|
p.product_id,
|
||||||
|
p.product_name,
|
||||||
|
p.product_category,
|
||||||
|
COUNT(st.transaction_id) as sales_count,
|
||||||
|
SUM(st.quantity) as units_sold,
|
||||||
|
MIN(st.sale_date) as first_sale,
|
||||||
|
MAX(st.sale_date) as last_sale,
|
||||||
|
EXTRACT(DAY FROM MAX(st.sale_date) - MIN(st.sale_date)) as days_on_market
|
||||||
|
FROM product_catalog p
|
||||||
|
LEFT JOIN sales_transactions st ON p.product_id = st.product_id
|
||||||
|
WHERE p.is_active = TRUE
|
||||||
|
GROUP BY p.product_id, p.product_name, p.product_category
|
||||||
|
)
|
||||||
|
SELECT
|
||||||
|
product_id,
|
||||||
|
product_name,
|
||||||
|
product_category,
|
||||||
|
sales_count,
|
||||||
|
units_sold,
|
||||||
|
days_on_market,
|
||||||
|
CASE
|
||||||
|
WHEN days_on_market > 0 THEN ROUND(units_sold::DECIMAL / days_on_market, 2)
|
||||||
|
ELSE 0
|
||||||
|
END as units_per_day,
|
||||||
|
CASE
|
||||||
|
WHEN days_on_market > 0 AND (units_sold::DECIMAL / days_on_market) > 10 THEN 'FAST'
|
||||||
|
WHEN days_on_market > 0 AND (units_sold::DECIMAL / days_on_market) > 5 THEN 'MEDIUM'
|
||||||
|
WHEN days_on_market > 0 THEN 'SLOW'
|
||||||
|
ELSE 'NO_SALES'
|
||||||
|
END as velocity_category
|
||||||
|
FROM product_sales
|
||||||
|
ORDER BY units_per_day DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 3.11: Discount Impact Analysis
|
||||||
|
**Objective:** Analyze the impact of discounts on sales
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
CASE
|
||||||
|
WHEN discount_amount = 0 THEN 'No Discount'
|
||||||
|
WHEN discount_amount / total_amount < 0.1 THEN '< 10%'
|
||||||
|
WHEN discount_amount / total_amount < 0.2 THEN '10-20%'
|
||||||
|
WHEN discount_amount / total_amount < 0.3 THEN '20-30%'
|
||||||
|
ELSE '> 30%'
|
||||||
|
END as discount_tier,
|
||||||
|
COUNT(*) as transaction_count,
|
||||||
|
AVG(total_amount) as avg_sale_value,
|
||||||
|
SUM(total_amount) as total_revenue,
|
||||||
|
SUM(discount_amount) as total_discounts,
|
||||||
|
AVG(quantity) as avg_quantity
|
||||||
|
FROM sales_transactions
|
||||||
|
GROUP BY discount_tier
|
||||||
|
ORDER BY
|
||||||
|
CASE discount_tier
|
||||||
|
WHEN 'No Discount' THEN 1
|
||||||
|
WHEN '< 10%' THEN 2
|
||||||
|
WHEN '10-20%' THEN 3
|
||||||
|
WHEN '20-30%' THEN 4
|
||||||
|
ELSE 5
|
||||||
|
END;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 3.12: Cross-Sell Opportunities
|
||||||
|
**Objective:** Find products frequently purchased together
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Hint:** Use self-join on transaction_id
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
p1.product_name as product_1,
|
||||||
|
p2.product_name as product_2,
|
||||||
|
COUNT(*) as times_purchased_together
|
||||||
|
FROM sales_transactions st1
|
||||||
|
JOIN sales_transactions st2 ON st1.transaction_id = st2.transaction_id
|
||||||
|
AND st1.product_id < st2.product_id
|
||||||
|
JOIN product_catalog p1 ON st1.product_id = p1.product_id
|
||||||
|
JOIN product_catalog p2 ON st2.product_id = p2.product_id
|
||||||
|
GROUP BY p1.product_id, p1.product_name, p2.product_id, p2.product_name
|
||||||
|
HAVING COUNT(*) > 10
|
||||||
|
ORDER BY times_purchased_together DESC
|
||||||
|
LIMIT 20;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Challenge Exercises
|
||||||
|
|
||||||
|
### Challenge 3.1: Sales Seasonality
|
||||||
|
Identify seasonal patterns in sales data by analyzing month-over-month and year-over-year trends
|
||||||
|
|
||||||
|
### Challenge 3.2: Customer Purchase Patterns
|
||||||
|
Analyze average time between purchases and identify customers with regular buying patterns
|
||||||
|
|
||||||
|
### Challenge 3.3: Product Launch Performance
|
||||||
|
Compare new product performance (launched in last 6 months) vs established products
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
Once you're comfortable with sales analysis, move on to:
|
||||||
|
- **Level 4:** KPI Dashboards & Metrics
|
||||||
|
- **Level 6:** Fraud Detection (Advanced)
|
||||||
|
|
||||||
@@ -0,0 +1,431 @@
|
|||||||
|
# Level 4: KPI Dashboards & Metrics
|
||||||
|
|
||||||
|
## Introduction
|
||||||
|
Learn to build and maintain KPI dashboards - a critical skill for Data Analysts supporting business decision-making.
|
||||||
|
|
||||||
|
## Learning Objectives
|
||||||
|
- Track key performance indicators
|
||||||
|
- Calculate trend metrics
|
||||||
|
- Build dashboard queries
|
||||||
|
- Monitor data quality
|
||||||
|
- Create executive reports
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 4.1: Current KPI Status
|
||||||
|
**Objective:** Get current status of all active KPIs
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
kd.kpi_name,
|
||||||
|
kd.kpi_category,
|
||||||
|
kd.target_value,
|
||||||
|
dm.metric_value as current_value,
|
||||||
|
dm.status,
|
||||||
|
kd.unit_of_measure,
|
||||||
|
ROUND(100.0 * dm.metric_value / kd.target_value, 2) as achievement_pct
|
||||||
|
FROM kpi_definitions kd
|
||||||
|
LEFT JOIN daily_metrics dm ON kd.kpi_id = dm.kpi_id
|
||||||
|
WHERE kd.is_active = TRUE
|
||||||
|
AND dm.metric_date = CURRENT_DATE
|
||||||
|
ORDER BY kd.kpi_category, kd.kpi_name;
|
||||||
|
```
|
||||||
|
|
||||||
|
**Expected Result:** Dashboard view of all KPIs with current status
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 4.2: KPIs Below Target
|
||||||
|
**Objective:** Identify KPIs that are underperforming
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
kd.kpi_name,
|
||||||
|
kd.kpi_category,
|
||||||
|
kd.target_value,
|
||||||
|
dm.metric_value as current_value,
|
||||||
|
kd.threshold_warning,
|
||||||
|
kd.threshold_critical,
|
||||||
|
dm.status,
|
||||||
|
(dm.metric_value - kd.target_value) as variance,
|
||||||
|
ROUND(100.0 * (dm.metric_value - kd.target_value) / kd.target_value, 2) as variance_pct
|
||||||
|
FROM kpi_definitions kd
|
||||||
|
JOIN daily_metrics dm ON kd.kpi_id = dm.kpi_id
|
||||||
|
WHERE kd.is_active = TRUE
|
||||||
|
AND dm.metric_date = CURRENT_DATE
|
||||||
|
AND dm.metric_value < kd.target_value
|
||||||
|
ORDER BY variance_pct ASC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 4.3: KPI Trend Analysis (7 Days)
|
||||||
|
**Objective:** Show 7-day trend for critical KPIs
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
kd.kpi_name,
|
||||||
|
dm.metric_date,
|
||||||
|
dm.metric_value,
|
||||||
|
kd.target_value,
|
||||||
|
dm.vs_previous_day_percentage,
|
||||||
|
dm.status
|
||||||
|
FROM kpi_definitions kd
|
||||||
|
JOIN daily_metrics dm ON kd.kpi_id = dm.kpi_id
|
||||||
|
WHERE kd.is_active = TRUE
|
||||||
|
AND dm.metric_date >= CURRENT_DATE - INTERVAL '7 days'
|
||||||
|
AND kd.kpi_category IN ('SALES', 'CUSTOMER')
|
||||||
|
ORDER BY kd.kpi_name, dm.metric_date DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 4.4: Monthly Performance Summary
|
||||||
|
**Objective:** Aggregate monthly business metrics
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
summary_year,
|
||||||
|
summary_month,
|
||||||
|
TO_CHAR(TO_DATE(summary_month::TEXT, 'MM'), 'Month') as month_name,
|
||||||
|
total_revenue,
|
||||||
|
total_transactions,
|
||||||
|
total_customers,
|
||||||
|
new_customers,
|
||||||
|
average_transaction_value,
|
||||||
|
ROUND(100.0 * new_customers / total_customers, 2) as new_customer_pct
|
||||||
|
FROM monthly_summaries
|
||||||
|
WHERE summary_year >= EXTRACT(YEAR FROM CURRENT_DATE) - 1
|
||||||
|
ORDER BY summary_year DESC, summary_month DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 4.5: Year-over-Year Comparison
|
||||||
|
**Objective:** Compare this year's performance to last year
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
WITH current_year AS (
|
||||||
|
SELECT
|
||||||
|
summary_month,
|
||||||
|
total_revenue,
|
||||||
|
total_transactions,
|
||||||
|
new_customers
|
||||||
|
FROM monthly_summaries
|
||||||
|
WHERE summary_year = EXTRACT(YEAR FROM CURRENT_DATE)
|
||||||
|
),
|
||||||
|
previous_year AS (
|
||||||
|
SELECT
|
||||||
|
summary_month,
|
||||||
|
total_revenue,
|
||||||
|
total_transactions,
|
||||||
|
new_customers
|
||||||
|
FROM monthly_summaries
|
||||||
|
WHERE summary_year = EXTRACT(YEAR FROM CURRENT_DATE) - 1
|
||||||
|
)
|
||||||
|
SELECT
|
||||||
|
cy.summary_month,
|
||||||
|
TO_CHAR(TO_DATE(cy.summary_month::TEXT, 'MM'), 'Month') as month_name,
|
||||||
|
cy.total_revenue as current_year_revenue,
|
||||||
|
py.total_revenue as previous_year_revenue,
|
||||||
|
(cy.total_revenue - py.total_revenue) as revenue_change,
|
||||||
|
ROUND(100.0 * (cy.total_revenue - py.total_revenue) / py.total_revenue, 2) as revenue_growth_pct,
|
||||||
|
cy.total_transactions as current_year_transactions,
|
||||||
|
py.total_transactions as previous_year_transactions
|
||||||
|
FROM current_year cy
|
||||||
|
LEFT JOIN previous_year py ON cy.summary_month = py.summary_month
|
||||||
|
ORDER BY cy.summary_month;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 4.6: Dashboard Snapshot
|
||||||
|
**Objective:** Create a pre-calculated dashboard snapshot
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
INSERT INTO dashboard_snapshots (
|
||||||
|
snapshot_date, dashboard_name, metric_name, metric_value, metric_category
|
||||||
|
)
|
||||||
|
SELECT
|
||||||
|
CURRENT_DATE as snapshot_date,
|
||||||
|
'Executive Dashboard' as dashboard_name,
|
||||||
|
kd.kpi_name as metric_name,
|
||||||
|
dm.metric_value,
|
||||||
|
kd.kpi_category as metric_category
|
||||||
|
FROM kpi_definitions kd
|
||||||
|
JOIN daily_metrics dm ON kd.kpi_id = dm.kpi_id
|
||||||
|
WHERE kd.is_active = TRUE
|
||||||
|
AND dm.metric_date = CURRENT_DATE;
|
||||||
|
|
||||||
|
-- View the snapshot
|
||||||
|
SELECT * FROM dashboard_snapshots
|
||||||
|
WHERE snapshot_date = CURRENT_DATE
|
||||||
|
ORDER BY metric_category, metric_name;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 4.7: Trend Detection
|
||||||
|
**Objective:** Identify metrics with consistent upward or downward trends
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
WITH trend_data AS (
|
||||||
|
SELECT
|
||||||
|
kd.kpi_name,
|
||||||
|
kd.kpi_category,
|
||||||
|
dm.metric_date,
|
||||||
|
dm.metric_value,
|
||||||
|
dm.vs_previous_day_percentage,
|
||||||
|
ROW_NUMBER() OVER (PARTITION BY kd.kpi_id ORDER BY dm.metric_date DESC) as day_rank
|
||||||
|
FROM kpi_definitions kd
|
||||||
|
JOIN daily_metrics dm ON kd.kpi_id = dm.kpi_id
|
||||||
|
WHERE dm.metric_date >= CURRENT_DATE - INTERVAL '7 days'
|
||||||
|
)
|
||||||
|
SELECT
|
||||||
|
kpi_name,
|
||||||
|
kpi_category,
|
||||||
|
COUNT(*) as days_tracked,
|
||||||
|
AVG(vs_previous_day_percentage) as avg_daily_change,
|
||||||
|
MIN(metric_value) as min_value,
|
||||||
|
MAX(metric_value) as max_value,
|
||||||
|
CASE
|
||||||
|
WHEN AVG(vs_previous_day_percentage) > 5 THEN 'STRONG UPWARD'
|
||||||
|
WHEN AVG(vs_previous_day_percentage) > 0 THEN 'UPWARD'
|
||||||
|
WHEN AVG(vs_previous_day_percentage) > -5 THEN 'DOWNWARD'
|
||||||
|
ELSE 'STRONG DOWNWARD'
|
||||||
|
END as trend_direction
|
||||||
|
FROM trend_data
|
||||||
|
GROUP BY kpi_name, kpi_category
|
||||||
|
ORDER BY avg_daily_change DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 4.8: Data Quality Monitoring
|
||||||
|
**Objective:** Track data quality metrics
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
check_date,
|
||||||
|
table_name,
|
||||||
|
check_type,
|
||||||
|
records_checked,
|
||||||
|
records_failed,
|
||||||
|
ROUND(100.0 * records_failed / records_checked, 2) as failure_rate,
|
||||||
|
status,
|
||||||
|
error_details
|
||||||
|
FROM data_quality_checks
|
||||||
|
WHERE check_date >= CURRENT_DATE - INTERVAL '7 days'
|
||||||
|
AND status IN ('WARNING', 'FAILED')
|
||||||
|
ORDER BY check_date DESC, failure_rate DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 4.9: Report Execution History
|
||||||
|
**Objective:** Monitor report generation and performance
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
rd.report_name,
|
||||||
|
rd.report_category,
|
||||||
|
re.execution_date,
|
||||||
|
re.execution_status,
|
||||||
|
re.execution_time_seconds,
|
||||||
|
re.records_generated,
|
||||||
|
re.file_size_kb
|
||||||
|
FROM report_definitions rd
|
||||||
|
JOIN report_executions re ON rd.report_id = re.report_id
|
||||||
|
WHERE re.execution_date >= CURRENT_DATE - INTERVAL '30 days'
|
||||||
|
ORDER BY re.execution_date DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 4.10: Executive Summary Dashboard
|
||||||
|
**Objective:** Create a comprehensive executive summary
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
WITH today_metrics AS (
|
||||||
|
SELECT
|
||||||
|
SUM(CASE WHEN kd.kpi_category = 'SALES' THEN dm.metric_value ELSE 0 END) as total_sales,
|
||||||
|
AVG(CASE WHEN kd.kpi_name = 'Customer Engagement Score' THEN dm.metric_value END) as avg_engagement,
|
||||||
|
AVG(CASE WHEN kd.kpi_name = 'Net Promoter Score' THEN dm.metric_value END) as avg_nps,
|
||||||
|
COUNT(CASE WHEN dm.status = 'CRITICAL' THEN 1 END) as critical_kpis,
|
||||||
|
COUNT(CASE WHEN dm.status = 'WARNING' THEN 1 END) as warning_kpis,
|
||||||
|
COUNT(CASE WHEN dm.status = 'ON_TARGET' THEN 1 END) as on_target_kpis
|
||||||
|
FROM kpi_definitions kd
|
||||||
|
JOIN daily_metrics dm ON kd.kpi_id = dm.kpi_id
|
||||||
|
WHERE dm.metric_date = CURRENT_DATE
|
||||||
|
),
|
||||||
|
monthly_summary AS (
|
||||||
|
SELECT
|
||||||
|
total_revenue,
|
||||||
|
total_transactions,
|
||||||
|
new_customers,
|
||||||
|
average_transaction_value
|
||||||
|
FROM monthly_summaries
|
||||||
|
WHERE summary_year = EXTRACT(YEAR FROM CURRENT_DATE)
|
||||||
|
AND summary_month = EXTRACT(MONTH FROM CURRENT_DATE)
|
||||||
|
)
|
||||||
|
SELECT
|
||||||
|
'Executive Summary' as report_title,
|
||||||
|
CURRENT_DATE as report_date,
|
||||||
|
tm.total_sales,
|
||||||
|
tm.avg_engagement,
|
||||||
|
tm.avg_nps,
|
||||||
|
tm.critical_kpis,
|
||||||
|
tm.warning_kpis,
|
||||||
|
tm.on_target_kpis,
|
||||||
|
ms.total_revenue as mtd_revenue,
|
||||||
|
ms.total_transactions as mtd_transactions,
|
||||||
|
ms.new_customers as mtd_new_customers
|
||||||
|
FROM today_metrics tm
|
||||||
|
CROSS JOIN monthly_summary ms;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 4.11: Moving Average Calculation
|
||||||
|
**Objective:** Calculate 7-day moving average for key metrics
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
kd.kpi_name,
|
||||||
|
dm.metric_date,
|
||||||
|
dm.metric_value,
|
||||||
|
AVG(dm.metric_value) OVER (
|
||||||
|
PARTITION BY kd.kpi_id
|
||||||
|
ORDER BY dm.metric_date
|
||||||
|
ROWS BETWEEN 6 PRECEDING AND CURRENT ROW
|
||||||
|
) as moving_avg_7day,
|
||||||
|
kd.target_value
|
||||||
|
FROM kpi_definitions kd
|
||||||
|
JOIN daily_metrics dm ON kd.kpi_id = dm.kpi_id
|
||||||
|
WHERE kd.kpi_name IN ('Daily Revenue', 'Transactions Per Day', 'Customer Engagement Score')
|
||||||
|
AND dm.metric_date >= CURRENT_DATE - INTERVAL '30 days'
|
||||||
|
ORDER BY kd.kpi_name, dm.metric_date DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Exercise 4.12: Anomaly Detection
|
||||||
|
**Objective:** Identify unusual metric values
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
WITH metric_stats AS (
|
||||||
|
SELECT
|
||||||
|
kpi_id,
|
||||||
|
AVG(metric_value) as avg_value,
|
||||||
|
STDDEV(metric_value) as stddev_value
|
||||||
|
FROM daily_metrics
|
||||||
|
WHERE metric_date >= CURRENT_DATE - INTERVAL '30 days'
|
||||||
|
GROUP BY kpi_id
|
||||||
|
)
|
||||||
|
SELECT
|
||||||
|
kd.kpi_name,
|
||||||
|
dm.metric_date,
|
||||||
|
dm.metric_value,
|
||||||
|
ms.avg_value,
|
||||||
|
ms.stddev_value,
|
||||||
|
(dm.metric_value - ms.avg_value) / NULLIF(ms.stddev_value, 0) as z_score,
|
||||||
|
CASE
|
||||||
|
WHEN ABS((dm.metric_value - ms.avg_value) / NULLIF(ms.stddev_value, 0)) > 3 THEN 'EXTREME ANOMALY'
|
||||||
|
WHEN ABS((dm.metric_value - ms.avg_value) / NULLIF(ms.stddev_value, 0)) > 2 THEN 'ANOMALY'
|
||||||
|
ELSE 'NORMAL'
|
||||||
|
END as anomaly_status
|
||||||
|
FROM kpi_definitions kd
|
||||||
|
JOIN daily_metrics dm ON kd.kpi_id = dm.kpi_id
|
||||||
|
JOIN metric_stats ms ON kd.kpi_id = ms.kpi_id
|
||||||
|
WHERE dm.metric_date >= CURRENT_DATE - INTERVAL '7 days'
|
||||||
|
AND ABS((dm.metric_value - ms.avg_value) / NULLIF(ms.stddev_value, 0)) > 2
|
||||||
|
ORDER BY ABS((dm.metric_value - ms.avg_value) / NULLIF(ms.stddev_value, 0)) DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Challenge Exercises
|
||||||
|
|
||||||
|
### Challenge 4.1: Custom Dashboard Builder
|
||||||
|
Create a flexible query that can generate different dashboard views based on parameters (date range, KPI category, etc.)
|
||||||
|
|
||||||
|
### Challenge 4.2: Predictive Alerting
|
||||||
|
Build a query that predicts which KPIs are likely to miss targets based on current trends
|
||||||
|
|
||||||
|
### Challenge 4.3: Correlation Analysis
|
||||||
|
Identify correlations between different KPIs (e.g., does customer engagement correlate with revenue?)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
You've now mastered the core Data Analyst skills! Continue with:
|
||||||
|
- **Level 5:** Advanced Analytics (Window Functions, CTEs)
|
||||||
|
- **Level 6:** Fraud Detection (Advanced Pattern Recognition)
|
||||||
|
- Build your own custom dashboards and reports!
|
||||||
|
|
||||||
+341
-2
@@ -1,6 +1,6 @@
|
|||||||
-- ============================================================================
|
-- ============================================================================
|
||||||
-- Financial Fraud Detection Database Schema
|
-- Business Analytics Database Schema
|
||||||
-- Purpose: Educational SQL learning with realistic fraud investigation scenarios
|
-- Purpose: Data Analyst training with routine/semi-routine analysis scenarios
|
||||||
-- ============================================================================
|
-- ============================================================================
|
||||||
-- This script is IDEMPOTENT - it will drop and recreate all objects
|
-- This script is IDEMPOTENT - it will drop and recreate all objects
|
||||||
-- ============================================================================
|
-- ============================================================================
|
||||||
@@ -23,6 +23,27 @@ DROP FUNCTION IF EXISTS check_suspicious_transaction() CASCADE;
|
|||||||
DROP FUNCTION IF EXISTS generate_fraud_score(DECIMAL, BOOLEAN, DECIMAL, INT, BOOLEAN) CASCADE;
|
DROP FUNCTION IF EXISTS generate_fraud_score(DECIMAL, BOOLEAN, DECIMAL, INT, BOOLEAN) CASCADE;
|
||||||
|
|
||||||
-- Drop tables in reverse dependency order
|
-- Drop tables in reverse dependency order
|
||||||
|
-- New analytics tables
|
||||||
|
DROP TABLE IF EXISTS report_executions CASCADE;
|
||||||
|
DROP TABLE IF EXISTS report_definitions CASCADE;
|
||||||
|
DROP TABLE IF EXISTS data_quality_checks CASCADE;
|
||||||
|
DROP TABLE IF EXISTS dashboard_snapshots CASCADE;
|
||||||
|
DROP TABLE IF EXISTS trend_analysis CASCADE;
|
||||||
|
DROP TABLE IF EXISTS monthly_summaries CASCADE;
|
||||||
|
DROP TABLE IF EXISTS daily_metrics CASCADE;
|
||||||
|
DROP TABLE IF EXISTS kpi_definitions CASCADE;
|
||||||
|
DROP TABLE IF EXISTS revenue_forecasts CASCADE;
|
||||||
|
DROP TABLE IF EXISTS sales_performance CASCADE;
|
||||||
|
DROP TABLE IF EXISTS sales_targets CASCADE;
|
||||||
|
DROP TABLE IF EXISTS sales_transactions CASCADE;
|
||||||
|
DROP TABLE IF EXISTS product_catalog CASCADE;
|
||||||
|
DROP TABLE IF EXISTS engagement_metrics CASCADE;
|
||||||
|
DROP TABLE IF EXISTS customer_satisfaction CASCADE;
|
||||||
|
DROP TABLE IF EXISTS churn_predictions CASCADE;
|
||||||
|
DROP TABLE IF EXISTS customer_lifetime_value CASCADE;
|
||||||
|
DROP TABLE IF EXISTS customer_segments CASCADE;
|
||||||
|
|
||||||
|
-- Existing tables
|
||||||
DROP TABLE IF EXISTS audit_log CASCADE;
|
DROP TABLE IF EXISTS audit_log CASCADE;
|
||||||
DROP TABLE IF EXISTS suspicious_activity_reports CASCADE;
|
DROP TABLE IF EXISTS suspicious_activity_reports CASCADE;
|
||||||
DROP TABLE IF EXISTS case_alerts CASCADE;
|
DROP TABLE IF EXISTS case_alerts CASCADE;
|
||||||
@@ -475,3 +496,321 @@ COMMENT ON TABLE cards IS 'Payment cards linked to accounts';
|
|||||||
COMMENT ON TABLE devices IS 'Device fingerprints for fraud detection';
|
COMMENT ON TABLE devices IS 'Device fingerprints for fraud detection';
|
||||||
COMMENT ON TABLE login_sessions IS 'Login history for account takeover detection';
|
COMMENT ON TABLE login_sessions IS 'Login history for account takeover detection';
|
||||||
|
|
||||||
|
-- ============================================================================
|
||||||
|
-- CUSTOMER ANALYTICS TABLES (for routine customer analysis)
|
||||||
|
-- ============================================================================
|
||||||
|
|
||||||
|
CREATE TABLE customer_segments (
|
||||||
|
segment_id SERIAL PRIMARY KEY,
|
||||||
|
segment_name VARCHAR(100) NOT NULL UNIQUE,
|
||||||
|
segment_description TEXT,
|
||||||
|
criteria_definition JSONB,
|
||||||
|
min_clv DECIMAL(15,2),
|
||||||
|
max_clv DECIMAL(15,2),
|
||||||
|
created_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
updated_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE customer_lifetime_value (
|
||||||
|
clv_id SERIAL PRIMARY KEY,
|
||||||
|
customer_id INT NOT NULL REFERENCES customers(customer_id),
|
||||||
|
calculation_date DATE NOT NULL,
|
||||||
|
total_revenue DECIMAL(15,2) DEFAULT 0,
|
||||||
|
total_transactions INT DEFAULT 0,
|
||||||
|
average_order_value DECIMAL(15,2) DEFAULT 0,
|
||||||
|
predicted_future_value DECIMAL(15,2),
|
||||||
|
clv_score DECIMAL(10,2),
|
||||||
|
segment_id INT REFERENCES customer_segments(segment_id),
|
||||||
|
UNIQUE(customer_id, calculation_date)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE churn_predictions (
|
||||||
|
prediction_id SERIAL PRIMARY KEY,
|
||||||
|
customer_id INT NOT NULL REFERENCES customers(customer_id),
|
||||||
|
prediction_date DATE NOT NULL,
|
||||||
|
churn_probability DECIMAL(5,2) CHECK (churn_probability BETWEEN 0 AND 100),
|
||||||
|
risk_level VARCHAR(20) CHECK (risk_level IN ('LOW', 'MEDIUM', 'HIGH', 'CRITICAL')),
|
||||||
|
last_transaction_date DATE,
|
||||||
|
days_since_last_transaction INT,
|
||||||
|
engagement_score DECIMAL(5,2),
|
||||||
|
recommended_action TEXT,
|
||||||
|
UNIQUE(customer_id, prediction_date)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE customer_satisfaction (
|
||||||
|
satisfaction_id SERIAL PRIMARY KEY,
|
||||||
|
customer_id INT NOT NULL REFERENCES customers(customer_id),
|
||||||
|
survey_date DATE NOT NULL,
|
||||||
|
nps_score INT CHECK (nps_score BETWEEN -100 AND 100),
|
||||||
|
csat_score DECIMAL(3,2) CHECK (csat_score BETWEEN 1 AND 5),
|
||||||
|
feedback_text TEXT,
|
||||||
|
category VARCHAR(50) CHECK (category IN ('PRODUCT', 'SERVICE', 'SUPPORT', 'BILLING', 'OTHER')),
|
||||||
|
sentiment VARCHAR(20) CHECK (sentiment IN ('POSITIVE', 'NEUTRAL', 'NEGATIVE'))
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE engagement_metrics (
|
||||||
|
metric_id SERIAL PRIMARY KEY,
|
||||||
|
customer_id INT NOT NULL REFERENCES customers(customer_id),
|
||||||
|
metric_date DATE NOT NULL,
|
||||||
|
login_count INT DEFAULT 0,
|
||||||
|
page_views INT DEFAULT 0,
|
||||||
|
time_spent_minutes INT DEFAULT 0,
|
||||||
|
features_used JSONB,
|
||||||
|
support_tickets_opened INT DEFAULT 0,
|
||||||
|
engagement_score DECIMAL(5,2),
|
||||||
|
UNIQUE(customer_id, metric_date)
|
||||||
|
);
|
||||||
|
|
||||||
|
-- ============================================================================
|
||||||
|
-- SALES & REVENUE ANALYTICS TABLES (for semi-routine sales reporting)
|
||||||
|
-- ============================================================================
|
||||||
|
|
||||||
|
CREATE TABLE product_catalog (
|
||||||
|
product_id SERIAL PRIMARY KEY,
|
||||||
|
product_name VARCHAR(200) NOT NULL,
|
||||||
|
product_category VARCHAR(100),
|
||||||
|
product_subcategory VARCHAR(100),
|
||||||
|
unit_price DECIMAL(15,2) NOT NULL,
|
||||||
|
cost_price DECIMAL(15,2),
|
||||||
|
margin_percentage DECIMAL(5,2),
|
||||||
|
is_active BOOLEAN DEFAULT TRUE,
|
||||||
|
launch_date DATE,
|
||||||
|
discontinued_date DATE
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE sales_transactions (
|
||||||
|
sale_id SERIAL PRIMARY KEY,
|
||||||
|
transaction_id INT REFERENCES transactions(transaction_id),
|
||||||
|
product_id INT REFERENCES product_catalog(product_id),
|
||||||
|
quantity INT NOT NULL DEFAULT 1,
|
||||||
|
unit_price DECIMAL(15,2) NOT NULL,
|
||||||
|
discount_amount DECIMAL(15,2) DEFAULT 0,
|
||||||
|
tax_amount DECIMAL(15,2) DEFAULT 0,
|
||||||
|
total_amount DECIMAL(15,2) NOT NULL,
|
||||||
|
sale_date TIMESTAMP NOT NULL,
|
||||||
|
sales_channel VARCHAR(50) CHECK (sales_channel IN ('ONLINE', 'STORE', 'PHONE', 'MOBILE_APP')),
|
||||||
|
sales_rep_id INT,
|
||||||
|
region VARCHAR(100)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE sales_targets (
|
||||||
|
target_id SERIAL PRIMARY KEY,
|
||||||
|
target_period VARCHAR(20) CHECK (target_period IN ('DAILY', 'WEEKLY', 'MONTHLY', 'QUARTERLY', 'YEARLY')),
|
||||||
|
start_date DATE NOT NULL,
|
||||||
|
end_date DATE NOT NULL,
|
||||||
|
product_category VARCHAR(100),
|
||||||
|
region VARCHAR(100),
|
||||||
|
target_revenue DECIMAL(15,2),
|
||||||
|
target_units INT,
|
||||||
|
target_customers INT,
|
||||||
|
created_by VARCHAR(100),
|
||||||
|
created_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE sales_performance (
|
||||||
|
performance_id SERIAL PRIMARY KEY,
|
||||||
|
period_date DATE NOT NULL,
|
||||||
|
period_type VARCHAR(20) CHECK (period_type IN ('DAILY', 'WEEKLY', 'MONTHLY', 'QUARTERLY')),
|
||||||
|
product_id INT REFERENCES product_catalog(product_id),
|
||||||
|
region VARCHAR(100),
|
||||||
|
total_revenue DECIMAL(15,2) DEFAULT 0,
|
||||||
|
total_units_sold INT DEFAULT 0,
|
||||||
|
total_transactions INT DEFAULT 0,
|
||||||
|
unique_customers INT DEFAULT 0,
|
||||||
|
average_order_value DECIMAL(15,2),
|
||||||
|
vs_target_percentage DECIMAL(5,2),
|
||||||
|
UNIQUE(period_date, period_type, product_id, region)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE revenue_forecasts (
|
||||||
|
forecast_id SERIAL PRIMARY KEY,
|
||||||
|
forecast_date DATE NOT NULL,
|
||||||
|
forecast_period_start DATE NOT NULL,
|
||||||
|
forecast_period_end DATE NOT NULL,
|
||||||
|
product_category VARCHAR(100),
|
||||||
|
region VARCHAR(100),
|
||||||
|
forecasted_revenue DECIMAL(15,2),
|
||||||
|
confidence_level VARCHAR(20) CHECK (confidence_level IN ('LOW', 'MEDIUM', 'HIGH')),
|
||||||
|
forecast_method VARCHAR(50) CHECK (forecast_method IN ('HISTORICAL', 'TREND', 'SEASONAL', 'ML')),
|
||||||
|
actual_revenue DECIMAL(15,2),
|
||||||
|
variance_percentage DECIMAL(5,2)
|
||||||
|
);
|
||||||
|
|
||||||
|
|
||||||
|
-- ============================================================================
|
||||||
|
-- OPERATIONAL METRICS & KPI TABLES (for dashboard creation)
|
||||||
|
-- ============================================================================
|
||||||
|
|
||||||
|
CREATE TABLE kpi_definitions (
|
||||||
|
kpi_id SERIAL PRIMARY KEY,
|
||||||
|
kpi_name VARCHAR(100) NOT NULL UNIQUE,
|
||||||
|
kpi_description TEXT,
|
||||||
|
kpi_category VARCHAR(50) CHECK (kpi_category IN ('SALES', 'CUSTOMER', 'OPERATIONAL', 'FINANCIAL')),
|
||||||
|
calculation_formula TEXT,
|
||||||
|
target_value DECIMAL(15,2),
|
||||||
|
threshold_warning DECIMAL(15,2),
|
||||||
|
threshold_critical DECIMAL(15,2),
|
||||||
|
unit_of_measure VARCHAR(50),
|
||||||
|
refresh_frequency VARCHAR(20) CHECK (refresh_frequency IN ('REALTIME', 'HOURLY', 'DAILY', 'WEEKLY')),
|
||||||
|
is_active BOOLEAN DEFAULT TRUE
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE daily_metrics (
|
||||||
|
metric_id SERIAL PRIMARY KEY,
|
||||||
|
metric_date DATE NOT NULL,
|
||||||
|
kpi_id INT NOT NULL REFERENCES kpi_definitions(kpi_id),
|
||||||
|
metric_value DECIMAL(15,2),
|
||||||
|
vs_previous_day_percentage DECIMAL(5,2),
|
||||||
|
vs_previous_week_percentage DECIMAL(5,2),
|
||||||
|
vs_previous_month_percentage DECIMAL(5,2),
|
||||||
|
status VARCHAR(20) CHECK (status IN ('ON_TARGET', 'WARNING', 'CRITICAL')),
|
||||||
|
notes TEXT,
|
||||||
|
UNIQUE(metric_date, kpi_id)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE monthly_summaries (
|
||||||
|
summary_id SERIAL PRIMARY KEY,
|
||||||
|
summary_month INT CHECK (summary_month BETWEEN 1 AND 12),
|
||||||
|
summary_year INT,
|
||||||
|
total_revenue DECIMAL(15,2) DEFAULT 0,
|
||||||
|
total_transactions INT DEFAULT 0,
|
||||||
|
total_customers INT DEFAULT 0,
|
||||||
|
new_customers INT DEFAULT 0,
|
||||||
|
churned_customers INT DEFAULT 0,
|
||||||
|
average_transaction_value DECIMAL(15,2),
|
||||||
|
customer_acquisition_cost DECIMAL(15,2),
|
||||||
|
customer_lifetime_value DECIMAL(15,2),
|
||||||
|
net_promoter_score DECIMAL(5,2),
|
||||||
|
gross_margin_percentage DECIMAL(5,2),
|
||||||
|
UNIQUE(summary_month, summary_year)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE trend_analysis (
|
||||||
|
trend_id SERIAL PRIMARY KEY,
|
||||||
|
kpi_id INT NOT NULL REFERENCES kpi_definitions(kpi_id),
|
||||||
|
analysis_date DATE NOT NULL,
|
||||||
|
period_start DATE NOT NULL,
|
||||||
|
period_end DATE NOT NULL,
|
||||||
|
trend_direction VARCHAR(20) CHECK (trend_direction IN ('UP', 'DOWN', 'FLAT')),
|
||||||
|
trend_strength VARCHAR(20) CHECK (trend_strength IN ('WEAK', 'MODERATE', 'STRONG')),
|
||||||
|
moving_average_7day DECIMAL(15,2),
|
||||||
|
moving_average_30day DECIMAL(15,2),
|
||||||
|
seasonality_detected BOOLEAN DEFAULT FALSE,
|
||||||
|
anomalies_detected BOOLEAN DEFAULT FALSE,
|
||||||
|
statistical_significance DECIMAL(5,4)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE dashboard_snapshots (
|
||||||
|
snapshot_id SERIAL PRIMARY KEY,
|
||||||
|
dashboard_name VARCHAR(100) NOT NULL,
|
||||||
|
snapshot_timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
data_payload JSONB,
|
||||||
|
refresh_duration_seconds DECIMAL(10,2),
|
||||||
|
row_count INT,
|
||||||
|
last_updated_by VARCHAR(100)
|
||||||
|
);
|
||||||
|
|
||||||
|
-- ============================================================================
|
||||||
|
-- BUSINESS INTELLIGENCE & REPORTING TABLES
|
||||||
|
-- ============================================================================
|
||||||
|
|
||||||
|
CREATE TABLE report_definitions (
|
||||||
|
report_id SERIAL PRIMARY KEY,
|
||||||
|
report_name VARCHAR(200) NOT NULL UNIQUE,
|
||||||
|
report_description TEXT,
|
||||||
|
report_category VARCHAR(100),
|
||||||
|
sql_query_template TEXT,
|
||||||
|
parameters JSONB,
|
||||||
|
output_format VARCHAR(20) CHECK (output_format IN ('PDF', 'EXCEL', 'CSV', 'HTML')),
|
||||||
|
schedule_frequency VARCHAR(50),
|
||||||
|
recipients TEXT,
|
||||||
|
is_active BOOLEAN DEFAULT TRUE,
|
||||||
|
created_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE report_executions (
|
||||||
|
execution_id SERIAL PRIMARY KEY,
|
||||||
|
report_id INT NOT NULL REFERENCES report_definitions(report_id),
|
||||||
|
execution_timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
parameters_used JSONB,
|
||||||
|
row_count INT,
|
||||||
|
execution_duration_seconds DECIMAL(10,2),
|
||||||
|
status VARCHAR(20) CHECK (status IN ('SUCCESS', 'FAILED', 'TIMEOUT')),
|
||||||
|
error_message TEXT,
|
||||||
|
output_file_path TEXT,
|
||||||
|
executed_by VARCHAR(100)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE data_quality_checks (
|
||||||
|
check_id SERIAL PRIMARY KEY,
|
||||||
|
check_name VARCHAR(200) NOT NULL,
|
||||||
|
table_name VARCHAR(100) NOT NULL,
|
||||||
|
column_name VARCHAR(100),
|
||||||
|
check_type VARCHAR(50) CHECK (check_type IN ('NULL_CHECK', 'RANGE_CHECK', 'UNIQUENESS', 'REFERENTIAL_INTEGRITY', 'FORMAT_CHECK')),
|
||||||
|
check_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
records_checked INT,
|
||||||
|
records_failed INT,
|
||||||
|
failure_percentage DECIMAL(5,2),
|
||||||
|
status VARCHAR(20) CHECK (status IN ('PASS', 'FAIL', 'WARNING')),
|
||||||
|
remediation_notes TEXT
|
||||||
|
);
|
||||||
|
|
||||||
|
-- ============================================================================
|
||||||
|
-- INDEXES FOR NEW ANALYTICS TABLES
|
||||||
|
-- ============================================================================
|
||||||
|
|
||||||
|
-- Customer Analytics indexes
|
||||||
|
CREATE INDEX idx_clv_customer ON customer_lifetime_value(customer_id);
|
||||||
|
CREATE INDEX idx_clv_date ON customer_lifetime_value(calculation_date);
|
||||||
|
CREATE INDEX idx_clv_segment ON customer_lifetime_value(segment_id);
|
||||||
|
CREATE INDEX idx_churn_customer ON churn_predictions(customer_id);
|
||||||
|
CREATE INDEX idx_churn_risk ON churn_predictions(risk_level);
|
||||||
|
CREATE INDEX idx_satisfaction_customer ON customer_satisfaction(customer_id);
|
||||||
|
CREATE INDEX idx_engagement_customer ON engagement_metrics(customer_id);
|
||||||
|
CREATE INDEX idx_engagement_date ON engagement_metrics(metric_date);
|
||||||
|
|
||||||
|
-- Sales Analytics indexes
|
||||||
|
CREATE INDEX idx_sales_trans_product ON sales_transactions(product_id);
|
||||||
|
CREATE INDEX idx_sales_trans_date ON sales_transactions(sale_date);
|
||||||
|
CREATE INDEX idx_sales_trans_channel ON sales_transactions(sales_channel);
|
||||||
|
CREATE INDEX idx_sales_perf_date ON sales_performance(period_date);
|
||||||
|
CREATE INDEX idx_sales_perf_product ON sales_performance(product_id);
|
||||||
|
CREATE INDEX idx_product_category ON product_catalog(product_category);
|
||||||
|
CREATE INDEX idx_product_active ON product_catalog(is_active);
|
||||||
|
|
||||||
|
-- KPI & Metrics indexes
|
||||||
|
CREATE INDEX idx_daily_metrics_date ON daily_metrics(metric_date);
|
||||||
|
CREATE INDEX idx_daily_metrics_kpi ON daily_metrics(kpi_id);
|
||||||
|
CREATE INDEX idx_monthly_summaries_period ON monthly_summaries(summary_year, summary_month);
|
||||||
|
CREATE INDEX idx_trend_kpi ON trend_analysis(kpi_id);
|
||||||
|
CREATE INDEX idx_trend_date ON trend_analysis(analysis_date);
|
||||||
|
|
||||||
|
-- Reporting indexes
|
||||||
|
CREATE INDEX idx_report_exec_report ON report_executions(report_id);
|
||||||
|
CREATE INDEX idx_report_exec_timestamp ON report_executions(execution_timestamp);
|
||||||
|
CREATE INDEX idx_data_quality_table ON data_quality_checks(table_name);
|
||||||
|
CREATE INDEX idx_data_quality_date ON data_quality_checks(check_date);
|
||||||
|
|
||||||
|
-- ============================================================================
|
||||||
|
-- COMMENTS FOR NEW TABLES
|
||||||
|
-- ============================================================================
|
||||||
|
|
||||||
|
COMMENT ON TABLE customer_segments IS 'Customer segmentation definitions for targeted analysis';
|
||||||
|
COMMENT ON TABLE customer_lifetime_value IS 'Customer lifetime value calculations and tracking';
|
||||||
|
COMMENT ON TABLE churn_predictions IS 'Customer churn risk predictions for retention analysis';
|
||||||
|
COMMENT ON TABLE customer_satisfaction IS 'Customer satisfaction scores and feedback';
|
||||||
|
COMMENT ON TABLE engagement_metrics IS 'Customer engagement tracking metrics';
|
||||||
|
COMMENT ON TABLE product_catalog IS 'Product master data for sales analysis';
|
||||||
|
COMMENT ON TABLE sales_transactions IS 'Detailed sales transaction records';
|
||||||
|
COMMENT ON TABLE sales_targets IS 'Sales performance targets and goals';
|
||||||
|
COMMENT ON TABLE sales_performance IS 'Aggregated sales performance metrics';
|
||||||
|
COMMENT ON TABLE revenue_forecasts IS 'Revenue forecasting and predictions';
|
||||||
|
COMMENT ON TABLE kpi_definitions IS 'Master list of tracked KPIs';
|
||||||
|
COMMENT ON TABLE daily_metrics IS 'Daily operational metrics for dashboards';
|
||||||
|
COMMENT ON TABLE monthly_summaries IS 'Monthly aggregated business summaries';
|
||||||
|
COMMENT ON TABLE trend_analysis IS 'Statistical trend analysis results';
|
||||||
|
COMMENT ON TABLE dashboard_snapshots IS 'Pre-calculated dashboard data snapshots';
|
||||||
|
COMMENT ON TABLE report_definitions IS 'Standard report catalog';
|
||||||
|
COMMENT ON TABLE report_executions IS 'Report execution history and logs';
|
||||||
|
COMMENT ON TABLE data_quality_checks IS 'Data quality validation tracking';
|
||||||
|
|
||||||
|
|||||||
@@ -5,6 +5,27 @@
|
|||||||
-- ============================================================================
|
-- ============================================================================
|
||||||
|
|
||||||
-- Clear existing reference data (in reverse dependency order)
|
-- Clear existing reference data (in reverse dependency order)
|
||||||
|
-- New analytics tables
|
||||||
|
TRUNCATE TABLE report_executions CASCADE;
|
||||||
|
TRUNCATE TABLE report_definitions RESTART IDENTITY CASCADE;
|
||||||
|
TRUNCATE TABLE data_quality_checks RESTART IDENTITY CASCADE;
|
||||||
|
TRUNCATE TABLE dashboard_snapshots RESTART IDENTITY CASCADE;
|
||||||
|
TRUNCATE TABLE trend_analysis RESTART IDENTITY CASCADE;
|
||||||
|
TRUNCATE TABLE monthly_summaries RESTART IDENTITY CASCADE;
|
||||||
|
TRUNCATE TABLE daily_metrics RESTART IDENTITY CASCADE;
|
||||||
|
TRUNCATE TABLE kpi_definitions RESTART IDENTITY CASCADE;
|
||||||
|
TRUNCATE TABLE revenue_forecasts RESTART IDENTITY CASCADE;
|
||||||
|
TRUNCATE TABLE sales_performance RESTART IDENTITY CASCADE;
|
||||||
|
TRUNCATE TABLE sales_targets RESTART IDENTITY CASCADE;
|
||||||
|
TRUNCATE TABLE sales_transactions RESTART IDENTITY CASCADE;
|
||||||
|
TRUNCATE TABLE product_catalog RESTART IDENTITY CASCADE;
|
||||||
|
TRUNCATE TABLE engagement_metrics RESTART IDENTITY CASCADE;
|
||||||
|
TRUNCATE TABLE customer_satisfaction RESTART IDENTITY CASCADE;
|
||||||
|
TRUNCATE TABLE churn_predictions RESTART IDENTITY CASCADE;
|
||||||
|
TRUNCATE TABLE customer_lifetime_value RESTART IDENTITY CASCADE;
|
||||||
|
TRUNCATE TABLE customer_segments RESTART IDENTITY CASCADE;
|
||||||
|
|
||||||
|
-- Existing tables
|
||||||
TRUNCATE TABLE suspicious_activity_reports CASCADE;
|
TRUNCATE TABLE suspicious_activity_reports CASCADE;
|
||||||
TRUNCATE TABLE case_alerts CASCADE;
|
TRUNCATE TABLE case_alerts CASCADE;
|
||||||
TRUNCATE TABLE case_transactions CASCADE;
|
TRUNCATE TABLE case_transactions CASCADE;
|
||||||
@@ -254,3 +275,91 @@ COMMENT ON FUNCTION generate_fraud_score IS 'Calculates fraud risk score based o
|
|||||||
COMMENT ON FUNCTION update_account_balance IS 'Automatically updates account balance after transaction';
|
COMMENT ON FUNCTION update_account_balance IS 'Automatically updates account balance after transaction';
|
||||||
COMMENT ON FUNCTION check_suspicious_transaction IS 'Generates alerts for suspicious transactions';
|
COMMENT ON FUNCTION check_suspicious_transaction IS 'Generates alerts for suspicious transactions';
|
||||||
|
|
||||||
|
-- ============================================================================
|
||||||
|
-- SEED DATA FOR NEW ANALYTICS TABLES
|
||||||
|
-- ============================================================================
|
||||||
|
|
||||||
|
-- Customer Segments
|
||||||
|
INSERT INTO customer_segments (segment_name, segment_description, min_clv, max_clv) VALUES
|
||||||
|
('VIP', 'Very Important Person - Highest value customers', 50000, NULL),
|
||||||
|
('High Value', 'High spending customers with strong loyalty', 10000, 49999),
|
||||||
|
('Medium Value', 'Regular customers with moderate spending', 2000, 9999),
|
||||||
|
('Low Value', 'Occasional customers with low spending', 500, 1999),
|
||||||
|
('At Risk', 'Previously active customers showing decline', NULL, NULL),
|
||||||
|
('New Customer', 'Recently acquired customers (< 90 days)', NULL, NULL),
|
||||||
|
('Dormant', 'Inactive customers (> 180 days)', NULL, NULL),
|
||||||
|
('Churned', 'Lost customers who have not transacted in 365+ days', NULL, NULL);
|
||||||
|
|
||||||
|
-- Product Catalog (Sample Products)
|
||||||
|
INSERT INTO product_catalog (product_name, product_category, product_subcategory, unit_price, cost_price, margin_percentage, is_active, launch_date) VALUES
|
||||||
|
-- Banking Products
|
||||||
|
('Premium Checking Account', 'Banking', 'Checking', 0.00, 0.00, 0.00, TRUE, '2020-01-01'),
|
||||||
|
('Basic Savings Account', 'Banking', 'Savings', 0.00, 0.00, 0.00, TRUE, '2020-01-01'),
|
||||||
|
('High-Yield Savings', 'Banking', 'Savings', 0.00, 0.00, 0.00, TRUE, '2021-06-01'),
|
||||||
|
('Business Checking', 'Banking', 'Business', 15.00, 5.00, 66.67, TRUE, '2020-01-01'),
|
||||||
|
('Student Checking', 'Banking', 'Checking', 0.00, 0.00, 0.00, TRUE, '2020-01-01'),
|
||||||
|
|
||||||
|
-- Credit Cards
|
||||||
|
('Platinum Credit Card', 'Credit', 'Premium', 99.00, 20.00, 79.80, TRUE, '2020-01-01'),
|
||||||
|
('Gold Credit Card', 'Credit', 'Standard', 49.00, 15.00, 69.39, TRUE, '2020-01-01'),
|
||||||
|
('Cash Back Card', 'Credit', 'Rewards', 0.00, 10.00, -100.00, TRUE, '2021-01-01'),
|
||||||
|
('Travel Rewards Card', 'Credit', 'Rewards', 95.00, 25.00, 73.68, TRUE, '2021-03-01'),
|
||||||
|
('Business Credit Card', 'Credit', 'Business', 75.00, 20.00, 73.33, TRUE, '2020-06-01'),
|
||||||
|
|
||||||
|
-- Loans
|
||||||
|
('Personal Loan', 'Lending', 'Personal', 0.00, 0.00, 0.00, TRUE, '2020-01-01'),
|
||||||
|
('Auto Loan', 'Lending', 'Auto', 0.00, 0.00, 0.00, TRUE, '2020-01-01'),
|
||||||
|
('Home Mortgage', 'Lending', 'Mortgage', 0.00, 0.00, 0.00, TRUE, '2020-01-01'),
|
||||||
|
('Small Business Loan', 'Lending', 'Business', 0.00, 0.00, 0.00, TRUE, '2020-01-01'),
|
||||||
|
|
||||||
|
-- Investment Products
|
||||||
|
('Index Fund', 'Investment', 'Mutual Funds', 0.00, 0.00, 0.00, TRUE, '2020-01-01'),
|
||||||
|
('Bond Fund', 'Investment', 'Mutual Funds', 0.00, 0.00, 0.00, TRUE, '2020-01-01'),
|
||||||
|
('Retirement Account (IRA)', 'Investment', 'Retirement', 0.00, 0.00, 0.00, TRUE, '2020-01-01'),
|
||||||
|
('401k Plan', 'Investment', 'Retirement', 0.00, 0.00, 0.00, TRUE, '2020-01-01'),
|
||||||
|
|
||||||
|
-- Services
|
||||||
|
('Wire Transfer', 'Services', 'Transfers', 25.00, 5.00, 80.00, TRUE, '2020-01-01'),
|
||||||
|
('International Transfer', 'Services', 'Transfers', 45.00, 10.00, 77.78, TRUE, '2020-01-01'),
|
||||||
|
('Overdraft Protection', 'Services', 'Protection', 35.00, 5.00, 85.71, TRUE, '2020-01-01'),
|
||||||
|
('Safe Deposit Box', 'Services', 'Security', 75.00, 20.00, 73.33, TRUE, '2020-01-01'),
|
||||||
|
('Financial Advisory', 'Services', 'Advisory', 150.00, 50.00, 66.67, TRUE, '2021-01-01'),
|
||||||
|
('Mobile Banking Premium', 'Services', 'Digital', 9.99, 2.00, 79.98, TRUE, '2022-01-01');
|
||||||
|
|
||||||
|
-- KPI Definitions
|
||||||
|
INSERT INTO kpi_definitions (kpi_name, kpi_description, kpi_category, calculation_formula, target_value, threshold_warning, threshold_critical, unit_of_measure, refresh_frequency, is_active) VALUES
|
||||||
|
-- Sales KPIs
|
||||||
|
('Daily Revenue', 'Total revenue generated per day', 'SALES', 'SUM(amount) FROM transactions WHERE DATE(transaction_date) = CURRENT_DATE', 500000, 400000, 300000, 'USD', 'DAILY', TRUE),
|
||||||
|
('Monthly Revenue', 'Total revenue for the month', 'SALES', 'SUM(amount) FROM transactions WHERE MONTH(transaction_date) = CURRENT_MONTH', 15000000, 12000000, 10000000, 'USD', 'DAILY', TRUE),
|
||||||
|
('Average Transaction Value', 'Average value per transaction', 'SALES', 'AVG(amount) FROM transactions', 150, 100, 75, 'USD', 'DAILY', TRUE),
|
||||||
|
('Transactions Per Day', 'Number of transactions per day', 'SALES', 'COUNT(*) FROM transactions WHERE DATE(transaction_date) = CURRENT_DATE', 10000, 7500, 5000, 'Count', 'DAILY', TRUE),
|
||||||
|
|
||||||
|
-- Customer KPIs
|
||||||
|
('New Customers', 'New customer acquisitions', 'CUSTOMER', 'COUNT(*) FROM customers WHERE registration_date >= CURRENT_DATE - 30', 1000, 750, 500, 'Count', 'DAILY', TRUE),
|
||||||
|
('Customer Churn Rate', 'Percentage of customers churning', 'CUSTOMER', '(Churned / Total) * 100', 5, 7, 10, 'Percentage', 'WEEKLY', TRUE),
|
||||||
|
('Customer Lifetime Value', 'Average CLV across all customers', 'CUSTOMER', 'AVG(clv_score) FROM customer_lifetime_value', 5000, 4000, 3000, 'USD', 'WEEKLY', TRUE),
|
||||||
|
('Net Promoter Score', 'Customer satisfaction metric', 'CUSTOMER', 'AVG(nps_score) FROM customer_satisfaction', 50, 30, 10, 'Score', 'WEEKLY', TRUE),
|
||||||
|
('Customer Engagement Score', 'Average engagement across customers', 'CUSTOMER', 'AVG(engagement_score) FROM engagement_metrics', 75, 60, 45, 'Score', 'DAILY', TRUE),
|
||||||
|
|
||||||
|
-- Operational KPIs
|
||||||
|
('Transaction Success Rate', 'Percentage of successful transactions', 'OPERATIONAL', '(Successful / Total) * 100', 99, 97, 95, 'Percentage', 'HOURLY', TRUE),
|
||||||
|
('Average Response Time', 'System response time', 'OPERATIONAL', 'AVG(response_time_ms)', 200, 500, 1000, 'Milliseconds', 'REALTIME', TRUE),
|
||||||
|
('Fraud Detection Rate', 'Percentage of fraud caught', 'OPERATIONAL', '(Detected / Total Fraud) * 100', 95, 90, 85, 'Percentage', 'DAILY', TRUE),
|
||||||
|
('Alert Resolution Time', 'Average time to resolve alerts', 'OPERATIONAL', 'AVG(resolution_time_hours)', 24, 48, 72, 'Hours', 'DAILY', TRUE),
|
||||||
|
|
||||||
|
-- Financial KPIs
|
||||||
|
('Gross Margin', 'Overall profit margin', 'FINANCIAL', '((Revenue - Cost) / Revenue) * 100', 75, 65, 55, 'Percentage', 'DAILY', TRUE),
|
||||||
|
('Customer Acquisition Cost', 'Cost to acquire new customer', 'FINANCIAL', 'Marketing Spend / New Customers', 50, 75, 100, 'USD', 'WEEKLY', TRUE),
|
||||||
|
('Return on Investment', 'ROI on marketing campaigns', 'FINANCIAL', '((Revenue - Cost) / Cost) * 100', 300, 200, 100, 'Percentage', 'WEEKLY', TRUE);
|
||||||
|
|
||||||
|
-- Report Definitions
|
||||||
|
INSERT INTO report_definitions (report_name, report_description, report_category, output_format, schedule_frequency, is_active) VALUES
|
||||||
|
('Daily Sales Summary', 'Daily sales performance report', 'Sales', 'PDF', 'Daily at 6 AM', TRUE),
|
||||||
|
('Weekly Customer Analytics', 'Customer behavior and segmentation analysis', 'Customer', 'EXCEL', 'Weekly on Monday', TRUE),
|
||||||
|
('Monthly Financial Summary', 'Comprehensive monthly financial report', 'Financial', 'PDF', 'Monthly on 1st', TRUE),
|
||||||
|
('Fraud Detection Report', 'Daily fraud alerts and cases', 'Risk', 'PDF', 'Daily at 8 AM', TRUE),
|
||||||
|
('KPI Dashboard', 'Executive KPI dashboard', 'Executive', 'HTML', 'Daily at 7 AM', TRUE),
|
||||||
|
('Customer Churn Analysis', 'At-risk customer identification', 'Customer', 'EXCEL', 'Weekly on Friday', TRUE),
|
||||||
|
('Product Performance', 'Product sales and profitability analysis', 'Sales', 'EXCEL', 'Monthly on 5th', TRUE),
|
||||||
|
('Data Quality Report', 'Data validation and quality metrics', 'Operations', 'CSV', 'Daily at 5 AM', TRUE);
|
||||||
|
|
||||||
|
|||||||
@@ -3,7 +3,7 @@
|
|||||||
# ============================================================================
|
# ============================================================================
|
||||||
# Database Setup Script - IDEMPOTENT
|
# Database Setup Script - IDEMPOTENT
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
# This script sets up the complete fraud detection database
|
# This script sets up the complete business analytics database
|
||||||
# It can be run multiple times safely - it will recreate everything
|
# It can be run multiple times safely - it will recreate everything
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
|
|
||||||
@@ -19,8 +19,8 @@ NC='\033[0m' # No Color
|
|||||||
# Configuration from environment or defaults
|
# Configuration from environment or defaults
|
||||||
DB_HOST="${POSTGRES_HOST:-localhost}"
|
DB_HOST="${POSTGRES_HOST:-localhost}"
|
||||||
DB_PORT="${POSTGRES_PORT:-5432}"
|
DB_PORT="${POSTGRES_PORT:-5432}"
|
||||||
DB_NAME="${POSTGRES_DB:-fraud_detection}"
|
DB_NAME="${POSTGRES_DB:-business_analytics}"
|
||||||
DB_USER="${POSTGRES_USER:-fraud_analyst}"
|
DB_USER="${POSTGRES_USER:-data_analyst}"
|
||||||
DB_PASSWORD="${POSTGRES_PASSWORD:-SecurePass123!}"
|
DB_PASSWORD="${POSTGRES_PASSWORD:-SecurePass123!}"
|
||||||
|
|
||||||
# Script directory
|
# Script directory
|
||||||
@@ -28,7 +28,7 @@ SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
|
|||||||
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
|
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
|
||||||
|
|
||||||
echo -e "${BLUE}============================================================================${NC}"
|
echo -e "${BLUE}============================================================================${NC}"
|
||||||
echo -e "${BLUE}Financial Fraud Detection Database - Setup Script${NC}"
|
echo -e "${BLUE}Business Analytics Database - Setup Script${NC}"
|
||||||
echo -e "${BLUE}============================================================================${NC}"
|
echo -e "${BLUE}============================================================================${NC}"
|
||||||
echo -e "Database: ${GREEN}$DB_NAME${NC}"
|
echo -e "Database: ${GREEN}$DB_NAME${NC}"
|
||||||
echo -e "Host: ${GREEN}$DB_HOST:$DB_PORT${NC}"
|
echo -e "Host: ${GREEN}$DB_HOST:$DB_PORT${NC}"
|
||||||
|
|||||||
@@ -3,7 +3,7 @@
|
|||||||
# ============================================================================
|
# ============================================================================
|
||||||
# Setup Verification Script
|
# Setup Verification Script
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
# Verifies that the fraud detection database is properly set up
|
# Verifies that the business analytics database is properly set up
|
||||||
# ============================================================================
|
# ============================================================================
|
||||||
|
|
||||||
set -e
|
set -e
|
||||||
@@ -18,12 +18,12 @@ NC='\033[0m' # No Color
|
|||||||
# Configuration
|
# Configuration
|
||||||
DB_HOST="${POSTGRES_HOST:-localhost}"
|
DB_HOST="${POSTGRES_HOST:-localhost}"
|
||||||
DB_PORT="${POSTGRES_PORT:-5432}"
|
DB_PORT="${POSTGRES_PORT:-5432}"
|
||||||
DB_NAME="${POSTGRES_DB:-fraud_detection}"
|
DB_NAME="${POSTGRES_DB:-business_analytics}"
|
||||||
DB_USER="${POSTGRES_USER:-fraud_analyst}"
|
DB_USER="${POSTGRES_USER:-data_analyst}"
|
||||||
DB_PASSWORD="${POSTGRES_PASSWORD:-SecurePass123!}"
|
DB_PASSWORD="${POSTGRES_PASSWORD:-SecurePass123!}"
|
||||||
|
|
||||||
echo -e "${BLUE}============================================================================${NC}"
|
echo -e "${BLUE}============================================================================${NC}"
|
||||||
echo -e "${BLUE}Financial Fraud Detection Database - Verification${NC}"
|
echo -e "${BLUE}Business Analytics Database - Verification${NC}"
|
||||||
echo -e "${BLUE}============================================================================${NC}"
|
echo -e "${BLUE}============================================================================${NC}"
|
||||||
echo ""
|
echo ""
|
||||||
|
|
||||||
@@ -34,7 +34,7 @@ execute_sql() {
|
|||||||
|
|
||||||
# Check 1: Docker containers
|
# Check 1: Docker containers
|
||||||
echo -e "${YELLOW}[1/10] Checking Docker containers...${NC}"
|
echo -e "${YELLOW}[1/10] Checking Docker containers...${NC}"
|
||||||
if docker ps | grep -q "fraud_detection_db"; then
|
if docker ps | grep -q "business_analytics_db"; then
|
||||||
echo -e "${GREEN}✓ PostgreSQL container is running${NC}"
|
echo -e "${GREEN}✓ PostgreSQL container is running${NC}"
|
||||||
else
|
else
|
||||||
echo -e "${RED}✗ PostgreSQL container is not running${NC}"
|
echo -e "${RED}✗ PostgreSQL container is not running${NC}"
|
||||||
@@ -42,7 +42,7 @@ else
|
|||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
if docker ps | grep -q "fraud_detection_ui"; then
|
if docker ps | grep -q "business_analytics_ui"; then
|
||||||
echo -e "${GREEN}✓ DB-UI container is running${NC}"
|
echo -e "${GREEN}✓ DB-UI container is running${NC}"
|
||||||
else
|
else
|
||||||
echo -e "${RED}✗ DB-UI container is not running${NC}"
|
echo -e "${RED}✗ DB-UI container is not running${NC}"
|
||||||
|
|||||||
Reference in New Issue
Block a user