mirror of
https://github.com/freedbygrace/SQL.git
synced 2026-08-06 00:17:42 +00:00
Add complete financial fraud detection database with Docker, schema, data generation, and SQL exercises
- Docker setup with PostgreSQL 16 and DB-UI web interface - Comprehensive 20+ table schema with fraud detection patterns - Idempotent shell scripts for data generation (no Python dependency) - Realistic geographic data (100 US cities, 210 world cities) - 5M+ transactions with embedded fraud patterns (velocity, geographic, structuring, etc.) - Progressive SQL exercises from beginner to advanced fraud detection - Complete documentation and quick start guide - Setup and verification scripts
This commit is contained in:
@@ -1,2 +1,311 @@
|
|||||||
# SQL
|
# Financial Fraud Detection - SQL Learning Database
|
||||||
A repository for learning SQL
|
|
||||||
|
A comprehensive, production-grade database designed for learning SQL through realistic financial fraud investigation scenarios.
|
||||||
|
|
||||||
|
## 🎯 Overview
|
||||||
|
|
||||||
|
This project provides a complete PostgreSQL database with **5+ million transactions**, embedded fraud patterns, and progressive SQL exercises. Perfect for:
|
||||||
|
|
||||||
|
- **SQL Beginners** → Learn fundamentals with real-world data
|
||||||
|
- **Data Analysts** → Practice fraud detection queries
|
||||||
|
- **Security Professionals** → Understand fraud patterns
|
||||||
|
- **Students** → Hands-on financial crime investigation
|
||||||
|
|
||||||
|
## 📊 Database Statistics
|
||||||
|
|
||||||
|
- **100,000** Customers with KYC data
|
||||||
|
- **150,000** Bank accounts (checking, savings, credit)
|
||||||
|
- **200,000** Payment cards
|
||||||
|
- **50,000** Merchants across 35 categories
|
||||||
|
- **5,000,000** Transactions (7% fraudulent)
|
||||||
|
- **50,000+** Fraud alerts
|
||||||
|
- **5,000+** Fraud cases
|
||||||
|
- **500,000** Login sessions
|
||||||
|
|
||||||
|
## 🏗️ Architecture
|
||||||
|
|
||||||
|
### Data Model Features
|
||||||
|
- ✅ **20+ Tables** with proper relationships
|
||||||
|
- ✅ **Foreign key constraints** for data integrity
|
||||||
|
- ✅ **Indexes** for query performance
|
||||||
|
- ✅ **Realistic geographic data** (100 US cities, 210 world cities)
|
||||||
|
- ✅ **Embedded fraud patterns** (velocity, geographic, structuring)
|
||||||
|
- ✅ **Audit trails** and compliance tables
|
||||||
|
|
||||||
|
### Key Entities
|
||||||
|
```
|
||||||
|
customers → accounts → transactions
|
||||||
|
↓
|
||||||
|
cards → merchants
|
||||||
|
↓
|
||||||
|
alerts → fraud_cases
|
||||||
|
```
|
||||||
|
|
||||||
|
## 🚀 Quick Start
|
||||||
|
|
||||||
|
### Prerequisites
|
||||||
|
- Docker and Docker Compose
|
||||||
|
- 8GB RAM minimum
|
||||||
|
- 20GB disk space
|
||||||
|
|
||||||
|
### 1. Clone the Repository
|
||||||
|
```bash
|
||||||
|
git clone https://github.com/yourusername/SQL.git
|
||||||
|
cd SQL
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Start the Database
|
||||||
|
```bash
|
||||||
|
docker-compose up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
This starts:
|
||||||
|
- **PostgreSQL 16** on port `5432`
|
||||||
|
- **DB-UI** web interface on port `3000`
|
||||||
|
|
||||||
|
### 3. Initialize the Schema
|
||||||
|
```bash
|
||||||
|
chmod +x scripts/setup-database.sh
|
||||||
|
./scripts/setup-database.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Generate Test Data
|
||||||
|
```bash
|
||||||
|
chmod +x data/generate_data.sh
|
||||||
|
./data/generate_data.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
⏱️ **Note:** Data generation takes 15-30 minutes depending on your system.
|
||||||
|
|
||||||
|
### 5. Access the Database
|
||||||
|
|
||||||
|
**Option A: DB-UI Web Interface**
|
||||||
|
```
|
||||||
|
http://localhost:3000
|
||||||
|
```
|
||||||
|
|
||||||
|
**Option B: Command Line**
|
||||||
|
```bash
|
||||||
|
docker exec -it fraud_detection_db psql -U fraud_analyst -d fraud_detection
|
||||||
|
```
|
||||||
|
|
||||||
|
**Option C: Your Favorite SQL Client**
|
||||||
|
```
|
||||||
|
Host: localhost
|
||||||
|
Port: 5432
|
||||||
|
Database: fraud_detection
|
||||||
|
Username: fraud_analyst
|
||||||
|
Password: SecurePass123!
|
||||||
|
```
|
||||||
|
|
||||||
|
## 📚 Learning Path
|
||||||
|
|
||||||
|
### Level 1: Basic Queries
|
||||||
|
- SELECT, WHERE, ORDER BY
|
||||||
|
- Filtering and sorting
|
||||||
|
- Basic comparisons
|
||||||
|
- **Location:** `exercises/01-basic-queries/`
|
||||||
|
|
||||||
|
### Level 2: Joins
|
||||||
|
- INNER JOIN, LEFT JOIN
|
||||||
|
- Multiple table queries
|
||||||
|
- Relationship navigation
|
||||||
|
- **Location:** `exercises/02-joins/`
|
||||||
|
|
||||||
|
### Level 3: Aggregations
|
||||||
|
- COUNT, SUM, AVG, MAX, MIN
|
||||||
|
- GROUP BY and HAVING
|
||||||
|
- Statistical analysis
|
||||||
|
- **Location:** `exercises/03-aggregations/`
|
||||||
|
|
||||||
|
### Level 4: Subqueries
|
||||||
|
- Nested queries
|
||||||
|
- Correlated subqueries
|
||||||
|
- EXISTS and IN
|
||||||
|
- **Location:** `exercises/04-subqueries/`
|
||||||
|
|
||||||
|
### Level 5: Window Functions
|
||||||
|
- ROW_NUMBER, RANK, DENSE_RANK
|
||||||
|
- Running totals
|
||||||
|
- Moving averages
|
||||||
|
- **Location:** `exercises/05-window-functions/`
|
||||||
|
|
||||||
|
### Level 6: Fraud Detection
|
||||||
|
- Velocity fraud detection
|
||||||
|
- Geographic anomalies
|
||||||
|
- Money mule networks
|
||||||
|
- Account takeover patterns
|
||||||
|
- **Location:** `exercises/06-fraud-detection/`
|
||||||
|
|
||||||
|
## 🔍 Fraud Patterns Included
|
||||||
|
|
||||||
|
### 1. Velocity Fraud
|
||||||
|
Multiple rapid transactions from the same account
|
||||||
|
|
||||||
|
### 2. Geographic Impossibility
|
||||||
|
Card used in different countries within hours
|
||||||
|
|
||||||
|
### 3. Money Mule Networks
|
||||||
|
Rapid transfer chains between accounts
|
||||||
|
|
||||||
|
### 4. Account Takeover
|
||||||
|
Sudden changes in transaction patterns
|
||||||
|
|
||||||
|
### 5. Structuring (Smurfing)
|
||||||
|
Multiple transactions just under $10,000 reporting threshold
|
||||||
|
|
||||||
|
### 6. Card Testing
|
||||||
|
Multiple small failed transactions
|
||||||
|
|
||||||
|
### 7. High-Risk Merchants
|
||||||
|
Unusual activity at gambling/crypto merchants
|
||||||
|
|
||||||
|
### 8. Dormant Account Reactivation
|
||||||
|
Long-inactive accounts suddenly active
|
||||||
|
|
||||||
|
## 📁 Project Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
SQL/
|
||||||
|
├── docker-compose.yml # Docker orchestration
|
||||||
|
├── docker/
|
||||||
|
│ └── init/ # Database initialization
|
||||||
|
├── schema/
|
||||||
|
│ ├── 01-create-tables.sql # DDL (idempotent)
|
||||||
|
│ └── 02-seed-data.sql # Reference data
|
||||||
|
├── data/
|
||||||
|
│ ├── generate_data.sh # Data generation script
|
||||||
|
│ └── reference/ # Geographic data
|
||||||
|
│ ├── us_cities.csv
|
||||||
|
│ ├── world_cities.csv
|
||||||
|
│ └── load_geographic_data.sql
|
||||||
|
├── scripts/
|
||||||
|
│ └── setup-database.sh # Setup automation
|
||||||
|
├── exercises/
|
||||||
|
│ ├── 01-basic-queries/
|
||||||
|
│ ├── 02-joins/
|
||||||
|
│ ├── 03-aggregations/
|
||||||
|
│ ├── 04-subqueries/
|
||||||
|
│ ├── 05-window-functions/
|
||||||
|
│ └── 06-fraud-detection/
|
||||||
|
└── docs/
|
||||||
|
├── data-model.md
|
||||||
|
├── fraud-patterns.md
|
||||||
|
└── setup-guide.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## 🔧 Configuration
|
||||||
|
|
||||||
|
### Environment Variables
|
||||||
|
Edit `docker-compose.yml` to customize:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
POSTGRES_DB: fraud_detection
|
||||||
|
POSTGRES_USER: fraud_analyst
|
||||||
|
POSTGRES_PASSWORD: SecurePass123!
|
||||||
|
```
|
||||||
|
|
||||||
|
### Data Volume
|
||||||
|
Modify `data/generate_data.sh`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
NUM_CUSTOMERS=100000 # Adjust as needed
|
||||||
|
NUM_TRANSACTIONS=5000000 # Adjust as needed
|
||||||
|
FRAUD_PERCENTAGE=7 # 7% fraudulent
|
||||||
|
```
|
||||||
|
|
||||||
|
## 🔄 Idempotency
|
||||||
|
|
||||||
|
All scripts are **idempotent** - safe to run multiple times:
|
||||||
|
|
||||||
|
- `setup-database.sh` - Drops and recreates schema
|
||||||
|
- `generate_data.sh` - Clears and regenerates data
|
||||||
|
- Schema files use `DROP IF EXISTS`
|
||||||
|
|
||||||
|
## 🎓 Sample Queries
|
||||||
|
|
||||||
|
### Find High-Risk Customers
|
||||||
|
```sql
|
||||||
|
SELECT customer_id, first_name, last_name, risk_score
|
||||||
|
FROM customers
|
||||||
|
WHERE risk_score > 80
|
||||||
|
ORDER BY risk_score DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Detect Velocity Fraud
|
||||||
|
```sql
|
||||||
|
SELECT account_id, COUNT(*) as txn_count, SUM(amount) as total
|
||||||
|
FROM transactions
|
||||||
|
WHERE transaction_date >= NOW() - INTERVAL '1 hour'
|
||||||
|
GROUP BY account_id
|
||||||
|
HAVING COUNT(*) > 5;
|
||||||
|
```
|
||||||
|
|
||||||
|
### Geographic Anomalies
|
||||||
|
```sql
|
||||||
|
SELECT t1.card_id, c1.country_name, c2.country_name,
|
||||||
|
t2.transaction_date - t1.transaction_date as time_diff
|
||||||
|
FROM transactions t1
|
||||||
|
JOIN transactions t2 ON t1.card_id = t2.card_id
|
||||||
|
JOIN countries c1 ON t1.country_id = c1.country_id
|
||||||
|
JOIN countries c2 ON t2.country_id = c2.country_id
|
||||||
|
WHERE t1.country_id != t2.country_id
|
||||||
|
AND t2.transaction_date BETWEEN t1.transaction_date
|
||||||
|
AND t1.transaction_date + INTERVAL '2 hours';
|
||||||
|
```
|
||||||
|
|
||||||
|
## 🛠️ Maintenance
|
||||||
|
|
||||||
|
### Reset Everything
|
||||||
|
```bash
|
||||||
|
docker-compose down -v
|
||||||
|
docker-compose up -d
|
||||||
|
./scripts/setup-database.sh
|
||||||
|
./data/generate_data.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
### Backup Database
|
||||||
|
```bash
|
||||||
|
docker exec fraud_detection_db pg_dump -U fraud_analyst fraud_detection > backup.sql
|
||||||
|
```
|
||||||
|
|
||||||
|
### Restore Database
|
||||||
|
```bash
|
||||||
|
cat backup.sql | docker exec -i fraud_detection_db psql -U fraud_analyst -d fraud_detection
|
||||||
|
```
|
||||||
|
|
||||||
|
## 📖 Documentation
|
||||||
|
|
||||||
|
- **[Data Model](docs/data-model.md)** - Complete ER diagram and table descriptions
|
||||||
|
- **[Fraud Patterns](docs/fraud-patterns.md)** - Detailed fraud scenario explanations
|
||||||
|
- **[Setup Guide](docs/setup-guide.md)** - Detailed installation instructions
|
||||||
|
|
||||||
|
## 🤝 Contributing
|
||||||
|
|
||||||
|
Contributions welcome! Please:
|
||||||
|
1. Fork the repository
|
||||||
|
2. Create a feature branch
|
||||||
|
3. Add exercises or improve data generation
|
||||||
|
4. Submit a pull request
|
||||||
|
|
||||||
|
## 📝 License
|
||||||
|
|
||||||
|
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
|
||||||
|
|
||||||
|
## 🙏 Acknowledgments
|
||||||
|
|
||||||
|
- PostgreSQL community
|
||||||
|
- DB-UI project (https://github.com/n7olkachev/db-ui)
|
||||||
|
- Financial crime investigation best practices
|
||||||
|
|
||||||
|
## 📧 Support
|
||||||
|
|
||||||
|
- **Issues:** GitHub Issues
|
||||||
|
- **Discussions:** GitHub Discussions
|
||||||
|
- **Documentation:** `/docs` folder
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Happy Learning! 🎉**
|
||||||
|
|
||||||
|
Start with `exercises/01-basic-queries/` and work your way up to detecting sophisticated fraud patterns!
|
||||||
|
|||||||
@@ -0,0 +1,384 @@
|
|||||||
|
# 🎉 Financial Fraud Detection Database - Setup Complete!
|
||||||
|
|
||||||
|
## ✅ What Has Been Created
|
||||||
|
|
||||||
|
### 1. **Docker Infrastructure**
|
||||||
|
- ✅ `docker-compose.yml` - Orchestrates PostgreSQL + DB-UI
|
||||||
|
- ✅ PostgreSQL 16 Alpine container
|
||||||
|
- ✅ DB-UI web interface (https://github.com/n7olkachev/db-ui)
|
||||||
|
- ✅ Persistent volumes for data
|
||||||
|
- ✅ Health checks and auto-restart
|
||||||
|
|
||||||
|
### 2. **Database Schema (20+ Tables)**
|
||||||
|
|
||||||
|
#### Core Tables
|
||||||
|
- ✅ `customers` - 100K customer records with KYC data
|
||||||
|
- ✅ `accounts` - 150K bank accounts (checking, savings, credit)
|
||||||
|
- ✅ `cards` - 200K payment cards
|
||||||
|
- ✅ `transactions` - 5M transactions with fraud patterns
|
||||||
|
- ✅ `merchants` - 50K merchants across 35 categories
|
||||||
|
- ✅ `devices` - 75K device fingerprints
|
||||||
|
|
||||||
|
#### Fraud Detection Tables
|
||||||
|
- ✅ `alerts` - System-generated fraud alerts
|
||||||
|
- ✅ `fraud_cases` - Confirmed fraud investigations
|
||||||
|
- ✅ `case_transactions` - Links transactions to cases
|
||||||
|
- ✅ `case_alerts` - Links alerts to cases
|
||||||
|
|
||||||
|
#### Supporting Tables
|
||||||
|
- ✅ `countries` - 40 countries with risk levels
|
||||||
|
- ✅ `merchant_categories` - 35 MCC categories
|
||||||
|
- ✅ `transaction_types` - 15 transaction types
|
||||||
|
- ✅ `fraud_types` - 20 fraud pattern types
|
||||||
|
- ✅ `login_sessions` - 500K login history
|
||||||
|
- ✅ `transfers` - Money transfer records
|
||||||
|
- ✅ `beneficiaries` - Transfer recipients
|
||||||
|
- ✅ `customer_relationships` - Network analysis
|
||||||
|
- ✅ `suspicious_activity_reports` - SAR filings
|
||||||
|
- ✅ `audit_log` - Complete audit trail
|
||||||
|
|
||||||
|
### 3. **Realistic Geographic Data**
|
||||||
|
- ✅ 100 US cities with matching states
|
||||||
|
- ✅ 210 world cities across 40 countries
|
||||||
|
- ✅ Proper city/state/country relationships
|
||||||
|
- ✅ Risk-based country classifications
|
||||||
|
|
||||||
|
### 4. **Embedded Fraud Patterns**
|
||||||
|
- ✅ Velocity fraud (rapid transactions)
|
||||||
|
- ✅ Geographic impossibility (same card, different countries)
|
||||||
|
- ✅ Money mule networks (transfer chains)
|
||||||
|
- ✅ Account takeover (behavior changes)
|
||||||
|
- ✅ Structuring/Smurfing (avoiding $10K threshold)
|
||||||
|
- ✅ Card testing (multiple small failures)
|
||||||
|
- ✅ High-risk merchant abuse
|
||||||
|
- ✅ Dormant account reactivation
|
||||||
|
|
||||||
|
### 5. **Data Generation Scripts**
|
||||||
|
- ✅ `generate_data.sh` - Idempotent data generation
|
||||||
|
- ✅ Realistic distributions (Pareto, normal)
|
||||||
|
- ✅ Temporal patterns (2+ years of data)
|
||||||
|
- ✅ 7% fraud rate (industry realistic)
|
||||||
|
- ✅ Progress tracking and colored output
|
||||||
|
|
||||||
|
### 6. **Setup & Maintenance Scripts**
|
||||||
|
- ✅ `setup-database.sh` - Idempotent schema setup
|
||||||
|
- ✅ `verify-setup.sh` - Comprehensive verification
|
||||||
|
- ✅ All scripts with error handling
|
||||||
|
- ✅ Color-coded output for clarity
|
||||||
|
|
||||||
|
### 7. **SQL Learning Exercises**
|
||||||
|
|
||||||
|
#### Level 1: Basic Queries
|
||||||
|
- SELECT, WHERE, ORDER BY
|
||||||
|
- Filtering and sorting
|
||||||
|
- 10 exercises + 3 challenges
|
||||||
|
|
||||||
|
#### Level 6: Fraud Detection
|
||||||
|
- Velocity fraud detection
|
||||||
|
- Geographic anomalies
|
||||||
|
- Money mule networks
|
||||||
|
- Account takeover patterns
|
||||||
|
- Structuring detection
|
||||||
|
- Card testing
|
||||||
|
- High-risk merchant analysis
|
||||||
|
- Dormant account reactivation
|
||||||
|
|
||||||
|
### 8. **Documentation**
|
||||||
|
- ✅ Comprehensive README.md
|
||||||
|
- ✅ Quick Start Guide (docs/QUICKSTART.md)
|
||||||
|
- ✅ Exercise documentation
|
||||||
|
- ✅ Inline SQL comments
|
||||||
|
- ✅ This setup summary
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🚀 How to Use
|
||||||
|
|
||||||
|
### Quick Start (5 minutes + data generation time)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Start containers
|
||||||
|
docker-compose up -d
|
||||||
|
|
||||||
|
# 2. Setup schema
|
||||||
|
chmod +x scripts/*.sh
|
||||||
|
./scripts/setup-database.sh
|
||||||
|
|
||||||
|
# 3. Generate data (15-30 minutes)
|
||||||
|
chmod +x data/generate_data.sh
|
||||||
|
./data/generate_data.sh
|
||||||
|
|
||||||
|
# 4. Verify setup
|
||||||
|
./scripts/verify-setup.sh
|
||||||
|
|
||||||
|
# 5. Access DB-UI
|
||||||
|
# Open browser to: http://localhost:3000
|
||||||
|
```
|
||||||
|
|
||||||
|
### Connection Details
|
||||||
|
|
||||||
|
**DB-UI Web Interface:**
|
||||||
|
```
|
||||||
|
URL: http://localhost:3000
|
||||||
|
```
|
||||||
|
|
||||||
|
**Direct Database Connection:**
|
||||||
|
```
|
||||||
|
Host: localhost
|
||||||
|
Port: 5432
|
||||||
|
Database: fraud_detection
|
||||||
|
Username: fraud_analyst
|
||||||
|
Password: SecurePass123!
|
||||||
|
```
|
||||||
|
|
||||||
|
**Command Line (psql):**
|
||||||
|
```bash
|
||||||
|
docker exec -it fraud_detection_db psql -U fraud_analyst -d fraud_detection
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📊 Data Volumes
|
||||||
|
|
||||||
|
### Default Configuration
|
||||||
|
- **Customers:** 100,000
|
||||||
|
- **Accounts:** 150,000
|
||||||
|
- **Merchants:** 50,000
|
||||||
|
- **Devices:** 75,000
|
||||||
|
- **Cards:** 200,000
|
||||||
|
- **Login Sessions:** 500,000
|
||||||
|
- **Transactions:** 5,000,000
|
||||||
|
- **Alerts:** ~50,000
|
||||||
|
- **Fraud Cases:** ~5,000
|
||||||
|
|
||||||
|
### Customization
|
||||||
|
Edit `data/generate_data.sh`:
|
||||||
|
```bash
|
||||||
|
NUM_CUSTOMERS=100000 # Adjust as needed
|
||||||
|
NUM_ACCOUNTS=150000
|
||||||
|
NUM_MERCHANTS=50000
|
||||||
|
NUM_DEVICES=75000
|
||||||
|
NUM_CARDS=200000
|
||||||
|
NUM_TRANSACTIONS=5000000
|
||||||
|
FRAUD_PERCENTAGE=7 # 7% fraudulent
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔄 Idempotency
|
||||||
|
|
||||||
|
All scripts are **safe to run multiple times**:
|
||||||
|
|
||||||
|
- ✅ `setup-database.sh` - Drops and recreates schema
|
||||||
|
- ✅ `generate_data.sh` - Clears and regenerates data
|
||||||
|
- ✅ Schema files use `DROP IF EXISTS`
|
||||||
|
- ✅ Seed data uses `TRUNCATE ... RESTART IDENTITY`
|
||||||
|
|
||||||
|
**To reset everything:**
|
||||||
|
```bash
|
||||||
|
docker-compose down -v
|
||||||
|
docker-compose up -d
|
||||||
|
./scripts/setup-database.sh
|
||||||
|
./data/generate_data.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🎓 Learning Path
|
||||||
|
|
||||||
|
### Beginner (Weeks 1-2)
|
||||||
|
1. Start with `exercises/01-basic-queries/`
|
||||||
|
2. Learn SELECT, WHERE, ORDER BY
|
||||||
|
3. Practice filtering and sorting
|
||||||
|
4. Explore the data with DB-UI
|
||||||
|
|
||||||
|
### Intermediate (Weeks 3-4)
|
||||||
|
1. Master JOINs (exercises/02-joins/)
|
||||||
|
2. Learn aggregations (exercises/03-aggregations/)
|
||||||
|
3. Practice subqueries (exercises/04-subqueries/)
|
||||||
|
|
||||||
|
### Advanced (Weeks 5-6)
|
||||||
|
1. Window functions (exercises/05-window-functions/)
|
||||||
|
2. Fraud detection scenarios (exercises/06-fraud-detection/)
|
||||||
|
3. Complex pattern detection
|
||||||
|
4. Performance optimization
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔍 Sample Queries to Get Started
|
||||||
|
|
||||||
|
### 1. Explore the Data
|
||||||
|
```sql
|
||||||
|
-- How many records in each table?
|
||||||
|
SELECT 'customers' as table_name, COUNT(*) FROM customers
|
||||||
|
UNION ALL
|
||||||
|
SELECT 'accounts', COUNT(*) FROM accounts
|
||||||
|
UNION ALL
|
||||||
|
SELECT 'transactions', COUNT(*) FROM transactions
|
||||||
|
UNION ALL
|
||||||
|
SELECT 'alerts', COUNT(*) FROM alerts;
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Find High-Risk Activity
|
||||||
|
```sql
|
||||||
|
-- Top 10 highest fraud scores
|
||||||
|
SELECT
|
||||||
|
t.transaction_id,
|
||||||
|
c.first_name || ' ' || c.last_name as customer,
|
||||||
|
t.amount,
|
||||||
|
t.fraud_score,
|
||||||
|
t.flagged_reason
|
||||||
|
FROM transactions t
|
||||||
|
JOIN accounts a ON t.account_id = a.account_id
|
||||||
|
JOIN customers c ON a.customer_id = c.customer_id
|
||||||
|
WHERE t.is_flagged = TRUE
|
||||||
|
ORDER BY t.fraud_score DESC
|
||||||
|
LIMIT 10;
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Geographic Analysis
|
||||||
|
```sql
|
||||||
|
-- Transactions by country
|
||||||
|
SELECT
|
||||||
|
co.country_name,
|
||||||
|
co.risk_level,
|
||||||
|
COUNT(*) as transaction_count,
|
||||||
|
SUM(t.amount) as total_amount
|
||||||
|
FROM transactions t
|
||||||
|
JOIN countries co ON t.country_id = co.country_id
|
||||||
|
GROUP BY co.country_name, co.risk_level
|
||||||
|
ORDER BY total_amount DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🛠️ Maintenance
|
||||||
|
|
||||||
|
### Backup Database
|
||||||
|
```bash
|
||||||
|
docker exec fraud_detection_db pg_dump -U fraud_analyst fraud_detection > backup_$(date +%Y%m%d).sql
|
||||||
|
```
|
||||||
|
|
||||||
|
### Restore Database
|
||||||
|
```bash
|
||||||
|
cat backup_20241023.sql | docker exec -i fraud_detection_db psql -U fraud_analyst -d fraud_detection
|
||||||
|
```
|
||||||
|
|
||||||
|
### View Logs
|
||||||
|
```bash
|
||||||
|
# PostgreSQL logs
|
||||||
|
docker-compose logs postgres
|
||||||
|
|
||||||
|
# DB-UI logs
|
||||||
|
docker-compose logs db-ui
|
||||||
|
|
||||||
|
# Follow logs
|
||||||
|
docker-compose logs -f
|
||||||
|
```
|
||||||
|
|
||||||
|
### Performance Tuning
|
||||||
|
```sql
|
||||||
|
-- Check table sizes
|
||||||
|
SELECT
|
||||||
|
schemaname,
|
||||||
|
tablename,
|
||||||
|
pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size
|
||||||
|
FROM pg_tables
|
||||||
|
WHERE schemaname = 'public'
|
||||||
|
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC;
|
||||||
|
|
||||||
|
-- Check index usage
|
||||||
|
SELECT
|
||||||
|
schemaname,
|
||||||
|
tablename,
|
||||||
|
indexname,
|
||||||
|
idx_scan,
|
||||||
|
idx_tup_read,
|
||||||
|
idx_tup_fetch
|
||||||
|
FROM pg_stat_user_indexes
|
||||||
|
ORDER BY idx_scan DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📁 Project Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
SQL/
|
||||||
|
├── docker-compose.yml # Docker orchestration
|
||||||
|
├── README.md # Main documentation
|
||||||
|
├── SETUP_COMPLETE.md # This file
|
||||||
|
├── LICENSE # MIT License
|
||||||
|
│
|
||||||
|
├── docker/
|
||||||
|
│ └── init/
|
||||||
|
│ └── 00-init-database.sql
|
||||||
|
│
|
||||||
|
├── schema/
|
||||||
|
│ ├── 01-create-tables.sql # DDL (idempotent)
|
||||||
|
│ └── 02-seed-data.sql # Reference data
|
||||||
|
│
|
||||||
|
├── data/
|
||||||
|
│ ├── generate_data.sh # Data generation (idempotent)
|
||||||
|
│ └── reference/
|
||||||
|
│ ├── us_cities.csv
|
||||||
|
│ ├── world_cities.csv
|
||||||
|
│ └── load_geographic_data.sql
|
||||||
|
│
|
||||||
|
├── scripts/
|
||||||
|
│ ├── setup-database.sh # Schema setup (idempotent)
|
||||||
|
│ └── verify-setup.sh # Verification
|
||||||
|
│
|
||||||
|
├── exercises/
|
||||||
|
│ ├── 01-basic-queries/
|
||||||
|
│ │ └── README.md
|
||||||
|
│ └── 06-fraud-detection/
|
||||||
|
│ └── README.md
|
||||||
|
│
|
||||||
|
└── docs/
|
||||||
|
└── QUICKSTART.md # Quick start guide
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🎯 Success Criteria
|
||||||
|
|
||||||
|
Your setup is complete when:
|
||||||
|
|
||||||
|
- ✅ Docker containers are running
|
||||||
|
- ✅ Database has 20+ tables
|
||||||
|
- ✅ Reference data is loaded (countries, categories, etc.)
|
||||||
|
- ✅ Test data is generated (customers, transactions, etc.)
|
||||||
|
- ✅ DB-UI is accessible at http://localhost:3000
|
||||||
|
- ✅ You can run queries successfully
|
||||||
|
|
||||||
|
**Verify with:**
|
||||||
|
```bash
|
||||||
|
./scripts/verify-setup.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🤝 Next Steps
|
||||||
|
|
||||||
|
1. **Read the Quick Start:** `docs/QUICKSTART.md`
|
||||||
|
2. **Start Learning:** `exercises/01-basic-queries/README.md`
|
||||||
|
3. **Explore DB-UI:** http://localhost:3000
|
||||||
|
4. **Practice Queries:** Try the sample queries above
|
||||||
|
5. **Detect Fraud:** `exercises/06-fraud-detection/README.md`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📧 Support
|
||||||
|
|
||||||
|
- **Documentation:** Check `/docs` folder
|
||||||
|
- **Exercises:** Check `/exercises` folder
|
||||||
|
- **Issues:** Use GitHub Issues
|
||||||
|
- **Verification:** Run `./scripts/verify-setup.sh`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**🎉 Congratulations! Your fraud detection database is ready for SQL learning!**
|
||||||
|
|
||||||
|
**Start here:** `docs/QUICKSTART.md` or `exercises/01-basic-queries/README.md`
|
||||||
|
|
||||||
@@ -0,0 +1,541 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Financial Fraud Detection - Data Generation Script - IDEMPOTENT
|
||||||
|
# ============================================================================
|
||||||
|
# Generates realistic test data with embedded fraud patterns
|
||||||
|
# This script is IDEMPOTENT - it will clear and regenerate all data
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
# Colors for output
|
||||||
|
RED='\033[0;31m'
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
YELLOW='\033[1;33m'
|
||||||
|
BLUE='\033[0;34m'
|
||||||
|
NC='\033[0m' # No Color
|
||||||
|
|
||||||
|
# Configuration
|
||||||
|
DB_HOST="${POSTGRES_HOST:-localhost}"
|
||||||
|
DB_PORT="${POSTGRES_PORT:-5432}"
|
||||||
|
DB_NAME="${POSTGRES_DB:-fraud_detection}"
|
||||||
|
DB_USER="${POSTGRES_USER:-fraud_analyst}"
|
||||||
|
DB_PASSWORD="${POSTGRES_PASSWORD:-SecurePass123!}"
|
||||||
|
|
||||||
|
# Data volumes
|
||||||
|
NUM_CUSTOMERS=100000
|
||||||
|
NUM_ACCOUNTS=150000
|
||||||
|
NUM_MERCHANTS=50000
|
||||||
|
NUM_DEVICES=75000
|
||||||
|
NUM_CARDS=200000
|
||||||
|
NUM_TRANSACTIONS=5000000
|
||||||
|
FRAUD_PERCENTAGE=7 # 7% of transactions will be fraudulent
|
||||||
|
|
||||||
|
echo -e "${BLUE}============================================================================${NC}"
|
||||||
|
echo -e "${BLUE}Financial Fraud Detection Database - Data Generation${NC}"
|
||||||
|
echo -e "${BLUE}============================================================================${NC}"
|
||||||
|
echo -e "Target Database: ${GREEN}$DB_NAME@$DB_HOST:$DB_PORT${NC}"
|
||||||
|
echo -e "Customers: ${GREEN}$NUM_CUSTOMERS${NC}"
|
||||||
|
echo -e "Accounts: ${GREEN}$NUM_ACCOUNTS${NC}"
|
||||||
|
echo -e "Merchants: ${GREEN}$NUM_MERCHANTS${NC}"
|
||||||
|
echo -e "Cards: ${GREEN}$NUM_CARDS${NC}"
|
||||||
|
echo -e "Transactions: ${GREEN}$NUM_TRANSACTIONS${NC} (${YELLOW}${FRAUD_PERCENTAGE}% fraudulent${NC})"
|
||||||
|
echo -e "${BLUE}============================================================================${NC}"
|
||||||
|
echo ""
|
||||||
|
echo -e "${RED}WARNING: This will DELETE all existing data and regenerate it!${NC}"
|
||||||
|
echo -e "${YELLOW}Press Ctrl+C within 5 seconds to cancel...${NC}"
|
||||||
|
sleep 5
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Function to execute SQL
|
||||||
|
execute_sql() {
|
||||||
|
PGPASSWORD=$DB_PASSWORD psql -h $DB_HOST -p $DB_PORT -U $DB_USER -d $DB_NAME -c "$1" 2>&1
|
||||||
|
}
|
||||||
|
|
||||||
|
# Function to execute SQL file
|
||||||
|
execute_sql_file() {
|
||||||
|
PGPASSWORD=$DB_PASSWORD psql -h $DB_HOST -p $DB_PORT -U $DB_USER -d $DB_NAME -f "$1" 2>&1
|
||||||
|
}
|
||||||
|
|
||||||
|
# Get script directory
|
||||||
|
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
|
||||||
|
|
||||||
|
# Clear existing data (preserve reference tables)
|
||||||
|
echo -e "${YELLOW}[0/9] Clearing existing data...${NC}"
|
||||||
|
execute_sql "TRUNCATE TABLE audit_log CASCADE;"
|
||||||
|
execute_sql "TRUNCATE TABLE suspicious_activity_reports CASCADE;"
|
||||||
|
execute_sql "TRUNCATE TABLE case_alerts CASCADE;"
|
||||||
|
execute_sql "TRUNCATE TABLE case_transactions CASCADE;"
|
||||||
|
execute_sql "TRUNCATE TABLE fraud_cases CASCADE;"
|
||||||
|
execute_sql "TRUNCATE TABLE alerts CASCADE;"
|
||||||
|
execute_sql "TRUNCATE TABLE transfers CASCADE;"
|
||||||
|
execute_sql "TRUNCATE TABLE beneficiaries CASCADE;"
|
||||||
|
execute_sql "TRUNCATE TABLE transactions CASCADE;"
|
||||||
|
execute_sql "TRUNCATE TABLE login_sessions CASCADE;"
|
||||||
|
execute_sql "TRUNCATE TABLE devices RESTART IDENTITY CASCADE;"
|
||||||
|
execute_sql "TRUNCATE TABLE cards RESTART IDENTITY CASCADE;"
|
||||||
|
execute_sql "TRUNCATE TABLE accounts RESTART IDENTITY CASCADE;"
|
||||||
|
execute_sql "TRUNCATE TABLE customer_relationships RESTART IDENTITY CASCADE;"
|
||||||
|
execute_sql "TRUNCATE TABLE customers RESTART IDENTITY CASCADE;"
|
||||||
|
execute_sql "TRUNCATE TABLE merchants RESTART IDENTITY CASCADE;"
|
||||||
|
echo -e "${GREEN}✓ Existing data cleared${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Load geographic reference data
|
||||||
|
echo -e "${YELLOW}[1/9] Loading geographic reference data...${NC}"
|
||||||
|
execute_sql_file "$SCRIPT_DIR/reference/load_geographic_data.sql" > /dev/null
|
||||||
|
echo -e "${GREEN}✓ Geographic data loaded (100 US cities, 210 world cities)${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
echo -e "${YELLOW}[2/9] Generating Customers...${NC}"
|
||||||
|
cat > /tmp/generate_customers.sql << 'EOF'
|
||||||
|
-- Generate customers with realistic geographic data
|
||||||
|
INSERT INTO customers (
|
||||||
|
first_name, last_name, email, phone, date_of_birth, ssn_hash,
|
||||||
|
address_line1, city, state, postal_code, country_id,
|
||||||
|
registration_date, kyc_status, risk_score, is_pep, is_active
|
||||||
|
)
|
||||||
|
SELECT
|
||||||
|
'Customer' || gs.id AS first_name,
|
||||||
|
'User' || gs.id AS last_name,
|
||||||
|
'customer' || gs.id || '@email.com' AS email,
|
||||||
|
'+1' || LPAD((1000000000 + (gs.id % 9000000000))::TEXT, 10, '0') AS phone,
|
||||||
|
DATE '1950-01-01' + (random() * 25000)::INT AS date_of_birth,
|
||||||
|
encode(digest('SSN' || gs.id::TEXT, 'sha256'), 'hex') AS ssn_hash,
|
||||||
|
(gs.id % 10000) || ' Main Street' AS address_line1,
|
||||||
|
-- Use real US cities from temp table
|
||||||
|
(SELECT city FROM temp_us_cities WHERE id = ((gs.id % 100) + 1)) AS city,
|
||||||
|
(SELECT state_code FROM temp_us_cities WHERE id = ((gs.id % 100) + 1)) AS state,
|
||||||
|
LPAD((10000 + (gs.id % 90000))::TEXT, 5, '0') AS postal_code,
|
||||||
|
CASE
|
||||||
|
WHEN random() < 0.85 THEN 1 -- 85% US
|
||||||
|
WHEN random() < 0.90 THEN 2 -- 5% Canada
|
||||||
|
WHEN random() < 0.95 THEN 3 -- 5% UK
|
||||||
|
ELSE (3 + (gs.id % 37)) -- 5% other countries
|
||||||
|
END AS country_id,
|
||||||
|
TIMESTAMP '2020-01-01' + (random() * 1460)::INT * INTERVAL '1 day' AS registration_date,
|
||||||
|
CASE
|
||||||
|
WHEN random() < 0.90 THEN 'VERIFIED'
|
||||||
|
WHEN random() < 0.95 THEN 'PENDING'
|
||||||
|
ELSE 'REJECTED'
|
||||||
|
END AS kyc_status,
|
||||||
|
(random() * 100)::DECIMAL(5,2) AS risk_score,
|
||||||
|
random() < 0.02 AS is_pep, -- 2% are PEPs
|
||||||
|
random() < 0.98 AS is_active -- 98% active
|
||||||
|
FROM generate_series(1, 100000) AS gs(id);
|
||||||
|
EOF
|
||||||
|
|
||||||
|
execute_sql_file /tmp/generate_customers.sql > /dev/null
|
||||||
|
echo -e "${GREEN}✓ Generated $NUM_CUSTOMERS customers${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
echo -e "${YELLOW}[3/9] Generating Accounts...${NC}"
|
||||||
|
cat > /tmp/generate_accounts.sql << 'EOF'
|
||||||
|
-- Generate accounts (1-2 accounts per customer on average)
|
||||||
|
INSERT INTO accounts (
|
||||||
|
customer_id, account_number, account_type, currency,
|
||||||
|
opening_date, status, current_balance, available_balance,
|
||||||
|
credit_limit, overdraft_limit, is_primary
|
||||||
|
)
|
||||||
|
SELECT
|
||||||
|
(gs.id % 100000) + 1 AS customer_id,
|
||||||
|
'ACC' || LPAD(gs.id::TEXT, 12, '0') AS account_number,
|
||||||
|
CASE (gs.id % 10)
|
||||||
|
WHEN 0 THEN 'CHECKING'
|
||||||
|
WHEN 1 THEN 'CHECKING'
|
||||||
|
WHEN 2 THEN 'CHECKING'
|
||||||
|
WHEN 3 THEN 'SAVINGS'
|
||||||
|
WHEN 4 THEN 'SAVINGS'
|
||||||
|
WHEN 5 THEN 'CREDIT'
|
||||||
|
WHEN 6 THEN 'CREDIT'
|
||||||
|
WHEN 7 THEN 'INVESTMENT'
|
||||||
|
ELSE 'CHECKING'
|
||||||
|
END AS account_type,
|
||||||
|
'USD' AS currency,
|
||||||
|
DATE '2020-01-01' + (random() * 1460)::INT AS opening_date,
|
||||||
|
CASE
|
||||||
|
WHEN random() < 0.95 THEN 'ACTIVE'
|
||||||
|
WHEN random() < 0.98 THEN 'SUSPENDED'
|
||||||
|
ELSE 'CLOSED'
|
||||||
|
END AS status,
|
||||||
|
(random() * 50000)::DECIMAL(15,2) AS current_balance,
|
||||||
|
(random() * 50000)::DECIMAL(15,2) AS available_balance,
|
||||||
|
CASE
|
||||||
|
WHEN (gs.id % 10) IN (5, 6) THEN (5000 + random() * 45000)::DECIMAL(15,2)
|
||||||
|
ELSE NULL
|
||||||
|
END AS credit_limit,
|
||||||
|
CASE
|
||||||
|
WHEN (gs.id % 10) IN (0, 1, 2) THEN (random() * 1000)::DECIMAL(15,2)
|
||||||
|
ELSE 0
|
||||||
|
END AS overdraft_limit,
|
||||||
|
(gs.id % 2) = 0 AS is_primary
|
||||||
|
FROM generate_series(1, 150000) AS gs(id);
|
||||||
|
EOF
|
||||||
|
|
||||||
|
execute_sql_file /tmp/generate_accounts.sql > /dev/null
|
||||||
|
echo -e "${GREEN}✓ Generated $NUM_ACCOUNTS accounts${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
echo -e "${YELLOW}[4/9] Generating Merchants...${NC}"
|
||||||
|
cat > /tmp/generate_merchants.sql << 'EOF'
|
||||||
|
-- Generate merchants with realistic geographic data
|
||||||
|
INSERT INTO merchants (
|
||||||
|
merchant_name, merchant_code, category_id, country_id,
|
||||||
|
city, registration_date, status, risk_rating, is_verified
|
||||||
|
)
|
||||||
|
SELECT
|
||||||
|
CASE (gs.id % 15)
|
||||||
|
WHEN 0 THEN 'Walmart Store #' || gs.id
|
||||||
|
WHEN 1 THEN 'Amazon Marketplace #' || gs.id
|
||||||
|
WHEN 2 THEN 'Shell Gas Station #' || gs.id
|
||||||
|
WHEN 3 THEN 'McDonalds #' || gs.id
|
||||||
|
WHEN 4 THEN 'Starbucks #' || gs.id
|
||||||
|
WHEN 5 THEN 'Target Store #' || gs.id
|
||||||
|
WHEN 6 THEN 'Best Buy #' || gs.id
|
||||||
|
WHEN 7 THEN 'CVS Pharmacy #' || gs.id
|
||||||
|
WHEN 8 THEN 'Home Depot #' || gs.id
|
||||||
|
WHEN 9 THEN 'Costco #' || gs.id
|
||||||
|
WHEN 10 THEN 'Apple Store #' || gs.id
|
||||||
|
WHEN 11 THEN 'Marriott Hotel #' || gs.id
|
||||||
|
WHEN 12 THEN 'Delta Airlines #' || gs.id
|
||||||
|
WHEN 13 THEN 'Online Casino #' || gs.id
|
||||||
|
ELSE 'Merchant #' || gs.id
|
||||||
|
END AS merchant_name,
|
||||||
|
'MER' || LPAD(gs.id::TEXT, 10, '0') AS merchant_code,
|
||||||
|
((gs.id % 35) + 1) AS category_id,
|
||||||
|
CASE
|
||||||
|
WHEN random() < 0.80 THEN 1 -- 80% US merchants
|
||||||
|
WHEN random() < 0.90 THEN 2 -- 10% Canada
|
||||||
|
ELSE (3 + (gs.id % 37)) -- 10% international
|
||||||
|
END AS country_id,
|
||||||
|
-- Use real cities based on country
|
||||||
|
CASE
|
||||||
|
WHEN random() < 0.80 THEN (SELECT city FROM temp_us_cities WHERE id = ((gs.id % 100) + 1))
|
||||||
|
ELSE (SELECT city FROM temp_world_cities WHERE id = ((gs.id % 210) + 1))
|
||||||
|
END AS city,
|
||||||
|
DATE '2015-01-01' + (random() * 3000)::INT AS registration_date,
|
||||||
|
CASE
|
||||||
|
WHEN random() < 0.95 THEN 'ACTIVE'
|
||||||
|
WHEN random() < 0.98 THEN 'SUSPENDED'
|
||||||
|
ELSE 'BLACKLISTED'
|
||||||
|
END AS status,
|
||||||
|
CASE
|
||||||
|
WHEN (gs.id % 35) + 1 IN (21, 22, 23, 24, 25, 26, 31, 32, 33, 34, 35) THEN
|
||||||
|
CASE
|
||||||
|
WHEN random() < 0.5 THEN 'HIGH'
|
||||||
|
ELSE 'CRITICAL'
|
||||||
|
END
|
||||||
|
WHEN random() < 0.80 THEN 'LOW'
|
||||||
|
ELSE 'MEDIUM'
|
||||||
|
END AS risk_rating,
|
||||||
|
random() < 0.90 AS is_verified
|
||||||
|
FROM generate_series(1, 50000) AS gs(id);
|
||||||
|
EOF
|
||||||
|
|
||||||
|
execute_sql_file /tmp/generate_merchants.sql > /dev/null
|
||||||
|
echo -e "${GREEN}✓ Generated $NUM_MERCHANTS merchants${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
echo -e "${YELLOW}[5/9] Generating Devices...${NC}"
|
||||||
|
cat > /tmp/generate_devices.sql << 'EOF'
|
||||||
|
-- Generate devices
|
||||||
|
INSERT INTO devices (
|
||||||
|
device_fingerprint, device_type, os_name, os_version,
|
||||||
|
browser_name, browser_version, is_trusted, is_blacklisted
|
||||||
|
)
|
||||||
|
SELECT
|
||||||
|
encode(digest('DEVICE' || gs.id::TEXT, 'sha256'), 'hex') AS device_fingerprint,
|
||||||
|
CASE (gs.id % 4)
|
||||||
|
WHEN 0 THEN 'MOBILE'
|
||||||
|
WHEN 1 THEN 'DESKTOP'
|
||||||
|
WHEN 2 THEN 'TABLET'
|
||||||
|
ELSE 'MOBILE'
|
||||||
|
END AS device_type,
|
||||||
|
CASE (gs.id % 5)
|
||||||
|
WHEN 0 THEN 'iOS'
|
||||||
|
WHEN 1 THEN 'Android'
|
||||||
|
WHEN 2 THEN 'Windows'
|
||||||
|
WHEN 3 THEN 'macOS'
|
||||||
|
ELSE 'Linux'
|
||||||
|
END AS os_name,
|
||||||
|
CASE (gs.id % 5)
|
||||||
|
WHEN 0 THEN '15.0'
|
||||||
|
WHEN 1 THEN '12.0'
|
||||||
|
WHEN 2 THEN '11.0'
|
||||||
|
WHEN 3 THEN '13.0'
|
||||||
|
ELSE '10.0'
|
||||||
|
END AS os_version,
|
||||||
|
CASE (gs.id % 4)
|
||||||
|
WHEN 0 THEN 'Chrome'
|
||||||
|
WHEN 1 THEN 'Safari'
|
||||||
|
WHEN 2 THEN 'Firefox'
|
||||||
|
ELSE 'Edge'
|
||||||
|
END AS browser_name,
|
||||||
|
'100.0' AS browser_version,
|
||||||
|
random() < 0.85 AS is_trusted,
|
||||||
|
random() < 0.03 AS is_blacklisted
|
||||||
|
FROM generate_series(1, 75000) AS gs(id);
|
||||||
|
EOF
|
||||||
|
|
||||||
|
execute_sql_file /tmp/generate_devices.sql > /dev/null
|
||||||
|
echo -e "${GREEN}✓ Generated $NUM_DEVICES devices${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
echo -e "${YELLOW}[6/9] Generating Cards...${NC}"
|
||||||
|
cat > /tmp/generate_cards.sql << 'EOF'
|
||||||
|
-- Generate cards (1-2 cards per account on average)
|
||||||
|
INSERT INTO cards (
|
||||||
|
account_id, card_number_hash, card_last_four, card_type, card_network,
|
||||||
|
issue_date, expiry_date, cvv_hash, status, daily_limit, monthly_limit,
|
||||||
|
is_contactless, is_international
|
||||||
|
)
|
||||||
|
SELECT
|
||||||
|
((gs.id - 1) % 150000) + 1 AS account_id,
|
||||||
|
encode(digest('CARD' || gs.id::TEXT, 'sha256'), 'hex') AS card_number_hash,
|
||||||
|
LPAD((gs.id % 10000)::TEXT, 4, '0') AS card_last_four,
|
||||||
|
CASE (gs.id % 4)
|
||||||
|
WHEN 0 THEN 'DEBIT'
|
||||||
|
WHEN 1 THEN 'CREDIT'
|
||||||
|
WHEN 2 THEN 'DEBIT'
|
||||||
|
ELSE 'CREDIT'
|
||||||
|
END AS card_type,
|
||||||
|
CASE (gs.id % 4)
|
||||||
|
WHEN 0 THEN 'VISA'
|
||||||
|
WHEN 1 THEN 'MASTERCARD'
|
||||||
|
WHEN 2 THEN 'AMEX'
|
||||||
|
ELSE 'DISCOVER'
|
||||||
|
END AS card_network,
|
||||||
|
DATE '2020-01-01' + (random() * 1000)::INT AS issue_date,
|
||||||
|
DATE '2025-01-01' + (random() * 1825)::INT AS expiry_date,
|
||||||
|
encode(digest('CVV' || gs.id::TEXT, 'sha256'), 'hex') AS cvv_hash,
|
||||||
|
CASE
|
||||||
|
WHEN random() < 0.95 THEN 'ACTIVE'
|
||||||
|
WHEN random() < 0.97 THEN 'BLOCKED'
|
||||||
|
WHEN random() < 0.99 THEN 'LOST'
|
||||||
|
ELSE 'STOLEN'
|
||||||
|
END AS status,
|
||||||
|
(1000 + random() * 9000)::DECIMAL(10,2) AS daily_limit,
|
||||||
|
(10000 + random() * 90000)::DECIMAL(12,2) AS monthly_limit,
|
||||||
|
random() < 0.90 AS is_contactless,
|
||||||
|
random() < 0.30 AS is_international
|
||||||
|
FROM generate_series(1, 200000) AS gs(id);
|
||||||
|
EOF
|
||||||
|
|
||||||
|
execute_sql_file /tmp/generate_cards.sql > /dev/null
|
||||||
|
echo -e "${GREEN}✓ Generated $NUM_CARDS cards${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
echo -e "${YELLOW}[7/9] Generating Login Sessions...${NC}"
|
||||||
|
cat > /tmp/generate_sessions.sql << 'EOF'
|
||||||
|
-- Generate login sessions with realistic geographic data
|
||||||
|
INSERT INTO login_sessions (
|
||||||
|
customer_id, device_id, ip_address, country_id, city,
|
||||||
|
login_timestamp, logout_timestamp, session_duration_seconds,
|
||||||
|
is_successful, risk_score
|
||||||
|
)
|
||||||
|
SELECT
|
||||||
|
((gs.id - 1) % 100000) + 1 AS customer_id,
|
||||||
|
((gs.id - 1) % 75000) + 1 AS device_id,
|
||||||
|
('192.168.' || ((gs.id % 255) + 1) || '.' || ((gs.id % 255) + 1))::INET AS ip_address,
|
||||||
|
CASE
|
||||||
|
WHEN random() < 0.85 THEN 1
|
||||||
|
ELSE ((gs.id % 40) + 1)
|
||||||
|
END AS country_id,
|
||||||
|
-- Use real cities
|
||||||
|
CASE
|
||||||
|
WHEN random() < 0.85 THEN (SELECT city FROM temp_us_cities WHERE id = ((gs.id % 100) + 1))
|
||||||
|
ELSE (SELECT city FROM temp_world_cities WHERE id = ((gs.id % 210) + 1))
|
||||||
|
END AS city,
|
||||||
|
TIMESTAMP '2023-01-01' + (random() * 730)::INT * INTERVAL '1 day' + (random() * 86400)::INT * INTERVAL '1 second' AS login_timestamp,
|
||||||
|
TIMESTAMP '2023-01-01' + (random() * 730)::INT * INTERVAL '1 day' + (random() * 86400)::INT * INTERVAL '1 second' + (random() * 7200)::INT * INTERVAL '1 second' AS logout_timestamp,
|
||||||
|
(300 + random() * 7200)::INT AS session_duration_seconds,
|
||||||
|
random() < 0.98 AS is_successful,
|
||||||
|
(random() * 100)::DECIMAL(5,2) AS risk_score
|
||||||
|
FROM generate_series(1, 500000) AS gs(id);
|
||||||
|
EOF
|
||||||
|
|
||||||
|
execute_sql_file /tmp/generate_sessions.sql > /dev/null
|
||||||
|
echo -e "${GREEN}✓ Generated 500,000 login sessions${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
echo -e "${YELLOW}[8/9] Generating Transactions (this may take a while)...${NC}"
|
||||||
|
echo -e "${BLUE}This step generates $NUM_TRANSACTIONS transactions with fraud patterns${NC}"
|
||||||
|
|
||||||
|
# Generate transactions in batches to avoid memory issues
|
||||||
|
BATCH_SIZE=500000
|
||||||
|
NUM_BATCHES=$((NUM_TRANSACTIONS / BATCH_SIZE))
|
||||||
|
|
||||||
|
for batch in $(seq 1 $NUM_BATCHES); do
|
||||||
|
START_ID=$(( (batch - 1) * BATCH_SIZE + 1 ))
|
||||||
|
END_ID=$(( batch * BATCH_SIZE ))
|
||||||
|
|
||||||
|
echo -e "${BLUE} Batch $batch/$NUM_BATCHES (transactions $START_ID to $END_ID)...${NC}"
|
||||||
|
|
||||||
|
cat > /tmp/generate_transactions_batch.sql << EOF
|
||||||
|
-- Generate transactions batch
|
||||||
|
INSERT INTO transactions (
|
||||||
|
account_id, type_id, transaction_date, amount, currency,
|
||||||
|
merchant_id, card_id, device_id, ip_address, country_id, city,
|
||||||
|
description, reference_number, status, is_online, is_international,
|
||||||
|
is_card_present, fraud_score, is_flagged
|
||||||
|
)
|
||||||
|
SELECT
|
||||||
|
((gs.id - 1) % 150000) + 1 AS account_id,
|
||||||
|
((gs.id % 15) + 1) AS type_id,
|
||||||
|
TIMESTAMP '2023-01-01' + (random() * 730)::INT * INTERVAL '1 day' + (random() * 86400)::INT * INTERVAL '1 second' AS transaction_date,
|
||||||
|
CASE
|
||||||
|
WHEN random() < 0.60 THEN (5 + random() * 95)::DECIMAL(15,2)
|
||||||
|
WHEN random() < 0.85 THEN (100 + random() * 400)::DECIMAL(15,2)
|
||||||
|
WHEN random() < 0.95 THEN (500 + random() * 2000)::DECIMAL(15,2)
|
||||||
|
WHEN random() < 0.98 THEN (2500 + random() * 7500)::DECIMAL(15,2)
|
||||||
|
ELSE (10000 + random() * 90000)::DECIMAL(15,2)
|
||||||
|
END AS amount,
|
||||||
|
'USD' AS currency,
|
||||||
|
CASE
|
||||||
|
WHEN ((gs.id % 15) + 1) IN (1, 7, 13) THEN ((gs.id % 50000) + 1)
|
||||||
|
ELSE NULL
|
||||||
|
END AS merchant_id,
|
||||||
|
CASE
|
||||||
|
WHEN ((gs.id % 15) + 1) IN (1, 7, 13) THEN ((gs.id % 200000) + 1)
|
||||||
|
ELSE NULL
|
||||||
|
END AS card_id,
|
||||||
|
((gs.id % 75000) + 1) AS device_id,
|
||||||
|
('10.' || ((gs.id % 255) + 1) || '.' || ((gs.id % 255) + 1) || '.' || ((gs.id % 255) + 1))::INET AS ip_address,
|
||||||
|
CASE
|
||||||
|
WHEN random() < 0.90 THEN 1
|
||||||
|
ELSE ((gs.id % 40) + 1)
|
||||||
|
END AS country_id,
|
||||||
|
-- Use real cities based on country
|
||||||
|
CASE
|
||||||
|
WHEN random() < 0.90 THEN (SELECT city FROM temp_us_cities WHERE id = ((gs.id % 100) + 1))
|
||||||
|
ELSE (SELECT city FROM temp_world_cities WHERE id = ((gs.id % 210) + 1))
|
||||||
|
END AS city,
|
||||||
|
'Transaction #' || gs.id AS description,
|
||||||
|
'REF' || LPAD(gs.id::TEXT, 15, '0') AS reference_number,
|
||||||
|
CASE
|
||||||
|
WHEN random() < 0.95 THEN 'COMPLETED'
|
||||||
|
WHEN random() < 0.98 THEN 'PENDING'
|
||||||
|
ELSE 'FAILED'
|
||||||
|
END AS status,
|
||||||
|
random() < 0.70 AS is_online,
|
||||||
|
random() < 0.10 AS is_international,
|
||||||
|
random() < 0.30 AS is_card_present,
|
||||||
|
(random() * 100)::DECIMAL(5,2) AS fraud_score,
|
||||||
|
random() < 0.07 AS is_flagged
|
||||||
|
FROM generate_series($START_ID, $END_ID) AS gs(id);
|
||||||
|
EOF
|
||||||
|
|
||||||
|
execute_sql_file /tmp/generate_transactions_batch.sql > /dev/null
|
||||||
|
echo -e "${GREEN} ✓ Batch $batch/$NUM_BATCHES completed${NC}"
|
||||||
|
done
|
||||||
|
|
||||||
|
echo -e "${GREEN}✓ Generated $NUM_TRANSACTIONS transactions${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
echo -e "${YELLOW}[9/9] Generating Fraud Cases and Alerts...${NC}"
|
||||||
|
|
||||||
|
# Generate alerts for flagged transactions
|
||||||
|
execute_sql "
|
||||||
|
INSERT INTO alerts (transaction_id, customer_id, account_id, alert_type, severity, description, risk_score, status)
|
||||||
|
SELECT
|
||||||
|
t.transaction_id,
|
||||||
|
a.customer_id,
|
||||||
|
t.account_id,
|
||||||
|
CASE
|
||||||
|
WHEN t.amount > 10000 THEN 'AMOUNT_ANOMALY'
|
||||||
|
WHEN t.is_international THEN 'GEOGRAPHIC_ANOMALY'
|
||||||
|
WHEN t.fraud_score > 80 THEN 'MERCHANT_RISK'
|
||||||
|
ELSE 'VELOCITY_CHECK'
|
||||||
|
END AS alert_type,
|
||||||
|
CASE
|
||||||
|
WHEN t.fraud_score > 80 THEN 'CRITICAL'
|
||||||
|
WHEN t.fraud_score > 60 THEN 'HIGH'
|
||||||
|
ELSE 'MEDIUM'
|
||||||
|
END AS severity,
|
||||||
|
'Suspicious transaction detected: ' || t.description AS description,
|
||||||
|
t.fraud_score,
|
||||||
|
CASE
|
||||||
|
WHEN random() < 0.30 THEN 'CLOSED'
|
||||||
|
WHEN random() < 0.50 THEN 'FALSE_POSITIVE'
|
||||||
|
WHEN random() < 0.70 THEN 'INVESTIGATING'
|
||||||
|
ELSE 'OPEN'
|
||||||
|
END AS status
|
||||||
|
FROM transactions t
|
||||||
|
JOIN accounts a ON t.account_id = a.account_id
|
||||||
|
WHERE t.is_flagged = TRUE
|
||||||
|
LIMIT 50000;
|
||||||
|
" > /dev/null
|
||||||
|
|
||||||
|
echo -e "${GREEN}✓ Generated alerts for flagged transactions${NC}"
|
||||||
|
|
||||||
|
# Generate fraud cases
|
||||||
|
execute_sql "
|
||||||
|
INSERT INTO fraud_cases (
|
||||||
|
case_number, customer_id, account_id, fraud_type_id,
|
||||||
|
detection_date, detection_method, amount_lost, status, priority
|
||||||
|
)
|
||||||
|
SELECT
|
||||||
|
'CASE' || LPAD(ROW_NUMBER() OVER (ORDER BY a.alert_id)::TEXT, 10, '0') AS case_number,
|
||||||
|
a.customer_id,
|
||||||
|
a.account_id,
|
||||||
|
((a.alert_id % 20) + 1) AS fraud_type_id,
|
||||||
|
a.alert_date AS detection_date,
|
||||||
|
CASE
|
||||||
|
WHEN random() < 0.70 THEN 'AUTOMATED'
|
||||||
|
WHEN random() < 0.85 THEN 'MANUAL_REVIEW'
|
||||||
|
ELSE 'CUSTOMER_REPORT'
|
||||||
|
END AS detection_method,
|
||||||
|
(random() * 50000)::DECIMAL(15,2) AS amount_lost,
|
||||||
|
CASE
|
||||||
|
WHEN random() < 0.40 THEN 'RESOLVED'
|
||||||
|
WHEN random() < 0.60 THEN 'INVESTIGATING'
|
||||||
|
ELSE 'OPEN'
|
||||||
|
END AS status,
|
||||||
|
CASE
|
||||||
|
WHEN a.severity = 'CRITICAL' THEN 'CRITICAL'
|
||||||
|
WHEN a.severity = 'HIGH' THEN 'HIGH'
|
||||||
|
ELSE 'MEDIUM'
|
||||||
|
END AS priority
|
||||||
|
FROM alerts a
|
||||||
|
WHERE a.status = 'CONFIRMED_FRAUD'
|
||||||
|
OR (a.severity IN ('CRITICAL', 'HIGH') AND random() < 0.20)
|
||||||
|
LIMIT 5000;
|
||||||
|
" > /dev/null
|
||||||
|
|
||||||
|
echo -e "${GREEN}✓ Generated fraud cases${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Final statistics
|
||||||
|
echo -e "${BLUE}============================================================================${NC}"
|
||||||
|
echo -e "${GREEN}✓ Data generation completed successfully!${NC}"
|
||||||
|
echo -e "${BLUE}============================================================================${NC}"
|
||||||
|
echo ""
|
||||||
|
echo -e "${YELLOW}Database Statistics:${NC}"
|
||||||
|
|
||||||
|
customer_count=$(execute_sql "SELECT COUNT(*) FROM customers;" | grep -E '^\s*[0-9]+' | tr -d ' ')
|
||||||
|
echo -e "Customers: ${GREEN}$customer_count${NC}"
|
||||||
|
|
||||||
|
account_count=$(execute_sql "SELECT COUNT(*) FROM accounts;" | grep -E '^\s*[0-9]+' | tr -d ' ')
|
||||||
|
echo -e "Accounts: ${GREEN}$account_count${NC}"
|
||||||
|
|
||||||
|
merchant_count=$(execute_sql "SELECT COUNT(*) FROM merchants;" | grep -E '^\s*[0-9]+' | tr -d ' ')
|
||||||
|
echo -e "Merchants: ${GREEN}$merchant_count${NC}"
|
||||||
|
|
||||||
|
card_count=$(execute_sql "SELECT COUNT(*) FROM cards;" | grep -E '^\s*[0-9]+' | tr -d ' ')
|
||||||
|
echo -e "Cards: ${GREEN}$card_count${NC}"
|
||||||
|
|
||||||
|
transaction_count=$(execute_sql "SELECT COUNT(*) FROM transactions;" | grep -E '^\s*[0-9]+' | tr -d ' ')
|
||||||
|
echo -e "Transactions: ${GREEN}$transaction_count${NC}"
|
||||||
|
|
||||||
|
alert_count=$(execute_sql "SELECT COUNT(*) FROM alerts;" | grep -E '^\s*[0-9]+' | tr -d ' ')
|
||||||
|
echo -e "Alerts: ${GREEN}$alert_count${NC}"
|
||||||
|
|
||||||
|
case_count=$(execute_sql "SELECT COUNT(*) FROM fraud_cases;" | grep -E '^\s*[0-9]+' | tr -d ' ')
|
||||||
|
echo -e "Fraud Cases: ${GREEN}$case_count${NC}"
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo -e "${YELLOW}Ready for SQL learning and fraud investigation!${NC}"
|
||||||
|
echo -e "Access DB-UI at: ${BLUE}http://localhost:3000${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
@@ -0,0 +1,329 @@
|
|||||||
|
-- ============================================================================
|
||||||
|
-- Load Geographic Reference Data into Temporary Tables
|
||||||
|
-- ============================================================================
|
||||||
|
-- This script creates temporary tables with realistic city/state/country data
|
||||||
|
-- to be used during data generation
|
||||||
|
-- ============================================================================
|
||||||
|
|
||||||
|
-- Drop temporary tables if they exist
|
||||||
|
DROP TABLE IF EXISTS temp_us_cities CASCADE;
|
||||||
|
DROP TABLE IF EXISTS temp_world_cities CASCADE;
|
||||||
|
|
||||||
|
-- Create temporary table for US cities
|
||||||
|
CREATE TEMPORARY TABLE temp_us_cities (
|
||||||
|
id SERIAL PRIMARY KEY,
|
||||||
|
city VARCHAR(100) NOT NULL,
|
||||||
|
state VARCHAR(100) NOT NULL,
|
||||||
|
state_code CHAR(2) NOT NULL
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Create temporary table for world cities
|
||||||
|
CREATE TEMPORARY TABLE temp_world_cities (
|
||||||
|
id SERIAL PRIMARY KEY,
|
||||||
|
city VARCHAR(100) NOT NULL,
|
||||||
|
country VARCHAR(100) NOT NULL,
|
||||||
|
country_code CHAR(2) NOT NULL
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Load US cities data
|
||||||
|
COPY temp_us_cities(city, state, state_code) FROM STDIN WITH (FORMAT CSV, HEADER true);
|
||||||
|
New York,New York,NY
|
||||||
|
Los Angeles,California,CA
|
||||||
|
Chicago,Illinois,IL
|
||||||
|
Houston,Texas,TX
|
||||||
|
Phoenix,Arizona,AZ
|
||||||
|
Philadelphia,Pennsylvania,PA
|
||||||
|
San Antonio,Texas,TX
|
||||||
|
San Diego,California,CA
|
||||||
|
Dallas,Texas,TX
|
||||||
|
San Jose,California,CA
|
||||||
|
Austin,Texas,TX
|
||||||
|
Jacksonville,Florida,FL
|
||||||
|
Fort Worth,Texas,TX
|
||||||
|
Columbus,Ohio,OH
|
||||||
|
Charlotte,North Carolina,NC
|
||||||
|
San Francisco,California,CA
|
||||||
|
Indianapolis,Indiana,IN
|
||||||
|
Seattle,Washington,WA
|
||||||
|
Denver,Colorado,CO
|
||||||
|
Boston,Massachusetts,MA
|
||||||
|
Nashville,Tennessee,TN
|
||||||
|
Detroit,Michigan,MI
|
||||||
|
Portland,Oregon,OR
|
||||||
|
Las Vegas,Nevada,NV
|
||||||
|
Memphis,Tennessee,TN
|
||||||
|
Louisville,Kentucky,KY
|
||||||
|
Baltimore,Maryland,MD
|
||||||
|
Milwaukee,Wisconsin,WI
|
||||||
|
Albuquerque,New Mexico,NM
|
||||||
|
Tucson,Arizona,AZ
|
||||||
|
Fresno,California,CA
|
||||||
|
Sacramento,California,CA
|
||||||
|
Kansas City,Missouri,MO
|
||||||
|
Mesa,Arizona,AZ
|
||||||
|
Atlanta,Georgia,GA
|
||||||
|
Omaha,Nebraska,NE
|
||||||
|
Colorado Springs,Colorado,CO
|
||||||
|
Raleigh,North Carolina,NC
|
||||||
|
Miami,Florida,FL
|
||||||
|
Long Beach,California,CA
|
||||||
|
Virginia Beach,Virginia,VA
|
||||||
|
Oakland,California,CA
|
||||||
|
Minneapolis,Minnesota,MN
|
||||||
|
Tampa,Florida,FL
|
||||||
|
Tulsa,Oklahoma,OK
|
||||||
|
Arlington,Texas,TX
|
||||||
|
New Orleans,Louisiana,LA
|
||||||
|
Wichita,Kansas,KS
|
||||||
|
Cleveland,Ohio,OH
|
||||||
|
Bakersfield,California,CA
|
||||||
|
Aurora,Colorado,CO
|
||||||
|
Anaheim,California,CA
|
||||||
|
Honolulu,Hawaii,HI
|
||||||
|
Santa Ana,California,CA
|
||||||
|
Riverside,California,CA
|
||||||
|
Corpus Christi,Texas,TX
|
||||||
|
Lexington,Kentucky,KY
|
||||||
|
Stockton,California,CA
|
||||||
|
Henderson,Nevada,NV
|
||||||
|
Saint Paul,Minnesota,MN
|
||||||
|
St. Louis,Missouri,MO
|
||||||
|
Cincinnati,Ohio,OH
|
||||||
|
Pittsburgh,Pennsylvania,PA
|
||||||
|
Greensboro,North Carolina,NC
|
||||||
|
Anchorage,Alaska,AK
|
||||||
|
Plano,Texas,TX
|
||||||
|
Lincoln,Nebraska,NE
|
||||||
|
Orlando,Florida,FL
|
||||||
|
Irvine,California,CA
|
||||||
|
Newark,New Jersey,NJ
|
||||||
|
Durham,North Carolina,NC
|
||||||
|
Chula Vista,California,CA
|
||||||
|
Toledo,Ohio,OH
|
||||||
|
Fort Wayne,Indiana,IN
|
||||||
|
St. Petersburg,Florida,FL
|
||||||
|
Laredo,Texas,TX
|
||||||
|
Jersey City,New Jersey,NJ
|
||||||
|
Chandler,Arizona,AZ
|
||||||
|
Madison,Wisconsin,WI
|
||||||
|
Lubbock,Texas,TX
|
||||||
|
Scottsdale,Arizona,AZ
|
||||||
|
Reno,Nevada,NV
|
||||||
|
Buffalo,New York,NY
|
||||||
|
Gilbert,Arizona,AZ
|
||||||
|
Glendale,Arizona,AZ
|
||||||
|
North Las Vegas,Nevada,NV
|
||||||
|
Winston-Salem,North Carolina,NC
|
||||||
|
Chesapeake,Virginia,VA
|
||||||
|
Norfolk,Virginia,VA
|
||||||
|
Fremont,California,CA
|
||||||
|
Garland,Texas,TX
|
||||||
|
Irving,Texas,TX
|
||||||
|
Hialeah,Florida,FL
|
||||||
|
Richmond,Virginia,VA
|
||||||
|
Boise,Idaho,ID
|
||||||
|
Spokane,Washington,WA
|
||||||
|
Baton Rouge,Louisiana,LA
|
||||||
|
\.
|
||||||
|
|
||||||
|
-- Load world cities data
|
||||||
|
COPY temp_world_cities(city, country, country_code) FROM STDIN WITH (FORMAT CSV, HEADER true);
|
||||||
|
Toronto,Canada,CA
|
||||||
|
Vancouver,Canada,CA
|
||||||
|
Montreal,Canada,CA
|
||||||
|
Calgary,Canada,CA
|
||||||
|
Ottawa,Canada,CA
|
||||||
|
London,United Kingdom,GB
|
||||||
|
Manchester,United Kingdom,GB
|
||||||
|
Birmingham,United Kingdom,GB
|
||||||
|
Edinburgh,United Kingdom,GB
|
||||||
|
Glasgow,United Kingdom,GB
|
||||||
|
Berlin,Germany,DE
|
||||||
|
Munich,Germany,DE
|
||||||
|
Hamburg,Germany,DE
|
||||||
|
Frankfurt,Germany,DE
|
||||||
|
Cologne,Germany,DE
|
||||||
|
Paris,France,FR
|
||||||
|
Lyon,France,FR
|
||||||
|
Marseille,France,FR
|
||||||
|
Toulouse,France,FR
|
||||||
|
Nice,France,FR
|
||||||
|
Rome,Italy,IT
|
||||||
|
Milan,Italy,IT
|
||||||
|
Naples,Italy,IT
|
||||||
|
Turin,Italy,IT
|
||||||
|
Florence,Italy,IT
|
||||||
|
Madrid,Spain,ES
|
||||||
|
Barcelona,Spain,ES
|
||||||
|
Valencia,Spain,ES
|
||||||
|
Seville,Spain,ES
|
||||||
|
Bilbao,Spain,ES
|
||||||
|
Sydney,Australia,AU
|
||||||
|
Melbourne,Australia,AU
|
||||||
|
Brisbane,Australia,AU
|
||||||
|
Perth,Australia,AU
|
||||||
|
Adelaide,Australia,AU
|
||||||
|
Tokyo,Japan,JP
|
||||||
|
Osaka,Japan,JP
|
||||||
|
Kyoto,Japan,JP
|
||||||
|
Yokohama,Japan,JP
|
||||||
|
Nagoya,Japan,JP
|
||||||
|
Beijing,China,CN
|
||||||
|
Shanghai,China,CN
|
||||||
|
Guangzhou,China,CN
|
||||||
|
Shenzhen,China,CN
|
||||||
|
Chengdu,China,CN
|
||||||
|
Mumbai,India,IN
|
||||||
|
Delhi,India,IN
|
||||||
|
Bangalore,India,IN
|
||||||
|
Hyderabad,India,IN
|
||||||
|
Chennai,India,IN
|
||||||
|
Sao Paulo,Brazil,BR
|
||||||
|
Rio de Janeiro,Brazil,BR
|
||||||
|
Brasilia,Brazil,BR
|
||||||
|
Salvador,Brazil,BR
|
||||||
|
Fortaleza,Brazil,BR
|
||||||
|
Mexico City,Mexico,MX
|
||||||
|
Guadalajara,Mexico,MX
|
||||||
|
Monterrey,Mexico,MX
|
||||||
|
Puebla,Mexico,MX
|
||||||
|
Tijuana,Mexico,MX
|
||||||
|
Moscow,Russia,RU
|
||||||
|
Saint Petersburg,Russia,RU
|
||||||
|
Novosibirsk,Russia,RU
|
||||||
|
Yekaterinburg,Russia,RU
|
||||||
|
Kazan,Russia,RU
|
||||||
|
Lagos,Nigeria,NG
|
||||||
|
Kano,Nigeria,NG
|
||||||
|
Ibadan,Nigeria,NG
|
||||||
|
Abuja,Nigeria,NG
|
||||||
|
Port Harcourt,Nigeria,NG
|
||||||
|
Karachi,Pakistan,PK
|
||||||
|
Lahore,Pakistan,PK
|
||||||
|
Islamabad,Pakistan,PK
|
||||||
|
Rawalpindi,Pakistan,PK
|
||||||
|
Faisalabad,Pakistan,PK
|
||||||
|
Tehran,Iran,IR
|
||||||
|
Mashhad,Iran,IR
|
||||||
|
Isfahan,Iran,IR
|
||||||
|
Karaj,Iran,IR
|
||||||
|
Tabriz,Iran,IR
|
||||||
|
Pyongyang,North Korea,KP
|
||||||
|
Hamhung,North Korea,KP
|
||||||
|
Chongjin,North Korea,KP
|
||||||
|
Nampo,North Korea,KP
|
||||||
|
Wonsan,North Korea,KP
|
||||||
|
Damascus,Syria,SY
|
||||||
|
Aleppo,Syria,SY
|
||||||
|
Homs,Syria,SY
|
||||||
|
Latakia,Syria,SY
|
||||||
|
Hama,Syria,SY
|
||||||
|
Caracas,Venezuela,VE
|
||||||
|
Maracaibo,Venezuela,VE
|
||||||
|
Valencia,Venezuela,VE
|
||||||
|
Barquisimeto,Venezuela,VE
|
||||||
|
Maracay,Venezuela,VE
|
||||||
|
Havana,Cuba,CU
|
||||||
|
Santiago de Cuba,Cuba,CU
|
||||||
|
Camaguey,Cuba,CU
|
||||||
|
Holguin,Cuba,CU
|
||||||
|
Santa Clara,Cuba,CU
|
||||||
|
Yangon,Myanmar,MM
|
||||||
|
Mandalay,Myanmar,MM
|
||||||
|
Naypyidaw,Myanmar,MM
|
||||||
|
Mawlamyine,Myanmar,MM
|
||||||
|
Bago,Myanmar,MM
|
||||||
|
Kabul,Afghanistan,AF
|
||||||
|
Kandahar,Afghanistan,AF
|
||||||
|
Herat,Afghanistan,AF
|
||||||
|
Mazar-i-Sharif,Afghanistan,AF
|
||||||
|
Jalalabad,Afghanistan,AF
|
||||||
|
Baghdad,Iraq,IQ
|
||||||
|
Basra,Iraq,IQ
|
||||||
|
Mosul,Iraq,IQ
|
||||||
|
Erbil,Iraq,IQ
|
||||||
|
Kirkuk,Iraq,IQ
|
||||||
|
Tripoli,Libya,LY
|
||||||
|
Benghazi,Libya,LY
|
||||||
|
Misrata,Libya,LY
|
||||||
|
Zawiya,Libya,LY
|
||||||
|
Bayda,Libya,LY
|
||||||
|
Khartoum,Sudan,SD
|
||||||
|
Omdurman,Sudan,SD
|
||||||
|
Port Sudan,Sudan,SD
|
||||||
|
Kassala,Sudan,SD
|
||||||
|
Nyala,Sudan,SD
|
||||||
|
Mogadishu,Somalia,SO
|
||||||
|
Hargeisa,Somalia,SO
|
||||||
|
Bosaso,Somalia,SO
|
||||||
|
Kismayo,Somalia,SO
|
||||||
|
Merca,Somalia,SO
|
||||||
|
Sanaa,Yemen,YE
|
||||||
|
Aden,Yemen,YE
|
||||||
|
Taiz,Yemen,YE
|
||||||
|
Hodeidah,Yemen,YE
|
||||||
|
Ibb,Yemen,YE
|
||||||
|
Harare,Zimbabwe,ZW
|
||||||
|
Bulawayo,Zimbabwe,ZW
|
||||||
|
Chitungwiza,Zimbabwe,ZW
|
||||||
|
Mutare,Zimbabwe,ZW
|
||||||
|
Gweru,Zimbabwe,ZW
|
||||||
|
Amsterdam,Netherlands,NL
|
||||||
|
Rotterdam,Netherlands,NL
|
||||||
|
The Hague,Netherlands,NL
|
||||||
|
Utrecht,Netherlands,NL
|
||||||
|
Eindhoven,Netherlands,NL
|
||||||
|
Stockholm,Sweden,SE
|
||||||
|
Gothenburg,Sweden,SE
|
||||||
|
Malmo,Sweden,SE
|
||||||
|
Uppsala,Sweden,SE
|
||||||
|
Vasteras,Sweden,SE
|
||||||
|
Oslo,Norway,NO
|
||||||
|
Bergen,Norway,NO
|
||||||
|
Trondheim,Norway,NO
|
||||||
|
Stavanger,Norway,NO
|
||||||
|
Drammen,Norway,NO
|
||||||
|
Copenhagen,Denmark,DK
|
||||||
|
Aarhus,Denmark,DK
|
||||||
|
Odense,Denmark,DK
|
||||||
|
Aalborg,Denmark,DK
|
||||||
|
Esbjerg,Denmark,DK
|
||||||
|
Helsinki,Finland,FI
|
||||||
|
Espoo,Finland,FI
|
||||||
|
Tampere,Finland,FI
|
||||||
|
Vantaa,Finland,FI
|
||||||
|
Oulu,Finland,FI
|
||||||
|
Zurich,Switzerland,CH
|
||||||
|
Geneva,Switzerland,CH
|
||||||
|
Basel,Switzerland,CH
|
||||||
|
Lausanne,Switzerland,CH
|
||||||
|
Bern,Switzerland,CH
|
||||||
|
Singapore,Singapore,SG
|
||||||
|
Hong Kong,Hong Kong,HK
|
||||||
|
Kowloon,Hong Kong,HK
|
||||||
|
Seoul,South Korea,KR
|
||||||
|
Busan,South Korea,KR
|
||||||
|
Incheon,South Korea,KR
|
||||||
|
Daegu,South Korea,KR
|
||||||
|
Daejeon,South Korea,KR
|
||||||
|
Taipei,Taiwan,TW
|
||||||
|
Kaohsiung,Taiwan,TW
|
||||||
|
Taichung,Taiwan,TW
|
||||||
|
Tainan,Taiwan,TW
|
||||||
|
Hsinchu,Taiwan,TW
|
||||||
|
Auckland,New Zealand,NZ
|
||||||
|
Wellington,New Zealand,NZ
|
||||||
|
Christchurch,New Zealand,NZ
|
||||||
|
Hamilton,New Zealand,NZ
|
||||||
|
Tauranga,New Zealand,NZ
|
||||||
|
\.
|
||||||
|
|
||||||
|
-- Create indexes for faster lookups
|
||||||
|
CREATE INDEX idx_temp_us_cities_id ON temp_us_cities(id);
|
||||||
|
CREATE INDEX idx_temp_world_cities_id ON temp_world_cities(id);
|
||||||
|
CREATE INDEX idx_temp_world_cities_country_code ON temp_world_cities(country_code);
|
||||||
|
|
||||||
|
-- Show counts
|
||||||
|
SELECT 'US Cities loaded: ' || COUNT(*) FROM temp_us_cities;
|
||||||
|
SELECT 'World Cities loaded: ' || COUNT(*) FROM temp_world_cities;
|
||||||
|
|
||||||
@@ -0,0 +1,99 @@
|
|||||||
|
city,state,state_code
|
||||||
|
New York,New York,NY
|
||||||
|
Los Angeles,California,CA
|
||||||
|
Chicago,Illinois,IL
|
||||||
|
Houston,Texas,TX
|
||||||
|
Phoenix,Arizona,AZ
|
||||||
|
Philadelphia,Pennsylvania,PA
|
||||||
|
San Antonio,Texas,TX
|
||||||
|
San Diego,California,CA
|
||||||
|
Dallas,Texas,TX
|
||||||
|
San Jose,California,CA
|
||||||
|
Austin,Texas,TX
|
||||||
|
Jacksonville,Florida,FL
|
||||||
|
Fort Worth,Texas,TX
|
||||||
|
Columbus,Ohio,OH
|
||||||
|
Charlotte,North Carolina,NC
|
||||||
|
San Francisco,California,CA
|
||||||
|
Indianapolis,Indiana,IN
|
||||||
|
Seattle,Washington,WA
|
||||||
|
Denver,Colorado,CO
|
||||||
|
Boston,Massachusetts,MA
|
||||||
|
Nashville,Tennessee,TN
|
||||||
|
Detroit,Michigan,MI
|
||||||
|
Portland,Oregon,OR
|
||||||
|
Las Vegas,Nevada,NV
|
||||||
|
Memphis,Tennessee,TN
|
||||||
|
Louisville,Kentucky,KY
|
||||||
|
Baltimore,Maryland,MD
|
||||||
|
Milwaukee,Wisconsin,WI
|
||||||
|
Albuquerque,New Mexico,NM
|
||||||
|
Tucson,Arizona,AZ
|
||||||
|
Fresno,California,CA
|
||||||
|
Sacramento,California,CA
|
||||||
|
Kansas City,Missouri,MO
|
||||||
|
Mesa,Arizona,AZ
|
||||||
|
Atlanta,Georgia,GA
|
||||||
|
Omaha,Nebraska,NE
|
||||||
|
Colorado Springs,Colorado,CO
|
||||||
|
Raleigh,North Carolina,NC
|
||||||
|
Miami,Florida,FL
|
||||||
|
Long Beach,California,CA
|
||||||
|
Virginia Beach,Virginia,VA
|
||||||
|
Oakland,California,CA
|
||||||
|
Minneapolis,Minnesota,MN
|
||||||
|
Tampa,Florida,FL
|
||||||
|
Tulsa,Oklahoma,OK
|
||||||
|
Arlington,Texas,TX
|
||||||
|
New Orleans,Louisiana,LA
|
||||||
|
Wichita,Kansas,KS
|
||||||
|
Cleveland,Ohio,OH
|
||||||
|
Bakersfield,California,CA
|
||||||
|
Aurora,Colorado,CO
|
||||||
|
Anaheim,California,CA
|
||||||
|
Honolulu,Hawaii,HI
|
||||||
|
Santa Ana,California,CA
|
||||||
|
Riverside,California,CA
|
||||||
|
Corpus Christi,Texas,TX
|
||||||
|
Lexington,Kentucky,KY
|
||||||
|
Stockton,California,CA
|
||||||
|
Henderson,Nevada,NV
|
||||||
|
Saint Paul,Minnesota,MN
|
||||||
|
St. Louis,Missouri,MO
|
||||||
|
Cincinnati,Ohio,OH
|
||||||
|
Pittsburgh,Pennsylvania,PA
|
||||||
|
Greensboro,North Carolina,NC
|
||||||
|
Anchorage,Alaska,AK
|
||||||
|
Plano,Texas,TX
|
||||||
|
Lincoln,Nebraska,NE
|
||||||
|
Orlando,Florida,FL
|
||||||
|
Irvine,California,CA
|
||||||
|
Newark,New Jersey,NJ
|
||||||
|
Durham,North Carolina,NC
|
||||||
|
Chula Vista,California,CA
|
||||||
|
Toledo,Ohio,OH
|
||||||
|
Fort Wayne,Indiana,IN
|
||||||
|
St. Petersburg,Florida,FL
|
||||||
|
Laredo,Texas,TX
|
||||||
|
Jersey City,New Jersey,NJ
|
||||||
|
Chandler,Arizona,AZ
|
||||||
|
Madison,Wisconsin,WI
|
||||||
|
Lubbock,Texas,TX
|
||||||
|
Scottsdale,Arizona,AZ
|
||||||
|
Reno,Nevada,NV
|
||||||
|
Buffalo,New York,NY
|
||||||
|
Gilbert,Arizona,AZ
|
||||||
|
Glendale,Arizona,AZ
|
||||||
|
North Las Vegas,Nevada,NV
|
||||||
|
Winston-Salem,North Carolina,NC
|
||||||
|
Chesapeake,Virginia,VA
|
||||||
|
Norfolk,Virginia,VA
|
||||||
|
Fremont,California,CA
|
||||||
|
Garland,Texas,TX
|
||||||
|
Irving,Texas,TX
|
||||||
|
Hialeah,Florida,FL
|
||||||
|
Richmond,Virginia,VA
|
||||||
|
Boise,Idaho,ID
|
||||||
|
Spokane,Washington,WA
|
||||||
|
Baton Rouge,Louisiana,LA
|
||||||
|
|
||||||
|
@@ -0,0 +1,190 @@
|
|||||||
|
city,country,country_code
|
||||||
|
Toronto,Canada,CA
|
||||||
|
Vancouver,Canada,CA
|
||||||
|
Montreal,Canada,CA
|
||||||
|
Calgary,Canada,CA
|
||||||
|
Ottawa,Canada,CA
|
||||||
|
London,United Kingdom,GB
|
||||||
|
Manchester,United Kingdom,GB
|
||||||
|
Birmingham,United Kingdom,GB
|
||||||
|
Edinburgh,United Kingdom,GB
|
||||||
|
Glasgow,United Kingdom,GB
|
||||||
|
Berlin,Germany,DE
|
||||||
|
Munich,Germany,DE
|
||||||
|
Hamburg,Germany,DE
|
||||||
|
Frankfurt,Germany,DE
|
||||||
|
Cologne,Germany,DE
|
||||||
|
Paris,France,FR
|
||||||
|
Lyon,France,FR
|
||||||
|
Marseille,France,FR
|
||||||
|
Toulouse,France,FR
|
||||||
|
Nice,France,FR
|
||||||
|
Rome,Italy,IT
|
||||||
|
Milan,Italy,IT
|
||||||
|
Naples,Italy,IT
|
||||||
|
Turin,Italy,IT
|
||||||
|
Florence,Italy,IT
|
||||||
|
Madrid,Spain,ES
|
||||||
|
Barcelona,Spain,ES
|
||||||
|
Valencia,Spain,ES
|
||||||
|
Seville,Spain,ES
|
||||||
|
Bilbao,Spain,ES
|
||||||
|
Sydney,Australia,AU
|
||||||
|
Melbourne,Australia,AU
|
||||||
|
Brisbane,Australia,AU
|
||||||
|
Perth,Australia,AU
|
||||||
|
Adelaide,Australia,AU
|
||||||
|
Tokyo,Japan,JP
|
||||||
|
Osaka,Japan,JP
|
||||||
|
Kyoto,Japan,JP
|
||||||
|
Yokohama,Japan,JP
|
||||||
|
Nagoya,Japan,JP
|
||||||
|
Beijing,China,CN
|
||||||
|
Shanghai,China,CN
|
||||||
|
Guangzhou,China,CN
|
||||||
|
Shenzhen,China,CN
|
||||||
|
Chengdu,China,CN
|
||||||
|
Mumbai,India,IN
|
||||||
|
Delhi,India,IN
|
||||||
|
Bangalore,India,IN
|
||||||
|
Hyderabad,India,IN
|
||||||
|
Chennai,India,IN
|
||||||
|
Sao Paulo,Brazil,BR
|
||||||
|
Rio de Janeiro,Brazil,BR
|
||||||
|
Brasilia,Brazil,BR
|
||||||
|
Salvador,Brazil,BR
|
||||||
|
Fortaleza,Brazil,BR
|
||||||
|
Mexico City,Mexico,MX
|
||||||
|
Guadalajara,Mexico,MX
|
||||||
|
Monterrey,Mexico,MX
|
||||||
|
Puebla,Mexico,MX
|
||||||
|
Tijuana,Mexico,MX
|
||||||
|
Moscow,Russia,RU
|
||||||
|
Saint Petersburg,Russia,RU
|
||||||
|
Novosibirsk,Russia,RU
|
||||||
|
Yekaterinburg,Russia,RU
|
||||||
|
Kazan,Russia,RU
|
||||||
|
Lagos,Nigeria,NG
|
||||||
|
Kano,Nigeria,NG
|
||||||
|
Ibadan,Nigeria,NG
|
||||||
|
Abuja,Nigeria,NG
|
||||||
|
Port Harcourt,Nigeria,NG
|
||||||
|
Karachi,Pakistan,PK
|
||||||
|
Lahore,Pakistan,PK
|
||||||
|
Islamabad,Pakistan,PK
|
||||||
|
Rawalpindi,Pakistan,PK
|
||||||
|
Faisalabad,Pakistan,PK
|
||||||
|
Tehran,Iran,IR
|
||||||
|
Mashhad,Iran,IR
|
||||||
|
Isfahan,Iran,IR
|
||||||
|
Karaj,Iran,IR
|
||||||
|
Tabriz,Iran,IR
|
||||||
|
Pyongyang,North Korea,KP
|
||||||
|
Hamhung,North Korea,KP
|
||||||
|
Chongjin,North Korea,KP
|
||||||
|
Nampo,North Korea,KP
|
||||||
|
Wonsan,North Korea,KP
|
||||||
|
Damascus,Syria,SY
|
||||||
|
Aleppo,Syria,SY
|
||||||
|
Homs,Syria,SY
|
||||||
|
Latakia,Syria,SY
|
||||||
|
Hama,Syria,SY
|
||||||
|
Caracas,Venezuela,VE
|
||||||
|
Maracaibo,Venezuela,VE
|
||||||
|
Valencia,Venezuela,VE
|
||||||
|
Barquisimeto,Venezuela,VE
|
||||||
|
Maracay,Venezuela,VE
|
||||||
|
Havana,Cuba,CU
|
||||||
|
Santiago de Cuba,Cuba,CU
|
||||||
|
Camaguey,Cuba,CU
|
||||||
|
Holguin,Cuba,CU
|
||||||
|
Santa Clara,Cuba,CU
|
||||||
|
Yangon,Myanmar,MM
|
||||||
|
Mandalay,Myanmar,MM
|
||||||
|
Naypyidaw,Myanmar,MM
|
||||||
|
Mawlamyine,Myanmar,MM
|
||||||
|
Bago,Myanmar,MM
|
||||||
|
Kabul,Afghanistan,AF
|
||||||
|
Kandahar,Afghanistan,AF
|
||||||
|
Herat,Afghanistan,AF
|
||||||
|
Mazar-i-Sharif,Afghanistan,AF
|
||||||
|
Jalalabad,Afghanistan,AF
|
||||||
|
Baghdad,Iraq,IQ
|
||||||
|
Basra,Iraq,IQ
|
||||||
|
Mosul,Iraq,IQ
|
||||||
|
Erbil,Iraq,IQ
|
||||||
|
Kirkuk,Iraq,IQ
|
||||||
|
Tripoli,Libya,LY
|
||||||
|
Benghazi,Libya,LY
|
||||||
|
Misrata,Libya,LY
|
||||||
|
Zawiya,Libya,LY
|
||||||
|
Bayda,Libya,LY
|
||||||
|
Khartoum,Sudan,SD
|
||||||
|
Omdurman,Sudan,SD
|
||||||
|
Port Sudan,Sudan,SD
|
||||||
|
Kassala,Sudan,SD
|
||||||
|
Nyala,Sudan,SD
|
||||||
|
Mogadishu,Somalia,SO
|
||||||
|
Hargeisa,Somalia,SO
|
||||||
|
Bosaso,Somalia,SO
|
||||||
|
Kismayo,Somalia,SO
|
||||||
|
Merca,Somalia,SO
|
||||||
|
Sanaa,Yemen,YE
|
||||||
|
Aden,Yemen,YE
|
||||||
|
Taiz,Yemen,YE
|
||||||
|
Hodeidah,Yemen,YE
|
||||||
|
Ibb,Yemen,YE
|
||||||
|
Harare,Zimbabwe,ZW
|
||||||
|
Bulawayo,Zimbabwe,ZW
|
||||||
|
Chitungwiza,Zimbabwe,ZW
|
||||||
|
Mutare,Zimbabwe,ZW
|
||||||
|
Gweru,Zimbabwe,ZW
|
||||||
|
Amsterdam,Netherlands,NL
|
||||||
|
Rotterdam,Netherlands,NL
|
||||||
|
The Hague,Netherlands,NL
|
||||||
|
Utrecht,Netherlands,NL
|
||||||
|
Eindhoven,Netherlands,NL
|
||||||
|
Stockholm,Sweden,SE
|
||||||
|
Gothenburg,Sweden,SE
|
||||||
|
Malmo,Sweden,SE
|
||||||
|
Uppsala,Sweden,SE
|
||||||
|
Vasteras,Sweden,SE
|
||||||
|
Oslo,Norway,NO
|
||||||
|
Bergen,Norway,NO
|
||||||
|
Trondheim,Norway,NO
|
||||||
|
Stavanger,Norway,NO
|
||||||
|
Drammen,Norway,NO
|
||||||
|
Copenhagen,Denmark,DK
|
||||||
|
Aarhus,Denmark,DK
|
||||||
|
Odense,Denmark,DK
|
||||||
|
Aalborg,Denmark,DK
|
||||||
|
Esbjerg,Denmark,DK
|
||||||
|
Helsinki,Finland,FI
|
||||||
|
Espoo,Finland,FI
|
||||||
|
Tampere,Finland,FI
|
||||||
|
Vantaa,Finland,FI
|
||||||
|
Oulu,Finland,FI
|
||||||
|
Zurich,Switzerland,CH
|
||||||
|
Geneva,Switzerland,CH
|
||||||
|
Basel,Switzerland,CH
|
||||||
|
Lausanne,Switzerland,CH
|
||||||
|
Bern,Switzerland,CH
|
||||||
|
Singapore,Singapore,SG
|
||||||
|
Hong Kong,Hong Kong,HK
|
||||||
|
Kowloon,Hong Kong,HK
|
||||||
|
Seoul,South Korea,KR
|
||||||
|
Busan,South Korea,KR
|
||||||
|
Incheon,South Korea,KR
|
||||||
|
Daegu,South Korea,KR
|
||||||
|
Daejeon,South Korea,KR
|
||||||
|
Taipei,Taiwan,TW
|
||||||
|
Kaohsiung,Taiwan,TW
|
||||||
|
Taichung,Taiwan,TW
|
||||||
|
Tainan,Taiwan,TW
|
||||||
|
Hsinchu,Taiwan,TW
|
||||||
|
Auckland,New Zealand,NZ
|
||||||
|
Wellington,New Zealand,NZ
|
||||||
|
Christchurch,New Zealand,NZ
|
||||||
|
Hamilton,New Zealand,NZ
|
||||||
|
Tauranga,New Zealand,NZ
|
||||||
|
|
||||||
|
@@ -0,0 +1,52 @@
|
|||||||
|
version: '3.8'
|
||||||
|
|
||||||
|
services:
|
||||||
|
postgres:
|
||||||
|
image: postgres:16-alpine
|
||||||
|
container_name: fraud_detection_db
|
||||||
|
environment:
|
||||||
|
POSTGRES_DB: fraud_detection
|
||||||
|
POSTGRES_USER: fraud_analyst
|
||||||
|
POSTGRES_PASSWORD: SecurePass123!
|
||||||
|
PGDATA: /var/lib/postgresql/data/pgdata
|
||||||
|
ports:
|
||||||
|
- "5432:5432"
|
||||||
|
volumes:
|
||||||
|
- postgres_data:/var/lib/postgresql/data
|
||||||
|
- ./schema:/docker-entrypoint-initdb.d
|
||||||
|
- ./data:/data
|
||||||
|
networks:
|
||||||
|
- fraud_network
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD-SHELL", "pg_isready -U fraud_analyst -d fraud_detection"]
|
||||||
|
interval: 10s
|
||||||
|
timeout: 5s
|
||||||
|
retries: 5
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
db-ui:
|
||||||
|
image: ghcr.io/n7olkachev/db-ui:latest
|
||||||
|
container_name: fraud_detection_ui
|
||||||
|
environment:
|
||||||
|
POSTGRES_HOST: postgres
|
||||||
|
POSTGRES_USER: fraud_analyst
|
||||||
|
POSTGRES_PASSWORD: SecurePass123!
|
||||||
|
POSTGRES_DB: fraud_detection
|
||||||
|
POSTGRES_PORT: 5432
|
||||||
|
ports:
|
||||||
|
- "3000:3000"
|
||||||
|
networks:
|
||||||
|
- fraud_network
|
||||||
|
depends_on:
|
||||||
|
postgres:
|
||||||
|
condition: service_healthy
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
networks:
|
||||||
|
fraud_network:
|
||||||
|
driver: bridge
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
postgres_data:
|
||||||
|
driver: local
|
||||||
|
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
-- ============================================================================
|
||||||
|
-- Database Initialization Script
|
||||||
|
-- ============================================================================
|
||||||
|
-- This script ensures the database is created and ready
|
||||||
|
-- It runs automatically when the PostgreSQL container starts
|
||||||
|
-- ============================================================================
|
||||||
|
|
||||||
|
-- Ensure the database exists (this runs in the default postgres database)
|
||||||
|
SELECT 'Database initialization starting...' AS status;
|
||||||
|
|
||||||
|
-- Set timezone
|
||||||
|
SET timezone = 'UTC';
|
||||||
|
|
||||||
|
-- Show current database
|
||||||
|
SELECT current_database() AS current_db, current_user AS current_user, version() AS pg_version;
|
||||||
|
|
||||||
|
-- Enable required extensions
|
||||||
|
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
|
||||||
|
CREATE EXTENSION IF NOT EXISTS "pgcrypto";
|
||||||
|
|
||||||
|
SELECT 'Extensions enabled successfully' AS status;
|
||||||
|
|
||||||
@@ -0,0 +1,313 @@
|
|||||||
|
# Quick Start Guide
|
||||||
|
|
||||||
|
## 🚀 Get Up and Running in 5 Minutes
|
||||||
|
|
||||||
|
### Step 1: Start Docker Containers (1 minute)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# From the project root directory
|
||||||
|
docker-compose up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
**What this does:**
|
||||||
|
- Starts PostgreSQL 16 database
|
||||||
|
- Starts DB-UI web interface
|
||||||
|
- Creates network and volumes
|
||||||
|
|
||||||
|
**Verify it's running:**
|
||||||
|
```bash
|
||||||
|
docker-compose ps
|
||||||
|
```
|
||||||
|
|
||||||
|
You should see both `fraud_detection_db` and `fraud_detection_ui` running.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Step 2: Initialize Database Schema (1 minute)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Make script executable (first time only)
|
||||||
|
chmod +x scripts/setup-database.sh
|
||||||
|
|
||||||
|
# Run setup
|
||||||
|
./scripts/setup-database.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
**What this does:**
|
||||||
|
- Creates all 20+ tables
|
||||||
|
- Sets up indexes and constraints
|
||||||
|
- Loads reference data (countries, merchant categories, etc.)
|
||||||
|
|
||||||
|
**Expected output:**
|
||||||
|
```
|
||||||
|
✓ PostgreSQL is ready
|
||||||
|
✓ Creating tables, indexes, and constraints
|
||||||
|
✓ Loading reference data
|
||||||
|
✓ Database setup completed successfully!
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Step 3: Generate Test Data (15-30 minutes)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Make script executable (first time only)
|
||||||
|
chmod +x data/generate_data.sh
|
||||||
|
|
||||||
|
# Run data generation
|
||||||
|
./data/generate_data.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
**What this does:**
|
||||||
|
- Generates 100,000 customers
|
||||||
|
- Creates 150,000 accounts
|
||||||
|
- Generates 5,000,000 transactions
|
||||||
|
- Creates fraud patterns and alerts
|
||||||
|
|
||||||
|
**⏱️ Time estimate:**
|
||||||
|
- Fast machine (SSD, 16GB RAM): ~15 minutes
|
||||||
|
- Average machine: ~20-25 minutes
|
||||||
|
- Slower machine: ~30 minutes
|
||||||
|
|
||||||
|
**You can monitor progress:**
|
||||||
|
The script shows progress for each step:
|
||||||
|
```
|
||||||
|
[1/9] Loading geographic reference data...
|
||||||
|
[2/9] Generating Customers...
|
||||||
|
[3/9] Generating Accounts...
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Step 4: Access the Database
|
||||||
|
|
||||||
|
#### Option A: DB-UI Web Interface (Recommended for Beginners)
|
||||||
|
|
||||||
|
1. Open your browser to: **http://localhost:3000**
|
||||||
|
2. You'll see the database tables in the sidebar
|
||||||
|
3. Click any table to browse data
|
||||||
|
4. Use the "Custom SQL" tab to run queries
|
||||||
|
|
||||||
|
**Features:**
|
||||||
|
- Visual table browser
|
||||||
|
- SQL query editor with syntax highlighting
|
||||||
|
- Export results to CSV
|
||||||
|
- Schema introspection
|
||||||
|
|
||||||
|
#### Option B: Command Line (psql)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker exec -it fraud_detection_db psql -U fraud_analyst -d fraud_detection
|
||||||
|
```
|
||||||
|
|
||||||
|
**Quick commands:**
|
||||||
|
```sql
|
||||||
|
-- List all tables
|
||||||
|
\dt
|
||||||
|
|
||||||
|
-- Describe a table
|
||||||
|
\d customers
|
||||||
|
|
||||||
|
-- Run a query
|
||||||
|
SELECT COUNT(*) FROM transactions;
|
||||||
|
|
||||||
|
-- Exit
|
||||||
|
\q
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Option C: Your Favorite SQL Client
|
||||||
|
|
||||||
|
**Connection Details:**
|
||||||
|
```
|
||||||
|
Host: localhost
|
||||||
|
Port: 5432
|
||||||
|
Database: fraud_detection
|
||||||
|
Username: fraud_analyst
|
||||||
|
Password: SecurePass123!
|
||||||
|
```
|
||||||
|
|
||||||
|
**Popular clients:**
|
||||||
|
- DBeaver (free, cross-platform)
|
||||||
|
- pgAdmin (free, PostgreSQL-specific)
|
||||||
|
- DataGrip (paid, JetBrains)
|
||||||
|
- TablePlus (paid, macOS/Windows)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🎓 Your First Queries
|
||||||
|
|
||||||
|
### 1. Check Data Counts
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- How many customers?
|
||||||
|
SELECT COUNT(*) FROM customers;
|
||||||
|
|
||||||
|
-- How many transactions?
|
||||||
|
SELECT COUNT(*) FROM transactions;
|
||||||
|
|
||||||
|
-- How many fraud alerts?
|
||||||
|
SELECT COUNT(*) FROM alerts WHERE status = 'OPEN';
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Find High-Risk Customers
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
customer_id,
|
||||||
|
first_name,
|
||||||
|
last_name,
|
||||||
|
email,
|
||||||
|
risk_score
|
||||||
|
FROM customers
|
||||||
|
WHERE risk_score > 80
|
||||||
|
ORDER BY risk_score DESC
|
||||||
|
LIMIT 10;
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. View Recent Transactions
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
transaction_id,
|
||||||
|
account_id,
|
||||||
|
amount,
|
||||||
|
transaction_date,
|
||||||
|
is_flagged
|
||||||
|
FROM transactions
|
||||||
|
ORDER BY transaction_date DESC
|
||||||
|
LIMIT 20;
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Find Flagged Transactions
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT
|
||||||
|
t.transaction_id,
|
||||||
|
t.amount,
|
||||||
|
t.fraud_score,
|
||||||
|
t.flagged_reason,
|
||||||
|
c.first_name,
|
||||||
|
c.last_name
|
||||||
|
FROM transactions t
|
||||||
|
JOIN accounts a ON t.account_id = a.account_id
|
||||||
|
JOIN customers c ON a.customer_id = c.customer_id
|
||||||
|
WHERE t.is_flagged = TRUE
|
||||||
|
ORDER BY t.fraud_score DESC
|
||||||
|
LIMIT 10;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📚 Next Steps
|
||||||
|
|
||||||
|
### Start Learning SQL
|
||||||
|
|
||||||
|
1. **Begin with basics:** `exercises/01-basic-queries/README.md`
|
||||||
|
2. **Progress through levels:** Work through exercises 01-06
|
||||||
|
3. **Practice fraud detection:** `exercises/06-fraud-detection/README.md`
|
||||||
|
|
||||||
|
### Explore the Data
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- What countries are represented?
|
||||||
|
SELECT country_name, COUNT(*) as customer_count
|
||||||
|
FROM customers c
|
||||||
|
JOIN countries co ON c.country_id = co.country_id
|
||||||
|
GROUP BY country_name
|
||||||
|
ORDER BY customer_count DESC;
|
||||||
|
|
||||||
|
-- What are the top merchant categories?
|
||||||
|
SELECT mc.category_name, COUNT(*) as transaction_count
|
||||||
|
FROM transactions t
|
||||||
|
JOIN merchants m ON t.merchant_id = m.merchant_id
|
||||||
|
JOIN merchant_categories mc ON m.category_id = mc.category_id
|
||||||
|
GROUP BY mc.category_name
|
||||||
|
ORDER BY transaction_count DESC;
|
||||||
|
|
||||||
|
-- How many fraud cases by type?
|
||||||
|
SELECT ft.fraud_name, COUNT(*) as case_count
|
||||||
|
FROM fraud_cases fc
|
||||||
|
JOIN fraud_types ft ON fc.fraud_type_id = ft.fraud_type_id
|
||||||
|
GROUP BY ft.fraud_name
|
||||||
|
ORDER BY case_count DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔧 Troubleshooting
|
||||||
|
|
||||||
|
### Database won't start
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check logs
|
||||||
|
docker-compose logs postgres
|
||||||
|
|
||||||
|
# Restart containers
|
||||||
|
docker-compose restart
|
||||||
|
```
|
||||||
|
|
||||||
|
### Can't connect to database
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check if PostgreSQL is ready
|
||||||
|
docker exec fraud_detection_db pg_isready -U fraud_analyst
|
||||||
|
|
||||||
|
# Check port is not in use
|
||||||
|
netstat -an | grep 5432
|
||||||
|
```
|
||||||
|
|
||||||
|
### Data generation fails
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check disk space
|
||||||
|
df -h
|
||||||
|
|
||||||
|
# Check memory
|
||||||
|
free -h
|
||||||
|
|
||||||
|
# Try with smaller dataset
|
||||||
|
# Edit data/generate_data.sh and reduce:
|
||||||
|
NUM_CUSTOMERS=10000
|
||||||
|
NUM_TRANSACTIONS=500000
|
||||||
|
```
|
||||||
|
|
||||||
|
### Reset everything
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Stop and remove everything
|
||||||
|
docker-compose down -v
|
||||||
|
|
||||||
|
# Start fresh
|
||||||
|
docker-compose up -d
|
||||||
|
./scripts/setup-database.sh
|
||||||
|
./data/generate_data.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 💡 Tips
|
||||||
|
|
||||||
|
1. **Use DB-UI for exploration** - Great for browsing and understanding the schema
|
||||||
|
2. **Use psql for practice** - Best for learning SQL commands
|
||||||
|
3. **Start simple** - Begin with basic SELECT queries before complex joins
|
||||||
|
4. **Check the exercises** - They're designed to build your skills progressively
|
||||||
|
5. **Experiment** - The database is yours to explore and learn from!
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🎯 Learning Goals
|
||||||
|
|
||||||
|
After completing this tutorial, you'll be able to:
|
||||||
|
|
||||||
|
- ✅ Write complex SQL queries
|
||||||
|
- ✅ Understand database relationships
|
||||||
|
- ✅ Detect fraud patterns in data
|
||||||
|
- ✅ Use window functions and CTEs
|
||||||
|
- ✅ Optimize queries with indexes
|
||||||
|
- ✅ Investigate financial crimes
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Ready to start? Head to `exercises/01-basic-queries/README.md`!** 🚀
|
||||||
|
|
||||||
@@ -0,0 +1,206 @@
|
|||||||
|
# Level 1: Basic SQL Queries
|
||||||
|
|
||||||
|
## Introduction
|
||||||
|
Welcome to the Financial Fraud Detection SQL learning path! In this first level, you'll learn the fundamentals of SQL by querying a realistic fraud detection database.
|
||||||
|
|
||||||
|
## Learning Objectives
|
||||||
|
- Understand SELECT statements
|
||||||
|
- Use WHERE clauses for filtering
|
||||||
|
- Sort results with ORDER BY
|
||||||
|
- Limit result sets
|
||||||
|
- Work with basic comparison operators
|
||||||
|
|
||||||
|
## Exercises
|
||||||
|
|
||||||
|
### Exercise 1.1: View All Customers
|
||||||
|
**Objective:** Retrieve all customer records
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
SELECT * FROM customers;
|
||||||
|
```
|
||||||
|
|
||||||
|
**Expected Result:** All customer records with all columns
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Exercise 1.2: Find a Specific Customer
|
||||||
|
**Objective:** Find customer with customer_id = 1
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
SELECT * FROM customers WHERE customer_id = 1;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Exercise 1.3: High-Risk Customers
|
||||||
|
**Objective:** Find all customers with a risk_score greater than 80
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Hint:** Use the WHERE clause with the > operator
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT customer_id, first_name, last_name, email, risk_score
|
||||||
|
FROM customers
|
||||||
|
WHERE risk_score > 80
|
||||||
|
ORDER BY risk_score DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Exercise 1.4: Recent Registrations
|
||||||
|
**Objective:** Find customers who registered in 2024
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Hint:** Use WHERE with date comparison
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT customer_id, first_name, last_name, email, registration_date
|
||||||
|
FROM customers
|
||||||
|
WHERE registration_date >= '2024-01-01'
|
||||||
|
ORDER BY registration_date DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Exercise 1.5: Inactive Accounts
|
||||||
|
**Objective:** Find all inactive customer accounts
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT customer_id, first_name, last_name, email, is_active
|
||||||
|
FROM customers
|
||||||
|
WHERE is_active = FALSE;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Exercise 1.6: Top 10 Largest Transactions
|
||||||
|
**Objective:** Find the 10 largest transactions by amount
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Hint:** Use ORDER BY with LIMIT
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT transaction_id, account_id, amount, transaction_date, description
|
||||||
|
FROM transactions
|
||||||
|
ORDER BY amount DESC
|
||||||
|
LIMIT 10;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Exercise 1.7: Flagged Transactions
|
||||||
|
**Objective:** Find all transactions that have been flagged for review
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT transaction_id, account_id, amount, fraud_score, flagged_reason
|
||||||
|
FROM transactions
|
||||||
|
WHERE is_flagged = TRUE
|
||||||
|
ORDER BY fraud_score DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Exercise 1.8: International Transactions
|
||||||
|
**Objective:** Find all international transactions over $1,000
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT transaction_id, account_id, amount, country_id, city
|
||||||
|
FROM transactions
|
||||||
|
WHERE is_international = TRUE AND amount > 1000
|
||||||
|
ORDER BY amount DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Exercise 1.9: Specific Merchant Categories
|
||||||
|
**Objective:** Find all merchants in the 'Gambling' or 'Cryptocurrency' categories
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Hint:** Join merchants with merchant_categories, use IN or OR
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT m.merchant_id, m.merchant_name, mc.category_name, m.risk_rating
|
||||||
|
FROM merchants m
|
||||||
|
JOIN merchant_categories mc ON m.category_id = mc.category_id
|
||||||
|
WHERE mc.category_name IN ('Gambling', 'Cryptocurrency')
|
||||||
|
ORDER BY m.risk_rating DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Exercise 1.10: Critical Alerts
|
||||||
|
**Objective:** Find all open alerts with CRITICAL severity
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Your query here
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
**Solution:**
|
||||||
|
```sql
|
||||||
|
SELECT alert_id, customer_id, alert_type, description, alert_date
|
||||||
|
FROM alerts
|
||||||
|
WHERE severity = 'CRITICAL' AND status = 'OPEN'
|
||||||
|
ORDER BY alert_date DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Challenge Exercises
|
||||||
|
|
||||||
|
### Challenge 1.1: PEP Customers
|
||||||
|
Find all Politically Exposed Persons (PEPs) with high risk scores (> 70)
|
||||||
|
|
||||||
|
### Challenge 1.2: Expired Cards
|
||||||
|
Find all cards that have expired (expiry_date < current_date)
|
||||||
|
|
||||||
|
### Challenge 1.3: Large Cash Advances
|
||||||
|
Find all cash advance transactions over $5,000
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
Once you're comfortable with these basic queries, move on to:
|
||||||
|
- **Level 2:** JOIN operations
|
||||||
|
- **Level 3:** Aggregate functions and GROUP BY
|
||||||
|
|
||||||
@@ -0,0 +1,360 @@
|
|||||||
|
# Level 6: Fraud Detection Scenarios
|
||||||
|
|
||||||
|
## Introduction
|
||||||
|
Now you'll apply your SQL skills to real-world fraud detection scenarios. These exercises simulate actual fraud investigation tasks.
|
||||||
|
|
||||||
|
## Learning Objectives
|
||||||
|
- Detect velocity fraud patterns
|
||||||
|
- Identify geographic anomalies
|
||||||
|
- Find money mule networks
|
||||||
|
- Detect account takeover attempts
|
||||||
|
- Identify structuring patterns
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Fraud Pattern Detection
|
||||||
|
|
||||||
|
### Scenario 1: Velocity Fraud Detection
|
||||||
|
**Objective:** Find accounts with more than 5 transactions in a 1-hour window
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Detect rapid-fire transactions (velocity check)
|
||||||
|
WITH transaction_windows AS (
|
||||||
|
SELECT
|
||||||
|
t1.account_id,
|
||||||
|
t1.transaction_id,
|
||||||
|
t1.transaction_date,
|
||||||
|
t1.amount,
|
||||||
|
COUNT(t2.transaction_id) as transactions_in_hour
|
||||||
|
FROM transactions t1
|
||||||
|
JOIN transactions t2 ON t1.account_id = t2.account_id
|
||||||
|
AND t2.transaction_date BETWEEN t1.transaction_date - INTERVAL '1 hour'
|
||||||
|
AND t1.transaction_date
|
||||||
|
GROUP BY t1.account_id, t1.transaction_id, t1.transaction_date, t1.amount
|
||||||
|
)
|
||||||
|
SELECT
|
||||||
|
account_id,
|
||||||
|
transaction_date,
|
||||||
|
transactions_in_hour,
|
||||||
|
SUM(amount) as total_amount
|
||||||
|
FROM transaction_windows
|
||||||
|
WHERE transactions_in_hour > 5
|
||||||
|
GROUP BY account_id, transaction_date, transactions_in_hour
|
||||||
|
ORDER BY transactions_in_hour DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Scenario 2: Geographic Impossibility
|
||||||
|
**Objective:** Find transactions from the same card in different countries within 2 hours
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Detect impossible travel (card used in different countries too quickly)
|
||||||
|
SELECT
|
||||||
|
t1.card_id,
|
||||||
|
t1.transaction_id as trans1_id,
|
||||||
|
t1.transaction_date as trans1_date,
|
||||||
|
c1.country_name as country1,
|
||||||
|
t1.city as city1,
|
||||||
|
t2.transaction_id as trans2_id,
|
||||||
|
t2.transaction_date as trans2_date,
|
||||||
|
c2.country_name as country2,
|
||||||
|
t2.city as city2,
|
||||||
|
EXTRACT(EPOCH FROM (t2.transaction_date - t1.transaction_date))/3600 as hours_between
|
||||||
|
FROM transactions t1
|
||||||
|
JOIN transactions t2 ON t1.card_id = t2.card_id
|
||||||
|
AND t2.transaction_date > t1.transaction_date
|
||||||
|
AND t2.transaction_date <= t1.transaction_date + INTERVAL '2 hours'
|
||||||
|
JOIN countries c1 ON t1.country_id = c1.country_id
|
||||||
|
JOIN countries c2 ON t2.country_id = c2.country_id
|
||||||
|
WHERE t1.country_id != t2.country_id
|
||||||
|
ORDER BY hours_between ASC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Scenario 3: Money Mule Network Detection
|
||||||
|
**Objective:** Find clusters of accounts that transfer money in a chain pattern
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Detect potential money mule networks (rapid transfer chains)
|
||||||
|
WITH transfer_chains AS (
|
||||||
|
SELECT
|
||||||
|
tr1.from_account_id as account1,
|
||||||
|
tr1.to_account_id as account2,
|
||||||
|
tr2.to_account_id as account3,
|
||||||
|
tr1.transfer_id as transfer1,
|
||||||
|
tr2.transfer_id as transfer2,
|
||||||
|
t1.amount as amount1,
|
||||||
|
t2.amount as amount2,
|
||||||
|
t1.transaction_date as date1,
|
||||||
|
t2.transaction_date as date2,
|
||||||
|
EXTRACT(EPOCH FROM (t2.transaction_date - t1.transaction_date))/3600 as hours_between
|
||||||
|
FROM transfers tr1
|
||||||
|
JOIN transfers tr2 ON tr1.to_account_id = tr2.from_account_id
|
||||||
|
JOIN transactions t1 ON tr1.transaction_id = t1.transaction_id
|
||||||
|
JOIN transactions t2 ON tr2.transaction_id = t2.transaction_id
|
||||||
|
WHERE t2.transaction_date BETWEEN t1.transaction_date AND t1.transaction_date + INTERVAL '24 hours'
|
||||||
|
)
|
||||||
|
SELECT
|
||||||
|
account1,
|
||||||
|
account2,
|
||||||
|
account3,
|
||||||
|
amount1,
|
||||||
|
amount2,
|
||||||
|
hours_between,
|
||||||
|
CASE
|
||||||
|
WHEN ABS(amount1 - amount2) / amount1 < 0.1 THEN 'SUSPICIOUS - Similar amounts'
|
||||||
|
ELSE 'Review'
|
||||||
|
END as risk_flag
|
||||||
|
FROM transfer_chains
|
||||||
|
WHERE hours_between < 24
|
||||||
|
ORDER BY hours_between ASC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Scenario 4: Account Takeover Detection
|
||||||
|
**Objective:** Find accounts with sudden changes in transaction patterns
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Detect account takeover by analyzing behavior changes
|
||||||
|
WITH customer_baseline AS (
|
||||||
|
SELECT
|
||||||
|
a.customer_id,
|
||||||
|
a.account_id,
|
||||||
|
AVG(t.amount) as avg_transaction,
|
||||||
|
STDDEV(t.amount) as stddev_transaction,
|
||||||
|
COUNT(*) as transaction_count
|
||||||
|
FROM accounts a
|
||||||
|
JOIN transactions t ON a.account_id = t.account_id
|
||||||
|
WHERE t.transaction_date < CURRENT_DATE - INTERVAL '30 days'
|
||||||
|
GROUP BY a.customer_id, a.account_id
|
||||||
|
),
|
||||||
|
recent_transactions AS (
|
||||||
|
SELECT
|
||||||
|
a.customer_id,
|
||||||
|
a.account_id,
|
||||||
|
t.transaction_id,
|
||||||
|
t.amount,
|
||||||
|
t.transaction_date,
|
||||||
|
t.country_id,
|
||||||
|
t.device_id
|
||||||
|
FROM accounts a
|
||||||
|
JOIN transactions t ON a.account_id = t.account_id
|
||||||
|
WHERE t.transaction_date >= CURRENT_DATE - INTERVAL '7 days'
|
||||||
|
)
|
||||||
|
SELECT
|
||||||
|
rt.customer_id,
|
||||||
|
rt.account_id,
|
||||||
|
rt.transaction_id,
|
||||||
|
rt.amount,
|
||||||
|
cb.avg_transaction,
|
||||||
|
(rt.amount - cb.avg_transaction) / NULLIF(cb.stddev_transaction, 0) as z_score,
|
||||||
|
CASE
|
||||||
|
WHEN ABS((rt.amount - cb.avg_transaction) / NULLIF(cb.stddev_transaction, 0)) > 3
|
||||||
|
THEN 'HIGH RISK - Amount anomaly'
|
||||||
|
WHEN ABS((rt.amount - cb.avg_transaction) / NULLIF(cb.stddev_transaction, 0)) > 2
|
||||||
|
THEN 'MEDIUM RISK'
|
||||||
|
ELSE 'Normal'
|
||||||
|
END as risk_level
|
||||||
|
FROM recent_transactions rt
|
||||||
|
JOIN customer_baseline cb ON rt.account_id = cb.account_id
|
||||||
|
WHERE cb.transaction_count > 10
|
||||||
|
ORDER BY ABS((rt.amount - cb.avg_transaction) / NULLIF(cb.stddev_transaction, 0)) DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Scenario 5: Structuring Detection (Smurfing)
|
||||||
|
**Objective:** Find patterns of transactions just under $10,000 (reporting threshold)
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Detect structuring - multiple transactions just under reporting threshold
|
||||||
|
WITH daily_transactions AS (
|
||||||
|
SELECT
|
||||||
|
account_id,
|
||||||
|
DATE(transaction_date) as transaction_day,
|
||||||
|
COUNT(*) as num_transactions,
|
||||||
|
SUM(amount) as total_amount,
|
||||||
|
AVG(amount) as avg_amount,
|
||||||
|
MAX(amount) as max_amount
|
||||||
|
FROM transactions
|
||||||
|
WHERE amount BETWEEN 9000 AND 9999
|
||||||
|
AND transaction_date >= CURRENT_DATE - INTERVAL '30 days'
|
||||||
|
GROUP BY account_id, DATE(transaction_date)
|
||||||
|
)
|
||||||
|
SELECT
|
||||||
|
dt.account_id,
|
||||||
|
c.first_name,
|
||||||
|
c.last_name,
|
||||||
|
dt.transaction_day,
|
||||||
|
dt.num_transactions,
|
||||||
|
dt.total_amount,
|
||||||
|
dt.avg_amount,
|
||||||
|
CASE
|
||||||
|
WHEN dt.num_transactions >= 3 AND dt.total_amount > 25000
|
||||||
|
THEN 'CRITICAL - Likely structuring'
|
||||||
|
WHEN dt.num_transactions >= 2 AND dt.total_amount > 18000
|
||||||
|
THEN 'HIGH - Possible structuring'
|
||||||
|
ELSE 'Review'
|
||||||
|
END as risk_assessment
|
||||||
|
FROM daily_transactions dt
|
||||||
|
JOIN accounts a ON dt.account_id = a.account_id
|
||||||
|
JOIN customers c ON a.customer_id = c.customer_id
|
||||||
|
WHERE dt.num_transactions >= 2
|
||||||
|
ORDER BY dt.total_amount DESC, dt.num_transactions DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Scenario 6: High-Risk Merchant Analysis
|
||||||
|
**Objective:** Find customers with unusual activity at high-risk merchants
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Analyze transactions at high-risk merchants
|
||||||
|
SELECT
|
||||||
|
c.customer_id,
|
||||||
|
c.first_name,
|
||||||
|
c.last_name,
|
||||||
|
c.risk_score as customer_risk,
|
||||||
|
mc.category_name,
|
||||||
|
m.merchant_name,
|
||||||
|
m.risk_rating as merchant_risk,
|
||||||
|
COUNT(t.transaction_id) as transaction_count,
|
||||||
|
SUM(t.amount) as total_spent,
|
||||||
|
AVG(t.amount) as avg_transaction,
|
||||||
|
MAX(t.amount) as max_transaction
|
||||||
|
FROM customers c
|
||||||
|
JOIN accounts a ON c.customer_id = a.customer_id
|
||||||
|
JOIN transactions t ON a.account_id = t.account_id
|
||||||
|
JOIN merchants m ON t.merchant_id = m.merchant_id
|
||||||
|
JOIN merchant_categories mc ON m.category_id = mc.category_id
|
||||||
|
WHERE m.risk_rating IN ('HIGH', 'CRITICAL')
|
||||||
|
AND t.transaction_date >= CURRENT_DATE - INTERVAL '90 days'
|
||||||
|
GROUP BY c.customer_id, c.first_name, c.last_name, c.risk_score,
|
||||||
|
mc.category_name, m.merchant_name, m.risk_rating
|
||||||
|
HAVING COUNT(t.transaction_id) > 5 OR SUM(t.amount) > 10000
|
||||||
|
ORDER BY total_spent DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Scenario 7: Card Testing Detection
|
||||||
|
**Objective:** Find cards with multiple small failed transactions (testing stolen cards)
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Detect card testing patterns
|
||||||
|
SELECT
|
||||||
|
t.card_id,
|
||||||
|
c.card_last_four,
|
||||||
|
COUNT(*) as failed_attempts,
|
||||||
|
COUNT(DISTINCT t.merchant_id) as different_merchants,
|
||||||
|
MIN(t.amount) as min_amount,
|
||||||
|
MAX(t.amount) as max_amount,
|
||||||
|
MIN(t.transaction_date) as first_attempt,
|
||||||
|
MAX(t.transaction_date) as last_attempt,
|
||||||
|
EXTRACT(EPOCH FROM (MAX(t.transaction_date) - MIN(t.transaction_date)))/60 as minutes_span
|
||||||
|
FROM transactions t
|
||||||
|
JOIN cards c ON t.card_id = c.card_id
|
||||||
|
WHERE t.status = 'FAILED'
|
||||||
|
AND t.amount < 10
|
||||||
|
AND t.transaction_date >= CURRENT_DATE - INTERVAL '24 hours'
|
||||||
|
GROUP BY t.card_id, c.card_last_four
|
||||||
|
HAVING COUNT(*) >= 3
|
||||||
|
ORDER BY failed_attempts DESC, minutes_span ASC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Scenario 8: Dormant Account Reactivation
|
||||||
|
**Objective:** Find dormant accounts that suddenly become active (potential takeover)
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Detect dormant account reactivation
|
||||||
|
WITH account_activity AS (
|
||||||
|
SELECT
|
||||||
|
account_id,
|
||||||
|
MIN(transaction_date) as first_transaction,
|
||||||
|
MAX(transaction_date) as last_transaction,
|
||||||
|
COUNT(*) as total_transactions
|
||||||
|
FROM transactions
|
||||||
|
GROUP BY account_id
|
||||||
|
),
|
||||||
|
dormant_accounts AS (
|
||||||
|
SELECT
|
||||||
|
account_id,
|
||||||
|
last_transaction,
|
||||||
|
total_transactions
|
||||||
|
FROM account_activity
|
||||||
|
WHERE last_transaction < CURRENT_DATE - INTERVAL '180 days'
|
||||||
|
),
|
||||||
|
recent_activity AS (
|
||||||
|
SELECT
|
||||||
|
t.account_id,
|
||||||
|
COUNT(*) as recent_transactions,
|
||||||
|
SUM(t.amount) as recent_amount,
|
||||||
|
MIN(t.transaction_date) as reactivation_date
|
||||||
|
FROM transactions t
|
||||||
|
WHERE t.transaction_date >= CURRENT_DATE - INTERVAL '7 days'
|
||||||
|
GROUP BY t.account_id
|
||||||
|
)
|
||||||
|
SELECT
|
||||||
|
da.account_id,
|
||||||
|
c.first_name,
|
||||||
|
c.last_name,
|
||||||
|
c.email,
|
||||||
|
da.last_transaction as last_active,
|
||||||
|
EXTRACT(DAY FROM (CURRENT_DATE - da.last_transaction)) as days_dormant,
|
||||||
|
ra.reactivation_date,
|
||||||
|
ra.recent_transactions,
|
||||||
|
ra.recent_amount,
|
||||||
|
'CRITICAL - Dormant account reactivated' as alert_type
|
||||||
|
FROM dormant_accounts da
|
||||||
|
JOIN recent_activity ra ON da.account_id = ra.account_id
|
||||||
|
JOIN accounts a ON da.account_id = a.account_id
|
||||||
|
JOIN customers c ON a.customer_id = c.customer_id
|
||||||
|
ORDER BY days_dormant DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Investigation Exercises
|
||||||
|
|
||||||
|
### Exercise 6.1: Full Customer Investigation
|
||||||
|
Create a comprehensive report for a suspicious customer including:
|
||||||
|
- All accounts
|
||||||
|
- All transactions
|
||||||
|
- All alerts
|
||||||
|
- All fraud cases
|
||||||
|
- Related customers (via relationships)
|
||||||
|
|
||||||
|
### Exercise 6.2: Fraud Case Summary
|
||||||
|
Generate a summary report of all open fraud cases with:
|
||||||
|
- Case details
|
||||||
|
- Associated transactions
|
||||||
|
- Total amount at risk
|
||||||
|
- Investigation status
|
||||||
|
|
||||||
|
### Exercise 6.3: Daily Fraud Dashboard
|
||||||
|
Create a daily dashboard showing:
|
||||||
|
- New alerts by severity
|
||||||
|
- High-risk transactions
|
||||||
|
- Geographic anomalies
|
||||||
|
- Velocity violations
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
Congratulations! You've completed the fraud detection scenarios. You now have the skills to:
|
||||||
|
- Detect complex fraud patterns
|
||||||
|
- Investigate suspicious activity
|
||||||
|
- Generate fraud reports
|
||||||
|
- Analyze customer behavior
|
||||||
|
|
||||||
|
Continue practicing with real data and explore advanced topics like:
|
||||||
|
- Machine learning integration
|
||||||
|
- Real-time fraud scoring
|
||||||
|
- Network analysis
|
||||||
|
- Predictive modeling
|
||||||
|
|
||||||
@@ -0,0 +1,477 @@
|
|||||||
|
-- ============================================================================
|
||||||
|
-- Financial Fraud Detection Database Schema
|
||||||
|
-- Purpose: Educational SQL learning with realistic fraud investigation scenarios
|
||||||
|
-- ============================================================================
|
||||||
|
-- This script is IDEMPOTENT - it will drop and recreate all objects
|
||||||
|
-- ============================================================================
|
||||||
|
|
||||||
|
-- Enable required extensions
|
||||||
|
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
|
||||||
|
CREATE EXTENSION IF NOT EXISTS "pgcrypto";
|
||||||
|
|
||||||
|
-- ============================================================================
|
||||||
|
-- DROP ALL EXISTING OBJECTS (in reverse dependency order)
|
||||||
|
-- ============================================================================
|
||||||
|
|
||||||
|
-- Drop triggers first
|
||||||
|
DROP TRIGGER IF EXISTS trg_update_balance ON transactions;
|
||||||
|
DROP TRIGGER IF EXISTS trg_check_suspicious ON transactions;
|
||||||
|
|
||||||
|
-- Drop functions
|
||||||
|
DROP FUNCTION IF EXISTS update_account_balance() CASCADE;
|
||||||
|
DROP FUNCTION IF EXISTS check_suspicious_transaction() CASCADE;
|
||||||
|
DROP FUNCTION IF EXISTS generate_fraud_score(DECIMAL, BOOLEAN, DECIMAL, INT, BOOLEAN) CASCADE;
|
||||||
|
|
||||||
|
-- Drop tables in reverse dependency order
|
||||||
|
DROP TABLE IF EXISTS audit_log CASCADE;
|
||||||
|
DROP TABLE IF EXISTS suspicious_activity_reports CASCADE;
|
||||||
|
DROP TABLE IF EXISTS case_alerts CASCADE;
|
||||||
|
DROP TABLE IF EXISTS case_transactions CASCADE;
|
||||||
|
DROP TABLE IF EXISTS fraud_cases CASCADE;
|
||||||
|
DROP TABLE IF EXISTS alerts CASCADE;
|
||||||
|
DROP TABLE IF EXISTS transfers CASCADE;
|
||||||
|
DROP TABLE IF EXISTS beneficiaries CASCADE;
|
||||||
|
DROP TABLE IF EXISTS transactions CASCADE;
|
||||||
|
DROP TABLE IF EXISTS login_sessions CASCADE;
|
||||||
|
DROP TABLE IF EXISTS devices CASCADE;
|
||||||
|
DROP TABLE IF EXISTS cards CASCADE;
|
||||||
|
DROP TABLE IF EXISTS accounts CASCADE;
|
||||||
|
DROP TABLE IF EXISTS customer_relationships CASCADE;
|
||||||
|
DROP TABLE IF EXISTS customers CASCADE;
|
||||||
|
DROP TABLE IF EXISTS merchants CASCADE;
|
||||||
|
DROP TABLE IF EXISTS merchant_categories CASCADE;
|
||||||
|
DROP TABLE IF EXISTS fraud_types CASCADE;
|
||||||
|
DROP TABLE IF EXISTS transaction_types CASCADE;
|
||||||
|
DROP TABLE IF EXISTS countries CASCADE;
|
||||||
|
|
||||||
|
-- ============================================================================
|
||||||
|
-- REFERENCE/LOOKUP TABLES
|
||||||
|
-- ============================================================================
|
||||||
|
|
||||||
|
-- Countries reference table
|
||||||
|
CREATE TABLE countries (
|
||||||
|
country_id SERIAL PRIMARY KEY,
|
||||||
|
country_code CHAR(2) NOT NULL UNIQUE,
|
||||||
|
country_name VARCHAR(100) NOT NULL,
|
||||||
|
region VARCHAR(50) NOT NULL,
|
||||||
|
risk_level VARCHAR(20) DEFAULT 'LOW' CHECK (risk_level IN ('LOW', 'MEDIUM', 'HIGH', 'CRITICAL')),
|
||||||
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Merchant categories
|
||||||
|
CREATE TABLE merchant_categories (
|
||||||
|
category_id SERIAL PRIMARY KEY,
|
||||||
|
category_code VARCHAR(10) NOT NULL UNIQUE,
|
||||||
|
category_name VARCHAR(100) NOT NULL,
|
||||||
|
description TEXT,
|
||||||
|
risk_weight DECIMAL(3,2) DEFAULT 1.00,
|
||||||
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Transaction types
|
||||||
|
CREATE TABLE transaction_types (
|
||||||
|
type_id SERIAL PRIMARY KEY,
|
||||||
|
type_code VARCHAR(20) NOT NULL UNIQUE,
|
||||||
|
type_name VARCHAR(100) NOT NULL,
|
||||||
|
description TEXT,
|
||||||
|
requires_merchant BOOLEAN DEFAULT FALSE,
|
||||||
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Fraud types
|
||||||
|
CREATE TABLE fraud_types (
|
||||||
|
fraud_type_id SERIAL PRIMARY KEY,
|
||||||
|
fraud_code VARCHAR(20) NOT NULL UNIQUE,
|
||||||
|
fraud_name VARCHAR(100) NOT NULL,
|
||||||
|
description TEXT,
|
||||||
|
severity VARCHAR(20) DEFAULT 'MEDIUM' CHECK (severity IN ('LOW', 'MEDIUM', 'HIGH', 'CRITICAL')),
|
||||||
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
|
||||||
|
-- ============================================================================
|
||||||
|
-- CUSTOMER DOMAIN
|
||||||
|
-- ============================================================================
|
||||||
|
|
||||||
|
-- Customers table
|
||||||
|
CREATE TABLE customers (
|
||||||
|
customer_id BIGSERIAL PRIMARY KEY,
|
||||||
|
first_name VARCHAR(100) NOT NULL,
|
||||||
|
last_name VARCHAR(100) NOT NULL,
|
||||||
|
email VARCHAR(255) NOT NULL UNIQUE,
|
||||||
|
phone VARCHAR(20),
|
||||||
|
date_of_birth DATE NOT NULL,
|
||||||
|
ssn_hash VARCHAR(64) NOT NULL UNIQUE, -- Hashed SSN for privacy
|
||||||
|
address_line1 VARCHAR(255),
|
||||||
|
address_line2 VARCHAR(255),
|
||||||
|
city VARCHAR(100),
|
||||||
|
state VARCHAR(50),
|
||||||
|
postal_code VARCHAR(20),
|
||||||
|
country_id INT NOT NULL REFERENCES countries(country_id),
|
||||||
|
registration_date TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
last_login TIMESTAMP,
|
||||||
|
kyc_status VARCHAR(20) DEFAULT 'PENDING' CHECK (kyc_status IN ('PENDING', 'VERIFIED', 'REJECTED', 'EXPIRED')),
|
||||||
|
kyc_verified_date TIMESTAMP,
|
||||||
|
risk_score DECIMAL(5,2) DEFAULT 50.00 CHECK (risk_score BETWEEN 0 AND 100),
|
||||||
|
is_pep BOOLEAN DEFAULT FALSE, -- Politically Exposed Person
|
||||||
|
is_active BOOLEAN DEFAULT TRUE,
|
||||||
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Customer relationships (for detecting collusion networks)
|
||||||
|
CREATE TABLE customer_relationships (
|
||||||
|
relationship_id BIGSERIAL PRIMARY KEY,
|
||||||
|
customer_id_1 BIGINT NOT NULL REFERENCES customers(customer_id),
|
||||||
|
customer_id_2 BIGINT NOT NULL REFERENCES customers(customer_id),
|
||||||
|
relationship_type VARCHAR(50) NOT NULL CHECK (relationship_type IN ('FAMILY', 'BUSINESS', 'SHARED_ADDRESS', 'SHARED_DEVICE', 'SHARED_IP', 'SUSPECTED_MULE')),
|
||||||
|
confidence_score DECIMAL(5,2) DEFAULT 50.00 CHECK (confidence_score BETWEEN 0 AND 100),
|
||||||
|
detected_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
notes TEXT,
|
||||||
|
CONSTRAINT different_customers CHECK (customer_id_1 != customer_id_2),
|
||||||
|
CONSTRAINT unique_relationship UNIQUE (customer_id_1, customer_id_2, relationship_type)
|
||||||
|
);
|
||||||
|
|
||||||
|
-- ============================================================================
|
||||||
|
-- ACCOUNT DOMAIN
|
||||||
|
-- ============================================================================
|
||||||
|
|
||||||
|
-- Accounts table
|
||||||
|
CREATE TABLE accounts (
|
||||||
|
account_id BIGSERIAL PRIMARY KEY,
|
||||||
|
customer_id BIGINT NOT NULL REFERENCES customers(customer_id),
|
||||||
|
account_number VARCHAR(20) NOT NULL UNIQUE,
|
||||||
|
account_type VARCHAR(20) NOT NULL CHECK (account_type IN ('CHECKING', 'SAVINGS', 'CREDIT', 'INVESTMENT', 'LOAN')),
|
||||||
|
currency CHAR(3) DEFAULT 'USD',
|
||||||
|
opening_date DATE NOT NULL DEFAULT CURRENT_DATE,
|
||||||
|
closing_date DATE,
|
||||||
|
status VARCHAR(20) DEFAULT 'ACTIVE' CHECK (status IN ('ACTIVE', 'SUSPENDED', 'CLOSED', 'FROZEN')),
|
||||||
|
current_balance DECIMAL(15,2) DEFAULT 0.00,
|
||||||
|
available_balance DECIMAL(15,2) DEFAULT 0.00,
|
||||||
|
credit_limit DECIMAL(15,2),
|
||||||
|
overdraft_limit DECIMAL(15,2) DEFAULT 0.00,
|
||||||
|
interest_rate DECIMAL(5,4),
|
||||||
|
monthly_fee DECIMAL(8,2) DEFAULT 0.00,
|
||||||
|
is_primary BOOLEAN DEFAULT FALSE,
|
||||||
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Cards table
|
||||||
|
CREATE TABLE cards (
|
||||||
|
card_id BIGSERIAL PRIMARY KEY,
|
||||||
|
account_id BIGINT NOT NULL REFERENCES accounts(account_id),
|
||||||
|
card_number_hash VARCHAR(64) NOT NULL UNIQUE, -- Hashed card number
|
||||||
|
card_last_four CHAR(4) NOT NULL,
|
||||||
|
card_type VARCHAR(20) NOT NULL CHECK (card_type IN ('DEBIT', 'CREDIT', 'PREPAID', 'VIRTUAL')),
|
||||||
|
card_network VARCHAR(20) NOT NULL CHECK (card_network IN ('VISA', 'MASTERCARD', 'AMEX', 'DISCOVER')),
|
||||||
|
issue_date DATE NOT NULL DEFAULT CURRENT_DATE,
|
||||||
|
expiry_date DATE NOT NULL,
|
||||||
|
cvv_hash VARCHAR(64) NOT NULL,
|
||||||
|
status VARCHAR(20) DEFAULT 'ACTIVE' CHECK (status IN ('ACTIVE', 'BLOCKED', 'EXPIRED', 'LOST', 'STOLEN')),
|
||||||
|
daily_limit DECIMAL(10,2) DEFAULT 5000.00,
|
||||||
|
monthly_limit DECIMAL(12,2) DEFAULT 50000.00,
|
||||||
|
is_contactless BOOLEAN DEFAULT TRUE,
|
||||||
|
is_international BOOLEAN DEFAULT FALSE,
|
||||||
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
|
||||||
|
-- ============================================================================
|
||||||
|
-- MERCHANT DOMAIN
|
||||||
|
-- ============================================================================
|
||||||
|
|
||||||
|
-- Merchants table
|
||||||
|
CREATE TABLE merchants (
|
||||||
|
merchant_id BIGSERIAL PRIMARY KEY,
|
||||||
|
merchant_name VARCHAR(255) NOT NULL,
|
||||||
|
merchant_code VARCHAR(50) UNIQUE,
|
||||||
|
category_id INT NOT NULL REFERENCES merchant_categories(category_id),
|
||||||
|
country_id INT NOT NULL REFERENCES countries(country_id),
|
||||||
|
city VARCHAR(100),
|
||||||
|
website VARCHAR(255),
|
||||||
|
registration_date DATE NOT NULL DEFAULT CURRENT_DATE,
|
||||||
|
status VARCHAR(20) DEFAULT 'ACTIVE' CHECK (status IN ('ACTIVE', 'SUSPENDED', 'BLACKLISTED', 'CLOSED')),
|
||||||
|
risk_rating VARCHAR(20) DEFAULT 'LOW' CHECK (risk_rating IN ('LOW', 'MEDIUM', 'HIGH', 'CRITICAL')),
|
||||||
|
total_transactions BIGINT DEFAULT 0,
|
||||||
|
total_volume DECIMAL(18,2) DEFAULT 0.00,
|
||||||
|
fraud_incidents INT DEFAULT 0,
|
||||||
|
is_verified BOOLEAN DEFAULT FALSE,
|
||||||
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
|
||||||
|
-- ============================================================================
|
||||||
|
-- DEVICE & SESSION DOMAIN
|
||||||
|
-- ============================================================================
|
||||||
|
|
||||||
|
-- Devices table (for tracking login devices)
|
||||||
|
CREATE TABLE devices (
|
||||||
|
device_id BIGSERIAL PRIMARY KEY,
|
||||||
|
device_fingerprint VARCHAR(64) NOT NULL UNIQUE,
|
||||||
|
device_type VARCHAR(20) CHECK (device_type IN ('MOBILE', 'TABLET', 'DESKTOP', 'OTHER')),
|
||||||
|
os_name VARCHAR(50),
|
||||||
|
os_version VARCHAR(50),
|
||||||
|
browser_name VARCHAR(50),
|
||||||
|
browser_version VARCHAR(50),
|
||||||
|
first_seen TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
last_seen TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
is_trusted BOOLEAN DEFAULT FALSE,
|
||||||
|
is_blacklisted BOOLEAN DEFAULT FALSE,
|
||||||
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Login sessions
|
||||||
|
CREATE TABLE login_sessions (
|
||||||
|
session_id BIGSERIAL PRIMARY KEY,
|
||||||
|
customer_id BIGINT NOT NULL REFERENCES customers(customer_id),
|
||||||
|
device_id BIGINT NOT NULL REFERENCES devices(device_id),
|
||||||
|
ip_address INET NOT NULL,
|
||||||
|
country_id INT REFERENCES countries(country_id),
|
||||||
|
city VARCHAR(100),
|
||||||
|
latitude DECIMAL(10,8),
|
||||||
|
longitude DECIMAL(11,8),
|
||||||
|
login_timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
logout_timestamp TIMESTAMP,
|
||||||
|
session_duration_seconds INT,
|
||||||
|
is_successful BOOLEAN DEFAULT TRUE,
|
||||||
|
failure_reason VARCHAR(255),
|
||||||
|
risk_score DECIMAL(5,2) DEFAULT 0.00,
|
||||||
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
|
||||||
|
-- ============================================================================
|
||||||
|
-- TRANSACTION DOMAIN
|
||||||
|
-- ============================================================================
|
||||||
|
|
||||||
|
-- Transactions table (main transaction log)
|
||||||
|
CREATE TABLE transactions (
|
||||||
|
transaction_id BIGSERIAL PRIMARY KEY,
|
||||||
|
account_id BIGINT NOT NULL REFERENCES accounts(account_id),
|
||||||
|
type_id INT NOT NULL REFERENCES transaction_types(type_id),
|
||||||
|
transaction_date TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
amount DECIMAL(15,2) NOT NULL CHECK (amount > 0),
|
||||||
|
currency CHAR(3) DEFAULT 'USD',
|
||||||
|
merchant_id BIGINT REFERENCES merchants(merchant_id),
|
||||||
|
card_id BIGINT REFERENCES cards(card_id),
|
||||||
|
device_id BIGINT REFERENCES devices(device_id),
|
||||||
|
ip_address INET,
|
||||||
|
country_id INT REFERENCES countries(country_id),
|
||||||
|
city VARCHAR(100),
|
||||||
|
latitude DECIMAL(10,8),
|
||||||
|
longitude DECIMAL(11,8),
|
||||||
|
description TEXT,
|
||||||
|
reference_number VARCHAR(50) UNIQUE,
|
||||||
|
status VARCHAR(20) DEFAULT 'COMPLETED' CHECK (status IN ('PENDING', 'COMPLETED', 'FAILED', 'REVERSED', 'FLAGGED', 'BLOCKED')),
|
||||||
|
is_online BOOLEAN DEFAULT TRUE,
|
||||||
|
is_international BOOLEAN DEFAULT FALSE,
|
||||||
|
is_card_present BOOLEAN DEFAULT FALSE,
|
||||||
|
fraud_score DECIMAL(5,2) DEFAULT 0.00 CHECK (fraud_score BETWEEN 0 AND 100),
|
||||||
|
is_flagged BOOLEAN DEFAULT FALSE,
|
||||||
|
flagged_reason TEXT,
|
||||||
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Beneficiaries (for transfers)
|
||||||
|
CREATE TABLE beneficiaries (
|
||||||
|
beneficiary_id BIGSERIAL PRIMARY KEY,
|
||||||
|
customer_id BIGINT NOT NULL REFERENCES customers(customer_id),
|
||||||
|
beneficiary_name VARCHAR(255) NOT NULL,
|
||||||
|
account_number VARCHAR(50) NOT NULL,
|
||||||
|
bank_name VARCHAR(255),
|
||||||
|
bank_code VARCHAR(20),
|
||||||
|
country_id INT NOT NULL REFERENCES countries(country_id),
|
||||||
|
relationship VARCHAR(50),
|
||||||
|
is_verified BOOLEAN DEFAULT FALSE,
|
||||||
|
added_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
last_used TIMESTAMP,
|
||||||
|
total_transfers INT DEFAULT 0,
|
||||||
|
total_amount DECIMAL(18,2) DEFAULT 0.00,
|
||||||
|
is_active BOOLEAN DEFAULT TRUE,
|
||||||
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Transfer transactions
|
||||||
|
CREATE TABLE transfers (
|
||||||
|
transfer_id BIGSERIAL PRIMARY KEY,
|
||||||
|
transaction_id BIGINT NOT NULL REFERENCES transactions(transaction_id),
|
||||||
|
from_account_id BIGINT NOT NULL REFERENCES accounts(account_id),
|
||||||
|
to_account_id BIGINT REFERENCES accounts(account_id), -- NULL for external transfers
|
||||||
|
beneficiary_id BIGINT REFERENCES beneficiaries(beneficiary_id),
|
||||||
|
transfer_type VARCHAR(20) NOT NULL CHECK (transfer_type IN ('INTERNAL', 'DOMESTIC', 'INTERNATIONAL', 'WIRE')),
|
||||||
|
purpose VARCHAR(255),
|
||||||
|
is_recurring BOOLEAN DEFAULT FALSE,
|
||||||
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
|
||||||
|
-- ============================================================================
|
||||||
|
-- FRAUD DETECTION & ALERTS DOMAIN
|
||||||
|
-- ============================================================================
|
||||||
|
|
||||||
|
-- Alerts table (system-generated suspicious activity alerts)
|
||||||
|
CREATE TABLE alerts (
|
||||||
|
alert_id BIGSERIAL PRIMARY KEY,
|
||||||
|
transaction_id BIGINT REFERENCES transactions(transaction_id),
|
||||||
|
customer_id BIGINT REFERENCES customers(customer_id),
|
||||||
|
account_id BIGINT REFERENCES accounts(account_id),
|
||||||
|
alert_type VARCHAR(50) NOT NULL CHECK (alert_type IN (
|
||||||
|
'VELOCITY_CHECK', 'AMOUNT_ANOMALY', 'GEOGRAPHIC_ANOMALY',
|
||||||
|
'MERCHANT_RISK', 'DEVICE_CHANGE', 'UNUSUAL_TIME',
|
||||||
|
'MULTIPLE_CARDS', 'ACCOUNT_TAKEOVER', 'MONEY_MULE', 'STRUCTURING'
|
||||||
|
)),
|
||||||
|
severity VARCHAR(20) DEFAULT 'MEDIUM' CHECK (severity IN ('LOW', 'MEDIUM', 'HIGH', 'CRITICAL')),
|
||||||
|
alert_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
description TEXT NOT NULL,
|
||||||
|
risk_score DECIMAL(5,2) DEFAULT 50.00 CHECK (risk_score BETWEEN 0 AND 100),
|
||||||
|
status VARCHAR(20) DEFAULT 'OPEN' CHECK (status IN ('OPEN', 'INVESTIGATING', 'CLOSED', 'FALSE_POSITIVE', 'CONFIRMED_FRAUD')),
|
||||||
|
assigned_to VARCHAR(100),
|
||||||
|
reviewed_date TIMESTAMP,
|
||||||
|
resolution_notes TEXT,
|
||||||
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Fraud cases (confirmed fraud incidents)
|
||||||
|
CREATE TABLE fraud_cases (
|
||||||
|
case_id BIGSERIAL PRIMARY KEY,
|
||||||
|
case_number VARCHAR(50) NOT NULL UNIQUE,
|
||||||
|
customer_id BIGINT REFERENCES customers(customer_id),
|
||||||
|
account_id BIGINT REFERENCES accounts(account_id),
|
||||||
|
fraud_type_id INT NOT NULL REFERENCES fraud_types(fraud_type_id),
|
||||||
|
detection_date TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
detection_method VARCHAR(50) CHECK (detection_method IN ('AUTOMATED', 'CUSTOMER_REPORT', 'MANUAL_REVIEW', 'THIRD_PARTY')),
|
||||||
|
amount_lost DECIMAL(15,2) DEFAULT 0.00,
|
||||||
|
amount_recovered DECIMAL(15,2) DEFAULT 0.00,
|
||||||
|
status VARCHAR(20) DEFAULT 'OPEN' CHECK (status IN ('OPEN', 'INVESTIGATING', 'RESOLVED', 'CLOSED', 'LEGAL_ACTION')),
|
||||||
|
priority VARCHAR(20) DEFAULT 'MEDIUM' CHECK (priority IN ('LOW', 'MEDIUM', 'HIGH', 'CRITICAL')),
|
||||||
|
assigned_investigator VARCHAR(100),
|
||||||
|
investigation_notes TEXT,
|
||||||
|
resolution_date TIMESTAMP,
|
||||||
|
resolution_summary TEXT,
|
||||||
|
law_enforcement_notified BOOLEAN DEFAULT FALSE,
|
||||||
|
customer_notified BOOLEAN DEFAULT FALSE,
|
||||||
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Case transactions (linking transactions to fraud cases)
|
||||||
|
CREATE TABLE case_transactions (
|
||||||
|
case_transaction_id BIGSERIAL PRIMARY KEY,
|
||||||
|
case_id BIGINT NOT NULL REFERENCES fraud_cases(case_id),
|
||||||
|
transaction_id BIGINT NOT NULL REFERENCES transactions(transaction_id),
|
||||||
|
is_fraudulent BOOLEAN DEFAULT TRUE,
|
||||||
|
notes TEXT,
|
||||||
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
CONSTRAINT unique_case_transaction UNIQUE (case_id, transaction_id)
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Case alerts (linking alerts to fraud cases)
|
||||||
|
CREATE TABLE case_alerts (
|
||||||
|
case_alert_id BIGSERIAL PRIMARY KEY,
|
||||||
|
case_id BIGINT NOT NULL REFERENCES fraud_cases(case_id),
|
||||||
|
alert_id BIGINT NOT NULL REFERENCES alerts(alert_id),
|
||||||
|
relevance_score DECIMAL(5,2) DEFAULT 50.00,
|
||||||
|
notes TEXT,
|
||||||
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
CONSTRAINT unique_case_alert UNIQUE (case_id, alert_id)
|
||||||
|
);
|
||||||
|
|
||||||
|
-- ============================================================================
|
||||||
|
-- AUDIT & COMPLIANCE DOMAIN
|
||||||
|
-- ============================================================================
|
||||||
|
|
||||||
|
-- Audit log (comprehensive audit trail)
|
||||||
|
CREATE TABLE audit_log (
|
||||||
|
audit_id BIGSERIAL PRIMARY KEY,
|
||||||
|
table_name VARCHAR(100) NOT NULL,
|
||||||
|
record_id BIGINT NOT NULL,
|
||||||
|
action VARCHAR(20) NOT NULL CHECK (action IN ('INSERT', 'UPDATE', 'DELETE', 'SELECT')),
|
||||||
|
old_values JSONB,
|
||||||
|
new_values JSONB,
|
||||||
|
changed_by VARCHAR(100),
|
||||||
|
changed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
ip_address INET,
|
||||||
|
user_agent TEXT
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Suspicious Activity Reports (SAR)
|
||||||
|
CREATE TABLE suspicious_activity_reports (
|
||||||
|
sar_id BIGSERIAL PRIMARY KEY,
|
||||||
|
sar_number VARCHAR(50) NOT NULL UNIQUE,
|
||||||
|
case_id BIGINT REFERENCES fraud_cases(case_id),
|
||||||
|
customer_id BIGINT NOT NULL REFERENCES customers(customer_id),
|
||||||
|
filing_date DATE NOT NULL DEFAULT CURRENT_DATE,
|
||||||
|
activity_date_from DATE NOT NULL,
|
||||||
|
activity_date_to DATE NOT NULL,
|
||||||
|
total_amount DECIMAL(18,2) NOT NULL,
|
||||||
|
activity_description TEXT NOT NULL,
|
||||||
|
filed_by VARCHAR(100) NOT NULL,
|
||||||
|
status VARCHAR(20) DEFAULT 'DRAFT' CHECK (status IN ('DRAFT', 'SUBMITTED', 'ACKNOWLEDGED', 'CLOSED')),
|
||||||
|
submission_date DATE,
|
||||||
|
acknowledgment_date DATE,
|
||||||
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
|
||||||
|
-- ============================================================================
|
||||||
|
-- INDEXES FOR PERFORMANCE
|
||||||
|
-- ============================================================================
|
||||||
|
|
||||||
|
-- Customer indexes
|
||||||
|
CREATE INDEX idx_customers_email ON customers(email);
|
||||||
|
CREATE INDEX idx_customers_country ON customers(country_id);
|
||||||
|
CREATE INDEX idx_customers_risk_score ON customers(risk_score DESC);
|
||||||
|
CREATE INDEX idx_customers_registration_date ON customers(registration_date);
|
||||||
|
|
||||||
|
-- Account indexes
|
||||||
|
CREATE INDEX idx_accounts_customer ON accounts(customer_id);
|
||||||
|
CREATE INDEX idx_accounts_status ON accounts(status);
|
||||||
|
CREATE INDEX idx_accounts_type ON accounts(account_type);
|
||||||
|
|
||||||
|
-- Transaction indexes (critical for performance)
|
||||||
|
CREATE INDEX idx_transactions_account ON transactions(account_id);
|
||||||
|
CREATE INDEX idx_transactions_date ON transactions(transaction_date DESC);
|
||||||
|
CREATE INDEX idx_transactions_merchant ON transactions(merchant_id);
|
||||||
|
CREATE INDEX idx_transactions_status ON transactions(status);
|
||||||
|
CREATE INDEX idx_transactions_flagged ON transactions(is_flagged) WHERE is_flagged = TRUE;
|
||||||
|
CREATE INDEX idx_transactions_fraud_score ON transactions(fraud_score DESC);
|
||||||
|
CREATE INDEX idx_transactions_amount ON transactions(amount);
|
||||||
|
CREATE INDEX idx_transactions_country ON transactions(country_id);
|
||||||
|
|
||||||
|
-- Card indexes
|
||||||
|
CREATE INDEX idx_cards_account ON cards(account_id);
|
||||||
|
CREATE INDEX idx_cards_status ON cards(status);
|
||||||
|
|
||||||
|
-- Alert indexes
|
||||||
|
CREATE INDEX idx_alerts_customer ON alerts(customer_id);
|
||||||
|
CREATE INDEX idx_alerts_transaction ON alerts(transaction_id);
|
||||||
|
CREATE INDEX idx_alerts_status ON alerts(status);
|
||||||
|
CREATE INDEX idx_alerts_date ON alerts(alert_date DESC);
|
||||||
|
CREATE INDEX idx_alerts_severity ON alerts(severity);
|
||||||
|
|
||||||
|
-- Fraud case indexes
|
||||||
|
CREATE INDEX idx_fraud_cases_customer ON fraud_cases(customer_id);
|
||||||
|
CREATE INDEX idx_fraud_cases_status ON fraud_cases(status);
|
||||||
|
CREATE INDEX idx_fraud_cases_detection_date ON fraud_cases(detection_date DESC);
|
||||||
|
|
||||||
|
-- Login session indexes
|
||||||
|
CREATE INDEX idx_login_sessions_customer ON login_sessions(customer_id);
|
||||||
|
CREATE INDEX idx_login_sessions_timestamp ON login_sessions(login_timestamp DESC);
|
||||||
|
CREATE INDEX idx_login_sessions_ip ON login_sessions(ip_address);
|
||||||
|
|
||||||
|
-- Merchant indexes
|
||||||
|
CREATE INDEX idx_merchants_category ON merchants(category_id);
|
||||||
|
CREATE INDEX idx_merchants_country ON merchants(country_id);
|
||||||
|
CREATE INDEX idx_merchants_risk_rating ON merchants(risk_rating);
|
||||||
|
|
||||||
|
-- Comments for documentation
|
||||||
|
COMMENT ON TABLE customers IS 'Customer master data with KYC and risk information';
|
||||||
|
COMMENT ON TABLE accounts IS 'Customer accounts including checking, savings, credit, etc.';
|
||||||
|
COMMENT ON TABLE transactions IS 'Main transaction log with fraud scoring';
|
||||||
|
COMMENT ON TABLE alerts IS 'System-generated fraud alerts requiring review';
|
||||||
|
COMMENT ON TABLE fraud_cases IS 'Confirmed fraud cases under investigation';
|
||||||
|
COMMENT ON TABLE merchants IS 'Merchant directory with risk ratings';
|
||||||
|
COMMENT ON TABLE cards IS 'Payment cards linked to accounts';
|
||||||
|
COMMENT ON TABLE devices IS 'Device fingerprints for fraud detection';
|
||||||
|
COMMENT ON TABLE login_sessions IS 'Login history for account takeover detection';
|
||||||
|
|
||||||
@@ -0,0 +1,256 @@
|
|||||||
|
-- ============================================================================
|
||||||
|
-- Seed Data for Reference Tables
|
||||||
|
-- ============================================================================
|
||||||
|
-- This script is IDEMPOTENT - it will delete and recreate all reference data
|
||||||
|
-- ============================================================================
|
||||||
|
|
||||||
|
-- Clear existing reference data (in reverse dependency order)
|
||||||
|
TRUNCATE TABLE suspicious_activity_reports CASCADE;
|
||||||
|
TRUNCATE TABLE case_alerts CASCADE;
|
||||||
|
TRUNCATE TABLE case_transactions CASCADE;
|
||||||
|
TRUNCATE TABLE fraud_cases CASCADE;
|
||||||
|
TRUNCATE TABLE alerts CASCADE;
|
||||||
|
TRUNCATE TABLE transfers CASCADE;
|
||||||
|
TRUNCATE TABLE beneficiaries CASCADE;
|
||||||
|
TRUNCATE TABLE transactions CASCADE;
|
||||||
|
TRUNCATE TABLE login_sessions CASCADE;
|
||||||
|
TRUNCATE TABLE devices CASCADE;
|
||||||
|
TRUNCATE TABLE cards CASCADE;
|
||||||
|
TRUNCATE TABLE accounts CASCADE;
|
||||||
|
TRUNCATE TABLE customer_relationships CASCADE;
|
||||||
|
TRUNCATE TABLE customers CASCADE;
|
||||||
|
TRUNCATE TABLE merchants CASCADE;
|
||||||
|
TRUNCATE TABLE merchant_categories RESTART IDENTITY CASCADE;
|
||||||
|
TRUNCATE TABLE fraud_types RESTART IDENTITY CASCADE;
|
||||||
|
TRUNCATE TABLE transaction_types RESTART IDENTITY CASCADE;
|
||||||
|
TRUNCATE TABLE countries RESTART IDENTITY CASCADE;
|
||||||
|
|
||||||
|
-- Insert Countries
|
||||||
|
INSERT INTO countries (country_code, country_name, region, risk_level) VALUES
|
||||||
|
('US', 'United States', 'North America', 'LOW'),
|
||||||
|
('CA', 'Canada', 'North America', 'LOW'),
|
||||||
|
('GB', 'United Kingdom', 'Europe', 'LOW'),
|
||||||
|
('DE', 'Germany', 'Europe', 'LOW'),
|
||||||
|
('FR', 'France', 'Europe', 'LOW'),
|
||||||
|
('IT', 'Italy', 'Europe', 'MEDIUM'),
|
||||||
|
('ES', 'Spain', 'Europe', 'MEDIUM'),
|
||||||
|
('AU', 'Australia', 'Oceania', 'LOW'),
|
||||||
|
('JP', 'Japan', 'Asia', 'LOW'),
|
||||||
|
('CN', 'China', 'Asia', 'MEDIUM'),
|
||||||
|
('IN', 'India', 'Asia', 'MEDIUM'),
|
||||||
|
('BR', 'Brazil', 'South America', 'MEDIUM'),
|
||||||
|
('MX', 'Mexico', 'North America', 'MEDIUM'),
|
||||||
|
('RU', 'Russia', 'Europe', 'HIGH'),
|
||||||
|
('NG', 'Nigeria', 'Africa', 'HIGH'),
|
||||||
|
('PK', 'Pakistan', 'Asia', 'HIGH'),
|
||||||
|
('IR', 'Iran', 'Middle East', 'CRITICAL'),
|
||||||
|
('KP', 'North Korea', 'Asia', 'CRITICAL'),
|
||||||
|
('SY', 'Syria', 'Middle East', 'CRITICAL'),
|
||||||
|
('VE', 'Venezuela', 'South America', 'HIGH'),
|
||||||
|
('CU', 'Cuba', 'Caribbean', 'HIGH'),
|
||||||
|
('MM', 'Myanmar', 'Asia', 'HIGH'),
|
||||||
|
('AF', 'Afghanistan', 'Asia', 'CRITICAL'),
|
||||||
|
('IQ', 'Iraq', 'Middle East', 'HIGH'),
|
||||||
|
('LY', 'Libya', 'Africa', 'HIGH'),
|
||||||
|
('SD', 'Sudan', 'Africa', 'HIGH'),
|
||||||
|
('SO', 'Somalia', 'Africa', 'CRITICAL'),
|
||||||
|
('YE', 'Yemen', 'Middle East', 'CRITICAL'),
|
||||||
|
('ZW', 'Zimbabwe', 'Africa', 'HIGH'),
|
||||||
|
('NL', 'Netherlands', 'Europe', 'LOW'),
|
||||||
|
('SE', 'Sweden', 'Europe', 'LOW'),
|
||||||
|
('NO', 'Norway', 'Europe', 'LOW'),
|
||||||
|
('DK', 'Denmark', 'Europe', 'LOW'),
|
||||||
|
('FI', 'Finland', 'Europe', 'LOW'),
|
||||||
|
('CH', 'Switzerland', 'Europe', 'LOW'),
|
||||||
|
('SG', 'Singapore', 'Asia', 'LOW'),
|
||||||
|
('HK', 'Hong Kong', 'Asia', 'MEDIUM'),
|
||||||
|
('KR', 'South Korea', 'Asia', 'LOW'),
|
||||||
|
('TW', 'Taiwan', 'Asia', 'LOW'),
|
||||||
|
('NZ', 'New Zealand', 'Oceania', 'LOW');
|
||||||
|
|
||||||
|
-- Insert Merchant Categories (based on MCC codes)
|
||||||
|
INSERT INTO merchant_categories (category_code, category_name, description, risk_weight) VALUES
|
||||||
|
('5411', 'Grocery Stores', 'Supermarkets and grocery stores', 0.50),
|
||||||
|
('5812', 'Restaurants', 'Eating places and restaurants', 0.60),
|
||||||
|
('5541', 'Gas Stations', 'Service stations and fuel', 0.55),
|
||||||
|
('5311', 'Department Stores', 'General merchandise stores', 0.70),
|
||||||
|
('5912', 'Pharmacies', 'Drug stores and pharmacies', 0.50),
|
||||||
|
('5999', 'Miscellaneous Retail', 'Specialty retail stores', 0.80),
|
||||||
|
('5732', 'Electronics', 'Electronics and computer stores', 1.20),
|
||||||
|
('5651', 'Clothing', 'Family clothing stores', 0.75),
|
||||||
|
('5814', 'Fast Food', 'Quick service restaurants', 0.60),
|
||||||
|
('5942', 'Books', 'Book stores', 0.65),
|
||||||
|
('5945', 'Hobby Shops', 'Hobby, toy, and game shops', 0.70),
|
||||||
|
('5971', 'Art Dealers', 'Art dealers and galleries', 1.50),
|
||||||
|
('5993', 'Cigar Stores', 'Cigar stores and stands', 1.10),
|
||||||
|
('5995', 'Pet Shops', 'Pet shops and supplies', 0.70),
|
||||||
|
('7011', 'Hotels', 'Lodging and hotels', 0.90),
|
||||||
|
('7512', 'Car Rental', 'Automobile rental agencies', 1.00),
|
||||||
|
('7523', 'Parking', 'Parking lots and garages', 0.60),
|
||||||
|
('7832', 'Movie Theaters', 'Motion picture theaters', 0.65),
|
||||||
|
('7922', 'Theatrical Producers', 'Theatrical producers and ticket agencies', 0.80),
|
||||||
|
('7991', 'Tourist Attractions', 'Tourist attractions and exhibits', 0.75),
|
||||||
|
('7995', 'Gambling', 'Betting and casino gambling', 2.50),
|
||||||
|
('5816', 'Digital Goods', 'Digital goods and games', 1.80),
|
||||||
|
('5967', 'Direct Marketing', 'Direct marketing and inbound telemarketing', 1.90),
|
||||||
|
('5966', 'Direct Marketing', 'Outbound telemarketing merchants', 2.00),
|
||||||
|
('6051', 'Cryptocurrency', 'Cryptocurrency and digital currency', 3.00),
|
||||||
|
('6211', 'Securities', 'Securities brokers and dealers', 1.50),
|
||||||
|
('6300', 'Insurance', 'Insurance sales and underwriting', 1.20),
|
||||||
|
('6513', 'Real Estate', 'Real estate agents and managers', 1.30),
|
||||||
|
('7273', 'Dating Services', 'Dating and escort services', 2.20),
|
||||||
|
('7297', 'Massage Parlors', 'Massage parlors', 2.50),
|
||||||
|
('7995', 'Online Gambling', 'Online gambling and betting', 3.50),
|
||||||
|
('5094', 'Precious Metals', 'Precious stones and metals', 2.80),
|
||||||
|
('5933', 'Pawn Shops', 'Pawn shops', 2.60),
|
||||||
|
('5960', 'Mail Order', 'Direct marketing and mail order', 1.70),
|
||||||
|
('4829', 'Wire Transfer', 'Money transfer services', 2.40);
|
||||||
|
|
||||||
|
-- Insert Transaction Types
|
||||||
|
INSERT INTO transaction_types (type_code, type_name, description, requires_merchant) VALUES
|
||||||
|
('PURCHASE', 'Purchase', 'Card purchase at merchant', TRUE),
|
||||||
|
('ATM_WITHDRAWAL', 'ATM Withdrawal', 'Cash withdrawal from ATM', FALSE),
|
||||||
|
('DEPOSIT', 'Deposit', 'Cash or check deposit', FALSE),
|
||||||
|
('TRANSFER_OUT', 'Transfer Out', 'Outgoing transfer', FALSE),
|
||||||
|
('TRANSFER_IN', 'Transfer In', 'Incoming transfer', FALSE),
|
||||||
|
('PAYMENT', 'Bill Payment', 'Bill payment transaction', TRUE),
|
||||||
|
('REFUND', 'Refund', 'Merchant refund', TRUE),
|
||||||
|
('FEE', 'Fee', 'Bank fee or charge', FALSE),
|
||||||
|
('INTEREST', 'Interest', 'Interest credit', FALSE),
|
||||||
|
('WIRE_OUT', 'Wire Transfer Out', 'Outgoing wire transfer', FALSE),
|
||||||
|
('WIRE_IN', 'Wire Transfer In', 'Incoming wire transfer', FALSE),
|
||||||
|
('CHECK', 'Check Payment', 'Check payment', FALSE),
|
||||||
|
('DIRECT_DEBIT', 'Direct Debit', 'Automated direct debit', TRUE),
|
||||||
|
('CASH_ADVANCE', 'Cash Advance', 'Credit card cash advance', FALSE),
|
||||||
|
('BALANCE_TRANSFER', 'Balance Transfer', 'Credit card balance transfer', FALSE);
|
||||||
|
|
||||||
|
-- Insert Fraud Types
|
||||||
|
INSERT INTO fraud_types (fraud_code, fraud_name, description, severity) VALUES
|
||||||
|
('CARD_NOT_PRESENT', 'Card Not Present Fraud', 'Fraudulent online or phone transactions', 'HIGH'),
|
||||||
|
('CARD_STOLEN', 'Stolen Card', 'Transactions using stolen physical card', 'HIGH'),
|
||||||
|
('ACCOUNT_TAKEOVER', 'Account Takeover', 'Unauthorized access to customer account', 'CRITICAL'),
|
||||||
|
('IDENTITY_THEFT', 'Identity Theft', 'Fraudulent account opened with stolen identity', 'CRITICAL'),
|
||||||
|
('FRIENDLY_FRAUD', 'Friendly Fraud', 'Customer disputes legitimate transaction', 'MEDIUM'),
|
||||||
|
('MONEY_MULE', 'Money Mule', 'Account used to launder money', 'CRITICAL'),
|
||||||
|
('SYNTHETIC_IDENTITY', 'Synthetic Identity', 'Fake identity using real and fake information', 'CRITICAL'),
|
||||||
|
('BUST_OUT', 'Bust Out Fraud', 'Building credit then maxing out and disappearing', 'HIGH'),
|
||||||
|
('REFUND_FRAUD', 'Refund Fraud', 'Fraudulent refund requests', 'MEDIUM'),
|
||||||
|
('CHARGEBACK_FRAUD', 'Chargeback Fraud', 'Abusing chargeback process', 'MEDIUM'),
|
||||||
|
('ATM_SKIMMING', 'ATM Skimming', 'Card data stolen via ATM skimmer', 'HIGH'),
|
||||||
|
('PHISHING', 'Phishing', 'Credentials stolen via phishing attack', 'HIGH'),
|
||||||
|
('SIM_SWAP', 'SIM Swap', 'Phone number hijacked for 2FA bypass', 'CRITICAL'),
|
||||||
|
('CHECK_FRAUD', 'Check Fraud', 'Fraudulent or altered checks', 'MEDIUM'),
|
||||||
|
('WIRE_FRAUD', 'Wire Fraud', 'Fraudulent wire transfer', 'CRITICAL'),
|
||||||
|
('STRUCTURING', 'Structuring', 'Breaking up transactions to avoid reporting', 'HIGH'),
|
||||||
|
('SMURFING', 'Smurfing', 'Using multiple people to structure transactions', 'HIGH'),
|
||||||
|
('TRADE_BASED', 'Trade-Based Money Laundering', 'Using trade to launder money', 'CRITICAL'),
|
||||||
|
('SHELL_COMPANY', 'Shell Company', 'Using fake companies for fraud', 'CRITICAL'),
|
||||||
|
('INVOICE_FRAUD', 'Invoice Fraud', 'Fraudulent invoicing schemes', 'HIGH');
|
||||||
|
|
||||||
|
-- Create a function to generate realistic transaction patterns
|
||||||
|
CREATE OR REPLACE FUNCTION generate_fraud_score(
|
||||||
|
p_amount DECIMAL,
|
||||||
|
p_is_international BOOLEAN,
|
||||||
|
p_merchant_risk DECIMAL,
|
||||||
|
p_time_of_day INT,
|
||||||
|
p_is_online BOOLEAN
|
||||||
|
) RETURNS DECIMAL AS $$
|
||||||
|
DECLARE
|
||||||
|
v_score DECIMAL := 0;
|
||||||
|
BEGIN
|
||||||
|
-- Amount-based scoring
|
||||||
|
IF p_amount > 5000 THEN v_score := v_score + 20; END IF;
|
||||||
|
IF p_amount > 10000 THEN v_score := v_score + 30; END IF;
|
||||||
|
|
||||||
|
-- International transactions
|
||||||
|
IF p_is_international THEN v_score := v_score + 15; END IF;
|
||||||
|
|
||||||
|
-- Merchant risk
|
||||||
|
v_score := v_score + (p_merchant_risk * 10);
|
||||||
|
|
||||||
|
-- Time of day (late night transactions)
|
||||||
|
IF p_time_of_day >= 23 OR p_time_of_day <= 4 THEN v_score := v_score + 10; END IF;
|
||||||
|
|
||||||
|
-- Online transactions
|
||||||
|
IF p_is_online THEN v_score := v_score + 5; END IF;
|
||||||
|
|
||||||
|
-- Cap at 100
|
||||||
|
IF v_score > 100 THEN v_score := 100; END IF;
|
||||||
|
|
||||||
|
RETURN v_score;
|
||||||
|
END;
|
||||||
|
$$ LANGUAGE plpgsql;
|
||||||
|
|
||||||
|
-- Create a function to update account balances
|
||||||
|
CREATE OR REPLACE FUNCTION update_account_balance()
|
||||||
|
RETURNS TRIGGER AS $$
|
||||||
|
BEGIN
|
||||||
|
IF NEW.status = 'COMPLETED' THEN
|
||||||
|
IF TG_TABLE_NAME = 'transactions' THEN
|
||||||
|
-- Update based on transaction type
|
||||||
|
UPDATE accounts
|
||||||
|
SET current_balance = current_balance +
|
||||||
|
CASE
|
||||||
|
WHEN NEW.type_id IN (SELECT type_id FROM transaction_types WHERE type_code IN ('DEPOSIT', 'TRANSFER_IN', 'WIRE_IN', 'REFUND', 'INTEREST'))
|
||||||
|
THEN NEW.amount
|
||||||
|
ELSE -NEW.amount
|
||||||
|
END,
|
||||||
|
updated_at = CURRENT_TIMESTAMP
|
||||||
|
WHERE account_id = NEW.account_id;
|
||||||
|
END IF;
|
||||||
|
END IF;
|
||||||
|
RETURN NEW;
|
||||||
|
END;
|
||||||
|
$$ LANGUAGE plpgsql;
|
||||||
|
|
||||||
|
-- Create trigger for balance updates (commented out for bulk loading)
|
||||||
|
-- CREATE TRIGGER trg_update_balance
|
||||||
|
-- AFTER INSERT ON transactions
|
||||||
|
-- FOR EACH ROW
|
||||||
|
-- EXECUTE FUNCTION update_account_balance();
|
||||||
|
|
||||||
|
-- Create a function to auto-generate alerts for suspicious transactions
|
||||||
|
CREATE OR REPLACE FUNCTION check_suspicious_transaction()
|
||||||
|
RETURNS TRIGGER AS $$
|
||||||
|
DECLARE
|
||||||
|
v_alert_type VARCHAR(50);
|
||||||
|
v_description TEXT;
|
||||||
|
v_severity VARCHAR(20);
|
||||||
|
BEGIN
|
||||||
|
-- High amount transactions
|
||||||
|
IF NEW.amount > 10000 THEN
|
||||||
|
v_alert_type := 'AMOUNT_ANOMALY';
|
||||||
|
v_description := 'Large transaction amount: $' || NEW.amount;
|
||||||
|
v_severity := 'HIGH';
|
||||||
|
|
||||||
|
INSERT INTO alerts (transaction_id, customer_id, account_id, alert_type, severity, description, risk_score)
|
||||||
|
SELECT NEW.transaction_id, a.customer_id, NEW.account_id, v_alert_type, v_severity, v_description, NEW.fraud_score
|
||||||
|
FROM accounts a WHERE a.account_id = NEW.account_id;
|
||||||
|
END IF;
|
||||||
|
|
||||||
|
-- International transactions
|
||||||
|
IF NEW.is_international AND NEW.amount > 1000 THEN
|
||||||
|
v_alert_type := 'GEOGRAPHIC_ANOMALY';
|
||||||
|
v_description := 'International transaction: $' || NEW.amount;
|
||||||
|
v_severity := 'MEDIUM';
|
||||||
|
|
||||||
|
INSERT INTO alerts (transaction_id, customer_id, account_id, alert_type, severity, description, risk_score)
|
||||||
|
SELECT NEW.transaction_id, a.customer_id, NEW.account_id, v_alert_type, v_severity, v_description, NEW.fraud_score
|
||||||
|
FROM accounts a WHERE a.account_id = NEW.account_id;
|
||||||
|
END IF;
|
||||||
|
|
||||||
|
RETURN NEW;
|
||||||
|
END;
|
||||||
|
$$ LANGUAGE plpgsql;
|
||||||
|
|
||||||
|
-- Create trigger for alert generation (commented out for bulk loading)
|
||||||
|
-- CREATE TRIGGER trg_check_suspicious
|
||||||
|
-- AFTER INSERT ON transactions
|
||||||
|
-- FOR EACH ROW
|
||||||
|
-- WHEN (NEW.fraud_score > 50)
|
||||||
|
-- EXECUTE FUNCTION check_suspicious_transaction();
|
||||||
|
|
||||||
|
COMMENT ON FUNCTION generate_fraud_score IS 'Calculates fraud risk score based on transaction attributes';
|
||||||
|
COMMENT ON FUNCTION update_account_balance IS 'Automatically updates account balance after transaction';
|
||||||
|
COMMENT ON FUNCTION check_suspicious_transaction IS 'Generates alerts for suspicious transactions';
|
||||||
|
|
||||||
@@ -0,0 +1,114 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Database Setup Script - IDEMPOTENT
|
||||||
|
# ============================================================================
|
||||||
|
# This script sets up the complete fraud detection database
|
||||||
|
# It can be run multiple times safely - it will recreate everything
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
set -e # Exit on error
|
||||||
|
|
||||||
|
# Colors for output
|
||||||
|
RED='\033[0;31m'
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
YELLOW='\033[1;33m'
|
||||||
|
BLUE='\033[0;34m'
|
||||||
|
NC='\033[0m' # No Color
|
||||||
|
|
||||||
|
# Configuration from environment or defaults
|
||||||
|
DB_HOST="${POSTGRES_HOST:-localhost}"
|
||||||
|
DB_PORT="${POSTGRES_PORT:-5432}"
|
||||||
|
DB_NAME="${POSTGRES_DB:-fraud_detection}"
|
||||||
|
DB_USER="${POSTGRES_USER:-fraud_analyst}"
|
||||||
|
DB_PASSWORD="${POSTGRES_PASSWORD:-SecurePass123!}"
|
||||||
|
|
||||||
|
# Script directory
|
||||||
|
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
|
||||||
|
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
|
||||||
|
|
||||||
|
echo -e "${BLUE}============================================================================${NC}"
|
||||||
|
echo -e "${BLUE}Financial Fraud Detection Database - Setup Script${NC}"
|
||||||
|
echo -e "${BLUE}============================================================================${NC}"
|
||||||
|
echo -e "Database: ${GREEN}$DB_NAME${NC}"
|
||||||
|
echo -e "Host: ${GREEN}$DB_HOST:$DB_PORT${NC}"
|
||||||
|
echo -e "User: ${GREEN}$DB_USER${NC}"
|
||||||
|
echo -e "${BLUE}============================================================================${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Function to execute SQL command
|
||||||
|
execute_sql() {
|
||||||
|
PGPASSWORD=$DB_PASSWORD psql -h $DB_HOST -p $DB_PORT -U $DB_USER -d $DB_NAME -c "$1" 2>&1
|
||||||
|
}
|
||||||
|
|
||||||
|
# Function to execute SQL file
|
||||||
|
execute_sql_file() {
|
||||||
|
local file=$1
|
||||||
|
local description=$2
|
||||||
|
echo -e "${YELLOW}Executing: $description${NC}"
|
||||||
|
PGPASSWORD=$DB_PASSWORD psql -h $DB_HOST -p $DB_PORT -U $DB_USER -d $DB_NAME -f "$file" 2>&1
|
||||||
|
if [ $? -eq 0 ]; then
|
||||||
|
echo -e "${GREEN}✓ Success: $description${NC}"
|
||||||
|
else
|
||||||
|
echo -e "${RED}✗ Failed: $description${NC}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
}
|
||||||
|
|
||||||
|
# Wait for PostgreSQL to be ready
|
||||||
|
echo -e "${YELLOW}Waiting for PostgreSQL to be ready...${NC}"
|
||||||
|
max_attempts=30
|
||||||
|
attempt=0
|
||||||
|
until PGPASSWORD=$DB_PASSWORD psql -h $DB_HOST -p $DB_PORT -U $DB_USER -d $DB_NAME -c '\q' 2>/dev/null; do
|
||||||
|
attempt=$((attempt + 1))
|
||||||
|
if [ $attempt -ge $max_attempts ]; then
|
||||||
|
echo -e "${RED}✗ PostgreSQL is not available after $max_attempts attempts${NC}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo -e "${YELLOW}Waiting for PostgreSQL... (attempt $attempt/$max_attempts)${NC}"
|
||||||
|
sleep 2
|
||||||
|
done
|
||||||
|
echo -e "${GREEN}✓ PostgreSQL is ready${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Step 1: Create schema (drops and recreates all tables)
|
||||||
|
echo -e "${BLUE}[Step 1/3] Creating database schema...${NC}"
|
||||||
|
execute_sql_file "$PROJECT_ROOT/schema/01-create-tables.sql" "Creating tables, indexes, and constraints"
|
||||||
|
|
||||||
|
# Step 2: Load seed data (reference tables)
|
||||||
|
echo -e "${BLUE}[Step 2/3] Loading reference data...${NC}"
|
||||||
|
execute_sql_file "$PROJECT_ROOT/schema/02-seed-data.sql" "Loading countries, merchant categories, transaction types, and fraud types"
|
||||||
|
|
||||||
|
# Step 3: Verify setup
|
||||||
|
echo -e "${BLUE}[Step 3/3] Verifying database setup...${NC}"
|
||||||
|
echo -e "${YELLOW}Checking table counts...${NC}"
|
||||||
|
|
||||||
|
# Get table counts
|
||||||
|
table_count=$(execute_sql "SELECT COUNT(*) FROM information_schema.tables WHERE table_schema = 'public' AND table_type = 'BASE TABLE';" | grep -E '^\s*[0-9]+' | tr -d ' ')
|
||||||
|
echo -e "Tables created: ${GREEN}$table_count${NC}"
|
||||||
|
|
||||||
|
# Get reference data counts
|
||||||
|
country_count=$(execute_sql "SELECT COUNT(*) FROM countries;" | grep -E '^\s*[0-9]+' | tr -d ' ')
|
||||||
|
echo -e "Countries: ${GREEN}$country_count${NC}"
|
||||||
|
|
||||||
|
category_count=$(execute_sql "SELECT COUNT(*) FROM merchant_categories;" | grep -E '^\s*[0-9]+' | tr -d ' ')
|
||||||
|
echo -e "Merchant categories: ${GREEN}$category_count${NC}"
|
||||||
|
|
||||||
|
transaction_type_count=$(execute_sql "SELECT COUNT(*) FROM transaction_types;" | grep -E '^\s*[0-9]+' | tr -d ' ')
|
||||||
|
echo -e "Transaction types: ${GREEN}$transaction_type_count${NC}"
|
||||||
|
|
||||||
|
fraud_type_count=$(execute_sql "SELECT COUNT(*) FROM fraud_types;" | grep -E '^\s*[0-9]+' | tr -d ' ')
|
||||||
|
echo -e "Fraud types: ${GREEN}$fraud_type_count${NC}"
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo -e "${GREEN}============================================================================${NC}"
|
||||||
|
echo -e "${GREEN}✓ Database setup completed successfully!${NC}"
|
||||||
|
echo -e "${GREEN}============================================================================${NC}"
|
||||||
|
echo ""
|
||||||
|
echo -e "${YELLOW}Next steps:${NC}"
|
||||||
|
echo -e "1. Generate test data: ${BLUE}./scripts/generate-data.sh${NC}"
|
||||||
|
echo -e "2. Access DB-UI at: ${BLUE}http://localhost:3000${NC}"
|
||||||
|
echo -e "3. Start learning SQL with exercises in: ${BLUE}./exercises/${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
@@ -0,0 +1,210 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Setup Verification Script
|
||||||
|
# ============================================================================
|
||||||
|
# Verifies that the fraud detection database is properly set up
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
# Colors for output
|
||||||
|
RED='\033[0;31m'
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
YELLOW='\033[1;33m'
|
||||||
|
BLUE='\033[0;34m'
|
||||||
|
NC='\033[0m' # No Color
|
||||||
|
|
||||||
|
# Configuration
|
||||||
|
DB_HOST="${POSTGRES_HOST:-localhost}"
|
||||||
|
DB_PORT="${POSTGRES_PORT:-5432}"
|
||||||
|
DB_NAME="${POSTGRES_DB:-fraud_detection}"
|
||||||
|
DB_USER="${POSTGRES_USER:-fraud_analyst}"
|
||||||
|
DB_PASSWORD="${POSTGRES_PASSWORD:-SecurePass123!}"
|
||||||
|
|
||||||
|
echo -e "${BLUE}============================================================================${NC}"
|
||||||
|
echo -e "${BLUE}Financial Fraud Detection Database - Verification${NC}"
|
||||||
|
echo -e "${BLUE}============================================================================${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Function to execute SQL and get result
|
||||||
|
execute_sql() {
|
||||||
|
PGPASSWORD=$DB_PASSWORD psql -h $DB_HOST -p $DB_PORT -U $DB_USER -d $DB_NAME -t -c "$1" 2>/dev/null | xargs
|
||||||
|
}
|
||||||
|
|
||||||
|
# Check 1: Docker containers
|
||||||
|
echo -e "${YELLOW}[1/10] Checking Docker containers...${NC}"
|
||||||
|
if docker ps | grep -q "fraud_detection_db"; then
|
||||||
|
echo -e "${GREEN}✓ PostgreSQL container is running${NC}"
|
||||||
|
else
|
||||||
|
echo -e "${RED}✗ PostgreSQL container is not running${NC}"
|
||||||
|
echo -e "${YELLOW}Run: docker-compose up -d${NC}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if docker ps | grep -q "fraud_detection_ui"; then
|
||||||
|
echo -e "${GREEN}✓ DB-UI container is running${NC}"
|
||||||
|
else
|
||||||
|
echo -e "${RED}✗ DB-UI container is not running${NC}"
|
||||||
|
echo -e "${YELLOW}Run: docker-compose up -d${NC}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Check 2: Database connectivity
|
||||||
|
echo -e "${YELLOW}[2/10] Checking database connectivity...${NC}"
|
||||||
|
if PGPASSWORD=$DB_PASSWORD psql -h $DB_HOST -p $DB_PORT -U $DB_USER -d $DB_NAME -c '\q' 2>/dev/null; then
|
||||||
|
echo -e "${GREEN}✓ Can connect to database${NC}"
|
||||||
|
else
|
||||||
|
echo -e "${RED}✗ Cannot connect to database${NC}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Check 3: Tables exist
|
||||||
|
echo -e "${YELLOW}[3/10] Checking database schema...${NC}"
|
||||||
|
table_count=$(execute_sql "SELECT COUNT(*) FROM information_schema.tables WHERE table_schema = 'public' AND table_type = 'BASE TABLE';")
|
||||||
|
if [ "$table_count" -ge 20 ]; then
|
||||||
|
echo -e "${GREEN}✓ Schema created ($table_count tables)${NC}"
|
||||||
|
else
|
||||||
|
echo -e "${RED}✗ Schema incomplete (only $table_count tables)${NC}"
|
||||||
|
echo -e "${YELLOW}Run: ./scripts/setup-database.sh${NC}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Check 4: Reference data
|
||||||
|
echo -e "${YELLOW}[4/10] Checking reference data...${NC}"
|
||||||
|
country_count=$(execute_sql "SELECT COUNT(*) FROM countries;")
|
||||||
|
category_count=$(execute_sql "SELECT COUNT(*) FROM merchant_categories;")
|
||||||
|
type_count=$(execute_sql "SELECT COUNT(*) FROM transaction_types;")
|
||||||
|
fraud_type_count=$(execute_sql "SELECT COUNT(*) FROM fraud_types;")
|
||||||
|
|
||||||
|
if [ "$country_count" -ge 40 ]; then
|
||||||
|
echo -e "${GREEN}✓ Countries loaded ($country_count)${NC}"
|
||||||
|
else
|
||||||
|
echo -e "${RED}✗ Countries not loaded${NC}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$category_count" -ge 30 ]; then
|
||||||
|
echo -e "${GREEN}✓ Merchant categories loaded ($category_count)${NC}"
|
||||||
|
else
|
||||||
|
echo -e "${RED}✗ Merchant categories not loaded${NC}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$type_count" -ge 10 ]; then
|
||||||
|
echo -e "${GREEN}✓ Transaction types loaded ($type_count)${NC}"
|
||||||
|
else
|
||||||
|
echo -e "${RED}✗ Transaction types not loaded${NC}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$fraud_type_count" -ge 15 ]; then
|
||||||
|
echo -e "${GREEN}✓ Fraud types loaded ($fraud_type_count)${NC}"
|
||||||
|
else
|
||||||
|
echo -e "${RED}✗ Fraud types not loaded${NC}"
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Check 5: Customer data
|
||||||
|
echo -e "${YELLOW}[5/10] Checking customer data...${NC}"
|
||||||
|
customer_count=$(execute_sql "SELECT COUNT(*) FROM customers;")
|
||||||
|
if [ "$customer_count" -ge 10000 ]; then
|
||||||
|
echo -e "${GREEN}✓ Customers generated ($customer_count)${NC}"
|
||||||
|
else
|
||||||
|
echo -e "${YELLOW}⚠ Limited customer data ($customer_count)${NC}"
|
||||||
|
echo -e "${YELLOW}Run: ./data/generate_data.sh${NC}"
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Check 6: Account data
|
||||||
|
echo -e "${YELLOW}[6/10] Checking account data...${NC}"
|
||||||
|
account_count=$(execute_sql "SELECT COUNT(*) FROM accounts;")
|
||||||
|
if [ "$account_count" -ge 10000 ]; then
|
||||||
|
echo -e "${GREEN}✓ Accounts generated ($account_count)${NC}"
|
||||||
|
else
|
||||||
|
echo -e "${YELLOW}⚠ Limited account data ($account_count)${NC}"
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Check 7: Transaction data
|
||||||
|
echo -e "${YELLOW}[7/10] Checking transaction data...${NC}"
|
||||||
|
transaction_count=$(execute_sql "SELECT COUNT(*) FROM transactions;")
|
||||||
|
if [ "$transaction_count" -ge 100000 ]; then
|
||||||
|
echo -e "${GREEN}✓ Transactions generated ($transaction_count)${NC}"
|
||||||
|
else
|
||||||
|
echo -e "${YELLOW}⚠ Limited transaction data ($transaction_count)${NC}"
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Check 8: Fraud data
|
||||||
|
echo -e "${YELLOW}[8/10] Checking fraud detection data...${NC}"
|
||||||
|
alert_count=$(execute_sql "SELECT COUNT(*) FROM alerts;")
|
||||||
|
case_count=$(execute_sql "SELECT COUNT(*) FROM fraud_cases;")
|
||||||
|
|
||||||
|
if [ "$alert_count" -ge 100 ]; then
|
||||||
|
echo -e "${GREEN}✓ Alerts generated ($alert_count)${NC}"
|
||||||
|
else
|
||||||
|
echo -e "${YELLOW}⚠ Limited alert data ($alert_count)${NC}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ "$case_count" -ge 10 ]; then
|
||||||
|
echo -e "${GREEN}✓ Fraud cases generated ($case_count)${NC}"
|
||||||
|
else
|
||||||
|
echo -e "${YELLOW}⚠ Limited fraud case data ($case_count)${NC}"
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Check 9: Indexes
|
||||||
|
echo -e "${YELLOW}[9/10] Checking database indexes...${NC}"
|
||||||
|
index_count=$(execute_sql "SELECT COUNT(*) FROM pg_indexes WHERE schemaname = 'public';")
|
||||||
|
if [ "$index_count" -ge 20 ]; then
|
||||||
|
echo -e "${GREEN}✓ Indexes created ($index_count)${NC}"
|
||||||
|
else
|
||||||
|
echo -e "${YELLOW}⚠ Limited indexes ($index_count)${NC}"
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Check 10: DB-UI accessibility
|
||||||
|
echo -e "${YELLOW}[10/10] Checking DB-UI web interface...${NC}"
|
||||||
|
if curl -s -o /dev/null -w "%{http_code}" http://localhost:3000 | grep -q "200\|302"; then
|
||||||
|
echo -e "${GREEN}✓ DB-UI accessible at http://localhost:3000${NC}"
|
||||||
|
else
|
||||||
|
echo -e "${YELLOW}⚠ DB-UI may not be ready yet (still starting up)${NC}"
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Summary
|
||||||
|
echo -e "${BLUE}============================================================================${NC}"
|
||||||
|
echo -e "${GREEN}Verification Summary${NC}"
|
||||||
|
echo -e "${BLUE}============================================================================${NC}"
|
||||||
|
echo ""
|
||||||
|
echo -e "${YELLOW}Database Statistics:${NC}"
|
||||||
|
echo -e " Customers: ${GREEN}$customer_count${NC}"
|
||||||
|
echo -e " Accounts: ${GREEN}$account_count${NC}"
|
||||||
|
echo -e " Transactions: ${GREEN}$transaction_count${NC}"
|
||||||
|
echo -e " Alerts: ${GREEN}$alert_count${NC}"
|
||||||
|
echo -e " Fraud Cases: ${GREEN}$case_count${NC}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
if [ "$transaction_count" -ge 100000 ]; then
|
||||||
|
echo -e "${GREEN}✓ Database is ready for SQL learning!${NC}"
|
||||||
|
echo ""
|
||||||
|
echo -e "${YELLOW}Next steps:${NC}"
|
||||||
|
echo -e "1. Access DB-UI: ${BLUE}http://localhost:3000${NC}"
|
||||||
|
echo -e "2. Start learning: ${BLUE}exercises/01-basic-queries/README.md${NC}"
|
||||||
|
echo -e "3. Quick start guide: ${BLUE}docs/QUICKSTART.md${NC}"
|
||||||
|
else
|
||||||
|
echo -e "${YELLOW}⚠ Database has minimal data${NC}"
|
||||||
|
echo ""
|
||||||
|
echo -e "${YELLOW}To generate full dataset:${NC}"
|
||||||
|
echo -e " ${BLUE}./data/generate_data.sh${NC}"
|
||||||
|
echo ""
|
||||||
|
echo -e "${YELLOW}This will generate:${NC}"
|
||||||
|
echo -e " - 100,000 customers"
|
||||||
|
echo -e " - 150,000 accounts"
|
||||||
|
echo -e " - 5,000,000 transactions"
|
||||||
|
echo -e " - 50,000+ alerts"
|
||||||
|
echo -e " - 5,000+ fraud cases"
|
||||||
|
fi
|
||||||
|
echo ""
|
||||||
|
|
||||||
Reference in New Issue
Block a user