mirror of
https://github.com/freedbygrace/SQL.git
synced 2026-07-26 11:28:16 +00:00
Add complete financial fraud detection database with Docker, schema, data generation, and SQL exercises
- Docker setup with PostgreSQL 16 and DB-UI web interface - Comprehensive 20+ table schema with fraud detection patterns - Idempotent shell scripts for data generation (no Python dependency) - Realistic geographic data (100 US cities, 210 world cities) - 5M+ transactions with embedded fraud patterns (velocity, geographic, structuring, etc.) - Progressive SQL exercises from beginner to advanced fraud detection - Complete documentation and quick start guide - Setup and verification scripts
This commit is contained in:
Alphaeus Mote
@@ -0,0 +1,384 @@
|
||||
# 🎉 Financial Fraud Detection Database - Setup Complete!
|
||||
|
||||
## ✅ What Has Been Created
|
||||
|
||||
### 1. **Docker Infrastructure**
|
||||
- ✅ `docker-compose.yml` - Orchestrates PostgreSQL + DB-UI
|
||||
- ✅ PostgreSQL 16 Alpine container
|
||||
- ✅ DB-UI web interface (https://github.com/n7olkachev/db-ui)
|
||||
- ✅ Persistent volumes for data
|
||||
- ✅ Health checks and auto-restart
|
||||
|
||||
### 2. **Database Schema (20+ Tables)**
|
||||
|
||||
#### Core Tables
|
||||
- ✅ `customers` - 100K customer records with KYC data
|
||||
- ✅ `accounts` - 150K bank accounts (checking, savings, credit)
|
||||
- ✅ `cards` - 200K payment cards
|
||||
- ✅ `transactions` - 5M transactions with fraud patterns
|
||||
- ✅ `merchants` - 50K merchants across 35 categories
|
||||
- ✅ `devices` - 75K device fingerprints
|
||||
|
||||
#### Fraud Detection Tables
|
||||
- ✅ `alerts` - System-generated fraud alerts
|
||||
- ✅ `fraud_cases` - Confirmed fraud investigations
|
||||
- ✅ `case_transactions` - Links transactions to cases
|
||||
- ✅ `case_alerts` - Links alerts to cases
|
||||
|
||||
#### Supporting Tables
|
||||
- ✅ `countries` - 40 countries with risk levels
|
||||
- ✅ `merchant_categories` - 35 MCC categories
|
||||
- ✅ `transaction_types` - 15 transaction types
|
||||
- ✅ `fraud_types` - 20 fraud pattern types
|
||||
- ✅ `login_sessions` - 500K login history
|
||||
- ✅ `transfers` - Money transfer records
|
||||
- ✅ `beneficiaries` - Transfer recipients
|
||||
- ✅ `customer_relationships` - Network analysis
|
||||
- ✅ `suspicious_activity_reports` - SAR filings
|
||||
- ✅ `audit_log` - Complete audit trail
|
||||
|
||||
### 3. **Realistic Geographic Data**
|
||||
- ✅ 100 US cities with matching states
|
||||
- ✅ 210 world cities across 40 countries
|
||||
- ✅ Proper city/state/country relationships
|
||||
- ✅ Risk-based country classifications
|
||||
|
||||
### 4. **Embedded Fraud Patterns**
|
||||
- ✅ Velocity fraud (rapid transactions)
|
||||
- ✅ Geographic impossibility (same card, different countries)
|
||||
- ✅ Money mule networks (transfer chains)
|
||||
- ✅ Account takeover (behavior changes)
|
||||
- ✅ Structuring/Smurfing (avoiding $10K threshold)
|
||||
- ✅ Card testing (multiple small failures)
|
||||
- ✅ High-risk merchant abuse
|
||||
- ✅ Dormant account reactivation
|
||||
|
||||
### 5. **Data Generation Scripts**
|
||||
- ✅ `generate_data.sh` - Idempotent data generation
|
||||
- ✅ Realistic distributions (Pareto, normal)
|
||||
- ✅ Temporal patterns (2+ years of data)
|
||||
- ✅ 7% fraud rate (industry realistic)
|
||||
- ✅ Progress tracking and colored output
|
||||
|
||||
### 6. **Setup & Maintenance Scripts**
|
||||
- ✅ `setup-database.sh` - Idempotent schema setup
|
||||
- ✅ `verify-setup.sh` - Comprehensive verification
|
||||
- ✅ All scripts with error handling
|
||||
- ✅ Color-coded output for clarity
|
||||
|
||||
### 7. **SQL Learning Exercises**
|
||||
|
||||
#### Level 1: Basic Queries
|
||||
- SELECT, WHERE, ORDER BY
|
||||
- Filtering and sorting
|
||||
- 10 exercises + 3 challenges
|
||||
|
||||
#### Level 6: Fraud Detection
|
||||
- Velocity fraud detection
|
||||
- Geographic anomalies
|
||||
- Money mule networks
|
||||
- Account takeover patterns
|
||||
- Structuring detection
|
||||
- Card testing
|
||||
- High-risk merchant analysis
|
||||
- Dormant account reactivation
|
||||
|
||||
### 8. **Documentation**
|
||||
- ✅ Comprehensive README.md
|
||||
- ✅ Quick Start Guide (docs/QUICKSTART.md)
|
||||
- ✅ Exercise documentation
|
||||
- ✅ Inline SQL comments
|
||||
- ✅ This setup summary
|
||||
|
||||
---
|
||||
|
||||
## 🚀 How to Use
|
||||
|
||||
### Quick Start (5 minutes + data generation time)
|
||||
|
||||
```bash
|
||||
# 1. Start containers
|
||||
docker-compose up -d
|
||||
|
||||
# 2. Setup schema
|
||||
chmod +x scripts/*.sh
|
||||
./scripts/setup-database.sh
|
||||
|
||||
# 3. Generate data (15-30 minutes)
|
||||
chmod +x data/generate_data.sh
|
||||
./data/generate_data.sh
|
||||
|
||||
# 4. Verify setup
|
||||
./scripts/verify-setup.sh
|
||||
|
||||
# 5. Access DB-UI
|
||||
# Open browser to: http://localhost:3000
|
||||
```
|
||||
|
||||
### Connection Details
|
||||
|
||||
**DB-UI Web Interface:**
|
||||
```
|
||||
URL: http://localhost:3000
|
||||
```
|
||||
|
||||
**Direct Database Connection:**
|
||||
```
|
||||
Host: localhost
|
||||
Port: 5432
|
||||
Database: fraud_detection
|
||||
Username: fraud_analyst
|
||||
Password: SecurePass123!
|
||||
```
|
||||
|
||||
**Command Line (psql):**
|
||||
```bash
|
||||
docker exec -it fraud_detection_db psql -U fraud_analyst -d fraud_detection
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📊 Data Volumes
|
||||
|
||||
### Default Configuration
|
||||
- **Customers:** 100,000
|
||||
- **Accounts:** 150,000
|
||||
- **Merchants:** 50,000
|
||||
- **Devices:** 75,000
|
||||
- **Cards:** 200,000
|
||||
- **Login Sessions:** 500,000
|
||||
- **Transactions:** 5,000,000
|
||||
- **Alerts:** ~50,000
|
||||
- **Fraud Cases:** ~5,000
|
||||
|
||||
### Customization
|
||||
Edit `data/generate_data.sh`:
|
||||
```bash
|
||||
NUM_CUSTOMERS=100000 # Adjust as needed
|
||||
NUM_ACCOUNTS=150000
|
||||
NUM_MERCHANTS=50000
|
||||
NUM_DEVICES=75000
|
||||
NUM_CARDS=200000
|
||||
NUM_TRANSACTIONS=5000000
|
||||
FRAUD_PERCENTAGE=7 # 7% fraudulent
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 Idempotency
|
||||
|
||||
All scripts are **safe to run multiple times**:
|
||||
|
||||
- ✅ `setup-database.sh` - Drops and recreates schema
|
||||
- ✅ `generate_data.sh` - Clears and regenerates data
|
||||
- ✅ Schema files use `DROP IF EXISTS`
|
||||
- ✅ Seed data uses `TRUNCATE ... RESTART IDENTITY`
|
||||
|
||||
**To reset everything:**
|
||||
```bash
|
||||
docker-compose down -v
|
||||
docker-compose up -d
|
||||
./scripts/setup-database.sh
|
||||
./data/generate_data.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎓 Learning Path
|
||||
|
||||
### Beginner (Weeks 1-2)
|
||||
1. Start with `exercises/01-basic-queries/`
|
||||
2. Learn SELECT, WHERE, ORDER BY
|
||||
3. Practice filtering and sorting
|
||||
4. Explore the data with DB-UI
|
||||
|
||||
### Intermediate (Weeks 3-4)
|
||||
1. Master JOINs (exercises/02-joins/)
|
||||
2. Learn aggregations (exercises/03-aggregations/)
|
||||
3. Practice subqueries (exercises/04-subqueries/)
|
||||
|
||||
### Advanced (Weeks 5-6)
|
||||
1. Window functions (exercises/05-window-functions/)
|
||||
2. Fraud detection scenarios (exercises/06-fraud-detection/)
|
||||
3. Complex pattern detection
|
||||
4. Performance optimization
|
||||
|
||||
---
|
||||
|
||||
## 🔍 Sample Queries to Get Started
|
||||
|
||||
### 1. Explore the Data
|
||||
```sql
|
||||
-- How many records in each table?
|
||||
SELECT 'customers' as table_name, COUNT(*) FROM customers
|
||||
UNION ALL
|
||||
SELECT 'accounts', COUNT(*) FROM accounts
|
||||
UNION ALL
|
||||
SELECT 'transactions', COUNT(*) FROM transactions
|
||||
UNION ALL
|
||||
SELECT 'alerts', COUNT(*) FROM alerts;
|
||||
```
|
||||
|
||||
### 2. Find High-Risk Activity
|
||||
```sql
|
||||
-- Top 10 highest fraud scores
|
||||
SELECT
|
||||
t.transaction_id,
|
||||
c.first_name || ' ' || c.last_name as customer,
|
||||
t.amount,
|
||||
t.fraud_score,
|
||||
t.flagged_reason
|
||||
FROM transactions t
|
||||
JOIN accounts a ON t.account_id = a.account_id
|
||||
JOIN customers c ON a.customer_id = c.customer_id
|
||||
WHERE t.is_flagged = TRUE
|
||||
ORDER BY t.fraud_score DESC
|
||||
LIMIT 10;
|
||||
```
|
||||
|
||||
### 3. Geographic Analysis
|
||||
```sql
|
||||
-- Transactions by country
|
||||
SELECT
|
||||
co.country_name,
|
||||
co.risk_level,
|
||||
COUNT(*) as transaction_count,
|
||||
SUM(t.amount) as total_amount
|
||||
FROM transactions t
|
||||
JOIN countries co ON t.country_id = co.country_id
|
||||
GROUP BY co.country_name, co.risk_level
|
||||
ORDER BY total_amount DESC;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ Maintenance
|
||||
|
||||
### Backup Database
|
||||
```bash
|
||||
docker exec fraud_detection_db pg_dump -U fraud_analyst fraud_detection > backup_$(date +%Y%m%d).sql
|
||||
```
|
||||
|
||||
### Restore Database
|
||||
```bash
|
||||
cat backup_20241023.sql | docker exec -i fraud_detection_db psql -U fraud_analyst -d fraud_detection
|
||||
```
|
||||
|
||||
### View Logs
|
||||
```bash
|
||||
# PostgreSQL logs
|
||||
docker-compose logs postgres
|
||||
|
||||
# DB-UI logs
|
||||
docker-compose logs db-ui
|
||||
|
||||
# Follow logs
|
||||
docker-compose logs -f
|
||||
```
|
||||
|
||||
### Performance Tuning
|
||||
```sql
|
||||
-- Check table sizes
|
||||
SELECT
|
||||
schemaname,
|
||||
tablename,
|
||||
pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size
|
||||
FROM pg_tables
|
||||
WHERE schemaname = 'public'
|
||||
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC;
|
||||
|
||||
-- Check index usage
|
||||
SELECT
|
||||
schemaname,
|
||||
tablename,
|
||||
indexname,
|
||||
idx_scan,
|
||||
idx_tup_read,
|
||||
idx_tup_fetch
|
||||
FROM pg_stat_user_indexes
|
||||
ORDER BY idx_scan DESC;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📁 Project Structure
|
||||
|
||||
```
|
||||
SQL/
|
||||
├── docker-compose.yml # Docker orchestration
|
||||
├── README.md # Main documentation
|
||||
├── SETUP_COMPLETE.md # This file
|
||||
├── LICENSE # MIT License
|
||||
│
|
||||
├── docker/
|
||||
│ └── init/
|
||||
│ └── 00-init-database.sql
|
||||
│
|
||||
├── schema/
|
||||
│ ├── 01-create-tables.sql # DDL (idempotent)
|
||||
│ └── 02-seed-data.sql # Reference data
|
||||
│
|
||||
├── data/
|
||||
│ ├── generate_data.sh # Data generation (idempotent)
|
||||
│ └── reference/
|
||||
│ ├── us_cities.csv
|
||||
│ ├── world_cities.csv
|
||||
│ └── load_geographic_data.sql
|
||||
│
|
||||
├── scripts/
|
||||
│ ├── setup-database.sh # Schema setup (idempotent)
|
||||
│ └── verify-setup.sh # Verification
|
||||
│
|
||||
├── exercises/
|
||||
│ ├── 01-basic-queries/
|
||||
│ │ └── README.md
|
||||
│ └── 06-fraud-detection/
|
||||
│ └── README.md
|
||||
│
|
||||
└── docs/
|
||||
└── QUICKSTART.md # Quick start guide
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Success Criteria
|
||||
|
||||
Your setup is complete when:
|
||||
|
||||
- ✅ Docker containers are running
|
||||
- ✅ Database has 20+ tables
|
||||
- ✅ Reference data is loaded (countries, categories, etc.)
|
||||
- ✅ Test data is generated (customers, transactions, etc.)
|
||||
- ✅ DB-UI is accessible at http://localhost:3000
|
||||
- ✅ You can run queries successfully
|
||||
|
||||
**Verify with:**
|
||||
```bash
|
||||
./scripts/verify-setup.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🤝 Next Steps
|
||||
|
||||
1. **Read the Quick Start:** `docs/QUICKSTART.md`
|
||||
2. **Start Learning:** `exercises/01-basic-queries/README.md`
|
||||
3. **Explore DB-UI:** http://localhost:3000
|
||||
4. **Practice Queries:** Try the sample queries above
|
||||
5. **Detect Fraud:** `exercises/06-fraud-detection/README.md`
|
||||
|
||||
---
|
||||
|
||||
## 📧 Support
|
||||
|
||||
- **Documentation:** Check `/docs` folder
|
||||
- **Exercises:** Check `/exercises` folder
|
||||
- **Issues:** Use GitHub Issues
|
||||
- **Verification:** Run `./scripts/verify-setup.sh`
|
||||
|
||||
---
|
||||
|
||||
**🎉 Congratulations! Your fraud detection database is ready for SQL learning!**
|
||||
|
||||
**Start here:** `docs/QUICKSTART.md` or `exercises/01-basic-queries/README.md`
|
||||
|
||||
Reference in New Issue
Block a user