 Alphaeus Mote
|
e2090cc6ad
|
Add Python dependency and start CSV-based data generation
DEPENDENCY INSTALLER UPDATES:
- Added Python 3 installation check and installer
- Python is required for fast CSV data generation
- Updated step numbers from [1/4] to [1/5]
- Python installed before PostgreSQL client
- Supports Ubuntu, Debian, CentOS, RHEL, Fedora, Arch, macOS
DEPLOY SCRIPT UPDATES:
- Added automatic dependency checking (Step 3/8)
- Checks for python3, psql, and docker
- Automatically runs install-dependencies.sh if missing
- Updated all step numbers to reflect 8 total steps
- Step 3: Check dependencies (NEW)
- Step 4: Fix permissions (was Step 3)
- Step 5: Start containers (was Step 4)
- Step 6: Initialize schema (was Step 5)
- Step 7: Generate data (was Step 6)
- Step 8: Verify deployment (was Step 7)
CSV DATA GENERATION (IN PROGRESS):
- Created scripts/generate_csv_data.py
- Python-based CSV generator for bulk import
- Much faster than SQL INSERT statements
- Generates realistic data with proper relationships
- Partial implementation (customers, accounts, merchants, cards)
- Next: Complete transactions and analytics data
RATIONALE:
- Current bash-based data generation has transaction commit issues
- Data shows as generated but doesn't persist (0 rows after completion)
- CSV + COPY command is PostgreSQL best practice for bulk loading
- 10-100x faster than INSERT statements
- More reliable - atomic COPY operations
- Easier to debug - can inspect CSV files
|
2025-10-24 12:43:39 -04:00 |
|