PSMinIO/README.md

PSMinIO

A comprehensive PowerShell module for MinIO object storage operations with enterprise-grade features. Built with custom REST API implementation for optimal PowerShell compatibility and performance.

🚀 Key Features

📦 Complete Object Management

• Single & Multipart Uploads: Automatic chunking for large files with resume capability
• Parallel Downloads: High-performance multipart downloads with progress tracking
• Bucket Directory Management: Create, manage, and remove nested folder structures
• Advanced Object Listing: Filter, sort, and paginate with comprehensive metadata

⚡ High-Performance Operations

• Intelligent Progress Tracking: Real-time 3-layer progress bars for concurrent operations
• Optimized Multipart Uploads: Configurable chunk sizes and parallel upload limits
• Performance Monitoring: Transfer speeds, timing metrics, and completion statistics
• Memory Efficient: Streaming operations for large files without memory bloat

🗂️ Bucket Directory Ecosystem

• Multi-Level Folder Creation: Support for complex nested directory structures
• Path Format Flexibility: Handle Windows (\Folder\Sub) and Unix (/Folder/Sub) paths
• Recursive Operations: Create or remove entire directory trees
• Automatic Sanitization: Clean and normalize all path inputs

🔒 Enterprise Security & Management

• Bucket Policy Management: Complete CRUD operations for access policies
• Presigned URL Generation: Secure temporary access with configurable expiration
• Connection Management: Secure credential handling with SSL/TLS support
• Comprehensive Error Handling: Detailed error reporting and recovery options

📊 Advanced Features

• ZIP Archive Integration: Create and extract archives with progress tracking
• Metadata Management: Custom headers and content type handling
• Professional Logging: Timestamped verbose output with intelligent formatting
• PowerShell Integration: Full ShouldProcess support with -WhatIf and -Confirm

Installation

From PowerShell Gallery

# Install from PowerShell Gallery
Install-Module -Name PSMinIO -Scope CurrentUser

# Import the module
Import-Module PSMinIO

From Source

# Clone the repository
git clone https://github.com/Grace-Solutions/PSMinIO.git
cd PSMinIO

# Build the module (automatically updates version)
.\scripts\Build.ps1

# Import the module
Import-Module .\Module\PSMinIO\PSMinIO.psd1

Direct Import

# Import from local path
Import-Module .\Module\PSMinIO\PSMinIO.psd1

🚀 Quick Start

Connection Setup

# Connect to MinIO server
Connect-MinIO -Endpoint "https://minio.example.com" -AccessKey "your-access-key" -SecretKey "your-secret-key"

# Connect with SSL verification disabled (development only)
Connect-MinIO -Endpoint "https://minio.local" -AccessKey "admin" -SecretKey "password" -SkipCertificateValidation

Bucket Management

# List all buckets
Get-MinIOBucket

# Create a new bucket
New-MinIOBucket -BucketName "my-bucket" -Region "us-east-1"

# Check if bucket exists
Test-MinIOBucketExists -BucketName "my-bucket"

Folder Management

# Create nested folder structure
New-MinIOBucketFolder -BucketName "my-bucket" -FolderPath "Documents/Projects/2024" -Recursive

# Remove folder and contents
Remove-MinIOBucketFolder -BucketName "my-bucket" -FolderPath "OldData" -Recursive -Force

File Upload Operations

# Simple file upload
New-MinIOObject -BucketName "my-bucket" -Path "C:\file.txt"

# Upload to specific bucket directory
New-MinIOObject -BucketName "my-bucket" -Path "C:\file.txt" -BucketDirectory "Documents/2024"

# Upload multiple files with progress
New-MinIOObject -BucketName "my-bucket" -Path @("file1.txt", "file2.txt") -Verbose

# Upload directory recursively
New-MinIOObject -BucketName "my-bucket" -Directory "C:\MyFolder" -Recursive -BucketDirectory "Backup"

# Large file multipart upload
New-MinIOObjectMultipart -BucketName "my-bucket" -FilePath "C:\large-file.zip" -ChunkSize 64MB -MaxParallelUploads 4 -BucketDirectory "Archives"

File Download Operations

# Simple download
Get-MinIOObjectContent -BucketName "my-bucket" -ObjectName "file.txt" -LocalPath "C:\Downloads\file.txt"

# Multipart download for large files
Get-MinIOObjectContentMultipart -BucketName "my-bucket" -ObjectName "large-file.zip" -LocalPath "C:\Downloads\" -ChunkSize 32MB

Object Listing & Management

# List all objects
Get-MinIOObject -BucketName "my-bucket"

# Advanced filtering and sorting
Get-MinIOObject -BucketName "my-bucket" -Prefix "documents/" -SortBy "LastModified" -Descending -MaxKeys 100 -ExcludeDirectories

📋 Available Cmdlets

🔌 Connection Management

• Connect-MinIO - Establish connection to MinIO server with SSL support and certificate validation options

🗂️ Bucket Management

• Get-MinIOBucket - List all buckets with filtering and metadata
• New-MinIOBucket - Create new buckets with region specification
• Test-MinIOBucketExists - Check bucket existence efficiently

📁 Bucket Directory Management

• New-MinIOBucketFolder - Create nested folder structures with multi-level support
• Remove-MinIOBucketFolder - Remove folders with recursive deletion options

📦 Object Operations

• Get-MinIOObject - List objects with advanced filtering, sorting, and pagination
• New-MinIOObject - Upload single/multiple files with directory support and progress tracking
• Get-MinIOObjectContent - Download objects with automatic directory creation

⚡ High-Performance Operations

• New-MinIOObjectMultipart - Large file uploads with chunking, parallelization, and resume capability
• Get-MinIOObjectContentMultipart - High-speed downloads with parallel chunk processing

📦 Archive Management

• New-MinIOZipArchive - Create ZIP archives with progress tracking and metadata
• Get-MinIOZipArchive - Extract and analyze ZIP archive contents

🔗 URL Management

• New-MinIOPresignedUrl - Generate secure temporary URLs for object access
• Get-MinIOPresignedUrl - Retrieve presigned URLs with configurable expiration

🔒 Security & Policy Management

• Get-MinIOBucketPolicy - Retrieve bucket access policies
• Set-MinIOBucketPolicy - Configure bucket access policies
• Remove-MinIOBucketPolicy - Remove bucket access policies

💡 Advanced Examples

📁 Comprehensive Bucket Directory Management

# Create complex nested structure
New-MinIOBucketFolder -BucketName "enterprise-data" -FolderPath "Departments/IT/Projects/2024/Q1" -Recursive -Verbose

# Upload files to specific directories
New-MinIOObject -BucketName "enterprise-data" -Path "C:\Reports\*.pdf" -BucketDirectory "Departments/Finance/Reports/2024"

# Multipart upload to nested directory
New-MinIOObjectMultipart -BucketName "enterprise-data" -FilePath "C:\Backups\database.bak" -BucketDirectory "Backups/Database/2024" -ChunkSize 128MB -MaxParallelUploads 6

# Clean up old directories
Remove-MinIOBucketFolder -BucketName "enterprise-data" -FolderPath "Departments/IT/Projects/2023" -Recursive -Force

⚡ High-Performance Multipart Operations

# Large file upload with optimal settings
New-MinIOObjectMultipart -BucketName "media-storage" -FilePath "C:\Videos\4K-movie.mkv" -ChunkSize 256MB -MaxParallelUploads 8 -ContentType "video/x-matroska" -Verbose

# Resume interrupted upload
$uploadResult = New-MinIOObjectMultipart -BucketName "media-storage" -FilePath "C:\Videos\4K-movie.mkv" -ResumeUploadId "abc123..." -CompletedParts $previousParts

# High-speed download with chunking
Get-MinIOObjectContentMultipart -BucketName "media-storage" -ObjectName "4K-movie.mkv" -LocalPath "C:\Downloads\" -ChunkSize 128MB -MaxParallelDownloads 6

• Progress tracking and timing information
• New-MinIOObjectChunked - Chunked uploads for large files
  • Configurable chunk sizes (1MB minimum)
  • Resume capability and multi-layer progress tracking
  • Automatic directory creation
• Get-MinIOObjectContent - Download objects with progress tracking
• Get-MinIOObjectContentChunked - Chunked downloads for large files
• Remove-MinIOObject - Delete objects with confirmation prompts
• New-MinIOFolder - Create folder structures in buckets

Security & Policy Management

• Get-MinIOBucketPolicy - Retrieve bucket access policies
• Set-MinIOBucketPolicy - Configure bucket access policies

Monitoring & Statistics

• Get-MinIOStats - Comprehensive server and bucket statistics with object counting limits

Key Features in Detail

🗂️ Advanced Directory Support

• Nested Folder Creation: Automatically create multi-level directory structures (e.g., documents/2025/january/reports)
• BucketDirectory Parameter: Specify target directories for uploads without manual folder creation
• Clean Directory Handling: Non-critical directory creation attempts with graceful fallback

⚡ Chunked Operations

• Large File Support: Handle files of any size with configurable chunk sizes
• Resume Capability: Interrupted transfers can be resumed (future enhancement)
• Multi-Layer Progress: Track collection progress, file progress, and chunk progress simultaneously
• Performance Optimization: Optimal chunk sizes for different network conditions

📊 Enhanced Object Listing

# Advanced filtering and sorting examples
Get-MinIOObject -BucketName "data" -Prefix "logs/" -SortBy "LastModified" -Descending -MaxObjects 50
Get-MinIOObject -BucketName "media" -ObjectsOnly -SortBy "Size" -Descending
Get-MinIOObject -BucketName "docs" -ObjectName "specific-file.pdf"

⏱️ Performance Metrics

All operations provide detailed timing information:

• Duration: Precise operation timing
• Transfer Speed: Formatted speed reporting (B/s, KB/s, MB/s, GB/s, TB/s)
• Progress Tracking: Real-time progress updates during transfers

Examples

See the scripts/examples directory for comprehensive usage examples:

• Basic Operations: Connection, bucket management, simple uploads/downloads
• Advanced Scenarios: Chunked transfers, directory management, bulk operations
• Enterprise Patterns: Policy management, monitoring, and automation scripts

Requirements

• PowerShell: 5.1+ or PowerShell 7+
• .NET Framework: 4.7.2+ (for PowerShell 5.1) or .NET Core/.NET 5+ (for PowerShell 7+)
• MinIO Server: Compatible with MinIO and Amazon S3 APIs

🚀 Performance Features

• 🔄 Multi-threaded Operations: Parallel uploads and downloads for maximum throughput
• 🧠 Intelligent Progress Tracking: Real-time 3-layer progress bars with concurrent chunk visibility
• ⚡ Optimized Multipart Operations: Configurable chunk sizes (1MB to 5GB) with parallel processing
• 🔄 Resume Capability: Interrupted transfers can be resumed from the last completed chunk
• 💾 Memory Efficiency: Streaming operations minimize memory usage for large files
• 🌐 Connection Management: Efficient HTTP connection handling with SSL/TLS support
• 📊 Performance Monitoring: Transfer speeds, timing metrics, and completion statistics

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

License

This project is licensed under the GNU General Public License v3.0 - see the LICENSE file for details.

Support

• 📖 Documentation: See docs/USAGE.md for detailed usage instructions
• 🐛 Issues: Report bugs and request features via GitHub Issues
• 💬 Discussions: Join the community discussions for questions and tips