mirror of
https://github.com/Grace-Solutions/PSMinIO.git
synced 2026-07-26 06:48:13 +00:00
Create comprehensive PowerShell threading documentation and patterns
� PERMANENT THREADING KNOWLEDGE BASE:
✅ CREATED COMPREHENSIVE DOCUMENTATION:
• docs/POWERSHELL-THREADING-RULES.md - Complete threading guide
• Enhanced ThreadSafeProgressCollector with usage examples
• Added to project structure documentation
• Saved to AI memory for future reference
� CRITICAL RULES DOCUMENTED:
• PowerShell cmdlets can ONLY call Write-* methods from main thread
• Background threads must QUEUE updates, never call Write-* directly
• ProcessQueuedUpdates() must ONLY be called from main thread
• Common mistakes and error symptoms clearly identified
� DESIGN PATTERNS PROVIDED:
• Periodic processing pattern (every 1 second)
• Completion-based processing pattern
• Manual processing points pattern
• Correct Task.WaitAll usage with timeouts
� IMPLEMENTATION CHECKLIST:
• Pre-implementation checklist for new cmdlets
• Debugging tips and error identification
• Code examples for correct and incorrect patterns
• Testing guidelines for threading compliance
� PREVENTS FUTURE REGRESSIONS:
• Clear documentation of the 'golden rule'
• Examples of common threading violations
• Patterns for all background operation types
• Reference for all future PowerShell module development
This ensures we never repeat the same threading mistakes!
This commit is contained in:
PSMinIO Developer
@@ -0,0 +1,175 @@
|
||||
# PowerShell Cmdlet Threading Rules and Design Patterns
|
||||
|
||||
## 🚨 CRITICAL THREADING RULE
|
||||
|
||||
**PowerShell cmdlets can ONLY call Write-Progress, Write-Verbose, Write-Object, Write-Error from the main cmdlet thread.**
|
||||
|
||||
**❌ NEVER call these methods from:**
|
||||
- Background threads (`Task.Run`, `Task.Factory.StartNew`)
|
||||
- Async callbacks (`async/await` continuations)
|
||||
- Timer callbacks
|
||||
- Event handlers from other threads
|
||||
- Any thread other than the main cmdlet thread
|
||||
|
||||
**✅ ONLY call these methods from:**
|
||||
- `BeginProcessing()` override
|
||||
- `ProcessRecord()` override
|
||||
- `EndProcessing()` override
|
||||
- Main cmdlet thread execution
|
||||
|
||||
## 🔧 DESIGN PATTERN: ThreadSafeProgressCollector
|
||||
|
||||
### Problem
|
||||
When performing background operations (file uploads, downloads, processing), you need to report progress from background threads, but PowerShell doesn't allow direct calls to Write-Progress from those threads.
|
||||
|
||||
### Solution
|
||||
Use the **ThreadSafeProgressCollector** pattern:
|
||||
|
||||
```csharp
|
||||
// Background thread - QUEUE updates (thread-safe)
|
||||
_progressCollector.QueueProgressUpdate(activityId, "Processing", "Status", percentage);
|
||||
_progressCollector.QueueVerboseMessage("Processing file: {0}", fileName);
|
||||
|
||||
// Main thread - PROCESS queued updates
|
||||
_progressCollector.ProcessQueuedUpdates(); // Only call from main thread!
|
||||
```
|
||||
|
||||
### Implementation Pattern
|
||||
|
||||
```csharp
|
||||
public class MyLongRunningCmdlet : PSCmdlet
|
||||
{
|
||||
private ThreadSafeProgressCollector _progressCollector;
|
||||
|
||||
protected override void BeginProcessing()
|
||||
{
|
||||
_progressCollector = new ThreadSafeProgressCollector(this);
|
||||
}
|
||||
|
||||
protected override void ProcessRecord()
|
||||
{
|
||||
// Start background work
|
||||
var tasks = new List<Task>();
|
||||
|
||||
for (int i = 0; i < workItems.Count; i++)
|
||||
{
|
||||
var task = Task.Run(() =>
|
||||
{
|
||||
// ✅ CORRECT: Queue from background thread
|
||||
_progressCollector.QueueProgressUpdate(1, "Processing", $"Item {i}", progress);
|
||||
_progressCollector.QueueVerboseMessage("Processing item {0}", i);
|
||||
|
||||
// ❌ WRONG: Never call directly from background thread
|
||||
// WriteProgress(...); // This will cause threading errors!
|
||||
// WriteVerbose(...); // This will cause threading errors!
|
||||
});
|
||||
tasks.Add(task);
|
||||
}
|
||||
|
||||
// ✅ CORRECT: Process updates from main thread with periodic updates
|
||||
while (!Task.WaitAll(tasks.ToArray(), 1000)) // 1 second intervals
|
||||
{
|
||||
_progressCollector.ProcessQueuedUpdates(); // Safe on main thread
|
||||
}
|
||||
|
||||
// Final processing
|
||||
_progressCollector.ProcessQueuedUpdates();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 🚨 COMMON MISTAKES TO AVOID
|
||||
|
||||
### ❌ Mistake 1: Direct calls from background threads
|
||||
```csharp
|
||||
Task.Run(() =>
|
||||
{
|
||||
WriteProgress(...); // THREADING ERROR!
|
||||
WriteVerbose(...); // THREADING ERROR!
|
||||
});
|
||||
```
|
||||
|
||||
### ❌ Mistake 2: ProcessQueuedUpdates from background threads
|
||||
```csharp
|
||||
Task.Run(() =>
|
||||
{
|
||||
_progressCollector.QueueProgressUpdate(...);
|
||||
_progressCollector.ProcessQueuedUpdates(); // THREADING ERROR!
|
||||
});
|
||||
```
|
||||
|
||||
### ❌ Mistake 3: Async/await callbacks
|
||||
```csharp
|
||||
await SomeAsyncOperation().ContinueWith(task =>
|
||||
{
|
||||
WriteProgress(...); // THREADING ERROR!
|
||||
});
|
||||
```
|
||||
|
||||
## ✅ CORRECT PATTERNS
|
||||
|
||||
### Pattern 1: Periodic Processing
|
||||
```csharp
|
||||
var tasks = StartBackgroundTasks();
|
||||
while (!Task.WaitAll(tasks, 1000))
|
||||
{
|
||||
_progressCollector.ProcessQueuedUpdates(); // Every 1 second
|
||||
}
|
||||
_progressCollector.ProcessQueuedUpdates(); // Final update
|
||||
```
|
||||
|
||||
### Pattern 2: Completion-based Processing
|
||||
```csharp
|
||||
var tasks = StartBackgroundTasks();
|
||||
Task.WaitAll(tasks);
|
||||
_progressCollector.ProcessQueuedUpdates(); // After completion
|
||||
```
|
||||
|
||||
### Pattern 3: Manual Processing Points
|
||||
```csharp
|
||||
foreach (var item in items)
|
||||
{
|
||||
ProcessItemInBackground(item); // Queues updates
|
||||
_progressCollector.ProcessQueuedUpdates(); // Process after each item
|
||||
}
|
||||
```
|
||||
|
||||
## 🎯 ERROR SYMPTOMS
|
||||
|
||||
If you violate these rules, you'll see errors like:
|
||||
- "The WriteObject and WriteError methods cannot be called from outside the overrides..."
|
||||
- "WriteProgress can only be called from within the same thread"
|
||||
- Cmdlet failures with threading exceptions
|
||||
- Progress bars not updating or appearing
|
||||
|
||||
## 📋 CHECKLIST FOR NEW CMDLETS
|
||||
|
||||
Before implementing background operations:
|
||||
|
||||
- [ ] Are you using ThreadSafeProgressCollector?
|
||||
- [ ] Are all Write* calls from main thread only?
|
||||
- [ ] Are you calling ProcessQueuedUpdates() only from main thread?
|
||||
- [ ] Do you have periodic progress processing for long operations?
|
||||
- [ ] Have you tested with verbose output enabled?
|
||||
- [ ] Have you tested with progress bars enabled?
|
||||
|
||||
## 🔧 DEBUGGING TIPS
|
||||
|
||||
1. **Enable verbose logging** to see threading violations
|
||||
2. **Test with progress bars** - they're most sensitive to threading issues
|
||||
3. **Use Task.WaitAll with timeouts** for periodic processing
|
||||
4. **Never call ProcessQueuedUpdates from callbacks**
|
||||
5. **Queue everything from background threads, process from main thread**
|
||||
|
||||
## 📚 RELATED PATTERNS
|
||||
|
||||
- **File Upload/Download**: Use periodic processing during Task.WaitAll
|
||||
- **Multipart Operations**: Queue from parallel tasks, process periodically
|
||||
- **Long-running Operations**: Process updates every 1-2 seconds
|
||||
- **Collection Processing**: Process after each item or batch
|
||||
|
||||
## 🎯 REMEMBER
|
||||
|
||||
**The golden rule: Background threads QUEUE, main thread PROCESSES.**
|
||||
|
||||
This pattern ensures PowerShell cmdlet compliance while providing real-time progress updates to users.
|
||||
@@ -42,6 +42,7 @@ PSMinIO/
|
||||
│ ├── USAGE.md # Comprehensive usage guide
|
||||
│ ├── RELEASE-NOTES.md # Release notes
|
||||
│ ├── POWERSHELL-GALLERY-RELEASE.md # Gallery release guide
|
||||
│ ├── POWERSHELL-THREADING-RULES.md # Critical PowerShell threading patterns
|
||||
│ └── PROJECT-STRUCTURE.md # This file
|
||||
│
|
||||
├── Artifacts/ # Build artifacts
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
# PSMinIO Basic Operations Example
|
||||
# This script demonstrates fundamental MinIO operations using PSMinIO
|
||||
|
||||
# Import the module
|
||||
Import-Module ..\..\Module\PSMinIO\PSMinIO.psd1
|
||||
|
||||
# Example connection details (replace with your actual values)
|
||||
$endpoint = "https://minio.example.com"
|
||||
$accessKey = "your-access-key"
|
||||
$secretKey = "your-secret-key"
|
||||
|
||||
try {
|
||||
# Connect to MinIO server
|
||||
$connection = Connect-MinIO -Endpoint $endpoint -AccessKey $accessKey -SecretKey $secretKey
|
||||
"Connected to MinIO server: $endpoint"
|
||||
|
||||
# List all buckets
|
||||
"Listing all buckets..."
|
||||
$buckets = Get-MinIOBucket
|
||||
$buckets | Format-Table Name, CreationDate, @{Name="Objects";Expression={"N/A"}}
|
||||
|
||||
# Create a new bucket
|
||||
$bucketName = "example-bucket-$(Get-Date -Format 'yyyyMMdd-HHmmss')"
|
||||
"Creating bucket: $bucketName"
|
||||
New-MinIOBucket -BucketName $bucketName
|
||||
|
||||
# Verify bucket creation
|
||||
if (Test-MinIOBucketExists -BucketName $bucketName) {
|
||||
"✅ Bucket '$bucketName' created successfully"
|
||||
}
|
||||
|
||||
# Create a sample file for upload
|
||||
$sampleFile = "sample-document.txt"
|
||||
"This is a sample document created on $(Get-Date)" | Out-File -FilePath $sampleFile -Encoding UTF8
|
||||
|
||||
# Upload the file
|
||||
"Uploading file: $sampleFile"
|
||||
$uploadResult = New-MinIOObject -BucketName $bucketName -Files $sampleFile
|
||||
$uploadResult | Format-Table Name, @{Name="Duration";Expression={$_.Duration}}, @{Name="Speed";Expression={$_.AverageSpeedFormatted}}
|
||||
|
||||
# List objects in the bucket
|
||||
"Listing objects in bucket '$bucketName':"
|
||||
$objects = Get-MinIOObject -BucketName $bucketName
|
||||
$objects | Format-Table Name, @{Name="Size";Expression={"$($_.Size) bytes"}}, LastModified
|
||||
|
||||
# Download the file
|
||||
$downloadPath = "downloaded-$sampleFile"
|
||||
"Downloading file to: $downloadPath"
|
||||
$downloadResult = Get-MinIOObjectContent -BucketName $bucketName -ObjectName $sampleFile -FilePath $downloadPath
|
||||
$downloadResult | Format-Table @{Name="LocalFile";Expression={$_.FilePath.Name}}, @{Name="Duration";Expression={$_.Duration}}, @{Name="Speed";Expression={$_.AverageSpeedFormatted}}
|
||||
|
||||
# Verify download
|
||||
if (Test-Path $downloadPath) {
|
||||
"✅ File downloaded successfully"
|
||||
"Content: $(Get-Content $downloadPath)"
|
||||
}
|
||||
|
||||
# Clean up
|
||||
"Cleaning up..."
|
||||
Remove-MinIOObject -BucketName $bucketName -ObjectName $sampleFile -Force
|
||||
Remove-MinIOBucket -BucketName $bucketName -Force
|
||||
Remove-Item $sampleFile, $downloadPath -Force -ErrorAction SilentlyContinue
|
||||
|
||||
"✅ Basic operations completed successfully"
|
||||
|
||||
} catch {
|
||||
"❌ Error: $($_.Exception.Message)"
|
||||
} finally {
|
||||
# Clean up any remaining files
|
||||
Remove-Item "sample-document.txt", "downloaded-sample-document.txt" -Force -ErrorAction SilentlyContinue
|
||||
}
|
||||
@@ -0,0 +1,108 @@
|
||||
# PSMinIO Advanced Object Listing Example
|
||||
# Demonstrates the powerful Get-MinIOObject cmdlet with filtering, sorting, and pagination
|
||||
|
||||
# Import the module
|
||||
Import-Module ..\..\Module\PSMinIO\PSMinIO.psd1
|
||||
|
||||
# Connection details (replace with your actual values)
|
||||
$endpoint = "https://minio.example.com"
|
||||
$accessKey = "your-access-key"
|
||||
$secretKey = "your-secret-key"
|
||||
|
||||
try {
|
||||
# Connect to MinIO
|
||||
$connection = Connect-MinIO -Endpoint $endpoint -AccessKey $accessKey -SecretKey $secretKey
|
||||
"Connected to MinIO server for advanced object listing demo"
|
||||
|
||||
# Create a demo bucket
|
||||
$bucketName = "demo-listing-$(Get-Date -Format 'yyyyMMdd-HHmmss')"
|
||||
New-MinIOBucket -BucketName $bucketName
|
||||
"Created demo bucket: $bucketName"
|
||||
|
||||
# Create sample files with different sizes and organize them in directories
|
||||
"Creating sample files..."
|
||||
|
||||
# Documents directory
|
||||
"Small document content" | Out-File -FilePath "small-doc.txt" -Encoding UTF8
|
||||
"Medium document content " * 50 | Out-File -FilePath "medium-doc.txt" -Encoding UTF8
|
||||
"Large document content " * 200 | Out-File -FilePath "large-doc.txt" -Encoding UTF8
|
||||
|
||||
# Images directory (simulated)
|
||||
"Image data " * 100 | Out-File -FilePath "image1.jpg" -Encoding UTF8
|
||||
"Image data " * 300 | Out-File -FilePath "image2.png" -Encoding UTF8
|
||||
|
||||
# Logs directory
|
||||
"Log entry " * 20 | Out-File -FilePath "app.log" -Encoding UTF8
|
||||
"Error log " * 80 | Out-File -FilePath "error.log" -Encoding UTF8
|
||||
|
||||
# Upload files to different directories
|
||||
New-MinIOObject -BucketName $bucketName -Files "small-doc.txt" -BucketDirectory "documents/2025"
|
||||
New-MinIOObject -BucketName $bucketName -Files "medium-doc.txt" -BucketDirectory "documents/2025"
|
||||
New-MinIOObject -BucketName $bucketName -Files "large-doc.txt" -BucketDirectory "documents/archive"
|
||||
New-MinIOObject -BucketName $bucketName -Files "image1.jpg" -BucketDirectory "media/images"
|
||||
New-MinIOObject -BucketName $bucketName -Files "image2.png" -BucketDirectory "media/images"
|
||||
New-MinIOObject -BucketName $bucketName -Files "app.log" -BucketDirectory "logs/application"
|
||||
New-MinIOObject -BucketName $bucketName -Files "error.log" -BucketDirectory "logs/system"
|
||||
|
||||
"Sample files uploaded to various directories"
|
||||
""
|
||||
|
||||
# Demonstrate different listing capabilities
|
||||
"=== 1. List All Objects ==="
|
||||
$allObjects = Get-MinIOObject -BucketName $bucketName
|
||||
$allObjects | Format-Table Name, @{Name="Size";Expression={"$($_.Size) B"}}, LastModified -AutoSize
|
||||
""
|
||||
|
||||
"=== 2. Filter by Prefix ==="
|
||||
$documentsObjects = Get-MinIOObject -BucketName $bucketName -Prefix "documents/"
|
||||
$documentsObjects | Format-Table Name, @{Name="Size";Expression={"$($_.Size) B"}}, LastModified -AutoSize
|
||||
""
|
||||
|
||||
"=== 3. Sort by Size (Descending) ==="
|
||||
$sortedBySize = Get-MinIOObject -BucketName $bucketName -SortBy "Size" -Descending
|
||||
$sortedBySize | Format-Table Name, @{Name="Size";Expression={"$($_.Size) B"}}, LastModified -AutoSize
|
||||
""
|
||||
|
||||
"=== 4. Sort by Name (Ascending) ==="
|
||||
$sortedByName = Get-MinIOObject -BucketName $bucketName -SortBy "Name"
|
||||
$sortedByName | Format-Table Name, @{Name="Size";Expression={"$($_.Size) B"}}, LastModified -AutoSize
|
||||
""
|
||||
|
||||
"=== 5. Limit Results ==="
|
||||
$limitedResults = Get-MinIOObject -BucketName $bucketName -MaxObjects 3 -SortBy "Size" -Descending
|
||||
$limitedResults | Format-Table Name, @{Name="Size";Expression={"$($_.Size) B"}}, LastModified -AutoSize
|
||||
""
|
||||
|
||||
"=== 6. Files Only (Exclude Directories) ==="
|
||||
$filesOnly = Get-MinIOObject -BucketName $bucketName -ObjectsOnly
|
||||
$filesOnly | Format-Table Name, @{Name="Size";Expression={"$($_.Size) B"}}, LastModified -AutoSize
|
||||
""
|
||||
|
||||
"=== 7. Specific Object Lookup ==="
|
||||
$specificObject = Get-MinIOObject -BucketName $bucketName -ObjectName "documents/2025/small-doc.txt"
|
||||
if ($specificObject) {
|
||||
$specificObject | Format-Table Name, @{Name="Size";Expression={"$($_.Size) B"}}, LastModified -AutoSize
|
||||
} else {
|
||||
"Object not found"
|
||||
}
|
||||
""
|
||||
|
||||
"=== 8. Complex Filtering Example ==="
|
||||
"Large files in media directory:"
|
||||
$largeMediaFiles = Get-MinIOObject -BucketName $bucketName -Prefix "media/" -SortBy "Size" -Descending | Where-Object { $_.Size -gt 1000 }
|
||||
$largeMediaFiles | Format-Table Name, @{Name="Size";Expression={"$($_.Size) B"}}, LastModified -AutoSize
|
||||
|
||||
# Clean up
|
||||
"Cleaning up demo files..."
|
||||
$allObjects = Get-MinIOObject -BucketName $bucketName
|
||||
foreach ($obj in $allObjects) {
|
||||
Remove-MinIOObject -BucketName $bucketName -ObjectName $obj.Name -Force
|
||||
}
|
||||
Remove-MinIOBucket -BucketName $bucketName -Force
|
||||
Remove-Item "small-doc.txt", "medium-doc.txt", "large-doc.txt", "image1.jpg", "image2.png", "app.log", "error.log" -Force -ErrorAction SilentlyContinue
|
||||
|
||||
"✅ Advanced object listing demo completed"
|
||||
|
||||
} catch {
|
||||
"❌ Error: $($_.Exception.Message)"
|
||||
}
|
||||
@@ -0,0 +1,135 @@
|
||||
# PSMinIO Directory and Folder Management Example
|
||||
# Demonstrates creating nested directory structures and organizing files
|
||||
|
||||
# Import the module
|
||||
Import-Module ..\..\Module\PSMinIO\PSMinIO.psd1
|
||||
|
||||
# Connection details (replace with your actual values)
|
||||
$endpoint = "https://minio.example.com"
|
||||
$accessKey = "your-access-key"
|
||||
$secretKey = "your-secret-key"
|
||||
|
||||
try {
|
||||
# Connect to MinIO
|
||||
$connection = Connect-MinIO -Endpoint $endpoint -AccessKey $accessKey -SecretKey $secretKey
|
||||
"Connected to MinIO server for directory management demo"
|
||||
|
||||
# Create a demo bucket
|
||||
$bucketName = "directory-demo-$(Get-Date -Format 'yyyyMMdd-HHmmss')"
|
||||
New-MinIOBucket -BucketName $bucketName
|
||||
"Created demo bucket: $bucketName"
|
||||
|
||||
# Create sample files for different scenarios
|
||||
"Creating sample files..."
|
||||
"Project documentation" | Out-File -FilePath "README.md" -Encoding UTF8
|
||||
"Configuration settings" | Out-File -FilePath "config.json" -Encoding UTF8
|
||||
"Application source code" | Out-File -FilePath "app.py" -Encoding UTF8
|
||||
"Test data" | Out-File -FilePath "test-data.csv" -Encoding UTF8
|
||||
"Build output" | Out-File -FilePath "app.exe" -Encoding UTF8
|
||||
|
||||
"=== 1. Creating Explicit Folder Structures ==="
|
||||
# Create explicit folder structures using New-MinIOFolder
|
||||
New-MinIOFolder -BucketName $bucketName -FolderName "projects/web-app/src"
|
||||
New-MinIOFolder -BucketName $bucketName -FolderName "projects/web-app/tests"
|
||||
New-MinIOFolder -BucketName $bucketName -FolderName "projects/web-app/docs"
|
||||
New-MinIOFolder -BucketName $bucketName -FolderName "projects/api/v1"
|
||||
New-MinIOFolder -BucketName $bucketName -FolderName "projects/api/v2"
|
||||
"Created explicit folder structures"
|
||||
""
|
||||
|
||||
"=== 2. Automatic Directory Creation with BucketDirectory ==="
|
||||
# Upload files with automatic directory creation
|
||||
New-MinIOObject -BucketName $bucketName -Files "README.md" -BucketDirectory "projects/web-app/docs"
|
||||
New-MinIOObject -BucketName $bucketName -Files "config.json" -BucketDirectory "projects/web-app/config"
|
||||
New-MinIOObject -BucketName $bucketName -Files "app.py" -BucketDirectory "projects/web-app/src"
|
||||
New-MinIOObject -BucketName $bucketName -Files "test-data.csv" -BucketDirectory "projects/web-app/tests/data"
|
||||
New-MinIOObject -BucketName $bucketName -Files "app.exe" -BucketDirectory "releases/v1.0/binaries"
|
||||
"Uploaded files with automatic directory creation"
|
||||
""
|
||||
|
||||
"=== 3. Multi-Level Directory Structure ==="
|
||||
# Create a complex directory structure
|
||||
$complexStructure = @(
|
||||
"company/departments/engineering/teams/backend",
|
||||
"company/departments/engineering/teams/frontend",
|
||||
"company/departments/engineering/teams/devops",
|
||||
"company/departments/marketing/campaigns/2025",
|
||||
"company/departments/hr/policies/remote-work"
|
||||
)
|
||||
|
||||
foreach ($folder in $complexStructure) {
|
||||
New-MinIOFolder -BucketName $bucketName -FolderName $folder
|
||||
}
|
||||
"Created complex multi-level directory structure"
|
||||
""
|
||||
|
||||
"=== 4. Viewing Directory Structure ==="
|
||||
# List all objects to see the directory structure
|
||||
$allObjects = Get-MinIOObject -BucketName $bucketName
|
||||
"Complete directory and file structure:"
|
||||
$allObjects | Sort-Object Name | Format-Table Name, @{Name="Type";Expression={if($_.IsDirectory){"Directory"}else{"File"}}}, @{Name="Size";Expression={if($_.IsDirectory){"-"}else{"$($_.Size) B"}}} -AutoSize
|
||||
""
|
||||
|
||||
"=== 5. Filtering by Directory ==="
|
||||
# Show objects in specific directories
|
||||
"Files in projects/web-app/:"
|
||||
$webAppFiles = Get-MinIOObject -BucketName $bucketName -Prefix "projects/web-app/"
|
||||
$webAppFiles | Format-Table Name, @{Name="Type";Expression={if($_.IsDirectory){"Directory"}else{"File"}}}, @{Name="Size";Expression={if($_.IsDirectory){"-"}else{"$($_.Size) B"}}} -AutoSize
|
||||
""
|
||||
|
||||
"Engineering team directories:"
|
||||
$engineeringDirs = Get-MinIOObject -BucketName $bucketName -Prefix "company/departments/engineering/teams/"
|
||||
$engineeringDirs | Format-Table Name, @{Name="Type";Expression={if($_.IsDirectory){"Directory"}else{"File"}}} -AutoSize
|
||||
""
|
||||
|
||||
"=== 6. Directory-Only Listing ==="
|
||||
# Show only directories (folders)
|
||||
$directoriesOnly = Get-MinIOObject -BucketName $bucketName | Where-Object { $_.IsDirectory }
|
||||
"Directories only:"
|
||||
$directoriesOnly | Sort-Object Name | Format-Table Name -AutoSize
|
||||
""
|
||||
|
||||
"=== 7. Files-Only Listing ==="
|
||||
# Show only files (excluding directories)
|
||||
$filesOnly = Get-MinIOObject -BucketName $bucketName -ObjectsOnly
|
||||
"Files only:"
|
||||
$filesOnly | Format-Table Name, @{Name="Size";Expression={"$($_.Size) B"}}, LastModified -AutoSize
|
||||
""
|
||||
|
||||
"=== 8. Organizing Files by Date ==="
|
||||
# Create date-based directory structure
|
||||
$currentDate = Get-Date
|
||||
$yearMonth = $currentDate.ToString("yyyy/MM")
|
||||
$dailyFolder = $currentDate.ToString("yyyy/MM/dd")
|
||||
|
||||
# Upload files to date-based directories
|
||||
"Sample log entry" | Out-File -FilePath "daily.log" -Encoding UTF8
|
||||
"Backup data" | Out-File -FilePath "backup.zip" -Encoding UTF8
|
||||
|
||||
New-MinIOObject -BucketName $bucketName -Files "daily.log" -BucketDirectory "logs/$dailyFolder"
|
||||
New-MinIOObject -BucketName $bucketName -Files "backup.zip" -BucketDirectory "backups/$yearMonth"
|
||||
|
||||
"Created date-based directory structure and uploaded files"
|
||||
""
|
||||
|
||||
# Show the final structure
|
||||
"=== Final Directory Structure ==="
|
||||
$finalStructure = Get-MinIOObject -BucketName $bucketName | Sort-Object Name
|
||||
$finalStructure | Format-Table Name, @{Name="Type";Expression={if($_.IsDirectory){"Directory"}else{"File"}}}, @{Name="Size";Expression={if($_.IsDirectory){"-"}else{"$($_.Size) B"}}} -AutoSize
|
||||
|
||||
# Clean up
|
||||
"Cleaning up demo files..."
|
||||
$allObjects = Get-MinIOObject -BucketName $bucketName
|
||||
foreach ($obj in $allObjects) {
|
||||
if (-not $obj.IsDirectory) {
|
||||
Remove-MinIOObject -BucketName $bucketName -ObjectName $obj.Name -Force
|
||||
}
|
||||
}
|
||||
Remove-MinIOBucket -BucketName $bucketName -Force
|
||||
Remove-Item "README.md", "config.json", "app.py", "test-data.csv", "app.exe", "daily.log", "backup.zip" -Force -ErrorAction SilentlyContinue
|
||||
|
||||
"✅ Directory management demo completed"
|
||||
|
||||
} catch {
|
||||
"❌ Error: $($_.Exception.Message)"
|
||||
}
|
||||
@@ -0,0 +1,153 @@
|
||||
# PSMinIO Chunked Operations Example
|
||||
# Demonstrates chunked uploads and downloads for large files with progress tracking
|
||||
|
||||
# Import the module
|
||||
Import-Module ..\..\Module\PSMinIO\PSMinIO.psd1
|
||||
|
||||
# Connection details (replace with your actual values)
|
||||
$endpoint = "https://minio.example.com"
|
||||
$accessKey = "your-access-key"
|
||||
$secretKey = "your-secret-key"
|
||||
|
||||
try {
|
||||
# Connect to MinIO
|
||||
$connection = Connect-MinIO -Endpoint $endpoint -AccessKey $accessKey -SecretKey $secretKey
|
||||
"Connected to MinIO server for chunked operations demo"
|
||||
|
||||
# Create a demo bucket
|
||||
$bucketName = "chunked-demo-$(Get-Date -Format 'yyyyMMdd-HHmmss')"
|
||||
New-MinIOBucket -BucketName $bucketName
|
||||
"Created demo bucket: $bucketName"
|
||||
|
||||
"=== 1. Creating Large Test Files ==="
|
||||
# Create files of different sizes for testing
|
||||
|
||||
# Small file (under chunk size)
|
||||
$smallContent = "Small file content " * 100 # ~2KB
|
||||
$smallContent | Out-File -FilePath "small-file.txt" -Encoding UTF8 -NoNewline
|
||||
|
||||
# Medium file (1-2 chunks)
|
||||
$mediumContent = "Medium file content " * 30000 # ~600KB
|
||||
$mediumContent | Out-File -FilePath "medium-file.txt" -Encoding UTF8 -NoNewline
|
||||
|
||||
# Large file (multiple chunks)
|
||||
$largeContent = "Large file content " * 100000 # ~2MB
|
||||
$largeContent | Out-File -FilePath "large-file.txt" -Encoding UTF8 -NoNewline
|
||||
|
||||
# Very large file (many chunks)
|
||||
$veryLargeContent = "Very large file content " * 250000 # ~6MB
|
||||
$veryLargeContent | Out-File -FilePath "very-large-file.txt" -Encoding UTF8 -NoNewline
|
||||
|
||||
"Created test files of various sizes"
|
||||
Get-ChildItem "*.txt" | Format-Table Name, @{Name="Size";Expression={"$([math]::Round($_.Length/1KB,2)) KB"}}
|
||||
""
|
||||
|
||||
"=== 2. Chunked Upload with Different Chunk Sizes ==="
|
||||
|
||||
# Upload with 1MB chunks (default)
|
||||
"Uploading medium file with 1MB chunks..."
|
||||
$result1 = New-MinIOObjectChunked -BucketName $bucketName -Files "medium-file.txt" -ChunkSize 1MB -BucketDirectory "uploads/1mb-chunks"
|
||||
$result1 | Format-Table Name, @{Name="SizeMB";Expression={[math]::Round($_.Size/1MB,2)}}, @{Name="Duration";Expression={$_.Duration}}, @{Name="Speed";Expression={$_.AverageSpeedFormatted}}
|
||||
""
|
||||
|
||||
# Upload with 512KB chunks
|
||||
"Uploading large file with 512KB chunks..."
|
||||
$result2 = New-MinIOObjectChunked -BucketName $bucketName -Files "large-file.txt" -ChunkSize 512KB -BucketDirectory "uploads/512kb-chunks"
|
||||
$result2 | Format-Table Name, @{Name="SizeMB";Expression={[math]::Round($_.Size/1MB,2)}}, @{Name="Duration";Expression={$_.Duration}}, @{Name="Speed";Expression={$_.AverageSpeedFormatted}}
|
||||
""
|
||||
|
||||
# Upload with 2MB chunks
|
||||
"Uploading very large file with 2MB chunks..."
|
||||
$result3 = New-MinIOObjectChunked -BucketName $bucketName -Files "very-large-file.txt" -ChunkSize 2MB -BucketDirectory "uploads/2mb-chunks"
|
||||
$result3 | Format-Table Name, @{Name="SizeMB";Expression={[math]::Round($_.Size/1MB,2)}}, @{Name="Duration";Expression={$_.Duration}}, @{Name="Speed";Expression={$_.AverageSpeedFormatted}}
|
||||
""
|
||||
|
||||
"=== 3. Multiple File Chunked Upload ==="
|
||||
# Upload multiple files in one operation
|
||||
"Uploading multiple files with chunked transfer..."
|
||||
$multiResult = New-MinIOObjectChunked -BucketName $bucketName -Files @("small-file.txt", "medium-file.txt") -ChunkSize 1MB -BucketDirectory "uploads/multi-file"
|
||||
$multiResult | Format-Table Name, @{Name="SizeMB";Expression={[math]::Round($_.Size/1MB,2)}}, @{Name="Duration";Expression={$_.Duration}}, @{Name="Speed";Expression={$_.AverageSpeedFormatted}}
|
||||
""
|
||||
|
||||
"=== 4. Chunked Download Operations ==="
|
||||
|
||||
# Download with chunked transfer
|
||||
"Downloading large file with chunked transfer..."
|
||||
$downloadResult = Get-MinIOObjectContentChunked -BucketName $bucketName -ObjectName "uploads/2mb-chunks/very-large-file.txt" -FilePath "downloaded-very-large.txt" -ChunkSize 1MB
|
||||
$downloadResult | Format-Table @{Name="LocalFile";Expression={$_.FilePath.Name}}, @{Name="SizeMB";Expression={[math]::Round($_.FilePath.Length/1MB,2)}}, @{Name="Duration";Expression={$_.Duration}}, @{Name="Speed";Expression={$_.AverageSpeedFormatted}}
|
||||
""
|
||||
|
||||
# Verify download integrity
|
||||
if (Test-Path "downloaded-very-large.txt") {
|
||||
$originalSize = (Get-Item "very-large-file.txt").Length
|
||||
$downloadedSize = (Get-Item "downloaded-very-large.txt").Length
|
||||
if ($originalSize -eq $downloadedSize) {
|
||||
"✅ Download integrity verified - sizes match ($originalSize bytes)"
|
||||
} else {
|
||||
"❌ Download integrity check failed - size mismatch"
|
||||
}
|
||||
}
|
||||
""
|
||||
|
||||
"=== 5. Performance Comparison ==="
|
||||
# Compare regular vs chunked upload for the same file
|
||||
"Comparing regular vs chunked upload performance..."
|
||||
|
||||
# Regular upload
|
||||
$regularStart = Get-Date
|
||||
$regularResult = New-MinIOObject -BucketName $bucketName -Files "large-file.txt" -BucketDirectory "comparison/regular"
|
||||
$regularEnd = Get-Date
|
||||
$regularDuration = $regularEnd - $regularStart
|
||||
|
||||
# Chunked upload
|
||||
$chunkedStart = Get-Date
|
||||
$chunkedResult = New-MinIOObjectChunked -BucketName $bucketName -Files "large-file.txt" -ChunkSize 1MB -BucketDirectory "comparison/chunked"
|
||||
$chunkedEnd = Get-Date
|
||||
$chunkedDuration = $chunkedEnd - $chunkedStart
|
||||
|
||||
"Performance Comparison for large-file.txt:"
|
||||
[PSCustomObject]@{
|
||||
Method = "Regular"
|
||||
Duration = $regularDuration.ToString("mm\:ss\.fff")
|
||||
Speed = $regularResult.AverageSpeedFormatted
|
||||
} | Format-Table
|
||||
|
||||
[PSCustomObject]@{
|
||||
Method = "Chunked"
|
||||
Duration = $chunkedDuration.ToString("mm\:ss\.fff")
|
||||
Speed = $chunkedResult.AverageSpeedFormatted
|
||||
} | Format-Table
|
||||
""
|
||||
|
||||
"=== 6. Listing Uploaded Files ==="
|
||||
# Show all uploaded files with their details
|
||||
$allUploads = Get-MinIOObject -BucketName $bucketName -Prefix "uploads/" -ObjectsOnly -SortBy "Size" -Descending
|
||||
"All uploaded files (sorted by size):"
|
||||
$allUploads | Format-Table Name, @{Name="SizeMB";Expression={[math]::Round($_.Size/1MB,2)}}, LastModified -AutoSize
|
||||
""
|
||||
|
||||
"=== 7. Chunk Size Recommendations ==="
|
||||
"Chunk Size Recommendations:"
|
||||
"• Files < 10MB: Use regular upload (New-MinIOObject)"
|
||||
"• Files 10MB - 100MB: Use 1-5MB chunks"
|
||||
"• Files 100MB - 1GB: Use 5-10MB chunks"
|
||||
"• Files > 1GB: Use 10-50MB chunks"
|
||||
"• Network considerations: Smaller chunks for unstable connections"
|
||||
""
|
||||
|
||||
# Clean up
|
||||
"Cleaning up demo files..."
|
||||
$allObjects = Get-MinIOObject -BucketName $bucketName
|
||||
foreach ($obj in $allObjects) {
|
||||
if (-not $obj.IsDirectory) {
|
||||
Remove-MinIOObject -BucketName $bucketName -ObjectName $obj.Name -Force
|
||||
}
|
||||
}
|
||||
Remove-MinIOBucket -BucketName $bucketName -Force
|
||||
Remove-Item "small-file.txt", "medium-file.txt", "large-file.txt", "very-large-file.txt", "downloaded-very-large.txt" -Force -ErrorAction SilentlyContinue
|
||||
|
||||
"✅ Chunked operations demo completed"
|
||||
|
||||
} catch {
|
||||
"❌ Error: $($_.Exception.Message)"
|
||||
}
|
||||
@@ -0,0 +1,205 @@
|
||||
# PSMinIO Bulk Operations Example
|
||||
# Demonstrates bulk file operations, batch processing, and automation scenarios
|
||||
|
||||
# Import the module
|
||||
Import-Module ..\..\Module\PSMinIO\PSMinIO.psd1
|
||||
|
||||
# Connection details (replace with your actual values)
|
||||
$endpoint = "https://minio.example.com"
|
||||
$accessKey = "your-access-key"
|
||||
$secretKey = "your-secret-key"
|
||||
|
||||
try {
|
||||
# Connect to MinIO
|
||||
$connection = Connect-MinIO -Endpoint $endpoint -AccessKey $accessKey -SecretKey $secretKey
|
||||
"Connected to MinIO server for bulk operations demo"
|
||||
|
||||
# Create a demo bucket
|
||||
$bucketName = "bulk-demo-$(Get-Date -Format 'yyyyMMdd-HHmmss')"
|
||||
New-MinIOBucket -BucketName $bucketName
|
||||
"Created demo bucket: $bucketName"
|
||||
|
||||
"=== 1. Creating Multiple Test Files ==="
|
||||
# Create a variety of files for bulk operations
|
||||
$testFiles = @()
|
||||
|
||||
# Create document files
|
||||
for ($i = 1; $i -le 5; $i++) {
|
||||
$fileName = "document-$i.txt"
|
||||
"Document $i content - $(Get-Date)" | Out-File -FilePath $fileName -Encoding UTF8
|
||||
$testFiles += $fileName
|
||||
}
|
||||
|
||||
# Create data files
|
||||
for ($i = 1; $i -le 3; $i++) {
|
||||
$fileName = "data-$i.csv"
|
||||
"Name,Value,Date`nItem$i,$(Get-Random -Minimum 100 -Maximum 1000),$(Get-Date)" | Out-File -FilePath $fileName -Encoding UTF8
|
||||
$testFiles += $fileName
|
||||
}
|
||||
|
||||
# Create log files
|
||||
for ($i = 1; $i -le 4; $i++) {
|
||||
$fileName = "log-$i.log"
|
||||
"$(Get-Date) - Log entry $i`n$(Get-Date) - Another log entry" | Out-File -FilePath $fileName -Encoding UTF8
|
||||
$testFiles += $fileName
|
||||
}
|
||||
|
||||
"Created $($testFiles.Count) test files"
|
||||
Get-ChildItem "*.txt", "*.csv", "*.log" | Format-Table Name, @{Name="Size";Expression={"$($_.Length) bytes"}} -AutoSize
|
||||
""
|
||||
|
||||
"=== 2. Bulk Upload to Different Directories ==="
|
||||
# Upload different file types to organized directories
|
||||
|
||||
# Upload documents
|
||||
$docFiles = Get-ChildItem "document-*.txt"
|
||||
if ($docFiles) {
|
||||
"Uploading $($docFiles.Count) document files..."
|
||||
$docResults = New-MinIOObject -BucketName $bucketName -Files $docFiles.Name -BucketDirectory "documents/$(Get-Date -Format 'yyyy/MM')"
|
||||
$docResults | Format-Table Name, @{Name="Duration";Expression={$_.Duration}}, @{Name="Speed";Expression={$_.AverageSpeedFormatted}}
|
||||
}
|
||||
""
|
||||
|
||||
# Upload data files
|
||||
$dataFiles = Get-ChildItem "data-*.csv"
|
||||
if ($dataFiles) {
|
||||
"Uploading $($dataFiles.Count) data files..."
|
||||
$dataResults = New-MinIOObject -BucketName $bucketName -Files $dataFiles.Name -BucketDirectory "data/exports"
|
||||
$dataResults | Format-Table Name, @{Name="Duration";Expression={$_.Duration}}, @{Name="Speed";Expression={$_.AverageSpeedFormatted}}
|
||||
}
|
||||
""
|
||||
|
||||
# Upload log files
|
||||
$logFiles = Get-ChildItem "log-*.log"
|
||||
if ($logFiles) {
|
||||
"Uploading $($logFiles.Count) log files..."
|
||||
$logResults = New-MinIOObject -BucketName $bucketName -Files $logFiles.Name -BucketDirectory "logs/application/$(Get-Date -Format 'yyyy-MM-dd')"
|
||||
$logResults | Format-Table Name, @{Name="Duration";Expression={$_.Duration}}, @{Name="Speed";Expression={$_.AverageSpeedFormatted}}
|
||||
}
|
||||
""
|
||||
|
||||
"=== 3. Bulk File Analysis ==="
|
||||
# Analyze uploaded files by type and location
|
||||
$allObjects = Get-MinIOObject -BucketName $bucketName -ObjectsOnly
|
||||
|
||||
"File distribution by directory:"
|
||||
$allObjects | Group-Object { ($_.Name -split '/')[0] } | Format-Table Name, Count -AutoSize
|
||||
""
|
||||
|
||||
"File distribution by extension:"
|
||||
$allObjects | Group-Object { [System.IO.Path]::GetExtension($_.Name) } | Format-Table Name, Count -AutoSize
|
||||
""
|
||||
|
||||
"Largest files:"
|
||||
$allObjects | Sort-Object Size -Descending | Select-Object -First 5 | Format-Table Name, @{Name="Size";Expression={"$($_.Size) bytes"}}, LastModified -AutoSize
|
||||
""
|
||||
|
||||
"=== 4. Batch Download Operations ==="
|
||||
# Download files by pattern/criteria
|
||||
|
||||
# Create download directory
|
||||
$downloadDir = "downloads"
|
||||
if (-not (Test-Path $downloadDir)) {
|
||||
New-Item -ItemType Directory -Path $downloadDir | Out-Null
|
||||
}
|
||||
|
||||
# Download all CSV files
|
||||
$csvObjects = Get-MinIOObject -BucketName $bucketName | Where-Object { $_.Name -like "*.csv" }
|
||||
"Downloading $($csvObjects.Count) CSV files..."
|
||||
foreach ($csvObj in $csvObjects) {
|
||||
$localFileName = [System.IO.Path]::GetFileName($csvObj.Name)
|
||||
$localPath = Join-Path $downloadDir $localFileName
|
||||
$downloadResult = Get-MinIOObjectContent -BucketName $bucketName -ObjectName $csvObj.Name -FilePath $localPath
|
||||
"Downloaded: $($csvObj.Name) -> $localPath"
|
||||
}
|
||||
""
|
||||
|
||||
# Download files from specific directory
|
||||
$docObjects = Get-MinIOObject -BucketName $bucketName -Prefix "documents/"
|
||||
"Downloading $($docObjects.Count) files from documents directory..."
|
||||
foreach ($docObj in $docObjects) {
|
||||
if (-not $docObj.IsDirectory) {
|
||||
$localFileName = [System.IO.Path]::GetFileName($docObj.Name)
|
||||
$localPath = Join-Path $downloadDir "doc-$localFileName"
|
||||
$downloadResult = Get-MinIOObjectContent -BucketName $bucketName -ObjectName $docObj.Name -FilePath $localPath
|
||||
"Downloaded: $($docObj.Name) -> $localPath"
|
||||
}
|
||||
}
|
||||
""
|
||||
|
||||
"=== 5. Bulk Operations with Filtering ==="
|
||||
# Demonstrate advanced bulk operations
|
||||
|
||||
# Find and process files by size
|
||||
$largeFiles = Get-MinIOObject -BucketName $bucketName -ObjectsOnly | Where-Object { $_.Size -gt 50 }
|
||||
"Processing $($largeFiles.Count) files larger than 50 bytes:"
|
||||
$largeFiles | Format-Table Name, @{Name="Size";Expression={"$($_.Size) bytes"}}, LastModified -AutoSize
|
||||
""
|
||||
|
||||
# Find files by date (last hour)
|
||||
$recentFiles = Get-MinIOObject -BucketName $bucketName -ObjectsOnly | Where-Object { $_.LastModified -gt (Get-Date).AddHours(-1) }
|
||||
"Files uploaded in the last hour: $($recentFiles.Count)"
|
||||
$recentFiles | Format-Table Name, LastModified -AutoSize
|
||||
""
|
||||
|
||||
"=== 6. Bulk Cleanup Operations ==="
|
||||
# Demonstrate selective cleanup
|
||||
|
||||
# Remove files by pattern
|
||||
$logObjects = Get-MinIOObject -BucketName $bucketName | Where-Object { $_.Name -like "*log*" -and -not $_.IsDirectory }
|
||||
"Removing $($logObjects.Count) log files..."
|
||||
foreach ($logObj in $logObjects) {
|
||||
Remove-MinIOObject -BucketName $bucketName -ObjectName $logObj.Name -Force
|
||||
"Removed: $($logObj.Name)"
|
||||
}
|
||||
""
|
||||
|
||||
# Remove files older than a certain date (simulated)
|
||||
$cutoffDate = (Get-Date).AddMinutes(-5) # 5 minutes ago for demo
|
||||
$oldFiles = Get-MinIOObject -BucketName $bucketName -ObjectsOnly | Where-Object { $_.LastModified -lt $cutoffDate }
|
||||
if ($oldFiles.Count -gt 0) {
|
||||
"Removing $($oldFiles.Count) files older than $cutoffDate..."
|
||||
foreach ($oldFile in $oldFiles) {
|
||||
Remove-MinIOObject -BucketName $bucketName -ObjectName $oldFile.Name -Force
|
||||
"Removed: $($oldFile.Name)"
|
||||
}
|
||||
} else {
|
||||
"No files older than $cutoffDate found"
|
||||
}
|
||||
""
|
||||
|
||||
"=== 7. Final Statistics ==="
|
||||
# Show final state
|
||||
$remainingObjects = Get-MinIOObject -BucketName $bucketName -ObjectsOnly
|
||||
"Remaining files after cleanup: $($remainingObjects.Count)"
|
||||
if ($remainingObjects.Count -gt 0) {
|
||||
$remainingObjects | Format-Table Name, @{Name="Size";Expression={"$($_.Size) bytes"}}, LastModified -AutoSize
|
||||
}
|
||||
""
|
||||
|
||||
"Downloaded files:"
|
||||
if (Test-Path $downloadDir) {
|
||||
Get-ChildItem $downloadDir | Format-Table Name, @{Name="Size";Expression={"$($_.Length) bytes"}}, LastWriteTime -AutoSize
|
||||
}
|
||||
|
||||
# Clean up
|
||||
"Cleaning up demo files..."
|
||||
$allObjects = Get-MinIOObject -BucketName $bucketName
|
||||
foreach ($obj in $allObjects) {
|
||||
if (-not $obj.IsDirectory) {
|
||||
Remove-MinIOObject -BucketName $bucketName -ObjectName $obj.Name -Force
|
||||
}
|
||||
}
|
||||
Remove-MinIOBucket -BucketName $bucketName -Force
|
||||
|
||||
# Clean up local files
|
||||
Remove-Item $testFiles -Force -ErrorAction SilentlyContinue
|
||||
if (Test-Path $downloadDir) {
|
||||
Remove-Item $downloadDir -Recurse -Force -ErrorAction SilentlyContinue
|
||||
}
|
||||
|
||||
"✅ Bulk operations demo completed"
|
||||
|
||||
} catch {
|
||||
"❌ Error: $($_.Exception.Message)"
|
||||
}
|
||||
@@ -0,0 +1,291 @@
|
||||
# PSMinIO Enterprise Automation Example
|
||||
# Demonstrates enterprise-grade automation scenarios including monitoring, policies, and scheduled operations
|
||||
|
||||
# Import the module
|
||||
Import-Module ..\..\Module\PSMinIO\PSMinIO.psd1
|
||||
|
||||
# Connection details (replace with your actual values)
|
||||
$endpoint = "https://minio.example.com"
|
||||
$accessKey = "your-access-key"
|
||||
$secretKey = "your-secret-key"
|
||||
|
||||
# Configuration
|
||||
$logFile = "minio-automation-$(Get-Date -Format 'yyyyMMdd').log"
|
||||
$reportFile = "minio-report-$(Get-Date -Format 'yyyyMMdd-HHmmss').html"
|
||||
|
||||
function Write-Log {
|
||||
param([string]$Message, [string]$Level = "INFO")
|
||||
$timestamp = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
|
||||
$logEntry = "[$timestamp] [$Level] $Message"
|
||||
$logEntry | Out-File -FilePath $logFile -Append -Encoding UTF8
|
||||
$logEntry
|
||||
}
|
||||
|
||||
try {
|
||||
Write-Log "Starting MinIO enterprise automation demo"
|
||||
|
||||
# Connect to MinIO
|
||||
$connection = Connect-MinIO -Endpoint $endpoint -AccessKey $accessKey -SecretKey $secretKey
|
||||
Write-Log "Connected to MinIO server: $endpoint"
|
||||
|
||||
"=== 1. Infrastructure Health Check ==="
|
||||
# Perform comprehensive health checks
|
||||
Write-Log "Performing infrastructure health check"
|
||||
|
||||
# Check server connectivity and basic operations
|
||||
try {
|
||||
$buckets = Get-MinIOBucket
|
||||
Write-Log "✅ Server connectivity: OK ($($buckets.Count) buckets found)"
|
||||
"✅ Server connectivity: OK ($($buckets.Count) buckets found)"
|
||||
} catch {
|
||||
Write-Log "❌ Server connectivity: FAILED - $($_.Exception.Message)" "ERROR"
|
||||
"❌ Server connectivity: FAILED"
|
||||
throw
|
||||
}
|
||||
|
||||
# Test bucket operations
|
||||
$testBucketName = "health-check-$(Get-Date -Format 'yyyyMMdd-HHmmss')"
|
||||
try {
|
||||
New-MinIOBucket -BucketName $testBucketName
|
||||
$testExists = Test-MinIOBucketExists -BucketName $testBucketName
|
||||
Remove-MinIOBucket -BucketName $testBucketName -Force
|
||||
Write-Log "✅ Bucket operations: OK"
|
||||
"✅ Bucket operations: OK"
|
||||
} catch {
|
||||
Write-Log "❌ Bucket operations: FAILED - $($_.Exception.Message)" "ERROR"
|
||||
"❌ Bucket operations: FAILED"
|
||||
}
|
||||
""
|
||||
|
||||
"=== 2. Storage Statistics and Monitoring ==="
|
||||
# Collect comprehensive storage statistics
|
||||
Write-Log "Collecting storage statistics"
|
||||
|
||||
$stats = Get-MinIOStats -MaxObjectsToCount 1000
|
||||
Write-Log "Storage stats collected: $($stats.TotalBuckets) buckets, $($stats.TotalObjects) objects, $($stats.TotalSizeFormatted)"
|
||||
|
||||
"Storage Overview:"
|
||||
$stats | Format-List TotalBuckets, TotalObjects, TotalSizeFormatted, AverageObjectSizeFormatted
|
||||
""
|
||||
|
||||
# Detailed bucket analysis
|
||||
"Bucket Analysis:"
|
||||
$bucketStats = Get-MinIOBucket -IncludeStatistics
|
||||
$bucketStats | Format-Table Name, @{Name="Objects";Expression={$_.ObjectCount}}, @{Name="Size";Expression={$_.SizeFormatted}}, CreationDate -AutoSize
|
||||
""
|
||||
|
||||
"=== 3. Automated Backup Operations ==="
|
||||
# Simulate automated backup scenario
|
||||
Write-Log "Starting automated backup operations"
|
||||
|
||||
$backupBucket = "automated-backups-$(Get-Date -Format 'yyyyMMdd')"
|
||||
New-MinIOBucket -BucketName $backupBucket
|
||||
Write-Log "Created backup bucket: $backupBucket"
|
||||
|
||||
# Create sample data to backup
|
||||
$backupData = @{
|
||||
"config-backup.json" = @{
|
||||
timestamp = Get-Date
|
||||
server = $env:COMPUTERNAME
|
||||
version = "1.0"
|
||||
} | ConvertTo-Json
|
||||
|
||||
"database-backup.sql" = "-- Database backup generated on $(Get-Date)`nSELECT * FROM users;"
|
||||
|
||||
"logs-backup.txt" = "Application logs backup`n$(Get-Date) - System started`n$(Get-Date) - Backup initiated"
|
||||
}
|
||||
|
||||
foreach ($file in $backupData.Keys) {
|
||||
$backupData[$file] | Out-File -FilePath $file -Encoding UTF8
|
||||
}
|
||||
|
||||
# Perform backup with organized directory structure
|
||||
$backupDate = Get-Date -Format "yyyy/MM/dd"
|
||||
$backupTime = Get-Date -Format "HH-mm"
|
||||
|
||||
$backupResults = New-MinIOObject -BucketName $backupBucket -Files $backupData.Keys -BucketDirectory "daily-backups/$backupDate/$backupTime"
|
||||
Write-Log "Backup completed: $($backupResults.Count) files uploaded"
|
||||
|
||||
"Backup Results:"
|
||||
$backupResults | Format-Table Name, @{Name="Duration";Expression={$_.Duration}}, @{Name="Speed";Expression={$_.AverageSpeedFormatted}}
|
||||
""
|
||||
|
||||
"=== 4. Data Lifecycle Management ==="
|
||||
# Simulate data lifecycle management
|
||||
Write-Log "Performing data lifecycle management"
|
||||
|
||||
# Create lifecycle demo bucket
|
||||
$lifecycleBucket = "lifecycle-demo-$(Get-Date -Format 'yyyyMMdd-HHmmss')"
|
||||
New-MinIOBucket -BucketName $lifecycleBucket
|
||||
|
||||
# Upload files with different "ages" (simulated by different directories)
|
||||
$lifecycleFiles = @(
|
||||
@{ Name = "current-data.txt"; Content = "Current data"; Directory = "current" }
|
||||
@{ Name = "week-old-data.txt"; Content = "Week old data"; Directory = "archive/weekly" }
|
||||
@{ Name = "month-old-data.txt"; Content = "Month old data"; Directory = "archive/monthly" }
|
||||
@{ Name = "year-old-data.txt"; Content = "Year old data"; Directory = "archive/yearly" }
|
||||
)
|
||||
|
||||
foreach ($file in $lifecycleFiles) {
|
||||
$file.Content | Out-File -FilePath $file.Name -Encoding UTF8
|
||||
New-MinIOObject -BucketName $lifecycleBucket -Files $file.Name -BucketDirectory $file.Directory
|
||||
Remove-Item $file.Name -Force
|
||||
}
|
||||
|
||||
# Analyze data by lifecycle stage
|
||||
$allLifecycleObjects = Get-MinIOObject -BucketName $lifecycleBucket -ObjectsOnly
|
||||
"Data Lifecycle Analysis:"
|
||||
$allLifecycleObjects | Group-Object { ($_.Name -split '/')[0] } | Format-Table Name, Count -AutoSize
|
||||
""
|
||||
|
||||
"=== 5. Security and Compliance Audit ==="
|
||||
# Perform security audit
|
||||
Write-Log "Performing security and compliance audit"
|
||||
|
||||
$auditResults = @()
|
||||
|
||||
# Check bucket policies
|
||||
foreach ($bucket in $buckets) {
|
||||
try {
|
||||
$policy = Get-MinIOBucketPolicy -BucketName $bucket.Name
|
||||
$auditResults += [PSCustomObject]@{
|
||||
Bucket = $bucket.Name
|
||||
HasPolicy = $policy -ne $null
|
||||
PolicySize = if ($policy) { $policy.Length } else { 0 }
|
||||
Status = if ($policy) { "Policy Configured" } else { "No Policy" }
|
||||
}
|
||||
} catch {
|
||||
$auditResults += [PSCustomObject]@{
|
||||
Bucket = $bucket.Name
|
||||
HasPolicy = $false
|
||||
PolicySize = 0
|
||||
Status = "Policy Check Failed"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
"Security Audit Results:"
|
||||
$auditResults | Format-Table Bucket, HasPolicy, Status -AutoSize
|
||||
Write-Log "Security audit completed: $($auditResults.Count) buckets audited"
|
||||
""
|
||||
|
||||
"=== 6. Performance Monitoring ==="
|
||||
# Monitor performance metrics
|
||||
Write-Log "Collecting performance metrics"
|
||||
|
||||
# Test upload/download performance
|
||||
$perfTestFile = "performance-test.txt"
|
||||
$perfTestContent = "Performance test data " * 1000 # ~20KB
|
||||
$perfTestContent | Out-File -FilePath $perfTestFile -Encoding UTF8 -NoNewline
|
||||
|
||||
$perfBucket = "performance-test-$(Get-Date -Format 'yyyyMMdd-HHmmss')"
|
||||
New-MinIOBucket -BucketName $perfBucket
|
||||
|
||||
# Measure upload performance
|
||||
$uploadStart = Get-Date
|
||||
$uploadResult = New-MinIOObject -BucketName $perfBucket -Files $perfTestFile
|
||||
$uploadEnd = Get-Date
|
||||
$uploadDuration = $uploadEnd - $uploadStart
|
||||
|
||||
# Measure download performance
|
||||
$downloadStart = Get-Date
|
||||
$downloadResult = Get-MinIOObjectContent -BucketName $perfBucket -ObjectName $perfTestFile -FilePath "downloaded-$perfTestFile"
|
||||
$downloadEnd = Get-Date
|
||||
$downloadDuration = $downloadEnd - $downloadStart
|
||||
|
||||
"Performance Metrics:"
|
||||
[PSCustomObject]@{
|
||||
Operation = "Upload"
|
||||
Duration = $uploadDuration.TotalMilliseconds
|
||||
Speed = $uploadResult.AverageSpeedFormatted
|
||||
FileSize = (Get-Item $perfTestFile).Length
|
||||
} | Format-Table
|
||||
|
||||
[PSCustomObject]@{
|
||||
Operation = "Download"
|
||||
Duration = $downloadDuration.TotalMilliseconds
|
||||
Speed = $downloadResult.AverageSpeedFormatted
|
||||
FileSize = (Get-Item "downloaded-$perfTestFile").Length
|
||||
} | Format-Table
|
||||
|
||||
Write-Log "Performance test completed - Upload: $($uploadDuration.TotalMilliseconds)ms, Download: $($downloadDuration.TotalMilliseconds)ms"
|
||||
""
|
||||
|
||||
"=== 7. Generating HTML Report ==="
|
||||
# Generate comprehensive HTML report
|
||||
Write-Log "Generating HTML report"
|
||||
|
||||
$htmlReport = @"
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<title>MinIO Enterprise Report - $(Get-Date -Format 'yyyy-MM-dd HH:mm:ss')</title>
|
||||
<style>
|
||||
body { font-family: Arial, sans-serif; margin: 20px; }
|
||||
.header { background-color: #f0f0f0; padding: 10px; border-radius: 5px; }
|
||||
.section { margin: 20px 0; }
|
||||
.metric { background-color: #e8f4f8; padding: 10px; margin: 5px 0; border-radius: 3px; }
|
||||
table { border-collapse: collapse; width: 100%; }
|
||||
th, td { border: 1px solid #ddd; padding: 8px; text-align: left; }
|
||||
th { background-color: #f2f2f2; }
|
||||
.success { color: green; }
|
||||
.warning { color: orange; }
|
||||
.error { color: red; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="header">
|
||||
<h1>MinIO Enterprise Automation Report</h1>
|
||||
<p>Generated: $(Get-Date -Format 'yyyy-MM-dd HH:mm:ss')</p>
|
||||
<p>Server: $endpoint</p>
|
||||
</div>
|
||||
|
||||
<div class="section">
|
||||
<h2>Storage Overview</h2>
|
||||
<div class="metric">Total Buckets: $($stats.TotalBuckets)</div>
|
||||
<div class="metric">Total Objects: $($stats.TotalObjects)</div>
|
||||
<div class="metric">Total Size: $($stats.TotalSizeFormatted)</div>
|
||||
</div>
|
||||
|
||||
<div class="section">
|
||||
<h2>Health Check Results</h2>
|
||||
<div class="metric success">✅ Server Connectivity: OK</div>
|
||||
<div class="metric success">✅ Bucket Operations: OK</div>
|
||||
</div>
|
||||
|
||||
<div class="section">
|
||||
<h2>Performance Metrics</h2>
|
||||
<div class="metric">Upload Performance: $($uploadResult.AverageSpeedFormatted)</div>
|
||||
<div class="metric">Download Performance: $($downloadResult.AverageSpeedFormatted)</div>
|
||||
</div>
|
||||
|
||||
<div class="section">
|
||||
<h2>Security Audit</h2>
|
||||
<p>$($auditResults.Count) buckets audited</p>
|
||||
<p>Buckets with policies: $(($auditResults | Where-Object HasPolicy).Count)</p>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
"@
|
||||
|
||||
$htmlReport | Out-File -FilePath $reportFile -Encoding UTF8
|
||||
Write-Log "HTML report generated: $reportFile"
|
||||
"📊 HTML report generated: $reportFile"
|
||||
""
|
||||
|
||||
# Clean up demo resources
|
||||
Write-Log "Cleaning up demo resources"
|
||||
Remove-MinIOBucket -BucketName $backupBucket -Force
|
||||
Remove-MinIOBucket -BucketName $lifecycleBucket -Force
|
||||
Remove-MinIOBucket -BucketName $perfBucket -Force
|
||||
Remove-Item $backupData.Keys, $perfTestFile, "downloaded-$perfTestFile" -Force -ErrorAction SilentlyContinue
|
||||
|
||||
Write-Log "Enterprise automation demo completed successfully"
|
||||
"✅ Enterprise automation demo completed successfully"
|
||||
"📋 Check the log file: $logFile"
|
||||
"📊 View the report: $reportFile"
|
||||
|
||||
} catch {
|
||||
Write-Log "❌ Enterprise automation demo failed: $($_.Exception.Message)" "ERROR"
|
||||
"❌ Error: $($_.Exception.Message)"
|
||||
}
|
||||
@@ -0,0 +1,231 @@
|
||||
# PSMinIO Examples
|
||||
|
||||
This directory contains comprehensive examples demonstrating various PSMinIO capabilities and usage patterns.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Before running these examples:
|
||||
|
||||
1. **Install PSMinIO**: Import the module using `Import-Module ..\..\Module\PSMinIO\PSMinIO.psd1`
|
||||
2. **MinIO Server**: Have access to a MinIO server or compatible S3 service
|
||||
3. **Credentials**: Update the connection details in each script with your actual values:
|
||||
```powershell
|
||||
$endpoint = "https://your-minio-server.com"
|
||||
$accessKey = "your-access-key"
|
||||
$secretKey = "your-secret-key"
|
||||
```
|
||||
|
||||
## Example Scripts
|
||||
|
||||
### 01-Basic-Operations.ps1
|
||||
**Fundamental MinIO operations for beginners**
|
||||
|
||||
Demonstrates:
|
||||
- Connecting to MinIO server
|
||||
- Creating and managing buckets
|
||||
- Basic file upload and download
|
||||
- Object listing and verification
|
||||
- Clean resource management
|
||||
|
||||
**Best for**: First-time users, basic automation scripts
|
||||
|
||||
---
|
||||
|
||||
### 02-Advanced-Object-Listing.ps1
|
||||
**Comprehensive object listing and filtering capabilities**
|
||||
|
||||
Demonstrates:
|
||||
- Advanced filtering with prefixes
|
||||
- Sorting by multiple criteria (Name, Size, LastModified, ETag)
|
||||
- Result pagination with MaxObjects
|
||||
- Files-only filtering (excluding directories)
|
||||
- Complex query combinations
|
||||
|
||||
**Best for**: Data discovery, content management, reporting
|
||||
|
||||
---
|
||||
|
||||
### 03-Directory-Management.ps1
|
||||
**Creating and managing directory structures**
|
||||
|
||||
Demonstrates:
|
||||
- Explicit folder creation with New-MinIOFolder
|
||||
- Automatic directory creation with BucketDirectory
|
||||
- Multi-level nested directory structures
|
||||
- Directory-based file organization
|
||||
- Date-based directory patterns
|
||||
|
||||
**Best for**: Organized file storage, hierarchical data management
|
||||
|
||||
---
|
||||
|
||||
### 04-Chunked-Operations.ps1
|
||||
**Large file handling with chunked transfers**
|
||||
|
||||
Demonstrates:
|
||||
- Chunked uploads with configurable chunk sizes
|
||||
- Chunked downloads for large files
|
||||
- Multi-layer progress tracking
|
||||
- Performance comparison (regular vs chunked)
|
||||
- Chunk size optimization guidelines
|
||||
|
||||
**Best for**: Large file transfers, bandwidth optimization, resume capability
|
||||
|
||||
---
|
||||
|
||||
### 05-Bulk-Operations.ps1
|
||||
**Batch processing and bulk file operations**
|
||||
|
||||
Demonstrates:
|
||||
- Multiple file uploads to organized directories
|
||||
- Bulk download operations with filtering
|
||||
- File analysis and statistics
|
||||
- Pattern-based file processing
|
||||
- Selective cleanup operations
|
||||
|
||||
**Best for**: Data migration, batch processing, automated workflows
|
||||
|
||||
---
|
||||
|
||||
### 06-Enterprise-Automation.ps1
|
||||
**Enterprise-grade automation and monitoring**
|
||||
|
||||
Demonstrates:
|
||||
- Infrastructure health checks
|
||||
- Storage statistics and monitoring
|
||||
- Automated backup operations
|
||||
- Data lifecycle management
|
||||
- Security and compliance auditing
|
||||
- Performance monitoring
|
||||
- HTML report generation
|
||||
|
||||
**Best for**: Enterprise environments, monitoring systems, compliance reporting
|
||||
|
||||
## Usage Patterns
|
||||
|
||||
### Quick Start
|
||||
```powershell
|
||||
# Run a basic example
|
||||
.\01-Basic-Operations.ps1
|
||||
```
|
||||
|
||||
### Customization
|
||||
Each script includes configuration variables at the top. Modify these for your environment:
|
||||
|
||||
```powershell
|
||||
# Connection details
|
||||
$endpoint = "https://your-minio-server.com"
|
||||
$accessKey = "your-access-key"
|
||||
$secretKey = "your-secret-key"
|
||||
|
||||
# Optional: Bucket naming
|
||||
$bucketName = "your-custom-bucket-name"
|
||||
```
|
||||
|
||||
### Integration
|
||||
These examples can be integrated into larger automation workflows:
|
||||
|
||||
```powershell
|
||||
# Source common functions
|
||||
. .\examples\06-Enterprise-Automation.ps1
|
||||
|
||||
# Use specific functions in your scripts
|
||||
Write-Log "Starting custom automation"
|
||||
$stats = Get-MinIOStats
|
||||
```
|
||||
|
||||
## Common Scenarios
|
||||
|
||||
### Development Environment Setup
|
||||
```powershell
|
||||
# Local MinIO setup
|
||||
$endpoint = "http://localhost:9000"
|
||||
$accessKey = "minioadmin"
|
||||
$secretKey = "minioadmin"
|
||||
```
|
||||
|
||||
### Production Environment
|
||||
```powershell
|
||||
# Production setup with SSL
|
||||
$endpoint = "https://minio.company.com"
|
||||
$accessKey = $env:MINIO_ACCESS_KEY
|
||||
$secretKey = $env:MINIO_SECRET_KEY
|
||||
```
|
||||
|
||||
### AWS S3 Compatibility
|
||||
```powershell
|
||||
# AWS S3 setup
|
||||
$endpoint = "https://s3.amazonaws.com"
|
||||
$accessKey = $env:AWS_ACCESS_KEY_ID
|
||||
$secretKey = $env:AWS_SECRET_ACCESS_KEY
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Error Handling
|
||||
All examples include comprehensive error handling:
|
||||
```powershell
|
||||
try {
|
||||
# MinIO operations
|
||||
} catch {
|
||||
"❌ Error: $($_.Exception.Message)"
|
||||
} finally {
|
||||
# Cleanup operations
|
||||
}
|
||||
```
|
||||
|
||||
### Resource Cleanup
|
||||
Examples automatically clean up created resources:
|
||||
- Remove test buckets
|
||||
- Delete uploaded files
|
||||
- Clean local temporary files
|
||||
|
||||
### Logging
|
||||
Enterprise examples include structured logging:
|
||||
```powershell
|
||||
function Write-Log {
|
||||
param([string]$Message, [string]$Level = "INFO")
|
||||
$timestamp = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
|
||||
"[$timestamp] [$Level] $Message"
|
||||
}
|
||||
```
|
||||
|
||||
### Performance Considerations
|
||||
- Use chunked operations for files > 10MB
|
||||
- Implement appropriate chunk sizes based on network conditions
|
||||
- Monitor transfer speeds and adjust accordingly
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Connection Issues
|
||||
- Verify endpoint URL and port
|
||||
- Check SSL/TLS configuration
|
||||
- Validate access credentials
|
||||
- Test network connectivity
|
||||
|
||||
### Permission Issues
|
||||
- Ensure proper bucket permissions
|
||||
- Verify access key has required privileges
|
||||
- Check bucket policies if applicable
|
||||
|
||||
### Performance Issues
|
||||
- Adjust chunk sizes for large files
|
||||
- Monitor network bandwidth
|
||||
- Consider parallel operations for bulk transfers
|
||||
|
||||
## Contributing
|
||||
|
||||
When adding new examples:
|
||||
1. Follow the existing naming convention
|
||||
2. Include comprehensive error handling
|
||||
3. Add resource cleanup
|
||||
4. Document the example purpose and usage
|
||||
5. Update this README with the new example
|
||||
|
||||
## Support
|
||||
|
||||
For issues with these examples:
|
||||
- Check the main [PSMinIO documentation](../docs/USAGE.md)
|
||||
- Review error messages and logs
|
||||
- Verify your MinIO server configuration
|
||||
- Test with basic operations first
|
||||
Reference in New Issue
Block a user