mirror of
https://github.com/Grace-Solutions/PSMinIO.git
synced 2026-07-26 14:58:13 +00:00
Initial commit: PSMinIO module with chunked transfer support
This commit is contained in:
@@ -0,0 +1,166 @@
|
||||
# PSMinIO Changelog
|
||||
|
||||
All notable changes to this project will be documented in this file.
|
||||
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [2025.07.10.1200] - 2025-07-10
|
||||
|
||||
### Added
|
||||
|
||||
#### Core Infrastructure
|
||||
- **MinIOConfiguration**: Singleton configuration management with JSON persistence
|
||||
- **MinIOLogger**: Centralized logging utility with timestamp formatting (yyyy/MM/dd HH:mm:ss.fff)
|
||||
- **MinIOClientWrapper**: Synchronous wrapper for async MinIO operations using `.GetAwaiter().GetResult()`
|
||||
- **MinIOBaseCmdlet**: Base class for all cmdlets with common functionality
|
||||
- **ProgressReporter**: Upload/download progress tracking with percentages and time estimates
|
||||
|
||||
#### Bucket Operations
|
||||
- **Get-MinIOBucket**: List buckets with optional statistics gathering
|
||||
- **New-MinIOBucket**: Create buckets with region support and validation
|
||||
- **Remove-MinIOBucket**: Delete buckets with optional object removal
|
||||
- **Test-MinIOBucketExists**: Check bucket existence with detailed information
|
||||
|
||||
#### Object Operations
|
||||
- **Get-MinIOObject**: List objects with filtering, sorting, and pagination
|
||||
- **New-MinIOObject**: Upload files with progress reporting and content type detection
|
||||
- **Get-MinIOObjectContent**: Download objects with progress reporting
|
||||
- **Remove-MinIOObject**: Delete objects with prefix support for batch operations
|
||||
|
||||
#### Security & Policy Management
|
||||
- **Get-MinIOBucketPolicy**: Retrieve bucket policies as JSON or structured objects
|
||||
- **Set-MinIOBucketPolicy**: Set policies from JSON, files, or predefined canned policies
|
||||
|
||||
#### Configuration & Utilities
|
||||
- **Set-MinIOConfig**: Configure MinIO connection with validation and testing
|
||||
- **Get-MinIOConfig**: View configuration with optional sensitive data masking
|
||||
- **Get-MinIOStats**: Comprehensive statistics with per-bucket details
|
||||
|
||||
#### PowerShell Integration
|
||||
- **Type Definitions**: Custom .ps1xml files for formatted output
|
||||
- **Format Definitions**: Table views for all major object types
|
||||
- **Parameter Validation**: Comprehensive input validation and error handling
|
||||
- **ShouldProcess Support**: All destructive operations support -WhatIf and -Confirm
|
||||
|
||||
### Design Decisions
|
||||
|
||||
#### .NET Standard 2.0 Compatibility
|
||||
- **Target Framework**: .NET Standard 2.0 for maximum compatibility
|
||||
- **PowerShell Support**: Compatible with PowerShell 5.1 (.NET Framework 4.7.2) and PowerShell 7+
|
||||
- **Dependency Management**: Uses PowerShellStandard.Library 5.1.1 for cmdlet base classes
|
||||
|
||||
#### Synchronous Operations Only
|
||||
- **No Async/Await**: All operations are synchronous for PowerShell compatibility
|
||||
- **Wrapper Strategy**: Uses `Task.Run().GetAwaiter().GetResult()` pattern
|
||||
- **Cancellation Support**: Implements CancellationToken for operation cancellation
|
||||
|
||||
#### Logging Strategy
|
||||
- **Conditional Logging**: Only logs when `-Verbose` is specified
|
||||
- **Timestamp Format**: Consistent yyyy/MM/dd HH:mm:ss.fff format
|
||||
- **Centralized Utility**: Single MinIOLogger class for all logging operations
|
||||
- **Error Categorization**: Proper PowerShell ErrorCategory assignment
|
||||
|
||||
#### Progress Reporting
|
||||
- **Upload/Download Progress**: Real-time progress with bytes transferred
|
||||
- **Time Estimates**: Calculates remaining time based on current speed
|
||||
- **Throttled Updates**: Updates every 100ms to avoid console flooding
|
||||
- **Formatted Display**: Human-readable size formatting (B, KB, MB, GB, etc.)
|
||||
|
||||
#### Configuration Management
|
||||
- **Singleton Pattern**: Single configuration instance across the module
|
||||
- **Persistent Storage**: JSON configuration file in user's AppData
|
||||
- **Validation**: Comprehensive validation before client creation
|
||||
- **Security**: Sensitive data masking in display output
|
||||
|
||||
#### Error Handling
|
||||
- **Comprehensive Validation**: Input validation at multiple levels
|
||||
- **Proper Error Categories**: Uses appropriate PowerShell ErrorCategory values
|
||||
- **Graceful Degradation**: Operations continue when possible, warn on failures
|
||||
- **Detailed Error Messages**: Includes context and suggestions for resolution
|
||||
|
||||
#### Performance Considerations
|
||||
- **Lazy Client Creation**: MinIO client created only when needed
|
||||
- **Resource Disposal**: Proper disposal of clients and resources
|
||||
- **Batch Operations**: Support for bulk operations with progress reporting
|
||||
- **Configurable Limits**: MaxObjects parameters to prevent performance issues
|
||||
|
||||
#### PowerShell Best Practices
|
||||
- **Parameter Sets**: Logical grouping of related parameters
|
||||
- **Pipeline Support**: ValueFromPipeline and ValueFromPipelineByPropertyName
|
||||
- **Aliases**: Common aliases for frequently used parameters
|
||||
- **Help Integration**: Comprehensive parameter documentation
|
||||
- **Output Types**: Strongly typed output objects
|
||||
|
||||
### Technical Implementation
|
||||
|
||||
#### Synchronous Wrapper Pattern
|
||||
```csharp
|
||||
public bool BucketExists(string bucketName)
|
||||
{
|
||||
var args = new BucketExistsArgs().WithBucket(bucketName);
|
||||
return Task.Run(async () =>
|
||||
await _client.BucketExistsAsync(args, CancellationToken))
|
||||
.GetAwaiter().GetResult();
|
||||
}
|
||||
```
|
||||
|
||||
#### Progress Reporting Implementation
|
||||
```csharp
|
||||
var progressReporter = new ProgressReporter(
|
||||
this, "Uploading Object", $"Uploading {fileInfo.Name}", fileSize, 1);
|
||||
|
||||
var etag = Client.UploadFile(BucketName, ObjectName, FilePath, ContentType,
|
||||
bytesTransferred => progressReporter.UpdateProgress(bytesTransferred));
|
||||
```
|
||||
|
||||
#### Logging Pattern
|
||||
```csharp
|
||||
MinIOLogger.WriteVerbose(this, "Operation started: {0}", operationName);
|
||||
// ... operation code ...
|
||||
MinIOLogger.WriteVerbose(this, "Operation completed: {0}", operationName);
|
||||
```
|
||||
|
||||
### Dependencies
|
||||
|
||||
- **Minio**: 5.0.0 - Core MinIO .NET SDK
|
||||
- **PowerShellStandard.Library**: 5.1.1 - PowerShell cmdlet base classes
|
||||
- **System.Text.Json**: 6.0.0 - JSON serialization for configuration and policies
|
||||
|
||||
### Breaking Changes
|
||||
|
||||
None - Initial release.
|
||||
|
||||
### Security Considerations
|
||||
|
||||
- **Credential Storage**: Configuration file stored in user's AppData directory
|
||||
- **Sensitive Data Masking**: Access keys and secret keys masked in output by default
|
||||
- **SSL by Default**: UseSSL defaults to true for secure connections
|
||||
- **Input Validation**: Comprehensive validation to prevent injection attacks
|
||||
|
||||
### Known Limitations
|
||||
|
||||
- **Large Bucket Performance**: Object counting can be slow for buckets with many objects
|
||||
- **Synchronous Only**: No async operations available (by design)
|
||||
- **Windows Paths**: File path handling optimized for Windows (cross-platform compatible)
|
||||
|
||||
### Future Enhancements
|
||||
|
||||
- **Multipart Upload Support**: For large files
|
||||
- **Presigned URL Generation**: For temporary access
|
||||
- **Server-Side Encryption**: Configuration and management
|
||||
- **Lifecycle Policies**: Bucket lifecycle management
|
||||
- **Notification Configuration**: Event notification setup
|
||||
|
||||
---
|
||||
|
||||
## Version Numbering
|
||||
|
||||
This project uses a date-based versioning scheme: `YYYY.MM.DD.HHMM`
|
||||
|
||||
- **YYYY**: Year (2025)
|
||||
- **MM**: Month (07)
|
||||
- **DD**: Day (10)
|
||||
- **HHMM**: Hour and minute (1200 = 12:00 PM)
|
||||
|
||||
This ensures chronological ordering and makes it easy to identify when a version was released.
|
||||
@@ -0,0 +1,401 @@
|
||||
# PSMinIO Examples
|
||||
|
||||
This document contains practical examples for using the PSMinIO PowerShell module.
|
||||
|
||||
## Basic Setup and Configuration
|
||||
|
||||
### Example 1: Initial Setup for Local MinIO
|
||||
|
||||
```powershell
|
||||
# Import the module
|
||||
Import-Module .\PSMinIO.psd1
|
||||
|
||||
# Configure for local MinIO instance
|
||||
Set-MinIOConfig -Endpoint "localhost:9000" `
|
||||
-AccessKey "minioadmin" `
|
||||
-SecretKey "minioadmin" `
|
||||
-NoSSL `
|
||||
-TestConnection `
|
||||
-SaveToDisk
|
||||
|
||||
# Verify configuration
|
||||
Get-MinIOConfig -Detailed
|
||||
```
|
||||
|
||||
### Example 2: Production Setup with SSL
|
||||
|
||||
```powershell
|
||||
# Configure for production MinIO cluster
|
||||
Set-MinIOConfig -Endpoint "minio.company.com:9000" `
|
||||
-AccessKey $env:MINIO_ACCESS_KEY `
|
||||
-SecretKey $env:MINIO_SECRET_KEY `
|
||||
-UseSSL `
|
||||
-Region "us-east-1" `
|
||||
-TimeoutSeconds 60 `
|
||||
-TestConnection
|
||||
|
||||
# Test the connection
|
||||
Get-MinIOConfig -TestConnection
|
||||
```
|
||||
|
||||
## Bucket Management Examples
|
||||
|
||||
### Example 3: Creating and Managing Buckets
|
||||
|
||||
```powershell
|
||||
# Create buckets for different purposes
|
||||
New-MinIOBucket -BucketName "company-documents" -Region "us-east-1" -PassThru
|
||||
New-MinIOBucket -BucketName "user-uploads" -Region "us-west-2" -PassThru
|
||||
New-MinIOBucket -BucketName "application-logs" -Region "eu-west-1" -PassThru
|
||||
|
||||
# List all buckets with statistics
|
||||
Get-MinIOBucket -IncludeStatistics
|
||||
|
||||
# Check if specific buckets exist
|
||||
$buckets = @("company-documents", "user-uploads", "temp-bucket")
|
||||
foreach ($bucket in $buckets) {
|
||||
$exists = Test-MinIOBucketExists -BucketName $bucket
|
||||
Write-Host "Bucket '$bucket' exists: $exists"
|
||||
}
|
||||
```
|
||||
|
||||
### Example 4: Bucket Cleanup
|
||||
|
||||
```powershell
|
||||
# Find and remove temporary buckets
|
||||
Get-MinIOBucket | Where-Object { $_.Name -like "temp-*" } | ForEach-Object {
|
||||
Write-Host "Removing temporary bucket: $($_.Name)"
|
||||
Remove-MinIOBucket -BucketName $_.Name -RemoveObjects -Force
|
||||
}
|
||||
|
||||
# Remove old test buckets (older than 30 days)
|
||||
Get-MinIOBucket | Where-Object {
|
||||
$_.Name -like "test-*" -and $_.Created -lt (Get-Date).AddDays(-30)
|
||||
} | ForEach-Object {
|
||||
Write-Host "Removing old test bucket: $($_.Name) (Created: $($_.Created))"
|
||||
Remove-MinIOBucket -BucketName $_.Name -RemoveObjects -Force
|
||||
}
|
||||
```
|
||||
|
||||
## File Upload and Download Examples
|
||||
|
||||
### Example 5: Bulk File Upload
|
||||
|
||||
```powershell
|
||||
# Upload all PDF files from a directory
|
||||
$sourceDir = "C:\Documents\Reports"
|
||||
$bucketName = "company-documents"
|
||||
|
||||
Get-ChildItem -Path $sourceDir -Filter "*.pdf" -Recurse | ForEach-Object {
|
||||
$relativePath = $_.FullName.Substring($sourceDir.Length + 1).Replace('\', '/')
|
||||
$objectName = "reports/$relativePath"
|
||||
|
||||
Write-Host "Uploading: $($_.Name) -> $objectName"
|
||||
|
||||
try {
|
||||
New-MinIOObject -BucketName $bucketName `
|
||||
-ObjectName $objectName `
|
||||
-FilePath $_.FullName `
|
||||
-Verbose
|
||||
Write-Host "✓ Successfully uploaded: $($_.Name)" -ForegroundColor Green
|
||||
} catch {
|
||||
Write-Host "✗ Failed to upload: $($_.Name) - $($_.Exception.Message)" -ForegroundColor Red
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Example 6: Backup Script
|
||||
|
||||
```powershell
|
||||
# Daily backup script
|
||||
param(
|
||||
[Parameter(Mandatory)]
|
||||
[string]$SourcePath,
|
||||
|
||||
[Parameter(Mandatory)]
|
||||
[string]$BucketName,
|
||||
|
||||
[string]$BackupPrefix = "backups"
|
||||
)
|
||||
|
||||
$timestamp = Get-Date -Format "yyyy-MM-dd_HH-mm-ss"
|
||||
$backupFolder = "$BackupPrefix/$timestamp"
|
||||
|
||||
Write-Host "Starting backup of '$SourcePath' to bucket '$BucketName'"
|
||||
|
||||
# Create a compressed archive
|
||||
$tempZip = "$env:TEMP\backup_$timestamp.zip"
|
||||
Compress-Archive -Path $SourcePath -DestinationPath $tempZip -Force
|
||||
|
||||
try {
|
||||
# Upload the backup
|
||||
$objectName = "$backupFolder/backup.zip"
|
||||
New-MinIOObject -BucketName $BucketName `
|
||||
-ObjectName $objectName `
|
||||
-FilePath $tempZip `
|
||||
-PassThru
|
||||
|
||||
Write-Host "✓ Backup completed successfully" -ForegroundColor Green
|
||||
|
||||
# Clean up old backups (keep last 7 days)
|
||||
$cutoffDate = (Get-Date).AddDays(-7)
|
||||
Get-MinIOObject -BucketName $BucketName -Prefix $BackupPrefix |
|
||||
Where-Object { $_.LastModified -lt $cutoffDate } |
|
||||
ForEach-Object {
|
||||
Write-Host "Removing old backup: $($_.Name)"
|
||||
Remove-MinIOObject -BucketName $BucketName -ObjectName $_.Name -Force
|
||||
}
|
||||
|
||||
} finally {
|
||||
# Clean up temporary file
|
||||
if (Test-Path $tempZip) {
|
||||
Remove-Item $tempZip -Force
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Example 7: Bulk Download
|
||||
|
||||
```powershell
|
||||
# Download all objects with specific prefix
|
||||
$bucketName = "company-documents"
|
||||
$prefix = "reports/2025/"
|
||||
$downloadDir = "C:\Downloads\Reports"
|
||||
|
||||
# Ensure download directory exists
|
||||
if (!(Test-Path $downloadDir)) {
|
||||
New-Item -ItemType Directory -Path $downloadDir -Force
|
||||
}
|
||||
|
||||
# Get all objects with the prefix
|
||||
$objects = Get-MinIOObject -BucketName $bucketName -Prefix $prefix
|
||||
|
||||
Write-Host "Found $($objects.Count) objects to download"
|
||||
|
||||
foreach ($obj in $objects) {
|
||||
# Skip directories
|
||||
if ($obj.IsDirectory) { continue }
|
||||
|
||||
# Create local file path
|
||||
$relativePath = $obj.Name.Substring($prefix.Length)
|
||||
$localPath = Join-Path $downloadDir $relativePath.Replace('/', '\')
|
||||
$localDir = Split-Path $localPath -Parent
|
||||
|
||||
# Ensure local directory exists
|
||||
if (!(Test-Path $localDir)) {
|
||||
New-Item -ItemType Directory -Path $localDir -Force
|
||||
}
|
||||
|
||||
Write-Host "Downloading: $($obj.Name) -> $localPath"
|
||||
|
||||
try {
|
||||
Get-MinIOObjectContent -BucketName $bucketName `
|
||||
-ObjectName $obj.Name `
|
||||
-FilePath $localPath `
|
||||
-Force
|
||||
Write-Host "✓ Downloaded: $($obj.GetFileName())" -ForegroundColor Green
|
||||
} catch {
|
||||
Write-Host "✗ Failed to download: $($obj.Name) - $($_.Exception.Message)" -ForegroundColor Red
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Security and Policy Examples
|
||||
|
||||
### Example 8: Setting Up Public Read Access
|
||||
|
||||
```powershell
|
||||
# Create a bucket for public assets
|
||||
New-MinIOBucket -BucketName "public-assets"
|
||||
|
||||
# Set read-only policy for public access
|
||||
Set-MinIOBucketPolicy -BucketName "public-assets" `
|
||||
-CannedPolicy "ReadOnly" `
|
||||
-Prefix "*"
|
||||
|
||||
# Verify the policy
|
||||
Get-MinIOBucketPolicy -BucketName "public-assets" -AsObject
|
||||
```
|
||||
|
||||
### Example 9: Custom Policy for Upload-Only Access
|
||||
|
||||
```powershell
|
||||
$uploadOnlyPolicy = @"
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Principal": {"AWS": "*"},
|
||||
"Action": ["s3:PutObject"],
|
||||
"Resource": ["arn:aws:s3:::user-uploads/uploads/*"]
|
||||
}
|
||||
]
|
||||
}
|
||||
"@
|
||||
|
||||
Set-MinIOBucketPolicy -BucketName "user-uploads" `
|
||||
-PolicyJson $uploadOnlyPolicy
|
||||
|
||||
# Test the policy
|
||||
Get-MinIOBucketPolicy -BucketName "user-uploads" -PrettyPrint
|
||||
```
|
||||
|
||||
## Monitoring and Statistics Examples
|
||||
|
||||
### Example 10: Storage Usage Report
|
||||
|
||||
```powershell
|
||||
# Generate comprehensive storage report
|
||||
$stats = Get-MinIOStats -IncludeBucketDetails -IncludeObjectCounts
|
||||
|
||||
Write-Host "=== MinIO Storage Report ===" -ForegroundColor Cyan
|
||||
Write-Host "Generated: $(Get-Date)" -ForegroundColor Gray
|
||||
Write-Host ""
|
||||
|
||||
Write-Host "Overall Statistics:" -ForegroundColor Yellow
|
||||
Write-Host " Total Buckets: $($stats.TotalBuckets)"
|
||||
Write-Host " Total Objects: $($stats.TotalObjects)"
|
||||
Write-Host " Total Size: $($stats.TotalSizeFormatted)"
|
||||
Write-Host " Average Object Size: $($stats.AverageObjectSizeFormatted)"
|
||||
Write-Host ""
|
||||
|
||||
Write-Host "Bucket Details:" -ForegroundColor Yellow
|
||||
$stats.BucketDetails | Sort-Object Size -Descending | ForEach-Object {
|
||||
Write-Host " $($_.Name):"
|
||||
Write-Host " Objects: $($_.ObjectCount)"
|
||||
Write-Host " Size: $($_.SizeFormatted)"
|
||||
Write-Host " Created: $($_.Created)"
|
||||
Write-Host ""
|
||||
}
|
||||
```
|
||||
|
||||
### Example 11: Health Check Script
|
||||
|
||||
```powershell
|
||||
# MinIO health check script
|
||||
function Test-MinIOHealth {
|
||||
param(
|
||||
[string]$TestBucketName = "health-check-$(Get-Date -Format 'yyyyMMdd')"
|
||||
)
|
||||
|
||||
$results = @{
|
||||
ConfigurationValid = $false
|
||||
ConnectionSuccessful = $false
|
||||
BucketOperations = $false
|
||||
ObjectOperations = $false
|
||||
OverallHealth = $false
|
||||
}
|
||||
|
||||
try {
|
||||
# Test 1: Configuration
|
||||
$config = Get-MinIOConfig
|
||||
$results.ConfigurationValid = $config.IsValid
|
||||
Write-Host "✓ Configuration is valid" -ForegroundColor Green
|
||||
|
||||
# Test 2: Connection
|
||||
$connectionTest = Get-MinIOConfig -TestConnection
|
||||
$results.ConnectionSuccessful = $connectionTest.ConnectionStatus -eq "Success"
|
||||
Write-Host "✓ Connection successful" -ForegroundColor Green
|
||||
|
||||
# Test 3: Bucket operations
|
||||
New-MinIOBucket -BucketName $TestBucketName -Force | Out-Null
|
||||
$bucketExists = Test-MinIOBucketExists -BucketName $TestBucketName
|
||||
$results.BucketOperations = $bucketExists
|
||||
Write-Host "✓ Bucket operations working" -ForegroundColor Green
|
||||
|
||||
# Test 4: Object operations
|
||||
$testFile = "$env:TEMP\minio-health-test.txt"
|
||||
"Health check test file - $(Get-Date)" | Out-File -FilePath $testFile
|
||||
|
||||
New-MinIOObject -BucketName $TestBucketName `
|
||||
-ObjectName "health-test.txt" `
|
||||
-FilePath $testFile | Out-Null
|
||||
|
||||
$objects = Get-MinIOObject -BucketName $TestBucketName -ObjectName "health-test.txt"
|
||||
$results.ObjectOperations = $objects.Count -eq 1
|
||||
Write-Host "✓ Object operations working" -ForegroundColor Green
|
||||
|
||||
# Cleanup
|
||||
Remove-MinIOObject -BucketName $TestBucketName -ObjectName "health-test.txt" -Force
|
||||
Remove-MinIOBucket -BucketName $TestBucketName -Force
|
||||
Remove-Item $testFile -Force
|
||||
|
||||
$results.OverallHealth = $results.ConfigurationValid -and
|
||||
$results.ConnectionSuccessful -and
|
||||
$results.BucketOperations -and
|
||||
$results.ObjectOperations
|
||||
|
||||
Write-Host "✓ Overall health check passed" -ForegroundColor Green
|
||||
|
||||
} catch {
|
||||
Write-Host "✗ Health check failed: $($_.Exception.Message)" -ForegroundColor Red
|
||||
}
|
||||
|
||||
return $results
|
||||
}
|
||||
|
||||
# Run health check
|
||||
Test-MinIOHealth
|
||||
```
|
||||
|
||||
## Advanced Automation Examples
|
||||
|
||||
### Example 12: Log Rotation and Archival
|
||||
|
||||
```powershell
|
||||
# Automated log rotation script
|
||||
param(
|
||||
[string]$LogBucket = "application-logs",
|
||||
[int]$RetentionDays = 90,
|
||||
[string]$ArchiveBucket = "archived-logs"
|
||||
)
|
||||
|
||||
$cutoffDate = (Get-Date).AddDays(-$RetentionDays)
|
||||
|
||||
Write-Host "Starting log rotation process..."
|
||||
Write-Host "Archiving logs older than: $cutoffDate"
|
||||
|
||||
# Get old log files
|
||||
$oldLogs = Get-MinIOObject -BucketName $LogBucket |
|
||||
Where-Object { $_.LastModified -lt $cutoffDate -and !$_.IsDirectory }
|
||||
|
||||
Write-Host "Found $($oldLogs.Count) log files to archive"
|
||||
|
||||
foreach ($log in $oldLogs) {
|
||||
try {
|
||||
# Download log file
|
||||
$tempFile = "$env:TEMP\$($log.GetFileName())"
|
||||
Get-MinIOObjectContent -BucketName $LogBucket `
|
||||
-ObjectName $log.Name `
|
||||
-FilePath $tempFile
|
||||
|
||||
# Compress the log file
|
||||
$compressedFile = "$tempFile.gz"
|
||||
# Note: You would use a compression library here
|
||||
# For this example, we'll just rename
|
||||
Move-Item $tempFile $compressedFile
|
||||
|
||||
# Upload to archive bucket with date prefix
|
||||
$archiveObjectName = "archived/$(Get-Date $log.LastModified -Format 'yyyy/MM/dd')/$($log.GetFileName()).gz"
|
||||
New-MinIOObject -BucketName $ArchiveBucket `
|
||||
-ObjectName $archiveObjectName `
|
||||
-FilePath $compressedFile
|
||||
|
||||
# Remove original log file
|
||||
Remove-MinIOObject -BucketName $LogBucket -ObjectName $log.Name -Force
|
||||
|
||||
# Clean up temp file
|
||||
Remove-Item $compressedFile -Force
|
||||
|
||||
Write-Host "✓ Archived: $($log.Name)" -ForegroundColor Green
|
||||
|
||||
} catch {
|
||||
Write-Host "✗ Failed to archive: $($log.Name) - $($_.Exception.Message)" -ForegroundColor Red
|
||||
}
|
||||
}
|
||||
|
||||
Write-Host "Log rotation completed"
|
||||
```
|
||||
|
||||
These examples demonstrate the practical usage of PSMinIO for various scenarios including setup, file management, security configuration, monitoring, and automation. Each example includes error handling and best practices for production use.
|
||||
+400
@@ -0,0 +1,400 @@
|
||||
# PSMinIO Usage Guide
|
||||
|
||||
This guide provides comprehensive examples and usage patterns for the PSMinIO PowerShell module.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Installation](#installation)
|
||||
- [Configuration](#configuration)
|
||||
- [Bucket Operations](#bucket-operations)
|
||||
- [Object Operations](#object-operations)
|
||||
- [Security and Policies](#security-and-policies)
|
||||
- [Statistics and Monitoring](#statistics-and-monitoring)
|
||||
- [Advanced Usage](#advanced-usage)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
|
||||
## Installation
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- PowerShell 5.1+ or PowerShell 7+
|
||||
- .NET Framework 4.7.2+ (for PowerShell 5.1) or .NET Core/.NET 5+ (for PowerShell 7+)
|
||||
|
||||
### Import the Module
|
||||
|
||||
```powershell
|
||||
# Import the module
|
||||
Import-Module .\PSMinIO.psd1
|
||||
|
||||
# Verify the module is loaded
|
||||
Get-Module PSMinIO
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Basic Configuration
|
||||
|
||||
```powershell
|
||||
# Set up MinIO connection
|
||||
Set-MinIOConfig -Endpoint "minio.example.com:9000" `
|
||||
-AccessKey "your-access-key" `
|
||||
-SecretKey "your-secret-key" `
|
||||
-UseSSL
|
||||
|
||||
# For local development (no SSL)
|
||||
Set-MinIOConfig -Endpoint "localhost:9000" `
|
||||
-AccessKey "minioadmin" `
|
||||
-SecretKey "minioadmin" `
|
||||
-NoSSL
|
||||
```
|
||||
|
||||
### Advanced Configuration
|
||||
|
||||
```powershell
|
||||
# Set configuration with custom region and timeout
|
||||
Set-MinIOConfig -Endpoint "minio.example.com:9000" `
|
||||
-AccessKey "your-access-key" `
|
||||
-SecretKey "your-secret-key" `
|
||||
-UseSSL `
|
||||
-Region "us-west-2" `
|
||||
-TimeoutSeconds 60 `
|
||||
-SaveToDisk `
|
||||
-TestConnection
|
||||
```
|
||||
|
||||
### View Current Configuration
|
||||
|
||||
```powershell
|
||||
# Basic configuration view
|
||||
Get-MinIOConfig
|
||||
|
||||
# Detailed configuration with connection test
|
||||
Get-MinIOConfig -Detailed -TestConnection
|
||||
|
||||
# Show sensitive information (use with caution)
|
||||
Get-MinIOConfig -ShowSensitive
|
||||
```
|
||||
|
||||
## Bucket Operations
|
||||
|
||||
### Create Buckets
|
||||
|
||||
```powershell
|
||||
# Create a simple bucket
|
||||
New-MinIOBucket -BucketName "my-data-bucket"
|
||||
|
||||
# Create bucket with specific region
|
||||
New-MinIOBucket -BucketName "eu-data-bucket" -Region "eu-west-1"
|
||||
|
||||
# Create bucket and return information
|
||||
New-MinIOBucket -BucketName "logs-bucket" -PassThru
|
||||
|
||||
# Force creation (no error if exists)
|
||||
New-MinIOBucket -BucketName "existing-bucket" -Force
|
||||
```
|
||||
|
||||
### List Buckets
|
||||
|
||||
```powershell
|
||||
# List all buckets
|
||||
Get-MinIOBucket
|
||||
|
||||
# Get specific bucket information
|
||||
Get-MinIOBucket -BucketName "my-data-bucket"
|
||||
|
||||
# Include statistics (object count and size)
|
||||
Get-MinIOBucket -IncludeStatistics
|
||||
|
||||
# Get bucket with statistics
|
||||
Get-MinIOBucket -BucketName "my-data-bucket" -IncludeStatistics
|
||||
```
|
||||
|
||||
### Check Bucket Existence
|
||||
|
||||
```powershell
|
||||
# Simple existence check
|
||||
Test-MinIOBucketExists -BucketName "my-data-bucket"
|
||||
|
||||
# Detailed existence information
|
||||
Test-MinIOBucketExists -BucketName "my-data-bucket" -Detailed
|
||||
```
|
||||
|
||||
### Remove Buckets
|
||||
|
||||
```powershell
|
||||
# Remove empty bucket
|
||||
Remove-MinIOBucket -BucketName "old-bucket"
|
||||
|
||||
# Remove bucket and all its objects
|
||||
Remove-MinIOBucket -BucketName "temp-bucket" -RemoveObjects
|
||||
|
||||
# Force removal without confirmation
|
||||
Remove-MinIOBucket -BucketName "test-bucket" -Force
|
||||
```
|
||||
|
||||
## Object Operations
|
||||
|
||||
### Upload Objects
|
||||
|
||||
```powershell
|
||||
# Upload a single file
|
||||
New-MinIOObject -BucketName "my-data-bucket" `
|
||||
-ObjectName "documents/report.pdf" `
|
||||
-FilePath "C:\Reports\monthly-report.pdf"
|
||||
|
||||
# Upload with custom content type
|
||||
New-MinIOObject -BucketName "web-assets" `
|
||||
-ObjectName "images/logo.png" `
|
||||
-FilePath "C:\Assets\logo.png" `
|
||||
-ContentType "image/png"
|
||||
|
||||
# Upload and return object information
|
||||
New-MinIOObject -BucketName "uploads" `
|
||||
-ObjectName "data.csv" `
|
||||
-FilePath "C:\Data\export.csv" `
|
||||
-PassThru
|
||||
|
||||
# Force overwrite existing object
|
||||
New-MinIOObject -BucketName "backups" `
|
||||
-ObjectName "backup.zip" `
|
||||
-FilePath "C:\Backups\latest.zip" `
|
||||
-Force
|
||||
```
|
||||
|
||||
### List Objects
|
||||
|
||||
```powershell
|
||||
# List all objects in a bucket
|
||||
Get-MinIOObject -BucketName "my-data-bucket"
|
||||
|
||||
# List objects with prefix
|
||||
Get-MinIOObject -BucketName "logs" -Prefix "2025/01/"
|
||||
|
||||
# List specific object
|
||||
Get-MinIOObject -BucketName "documents" -ObjectName "report.pdf"
|
||||
|
||||
# List with filtering and sorting
|
||||
Get-MinIOObject -BucketName "media" `
|
||||
-Prefix "images/" `
|
||||
-ObjectsOnly `
|
||||
-SortBy "Size" `
|
||||
-Descending `
|
||||
-MaxObjects 50
|
||||
```
|
||||
|
||||
### Download Objects
|
||||
|
||||
```powershell
|
||||
# Download a single object
|
||||
Get-MinIOObjectContent -BucketName "documents" `
|
||||
-ObjectName "report.pdf" `
|
||||
-FilePath "C:\Downloads\report.pdf"
|
||||
|
||||
# Download with overwrite
|
||||
Get-MinIOObjectContent -BucketName "backups" `
|
||||
-ObjectName "backup.zip" `
|
||||
-FilePath "C:\Restore\backup.zip" `
|
||||
-Force
|
||||
|
||||
# Download and return file information
|
||||
Get-MinIOObjectContent -BucketName "data" `
|
||||
-ObjectName "export.csv" `
|
||||
-FilePath "C:\Import\data.csv" `
|
||||
-PassThru
|
||||
```
|
||||
|
||||
### Remove Objects
|
||||
|
||||
```powershell
|
||||
# Remove a single object
|
||||
Remove-MinIOObject -BucketName "temp" -ObjectName "old-file.txt"
|
||||
|
||||
# Remove all objects with prefix
|
||||
Remove-MinIOObject -BucketName "logs" `
|
||||
-ObjectName "2024/" `
|
||||
-RemovePrefix
|
||||
|
||||
# Force removal without confirmation
|
||||
Remove-MinIOObject -BucketName "cache" `
|
||||
-ObjectName "temp-data.json" `
|
||||
-Force
|
||||
```
|
||||
|
||||
## Security and Policies
|
||||
|
||||
### Get Bucket Policies
|
||||
|
||||
```powershell
|
||||
# Get policy as JSON
|
||||
Get-MinIOBucketPolicy -BucketName "public-bucket"
|
||||
|
||||
# Get policy as structured object
|
||||
Get-MinIOBucketPolicy -BucketName "public-bucket" -AsObject
|
||||
|
||||
# Get pretty-printed JSON
|
||||
Get-MinIOBucketPolicy -BucketName "public-bucket" -PrettyPrint
|
||||
```
|
||||
|
||||
### Set Bucket Policies
|
||||
|
||||
```powershell
|
||||
# Set policy from JSON string
|
||||
$policy = @"
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Principal": {"AWS": "*"},
|
||||
"Action": ["s3:GetObject"],
|
||||
"Resource": ["arn:aws:s3:::public-bucket/*"]
|
||||
}
|
||||
]
|
||||
}
|
||||
"@
|
||||
|
||||
Set-MinIOBucketPolicy -BucketName "public-bucket" -PolicyJson $policy
|
||||
|
||||
# Set policy from file
|
||||
Set-MinIOBucketPolicy -BucketName "secure-bucket" `
|
||||
-PolicyFilePath "C:\Policies\secure-policy.json"
|
||||
|
||||
# Use canned policies
|
||||
Set-MinIOBucketPolicy -BucketName "readonly-bucket" `
|
||||
-CannedPolicy "ReadOnly"
|
||||
|
||||
Set-MinIOBucketPolicy -BucketName "upload-bucket" `
|
||||
-CannedPolicy "WriteOnly" `
|
||||
-Prefix "uploads/*"
|
||||
|
||||
# Validate policy without setting
|
||||
Set-MinIOBucketPolicy -BucketName "test-bucket" `
|
||||
-PolicyJson $policy `
|
||||
-ValidateOnly
|
||||
```
|
||||
|
||||
## Statistics and Monitoring
|
||||
|
||||
### Basic Statistics
|
||||
|
||||
```powershell
|
||||
# Get basic statistics
|
||||
Get-MinIOStats
|
||||
|
||||
# Include object counts (may be slow)
|
||||
Get-MinIOStats -IncludeObjectCounts
|
||||
|
||||
# Detailed statistics with per-bucket information
|
||||
Get-MinIOStats -IncludeBucketDetails -IncludeObjectCounts
|
||||
|
||||
# Limit object counting for performance
|
||||
Get-MinIOStats -IncludeObjectCounts -MaxObjectsToCount 1000
|
||||
```
|
||||
|
||||
## Advanced Usage
|
||||
|
||||
### Batch Operations
|
||||
|
||||
```powershell
|
||||
# Upload multiple files
|
||||
$files = Get-ChildItem "C:\Data\*.csv"
|
||||
foreach ($file in $files) {
|
||||
New-MinIOObject -BucketName "data-lake" `
|
||||
-ObjectName "csv-files/$($file.Name)" `
|
||||
-FilePath $file.FullName `
|
||||
-Verbose
|
||||
}
|
||||
|
||||
# Download all objects with specific prefix
|
||||
$objects = Get-MinIOObject -BucketName "backups" -Prefix "2025/01/"
|
||||
foreach ($obj in $objects) {
|
||||
$localPath = "C:\Restore\$($obj.Name)"
|
||||
$localDir = Split-Path $localPath -Parent
|
||||
if (!(Test-Path $localDir)) { New-Item -ItemType Directory -Path $localDir -Force }
|
||||
|
||||
Get-MinIOObjectContent -BucketName "backups" `
|
||||
-ObjectName $obj.Name `
|
||||
-FilePath $localPath `
|
||||
-Verbose
|
||||
}
|
||||
```
|
||||
|
||||
### Pipeline Usage
|
||||
|
||||
```powershell
|
||||
# Pipeline bucket operations
|
||||
Get-MinIOBucket | Where-Object { $_.Name -like "temp-*" } | Remove-MinIOBucket -Force
|
||||
|
||||
# Pipeline object operations
|
||||
Get-MinIOObject -BucketName "logs" -Prefix "old/" |
|
||||
ForEach-Object { Remove-MinIOObject -BucketName $_.BucketName -ObjectName $_.Name -Force }
|
||||
```
|
||||
|
||||
### Error Handling
|
||||
|
||||
```powershell
|
||||
try {
|
||||
New-MinIOBucket -BucketName "test-bucket" -ErrorAction Stop
|
||||
Write-Host "Bucket created successfully"
|
||||
} catch {
|
||||
Write-Error "Failed to create bucket: $($_.Exception.Message)"
|
||||
}
|
||||
|
||||
# Using -WhatIf for testing
|
||||
New-MinIOObject -BucketName "test" `
|
||||
-ObjectName "test.txt" `
|
||||
-FilePath "C:\test.txt" `
|
||||
-WhatIf
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Configuration Problems**
|
||||
```powershell
|
||||
# Test configuration
|
||||
Get-MinIOConfig -TestConnection
|
||||
|
||||
# Reset configuration
|
||||
Set-MinIOConfig -Endpoint "localhost:9000" `
|
||||
-AccessKey "minioadmin" `
|
||||
-SecretKey "minioadmin" `
|
||||
-NoSSL `
|
||||
-TestConnection
|
||||
```
|
||||
|
||||
2. **Connection Issues**
|
||||
```powershell
|
||||
# Check endpoint accessibility
|
||||
Test-NetConnection -ComputerName "minio.example.com" -Port 9000
|
||||
|
||||
# Verify SSL settings
|
||||
Get-MinIOConfig -Detailed
|
||||
```
|
||||
|
||||
3. **Permission Issues**
|
||||
```powershell
|
||||
# Check bucket policy
|
||||
Get-MinIOBucketPolicy -BucketName "problem-bucket" -AsObject
|
||||
|
||||
# Test with different credentials
|
||||
Set-MinIOConfig -AccessKey "admin" -SecretKey "admin-password"
|
||||
```
|
||||
|
||||
### Verbose Logging
|
||||
|
||||
```powershell
|
||||
# Enable verbose output for troubleshooting
|
||||
Get-MinIOBucket -Verbose
|
||||
New-MinIOObject -BucketName "test" -ObjectName "test.txt" -FilePath "C:\test.txt" -Verbose
|
||||
```
|
||||
|
||||
### Performance Tips
|
||||
|
||||
1. **Use appropriate batch sizes for large operations**
|
||||
2. **Limit object counting with `-MaxObjectsToCount` for large buckets**
|
||||
3. **Use `-Force` parameter to avoid confirmation prompts in scripts**
|
||||
4. **Test operations with `-WhatIf` before execution**
|
||||
|
||||
For more information, see the [API Reference](API.md) and [Examples](EXAMPLES.md).
|
||||
Reference in New Issue
Block a user