Adds TestMemoryStatusEffectiveAvailable_RegressionIssue435 with 5 test scenarios
covering all memory calculation edge cases reported in GitHub issue #435:
1. Proxmox 8.x with 'available' field (most common)
2. Older Proxmox with 'avail' field
3. Derived calculation from free+buffers+cached
4. Real user case: 86% displayed when actual usage was 42%
5. Missing all cache fields (fallback behavior)
Tests verify EffectiveAvailable() correctly returns cache-aware memory values
and calculates realistic usage percentages that exclude reclaimable cache.
These tests ensure future changes don't regress the cache-aware behavior
that was fixed in v4.15.0-rc.3.
Add comprehensive PMG monitoring with mail statistics, queue depth tracking,
spam distribution analysis, and quarantine monitoring. Includes full discovery
support and UI consistency improvements across all Proxmox products.
Backend:
- Add pkg/pmg package with complete API client for PMG operations
- Implement mail statistics collection (inbound/outbound, spam, virus, bounces)
- Add queue depth monitoring (active, deferred, hold, incoming queues)
- Support spam score distribution and quarantine totals
- Add PMG-specific discovery logic to differentiate from PVE on port 8006
- Extend mock data generator with realistic PMG instances and metrics
- Add PMG node configuration support in config system
Frontend:
- Create MailGateway.tsx component with detailed PMG dashboard
- Display mail flow statistics with time-series charts
- Show queue depth with color-coded warnings (>50 messages or >30min age)
- Add spam distribution histogram and quarantine status
- Support cluster node status with individual queue monitoring
- Add PMG to network discovery with purple branding and mail icon
- Implement conditional navigation (hide PMG tab when no instances configured)
- Standardize discovery UI controls across PVE/PBS/PMG settings pages
API:
- Add /api/config/pmg endpoints for node configuration
- Support PMG-specific monitoring toggles (mail stats, queues, quarantine)
- Extend system settings with PMG configuration options
Discovery:
- Detect PMG vs PVE on shared port 8006 using /api2/json/statistics/mail endpoint
- Return 'pmg' type for mail gateway servers in discovery results
- Update DiscoveryModal to display PMG servers with appropriate styling
This completes ecosystem monitoring support for all three Proxmox products:
Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
Implements comprehensive Docker monitoring with a dedicated agent that collects
container metrics and reports them to the main Pulse server. Adds Docker-specific
alert rules and threshold management with a redesigned UI.
Backend changes:
- Add Docker agent binary with container metrics collection
- Implement Docker host and container models with CPU/memory tracking
- Add Docker-specific alert types (offline, state, health)
- Extend threshold system to support Docker resources
- Add WebSocket message types for Docker agent communication
- Implement Docker agent API endpoints for registration and metrics
Frontend changes:
- Add Docker monitoring page with host/container views
- Add Docker agent settings panel for configuration
- Reorganize thresholds page with Proxmox/Docker tabs
- Add Docker-specific alert threshold management
- Improve layout consistency with vertical stacking
- Fix defensive null checks and TypeScript errors
This change enables monitoring of Docker containers across multiple hosts
with the same alerting and threshold capabilities as Proxmox resources.
This commit addresses all issues reported in GitHub issue #485:
1. **SMART Status Recognition**
- Fix disk health check to accept both "PASSED" and "OK" status
- Previously only "PASSED" was recognized as healthy
- Location: internal/monitoring/monitor.go:1255
2. **ZFS Spare Device False Alerts**
- Skip ZFS SPARE devices unless they have actual errors
- SPARE devices are intentional and should not trigger alerts
- Updated in two locations:
- pkg/proxmox/zfs.go:154 (device filtering)
- internal/alerts/alerts.go:1077 (alert generation)
3. **Memory Display Granularity**
- Increase byte formatting precision from 0 to 1 decimal place
- Improves accuracy (e.g., "1.7 GB" instead of "1 GB" for 86% of 2GB)
- Location: frontend-modern/src/utils/format.ts:3
4. **Custom Alert Rules Evaluation**
- Add ReevaluateGuestAlert() method for proper threshold reevaluation
- Add comments explaining custom rules evaluation limitations
- Next poll cycle will properly clear stale alerts with new thresholds
Additional improvements:
- Fix ZFS pool alert locking to prevent deadlocks
- Prevent discovery service from running in mock mode
- Restore discovery service when exiting mock mode
Fixes#485
Fixes#483 - PBS syslog was being flooded with 400 Bad Request errors
because the cf parameter value 'average' is not in the valid enumeration.
Changed to 'AVERAGE' (all caps) per PBS API specification.
addresses #449
proxmox returns 'N/A' or empty string for the wearout field on disks that
don't support wear reporting (HDDs, hardware RAID controllers, etc). pulse
was expecting an integer, causing JSON unmarshal errors that prevented ALL
disks from being displayed on affected nodes.
added custom UnmarshalJSON method for the Disk type to gracefully handle:
- numeric values (SSDs with wear reporting)
- string values like 'N/A' (HDDs, RAID controllers) - converts to 0
- null values - converts to 0
this allows nodes with mixed disk types (SSDs, HDDs, RAID) to display all
their disks correctly. wearout value of 0 indicates no wear reporting
available, which is expected for HDDs.
- Added streaming discovery that shows servers as they're found
- Backend sends WebSocket updates for each discovered server
- Frontend displays servers immediately without waiting for full scan
- Created sync-production-config.sh to preserve nodes when switching modes
- Updated toggle-mock.sh to sync config when disabling mock mode
- Dev environment now maintains separate config that syncs from production
- Enabled discovery service in dev environment by default
addresses real-time discovery UX and mock/production mode configuration persistence
The 15-second timeout introduced to handle unavailable NFS storage was too aggressive and caused legitimate storage queries to timeout on nodes with many storage backends or higher latency. This was causing storage to not be displayed for affected nodes.
Increased timeout to 30 seconds as a better balance between responsiveness and reliability.
- Add debug logging to guest agent filesystem API responses
- Better handle Windows drive mountpoints (C:\, D:\, etc.)
- Improve empty filesystem list detection and logging
- Add specific handling for Windows filesystems that may report differently
This should help diagnose why some VMs with guest agents installed still show 0% or missing disk usage, particularly on Windows systems.
- handle PBS node status endpoint permission errors gracefully (returns nil instead of error for 403s)
- add required cf and timeframe parameters to RRD endpoint calls
- properly handle nil nodeStatus returns in monitor.go
these API calls now fail silently as PBS API tokens often lack the required permissions for these endpoints, which is expected behavior
The cluster client was incorrectly marking nodes as unhealthy when encountering
VM-specific QEMU guest agent errors. This caused storage and backup operations
to fail with "no healthy nodes available" even though the nodes were actually
accessible.
Changes:
- Added broader detection for guest agent errors in executeWithFailover
- Updated recovery logic to ignore VM-specific errors when recovering nodes
- Guest agent errors no longer affect node health status
This fixes the issue where users with clusters would see storage and backup
operations fail after any VM without a guest agent was queried.
- Add Available field to MemoryStatus struct to capture memory available for allocation
- Update node memory calculation to use Available memory when present
- This excludes non-reclaimable cache/buffers from used memory calculation
- Provides more accurate memory pressure indication, avoiding false alerts
- Falls back to traditional used memory if Available field is missing (older Proxmox versions)
- Added disk polling to monitoring cycle using Proxmox API
- Created CheckDiskHealth() alert manager for failing drives and low SSD life
- Added PhysicalDisk model to state with proper serialization
- Implemented DiskList component with health indicators and SSD wearout bars
- Added Physical Disks tab to Storage page with toggle between pools and disks
- Added ZFS health badges to storage cards for degraded/failed pools
- Alerts trigger for health != PASSED and SSD wearout < 10%
- Frontend displays disk model, type, temperature, and usage information
- Always query guest agent for running VMs (cluster/resources API always returns 0)
- Show allocated disk size when guest agent unavailable (instead of misleading 0%)
- Fix duplicate mount point counting issue (#425)
- Add comprehensive logging for guest agent queries
- Include diagnostic script for troubleshooting VM disk issues
- Update both monitor.go and monitor_optimized.go for consistency
- Implement proper API integration with list and detail endpoints
- Add ZFS pool and device status conversion
- Enable by default with PULSE_DISABLE_ZFS_MONITORING opt-out
- Test with real Proxmox nodes and verify functionality
- Add comprehensive error handling and logging
- Document feature configuration and requirements
The feature now properly:
- Fetches ZFS pool status from Proxmox API
- Detects degraded/faulted pools and devices
- Tracks read/write/checksum errors
- Generates appropriate alerts
- Displays issues in the Storage tab UI
Tested and verified working with real Proxmox clusters.
- Added debug mode: localStorage.setItem('debug-pmg', 'true')
- Robust VMID=0 detection handles string and number types
- Debug logging shows exactly what's happening with PMG backups
- Created test suite that verifies all PMG backup scenarios
- All test cases pass including PBS 'ct' type with VMID='0'
Users experiencing issues can enable debug mode to help diagnose:
1. Open browser console
2. Run: localStorage.setItem('debug-pmg', 'true')
3. Reload page and check for [PMG Debug] messages
4. Share debug output if still showing as LXC
Test results:
✓ PBS PMG backup (ct type with VMID 0) → Host
✓ PBS PMG backup (ct type with numeric VMID 0) → Host
✓ Storage PMG backup (host type) → Host
✓ Storage PMG backup (lxc type with VMID 0) → Host
✓ Regular LXC backup → LXC
- Handle VMID as both string and number types consistently
- Check for both 'ct' and 'lxc' backup types (PBS uses 'ct')
- Check for both 'vm' and 'qemu' backup types for consistency
- Always check VMID=0 first before checking backup type
- PBS stores PMG backups as 'ct' type with VMID='0' (string)
This should properly identify all PMG host config backups regardless
of whether they come from PBS or regular storage, and regardless
of whether VMID is a string or number.
- Add PULSE_ENABLE_ZFS_MONITORING env var (disabled by default)
- Fix API field mapping (health vs state, cksum vs checksum)
- Add proper API endpoint structures for list and detail
- Mark feature as experimental due to API complexity
- Simplify conversion to handle basic health status only
This is a safer approach until we can fully test with real Proxmox nodes
- Add ZFS pool status data structures to models
- Implement ZFS pool data collection via Proxmox API
- Add ZFS pool health alerts for degraded/faulted states
- Add ZFS device error detection and alerting
- Display ZFS pool status in Storage tab when issues detected
- Add mock data generation for testing ZFS monitoring
- Alert on read/write/checksum errors for pools and devices
The real issue was not the overall timeout duration, but that DNS resolution and TLS handshake could hang indefinitely. Added specific timeouts for:
- DNS resolution/connection: 10 seconds
- TLS handshake: 10 seconds
- Response headers: 10 seconds
This prevents the connection from hanging on DNS lookup (like with pve-backup.lan) or during TLS negotiation, which was causing the 'context deadline exceeded' errors. (addresses #424)
- Made cluster health checks less aggressive to prevent false unhealthy states
- Fixed JSON unmarshal error when Proxmox returns object instead of array for VMFileSystem
- Increased initial health check timeouts from 2s to 5s for better reliability
- Added handling for JSON unmarshal errors as data format issues, not connectivity problems
- Improved recovery check interval from 5s to 10s to reduce excessive health checks
- Changed log levels from WARN to DEBUG for transient connectivity issues
- Reduced storage API timeout from 120s to 15s to prevent blocking when storage mounts are unavailable
- Added graceful error handling for storage timeouts - continues with partial data instead of failing
- Improved error messages to clarify when timeouts are likely due to unavailable storage (e.g., NFS mounts)
This prevents Pulse from marking nodes as unhealthy when storage endpoints timeout due to temporarily unavailable network storage.
- Use main host for cluster operations when node endpoints lack FQDNs/IPs
- Skip initial health check for single-endpoint clusters (main host routing)
- Return empty lists instead of errors when cluster nodes are unreachable
- Prevent VMs/containers from disappearing when cluster has connectivity issues
- Fix the 'Instance marked as cluster but is actually standalone' false warning
- Fixed issue where QEMU guest agent errors incorrectly marked nodes as unhealthy
- Nodes with VMs missing guest agents no longer affect cluster health status
- Reduced health check retry interval from 30s to 5s for faster recovery
- Storage and backup polling now works correctly even when some VMs lack guest agents
When a VM doesn't have QEMU guest agent configured, Proxmox returns a 500 error.
This was incorrectly marking the entire cluster node as unhealthy, preventing
all operations on that node. Now we treat these as VM-specific errors that
don't affect node health status.
- Storage queries can timeout on large clusters or slow storage backends
- Extended timeout specifically for GetStorage, GetStorageContent, and GetAllStorage
- Preserves existing context deadlines if they're shorter than 120s
- Should resolve 'context deadline exceeded' errors during storage polling
- Added storage permission errors (403) to exception list
- Permission denied errors no longer mark nodes as unhealthy
- Storage polling can now continue even with permission issues
- Prevents cascading failures when storage permissions are missing
- Nodes remain healthy for VM/container operations even if storage fails
- Changed cluster client initialization to be optimistic (assume healthy)
- Nodes now start as healthy and are marked unhealthy only on actual failures
- This prevents the issue where all nodes were marked unhealthy during init
- Storage operations can now proceed even if initial health checks fail
- Allows recovery from temporary network or auth issues during startup
- Increased default HTTP client timeout from 30s to 60s
- Added CreateHTTPClientWithTimeout function to properly set custom timeouts
- Updated Proxmox and PBS clients to use configured timeout values
- Increased default connection timeout from 45s to 60s in config
This prevents "context deadline exceeded" errors when connecting to slow or overloaded Proxmox/PBS nodes.
addresses #379 - better handling of offline nodes in clusters
- Skip polling VMs/containers from offline nodes to avoid 595 errors
- Improved error message for 595 to distinguish between auth failures and offline node access
addresses #389 - improved error messaging
- Better detection of whether 595 is an auth issue or offline node issue
- Clearer error messages to help users diagnose the actual problem
The 595 error can occur when:
1. Authentication actually fails (wrong credentials)
2. Trying to access resources on an offline node through another node in the cluster
addresses #388 - LXC containers not showing due to VMID type mismatch
- Changed Container.VMID from int to FlexInt to handle string VMIDs from older Proxmox versions
- Updated all code that references Container.VMID to cast to int where needed
addresses #389 - connection timeout errors with Proxmox nodes
- Increased default CONNECTION_TIMEOUT from 10s to 30s to handle slower networks
- This should resolve "context deadline exceeded" errors when polling nodes
addresses #379 - authentication errors may have been related to timeouts
Addresses #379 - Added clearer error messages for common authentication issues:
- 403 errors now explain that permissions must be set on the USER (not just the token) in Proxmox GUI
- 595 errors indicate authentication failure
- 401 errors indicate invalid credentials
While the setup script handles this correctly, these messages help users who manually configure permissions.
- Add expandable namespace rows to PBS instances table
- Show deduplication factor from PBS GC status (calculated from index-data-bytes/disk-bytes)
- Move deduplication display to bottom left of backup frequency chart
- Add namespace highlighting when filtered (blue background, filtering indicator)
- Fix backup frequency chart to properly handle PBS namespace filters
- Allow clicking namespace again to clear filter (toggle behavior)
- Improve visual feedback for selected namespaces with color changes
rcourtman