- Add debug logging to guest agent filesystem API responses
- Better handle Windows drive mountpoints (C:\, D:\, etc.)
- Improve empty filesystem list detection and logging
- Add specific handling for Windows filesystems that may report differently
This should help diagnose why some VMs with guest agents installed still show 0% or missing disk usage, particularly on Windows systems.
- Add comprehensive disk detection diagnostics to /api/diagnostics
- Shows which nodes return disks and which don't
- Provides specific error messages and API responses
- Includes targeted recommendations based on failure type
- Helps users provide better debugging info when reporting issues
- Add specific warning for permission errors (401/403)
- Add info message for nodes without disk monitoring support
- Elevate error logging from debug to warn level for visibility
- Helps users understand why disks might not appear for certain nodes
- Increase WebSocket buffer sizes from 64KB to 4MB to handle large mock data
- Add robust reconnection logic with exponential backoff
- Implement heartbeat mechanism to detect stale connections faster
- Add manual reconnect button in UI when connection fails
- Fix unused variable warnings in monitor code
- Add debug logging to trace WebSocket state initialization
This resolves the issue where the frontend would hang after code changes
during hot-reload, especially when using mock mode with many nodes.
- Storage polling now preserves existing data when nodes are temporarily offline
- Backup polling preserves existing backups for unpolled nodes
- Prevents storage and backups from disappearing when nodes have connectivity issues
- Similar to the physical disk preservation fix
When a node is temporarily offline or has connectivity issues, preserve its last known physical disk data instead of removing it from the table. This prevents nodes from disappearing from the Physical Disks table when they have transient connection problems.
- VMs/Containers default to 10 seconds
- Nodes default to 15 seconds
- Storage defaults to 30 seconds
- PBS servers default to 30 seconds
This allows more appropriate delays for different resource types instead of a single global delay that doesn't fit all use cases. Storage and PBS can have longer delays since they're less critical and more prone to transient spikes during operations.
The 100% threshold disabling feature was incorrectly implemented and doesn't
make logical sense - metrics can legitimately reach 100% (CPU, memory, storage)
and those are critical conditions that should trigger alerts.
The correct way to disable specific alerts is already implemented:
- Set threshold to 0 or negative to disable a metric type globally
- Use per-resource overrides to disable specific metrics for specific resources
- Example: Overrides[guest-id].Memory = {Trigger: 0} disables memory alerts for that guest
This removes the confusing behavior where 100% thresholds would disable alerts
instead of alerting on actual 100% usage conditions.
Two critical issues fixed:
1. Storage threshold overrides were being loaded but never applied - the code
always used the default threshold instead of checking for overrides
2. Setting a threshold to 100% now properly disables alerts for that metric,
allowing users to suppress specific alerts they don't want
This fixes both the storage threshold persistence issue and the inability
to disable alerts by setting thresholds to 100%.
The issue was that PBS monitoring uses name-based IDs (pbs-<name>) while
the config system uses index-based IDs (pbs-0, pbs-1). When updating PBS
node configuration, the alert overrides were already being preserved but
the ID mismatch wasn't properly documented. Added explicit logging to
track PBS override preservation using the correct monitoring ID.
When detecting Proxmox cluster nodes, the Host field was being set to just the node name without a port. This caused validation to fail with "invalid Port number" error when qdevices were running.
Now cluster endpoints properly include the port (8006) in the Host field, allowing clusters with qdevices to be added successfully.
- Limit alert checking to 50 guests per cycle to prevent blocking
- Remove unnecessary state broadcast when alerts are resolved
- Fix deadlock in GetActiveAlerts by releasing lock quickly
- Enable handling of 800+ mock guests with sub-10ms response times
This allows Pulse to handle large-scale deployments efficiently for testing and production use.
The alert acknowledgment endpoints were hanging because GetState() was called
synchronously to broadcast updates via WebSocket, which could take significant
time with many nodes/guests. This caused the HTTP response to timeout, showing
an error to users even though the alert was successfully acknowledged.
Fixed by:
- Sending HTTP response immediately after acknowledging the alert
- Moving WebSocket broadcast to a goroutine to avoid blocking
- Applied fix to all alert endpoints (acknowledge, unacknowledge, clear, bulk ops)
This resolves the issue where users saw 'Failed to acknowledge alert' errors
but the alert was actually acknowledged (disappeared on refresh).
When updating PBS nodes through the node configuration UI, alert thresholds
were being reset to defaults. This was because alert overrides are stored
separately from node configuration and weren't being preserved during node updates.
The fix ensures that when a node is updated, the alert configuration (including
any custom threshold overrides) is reloaded and preserved. This applies to both
PBS and PVE nodes to ensure consistent behavior.
- handle PBS node status endpoint permission errors gracefully (returns nil instead of error for 403s)
- add required cf and timeframe parameters to RRD endpoint calls
- properly handle nil nodeStatus returns in monitor.go
these API calls now fail silently as PBS API tokens often lack the required permissions for these endpoints, which is expected behavior
- Add retry logic with delays to detectPVECluster function to handle API permission propagation
- Periodically re-check standalone nodes to detect if they're actually part of a cluster
- Increase timeout from 3 to 5 seconds for cluster detection attempts
- Skip retries for definitively standalone nodes (501 not implemented errors)
This addresses the issue where adding a PVE cluster doesn't detect it properly on first attempt,
requiring deletion and re-adding to work correctly. The retry mechanism gives time for
API permissions to fully propagate in Proxmox.
- Use centralized detectServiceName() function instead of duplicate logic
- Automatically detect whether system uses 'pulse' or 'pulse-backend' service
- Improves compatibility between official and community installer scripts
- Reduces confusion when users mix installation methods
- Calculate memory as (Total - Available) instead of raw Used value
- Excludes buffer/cache memory that Linux can reclaim when needed
- Prevents false alerts from Linux cache usage
- Falls back to traditional calculation on older Proxmox versions
- VMs already use FreeMem from guest agent when available
- Memory usage will appear lower but more accurate (e.g., 56% instead of 84%)
- Users may need to adjust alert thresholds accordingly
- Add Available field to MemoryStatus struct to capture memory available for allocation
- Update node memory calculation to use Available memory when present
- This excludes non-reclaimable cache/buffers from used memory calculation
- Provides more accurate memory pressure indication, avoiding false alerts
- Falls back to traditional used memory if Available field is missing (older Proxmox versions)
When a threshold is set to 100%, it now effectively disables alerts for that metric.
This allows users to turn off specific alerts without disabling all alerts for a resource.
Also clears any existing alerts when threshold is changed to 100%.
- Changed logic to always query guest agent when available, not just when disk is 0
- This fixes issue where Proxmox returns incorrect non-zero values from cluster/resources
- Guest agent data is now preferred over cluster/resources data for all running VMs
- Improved logging to show when we're replacing cluster data with guest agent data
This should resolve the issue reported by FaboulousSan where VMs were showing
host disk space instead of actual VM disk usage.
- Add comprehensive filtering for network filesystems (NFS, CIFS, SMB, FUSE, 9p)
- Skip Docker volumes, snap mounts, and other special mountpoints
- Add detailed logging to track which filesystems are included/excluded
- Add sanity check to detect when reported disk is way larger than allocated
- Improve logging with GB values and more context for debugging
This should prevent Pulse from accidentally including host disk space or
network shares when calculating VM disk usage. Users can use the existing
diagnostics system in the UI to troubleshoot VM disk issues.
The disk monitoring backend was working but frontend wasn't updating because the WebSocket store was missing the handler for physicalDisks data. Also added physicalDisks count to broadcast logging for better debugging.
- Added disk polling to monitoring cycle using Proxmox API
- Created CheckDiskHealth() alert manager for failing drives and low SSD life
- Added PhysicalDisk model to state with proper serialization
- Implemented DiskList component with health indicators and SSD wearout bars
- Added Physical Disks tab to Storage page with toggle between pools and disks
- Added ZFS health badges to storage cards for degraded/failed pools
- Alerts trigger for health != PASSED and SSD wearout < 10%
- Frontend displays disk model, type, temperature, and usage information
- Add API validation for cluster nodes to filter out qdevice VMs
- Only include nodes with working Proxmox APIs in cluster endpoints
- Prevent connection failures when cluster has non-Proxmox participants
- Add detailed logging for cluster node validation process
This resolves issues where Proxmox clusters using corosync qdevice
(external quorum device) would fail to connect because Pulse tried
to connect to the qdevice VM which has no Proxmox API.
- Add client-side URL validation with instant feedback
- Show validation errors inline below URL input fields
- Prevent saving when URLs have validation errors
- Improve error message extraction in API client
- Handle incomplete URLs like 'https://emby.' gracefully
- Backend already had validation, now frontend shows it properly
- Add more specific error messages when metadata save fails
- Better handling of permission and disk space errors
- This should help diagnose why guest URLs fail to save in some cases
- The atomic write operation was already in place but errors weren't clear
- Fixed PBS alert toggle not responding in thresholds settings
- PBS servers now use connectivity toggle like nodes instead of disabled toggle
- Added support for disableConnectivity flag on PBS instances in backend
- Fixed PBS ID format mismatch between frontend and backend
- PBS offline alerts now properly respect the disableConnectivity setting
- Prevents spam alerts by checking disableConnectivity flag for PBS offline alerts
- Always query guest agent for running VMs instead of only when disk is 0
- Add duplicate mount point detection to prevent inflated disk totals
- Show allocated disk size as fallback when guest agent unavailable
- Add comprehensive logging for guest agent disk queries
- Include diagnostic script for troubleshooting VM disk issues
- Always query guest agent for running VMs (cluster/resources API always returns 0)
- Show allocated disk size when guest agent unavailable (instead of misleading 0%)
- Fix duplicate mount point counting issue (#425)
- Add comprehensive logging for guest agent queries
- Include diagnostic script for troubleshooting VM disk issues
- Update both monitor.go and monitor_optimized.go for consistency
- Prevent path traversal attacks by cleaning and validating URL paths
- Use secure token comparison to prevent timing attacks
- Return appropriate HTTP status codes for different attack vectors
- Add comprehensive logging for security events
The parallel optimization introduced in commit 634e0dd37 accidentally removed
all guest agent filesystem fetching logic from the optimized monitor code.
This caused VMs with guest agents to show no disk stats after v4.12.1.
Added back the guest agent fetching logic to pollVMsWithNodesOptimized:
- Fetches filesystem info when VM disk stats are 0
- Aggregates disk usage from all valid filesystems
- Skips special filesystems and Windows System Reserved partitions
- Uses guest agent data when available to show accurate disk usage
This restores disk stats display for VMs with working QEMU guest agents.
- Added debug mode: localStorage.setItem('debug-pmg', 'true')
- Robust VMID=0 detection handles string and number types
- Debug logging shows exactly what's happening with PMG backups
- Created test suite that verifies all PMG backup scenarios
- All test cases pass including PBS 'ct' type with VMID='0'
Users experiencing issues can enable debug mode to help diagnose:
1. Open browser console
2. Run: localStorage.setItem('debug-pmg', 'true')
3. Reload page and check for [PMG Debug] messages
4. Share debug output if still showing as LXC
Test results:
✓ PBS PMG backup (ct type with VMID 0) → Host
✓ PBS PMG backup (ct type with numeric VMID 0) → Host
✓ Storage PMG backup (host type) → Host
✓ Storage PMG backup (lxc type with VMID 0) → Host
✓ Regular LXC backup → LXC
- Add PULSE_ENABLE_ZFS_MONITORING env var (disabled by default)
- Fix API field mapping (health vs state, cksum vs checksum)
- Add proper API endpoint structures for list and detail
- Mark feature as experimental due to API complexity
- Simplify conversion to handle basic health status only
This is a safer approach until we can fully test with real Proxmox nodes
- Add ZFS pool status data structures to models
- Implement ZFS pool data collection via Proxmox API
- Add ZFS pool health alerts for degraded/faulted states
- Add ZFS device error detection and alerting
- Display ZFS pool status in Storage tab when issues detected
- Add mock data generation for testing ZFS monitoring
- Alert on read/write/checksum errors for pools and devices
The real issue was not the overall timeout duration, but that DNS resolution and TLS handshake could hang indefinitely. Added specific timeouts for:
- DNS resolution/connection: 10 seconds
- TLS handshake: 10 seconds
- Response headers: 10 seconds
This prevents the connection from hanging on DNS lookup (like with pve-backup.lan) or during TLS negotiation, which was causing the 'context deadline exceeded' errors. (addresses #424)
PBS servers can be slow to respond, especially under load or over slower connections. Increased the timeout from 10 seconds to 30 seconds specifically for PBS version checks during diagnostics.
- Reduced storage API timeout from 120s to 15s to prevent blocking when storage mounts are unavailable
- Added graceful error handling for storage timeouts - continues with partial data instead of failing
- Improved error messages to clarify when timeouts are likely due to unavailable storage (e.g., NFS mounts)
This prevents Pulse from marking nodes as unhealthy when storage endpoints timeout due to temporarily unavailable network storage.
- Use main host for cluster operations when node endpoints lack FQDNs/IPs
- Skip initial health check for single-endpoint clusters (main host routing)
- Return empty lists instead of errors when cluster nodes are unreachable
- Prevent VMs/containers from disappearing when cluster has connectivity issues
- Fix the 'Instance marked as cluster but is actually standalone' false warning
The /api/state and /api/guests/metadata endpoints are now excluded from
rate limiting as they are polled frequently by the UI for real-time updates.
This prevents the "Loading..." issue when users with multiple nodes access
the application.
- Added skip list in UniversalRateLimitMiddleware for real-time endpoints
- Removed duplicate rate limiting logic from router's ServeHTTP
- Consolidated all rate limiting into the universal middleware
Pulse Monitor