rustguac/patches

guacamole-server patches

These patches are applied to the apache/guacamole-server source tree before building guacd. They fix compilation and runtime issues when building against FreeRDP 3.x on Debian 13 (Trixie).

001-freerdp3-debian13.patch

Problem: guacamole-server 1.6.1 fails to compile against FreeRDP 3.15+ (as shipped in Debian 13) due to:

1. Autoconf feature detection failure — FreeRDP 3.15 marks codecs_free() as deprecated. The -Werror flag in configure.ac causes all compile-time feature-detection tests to fail, cascading into 10+ undefined macros and wrong #ifdef code paths.

2. Deprecated function pointer API — FreeRDP 3.x deprecates direct struct member access for ->input->KeyboardEvent(), ->input->MouseEvent(), etc. behind WITH_FREERDP_DEPRECATED. The safe replacement functions are freerdp_input_send_keyboard_event(), freerdp_input_send_mouse_event(), etc.

3. NULL pointer dereference in display channel — FreeRDP 3.x fires PubSub ChannelConnected events before guac_rdp_disp is allocated, causing a segfault when the callback writes to disp->requested_width (offset 0x18 of NULL).

Files patched:

File	Fix
configure.ac	Add -Wno-error=deprecated-declarations to both FreeRDP 2.x and 3.x PKG_CHECK_MODULES blocks so autoconf feature detection works
src/protocols/rdp/Makefile.am	Add -Wno-error=deprecated-declarations to all three CFLAGS targets
src/protocols/rdp/tests/Makefile.am	Same for test CFLAGS
src/protocols/rdp/keyboard.c	Replace ->input->KeyboardEvent(), ->input->UnicodeKeyboardEvent(), ->input->SynchronizeEvent() with safe API functions
src/protocols/rdp/input-queue.c	Replace all ->input->MouseEvent() calls with freerdp_input_send_mouse_event()
src/protocols/rdp/channels/disp.c	Add NULL guards in guac_rdp_disp_channel_connected() and guac_rdp_disp_channel_disconnected()

002-kerberos-nla.patch

Feature: Adds Kerberos NLA authentication support to guacd's RDP protocol, based on GUACAMOLE-2057 (PR #581). This allows RDP connections to use Kerberos instead of NTLM for NLA, which is required as Microsoft phases out NTLM.

Three new connection parameters:

Parameter	Values	FreeRDP3 Setting
auth-pkg	"" (negotiate), "kerberos", "ntlm"	FreeRDP_AuthenticationPackageList
kdc-url	KDC server URL (optional)	FreeRDP_KerberosKdcUrl
kerberos-cache	Path to ccache file (optional)	FreeRDP_KerberosCache

Files patched:

File	Fix
src/protocols/rdp/settings.h	Add guac_rdp_auth_package enum, add auth_pkg, kdc_url, kerberos_cache fields to guac_rdp_settings
src/protocols/rdp/settings.c	Add connection parameter parsing, FreeRDP3 settings push, memory cleanup

Differences from upstream PR #581:

• Dropped FreeRDP2 code path (not needed on Debian 13)
• Fixed guac_strdup() leak in freerdp_settings_set_string() calls (FreeRDP3 copies internally)
• Fixed typos ("NTML" -> "NTLM", "negotiatoin" -> "negotiation")

Requires: FreeRDP 3.x built with Kerberos support (-DWITH_KRB5=ON). Debian 13's freerdp3-dev includes this by default.

003-null-guard-and-config-h.patch

Problem: Two related issues causing RDP display resize to silently fail:

1. Missing config.h include — Several RDP channel source files and input.c do not include config.h, so ENABLE_COMMON_SSH is undefined in those compilation units. This causes the guac_rdp_client struct to have a different layout (missing 3 SSH pointer fields = 24 bytes), making all field accesses after the #ifdef ENABLE_COMMON_SSH block read/write wrong memory offsets. Specifically, rdp_client->disp reads NULL (actually the recording field), so RDP display resizing silently fails.

2. Early size instructions — Browser may send size instructions before the RDP connection is fully established, causing NULL pointer dereferences in the resize handler.

Files patched:

File	Fix
src/protocols/rdp/channels/common-svc.c	Add #include "config.h"
src/protocols/rdp/channels/disp.c	Add #include "config.h", add NULL guard in guac_rdp_disp_set_size()
src/protocols/rdp/channels/pipe-svc.c	Add #include "config.h"
src/protocols/rdp/channels/rdpei.c	Add #include "config.h"
src/protocols/rdp/channels/rdpgfx.c	Add #include "config.h"
src/protocols/rdp/input.c	Add #include "config.h", add NULL guard in guac_rdp_user_size_handler()

004-h264-passthrough.patch

Feature: end-to-end H.264. When an RDP server sends H.264 over the Graphics Pipeline, the raw NAL units are forwarded to the browser's WebCodecs VideoDecoder instead of being decoded by guacd and re-encoded as JPEG/PNG/WebP.

Measured with 1080p video playing, guacd session CPU over 30s:

	decode + re-encode	passthrough
xrdp (AVC420)	~100% of a core	2.0%
Windows 11 (AVC444)	90.6% of a core	2.1%

How it works. guac_rdp_gfx_surface_command() wraps FreeRDP's SurfaceCommand handler, copies the NAL data out of cmd->extra before the original handler can free it, and skips the GDI decode entirely for commands it captured. The frames are queued on the display layer and sent during the frame flush as a custom h264 instruction:

h264 <stream> <layer> <keyframe> <x> <y> <width> <height>
     <view> <numrects> [<x> <y> <width> <height>]...

Region rects identify which parts of the decoded picture are valid — the picture is always full-surface sized, so a server encoding only part of the screen leaves the rest holding nothing meaningful. A count of zero means the whole picture is valid.

AVC444. Windows only offers H.264 when the client advertises the AVC444 capability: FreeRDP emits the RDPGFX V10 capability sets only when GfxAVC444 is set, and Windows offers nothing below V10 (verified by advertising AVC420 alone at V8.1 and receiving zero H.264 — only CLEARCODEC and CAPROGRESSIVE). AVC444 is therefore always advertised, and AVC444 streams are handled rather than avoided.

An AVC444 picture is split across two views inside one H.264 sequence — FreeRDP decodes both through the same H264_CONTEXT. Both views must therefore reach the client's decoder, in order; dropping either leaves later pictures referencing data it never received, which renders as blocky wrongly-coloured macroblocks and reports no decode error, since what arrives is well formed. The view field marks which is which:

view	meaning	drawn
0	AVC420, or the main view of AVC444	yes
1	AVC444 auxiliary chroma, v1 layout	no
2	AVC444 auxiliary chroma, v2 layout	no

The client decodes every view. View 0 is drawn; the auxiliary view supplies the chroma samples it lacks, and static/guac/Yuv444.js combines the two into full 4:4:4 in a single WebGL2 pass, inverting the encoder's chroma filter to recover the one sample per 2x2 block that neither view carries. That matters for subpixel-antialiased text, which reads as fringed at 4:2:0. The path falls back to drawing view 0 alone at 4:2:0 on missing WebGL2, a lost GL context, or a pixel format it cannot read.

Threading. The frame queue has its own lock, deliberately not the display's pending_frame.lock. The render thread holds that one across the whole flush including image encoding, so queueing under it blocked the protocol thread — which for RDP is the thread that sends RDPGFX_FRAME_ACKNOWLEDGE, and a server throttles when frames go unacknowledged. The NAL copy happens before locking, and the flush detaches the queue and sends outside it.

Frame signalling. Two non-obvious requirements, both of which stall the stream if missed:

• Signal via rdp_client->gdi_modified, not a direct notify_modified(). EGFX has no explicit frame boundary, so per-surface-command notifies keep FRAME_MODIFIED permanently set and every flush waits out MAX_FRAME_DURATION (100ms).
• An H.264-only frame must count toward frame_nonempty, or no NOP is enqueued, no worker runs, and sync is never sent — which stops display.flush() client-side and inflates guac_client_get_processing_lag().

Keyframes are detected as an IDR slice (NAL type 5) only. Treating an SPS (type 7) as sufficient marks ordinary delta frames as keyframes, and handing a decoder a delta labelled key makes it discard queued work.

Configuration: none. Passthrough follows the per-connection enable-h264 argument, which rustguac sets from the connection entry. There are no environment variables.

Requires: a server sending H.264 over RDPGFX, and a browser with WebCodecs (Chrome/Edge 94+, Firefox 130+). See docs/rdp-h264.md for the Windows host settings, which are not optional.

Files patched:

File	Change
src/libguac/guacamole/display.h	guac_display_layer_set_h264(), view constants
src/libguac/display-priv.h	H.264 frame queue and its lock on the layer
src/libguac/display-layer.c	Queue frames without blocking on the display lock
src/libguac/display-layer-list.c	Init/destroy the queue lock; free queued frames
src/libguac/display-plan.c	Send queued frames during flush; walk pending-frame layers
src/libguac/display-flush.c	Count H.264-only frames toward frame_nonempty
src/protocols/rdp/channels/rdpgfx.c	Capture NAL data, skip the GDI decode, forward both AVC444 views
src/protocols/rdp/rdp.h	Store the original SurfaceCommand/CapsConfirm callbacks
src/protocols/rdp/settings.c	Advertise GfxH264 and GfxAVC444 when enable-h264 is set
src/protocols/rdp/settings.h	enable_h264 connection parameter

005-rdp-resize-dirty-flush.patch

Problem: After a dynamic RDP display resize (browser window resized, resize-method=display-update), regions of the desktop render as solid black until something repaints them. guac_rdp_gdi_desktop_resize() resizes the FreeRDP GDI buffer and the guac display layer but never marks the layer dirty, so guac_display_layer_close_raw() flushes nothing and the client keeps its stale/blank canvas for the resized layer. See sol1/rustguac#118 (reported by @Bails309, who diagnosed the root cause and supplied the fix).

Fix: In guac_rdp_gdi_desktop_resize(), after the layer resize and before guac_display_layer_close_raw():

1. Mark the entire layer dirty (guac_rect_init(&current_context->dirty, ...)) so a full repaint is flushed to the client.
2. Issue a RefreshRect for the full new desktop so the server re-sends authoritative pixels (legacy bitmap update path).

Scope: Fixes the legacy bitmap update path, which is rustguac's default (enable_gfx defaults to false). The RDPGFX surface cache ignores RefreshRect, so GFX sessions are not addressed by this patch; in practice GFX sessions have not reproduced the artifact.

Files patched:

File	Fix
src/protocols/rdp/gdi.c	Mark layer dirty + RefreshRect after resize in guac_rdp_gdi_desktop_resize()

007-rdp-disp-mod16.patch

Problem: When the negotiated RDP display dimensions aren't a multiple of 16, the H.264 graphics pipeline (16x16 macroblocks) pads encoded frames with all-zero YUV macroblocks. The chroma plane straddling the real/padding boundary contaminates the bottom-most real chroma row, which after client-side bilinear scaling spreads into a saturated green band (YUV(0,0,0) -> RGB ~ #008700) along the bottom edge. Mod-2 rounding (the upstream default) is insufficient — the whole bottom 16-row macroblock strip is affected.

Fix: In guac_rdp_disp_set_size(), round both width and height down to a multiple of 16 (replacing the existing "width must be even" mod-2 rounding). Costs up to 15px of unused canvas margin, avoidable by sizing the viewport so the requested height is already mod-16.

Ported from pletch/guacamole-server@b28bdac (fixes-1.6.0). Complements 005-rdp-resize-dirty-flush.patch: 005 fixes black regions on the legacy bitmap path, 007 fixes the green band on the H.264/GFX path.

Files patched:

File	Fix
src/protocols/rdp/channels/disp.c	Round display dimensions down to mod-16 in guac_rdp_disp_set_size()

008-spice-protocol.patch

Feature: Adds native SPICE protocol support (libguac-client-spice), vendored from the upstream PR apache/guacamole-server#688 (GUACAMOLE-261). Enables connecting to SPICE displays (e.g. Proxmox VE / QEMU consoles). Requires libspice-client-glib-2.0-dev (>= 0.38) at build time; guacd is configured --with-spice.

Vendored as the diff of the PR branch against its merge-base with our pinned guacd. The PR's incidental, non-SPICE change to src/terminal/terminal.c (SSH terminal keyboard-modifier handling) is excluded here: it is unrelated to SPICE and conflicted with our pinned base. The bundled guacclip tool is included in the source but not built (--disable-guacclip, matching how we treat guacenc/guaclog); see sol1/rustguac#181 for a possible future clipboard-audit feature.

Files patched: new src/protocols/spice/* and src/guacclip/* trees, plus additive hooks in configure.ac, Makefile.am, src/libguac/* (protocol constants, user handlers, rect), and per-protocol input.c.

009-spice-empty-port.patch

Bug: For TLS-only SPICE (e.g. Proxmox VE consoles) rustguac sends an empty port connect arg so guacd connects via tls-port. guac_spice_session_configure() set the spice-gtk port property whenever settings->port != NULL, but the parsed value for an omitted arg is an empty string (non-NULL), so spice-gtk logged GSpice: Invalid port value on every channel while parsing "".

Fix: Only set the plain port when it is non-empty (settings->port[0] != '\0'), so TLS-only connections use tls-port cleanly with no warning.

Files patched: src/protocols/spice/auth.c.

010-rdp-multimonitor.patch

Feature: Adds RDP multi-monitor support. A secondary-monitors arg enables it; the Display Update module (channels/disp.c) tracks a per-monitor layout (tiled left-to-right, top-aligned, with RDP-valid geometry) and sends the full DISPLAY_CONTROL_MONITOR_LAYOUT array via SendMonitorLayout instead of a single monitor. The RDP host extends the desktop across the monitors and streams one combined framebuffer, so no client-side compositing is needed (unlike SPICE). guacd advertises secondary-monitors on user join and publishes the multimon-layout layer parameter so a multi-monitor client can split the framebuffer into per-monitor windows. Reuses the protocol-agnostic client machinery added with 008.

Files patched: src/protocols/rdp/settings.{c,h}, src/protocols/rdp/channels/disp.{c,h}, src/protocols/rdp/input.c, src/protocols/rdp/user.c.

011-spice-ca-pem.patch

Bug: rustguac sends the SPICE CA certificate (the connection entry's PEM, or the Proxmox cluster CA from spiceproxy) as the ca-cert connect arg, and the plugin from 008 passed it to spice-gtk's ca-file property. ca-file is a path, so OpenSSL failed to open a "file" named after the PEM text, spice-gtk logged loading ca certs from ... failed, fell back to the system trust store, and rejected servers signed by a private CA. See sol1/rustguac#308.

Fix: When ca-cert contains a PEM header (-----BEGIN ), pass it in memory through spice-gtk's ca property (a GByteArray of PEM data, available since spice-gtk 0.15), which adds each certificate to the TLS trust store. Any other value is still treated as a path and passed as ca-file. The settings field is renamed ca_file -> ca_cert to match.

Files patched: src/protocols/spice/auth.c, src/protocols/spice/settings.{c,h}.

Applying patches

Patches are applied automatically by all build scripts (build-deb.sh, build-rpm.sh, install.sh, dev.sh, Dockerfile). To apply manually:

cd ../guacamole-server
git apply ../rustguac/patches/001-freerdp3-debian13.patch

To check if patches are already applied:

cd ../guacamole-server
git apply --check ../rustguac/patches/001-freerdp3-debian13.patch 2>&1 || echo "Already applied or conflict"

Adding new patches

1. Make changes in the ../guacamole-server working tree
2. Export: cd ../guacamole-server && git diff > ../rustguac/patches/NNN-description.patch
3. Patches are applied in numeric order by the build scripts