rustguac/patches

guacamole-server patches

These patches are applied to the apache/guacamole-server source tree before building guacd. They fix compilation and runtime issues when building against FreeRDP 3.x on Debian 13 (Trixie).

001-freerdp3-debian13.patch

Problem: guacamole-server 1.6.1 fails to compile against FreeRDP 3.15+ (as shipped in Debian 13) due to:

1. Autoconf feature detection failure — FreeRDP 3.15 marks codecs_free() as deprecated. The -Werror flag in configure.ac causes all compile-time feature-detection tests to fail, cascading into 10+ undefined macros and wrong #ifdef code paths.

2. Deprecated function pointer API — FreeRDP 3.x deprecates direct struct member access for ->input->KeyboardEvent(), ->input->MouseEvent(), etc. behind WITH_FREERDP_DEPRECATED. The safe replacement functions are freerdp_input_send_keyboard_event(), freerdp_input_send_mouse_event(), etc.

3. NULL pointer dereference in display channel — FreeRDP 3.x fires PubSub ChannelConnected events before guac_rdp_disp is allocated, causing a segfault when the callback writes to disp->requested_width (offset 0x18 of NULL).

Files patched:

File	Fix
configure.ac	Add -Wno-error=deprecated-declarations to both FreeRDP 2.x and 3.x PKG_CHECK_MODULES blocks so autoconf feature detection works
src/protocols/rdp/Makefile.am	Add -Wno-error=deprecated-declarations to all three CFLAGS targets
src/protocols/rdp/tests/Makefile.am	Same for test CFLAGS
src/protocols/rdp/keyboard.c	Replace ->input->KeyboardEvent(), ->input->UnicodeKeyboardEvent(), ->input->SynchronizeEvent() with safe API functions
src/protocols/rdp/input-queue.c	Replace all ->input->MouseEvent() calls with freerdp_input_send_mouse_event()
src/protocols/rdp/channels/disp.c	Add NULL guards in guac_rdp_disp_channel_connected() and guac_rdp_disp_channel_disconnected()

002-kerberos-nla.patch

Feature: Adds Kerberos NLA authentication support to guacd's RDP protocol, based on GUACAMOLE-2057 (PR #581). This allows RDP connections to use Kerberos instead of NTLM for NLA, which is required as Microsoft phases out NTLM.

Three new connection parameters:

Parameter	Values	FreeRDP3 Setting
auth-pkg	"" (negotiate), "kerberos", "ntlm"	FreeRDP_AuthenticationPackageList
kdc-url	KDC server URL (optional)	FreeRDP_KerberosKdcUrl
kerberos-cache	Path to ccache file (optional)	FreeRDP_KerberosCache

Files patched:

File	Fix
src/protocols/rdp/settings.h	Add guac_rdp_auth_package enum, add auth_pkg, kdc_url, kerberos_cache fields to guac_rdp_settings
src/protocols/rdp/settings.c	Add connection parameter parsing, FreeRDP3 settings push, memory cleanup

Differences from upstream PR #581:

• Dropped FreeRDP2 code path (not needed on Debian 13)
• Fixed guac_strdup() leak in freerdp_settings_set_string() calls (FreeRDP3 copies internally)
• Fixed typos ("NTML" -> "NTLM", "negotiatoin" -> "negotiation")

Requires: FreeRDP 3.x built with Kerberos support (-DWITH_KRB5=ON). Debian 13's freerdp3-dev includes this by default.

003-null-guard-and-config-h.patch

Problem: Two related issues causing RDP display resize to silently fail:

1. Missing config.h include — Several RDP channel source files and input.c do not include config.h, so ENABLE_COMMON_SSH is undefined in those compilation units. This causes the guac_rdp_client struct to have a different layout (missing 3 SSH pointer fields = 24 bytes), making all field accesses after the #ifdef ENABLE_COMMON_SSH block read/write wrong memory offsets. Specifically, rdp_client->disp reads NULL (actually the recording field), so RDP display resizing silently fails.

2. Early size instructions — Browser may send size instructions before the RDP connection is fully established, causing NULL pointer dereferences in the resize handler.

Files patched:

File	Fix
src/protocols/rdp/channels/common-svc.c	Add #include "config.h"
src/protocols/rdp/channels/disp.c	Add #include "config.h", add NULL guard in guac_rdp_disp_set_size()
src/protocols/rdp/channels/pipe-svc.c	Add #include "config.h"
src/protocols/rdp/channels/rdpei.c	Add #include "config.h"
src/protocols/rdp/channels/rdpgfx.c	Add #include "config.h"
src/protocols/rdp/input.c	Add #include "config.h", add NULL guard in guac_rdp_user_size_handler()

004-h264-display-worker.patch

Feature: H.264 passthrough via guac_display worker integration. When the RDP server sends AVC420 encoded frames (H.264), the raw NAL units are passed through to the browser's WebCodecs VideoDecoder instead of being decoded server-side and re-encoded as JPEG/PNG/WebP.

AVC420 only — AVC444 is deliberately disabled. The passthrough forwards a single H.264 bitstream per surface command. AVC420 carries a complete YUV420 frame, which WebCodecs decodes directly. AVC444 instead splits the image across two bitstreams (bitstream[0] luma/main view + bitstream[1] auxiliary chroma) to reconstruct YUV444; the passthrough only forwards bitstream[0], so a Windows host that negotiated AVC444 renders as a corrupted luma+chroma split — two blocks with green and magenta casts. We therefore advertise GfxH264 without GfxAVC444/GfxAVC444v2 so the server always uses AVC420. (RemoteFX/RFX is unaffected — it is a separate codec path and renders correctly.) Properly supporting AVC444 would require decoding both bitstreams and recombining the chroma planes in the browser (e.g. a second VideoDecoder + a WebGL merge shader), which is not implemented.

Architecture: The SurfaceCommand callback intercepts H.264 data and stores it on the display layer. During the normal frame flush cycle (guac_display_plan_apply), the H.264 data is sent to clients as a custom h264 instruction before worker threads start encoding. All IMG operations for the H.264 layer are skipped, eliminating the decode→re-encode overhead.

This approach avoids the socket contention issue that occurred when H.264 was sent directly from FreeRDP's SurfaceCommand callback thread, which raced with guac_display's worker threads writing to the same socket.

Files patched:

File	Fix
src/libguac/display-priv.h	Add H.264 buffer fields to guac_display_layer (data, length, keyframe, rect)
src/libguac/guacamole/display.h	Add guac_display_layer_set_h264() public API
src/libguac/display-layer.c	Implement guac_display_layer_set_h264() with lock management
src/libguac/display-layer-list.c	Free H.264 data in layer cleanup
src/libguac/display-plan.c	Send H.264 data during plan apply, skip IMG ops for H.264 layers
src/protocols/rdp/channels/rdpgfx.c	Wrap SurfaceCommand to store H.264 on display layer after GDI decode
src/protocols/rdp/settings.c	Enable GfxH264 (AVC420) in FreeRDP settings; leave GfxAVC444 disabled (see AVC420-only note above)

Requires: RDP server with H.264 support (xrdp with x264, or Windows with AVC hardware encoder). Browser must support WebCodecs VideoDecoder (Chrome/Edge 94+, Firefox 130+).

005-rdp-resize-dirty-flush.patch

Problem: After a dynamic RDP display resize (browser window resized, resize-method=display-update), regions of the desktop render as solid black until something repaints them. guac_rdp_gdi_desktop_resize() resizes the FreeRDP GDI buffer and the guac display layer but never marks the layer dirty, so guac_display_layer_close_raw() flushes nothing and the client keeps its stale/blank canvas for the resized layer. See sol1/rustguac#118 (reported by @Bails309, who diagnosed the root cause and supplied the fix).

Fix: In guac_rdp_gdi_desktop_resize(), after the layer resize and before guac_display_layer_close_raw():

1. Mark the entire layer dirty (guac_rect_init(&current_context->dirty, ...)) so a full repaint is flushed to the client.
2. Issue a RefreshRect for the full new desktop so the server re-sends authoritative pixels (legacy bitmap update path).

Scope: Fixes the legacy bitmap update path, which is rustguac's default (enable_gfx defaults to false). The RDPGFX surface cache ignores RefreshRect, so GFX sessions are not addressed by this patch; in practice GFX sessions have not reproduced the artifact.

Files patched:

File	Fix
src/protocols/rdp/gdi.c	Mark layer dirty + RefreshRect after resize in guac_rdp_gdi_desktop_resize()

007-rdp-disp-mod16.patch

Problem: When the negotiated RDP display dimensions aren't a multiple of 16, the H.264 graphics pipeline (16x16 macroblocks) pads encoded frames with all-zero YUV macroblocks. The chroma plane straddling the real/padding boundary contaminates the bottom-most real chroma row, which after client-side bilinear scaling spreads into a saturated green band (YUV(0,0,0) -> RGB ~ #008700) along the bottom edge. Mod-2 rounding (the upstream default) is insufficient — the whole bottom 16-row macroblock strip is affected.

Fix: In guac_rdp_disp_set_size(), round both width and height down to a multiple of 16 (replacing the existing "width must be even" mod-2 rounding). Costs up to 15px of unused canvas margin, avoidable by sizing the viewport so the requested height is already mod-16.

Ported from pletch/guacamole-server@b28bdac (fixes-1.6.0). Complements 005-rdp-resize-dirty-flush.patch: 005 fixes black regions on the legacy bitmap path, 007 fixes the green band on the H.264/GFX path.

Files patched:

File	Fix
src/protocols/rdp/channels/disp.c	Round display dimensions down to mod-16 in guac_rdp_disp_set_size()

Applying patches

Patches are applied automatically by all build scripts (build-deb.sh, build-rpm.sh, install.sh, dev.sh, Dockerfile). To apply manually:

cd ../guacamole-server
git apply ../rustguac/patches/001-freerdp3-debian13.patch

To check if patches are already applied:

cd ../guacamole-server
git apply --check ../rustguac/patches/001-freerdp3-debian13.patch 2>&1 || echo "Already applied or conflict"

Adding new patches

1. Make changes in the ../guacamole-server working tree
2. Export: cd ../guacamole-server && git diff > ../rustguac/patches/NNN-description.patch
3. Patches are applied in numeric order by the build scripts