H.264 passthrough for RDP

rustguac can forward an RDP server's H.264 stream straight to the browser's WebCodecs decoder instead of having guacd decode it and re-encode the pixels as JPEG/WebP. With 1080p video playing, that takes guacd from roughly a full CPU core down to about 2% of one — measured on both xrdp and Windows 11.

Enable it per connection with the H.264 checkbox on the entry. There is no server-side configuration and no environment variables.

What the server has to send#

Passthrough only engages when the server actually sends H.264 over the RDPGFX Graphics Pipeline. Both AVC420 and AVC444 work.

• xrdp — set a GFX codec order listing H.264 in /etc/xrdp/gfx.toml. Nothing else is needed; xrdp sends AVC420 and passthrough applies to every frame.
• Windows — needs the host settings below. Without them Windows sends CLEARCODEC and CAPROGRESSIVE instead, and guacd falls back to decoding and re-encoding. The display is still correct, just far more expensive.

Windows 11 host settings#

These are not optional, and two of them are non-obvious. Verified on Windows 11 Pro with an NVIDIA RTX 3070.

1. Disable the WDDM display driver for Remote Desktop#

Group Policy → Computer Configuration → Administrative Templates → Windows Components → Remote Desktop Services → Remote Desktop Session Host → Remote Session Environment → "Use WDDM graphics display driver for Remote Desktop Connections" → Disabled. Reboot.

This is what allows hardware H.264 encoding to engage at all. With the WDDM driver in place, the session runs on the GPU for rendering but never reaches the encoder: Task Manager shows GPU 3D under load while Video Encode stays at 0% and nvidia-smi encodersessions reports nothing. Setting the registry values below without also disabling WDDM changes nothing — the policy is accepted and has no effect.

Note this changes the display pipeline, so check dynamic resize and multi-monitor behaviour after enabling it.

2. Prefer AVC 4:4:4#

HKLM\SOFTWARE\Policies\Microsoft\Windows NT\Terminal Services
    AVC444ModePreferred = 1   (DWORD)

Counter-intuitively this is required for hardware encoding, not for image quality: setting it to 0 on the test host stopped NVENC entirely and fell back to software encoding. Windows then sends AVC444, which rustguac handles — both views are forwarded to the browser and combined there into full 4:4:4 chroma, so leaving this on costs nothing in image quality and is what makes hardware encoding engage.

3. Raise the frame rate cap#

HKLM\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations
    DWMFRAMEINTERVAL = 15   (DWORD)   # 60fps; default is 30

4. Hardware encoding policy#

HKLM\SOFTWARE\Policies\Microsoft\Windows NT\Terminal Services
    AVCHardwareEncodePreferred = 1   (DWORD)
    bEnumerateHWBeforeSW       = 1   (DWORD)

contrib/setup-rdp-performance.ps1 applies items 2–4; item 1 is a Group Policy change and must be made separately. Run the script with -Report first to see what is already configured.

What you do not need#

The "Prioritize H.264/AVC 444 Graphics Mode for Remote Desktop Connections" policy does not need to be Disabled. It was used during development to force AVC420, but rustguac handles AVC444 directly now, and on the test host Windows ignored the client's AVC420 request regardless — it confirmed RDPGFX capability version 10.7 with the AVC_THINCLIENT flag set and sent AVC444 anyway.

Verifying#

After connecting, on the Windows host: Task Manager → Performance → GPU → Video Encode should be non-zero during activity. If it reads 0%, hardware encoding is not engaged — check the WDDM policy first.

On the rustguac host, with guacd running at trace level (-L trace, e.g. via systemctl edit rustguac-guacd to override ExecStart):

# Which codecs the server is sending. 11 = AVC420, 14/15 = AVC444(v2).
# 8 = CLEARCODEC and 9 = CAPROGRESSIVE mean H.264 is not being used.
journalctl -u rustguac-guacd --since '1 min ago' | grep "TRACE:" \
  | grep -oP 'codec=\K[0-9]+' | sort -n | uniq -c

# The RDPGFX capability version the server confirmed
journalctl -u rustguac-guacd --since '2 min ago' | grep "RDPGFX capability version"

# guacd CPU with the workload running
./contrib/measure-guacd-cpu.sh

display-wrk near zero in that last output is the sign that passthrough is working: it means image encoding is not happening.

In the browser console, __h264.stats() reports decoder health — avgDecodeLatencyMs in the low single digits, framesDropped and gcLeaks at zero, auxViewsDecoded counting AVC444 auxiliary views.

Recording#

Session recordings capture the raw stream, so a recording of an H.264 session contains h264 instructions and needs the same WebCodecs decoder to play back.