5 Commits

Author SHA256 Message Date
Alphaeus Mote d5274dfbea test(backend): presigned link expiry tolerates slow runners [skip ci]
CI #181 measured 298 s left on a 300 s link; allow 10 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 14:57:02 -05:00
Alphaeus Mote d1f09eec2b feat(screen): H.264 video for Remote View, RemoteFX fallback at 30 fps [skip ci]
The screen goes as H.264 where the device can encode it: Media Foundation,
the GPU's encoder MFT first (async; NVIDIA, Intel, AMD), else Microsoft's
software encoder. The browser decodes with WebCodecs onto a canvas over the
RDP canvas; RDP keeps input, cursor, clipboard and Ctrl+Alt+Del.

- Negotiated in band: the browser asks with destination "screen;video=h264";
  a host with an encoder answers EXRMUX01 and frames stdio (channel 0 RDP,
  channel 1 video). Old hosts and hosts without an encoder (Windows PE,
  Server Core) answer plainly; the relay is unchanged.
- Baseline, no B-frames, low latency, CBR ~0.1 bit/pixel/frame; only changes
  are encoded, plus three refinement frames; at most 4 frames unacknowledged.
- Fallbacks: hardware -> software encoder; no encoder for a size -> STOP and
  RDP bitmaps; browser that cannot decode sends OFF -> RDP bitmaps.
- mfplat/ole32 loaded at run time from System32: static imports unchanged,
  so the exe still starts on Windows PE.
- Default frame rate 15 -> 30 (video and RemoteFX).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 13:56:29 -05:00
Alphaeus Mote cfba9bb51e feat(screen): text clipboard both ways in Remote View control sessions [skip ci]
The screen host now serves RDP's clipboard channel on Windows: a copy on the
device is announced to the operator's browser and a copy in the browser is
placed on the device, plain text only, up to 1 MiB each way. View-only
sessions get no clipboard channel. Device changes are polled (no listener
window); each write uses a message-only window made and destroyed for it,
the one exception to the no-windows rule, now documented.

The web view waits for the component's clipboard set-up before connecting:
it fires "ready" first, and a connection made then asks for no clipboard
channel. Verified both ways live through the stdio bridge and headless
Chrome against this machine's clipboard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 11:14:23 -05:00
Alphaeus Mote 61da8b704f chore(screen): log the client's offered codecs and the codec picked [skip ci]
The IronRDP web client 0.7 offers RemoteFX only, so that is what every
session encodes with; the log makes that visible when a session is slow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 10:53:04 -05:00
Alphaeus Mote 8de2214c93 feat(webui): right-click menus and copy paths in Files and Registry [skip ci]
Files: right-click a row for its menu (the whole selection when the row is
part of it), with Copy Path(s) and Copy Name; right-click the folder for
Copy Folder Path, New File/Folder, Upload and Refresh; a Copy Folder Path
button sits next to the path.
Registry: Copy Key Path/Name on keys, Copy Name/Data/Value Path on values
(the whole value is fetched when the list only holds its start), and a
Copy Key Path button next to the path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 10:53:03 -05:00
26 changed files with 2653 additions and 49 deletions
+17 -4
View File
@@ -3309,7 +3309,8 @@ newer agent.* Upgrade the agent.
keyboard and mouse (**Control**) or without (**View Only**). It works at the
sign-in, lock and UAC screens, when nobody is signed in, and in Windows PE.
**How it works.** The pixels and input are RDP. The console runs Devolutions'
**How it works.** Input, cursor and clipboard are RDP; the picture is H.264
video where the device can encode it (below), RDP bitmaps otherwise. The console runs Devolutions'
IronRDP web client (`@devolutions/iron-remote-desktop` with the
`@devolutions/iron-remote-desktop-rdp` WASM backend, loaded only on this page),
which connects over a WebSocket using RDCleanPath. On the device the agent
@@ -3393,9 +3394,21 @@ agent), a device without the screen host or where capture fails shows why.
changed."); reconnect to follow it.
- When the host can't open the input desktop, the console shows **Secure Screen**
and keeps the session open.
- No clipboard in this version.
- **Windows PE**: the portable package carries the screen host; it starts in the
agent's session on the interactive desktop.
- **Clipboard** (Windows, control sessions): copy text on either side and paste on
the other, up to 1 MiB; no files or images. The browser asks once for clipboard
access (Chromium; HTTPS only). View-only sessions have no clipboard.
- **Windows PE**: the portable package carries the screen host; the agent's service
starts it in the console session, like on full Windows.
- **Video.** On Windows with Media Foundation the screen goes as **H.264**: the
GPU's encoder when there is one (NVIDIA, Intel, AMD), else Microsoft's software
encoder; the browser decodes it with WebCodecs (in hardware where it can). The
toolbar shows **H.264** or **H.264 GPU** (hover for the encoder). Only changes
are sent, at up to 30 fps and about 0.1 bit per pixel per frame (about 6 Mbit/s
at most for a busy 1080p screen, next to nothing for a still one). It travels in
the same socket, so nothing else is opened. Windows PE and Server Core have no
encoder, browsers without WebCodecs (or over plain HTTP) can't decode it, and
older agents don't offer it: those sessions use RDP bitmaps (RemoteFX) at up to
30 fps, as does any session whose encoder or decoder fails.
**Security.**
+2
View File
@@ -772,6 +772,7 @@ dependencies = [
"async-trait",
"bytes",
"ironrdp-async",
"ironrdp-cliprdr",
"ironrdp-connector",
"ironrdp-core",
"ironrdp-graphics",
@@ -780,6 +781,7 @@ dependencies = [
"ironrdp-server",
"ironrdp-session",
"ironrdp-tokio",
"rayon",
"rcgen",
"serde",
"serde_json",
+8
View File
@@ -21,12 +21,14 @@ ironrdp-server = { version = "0.13", default-features = false, features = ["rayo
ironrdp-pdu = "0.9"
ironrdp-core = "0.2"
ironrdp-rdcleanpath = "0.3.1"
ironrdp-cliprdr = "0.7"
tokio = { version = "1", features = ["rt", "macros", "io-util", "io-std", "sync", "time"] }
tokio-rustls = { version = "0.26", default-features = false, features = ["ring", "tls12"] }
rcgen = { version = "0.14", default-features = false, features = ["ring", "crypto"] }
anyhow = "1"
async-trait = "0.1"
bytes = "1"
rayon = "1"
serde = { version = "1", features = ["derive"] }
serde_json = "1"
tracing = "0.1"
@@ -41,12 +43,18 @@ features = [
"Win32_Graphics_Dxgi",
"Win32_Graphics_Dxgi_Common",
"Win32_Graphics_Gdi",
"Win32_Media_MediaFoundation",
"Win32_Security",
"Win32_System_Com",
"Win32_System_DataExchange",
"Win32_System_Memory",
"Win32_System_LibraryLoader",
"Win32_System_Ole",
"Win32_System_Registry",
"Win32_System_RemoteDesktop",
"Win32_System_StationsAndDesktops",
"Win32_System_SystemInformation",
"Win32_System_Variant",
"Win32_System_Threading",
"Win32_UI_HiDpi",
"Win32_UI_Input_KeyboardAndMouse",
+53 -14
View File
@@ -23,7 +23,7 @@ ExecuteR-Screen --version
| `--stdio` | Required. stdin/stdout carry the WebSocket byte stream (the only transport). |
| `--view-only` | All keyboard/mouse input from the client is dropped here (the relay cannot see inside RDP). |
| `--monitor -1` | Default. All monitors as one virtual desktop. `--monitor N` serves monitor `N` (index from the `state` event). |
| `--max-fps n` | Frame rate cap, 1-60, default 15. |
| `--max-fps n` | Frame rate cap, 1-60, default 30 (video and RDP bitmaps). |
| `--log-level` | Human log verbosity on stderr, default `info`. |
Options come only from the command line. Nothing read from stdin is ever interpreted as an
@@ -53,6 +53,14 @@ Exactly the bytes the IronRDP web client exchanges with an RDCleanPath proxy ove
x224_connection_confirm, [server_certificate])`.
3. Plaintext RDP in both directions (the client skips its own TLS upgrade, as with the gateway).
**Video (H.264).** A browser that can decode H.264 sends `destination` = `screen;video=h264`.
When this host has an H.264 encoder it then writes `EXRMUX01` before the RDCleanPath response,
and from there on both directions are frames `[channel u8][length u32 LE][payload]`: channel 0
carries the bytes above (RDCleanPath response, then RDP), channel 1 the video messages
(`src/video.rs`: `START`, `FRAME` with Annex B H.264, `STOP`; from the browser `ACK`, `KEY`,
`OFF`). Hosts without an encoder, and older hosts (the field is opaque to them), answer plainly,
so every combination of console and agent version works. The relay passes the bytes unchanged.
Refusals: a request without an X.224 connection request (VMConnect) gets `new_http_error(400)`;
a client that does not offer TLS security gets `new_negotiation_error(X.224 failure
SSL_REQUIRED_BY_SERVER)`. Bytes that are not an RDCleanPath PDU get no answer (like the gateway).
@@ -102,7 +110,24 @@ stdout ◄─ (X.224 confirm wrapped │ ▲
handshake, socket or listener is involved. A self-signed certificate (rcgen, in memory, per
process) is what the RDCleanPath response carries; the client only parses its public key.
`server_addr` is reported as `127.0.0.1:3389` (the client ignores it).
* **Encoding.** ironrdp-server negotiates RemoteFX (the web client's default) and falls back to
* **Video.** With a framed stream (above) the capture pump sends the screen as H.264 instead of
RDP bitmaps (`src/video.rs`, `src/platform/windows/h264.rs`); RDP keeps the desktop size,
cursor, input and clipboard. BGRX is converted to NV12 (BT.709, limited range, parallel rows)
and encoded by Media Foundation: the first hardware encoder MFT that takes the size
(asynchronous; NVIDIA, Intel, AMD), else Microsoft's software encoder. Constrained to what the
browser decodes without delay: Baseline profile, no B-frames, low-latency mode, CBR at about
0.1 bit per pixel per frame (1.5-12 Mbit/s). Only changes are encoded, plus three frames after
the screen settles so it sharpens. At most 4 frames may be unacknowledged by the browser (it
acknowledges each frame it draws), so a slow link or decoder skips frames instead of falling
behind. A hardware encoder that fails, or gives no output 15 times in a row, is replaced by the
software one; with no encoder for a size (or any failure there) the host sends `STOP` and RDP
bitmaps serve it; the browser sends `OFF` (no picture 5 s after `START`, or 3 decoder failures
in a row) and RDP bitmaps serve the rest of the session. The browser draws on a canvas over the
RDP canvas with `pointer-events: none`, so the mouse still reaches the RDP client. Measured
(release, this machine's software encoder, no GPU): 1080p with a 400x400 textured block moving
encodes in about 10 ms a frame including NV12 conversion, about 1.6 Mbit/s at 30 fps
(`h264_1080p_moving_block`).
* **Encoding.** Without video, ironrdp-server negotiates RemoteFX (the web client's default) and falls back to
RLE bitmaps. The pump sends dirty regions only (DXGI dirty/move rectangles, or 64x64 tile diffs
for GDI), merged and capped, and ironrdp-server diffs them against its framebuffer again.
Frames are paced by `--max-fps`; the update channel holds at most two frames, so a slow link
@@ -116,16 +141,24 @@ stdout ◄─ (X.224 confirm wrapped │ ▲
screen's aspect ratio (capped at the window's height), so it fills the page without bars below.
* **Pointer.** Cursor shapes are sent as 32-bpp pointer updates using the client pointer cache;
hidden cursors are hidden. The cursor position is not pushed (the browser draws its own pointer).
* **Clipboard: not implemented.** ironrdp-server has a CLIPRDR channel, but writing the Windows
clipboard (`SetClipboardData`) requires a window to own it, which the security rules below forbid.
When added, it must be plain text only, capped at 1 MiB, and disabled with `--view-only`.
* **Clipboard (Windows).** Plain text both ways over CLIPRDR (`src/clipboard.rs`): a copy on either
side is announced to the other, which fetches it (the operator's text is placed on the device as
soon as it is announced, so it is there before they paste). Capped at 1 MiB (UTF-8) each way; no
files or images; no clipboard channel at all with `--view-only`. Device changes are found by
polling the clipboard sequence number (4 per second); writes go through a message-only window
made and destroyed for that one write (see Security 1). The web client needs its clipboard set
up before it connects, or it asks for no clipboard channel (`RemoteScreenView.tsx` waits for it).
Linux: no clipboard yet.
* **Codec.** The IronRDP web client 0.7 offers RemoteFX only (logged as `client bitmap codecs` and
`bitmap codec`), so RDP bitmaps are RemoteFX whatever else the host could do; hence the separate
video channel.
## Platforms
| Platform | Capture | Input | Notes |
|---|---|---|---|
| Windows 10/11 (priority 1) | DXGI Desktop Duplication (dirty + move rects), GDI `BitBlt` fallback (secure desktop if duplication is refused, VMs, rotated screens) | `SendInput`: scancodes incl. extended keys, Unicode, absolute mouse over the virtual desktop (negative origins), wheel, horizontal wheel, X buttons, lock-key sync; Ctrl+Alt+Del via `SendSAS` | Capture and injection threads follow the input desktop (`OpenInputDesktop` + `SetThreadDesktop`): lock screen, UAC prompt and sign-in screen are served when the host runs as SYSTEM in the console session. Per-monitor DPI aware. Tested: unit, in-memory e2e, and a live session over real stdio on this machine. |
| Windows PE (priority 1) | GDI only (no DWM) | `SendInput` | Detected via `HKLM\SYSTEM\CurrentControlSet\Control\MiniNT` (`winpe:true`). Static CRT, no VC++ runtime. Static imports: kernel32, user32, gdi32, advapi32, ntdll, bcrypt, bcryptprimitives, oleaut32, api-ms-win-core-synch-l1-2-0. dxgi/d3d11/sas are loaded at run time only on full Windows. The agent runs as a service in session 0 there too, so it starts the host in the console session (session 0's desktop is never on the screen). |
| Windows PE (priority 1) | GDI only (no DWM); no Media Foundation, so RDP bitmaps (RemoteFX) instead of H.264 | `SendInput` | Detected via `HKLM\SYSTEM\CurrentControlSet\Control\MiniNT` (`winpe:true`). Static CRT, no VC++ runtime. Static imports: kernel32, user32, gdi32, advapi32, ntdll, bcrypt, bcryptprimitives, oleaut32, api-ms-win-core-synch-l1-2-0. dxgi/d3d11/sas and mfplat/ole32 (H.264) are loaded at run time only on full Windows. The agent runs as a service in session 0 there too, so it starts the host in the console session (session 0's desktop is never on the screen). |
| Linux X11 (priority 2) | `GetImage` on the root window + tile diff, monitors from RandR, cursor from XFixes | XTest (keys via evdev keycodes, buttons, wheels) | Pure-Rust X11 client (`x11rb`). Type-checked and clippy-clean for `x86_64-unknown-linux-gnu`; **not run on Linux yet**. No XShm (slower at large sizes), no Unicode typing. |
| Linux Wayland, macOS | `probe` reports `capture:false` with a reason; `serve` exits 2. | | |
@@ -151,22 +184,26 @@ stdout ◄─ (X.224 confirm wrapped │ ▲
ExecuteR-Screen runs as SYSTEM on the user's desktop, so it is treated as an elevation-of-privilege surface:
1. **No windows, no message pump.** The Windows code creates no HWND, hook or message loop
(cursor, capture and input use `GetCursorInfo`, DXGI/GDI and `SendInput`). This is also why
clipboard support is left out (see above).
1. **No lasting windows, no message pump.** Cursor, capture and input use `GetCursorInfo`,
DXGI/GDI and `SendInput`: no HWND, hook or message loop. The one exception is the clipboard
write: `SetClipboardData` needs an owner window after `EmptyClipboard`, so each write creates a
message-only window (default procedure, never pumped), places the text and destroys it at once.
Clipboard changes are polled, not listened for.
2. **DLL search path.** First thing in `main`: `SetDefaultDllDirectories(LOAD_LIBRARY_SEARCH_SYSTEM32)`
and `SetDllDirectoryW("")`. The executable is linked with `/DEPENDENTLOADFLAG:0x800`, so its static
imports also resolve from System32 only. Run-time DLLs (dxgi, d3d11, sas) are loaded with
`LoadLibraryExW(..., LOAD_LIBRARY_SEARCH_SYSTEM32)`. Nothing is loaded from the current directory,
`LoadLibraryExW(..., LOAD_LIBRARY_SEARCH_SYSTEM32)`, as are mfplat and ole32 for H.264 (Media
Foundation and COM, multithreaded apartment on the capture thread: no windows). Nothing is loaded from the current directory,
`PATH` or the user profile.
3. **stdin/stdout carry only RDCleanPath + RDP.** No file path, command or option is ever taken
3. **stdin/stdout carry only RDCleanPath + RDP** (and, framed beside them, video frames out and
acknowledgements in: fixed message types, frames capped at 1 MiB from the browser). No file path, command or option is ever taken
from the byte stream; the RDCleanPath fields are opaque and only logged (the token never).
The request is size-capped (64 KiB). Options come only from the agent's command line.
4. **No IPC objects or persistence.** No named pipes, no sockets at all (the RDP acceptor runs on
stdio; no loopback listener exists), no files, no registry writes (the only registry access is
a read of the WinPE marker key). The TLS key pair lives in memory only.
5. **Clipboard.** Not implemented. Any future implementation must be plain text only, size-capped
(1 MiB), and disabled in `--view-only`.
5. **Clipboard.** Plain text only (CF_UNICODETEXT), 1 MiB cap each way, off in `--view-only`
(no CLIPRDR channel is offered). File and image formats are refused.
6. **View-only** is enforced in this process: every keyboard and mouse event is dropped before mapping.
## Build
@@ -178,7 +215,8 @@ cd src/ScreenHost
cargo build --release # target/release/ExecuteR-Screen.exe
cargo test # unit + e2e + live stdio test (skips without a capturable screen)
cargo clippy --all-targets -- -D warnings
cargo test --release --test e2e -- --ignored --nocapture # 1080p throughput measurement
cargo test --release --test e2e -- --ignored --nocapture # 1080p throughput measurement (RemoteFX)
cargo test --release --lib h264_1080p -- --ignored --nocapture # 1080p H.264 encode measurement
```
`.cargo/config.toml` sets `-C target-feature=+crt-static` for the Windows MSVC targets (no VC++
@@ -213,6 +251,7 @@ Top-level dependencies (all permissive):
| rustls (via tokio-rustls) | Apache-2.0 OR ISC OR MIT | TLS types for the acceptor |
| ring | Apache-2.0 AND ISC | crypto provider |
| rcgen | MIT OR Apache-2.0 | self-signed certificate |
| rayon | MIT OR Apache-2.0 | parallel NV12 conversion |
| windows (Windows) | MIT OR Apache-2.0 | Win32 API |
| x11rb (Linux) | MIT OR Apache-2.0 | X11 protocol |
+12 -1
View File
@@ -133,6 +133,8 @@ pub type UpdateBatch = Vec<DisplayUpdate>;
pub struct PumpConfig {
pub max_fps: u32,
/// H.264 video, when the host can encode it: used once the browser takes it.
pub video: Option<crate::video::VideoPath>,
}
/// Number of client pointer-cache slots we use (clients advertise at least 25 for the
@@ -180,7 +182,7 @@ pub fn run_pump(
mut source: Box<dyn ScreenSource>,
tx: mpsc::Sender<UpdateBatch>,
ctl: Arc<PumpControl>,
cfg: PumpConfig,
mut cfg: PumpConfig,
mut on_state: impl FnMut(&SourceState),
mut on_origin: impl FnMut(i32, i32),
) {
@@ -258,7 +260,16 @@ pub fn run_pump(
if !fits {
// Layout changed under us; the next poll_state will resize. Ask for a full frame then.
ctl.full_refresh.store(true, Ordering::Relaxed);
} else if let Some(video) = cfg.video.as_mut().filter(|v| v.handles(&frame)) {
// Video carries the picture; RDP keeps the size, the cursor and input.
let changed = full || !matches!(captured, Capture::Unchanged);
if !video.tick(&frame, changed) {
ctl.full_refresh.store(true, Ordering::Relaxed);
}
sent_first_frame = true;
} else {
// Video stopped carrying the picture (the browser gave up on it): a complete frame now.
let full = full || cfg.video.as_mut().is_some_and(crate::video::VideoPath::hand_back);
let rects = if full {
vec![bounds]
} else {
+2 -2
View File
@@ -45,7 +45,7 @@ pub struct ServeArgs {
pub log_level: LogLevel,
}
pub const DEFAULT_MAX_FPS: u32 = 15;
pub const DEFAULT_MAX_FPS: u32 = 30;
pub const MAX_MAX_FPS: u32 = 60;
#[derive(Clone, Debug, PartialEq, Eq)]
@@ -187,7 +187,7 @@ mod tests {
let s = serve(&["--stdio"]).unwrap();
assert_eq!(
s,
ServeArgs { view_only: false, monitor: MonitorSelection::All, max_fps: 15, log_level: LogLevel::Info }
ServeArgs { view_only: false, monitor: MonitorSelection::All, max_fps: 30, log_level: LogLevel::Info }
);
}
+265
View File
@@ -0,0 +1,265 @@
//! Text clipboard between the operator and the device, over RDP's clipboard channel
//! (CLIPRDR). Copying on either side announces text to the other; the other side asks for
//! it when it pastes (the web client: when its tab regains focus). Text only (Unicode); no
//! files or images; at most [`MAX_TEXT_BYTES`] either way. Only in control sessions: a
//! view-only session has no clipboard channel at all.
use std::sync::{Arc, Mutex};
use ironrdp_cliprdr::backend::{ClipboardMessage, CliprdrBackend, CliprdrBackendFactory};
use ironrdp_cliprdr::pdu::{
ClipboardFormat, ClipboardFormatId, ClipboardGeneralCapabilityFlags, FileContentsRequest, FileContentsResponse,
FormatDataRequest, FormatDataResponse, LockDataId, OwnedFormatDataResponse,
};
use ironrdp_server::{CliprdrServerFactory, ServerEvent, ServerEventSender};
use tokio::sync::mpsc::UnboundedSender;
/// Text larger than this (UTF-8 bytes) is neither offered nor taken: a clipboard is for
/// snippets, files have their own tools (README, Security).
pub const MAX_TEXT_BYTES: usize = 1024 * 1024;
/// The device's clipboard, as the bridge needs it.
pub trait OsClipboard: Send + Sync {
/// The clipboard's text, if it holds any.
fn read_text(&self) -> Option<String>;
/// Replaces the clipboard with `text`. The change this makes is not reported back
/// through `watch`.
fn write_text(&self, text: &str);
/// Calls `changed` whenever something else changes the clipboard.
fn watch(&self, changed: Box<dyn Fn() + Send + Sync>);
}
/// What the server-side bridge shares with every backend it builds.
struct Shared {
os: Arc<dyn OsClipboard>,
sender: Mutex<Option<UnboundedSender<ServerEvent>>>,
}
impl Shared {
fn send(&self, message: ClipboardMessage) {
if let Some(sender) = self.sender.lock().ok().and_then(|s| s.clone()) {
let _ = sender.send(ServerEvent::Clipboard(message));
}
}
/// Tells the client the device has text to paste.
fn announce(&self) {
if self.os.read_text().is_some_and(|t| !t.is_empty() && t.len() <= MAX_TEXT_BYTES) {
self.send(ClipboardMessage::SendInitiateCopy(vec![ClipboardFormat::new(
ClipboardFormatId::CF_UNICODETEXT,
)]));
}
}
}
/// Builds the clipboard channel for a session on top of the device's clipboard.
pub struct ClipboardFactory {
shared: Arc<Shared>,
}
impl ClipboardFactory {
pub fn new(os: Arc<dyn OsClipboard>) -> Self {
let shared = Arc::new(Shared { os, sender: Mutex::new(None) });
let watcher = Arc::downgrade(&shared);
shared.os.watch(Box::new(move || {
if let Some(shared) = watcher.upgrade() {
tracing::debug!("device clipboard changed");
shared.announce();
}
}));
Self { shared }
}
}
impl CliprdrBackendFactory for ClipboardFactory {
fn build_cliprdr_backend(&self) -> Box<dyn CliprdrBackend> {
Box::new(ClipboardBackend { shared: Arc::clone(&self.shared) })
}
}
impl ServerEventSender for ClipboardFactory {
fn set_sender(&mut self, sender: UnboundedSender<ServerEvent>) {
if let Ok(mut slot) = self.shared.sender.lock() {
*slot = Some(sender);
}
}
}
impl CliprdrServerFactory for ClipboardFactory {}
struct ClipboardBackend {
shared: Arc<Shared>,
}
impl core::fmt::Debug for ClipboardBackend {
fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
f.write_str("ClipboardBackend")
}
}
ironrdp_core::impl_as_any!(ClipboardBackend);
impl CliprdrBackend for ClipboardBackend {
fn temporary_directory(&self) -> &str {
""
}
fn client_capabilities(&self) -> ClipboardGeneralCapabilityFlags {
// Text only: no file streams, no locking.
ClipboardGeneralCapabilityFlags::empty()
}
fn on_ready(&mut self) {
self.shared.announce();
}
fn on_request_format_list(&mut self) {
self.shared.announce();
}
fn on_process_negotiated_capabilities(&mut self, _capabilities: ClipboardGeneralCapabilityFlags) {}
fn on_remote_copy(&mut self, available_formats: &[ClipboardFormat]) {
// The operator copied: fetch the text now so it is on the device before they paste.
if available_formats.iter().any(|f| f.id == ClipboardFormatId::CF_UNICODETEXT) {
self.shared.send(ClipboardMessage::SendInitiatePaste(ClipboardFormatId::CF_UNICODETEXT));
}
}
fn on_format_data_request(&mut self, request: FormatDataRequest) {
let reply =
match (request.format == ClipboardFormatId::CF_UNICODETEXT).then(|| self.shared.os.read_text()).flatten() {
Some(text) if text.len() <= MAX_TEXT_BYTES => OwnedFormatDataResponse::new_unicode_string(&text),
_ => OwnedFormatDataResponse::new_error(),
};
self.shared.send(ClipboardMessage::SendFormatData(reply));
}
fn on_format_data_response(&mut self, response: FormatDataResponse<'_>) {
if response.is_error() {
return;
}
match response.to_unicode_string() {
Ok(text) if text.len() <= MAX_TEXT_BYTES => {
tracing::debug!(chars = text.chars().count(), "operator clipboard text placed on the device");
self.shared.os.write_text(&text);
}
Ok(_) => tracing::info!("operator clipboard text too large; not placed on the device"),
Err(e) => tracing::debug!(error = %e, "operator clipboard data was not text"),
}
}
fn on_file_contents_request(&mut self, request: FileContentsRequest) {
// No file copy: answer so the client does not wait.
self.shared
.send(ClipboardMessage::SendFileContentsResponse(FileContentsResponse::new_error(request.stream_id)));
}
fn on_file_contents_response(&mut self, _response: FileContentsResponse<'_>) {}
fn on_lock(&mut self, _data_id: LockDataId) {}
fn on_unlock(&mut self, _data_id: LockDataId) {}
}
#[cfg(test)]
mod tests {
use super::*;
use tokio::sync::mpsc;
#[derive(Default)]
struct FakeClipboard {
text: Mutex<Option<String>>,
watcher: Mutex<Option<Box<dyn Fn() + Send + Sync>>>,
}
impl FakeClipboard {
/// Something on the device copied.
fn local_copy(&self, text: &str) {
*self.text.lock().unwrap() = Some(text.to_owned());
if let Some(changed) = self.watcher.lock().unwrap().as_ref() {
changed();
}
}
}
impl OsClipboard for FakeClipboard {
fn read_text(&self) -> Option<String> {
self.text.lock().unwrap().clone()
}
fn write_text(&self, text: &str) {
*self.text.lock().unwrap() = Some(text.to_owned());
}
fn watch(&self, changed: Box<dyn Fn() + Send + Sync>) {
*self.watcher.lock().unwrap() = Some(changed);
}
}
fn setup() -> (Arc<FakeClipboard>, Box<dyn CliprdrBackend>, mpsc::UnboundedReceiver<ServerEvent>) {
let os = Arc::new(FakeClipboard::default());
let mut factory = ClipboardFactory::new(Arc::clone(&os) as Arc<dyn OsClipboard>);
let (tx, rx) = mpsc::unbounded_channel();
factory.set_sender(tx);
let backend = factory.build_cliprdr_backend();
// The factory must stay alive for the watcher (the server keeps it for the session).
std::mem::forget(factory);
(os, backend, rx)
}
fn next(rx: &mut mpsc::UnboundedReceiver<ServerEvent>) -> ClipboardMessage {
match rx.try_recv().expect("an event") {
ServerEvent::Clipboard(m) => m,
other => panic!("not a clipboard event: {other:?}"),
}
}
#[test]
fn device_copy_is_announced_and_served_as_text() {
let (os, mut backend, mut rx) = setup();
os.local_copy("héllo device");
assert!(
matches!(next(&mut rx), ClipboardMessage::SendInitiateCopy(f) if f[0].id == ClipboardFormatId::CF_UNICODETEXT)
);
backend.on_format_data_request(FormatDataRequest { format: ClipboardFormatId::CF_UNICODETEXT });
let ClipboardMessage::SendFormatData(reply) = next(&mut rx) else { panic!("no data") };
assert_eq!(reply.to_unicode_string().unwrap(), "héllo device");
}
#[test]
fn operator_copy_is_fetched_and_placed_on_the_device() {
let (os, mut backend, mut rx) = setup();
backend.on_remote_copy(&[ClipboardFormat::new(ClipboardFormatId::CF_UNICODETEXT)]);
assert!(
matches!(next(&mut rx), ClipboardMessage::SendInitiatePaste(id) if id == ClipboardFormatId::CF_UNICODETEXT)
);
let data = OwnedFormatDataResponse::new_unicode_string("from the operator");
backend.on_format_data_response(data);
assert_eq!(os.read_text().as_deref(), Some("from the operator"));
}
#[test]
fn text_over_the_cap_is_neither_offered_nor_taken() {
let (os, mut backend, mut rx) = setup();
let big = "x".repeat(MAX_TEXT_BYTES + 1);
os.local_copy(&big);
assert!(rx.try_recv().is_err(), "too large to offer");
backend.on_format_data_response(OwnedFormatDataResponse::new_unicode_string(&big));
assert_eq!(os.read_text().map(|t| t.len()), Some(MAX_TEXT_BYTES + 1), "the device keeps its own text");
}
#[test]
fn nothing_to_offer_and_other_formats_are_refused() {
let (_os, mut backend, mut rx) = setup();
backend.on_ready();
assert!(rx.try_recv().is_err(), "an empty clipboard is not announced");
backend.on_remote_copy(&[ClipboardFormat::new(ClipboardFormatId::new(2))]);
assert!(rx.try_recv().is_err(), "a bitmap is not fetched");
backend.on_format_data_request(FormatDataRequest { format: ClipboardFormatId::CF_UNICODETEXT });
let ClipboardMessage::SendFormatData(reply) = next(&mut rx) else { panic!("no reply") };
assert!(reply.is_error());
}
}
+3
View File
@@ -6,14 +6,17 @@
pub mod capture;
pub mod cli;
pub mod clipboard;
pub mod events;
pub mod geometry;
pub mod input;
pub mod keymap;
pub mod mux;
pub mod platform;
pub mod rdcleanpath;
pub mod session;
pub mod tls;
pub mod video;
/// Initialises stderr logging at the given level.
pub fn init_logging(level: cli::LogLevel) {
+25 -1
View File
@@ -11,6 +11,7 @@ use executer_screen::input::{CoordMap, GatedInput};
use executer_screen::platform::{self, SourceError};
use executer_screen::session::{self, SessionEnd};
use executer_screen::tls;
use executer_screen::video::{VideoControl, VideoPath};
const EXIT_OK: u8 = 0;
const EXIT_ERROR: u8 = 1;
@@ -90,6 +91,18 @@ fn serve(args: &ServeArgs) -> u8 {
};
let ctl = PumpControl::new(ironrdp_server_size_placeholder());
// H.264 when this device has an encoder (none on Windows PE): offered to the browser, used if it asks.
let (video_control, video_path) = match platform::video_encoders() {
Some(factory) => {
let control = Arc::new(VideoControl::default());
let path = VideoPath::new(Arc::clone(&control), factory, args.max_fps);
(Some(control), Some(path))
}
None => {
tracing::info!("no H.264 encoder: the screen goes as RDP bitmaps");
(None, None)
}
};
let coords = CoordMap::default();
let (tx, rx) = tokio::sync::mpsc::channel(2);
@@ -99,7 +112,7 @@ fn serve(args: &ServeArgs) -> u8 {
let ctl = Arc::clone(&ctl);
let coords = coords.clone();
let view_only = args.view_only;
let cfg = PumpConfig { max_fps: args.max_fps };
let cfg = PumpConfig { max_fps: args.max_fps, video: video_path };
std::thread::Builder::new().name("capture".into()).spawn(move || {
capture::run_pump(
source,
@@ -123,6 +136,15 @@ fn serve(args: &ServeArgs) -> u8 {
let input = Box::new(GatedInput::new(args.view_only, coords, platform::create_input_sink()));
let display = HostDisplay::new(Arc::clone(&ctl), rx);
// Control sessions only: a view-only session shares no clipboard either way.
let clipboard: Option<Box<dyn ironrdp_server::CliprdrServerFactory>> = if args.view_only {
None
} else {
platform::create_clipboard().map(|os| {
Box::new(executer_screen::clipboard::ClipboardFactory::new(os))
as Box<dyn ironrdp_server::CliprdrServerFactory>
})
};
let runtime = match tokio::runtime::Builder::new_current_thread().enable_all().build() {
Ok(rt) => rt,
@@ -136,6 +158,8 @@ fn serve(args: &ServeArgs) -> u8 {
display,
Arc::clone(&ctl),
input,
clipboard,
video_control,
));
ctl.stop.store(true, Ordering::Relaxed);
+255
View File
@@ -0,0 +1,255 @@
//! Framing of stdin/stdout when the session carries H.264 video beside RDP.
//!
//! The browser asks for video in the RDCleanPath request's `destination` (`screen;video=h264`).
//! Hosts that do not know the hint ignore it (the field is opaque) and speak plain
//! RDCleanPath + RDP, so a new console works with an old host. A host that can encode H.264
//! answers with [`MAGIC`] before anything else; from then on both directions are frames of
//! `[channel u8][length u32 LE][payload]`: channel [`RDP`] carries the bytes the RDP client and
//! server exchange (starting with the RDCleanPath response), channel [`VIDEO`] the video
//! messages (`crate::video`). The relay (agent, server) passes the bytes through unchanged.
use std::io;
use bytes::{Buf as _, Bytes, BytesMut};
use tokio::io::{AsyncRead, AsyncReadExt as _, AsyncWrite, AsyncWriteExt as _, DuplexStream};
use tokio::sync::mpsc;
/// What the host writes first when it accepts video.
pub const MAGIC: &[u8; 8] = b"EXRMUX01";
pub const RDP: u8 = 0;
pub const VIDEO: u8 = 1;
const HEADER: usize = 5;
/// The largest frame the browser may send (RDP input and video acknowledgements are small).
pub const MAX_INBOUND: usize = 1 << 20;
/// Bytes buffered between the RDP server and the framing tasks, each way.
const PIPE: usize = 256 * 1024;
/// Whether the RDCleanPath `destination` asks for H.264 video (`<name>;video=h264`).
pub fn wants_video(destination: Option<&str>) -> bool {
destination.is_some_and(|d| d.split(';').skip(1).any(|p| p.trim().eq_ignore_ascii_case("video=h264")))
}
/// One frame: header and payload.
pub fn frame(channel: u8, payload: &[u8]) -> Vec<u8> {
let mut out = Vec::with_capacity(HEADER + payload.len());
out.push(channel);
out.extend_from_slice(&u32::try_from(payload.len()).unwrap_or(u32::MAX).to_le_bytes());
out.extend_from_slice(payload);
out
}
/// Splits the browser's bytes into frames (they arrive in any chunking).
#[derive(Default)]
pub struct Deframer {
buf: BytesMut,
}
impl Deframer {
pub fn push(&mut self, data: &[u8]) {
self.buf.extend_from_slice(data);
}
/// The next complete frame, if there is one.
pub fn next_frame(&mut self) -> io::Result<Option<(u8, Bytes)>> {
if self.buf.len() < HEADER {
return Ok(None);
}
let channel = self.buf[0];
let len = u32::from_le_bytes([self.buf[1], self.buf[2], self.buf[3], self.buf[4]]) as usize;
if channel > VIDEO || len > MAX_INBOUND {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
format!("bad frame (channel {channel}, {len} bytes)"),
));
}
if self.buf.len() < HEADER + len {
return Ok(None);
}
self.buf.advance(HEADER);
Ok(Some((channel, self.buf.split_to(len).freeze())))
}
}
/// Video messages from the browser.
pub type VideoHandler = Box<dyn Fn(&[u8]) + Send>;
/// Sends frames to the browser; usable from any thread.
#[derive(Clone)]
pub struct Outbound(mpsc::Sender<(u8, Bytes)>);
impl Outbound {
/// Queues a video message, waiting while the queue is full. False once stdout is gone.
pub fn send_video(&self, payload: Vec<u8>) -> bool {
self.0.blocking_send((VIDEO, Bytes::from(payload))).is_ok()
}
#[cfg(test)]
pub fn for_tests(sender: mpsc::Sender<(u8, Bytes)>) -> Self {
Self(sender)
}
}
/// The RDP server's side of a framed stream, plus the way out for video.
pub struct Muxed {
pub reader: DuplexStream,
pub writer: DuplexStream,
pub out: Outbound,
}
/// Starts framing `reader`/`writer` (stdin/stdout) on the current runtime. `leftover` is what
/// the browser sent after its RDCleanPath request (already framed). Video messages from the
/// browser go to `on_video`. The returned reader ends when stdin does (or sends a bad frame);
/// the writer fails with a broken pipe once stdout is gone.
pub fn start<R, W>(reader: R, writer: W, leftover: Vec<u8>, on_video: VideoHandler) -> Muxed
where
R: AsyncRead + Unpin + Send + 'static,
W: AsyncWrite + Unpin + Send + 'static,
{
let (queue, outgoing) = mpsc::channel::<(u8, Bytes)>(64);
let (rdp_reader, mut to_rdp) = tokio::io::duplex(PIPE);
let (rdp_writer, mut from_rdp) = tokio::io::duplex(PIPE);
tokio::spawn(write_frames(writer, outgoing));
let rdp_out = queue.clone();
tokio::spawn(async move {
let mut buf = vec![0u8; 64 * 1024];
loop {
match from_rdp.read(&mut buf).await {
Ok(0) | Err(_) => return,
Ok(n) => {
if rdp_out.send((RDP, Bytes::copy_from_slice(&buf[..n]))).await.is_err() {
return;
}
}
}
}
});
tokio::spawn(async move {
let mut reader = reader;
let mut deframer = Deframer::default();
deframer.push(&leftover);
let mut buf = vec![0u8; 64 * 1024];
loop {
loop {
match deframer.next_frame() {
Ok(Some((RDP, payload))) => {
if to_rdp.write_all(&payload).await.is_err() {
return;
}
}
Ok(Some((_, payload))) => on_video(&payload),
Ok(None) => break,
Err(e) => {
tracing::warn!(error = %e, "the browser's stream is not framed; ending the session");
return;
}
}
}
match reader.read(&mut buf).await {
Ok(0) | Err(_) => return,
Ok(n) => deframer.push(&buf[..n]),
}
}
});
Muxed { reader: rdp_reader, writer: rdp_writer, out: Outbound(queue) }
}
async fn write_frames<W: AsyncWrite + Unpin>(mut writer: W, mut outgoing: mpsc::Receiver<(u8, Bytes)>) {
if writer.write_all(MAGIC).await.is_err() {
return;
}
while let Some((channel, payload)) = outgoing.recv().await {
let mut chunk = frame(channel, &payload);
// Whatever else is already queued goes in the same write.
while chunk.len() < 256 * 1024 {
match outgoing.try_recv() {
Ok((c, p)) => chunk.extend_from_slice(&frame(c, &p)),
Err(_) => break,
}
}
if writer.write_all(&chunk).await.is_err() || writer.flush().await.is_err() {
return;
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn the_hint_is_a_destination_parameter() {
assert!(wants_video(Some("screen;video=h264")));
assert!(wants_video(Some("screen; VIDEO=H264")));
assert!(!wants_video(Some("screen")));
assert!(!wants_video(Some("video=h264")));
assert!(!wants_video(None));
}
#[test]
fn frames_survive_any_chunking() {
let mut bytes = frame(RDP, b"hello");
bytes.extend(frame(VIDEO, &[1, 2, 3]));
bytes.extend(frame(RDP, b""));
let mut d = Deframer::default();
let mut got = Vec::new();
for b in bytes {
d.push(&[b]);
while let Some(f) = d.next_frame().unwrap() {
got.push(f);
}
}
assert_eq!(
got,
vec![(RDP, Bytes::from_static(b"hello")), (VIDEO, Bytes::from_static(&[1, 2, 3])), (RDP, Bytes::new())]
);
}
#[test]
fn bad_frames_are_refused() {
let mut d = Deframer::default();
d.push(&[7, 0, 0, 0, 0]);
assert!(d.next_frame().is_err());
let mut d = Deframer::default();
d.push(&frame(RDP, &[])[..1]);
d.push(&u32::try_from(MAX_INBOUND + 1).unwrap().to_le_bytes());
assert!(d.next_frame().is_err());
}
#[tokio::test(flavor = "current_thread")]
async fn rdp_and_video_share_the_stream() {
let (host_in, mut browser_out) = tokio::io::duplex(1 << 16);
let (mut browser_in, host_out) = tokio::io::duplex(1 << 16);
let seen = std::sync::Arc::new(std::sync::Mutex::new(Vec::new()));
let sink = std::sync::Arc::clone(&seen);
let mut m = start(
host_in,
host_out,
frame(RDP, b"early"),
Box::new(move |p| sink.lock().unwrap().extend_from_slice(p)),
);
browser_out.write_all(&frame(VIDEO, b"ack")).await.unwrap();
browser_out.write_all(&frame(RDP, b"-input")).await.unwrap();
let mut got = [0u8; 11];
m.reader.read_exact(&mut got).await.unwrap();
assert_eq!(&got, b"early-input");
assert_eq!(seen.lock().unwrap().as_slice(), b"ack");
m.writer.write_all(b"rdp-out").await.unwrap();
let mut head = [0u8; 8];
browser_in.read_exact(&mut head).await.unwrap();
assert_eq!(&head, MAGIC);
let mut f = vec![0u8; HEADER + 7];
browser_in.read_exact(&mut f).await.unwrap();
assert_eq!(f, frame(RDP, b"rdp-out"));
drop(browser_out);
let mut rest = Vec::new();
m.reader.read_to_end(&mut rest).await.unwrap();
assert!(rest.is_empty(), "stdin EOF ends the RDP reader");
}
}
+29
View File
@@ -66,6 +66,35 @@ pub fn probe() -> ProbeReport {
imp::probe()
}
/// The device clipboard for Remote View's clipboard channel; None where there is none yet
/// (Linux, macOS).
pub fn create_clipboard() -> Option<std::sync::Arc<dyn crate::clipboard::OsClipboard>> {
#[cfg(windows)]
{
imp::create_clipboard()
}
#[cfg(not(windows))]
{
None
}
}
/// H.264 encoders for Remote View video, when this device has one (Windows with Media
/// Foundation; none on Windows PE, Server Core, Linux, macOS).
pub fn video_encoders() -> Option<crate::video::EncoderFactory> {
#[cfg(windows)]
{
imp::h264::available().then(|| {
Box::new(|w: u32, h: u32, fps: u32, hw: bool| imp::h264::create(w, h, fps, hw))
as crate::video::EncoderFactory
})
}
#[cfg(not(windows))]
{
None
}
}
/// The served region for a selection: the union of all monitors, or one monitor.
pub fn select_bounds(monitors: &[MonitorInfo], selection: MonitorSelection) -> Option<Rect> {
match selection {
@@ -0,0 +1,204 @@
//! The device's clipboard (Unicode text) for the Remote View clipboard channel.
//!
//! The host runs as SYSTEM on the user's desktop, so it keeps no window and no message pump
//! (README, Security). Changes are found by polling the clipboard's sequence number. A write
//! needs a window to own the clipboard after `EmptyClipboard`, so each write makes a
//! message-only window, places the text and destroys the window straight away; the text stays
//! on the clipboard (it is not delay-rendered). The clipboard belongs to the window station
//! (winsta0), so it is the same on the user's desktop, the lock screen and UAC prompts.
use std::sync::mpsc::{RecvTimeoutError, Sender, channel};
use std::sync::{Arc, Mutex};
use std::time::Duration;
use windows::Win32::Foundation::{GlobalFree, HANDLE, HGLOBAL, HWND, LPARAM, LRESULT, WPARAM};
use windows::Win32::System::DataExchange::{
CloseClipboard, EmptyClipboard, GetClipboardData, GetClipboardSequenceNumber, IsClipboardFormatAvailable,
OpenClipboard, SetClipboardData,
};
use windows::Win32::System::LibraryLoader::GetModuleHandleW;
use windows::Win32::System::Memory::{GMEM_MOVEABLE, GlobalAlloc, GlobalLock, GlobalSize, GlobalUnlock};
use windows::Win32::UI::WindowsAndMessaging::{
CreateWindowExW, DefWindowProcW, DestroyWindow, HWND_MESSAGE, RegisterClassW, WINDOW_EX_STYLE, WINDOW_STYLE,
WNDCLASSW,
};
use windows::core::{PCWSTR, w};
use crate::clipboard::OsClipboard;
/// CF_UNICODETEXT.
const CF_UNICODETEXT: u32 = 13;
/// How often the clipboard's sequence number is checked.
const POLL: Duration = Duration::from_millis(250);
const CLASS: PCWSTR = w!("ExecuteRScreenClipboardOwner");
type Watcher = Box<dyn Fn() + Send + Sync>;
pub struct WindowsClipboard {
writes: Sender<String>,
watcher: Arc<Mutex<Option<Watcher>>>,
}
impl WindowsClipboard {
/// Starts the clipboard thread (it ends with the process).
pub fn start() -> Option<Arc<Self>> {
let (writes, queue) = channel::<String>();
let watcher: Arc<Mutex<Option<Watcher>>> = Arc::new(Mutex::new(None));
let notify = Arc::clone(&watcher);
std::thread::Builder::new()
.name("clipboard".into())
.spawn(move || {
register_class();
// SAFETY: plain Win32 call.
let mut seen = unsafe { GetClipboardSequenceNumber() };
loop {
match queue.recv_timeout(POLL) {
Ok(text) => {
// Only the newest of several queued writes matters.
let text = queue.try_iter().last().unwrap_or(text);
if write_now(&text) {
// Our own change is not reported back.
// SAFETY: plain Win32 call.
seen = unsafe { GetClipboardSequenceNumber() };
}
}
Err(RecvTimeoutError::Timeout) => {}
Err(RecvTimeoutError::Disconnected) => return,
}
// SAFETY: plain Win32 call.
let now = unsafe { GetClipboardSequenceNumber() };
if now != seen {
seen = now;
if let Some(changed) = notify.lock().ok().as_ref().and_then(|w| w.as_ref()) {
changed();
}
}
}
})
.ok()?;
Some(Arc::new(Self { writes, watcher }))
}
}
unsafe extern "system" fn owner_proc(hwnd: HWND, msg: u32, wparam: WPARAM, lparam: LPARAM) -> LRESULT {
// SAFETY: default handling; the window lives only for one write and is never pumped.
unsafe { DefWindowProcW(hwnd, msg, wparam, lparam) }
}
fn register_class() {
// SAFETY: registering a window class with a static name and a default procedure.
unsafe {
let instance = GetModuleHandleW(None).ok();
let class = WNDCLASSW {
lpfnWndProc: Some(owner_proc),
hInstance: instance.map(Into::into).unwrap_or_default(),
lpszClassName: CLASS,
..Default::default()
};
RegisterClassW(&class);
}
}
/// Opens the clipboard, retrying briefly while another program holds it.
fn open(owner: Option<HWND>) -> bool {
for _ in 0..10 {
// SAFETY: plain Win32 call; closed by the caller.
if unsafe { OpenClipboard(owner) }.is_ok() {
return true;
}
std::thread::sleep(Duration::from_millis(15));
}
false
}
/// Places `text` on the clipboard through a window made and destroyed for this one write.
fn write_now(text: &str) -> bool {
let wide: Vec<u16> = text.encode_utf16().chain(Some(0)).collect();
// SAFETY: the window is created and destroyed on this thread; the clipboard is opened
// and closed here; the global block goes to the clipboard on success and is freed by us
// otherwise.
unsafe {
let instance = GetModuleHandleW(None).ok();
let Ok(hwnd) = CreateWindowExW(
WINDOW_EX_STYLE(0),
CLASS,
PCWSTR::null(),
WINDOW_STYLE(0),
0,
0,
0,
0,
Some(HWND_MESSAGE),
None,
instance.map(Into::into),
None,
) else {
tracing::info!("the operator's text could not be placed on the device clipboard (no owner window)");
return false;
};
let placed = if open(Some(hwnd)) {
let _ = EmptyClipboard();
let result = (|| -> windows::core::Result<()> {
let block = GlobalAlloc(GMEM_MOVEABLE, wide.len() * 2)?;
let target = GlobalLock(block).cast::<u16>();
if target.is_null() {
let _ = GlobalFree(Some(block));
return Err(windows::core::Error::from_thread());
}
core::ptr::copy_nonoverlapping(wide.as_ptr(), target, wide.len());
let _ = GlobalUnlock(block);
if let Err(e) = SetClipboardData(CF_UNICODETEXT, Some(HANDLE(block.0))) {
let _ = GlobalFree(Some(block));
return Err(e);
}
Ok(())
})();
let _ = CloseClipboard();
if let Err(e) = &result {
tracing::info!(error = %e, "the operator's text could not be placed on the device clipboard");
}
result.is_ok()
} else {
tracing::info!("the device clipboard is busy; the operator's text was not placed");
false
};
let _ = DestroyWindow(hwnd);
placed
}
}
impl OsClipboard for WindowsClipboard {
fn read_text(&self) -> Option<String> {
// SAFETY: plain Win32 calls; the clipboard is closed before returning and the data is
// copied out while locked, within the block's size.
unsafe {
if IsClipboardFormatAvailable(CF_UNICODETEXT).is_err() || !open(None) {
return None;
}
let text = GetClipboardData(CF_UNICODETEXT).ok().and_then(|handle| {
let block = HGLOBAL(handle.0);
let data = GlobalLock(block).cast::<u16>();
if data.is_null() {
return None;
}
let max = GlobalSize(block) / 2;
let len = (0..max).find(|&i| *data.add(i) == 0).unwrap_or(max);
let text = String::from_utf16_lossy(core::slice::from_raw_parts(data, len));
let _ = GlobalUnlock(block);
Some(text)
});
let _ = CloseClipboard();
text
}
}
fn write_text(&self, text: &str) {
let _ = self.writes.send(text.to_owned());
}
fn watch(&self, changed: Watcher) {
if let Ok(mut slot) = self.watcher.lock() {
*slot = Some(changed);
}
}
}
+601
View File
@@ -0,0 +1,601 @@
//! H.264 through Media Foundation: the GPU's encoder when there is one (an asynchronous
//! hardware MFT: NVIDIA, Intel Quick Sync, AMD), else Microsoft's software encoder.
//!
//! Media Foundation is loaded at run time (mfplat.dll, ole32.dll from System32): Windows PE
//! and Server Core have none, and the executable must still start there (README, Platforms).
//! Constrained to what the browser decodes with the least delay: Baseline profile (no
//! reordering), no B-frames, low-latency mode, constant bit rate scaled to the frame size.
use core::ffi::c_void;
use core::mem::ManuallyDrop;
use std::sync::OnceLock;
use std::time::{Duration, Instant};
use anyhow::{Context as _, Result, anyhow, bail};
use windows::Win32::Media::MediaFoundation::{
CODECAPI_AVEncCommonMeanBitRate, CODECAPI_AVEncCommonQualityVsSpeed, CODECAPI_AVEncCommonRateControlMode,
CODECAPI_AVEncMPVDefaultBPictureCount, CODECAPI_AVEncMPVGOPSize, CODECAPI_AVEncVideoForceKeyFrame,
CODECAPI_AVLowLatencyMode, ICodecAPI, IMFActivate, IMFMediaBuffer, IMFMediaEventGenerator, IMFMediaType, IMFSample,
IMFTransform, METransformHaveOutput, METransformNeedInput, MF_E_NO_EVENTS_AVAILABLE, MF_E_NOTACCEPTING,
MF_E_TRANSFORM_NEED_MORE_INPUT, MF_E_TRANSFORM_STREAM_CHANGE, MF_EVENT_FLAG_NO_WAIT, MF_LOW_LATENCY,
MF_MT_AVG_BITRATE, MF_MT_FRAME_RATE, MF_MT_FRAME_SIZE, MF_MT_INTERLACE_MODE, MF_MT_MAJOR_TYPE,
MF_MT_MPEG_SEQUENCE_HEADER, MF_MT_MPEG2_PROFILE, MF_MT_PIXEL_ASPECT_RATIO, MF_MT_SUBTYPE, MF_TRANSFORM_ASYNC,
MF_TRANSFORM_ASYNC_UNLOCK, MF_VERSION, MFMediaType_Video, MFSTARTUP_NOSOCKET, MFSampleExtension_CleanPoint,
MFT_CATEGORY_VIDEO_ENCODER, MFT_ENUM_FLAG_HARDWARE, MFT_ENUM_FLAG_LOCALMFT, MFT_ENUM_FLAG_SORTANDFILTER,
MFT_ENUM_FLAG_SYNCMFT, MFT_FRIENDLY_NAME_Attribute, MFT_MESSAGE_NOTIFY_BEGIN_STREAMING,
MFT_MESSAGE_NOTIFY_START_OF_STREAM, MFT_OUTPUT_DATA_BUFFER, MFT_OUTPUT_STREAM_CAN_PROVIDE_SAMPLES,
MFT_OUTPUT_STREAM_PROVIDES_SAMPLES, MFT_REGISTER_TYPE_INFO, MFVideoFormat_H264, MFVideoFormat_NV12,
MFVideoInterlace_Progressive, eAVEncCommonRateControlMode_CBR, eAVEncH264VProfile_Base,
};
use windows::Win32::System::LibraryLoader::{GetProcAddress, LOAD_LIBRARY_SEARCH_SYSTEM32, LoadLibraryExW};
use windows::Win32::System::Variant::VARIANT;
use windows::core::{GUID, HRESULT, Interface, PWSTR, s, w};
use crate::video::{EncodedFrame, VideoEncoder};
type MfStartupFn = unsafe extern "system" fn(u32, u32) -> HRESULT;
type MftEnumExFn = unsafe extern "system" fn(
GUID,
u32,
*const MFT_REGISTER_TYPE_INFO,
*const MFT_REGISTER_TYPE_INFO,
*mut *mut *mut c_void,
*mut u32,
) -> HRESULT;
type MfCreateFn = unsafe extern "system" fn(*mut *mut c_void) -> HRESULT;
type MfCreateMemoryBufferFn = unsafe extern "system" fn(u32, *mut *mut c_void) -> HRESULT;
type CoInitializeExFn = unsafe extern "system" fn(*const c_void, u32) -> HRESULT;
type CoTaskMemFreeFn = unsafe extern "system" fn(*const c_void);
/// The Media Foundation entry points, loaded once.
struct Api {
enum_ex: MftEnumExFn,
create_sample: MfCreateFn,
create_media_type: MfCreateFn,
create_memory_buffer: MfCreateMemoryBufferFn,
co_initialize: CoInitializeExFn,
co_free: CoTaskMemFreeFn,
}
// SAFETY: plain function pointers into system DLLs that stay loaded.
unsafe impl Send for Api {}
// SAFETY: as above.
unsafe impl Sync for Api {}
fn api() -> Option<&'static Api> {
static API: OnceLock<Option<Api>> = OnceLock::new();
API.get_or_init(|| match load() {
Ok(api) => Some(api),
Err(e) => {
tracing::info!(error = format!("{e:#}"), "no Media Foundation: no H.264 video");
None
}
})
.as_ref()
}
fn load() -> Result<Api> {
// SAFETY: System32 DLLs only; each symbol has the documented signature declared above.
unsafe {
let mf = LoadLibraryExW(w!("mfplat.dll"), None, LOAD_LIBRARY_SEARCH_SYSTEM32).context("mfplat.dll")?;
let ole = LoadLibraryExW(w!("ole32.dll"), None, LOAD_LIBRARY_SEARCH_SYSTEM32).context("ole32.dll")?;
macro_rules! symbol {
($module:expr, $name:literal, $ty:ty) => {
core::mem::transmute::<unsafe extern "system" fn() -> isize, $ty>(
GetProcAddress($module, s!($name)).ok_or_else(|| anyhow!(concat!($name, " missing")))?,
)
};
}
let startup = symbol!(mf, "MFStartup", MfStartupFn);
let api = Api {
enum_ex: symbol!(mf, "MFTEnumEx", MftEnumExFn),
create_sample: symbol!(mf, "MFCreateSample", MfCreateFn),
create_media_type: symbol!(mf, "MFCreateMediaType", MfCreateFn),
create_memory_buffer: symbol!(mf, "MFCreateMemoryBuffer", MfCreateMemoryBufferFn),
co_initialize: symbol!(ole, "CoInitializeEx", CoInitializeExFn),
co_free: symbol!(ole, "CoTaskMemFree", CoTaskMemFreeFn),
};
com(&api);
startup(MF_VERSION, MFSTARTUP_NOSOCKET).ok().context("MFStartup")?;
Ok(api)
}
}
/// COM for this thread (multithreaded apartment), once per thread.
fn com(api: &Api) {
thread_local!(static DONE: core::cell::Cell<bool> = const { core::cell::Cell::new(false) });
DONE.with(|done| {
if !done.replace(true) {
// COINIT_MULTITHREADED = 0. S_FALSE (already initialised) is fine.
// SAFETY: documented signature; never uninitialised (the thread ends with the process).
let _ = unsafe { (api.co_initialize)(core::ptr::null(), 0) };
}
});
}
/// The registered H.264 encoders that take NV12, best first.
fn encoders(api: &Api, hardware: bool) -> Vec<IMFActivate> {
com(api);
let flags = if hardware {
MFT_ENUM_FLAG_HARDWARE.0 | MFT_ENUM_FLAG_SORTANDFILTER.0
} else {
MFT_ENUM_FLAG_SYNCMFT.0 | MFT_ENUM_FLAG_LOCALMFT.0 | MFT_ENUM_FLAG_SORTANDFILTER.0
};
let input = MFT_REGISTER_TYPE_INFO { guidMajorType: MFMediaType_Video, guidSubtype: MFVideoFormat_NV12 };
let output = MFT_REGISTER_TYPE_INFO { guidMajorType: MFMediaType_Video, guidSubtype: MFVideoFormat_H264 };
let mut list: *mut *mut c_void = core::ptr::null_mut();
let mut count = 0u32;
// SAFETY: documented signature; the array and each reference it holds become ours.
unsafe {
if (api.enum_ex)(MFT_CATEGORY_VIDEO_ENCODER, flags as u32, &input, &output, &mut list, &mut count).is_err()
|| list.is_null()
{
return Vec::new();
}
let found = (0..count as usize)
.map(|i| *list.add(i))
.filter(|p| !p.is_null())
.map(|p| IMFActivate::from_raw(p))
.collect();
(api.co_free)(list.cast());
found
}
}
/// Whether an H.264 encoder is there at all (the browser is offered video only then).
pub fn available() -> bool {
api().is_some_and(|api| !encoders(api, true).is_empty() || !encoders(api, false).is_empty())
}
/// An encoder for `width` x `height` (even) at `fps`: hardware first when allowed.
pub fn create(width: u32, height: u32, fps: u32, hardware: bool) -> Result<Box<dyn VideoEncoder>> {
let api = api().ok_or_else(|| anyhow!("Media Foundation is not available"))?;
let kinds: &[bool] = if hardware { &[true, false] } else { &[false] };
let mut tried = Vec::new();
for &hw in kinds {
for activate in encoders(api, hw) {
let name = friendly_name(api, &activate).unwrap_or_else(|| "H.264 encoder".into());
match MfEncoder::new(api, activate, name.clone(), hw, width, height, fps) {
Ok(encoder) => return Ok(Box::new(encoder)),
Err(e) => {
tracing::info!(encoder = name, error = format!("{e:#}"), "H.264 encoder not usable");
tried.push(name);
}
}
}
}
if tried.is_empty() {
bail!("no H.264 encoder is installed")
} else {
bail!("no H.264 encoder took {width}x{height} (tried {})", tried.join(", "))
}
}
fn friendly_name(api: &Api, activate: &IMFActivate) -> Option<String> {
let mut text = PWSTR::null();
let mut len = 0u32;
// SAFETY: the string is allocated by the callee and freed here.
unsafe {
activate.GetAllocatedString(&MFT_FRIENDLY_NAME_Attribute, &mut text, &mut len).ok()?;
let name = text.to_string().ok();
(api.co_free)(text.0.cast());
name
}
}
/// Bits per second for a frame size: about 0.1 bit per pixel per frame, 1.5-12 Mbit/s. Only
/// frames that change are encoded, so a still screen costs next to nothing.
pub fn bitrate(width: u32, height: u32, fps: u32) -> u32 {
let bits = u64::from(width) * u64::from(height) * u64::from(fps) / 10;
u32::try_from(bits.clamp(1_500_000, 12_000_000)).unwrap_or(12_000_000)
}
struct MfEncoder {
api: &'static Api,
activate: IMFActivate,
transform: IMFTransform,
codec: Option<ICodecAPI>,
events: Option<IMFMediaEventGenerator>,
name: String,
hardware: bool,
input_id: u32,
output_id: u32,
sequence_header: Vec<u8>,
frame_duration: i64,
need_input: u32,
output_capacity: u32,
}
// SAFETY: made, used and dropped on the capture thread (a COM multithreaded apartment); the
// pump is moved to that thread before any encoder exists.
unsafe impl Send for MfEncoder {}
impl MfEncoder {
fn new(
api: &'static Api,
activate: IMFActivate,
name: String,
hardware: bool,
width: u32,
height: u32,
fps: u32,
) -> Result<Self> {
// SAFETY: COM calls on interfaces this thread owns; arguments are valid for each call.
unsafe {
let transform: IMFTransform = activate.ActivateObject().context("activate")?;
let mut encoder = Self {
api,
activate,
codec: transform.cast().ok(),
transform,
events: None,
name,
hardware,
input_id: 0,
output_id: 0,
sequence_header: Vec::new(),
frame_duration: 10_000_000 / i64::from(fps.max(1)),
need_input: 0,
output_capacity: width * height * 3 / 2,
};
encoder.configure(width, height, fps)?;
Ok(encoder)
}
}
unsafe fn configure(&mut self, width: u32, height: u32, fps: u32) -> Result<()> {
// SAFETY: as in `new`.
unsafe {
let t = self.transform.clone();
if let Ok(attributes) = t.GetAttributes() {
if attributes.GetUINT32(&MF_TRANSFORM_ASYNC).unwrap_or(0) != 0 {
attributes.SetUINT32(&MF_TRANSFORM_ASYNC_UNLOCK, 1).context("unlock async MFT")?;
self.events = Some(t.cast().context("async MFT without events")?);
}
let _ = attributes.SetUINT32(&MF_LOW_LATENCY, 1);
}
let (mut inputs, mut outputs) = ([0u32], [0u32]);
if t.GetStreamIDs(&mut inputs, &mut outputs).is_ok() {
(self.input_id, self.output_id) = (inputs[0], outputs[0]);
}
let rate = bitrate(width, height, fps);
if let Some(codec) = &self.codec {
// Best effort: an encoder that lacks a setting keeps its default.
let set = |api: &GUID, value: VARIANT| {
let _ = codec.SetValue(api, &value);
};
set(&CODECAPI_AVEncCommonRateControlMode, VARIANT::from(eAVEncCommonRateControlMode_CBR.0 as u32));
set(&CODECAPI_AVEncCommonMeanBitRate, VARIANT::from(rate));
set(&CODECAPI_AVLowLatencyMode, VARIANT::from(true));
set(&CODECAPI_AVEncMPVDefaultBPictureCount, VARIANT::from(0u32));
set(&CODECAPI_AVEncMPVGOPSize, VARIANT::from(fps.max(1) * 60));
set(&CODECAPI_AVEncCommonQualityVsSpeed, VARIANT::from(if self.hardware { 50u32 } else { 20u32 }));
}
let size = (u64::from(width) << 32) | u64::from(height);
let frame_rate = (u64::from(fps.max(1)) << 32) | 1;
let output = self.media_type()?;
output.SetGUID(&MF_MT_MAJOR_TYPE, &MFMediaType_Video)?;
output.SetGUID(&MF_MT_SUBTYPE, &MFVideoFormat_H264)?;
output.SetUINT32(&MF_MT_AVG_BITRATE, rate)?;
output.SetUINT64(&MF_MT_FRAME_SIZE, size)?;
output.SetUINT64(&MF_MT_FRAME_RATE, frame_rate)?;
output.SetUINT64(&MF_MT_PIXEL_ASPECT_RATIO, (1 << 32) | 1)?;
output.SetUINT32(&MF_MT_INTERLACE_MODE, MFVideoInterlace_Progressive.0 as u32)?;
output.SetUINT32(&MF_MT_MPEG2_PROFILE, eAVEncH264VProfile_Base.0 as u32)?;
t.SetOutputType(self.output_id, &output, 0).context("output type")?;
let input = self.media_type()?;
input.SetGUID(&MF_MT_MAJOR_TYPE, &MFMediaType_Video)?;
input.SetGUID(&MF_MT_SUBTYPE, &MFVideoFormat_NV12)?;
input.SetUINT64(&MF_MT_FRAME_SIZE, size)?;
input.SetUINT64(&MF_MT_FRAME_RATE, frame_rate)?;
input.SetUINT64(&MF_MT_PIXEL_ASPECT_RATIO, (1 << 32) | 1)?;
input.SetUINT32(&MF_MT_INTERLACE_MODE, MFVideoInterlace_Progressive.0 as u32)?;
t.SetInputType(self.input_id, &input, 0).context("input type")?;
self.read_sequence_header();
t.ProcessMessage(MFT_MESSAGE_NOTIFY_BEGIN_STREAMING, 0).context("begin streaming")?;
t.ProcessMessage(MFT_MESSAGE_NOTIFY_START_OF_STREAM, 0).context("start of stream")?;
Ok(())
}
}
fn media_type(&self) -> Result<IMFMediaType> {
let mut raw = core::ptr::null_mut();
// SAFETY: documented signature; the reference becomes ours.
unsafe {
(self.api.create_media_type)(&mut raw).ok().context("MFCreateMediaType")?;
Ok(IMFMediaType::from_raw(raw))
}
}
/// SPS/PPS from the output type, for key frames that come without them.
fn read_sequence_header(&mut self) {
// SAFETY: COM calls; the blob is copied into a buffer of its reported size.
unsafe {
if let Ok(current) = self.transform.GetOutputCurrentType(self.output_id)
&& let Ok(size) = current.GetBlobSize(&MF_MT_MPEG_SEQUENCE_HEADER)
{
let mut blob = vec![0u8; size as usize];
if current.GetBlob(&MF_MT_MPEG_SEQUENCE_HEADER, &mut blob, None).is_ok() {
self.sequence_header = blob;
}
}
}
}
fn sample(&self, data: Option<&[u8]>, capacity: u32) -> Result<IMFSample> {
// SAFETY: documented signatures; the buffer is locked only while copying into it.
unsafe {
let mut raw = core::ptr::null_mut();
(self.api.create_memory_buffer)(capacity, &mut raw).ok().context("MFCreateMemoryBuffer")?;
let buffer = IMFMediaBuffer::from_raw(raw);
if let Some(data) = data {
let mut target = core::ptr::null_mut();
buffer.Lock(&mut target, None, None)?;
core::ptr::copy_nonoverlapping(data.as_ptr(), target, data.len());
buffer.Unlock()?;
buffer.SetCurrentLength(u32::try_from(data.len())?)?;
}
let mut raw = core::ptr::null_mut();
(self.api.create_sample)(&mut raw).ok().context("MFCreateSample")?;
let sample = IMFSample::from_raw(raw);
sample.AddBuffer(&buffer)?;
Ok(sample)
}
}
/// One ProcessOutput: Ok(None) when the encoder needs more input.
fn pull(&mut self) -> Result<Option<EncodedFrame>> {
// SAFETY: COM calls; the output buffer's sample and events are taken back out of the
// ManuallyDrop fields so they are released exactly once.
unsafe {
for _ in 0..3 {
let info = self.transform.GetOutputStreamInfo(self.output_id)?;
let provides = info.dwFlags
& (MFT_OUTPUT_STREAM_PROVIDES_SAMPLES.0 | MFT_OUTPUT_STREAM_CAN_PROVIDE_SAMPLES.0) as u32
!= 0;
let ours =
if provides { None } else { Some(self.sample(None, info.cbSize.max(self.output_capacity))?) };
let mut buffers = [MFT_OUTPUT_DATA_BUFFER {
dwStreamID: self.output_id,
pSample: ManuallyDrop::new(ours),
dwStatus: 0,
pEvents: ManuallyDrop::new(None),
}];
let mut status = 0u32;
let result = self.transform.ProcessOutput(0, &mut buffers, &mut status);
let sample = buffers[0].pSample.take();
drop(buffers[0].pEvents.take());
match result {
Ok(()) => {}
Err(e) if e.code() == MF_E_TRANSFORM_NEED_MORE_INPUT => return Ok(None),
Err(e) if e.code() == MF_E_TRANSFORM_STREAM_CHANGE => {
// Hardware encoders may settle their output type once streaming.
let available = self.transform.GetOutputAvailableType(self.output_id, 0)?;
self.transform.SetOutputType(self.output_id, &available, 0)?;
self.read_sequence_header();
continue;
}
Err(e) => return Err(e).context("ProcessOutput"),
}
let Some(sample) = sample else { return Ok(None) };
let key = sample.GetUINT32(&MFSampleExtension_CleanPoint).unwrap_or(0) != 0;
let buffer = sample.ConvertToContiguousBuffer()?;
let mut data = core::ptr::null_mut();
let mut len = 0u32;
buffer.Lock(&mut data, None, Some(&mut len))?;
let bytes = core::slice::from_raw_parts(data, len as usize).to_vec();
buffer.Unlock()?;
let bytes = if key && !has_sps(&bytes) && !self.sequence_header.is_empty() {
let mut with = self.sequence_header.clone();
with.extend_from_slice(&bytes);
with
} else {
bytes
};
let key = key || has_sps(&bytes);
return Ok(Some(EncodedFrame { data: bytes, key }));
}
bail!("the encoder kept changing its output type")
}
}
/// Handles queued events of an asynchronous encoder.
fn pump_events(&mut self, out: &mut Vec<EncodedFrame>) -> Result<()> {
let Some(events) = self.events.clone() else { return Ok(()) };
loop {
// SAFETY: COM call on an interface this thread owns.
let event = match unsafe { events.GetEvent(MF_EVENT_FLAG_NO_WAIT) } {
Ok(e) => e,
Err(e) if e.code() == MF_E_NO_EVENTS_AVAILABLE => return Ok(()),
Err(e) => return Err(e).context("GetEvent"),
};
// SAFETY: as above.
let kind = unsafe { event.GetType() }?;
if kind == METransformNeedInput.0 as u32 {
self.need_input += 1;
} else if kind == METransformHaveOutput.0 as u32
&& let Some(frame) = self.pull()?
{
out.push(frame);
}
}
}
}
impl VideoEncoder for MfEncoder {
fn name(&self) -> &str {
&self.name
}
fn hardware(&self) -> bool {
self.hardware
}
fn encode(&mut self, nv12: &[u8], time: i64, key: bool) -> Result<Vec<EncodedFrame>> {
let mut out = Vec::new();
if key && let Some(codec) = &self.codec {
// SAFETY: COM call with a valid VARIANT.
let _ = unsafe { codec.SetValue(&CODECAPI_AVEncVideoForceKeyFrame, &VARIANT::from(1u32)) };
}
let sample = self.sample(Some(nv12), u32::try_from(nv12.len())?)?;
// SAFETY: COM calls on interfaces this thread owns.
unsafe {
sample.SetSampleTime(time)?;
sample.SetSampleDuration(self.frame_duration)?;
}
if self.events.is_some() {
// Asynchronous (hardware): input only when asked for; outputs as they come.
let deadline = Instant::now() + Duration::from_millis(100);
while self.need_input == 0 {
self.pump_events(&mut out)?;
if self.need_input > 0 {
break;
}
if Instant::now() >= deadline {
return Ok(out); // The encoder is busy: this picture is skipped.
}
std::thread::sleep(Duration::from_millis(1));
}
self.need_input -= 1;
// SAFETY: as above.
unsafe { self.transform.ProcessInput(self.input_id, &sample, 0) }.context("ProcessInput")?;
let deadline = Instant::now() + Duration::from_millis(40);
let before = out.len();
while out.len() == before && Instant::now() < deadline {
self.pump_events(&mut out)?;
if out.len() == before {
std::thread::sleep(Duration::from_millis(1));
}
}
return Ok(out);
}
// Synchronous (software).
// SAFETY: as above.
match unsafe { self.transform.ProcessInput(self.input_id, &sample, 0) } {
Ok(()) => {}
Err(e) if e.code() == MF_E_NOTACCEPTING => {
while let Some(frame) = self.pull()? {
out.push(frame);
}
// SAFETY: as above.
unsafe { self.transform.ProcessInput(self.input_id, &sample, 0) }.context("ProcessInput")?;
}
Err(e) => return Err(e).context("ProcessInput"),
}
while let Some(frame) = self.pull()? {
out.push(frame);
}
Ok(out)
}
}
impl Drop for MfEncoder {
fn drop(&mut self) {
// SAFETY: releases the encoder's resources; the transform is not used afterwards.
unsafe {
let _ = self.activate.ShutdownObject();
}
}
}
/// Whether Annex B data holds a sequence parameter set (NAL type 7).
fn has_sps(data: &[u8]) -> bool {
data.windows(4).any(|w| w[..3] == [0, 0, 1] && w[3] & 0x1f == 7)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn bitrate_scales_with_the_picture() {
assert_eq!(bitrate(1920, 1080, 30), 6_220_800);
assert_eq!(bitrate(640, 480, 30), 1_500_000);
assert_eq!(bitrate(3840, 2160, 30), 12_000_000);
}
#[test]
fn sps_is_found_after_any_start_code() {
assert!(has_sps(&[0, 0, 0, 1, 0x67, 0x42]));
assert!(has_sps(&[0, 0, 1, 0x27]));
assert!(!has_sps(&[0, 0, 0, 1, 0x65, 0x88]));
}
/// Throughput at 1080p with a 400x400 textured block moving (what `throughput_1080p_15fps`
/// measures for RemoteFX): `cargo test --release --lib h264_1080p -- --ignored --nocapture`.
#[test]
#[ignore = "benchmark"]
fn h264_1080p_moving_block() {
if !available() {
return;
}
let (w, h) = (1920usize, 1080usize);
let mut encoder = create(1920, 1080, 30, true).expect("an encoder");
let mut bgrx = vec![0x30u8; w * h * 4];
let mut nv12 = Vec::new();
let (mut bytes, mut frames) = (0usize, 0usize);
let start = Instant::now();
for i in 0..120usize {
bgrx.fill(0x30);
let (bx, by) = ((i * 12) % (w - 400), (i * 5) % (h - 400));
for y in 0..400 {
for x in 0..400 {
let p = ((by + y) * w + bx + x) * 4;
bgrx[p..p + 4].copy_from_slice(&[(x ^ y) as u8, (x * 2) as u8, (y * 3) as u8, 0]);
}
}
let frame = crate::capture::FrameRef { data: &bgrx, stride: w * 4, width: w, height: h };
crate::video::bgrx_to_nv12(&frame, (1920, 1080), &mut nv12);
for f in encoder.encode(&nv12, i as i64 * 333_333, i == 0).expect("encode") {
bytes += f.data.len();
frames += 1;
}
}
let secs = start.elapsed().as_secs_f64();
eprintln!(
"{} (hardware: {}): {frames} frames in {secs:.2}s = {:.1} fps max, {:.1} ms/frame, {:.2} Mbit/s at 30 fps",
encoder.name(),
encoder.hardware(),
frames as f64 / secs,
secs * 1000.0 / 120.0,
bytes as f64 * 8.0 / (120.0 / 30.0) / 1e6
);
}
/// Encodes real frames with whatever encoder this machine has (Windows only; skipped
/// where Media Foundation is missing).
#[test]
fn encodes_a_key_frame_then_deltas() {
if !available() {
eprintln!("no H.264 encoder here; skipped");
return;
}
let (width, height) = (1280u32, 720u32);
let mut encoder = create(width, height, 30, true).expect("an encoder");
let mut frames = Vec::new();
let mut nv12 = vec![0u8; (width * height * 3 / 2) as usize];
for i in 0..10u8 {
nv12[..(width * height) as usize]
.iter_mut()
.enumerate()
.for_each(|(p, v)| *v = (p as u8).wrapping_add(i * 9));
frames.extend(encoder.encode(&nv12, i64::from(i) * 333_333, i == 0).expect("encode"));
}
assert!(!frames.is_empty(), "{} produced nothing", encoder.name());
assert!(frames[0].key && has_sps(&frames[0].data), "the first frame is a key frame with SPS");
assert!(frames[1..].iter().any(|f| !f.key), "later frames are deltas");
eprintln!(
"{} (hardware: {}): {} frames, {} bytes",
encoder.name(),
encoder.hardware(),
frames.len(),
frames.iter().map(|f| f.data.len()).sum::<usize>()
);
}
}
@@ -4,10 +4,12 @@
//! to the input desktop with `SetThreadDesktop`. Optional DLLs (dxgi, d3d11, sas) are loaded
//! from System32 only.
mod clipboard;
mod cursor;
mod desktop;
mod dxgi;
mod gdi;
pub(crate) mod h264;
mod input;
mod monitors;
mod source;
@@ -108,6 +110,11 @@ pub fn create_input_sink() -> Box<dyn InputSink> {
Box::new(input::create())
}
/// The device clipboard (text), or None when its window cannot be made.
pub fn create_clipboard() -> Option<std::sync::Arc<dyn crate::clipboard::OsClipboard>> {
clipboard::WindowsClipboard::start().map(|c| c as std::sync::Arc<dyn crate::clipboard::OsClipboard>)
}
pub fn probe() -> ProbeReport {
let winpe = is_winpe();
let monitors = monitors::enumerate();
+28 -11
View File
@@ -4,12 +4,14 @@ use std::sync::Arc;
use std::sync::atomic::Ordering;
use anyhow::{Context as _, Result};
use ironrdp_server::{RdpServer, RdpServerInputHandler, TransportTls};
use ironrdp_server::{CliprdrServerFactory, RdpServer, RdpServerInputHandler, TransportTls};
use tokio::io::{AsyncRead, AsyncWrite};
use crate::capture::{HostDisplay, PumpControl};
use crate::mux;
use crate::rdcleanpath::{self, CleanPathStream};
use crate::tls::ServerIdentity;
use crate::video::VideoControl;
/// The `server_addr` field of our RDCleanPath response. The gateway reports the resolved
/// address of the RDP server it dialled; the client does not use it. The server is this
@@ -27,8 +29,14 @@ pub enum SessionEnd {
Refused(String),
}
/// Runs one session to completion. The returned future is `!Send` (ironrdp-server uses
/// `Rc` internally): drive it on a current-thread runtime.
type BoxedReader = Box<dyn AsyncRead + Unpin + Send + Sync>;
type BoxedWriter = Box<dyn AsyncWrite + Unpin + Send + Sync>;
/// Runs one session to completion. `video` is set when this host can encode H.264: a browser
/// that asks for it then gets the framed stream (`crate::mux`) and video beside RDP. The
/// returned future is `!Send` (ironrdp-server uses `Rc` internally): drive it on a
/// current-thread runtime.
#[allow(clippy::too_many_arguments)]
pub async fn run_session<R, W>(
mut reader: R,
mut writer: W,
@@ -36,6 +44,8 @@ pub async fn run_session<R, W>(
display: HostDisplay,
ctl: Arc<PumpControl>,
input: Box<dyn RdpServerInputHandler>,
clipboard: Option<Box<dyn CliprdrServerFactory>>,
video: Option<Arc<VideoControl>>,
) -> Result<SessionEnd>
where
R: AsyncRead + Unpin + Send + Sync + 'static,
@@ -63,14 +73,20 @@ where
}
};
let stream = CleanPathStream::new(
reader,
writer,
accepted,
request.leftover,
SERVER_ADDR.to_owned(),
identity.cert_chain.clone(),
);
let (reader, writer, leftover): (BoxedReader, BoxedWriter, Vec<u8>) =
match video.filter(|_| mux::wants_video(request.pdu.destination.as_deref())) {
Some(control) => {
tracing::info!("the browser takes H.264 video");
let on_video = Arc::clone(&control);
let muxed = mux::start(reader, writer, request.leftover, Box::new(move |m| on_video.on_message(m)));
control.attach(muxed.out);
(Box::new(muxed.reader), Box::new(muxed.writer), Vec::new())
}
None => (Box::new(reader), Box::new(writer), request.leftover),
};
let stream =
CleanPathStream::new(reader, writer, accepted, leftover, SERVER_ADDR.to_owned(), identity.cert_chain.clone());
let eof = stream.eof_flag();
let peer_closed = stream.peer_closed_flag();
@@ -80,6 +96,7 @@ where
.with_input_handler(InputBox(input))
.with_display_handler(display)
.with_display_suppressed_handle(Arc::clone(&ctl.suppressed))
.with_cliprdr_factory(clipboard)
.build();
let result = server.run_connection_with(stream, TransportTls::AlreadyDone).await.context("RDP connection");
+462
View File
@@ -0,0 +1,462 @@
//! H.264 video for the browser beside RDP (README, Video).
//!
//! When the browser asked for it and an encoder exists, the capture pump sends the screen as
//! H.264 on the [`crate::mux::VIDEO`] channel instead of RDP bitmaps; the browser decodes it
//! with WebCodecs (in hardware where it can) and draws it over the RDP client's canvas, which
//! keeps doing input, cursor and clipboard. Without an encoder (Windows PE, Server Core) or
//! after any encoder failure, RDP bitmaps (RemoteFX) carry the screen as before.
//!
//! Messages on the video channel (first byte is the type):
//!
//! | Type | Direction | Payload |
//! |---|---|---|
//! | 1 `START` | host -> browser | JSON `{width,height,codedWidth,codedHeight,encoder,hardware}`: a new stream; a key frame follows |
//! | 2 `FRAME` | host -> browser | flags u8 (bit 0: key frame), sequence u32 LE, H.264 Annex B |
//! | 3 `STOP` | host -> browser | JSON `{reason}`: RDP bitmaps carry the screen again |
//! | 0x81 `ACK` | browser -> host | sequence u32 LE of a frame the browser has drawn (or dropped) |
//! | 0x82 `KEY` | browser -> host | none: the decoder needs a key frame |
//! | 0x83 `OFF` | browser -> host | none: this browser cannot show the video; RDP bitmaps for the rest of the session |
use std::sync::atomic::{AtomicBool, AtomicU32, Ordering};
use std::sync::{Arc, OnceLock};
use std::time::Instant;
use anyhow::Result;
use rayon::prelude::*;
use crate::capture::FrameRef;
use crate::mux::Outbound;
pub const START: u8 = 1;
pub const FRAME: u8 = 2;
pub const STOP: u8 = 3;
pub const ACK: u8 = 0x81;
pub const KEY: u8 = 0x82;
pub const OFF: u8 = 0x83;
/// Frames the browser may have unacknowledged before the pump waits: past this the link
/// or the decoder is behind, and skipping frames keeps the picture current instead of late.
const MAX_IN_FLIGHT: u32 = 4;
/// Frames encoded after the screen stops changing, so the last picture sharpens.
const REFINE_FRAMES: u32 = 3;
/// Encodes in a row without any output before an encoder is considered broken.
const SILENT_LIMIT: u32 = 15;
/// One encoded access unit.
pub struct EncodedFrame {
/// H.264 Annex B; a key frame carries its SPS and PPS.
pub data: Vec<u8>,
pub key: bool,
}
/// An H.264 encoder for one frame size.
pub trait VideoEncoder: Send {
/// Shown to the operator (e.g. "NVIDIA H.264 Encoder MFT").
fn name(&self) -> &str;
fn hardware(&self) -> bool;
/// Encodes one NV12 picture (`coded` size) at `time` (100 ns units); returns whatever the
/// encoder has finished (usually this picture, possibly nothing yet).
fn encode(&mut self, nv12: &[u8], time: i64, key: bool) -> Result<Vec<EncodedFrame>>;
}
/// Makes an encoder for (coded width, coded height, fps, hardware allowed).
pub type EncoderFactory = Box<dyn Fn(u32, u32, u32, bool) -> Result<Box<dyn VideoEncoder>> + Send>;
/// State shared between the session (browser messages) and the capture pump.
#[derive(Default)]
pub struct VideoControl {
out: OnceLock<Outbound>,
keyframe: AtomicBool,
/// The browser gave up on the video.
off: AtomicBool,
/// One past the highest sequence the browser acknowledged.
acked: AtomicU32,
}
impl VideoControl {
/// The browser accepted video: frames can go out now.
pub fn attach(&self, out: Outbound) {
let _ = self.out.set(out);
}
pub fn attached(&self) -> bool {
self.out.get().is_some()
}
/// A message from the browser.
pub fn on_message(&self, message: &[u8]) {
match message {
[ACK, a, b, c, d, ..] => {
self.acked.fetch_max(u32::from_le_bytes([*a, *b, *c, *d]).wrapping_add(1), Ordering::Relaxed);
}
[KEY, ..] => self.keyframe.store(true, Ordering::Relaxed),
[OFF, ..] => {
tracing::warn!("the browser cannot show the H.264 video; using RDP bitmaps");
self.off.store(true, Ordering::Relaxed);
}
_ => {}
}
}
fn send(&self, message: Vec<u8>) -> bool {
self.out.get().is_some_and(|o| o.send_video(message))
}
}
/// The pump's video state: owns the encoder and decides when to encode.
pub struct VideoPath {
control: Arc<VideoControl>,
factory: EncoderFactory,
fps: u32,
encoder: Option<(Box<dyn VideoEncoder>, (u32, u32))>,
/// A frame size no encoder could take (RDP bitmaps serve it).
refused: Option<(u32, u32)>,
/// A hardware encoder failed this session: software only from now on.
software_only: bool,
nv12: Vec<u8>,
seq: u32,
pending: bool,
refine: u32,
silent: u32,
started: Instant,
}
/// Coded size for a frame: even (4:2:0 chroma), the extra row/column repeats the edge.
pub fn coded_size(width: usize, height: usize) -> (u32, u32) {
let even = |v: usize| u32::try_from(v + (v & 1)).unwrap_or(u32::MAX);
(even(width), even(height))
}
impl VideoPath {
pub fn new(control: Arc<VideoControl>, factory: EncoderFactory, fps: u32) -> Self {
Self {
control,
factory,
fps,
encoder: None,
refused: None,
software_only: false,
nv12: Vec::new(),
seq: 0,
pending: true,
refine: 0,
silent: 0,
started: Instant::now(),
}
}
/// Whether video carries a frame of this size (else RDP bitmaps do).
pub fn handles(&self, frame: &FrameRef<'_>) -> bool {
self.control.attached()
&& !self.control.off.load(Ordering::Relaxed)
&& self.refused != Some(coded_size(frame.width, frame.height))
}
/// RDP bitmaps take over: true when video was carrying the picture until now (so RDP must
/// send a complete frame).
pub fn hand_back(&mut self) -> bool {
self.encoder.take().is_some()
}
/// One pump tick with video in charge. `changed`: the screen changed since the last tick.
/// Returns false when video just gave up on this size: RDP must send a full frame now.
pub fn tick(&mut self, frame: &FrameRef<'_>, changed: bool) -> bool {
let size = coded_size(frame.width, frame.height);
if changed {
self.pending = true;
self.refine = REFINE_FRAMES;
}
let mut key = self.control.keyframe.swap(false, Ordering::Relaxed);
if self.encoder.as_ref().is_none_or(|(_, s)| *s != size) {
self.encoder = None;
match (self.factory)(size.0, size.1, self.fps, !self.software_only) {
Ok(encoder) => {
tracing::info!(
encoder = encoder.name(),
hardware = encoder.hardware(),
width = size.0,
height = size.1,
"H.264 video"
);
let start = serde_json::json!({
"width": frame.width, "height": frame.height,
"codedWidth": size.0, "codedHeight": size.1,
"encoder": encoder.name(), "hardware": encoder.hardware(),
});
let mut message = vec![START];
message.extend_from_slice(start.to_string().as_bytes());
self.control.send(message);
self.encoder = Some((encoder, size));
self.pending = true;
self.silent = 0;
key = true;
}
Err(e) => return self.give_up(size, &format!("{e:#}")),
}
}
if !self.pending && self.refine == 0 && !key {
return true;
}
let in_flight = self.seq.saturating_sub(self.control.acked.load(Ordering::Relaxed));
if in_flight >= MAX_IN_FLIGHT && !key {
// The browser is behind: skip this tick, the change stays pending.
return true;
}
bgrx_to_nv12(frame, size, &mut self.nv12);
let time = i64::try_from(self.started.elapsed().as_nanos() / 100).unwrap_or(i64::MAX);
let Some((encoder, _)) = self.encoder.as_mut() else { return true };
let hardware = encoder.hardware();
let frames = match encoder.encode(&self.nv12, time, key) {
Ok(f) => f,
Err(e) => return self.encoder_failed(size, hardware, &format!("{e:#}")),
};
if frames.is_empty() {
self.silent += 1;
if self.silent >= SILENT_LIMIT {
return self.encoder_failed(size, hardware, "the encoder produced no output");
}
} else {
self.silent = 0;
}
for f in frames {
let mut message = Vec::with_capacity(f.data.len() + 6);
message.push(FRAME);
message.push(u8::from(f.key));
message.extend_from_slice(&self.seq.to_le_bytes());
message.extend_from_slice(&f.data);
self.seq = self.seq.wrapping_add(1);
if !self.control.send(message) {
break;
}
}
if self.pending {
self.pending = false;
} else {
self.refine = self.refine.saturating_sub(1);
}
true
}
fn encoder_failed(&mut self, size: (u32, u32), hardware: bool, reason: &str) -> bool {
self.encoder = None;
if hardware {
// Try the software encoder next tick (a new stream starts with a key frame).
tracing::warn!(reason, "hardware H.264 encoder failed; using software");
self.software_only = true;
self.pending = true;
return true;
}
self.give_up(size, reason)
}
fn give_up(&mut self, size: (u32, u32), reason: &str) -> bool {
tracing::warn!(
reason,
width = size.0,
height = size.1,
"no H.264 video for this screen size; using RDP bitmaps"
);
self.encoder = None;
self.refused = Some(size);
let mut message = vec![STOP];
message.extend_from_slice(serde_json::json!({ "reason": reason }).to_string().as_bytes());
self.control.send(message);
false
}
}
/// BGRX (32 bpp) to NV12 at `coded` size (BT.709, limited range, 2x2 averaged chroma); the
/// rows and columns past the frame repeat its last row and column.
pub fn bgrx_to_nv12(frame: &FrameRef<'_>, coded: (u32, u32), out: &mut Vec<u8>) {
let (cw, ch) = (coded.0 as usize, coded.1 as usize);
out.resize(cw * ch * 3 / 2, 0);
if frame.width == 0 || frame.height == 0 {
return;
}
let (luma, chroma) = out.split_at_mut(cw * ch);
let last_x = frame.width - 1;
let last_y = frame.height - 1;
let pixel = |x: usize, y: usize| -> (i32, i32, i32) {
let i = y.min(last_y) * frame.stride + x.min(last_x) * 4;
let p = &frame.data[i..i + 3];
(i32::from(p[2]), i32::from(p[1]), i32::from(p[0]))
};
luma.par_chunks_mut(cw * 2).zip(chroma.par_chunks_mut(cw)).enumerate().for_each(|(pair, (y2, uv))| {
let y0 = pair * 2;
for (row, line) in y2.chunks_mut(cw).enumerate() {
for (x, out) in line.iter_mut().enumerate() {
let (r, g, b) = pixel(x, y0 + row);
*out = (((47 * r + 157 * g + 16 * b + 128) >> 8) + 16) as u8;
}
}
for (cx, out) in uv.as_chunks_mut::<2>().0.iter_mut().enumerate() {
let (mut r, mut g, mut b) = (0, 0, 0);
for (dx, dy) in [(0, 0), (1, 0), (0, 1), (1, 1)] {
let p = pixel(cx * 2 + dx, y0 + dy);
r += p.0;
g += p.1;
b += p.2;
}
out[0] = (((-26 * r - 86 * g + 112 * b + 512) >> 10) + 128).clamp(0, 255) as u8;
out[1] = (((112 * r - 102 * g - 10 * b + 512) >> 10) + 128).clamp(0, 255) as u8;
}
});
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::Mutex;
fn solid(width: usize, height: usize, bgrx: [u8; 4]) -> Vec<u8> {
bgrx.repeat(width * height)
}
#[test]
fn colours_convert_to_limited_range_bt709() {
for (bgrx, y, u, v) in [
([255, 255, 255, 0], 235, 128, 128),
([0, 0, 0, 0], 16, 128, 128),
([0, 0, 255, 0], 63, 102, 240), // red
([255, 0, 0, 0], 32, 240, 118), // blue
] {
let data = solid(3, 3, bgrx);
let frame = FrameRef { data: &data, stride: 12, width: 3, height: 3 };
let mut out = Vec::new();
bgrx_to_nv12(&frame, coded_size(3, 3), &mut out);
assert_eq!(out.len(), 4 * 4 * 3 / 2);
assert!(out[..16].iter().all(|&p| p == y), "{bgrx:?}: luma {:?}", &out[..16]);
assert!(out[16..].chunks(2).all(|c| c == [u, v]), "{bgrx:?}: chroma {:?}", &out[16..]);
}
}
#[test]
fn odd_sizes_are_padded_by_repeating_the_edge() {
assert_eq!(coded_size(1366, 767), (1366, 768));
let mut data = solid(3, 1, [0, 0, 0, 0]);
data[8..12].copy_from_slice(&[255, 255, 255, 0]);
let frame = FrameRef { data: &data, stride: 12, width: 3, height: 1 };
let mut out = Vec::new();
bgrx_to_nv12(&frame, coded_size(3, 1), &mut out);
assert_eq!(&out[..8], &[16, 16, 235, 235, 16, 16, 235, 235]);
}
struct FakeEncoder {
log: Arc<Mutex<Vec<bool>>>,
hardware: bool,
fail: bool,
}
impl VideoEncoder for FakeEncoder {
fn name(&self) -> &str {
"fake"
}
fn hardware(&self) -> bool {
self.hardware
}
fn encode(&mut self, _: &[u8], _: i64, key: bool) -> Result<Vec<EncodedFrame>> {
if self.fail {
anyhow::bail!("broken");
}
self.log.lock().unwrap().push(key);
Ok(vec![EncodedFrame { data: vec![0, 0, 0, 1, 0x65], key }])
}
}
type Outbox = tokio::sync::mpsc::Receiver<(u8, bytes::Bytes)>;
fn path(hardware_fails: bool) -> (VideoPath, Arc<VideoControl>, Arc<Mutex<Vec<bool>>>, Outbox) {
let control = Arc::new(VideoControl::default());
let (tx, rx) = tokio::sync::mpsc::channel(256);
control.attach(crate::mux::Outbound::for_tests(tx));
let log = Arc::new(Mutex::new(Vec::new()));
let encodes = Arc::clone(&log);
let factory: EncoderFactory = Box::new(move |_, _, _, hw| {
Ok(Box::new(FakeEncoder { log: Arc::clone(&encodes), hardware: hw, fail: hw && hardware_fails })
as Box<dyn VideoEncoder>)
});
(VideoPath::new(Arc::clone(&control), factory, 30), control, log, rx)
}
fn kinds(rx: &mut Outbox) -> Vec<u8> {
std::iter::from_fn(|| rx.try_recv().ok()).map(|(_, m)| m[0]).collect()
}
#[test]
fn starts_with_a_key_frame_refines_then_idles() {
let (mut video, control, log, mut rx) = path(false);
let data = solid(4, 2, [1, 2, 3, 0]);
let frame = FrameRef { data: &data, stride: 16, width: 4, height: 2 };
assert!(video.tick(&frame, true));
for _ in 0..10 {
control.on_message(&[ACK, 0, 0, 0, 0]);
assert!(video.tick(&frame, false));
}
assert_eq!(
*log.lock().unwrap(),
vec![true, false, false, false],
"key frame, then three refinements, then nothing"
);
assert_eq!(kinds(&mut rx), vec![START, FRAME, FRAME, FRAME, FRAME]);
control.on_message(&[KEY]);
assert!(video.tick(&frame, false));
assert_eq!(log.lock().unwrap().last(), Some(&true), "the browser's key frame request is honoured");
}
#[test]
fn waits_for_the_browser_when_too_many_frames_are_unacknowledged() {
let (mut video, control, log, _rx) = path(false);
let data = solid(4, 2, [1, 2, 3, 0]);
let frame = FrameRef { data: &data, stride: 16, width: 4, height: 2 };
for _ in 0..10 {
video.tick(&frame, true);
}
assert_eq!(log.lock().unwrap().len(), MAX_IN_FLIGHT as usize);
control.on_message(&[ACK, 1, 0, 0, 0]);
video.tick(&frame, false);
assert_eq!(
log.lock().unwrap().len(),
MAX_IN_FLIGHT as usize + 1,
"the pending change goes out once acknowledged"
);
}
#[test]
fn a_failing_hardware_encoder_falls_back_to_software() {
let (mut video, _control, log, mut rx) = path(true);
let data = solid(4, 2, [1, 2, 3, 0]);
let frame = FrameRef { data: &data, stride: 16, width: 4, height: 2 };
assert!(video.tick(&frame, true), "hardware failed, software comes next");
assert!(video.tick(&frame, false));
assert_eq!(*log.lock().unwrap(), vec![true]);
assert_eq!(kinds(&mut rx), vec![START, START, FRAME]);
}
#[test]
fn a_browser_that_cannot_decode_turns_video_off() {
let (mut video, control, _log, _rx) = path(false);
let data = solid(4, 2, [1, 2, 3, 0]);
let frame = FrameRef { data: &data, stride: 16, width: 4, height: 2 };
assert!(video.tick(&frame, true));
control.on_message(&[OFF]);
assert!(!video.handles(&frame));
assert!(video.hand_back(), "RDP sends a complete frame");
assert!(!video.hand_back(), "only once");
}
#[test]
fn no_encoder_hands_the_size_back_to_rdp() {
let control = Arc::new(VideoControl::default());
let (tx, mut rx) = tokio::sync::mpsc::channel(8);
control.attach(crate::mux::Outbound::for_tests(tx));
let mut video = VideoPath::new(Arc::clone(&control), Box::new(|_, _, _, _| anyhow::bail!("too large")), 30);
let data = solid(4, 2, [1, 2, 3, 0]);
let frame = FrameRef { data: &data, stride: 16, width: 4, height: 2 };
assert!(video.handles(&frame));
assert!(!video.tick(&frame, true));
assert!(!video.handles(&frame));
assert_eq!(kinds(&mut rx), vec![STOP]);
}
}
+3 -2
View File
@@ -129,7 +129,7 @@ async fn run_e2e(pattern: TestPattern, view_only: bool, max_fps: u32, min_update
Box::new(pattern),
tx,
ctl,
PumpConfig { max_fps },
PumpConfig { max_fps, video: None },
|_| {},
|x, y| coords.set_origin(x, y),
);
@@ -145,7 +145,8 @@ async fn run_e2e(pattern: TestPattern, view_only: bool, max_fps: u32, min_update
let (server_read, server_write) = tokio::io::split(server_side);
let start = Instant::now();
let server = session::run_session(server_read, server_write, identity, display, Arc::clone(&ctl), input);
let server =
session::run_session(server_read, server_write, identity, display, Arc::clone(&ctl), input, None, None);
let client = common::run_client(client_side, true, min_updates, Duration::from_secs(60));
let (server_result, client_result) = tokio::join!(server, client);
let elapsed = start.elapsed();
+16
View File
@@ -158,6 +158,22 @@ impl UpdateEncoder {
BitmapUpdater::Bitmap(BitmapHandler::new())
};
// ExecuteR patch: name the negotiated codec (the first thing to check when a session feels slow).
tracing::info!(
codec = match &bitmap_updater {
BitmapUpdater::None(_) => "none",
BitmapUpdater::Bitmap(_) => "bitmap",
BitmapUpdater::RemoteFx(_) => "remotefx",
#[cfg(feature = "qoi")]
BitmapUpdater::Qoi(_) => "qoi",
#[cfg(feature = "qoiz")]
BitmapUpdater::Qoiz(_) => "qoiz",
#[cfg(feature = "nscodec")]
BitmapUpdater::NsCodec(_) => "nscodec",
},
"bitmap codec"
);
Ok(Self {
desktop_size,
framebuffer: None,
+2
View File
@@ -1422,6 +1422,8 @@ impl RdpServer {
surface_flags = c.flags;
}
CapabilitySet::BitmapCodecs(BitmapCodecs(codecs)) => {
// ExecuteR patch: what the client offers, next to what the encoder picks.
debug!(offered = ?codecs.iter().map(|c| format!("{:?}", c.property).chars().take(24).collect::<String>()).collect::<Vec<_>>(), "client bitmap codecs");
for codec in codecs {
match codec.property {
// FIXME: The encoder operates in image mode only.
+9
View File
@@ -23,6 +23,7 @@ import useSession from 'hooks/useSession';
import DeviceToolFrame, { useToolSurface } from 'sections/deviceTools/DeviceToolFrame';
import { useIsPopOut } from 'sections/deviceTools/popOut';
import RemoteScreenView, { type RemoteScreenHandle, type ScreenScale, type ViewPhase } from 'sections/screen/RemoteScreenView';
import type { VideoInfo } from 'sections/screen/videoChannel';
import { screenGates } from 'sections/screen/screenGate';
import { api, errorMessage, isUnsupported, keys } from 'api/client';
import type { DeviceDetailDto, ScreenMonitor, ScreenSessionDto, ScreenTicketDto } from 'api/types';
@@ -92,6 +93,7 @@ function ScreenView({ clientId, viewOnly }: { clientId: string; viewOnly: boolea
const [reason, setReason] = useState<string>();
const [scale, setScale] = useState<ScreenScale>('fit');
const [desktop, setDesktop] = useState<{ width: number; height: number }>();
const [video, setVideo] = useState<VideoInfo | null>(null);
const [monitor, setMonitor] = useState(-1);
const [openError, setOpenError] = useState<unknown>();
const view = useRef<RemoteScreenHandle>(null);
@@ -262,6 +264,11 @@ function ScreenView({ clientId, viewOnly }: { clientId: string; viewOnly: boolea
!gate && (
<Stack direction="row" component="span" sx={{ gap: 1, alignItems: 'center' }}>
<StatusChip status={status.status} label={status.label} tooltip={reason} />
{video && (
<Tooltip title={`${video.hardware ? 'GPU' : 'Software'} encoder on the device: ${video.encoder}`}>
<Chip size="small" variant="light" color="success" label={video.hardware ? 'H.264 GPU' : 'H.264'} />
</Tooltip>
)}
{viewOnly && (
<Tooltip title="Keyboard and mouse are not sent to the device.">
<Chip size="small" variant="light" color="info" icon={<EyeOutlined />} label="View Only" />
@@ -362,6 +369,7 @@ function ScreenView({ clientId, viewOnly }: { clientId: string; viewOnly: boolea
<Box ref={frame} sx={{ bgcolor: '#000', ...(hug ? hugging(hug) : surface.body) }}>
{showing && (
<RemoteScreenView
clipboard={!viewOnly}
ref={view}
key={ticket.token}
ticket={ticket}
@@ -369,6 +377,7 @@ function ScreenView({ clientId, viewOnly }: { clientId: string; viewOnly: boolea
scale={scale}
onPhase={onPhase}
onDesktopSize={setDesktop}
onVideo={setVideo}
/>
)}
</Box>
+115 -2
View File
@@ -30,6 +30,7 @@ import useApi from 'hooks/useApi';
import useSession, { useGate } from 'hooks/useSession';
import { api, errorMessage, keys, request } from 'api/client';
import type { DeviceDto, DeviceFileEntry, DeviceFileReply, DeviceFileRoot } from 'api/types';
import { copyText } from 'utils/clipboard';
import { formatAbsolute, formatBytes, plural } from 'utils/format';
import SendFromS3Dialog from './SendFromS3Dialog';
import TextFileDialog from './TextFileDialog';
@@ -39,6 +40,7 @@ import UploadsCard, { UploadConflictDialog } from 'sections/uploads/UploadsCard'
// assets
import CloudDownloadOutlined from '@ant-design/icons/CloudDownloadOutlined';
import CopyOutlined from '@ant-design/icons/CopyOutlined';
import CloudServerOutlined from '@ant-design/icons/CloudServerOutlined';
import CloudUploadOutlined from '@ant-design/icons/CloudUploadOutlined';
import DeleteOutlined from '@ant-design/icons/DeleteOutlined';
@@ -90,7 +92,11 @@ export default function DeviceFiles({ device }: { device: DeviceDto }) {
const [take, setTake] = useState(PAGE);
const [chosen, setChosen] = useState<ReadonlySet<string>>(new Set());
const [address, setAddress] = useState<string | null>(null);
const [menu, setMenu] = useState<{ anchor: HTMLElement; entries: DeviceFileEntry[] } | null>(null);
// A row's "…" or Actions button anchors the menu; a right-click opens it where the pointer is.
const [menu, setMenu] = useState<{ anchor?: HTMLElement; position?: { top: number; left: number }; entries: DeviceFileEntry[] } | null>(
null
);
const [folderMenu, setFolderMenu] = useState<{ top: number; left: number } | null>(null);
const [uploadMenu, setUploadMenu] = useState<HTMLElement | null>(null);
const [nameDialog, setNameDialog] = useState<NameDialog | null>(null);
const [nameValue, setNameValue] = useState('');
@@ -285,6 +291,23 @@ export default function DeviceFiles({ device }: { device: DeviceDto }) {
});
}
items.push({ kind: 'divider' });
items.push({
kind: 'item',
id: 'copy-path',
label: one ? 'Copy Path' : 'Copy Paths',
icon: <CopyOutlined />,
onSelect: close(() => void copyText(targets.map((t) => t.path).join('\n'), one ? 'Path' : 'Paths'))
});
if (one) {
items.push({
kind: 'item',
id: 'copy-name',
label: 'Copy Name',
icon: <CopyOutlined />,
onSelect: close(() => void copyText(one.name, 'Name'))
});
}
items.push({ kind: 'divider' });
items.push({
kind: 'item',
id: 'delete',
@@ -300,6 +323,67 @@ export default function DeviceFiles({ device }: { device: DeviceDto }) {
return items;
};
/** Right-click on the folder itself (not on a row). */
const folderEntries = (): MenuEntry[] => {
const close = (action: () => void) => () => {
setFolderMenu(null);
action();
};
const items: MenuEntry[] = [];
if (path)
items.push({
kind: 'item',
id: 'copy-folder',
label: 'Copy Folder Path',
icon: <CopyOutlined />,
onSelect: close(() => void copyText(path, 'Folder Path'))
});
items.push(
{ kind: 'divider' },
{
kind: 'item',
id: 'new-file',
label: 'New File…',
icon: <FileAddOutlined />,
disabledReason: manageGate,
onSelect: close(() => {
setNameValue('');
setNameDialog({ kind: 'file' });
})
},
{
kind: 'item',
id: 'new-folder',
label: 'New Folder…',
icon: <FolderAddOutlined />,
disabledReason: manageGate,
onSelect: close(() => {
setNameValue('');
setNameDialog({ kind: 'folder' });
})
},
{
kind: 'item',
id: 'upload-files',
label: 'Upload Files…',
icon: <UploadOutlined />,
disabledReason: manageGate,
onSelect: close(() => fileInput.current?.click())
},
{
kind: 'item',
id: 'upload-folder',
label: 'Upload Folder…',
icon: <FolderOutlined />,
disabledReason: manageGate,
onSelect: close(() => folderInput.current?.click())
},
{ kind: 'divider' },
{ kind: 'item', id: 'refresh', label: 'Refresh', icon: <ReloadOutlined />, onSelect: close(() => void reload()) }
);
return items;
};
const submitName = async () => {
if (!nameDialog || !path) return;
const name = nameValue.trim();
@@ -612,6 +696,18 @@ export default function DeviceFiles({ device }: { device: DeviceDto }) {
<EditOutlined />
</IconButton>
</Tooltip>
{path && (
<Tooltip title="Copy Folder Path">
<IconButton
size="small"
color="secondary"
aria-label="Copy folder path"
onClick={() => void copyText(path, 'Folder Path')}
>
<CopyOutlined />
</IconButton>
</Tooltip>
)}
<Tooltip title="Refresh">
<IconButton size="small" color="secondary" aria-label="Refresh" onClick={() => void reload()}>
<ReloadOutlined />
@@ -631,6 +727,12 @@ export default function DeviceFiles({ device }: { device: DeviceDto }) {
if (!e.currentTarget.contains(e.relatedTarget as Node)) setDragging(false);
}}
onDrop={onDrop}
onContextMenu={(e) => {
// A row's own right-click already opened its menu.
if (e.defaultPrevented || (e.target as HTMLElement).closest('input, textarea')) return;
e.preventDefault();
setFolderMenu({ top: e.clientY, left: e.clientX });
}}
sx={{ outline: dragging ? '2px dashed' : 'none', outlineColor: 'primary.main', outlineOffset: -4 }}
>
<DataTable
@@ -644,6 +746,9 @@ export default function DeviceFiles({ device }: { device: DeviceDto }) {
paged={false}
dense
onRowClick={open}
onRowContextMenu={(row, position) =>
setMenu({ position, entries: chosen.has(row.path) && chosenEntries.length > 1 ? chosenEntries : [row] })
}
rowLabel={(e) => `Choose ${e.name}`}
selection={{
selected: chosen,
@@ -725,7 +830,15 @@ export default function DeviceFiles({ device }: { device: DeviceDto }) {
/>
)}
{menu && <ActionMenu anchorEl={menu.anchor} onClose={() => setMenu(null)} entries={menuEntries(menu.entries)} />}
{menu && (
<ActionMenu
anchorEl={menu.anchor}
anchorPosition={menu.position}
onClose={() => setMenu(null)}
entries={menuEntries(menu.entries)}
/>
)}
{folderMenu && <ActionMenu anchorPosition={folderMenu} onClose={() => setFolderMenu(null)} entries={folderEntries()} />}
<ConfirmDialog
open={!!nameDialog}
@@ -29,6 +29,7 @@ import TreeList, { type TreeMenuAnchor, type TreeNode } from 'components/TreeLis
import { useGate } from 'hooks/useSession';
import { api, downloads, errorMessage, isUnsupported, keys, request } from 'api/client';
import type { DeviceDto, RegistryKeyEntry, RegistryReply, RegistryValue, RegistryView } from 'api/types';
import { copyText } from 'utils/clipboard';
import { formatAbsolute } from 'utils/format';
import RegistryImportDialog from './RegistryImportDialog';
import RegistryValueDialog from './RegistryValueDialog';
@@ -52,6 +53,7 @@ import {
} from './registryModel';
// assets
import CopyOutlined from '@ant-design/icons/CopyOutlined';
import DeleteOutlined from '@ant-design/icons/DeleteOutlined';
import EditOutlined from '@ant-design/icons/EditOutlined';
import ExportOutlined from '@ant-design/icons/ExportOutlined';
@@ -260,6 +262,23 @@ export default function DeviceRegistry({ device }: { device: DeviceDto }) {
downloads.registryExport(clientId, keyPath, view).catch((e) => setNotice(errorMessage(e, 'The export failed.')));
};
/** The value's data as text: strings as they are, multi-strings one per line, numbers as shown, binary as hex. */
const copyValueData = async (value: RegistryValue) => {
let whole = value;
if (value.truncated) {
try {
whole = (await api.registryValue(clientId, current, view, value.name)).value ?? value;
} catch (e) {
setNotice(errorMessage(e, 'The value could not be read.'));
return;
}
}
const text = whole.lines
? whole.lines.join('\n')
: (whole.text ?? displayData({ ...whole, truncated: false }, Number.MAX_SAFE_INTEGER));
await copyText(text, 'Data');
};
const openValue = async (value: RegistryValue) => {
setNotice('');
let whole = value;
@@ -303,6 +322,21 @@ export default function DeviceRegistry({ device }: { device: DeviceDto }) {
},
{ kind: 'item', id: 'export', label: 'Export…', icon: <ExportOutlined />, onSelect: () => exportKey(target.path) },
{ kind: 'divider' },
{
kind: 'item',
id: 'copy-path',
label: 'Copy Key Path',
icon: <CopyOutlined />,
onSelect: () => void copyText(target.path, 'Key Path')
},
{
kind: 'item',
id: 'copy-name',
label: 'Copy Key Name',
icon: <CopyOutlined />,
onSelect: () => void copyText(nameOf(target.path), 'Key Name')
},
{ kind: 'divider' },
{
kind: 'item',
id: 'delete',
@@ -339,6 +373,23 @@ export default function DeviceRegistry({ device }: { device: DeviceDto }) {
}
},
{ kind: 'divider' },
{
kind: 'item',
id: 'copy-name',
label: 'Copy Name',
icon: <CopyOutlined />,
disabledReason: value.name === '' ? 'The default value has no name.' : undefined,
onSelect: () => void copyText(value.name, 'Name')
},
{ kind: 'item', id: 'copy-data', label: 'Copy Data', icon: <CopyOutlined />, onSelect: () => void copyValueData(value) },
{
kind: 'item',
id: 'copy-path',
label: 'Copy Value Path',
icon: <CopyOutlined />,
onSelect: () => void copyText(`${current}\\${value.name || '(Default)'}`, 'Value Path')
},
{ kind: 'divider' },
{
kind: 'item',
id: 'delete',
@@ -613,6 +664,11 @@ export default function DeviceRegistry({ device }: { device: DeviceDto }) {
<EditOutlined />
</IconButton>
</Tooltip>
<Tooltip title="Copy Key Path">
<IconButton size="small" color="secondary" aria-label="Copy key path" onClick={() => void copyText(current, 'Key Path')}>
<CopyOutlined />
</IconButton>
</Tooltip>
<Tooltip title="Refresh">
<IconButton
size="small"
@@ -8,6 +8,8 @@ import type { UserInteraction } from '@devolutions/iron-remote-desktop';
import type { ScreenTicketDto } from 'api/types';
import { h264Supported, routeSocket, VIDEO_DESTINATION, VideoSink, type VideoInfo } from './videoChannel';
/** What the toolbar can do to a running view. */
export interface RemoteScreenHandle {
ctrlAltDel: () => void;
@@ -36,6 +38,10 @@ div:has(> .screen-wrapper) { width: 100%; height: 100%; }
width: var(--rv-w) !important; height: var(--rv-h) !important;
min-width: var(--rv-w) !important; min-height: var(--rv-h) !important;
max-width: var(--rv-w) !important; max-height: var(--rv-h) !important;
}
.screen-viewer { position: relative !important; }
.screen-viewer canvas.exr-video {
position: absolute; inset: 0; width: 100% !important; height: 100% !important; pointer-events: none;
}`;
/** Sizes the screen to the container: Fit scales the desktop into it, Real keeps its pixels (and scrolls). */
@@ -61,11 +67,28 @@ interface Props {
sessionId: string;
scale: ScreenScale;
onPhase: (phase: ViewPhase, reason?: string) => void;
/** Text clipboard both ways (control sessions); read when the connection starts. */
clipboard?: boolean;
/** The device's H.264 encoder while video carries the screen; null when RDP bitmaps do. */
onVideo?: (info: VideoInfo | null) => void;
/** The device's desktop size once connected, and again whenever it changes. */
onDesktopSize?: (size: { width: number; height: number }) => void;
ref?: Ref<RemoteScreenHandle>;
}
/** Waits out the component's own clipboard set-up: the same permission query it makes, then a beat. */
async function clipboardSetUp() {
try {
await Promise.race([
navigator.permissions.query({ name: 'clipboard-read' as PermissionName }),
new Promise((resolve) => setTimeout(resolve, 1000))
]);
} catch {
// Not a permission this browser knows (Firefox): the component falls back the same way.
}
await new Promise((resolve) => setTimeout(resolve, 100));
}
/** An IronError (kind/backtrace), an Error, or anything else, as one line. */
function describe(error: unknown): string {
if (error && typeof error === 'object' && 'backtrace' in error && typeof error.backtrace === 'function') {
@@ -81,14 +104,18 @@ function describe(error: unknown): string {
* imported here, dynamically, so they load only with this view (the RDP backend
* inlines its WASM as a data URL: no asset to serve, and nothing in the main bundle).
*/
export default function RemoteScreenView({ ticket, sessionId, scale, onPhase, onDesktopSize, ref }: Props) {
export default function RemoteScreenView({ ticket, sessionId, scale, clipboard = false, onPhase, onDesktopSize, onVideo, ref }: Props) {
const hostRef = useRef<HTMLDivElement>(null);
const interaction = useRef<UserInteraction | null>(null);
const phase = useRef(onPhase);
phase.current = onPhase;
const desktopSize = useRef(onDesktopSize);
desktopSize.current = onDesktopSize;
const videoInfo = useRef(onVideo);
videoInfo.current = onVideo;
const scaleRef = useRef(scale);
const clipboardRef = useRef(clipboard);
clipboardRef.current = clipboard;
scaleRef.current = scale;
const elementRef = useRef<IronElement | null>(null);
@@ -114,6 +141,7 @@ export default function RemoteScreenView({ ticket, sessionId, scale, onPhase, on
let element: IronElement | null = null;
let connected = false;
const observers: { disconnect: () => void }[] = [];
let sink: VideoSink | undefined;
// The canvas holds 800x600 until the session sizes it: only a connected session's size is the device's.
const reportSize = (canvas: HTMLCanvasElement) => {
if (connected && canvas.width && canvas.height) desktopSize.current?.({ width: canvas.width, height: canvas.height });
@@ -121,9 +149,15 @@ export default function RemoteScreenView({ ticket, sessionId, scale, onPhase, on
phase.current('loading');
(async () => {
const [, rdp] = await Promise.all([import('@devolutions/iron-remote-desktop'), import('@devolutions/iron-remote-desktop-rdp')]);
const [, rdp, video] = await Promise.all([
import('@devolutions/iron-remote-desktop'),
import('@devolutions/iron-remote-desktop-rdp'),
h264Supported()
]);
await rdp.init('WARN');
if (stopped) return;
// H.264 from hosts that can encode it, drawn over the RDP canvas (RDP keeps input, cursor, clipboard).
if (video) sink = new VideoSink((info) => videoInfo.current?.(info));
element = document.createElement('iron-remote-desktop') as IronElement;
// Set before it is attached: the component reads its backend when it mounts.
@@ -145,6 +179,7 @@ export default function RemoteScreenView({ ticket, sessionId, scale, onPhase, on
const style = document.createElement('style');
style.textContent = SHADOW_CSS;
shadow.appendChild(style);
if (sink) canvas.after(sink.canvas);
const relayout = () => element && layout(element, host, scaleRef.current);
// The desktop size arrives (and changes) as the canvas' width and height.
const resized = new MutationObserver(() => {
@@ -159,20 +194,32 @@ export default function RemoteScreenView({ ticket, sessionId, scale, onPhase, on
void (async () => {
try {
const scheme = window.location.protocol === 'https:' ? 'wss' : 'ws';
const proxy = `${scheme}://${window.location.host}${ticket.socketPath}`;
const config = ui
.configBuilder()
.withUsername('remote-view')
.withPassword('-')
.withDestination('screen')
.withDestination(sink ? VIDEO_DESTINATION : 'screen')
.withServerDomain('')
.withProxyAddress(`${scheme}://${window.location.host}${ticket.socketPath}`)
.withProxyAddress(proxy)
.withAuthToken(sessionId)
.withDesktopSize({ width: Math.max(640, host.clientWidth), height: Math.max(480, host.clientHeight) })
.build();
// The screen host has no clipboard in v1.
ui.setEnableClipboard(false);
// Copy on either side, paste on the other: the component reads this browser's clipboard when the tab
// regains focus and writes the device's copies to it (Chromium asks the operator once).
ui.setEnableClipboard(clipboardRef.current);
phase.current('connecting');
const session = await ui.connect(config);
// The component fires "ready" before it sets up its clipboard handling (a permission query), and a
// connection made before that asks for no clipboard channel at all.
if (clipboardRef.current) await clipboardSetUp();
if (stopped) return;
const unroute = sink ? routeSocket(proxy, sink) : undefined;
let session: Awaited<ReturnType<typeof ui.connect>>;
try {
session = await ui.connect(config);
} finally {
unroute?.();
}
if (stopped) return;
phase.current('connected');
connected = true;
@@ -201,6 +248,8 @@ export default function RemoteScreenView({ ticket, sessionId, scale, onPhase, on
// Already down.
}
interaction.current = null;
sink?.close();
videoInfo.current?.(null);
observers.forEach((observer) => observer.disconnect());
elementRef.current = null;
element?.remove();
@@ -0,0 +1,407 @@
/**
* H.264 video beside the RDP client (ScreenHost README, Video).
*
* The browser asks for video in the RDCleanPath destination (`screen;video=h264`). A screen host
* that can encode answers `EXRMUX01` first and frames the stream from then on
* (`[channel u8][length u32 LE][payload]`: 0 RDP, 1 video); an older host, or one without an
* encoder (Windows PE), answers plain RDCleanPath and nothing changes. The RDP client (IronRDP,
* WASM) opens its own WebSocket, so {@link routeSocket} hands it a stand-in that unframes channel 0
* for it and gives channel 1 to a {@link VideoSink}: WebCodecs decodes the frames (in hardware
* where it can) onto a canvas over the RDP canvas, which keeps input, cursor and clipboard.
*/
const MAGIC = new TextEncoder().encode('EXRMUX01');
const RDP = 0;
const VIDEO = 1;
const HEADER = 5;
const START = 1;
const FRAME = 2;
const STOP = 3;
const ACK = 0x81;
const KEY = 0x82;
const OFF = 0x83;
/** No picture this long after the stream starts: this browser cannot show it. */
const FIRST_PICTURE_MS = 5000;
/** Decoder failures in a row (without a picture between) before giving up. */
const MAX_RECOVERIES = 3;
/** The destination hint the host understands. */
export const VIDEO_DESTINATION = 'screen;video=h264';
/** What the device encodes with, once video is on. */
export interface VideoInfo {
encoder: string;
hardware: boolean;
}
/** Whether this browser can show the host's H.264 (WebCodecs needs a secure context). */
export async function h264Supported(): Promise<boolean> {
if (!window.isSecureContext || typeof VideoDecoder === 'undefined' || typeof EncodedVideoChunk === 'undefined') return false;
try {
const result = await VideoDecoder.isConfigSupported({ codec: 'avc1.42E033', optimizeForLatency: true });
return Boolean(result.supported);
} catch {
return false;
}
}
/** The `avc1.PPCCLL` codec string from a key frame's SPS (profile, constraints, level). */
export function codecOf(annexB: Uint8Array): string {
for (let i = 0; i + 6 < annexB.length; i++) {
if (annexB[i] === 0 && annexB[i + 1] === 0 && annexB[i + 2] === 1 && (annexB[i + 3] & 0x1f) === 7) {
const hex = (v: number) => v.toString(16).padStart(2, '0').toUpperCase();
return `avc1.${hex(annexB[i + 4])}${hex(annexB[i + 5])}${hex(annexB[i + 6])}`;
}
}
return 'avc1.42E033';
}
function header(channel: number, length: number): Uint8Array<ArrayBuffer> {
const out = new Uint8Array(HEADER);
out[0] = channel;
new DataView(out.buffer).setUint32(1, length, true);
return out;
}
function frame(channel: number, payload: Uint8Array): Uint8Array<ArrayBuffer> {
const out = new Uint8Array(HEADER + payload.length);
out.set(header(channel, payload.length));
out.set(payload, HEADER);
return out;
}
/** Splits the host's bytes into frames (they arrive in any chunking). */
export class Deframer {
private buf = new Uint8Array(0);
private start = 0;
push(bytes: Uint8Array) {
const kept = this.buf.length - this.start;
const next = new Uint8Array(kept + bytes.length);
next.set(this.buf.subarray(this.start));
next.set(bytes, kept);
this.buf = next;
this.start = 0;
}
next(): { channel: number; payload: Uint8Array } | undefined {
const available = this.buf.length - this.start;
if (available < HEADER) return undefined;
const view = new DataView(this.buf.buffer, this.buf.byteOffset + this.start);
const length = view.getUint32(1, true);
if (available < HEADER + length) return undefined;
const channel = this.buf[this.start];
const payload = this.buf.subarray(this.start + HEADER, this.start + HEADER + length);
this.start += HEADER + length;
return { channel, payload };
}
}
/** Draws the host's video; made per connection. */
export class VideoSink {
readonly canvas: HTMLCanvasElement;
private readonly context: CanvasRenderingContext2D | null;
private decoder?: VideoDecoder;
private codec?: string;
private waitingForKey = true;
private lastSubmitted = -1;
private info?: VideoInfo;
private shown = false;
private off = false;
private recoveries = 0;
private watchdog?: ReturnType<typeof setTimeout>;
private send?: (message: Uint8Array) => void;
constructor(private readonly onInfo: (info: VideoInfo | null) => void) {
this.canvas = document.createElement('canvas');
this.canvas.className = 'exr-video';
this.canvas.style.display = 'none';
this.context = this.canvas.getContext('2d', { alpha: false, desynchronized: true });
}
/** The way back to the host (framed on channel 1). */
attach(send: (message: Uint8Array) => void) {
this.send = send;
}
message(payload: Uint8Array) {
if (this.off) return;
switch (payload[0]) {
case START:
this.start(payload.subarray(1));
break;
case FRAME:
this.frame(payload);
break;
case STOP:
this.stop();
break;
default:
}
}
close() {
clearTimeout(this.watchdog);
this.closeDecoder();
this.send = undefined;
}
private start(json: Uint8Array) {
try {
const start = JSON.parse(new TextDecoder().decode(json)) as { width: number; height: number; encoder: string; hardware: boolean };
this.canvas.width = start.width;
this.canvas.height = start.height;
this.info = { encoder: start.encoder, hardware: start.hardware };
} catch {
this.info = { encoder: 'H.264', hardware: false };
}
this.closeDecoder();
this.waitingForKey = true;
clearTimeout(this.watchdog);
this.watchdog = setTimeout(() => {
if (!this.shown) this.giveUp();
}, FIRST_PICTURE_MS);
}
private frame(payload: Uint8Array) {
const key = (payload[1] & 1) === 1;
const seq = new DataView(payload.buffer, payload.byteOffset).getUint32(2, true);
const data = payload.slice(6);
if (this.waitingForKey && !key) {
this.ack(seq);
return;
}
try {
if (key) {
const codec = codecOf(data);
if (!this.decoder || this.decoder.state === 'closed' || codec !== this.codec) this.configure(codec);
this.waitingForKey = false;
}
this.decoder!.decode(new EncodedVideoChunk({ type: key ? 'key' : 'delta', timestamp: seq, data }));
this.lastSubmitted = seq;
} catch {
this.ack(seq);
this.recover();
}
}
private configure(codec: string) {
this.closeDecoder();
this.codec = codec;
this.decoder = new VideoDecoder({ output: (picture) => this.draw(picture), error: () => this.recover() });
this.decoder.configure({
codec,
optimizeForLatency: true,
// What the host's NV12 conversion uses (BT.709, limited range).
colorSpace: { primaries: 'bt709', transfer: 'bt709', matrix: 'bt709', fullRange: false }
});
}
private draw(picture: VideoFrame) {
const seq = picture.timestamp;
try {
this.context?.drawImage(picture, 0, 0);
} finally {
picture.close();
}
this.recoveries = 0;
if (!this.shown) {
this.shown = true;
this.canvas.style.display = 'block';
this.onInfo(this.info ?? null);
}
this.ack(seq);
}
/** The decoder failed: start again from a key frame (and release the host's frames in flight). */
private recover() {
if (++this.recoveries > MAX_RECOVERIES) {
this.giveUp();
return;
}
this.closeDecoder();
this.waitingForKey = true;
if (this.lastSubmitted >= 0) this.ack(this.lastSubmitted);
this.send?.(new Uint8Array([KEY]));
}
/** This browser cannot show the video: the host goes back to RDP bitmaps for the session. */
private giveUp() {
if (this.off) return;
this.off = true;
this.stop();
this.send?.(new Uint8Array([OFF]));
}
private stop() {
clearTimeout(this.watchdog);
this.closeDecoder();
this.waitingForKey = true;
this.shown = false;
this.canvas.style.display = 'none';
this.onInfo(null);
}
private closeDecoder() {
if (this.decoder && this.decoder.state !== 'closed') {
try {
this.decoder.close();
} catch {
// Already closed.
}
}
this.decoder = undefined;
this.codec = undefined;
}
private ack(seq: number) {
const message = new Uint8Array(5);
message[0] = ACK;
new DataView(message.buffer).setUint32(1, seq >>> 0, true);
this.send?.(message);
}
}
function bytesOf(data: unknown): Uint8Array | undefined {
if (data instanceof ArrayBuffer) return new Uint8Array(data);
if (ArrayBuffer.isView(data)) return new Uint8Array(data.buffer, data.byteOffset, data.byteLength);
return undefined;
}
function startsWith(bytes: Uint8Array, prefix: Uint8Array, length = prefix.length) {
for (let i = 0; i < Math.min(length, bytes.length); i++) if (bytes[i] !== prefix[i]) return false;
return true;
}
/**
* Stands in for the RDP client's WebSocket: what the WASM glue uses of one (events through
* addEventListener, send, close, readyState, binaryType), passing RDP bytes through and video to the sink.
*/
class MuxSocket extends EventTarget {
binaryType: BinaryType = 'blob';
private mode: 'detect' | 'raw' | 'mux' = 'detect';
private head = new Uint8Array(0);
private readonly inbound = new Deframer();
constructor(
private readonly socket: WebSocket,
private readonly sink: VideoSink
) {
super();
socket.binaryType = 'arraybuffer';
socket.addEventListener('open', () => this.dispatchEvent(new Event('open')));
socket.addEventListener('error', () => this.dispatchEvent(new Event('error')));
socket.addEventListener('close', (e) => {
sink.close();
this.dispatchEvent(new CloseEvent('close', { code: e.code, reason: e.reason, wasClean: e.wasClean }));
});
socket.addEventListener('message', (e: MessageEvent<ArrayBuffer | string>) => this.receive(e.data));
sink.attach((message) => {
if (this.mode === 'mux' && socket.readyState === WebSocket.OPEN) socket.send(frame(VIDEO, message));
});
}
get readyState() {
return this.socket.readyState;
}
get url() {
return this.socket.url;
}
get bufferedAmount() {
return this.socket.bufferedAmount;
}
send(data: string | ArrayBufferLike | Blob | ArrayBufferView) {
if (this.mode !== 'mux') {
this.socket.send(data as string | ArrayBuffer | Blob | ArrayBufferView<ArrayBuffer>);
} else if (data instanceof Blob) {
// The RDP client sends Blobs: framed without reading them, so the order stays.
this.socket.send(new Blob([header(RDP, data.size), data]));
} else {
const bytes = bytesOf(data);
if (bytes) this.socket.send(frame(RDP, bytes));
}
}
close(code?: number, reason?: string) {
this.socket.close(code, reason);
}
private receive(data: ArrayBuffer | string) {
if (typeof data === 'string') {
this.deliver(data);
return;
}
let bytes = new Uint8Array(data);
if (this.mode === 'detect') {
const joined = new Uint8Array(this.head.length + bytes.length);
joined.set(this.head);
joined.set(bytes, this.head.length);
if (!startsWith(joined, MAGIC)) {
this.mode = 'raw';
this.deliver(joined);
return;
}
if (joined.length < MAGIC.length) {
this.head = joined;
return;
}
this.mode = 'mux';
bytes = joined.subarray(MAGIC.length);
}
if (this.mode === 'raw') {
this.deliver(bytes);
return;
}
this.inbound.push(bytes);
for (let f = this.inbound.next(); f; f = this.inbound.next()) {
if (f.channel === RDP) this.deliver(f.payload);
else if (f.channel === VIDEO) this.sink.message(f.payload);
}
}
private deliver(payload: Uint8Array | string) {
let data: string | ArrayBuffer | Blob = payload as string;
if (payload instanceof Uint8Array) {
const copy = payload.slice().buffer;
data = this.binaryType === 'blob' ? new Blob([copy]) : copy;
}
this.dispatchEvent(new MessageEvent('message', { data }));
}
}
const routes = new Map<string, VideoSink>();
let installed = false;
/** Replaces `window.WebSocket` once with a constructor that hands routed URLs a {@link MuxSocket}. */
function install() {
if (installed) return;
installed = true;
const Native = window.WebSocket;
function Routing(url: string | URL, protocols?: string | string[]) {
const target = String(url);
for (const [prefix, sink] of routes) {
if (target.startsWith(prefix)) {
routes.delete(prefix);
return new MuxSocket(new Native(url, protocols), sink);
}
}
return new Native(url, protocols);
}
Object.assign(Routing, { CONNECTING: 0, OPEN: 1, CLOSING: 2, CLOSED: 3 });
Routing.prototype = Native.prototype;
window.WebSocket = Routing as unknown as typeof WebSocket;
}
/** The next WebSocket opened to `url` carries video for `sink`; returns a function that withdraws the route. */
export function routeSocket(url: string, sink: VideoSink): () => void {
install();
routes.set(url, sink);
return () => {
if (routes.get(url) === sink) routes.delete(url);
};
}
+11
View File
@@ -0,0 +1,11 @@
import { enqueueSnackbar } from 'notistack';
/** Copies text to the operator's clipboard and says so (a blocked clipboard, e.g. plain HTTP, says that instead). */
export async function copyText(text: string, what = 'Text'): Promise<void> {
try {
await navigator.clipboard.writeText(text);
enqueueSnackbar(`${what} Copied`, { variant: 'success', autoHideDuration: 1500 });
} catch {
enqueueSnackbar('The browser blocked the clipboard.', { variant: 'warning' });
}
}
+5 -5
View File
@@ -138,7 +138,7 @@ public class AgentOffloadTests(BackendTestFixture fixture) : IClassFixture<Backe
var link = upgrade.Headers.Location!;
Assert.StartsWith($"{Endpoint}/{Bucket}/executer/agent/{V2}/Agent.zip?", link.ToString());
var query = Query(link);
Assert.InRange(int.Parse(query["X-Amz-Expires"]), 299, 300); // a second can pass while signing
Assert.InRange(int.Parse(query["X-Amz-Expires"]), 290, 300); // time passes while signing (slow CI runners)
Assert.Contains("X-Amz-Signature", query.Keys);
var signed = DateTime.ParseExact(query["X-Amz-Date"], "yyyyMMdd'T'HHmmss'Z'",
System.Globalization.CultureInfo.InvariantCulture, System.Globalization.DateTimeStyles.AdjustToUniversal | System.Globalization.DateTimeStyles.AssumeUniversal);
@@ -153,7 +153,7 @@ public class AgentOffloadTests(BackendTestFixture fixture) : IClassFixture<Backe
// The lifetime is a setting.
await ConfigureAsync(offload: true, linkSeconds: 120);
var shorter = await client.GetAsync(ApiRoutes.For.AgentPackageBySha(package.Sha256), ct);
Assert.InRange(int.Parse(Query(shorter.Headers.Location!)["X-Amz-Expires"]), 119, 120);
Assert.InRange(int.Parse(Query(shorter.Headers.Location!)["X-Amz-Expires"]), 110, 120);
using var admin = await fixture.SignedInAsync("offload-admin");
var listed = await (await admin.GetAsync(ApiRoutes.AgentVersions, ct)).ReadAsync<List<AgentVersionDto>>();
@@ -232,7 +232,7 @@ public class AgentOffloadTests(BackendTestFixture fixture) : IClassFixture<Backe
var onServer = new Uri(AgentPackageService.PresignedUrl(builtIn, key, "https://control.contoso.com/", expires));
Assert.StartsWith($"https://control.contoso.com{ApiRoutes.S3Store}/{EmbeddedS3Store.Bucket}/executer/agent/1.3.0/Agent.msi?", onServer.ToString());
Assert.InRange(int.Parse(Query(onServer)["X-Amz-Expires"]), 299, 300);
Assert.InRange(int.Parse(Query(onServer)["X-Amz-Expires"]), 290, 300);
Assert.DoesNotContain("127.0.0.1", onServer.ToString());
var elsewhere = new Uri(AgentPackageService.PresignedUrl(External, key, "https://control.contoso.com", expires));
@@ -309,7 +309,7 @@ public sealed class AgentOffloadBuiltInStoreTests(EmbeddedS3StoreTests.Fixture s
Assert.Equal(HttpStatusCode.Found, redirect.StatusCode);
var link = redirect.Headers.Location!;
Assert.StartsWith($"http://edge.example.net:8443{ApiRoutes.S3Store}/{EmbeddedS3Store.Bucket}/executer/agent/{V2}/Agent.zip?", link.ToString());
Assert.InRange(int.Parse(System.Web.HttpUtility.ParseQueryString(link.Query)["X-Amz-Expires"]!), 119, 120); // a second can pass while signing
Assert.InRange(int.Parse(System.Web.HttpUtility.ParseQueryString(link.Query)["X-Amz-Expires"]!), 110, 120); // time passes while signing (slow CI runners)
// It downloads, with no session, through this server.
Assert.Equal("zip v2", await client.GetStringAsync(link.PathAndQuery, ct));
@@ -344,6 +344,6 @@ public sealed class AgentOffloadBuiltInStoreTests(EmbeddedS3StoreTests.Fixture s
var upgrade = await client.GetAsync(ApiRoutes.For.AgentPackageBySha(package.Sha256), ct);
Assert.StartsWith("https://control.contoso.com/", upgrade.Headers.Location!.ToString());
Assert.InRange(int.Parse(System.Web.HttpUtility.ParseQueryString(upgrade.Headers.Location!.Query)["X-Amz-Expires"]!), 299, 300); // a second can pass while signing
Assert.InRange(int.Parse(System.Web.HttpUtility.ParseQueryString(upgrade.Headers.Location!.Query)["X-Amz-Expires"]!), 290, 300); // time passes while signing (slow CI runners)
}
}