Incremental UI Snapshot Benchmark¶

Incremental snapshots exist to reduce the repeated UI context sent to an agent. A complete accessibility tree is still captured on every request, but mode=auto compares it with the remembered semantic tree and returns kind=diff only when the change list is safe and less than 80% of the complete semantic response. Otherwise it returns kind=full with a complete simplified view. Explicit mode=full calls remain unchanged. This preserves correctness while making payload savings workload-dependent rather than guaranteed.

The full response contains the documented compact tree and no longer serializes the former redundant full-detail elements copy. Consumers that relied on that duplicate should migrate to tree, or call ui_find for a flat result.

Result¶

Four representative workflows were run five times per comparison arm on Windows 10.0.26220.0. The browser pages were the public microsoft/vscode GitHub repository, not a synthetic TodoMVC page.

Chrome uses the median of five paired run-level reductions: every automatic run is compared with the complete trees captured by those same requests. This prevents live GitHub variation between separately launched arms from becoming fake savings. The deterministic Electron and Office rows use their full-arm medians, whose payloads vary negligibly. The final row averages the four workload percentages so each workflow has equal weight. Tokens are a SharpToken cl100k_base approximation, not universal model billing tokens.

The measured benefit is strong but not universal. Electron, Word, and Excel produced a diff after every action, reducing median payloads by 84-96%. Chrome returned two diffs and 18 complete simplified responses. Removing layout-only containers from those complete automatic responses improved Chrome from the previous 3.9% byte reduction to 13.1%, and from 3.6% to 13.4% for approximate tokens. Across the four equally weighted workloads, the average reductions were 70.6% and 71.4%. A separate regression confirms that a same-page GitHub search-field edit returns a scoped diff in both Edge and Chrome; it is not included as another benchmark workload.

Further Chromium cleanup¶

A second five-run Chrome experiment measured conservative display cleanup separately from the semantic wrapper removal above. Readiness now requires an exact page control returned by Windows UI Automation; a partial match against the browser tab title no longer counts as a ready webpage. The run used Chrome for Testing 151.0.7922.174 and the same four GitHub destinations.

The first row is the decision metric: cleanup alone exceeded the 5% keep threshold. It removes click coordinates from leaf items that Windows says cannot be acted on. It also removes a child label that exactly repeats its parent's label, and blank leaf images, but only when they have no action, state, developer identifier, or children. Readable text, values, toggle state, controls, and element IDs remain. Explicit mode=full output is unchanged.

All 20 responses were complete automatic views because each action navigated to a different page. That makes this a useful worst case: the savings do not depend on a navigation being mistaken for a small update. Each automatic capture was also serialized before display cleanup, so live page variation cannot become fake savings.

Two other Chromium experiments were rejected:

• Page-only snapshots: Chrome 152 exposed the address-bar popup as a webpage root in one run, while Edge 152 exposed no dependable webpage root. A screen-position fallback could select another application covering the page. Because this could return the wrong content, page-only scope is not shipped and whole-window snapshots remain the default.
• Cache-only Windows elements: asking Windows to return cached properties without live automation objects made the Electron safety test fail with 0x80004005. The speed comparison was stopped because correctness had already failed; normal cached tree capture remains in use.

Chromium noise spike¶

We tested whether Windows UI Automation's smaller "content view" could remove Chromium layout noise before snapshots were compared. The same GitHub navigation workflow was run five times per arm with that view enabled.

This did not produce a single diff. More importantly, a safety check found that the content-view tree contained the browser frame but omitted GitHub's page controls, including the Code link, in both Edge and Chrome. Chrome's median full payload was about 6% smaller than the original control view, but the missing page made that reduction unusable. The experiment was therefore rejected and is not enabled in production. Edge's live tree differed too much between runs to make a reliable size comparison.

Playwright's ARIA snapshot implementation suggested the safer direction now used. It creates a small role, name, text, and state tree rather than comparing raw browser nodes. Its distiller joins adjacent text, normalizes whitespace, removes empty text, and unwraps low-information layout containers with one child. Action references are handled separately and are renewed after navigation. Playwright's loose role-and-name matching is suitable for test assertions, but not for carrying an action ID across duplicate controls.

We first tested post-capture cleanup rather than Windows content-view filtering. That experiment unwrapped only unnamed one-child Pane and Group containers when the wrapper and child had the same bounds, visibility, and enabled state, the wrapper had no developer ID, and Windows reported no supported action pattern. It preserved GitHub's Code and Issues controls in both browsers, but still produced 40 complete responses and no diffs. Checking action patterns also added provider calls to each candidate wrapper. The cleanup was rejected because it added work without improving incremental responses.

The production approach instead keeps an internal comparison view and a smaller response view from the same capture. Explicit full mode retains the complete Windows accessibility tree and all IDs. Automatic mode removes unnamed Pane and Group containers only when they have no developer ID and no direct Invoke, Expand/Collapse, Selection, Toggle, or Value action. It then applies the conservative display cleanup measured above. Chromium's widely reported ScrollItem capability merely brings a node into view, so it does not make an otherwise anonymous wrapper a user-facing action. Named controls and direct action references remain in the tree, and uncertain duplicate controls still force a complete response. A live Chrome test checks that GitHub page controls survive this projection.

This experiment also exposed a benchmark problem. Chromium's recommended depth of 15 reached the browser frame but not GitHub's page controls. The browser scenarios now explicitly use depth 20 and wait until a real page control appears before each measured snapshot. An integration test verifies that GitHub's Code control is present. The browser and aggregate results in this document are the corrected page-content measurements.

Method¶

Each scenario executes the same four state changes under three arms:

1. Action-only control performs the actions without a snapshot. It is a latency floor only and is never the payload-savings denominator.
2. Full baseline requests mode=full after every action.
3. Automatic treatment establishes an unmeasured baseline with mode=reset, then requests mode=auto after every action. Production logic decides whether each response is a diff or a safe full fallback.

Every arm starts from equivalent application content. Browser and Office scenarios launch isolated processes and temporary state; Electron resets its dedicated test harness. Arm order rotates between samples to limit warm-cache sequence bias. Measurements serialize the actual UIAutomationResult with the production JSON options. Action and snapshot elapsed time are recorded separately.

The workflows are:

• Electron: navigate the real Electron harness through Forms, Data, Settings, and Home.
• Chrome: navigate the public microsoft/vscode GitHub repository through Issues, Pull requests, Actions, and Code in an isolated browser profile. The scenario uses depth 20 so the snapshot includes the webpage, not only the browser frame. Edge and Chrome share Chromium, so the expensive benchmark uses Chrome only; short live regressions still cover both because their Windows accessibility output is not identical.
• Word: edit, append to, undo in, and edit a dedicated temporary RTF document.
• Excel: enter four values into a dedicated temporary CSV workbook.

These are median totals for four actions or snapshots, not per-call values. Automatic mode still captures a complete accessibility tree before comparing it, so it is designed to reduce response payload and agent context, not capture time.

Raw samples¶

Each row is one complete four-action run. Full/diff counts the response kinds returned during that run.

Electron¶

Chrome¶

Word¶

Excel¶

Reproduce¶

The benchmarks require an interactive Windows desktop. Chrome is opt-in, and Word or Excel tests skip explicitly when the corresponding desktop application is unavailable. Run the workloads sequentially because they share the foreground desktop:

$project = '.\tests\Sbroenne.WindowsMcp.Tests\Sbroenne.WindowsMcp.Tests.csproj'
$env:MCP_TEST_CHROME = '1'
# Optional: pin an official build when the installed Chromium version does not expose page controls.
# $env:MCP_TEST_CHROME_PATH = 'C:\path\to\chrome.exe'
$env:MCP_SNAPSHOT_BENCHMARK_OUTPUT = "$env:TEMP\mcp-windows-snapshot-benchmark"

dotnet build $project
dotnet test $project --no-build --filter 'FullyQualifiedName~ElectronSnapshotBenchmarkTests'
dotnet test $project --no-build --filter 'FullyQualifiedName~Benchmark_PublicGitHubRepositoryWorkflow_Chrome'
dotnet test $project --no-build --filter 'FullyQualifiedName~OfficeSnapshotBenchmarkTests&DisplayName~Word'
dotnet test $project --no-build --filter 'FullyQualifiedName~OfficeSnapshotBenchmarkTests&DisplayName~Excel'

Each test writes a Markdown report containing medians and raw samples to MCP_SNAPSHOT_BENCHMARK_OUTPUT.

Limitations¶

• Live GitHub content, network conditions, application builds, accessibility trees, and machine load vary. This benchmark characterizes these runs; it is not a fixed performance promise.
• Chrome's live accessibility tree varied substantially between runs. The safe fallback prevented an oversized or structurally unsafe diff from being returned; the semantic projection reduces these complete automatic responses without pretending that a full-page navigation was a small change.
• Office actions edit temporary local files through keyboard input. They do not exercise every ribbon, dialog, formula, or workbook feature.
• Electron uses the repository's deterministic harness rather than a large third-party Electron application, but its navigation changes real renderer accessibility state.
• Payload savings do not imply snapshot-latency savings because automatic mode must still capture the current full tree before computing the response.