Token measurements¶

This benchmark asks an AI to finish work in Notepad, Word, PowerPoint, and Chrome, then checks the saved result independently. It measures the whole conversation and its Windows actions, not just reading a saved image.

Results¶

The 30 September 2026 run completed all 16 trials. The controls-first route completed 8 of 8 tasks. The screenshot-only route completed 7 of 8. GPT-6 Luna reached the 80-call limit without saving the PowerPoint copy in its screenshot-only trial. That failure remains part of the result.

Across the seven app/model pairs where both routes completed, controls first used a median 34.9% fewer input tokens and took a median 34.3% less framework session time.

This is one run per combination, not a general model ranking. PowerPoint with GPT-6.1 Sol is an important counterexample: controls completed the task but used more tokens and time than screenshots. Token counts measure model input, not price.

Download the privacy-safe evidence. It contains settings, counts, hashes, request-audit metadata, and stop reasons. Raw screenshots, messages, desktop text, and local paths remain private.

Tasks¶

The agent does not get the complete source document text in its task prompt. It must inspect the open application, make the changes, and save or submit them. A claim that the task is finished does not count as success.

Two routes through the same server¶

Controls first: the agent can find, read, click, type into, and select Windows controls. It can request complete or changes-only views and use the file-saving helper. Screenshots, mouse, and keyboard input remain available.

Screenshots: the agent gets unannotated images and uses mouse and keyboard input. It cannot read control names, values, element IDs, or text through the control tools. It must handle save dialogs itself.

Both routes use Windows MCP. This compares two tool sets within the same product, not separate products.

The benchmark adapter restricts actions, window listings, and screenshots to the test-owned application. Batch operations, macros, the clipboard, launching other apps, shell commands, Office APIs, browser page source, and developer tools are unavailable to the agent.

Models and run order¶

The matrix has 16 trials: four apps, two models, and two routes. The model IDs are gpt-6.1-sol and gpt-6-luna. Every trial starts a fresh, tool-restricted model session with medium reasoning, high-detail images, a ten-minute limit, and an 80-tool-call limit.

Route order alternates across app/model pairs. The apps run one at a time on the same reserved desktop. Each pair starts with copies of the same input file. Notepad, Word, and PowerPoint must not already be open. Chrome uses a separate test profile and local booking page.

What is counted¶

Completion comes first. Saved files and submitted form values are checked outside the model session. The original input must remain unchanged.

The framework records the actual model requests, tools, reasoning setting, image detail, usage, stop reason, and admitted tool calls. Unsupported or incomplete request formats stop the run rather than producing incomplete evidence.

Token and time comparisons are calculated only for matching app/model pairs where both routes complete within their limits. Failed attempts remain in the run and are not credited as efficiency wins. One trial per combination is an initial comparison, not a general model ranking.

Framework session time includes model-runtime startup and cleanup. It excludes Windows app setup and the independent saved-result check.

Evidence and privacy¶

Keep the framework JSON, JUnit report, task artifacts, and app checks together in a fresh private directory. Raw reports may contain local paths, screenshots, model messages, or unrelated text visible in dialogs, so do not publish them.

Any future public export must contain only settings, counts, request-audit metadata, and result hashes. It must exclude desktop images, raw window text, model messages, authentication, and local file paths.

Reproduce the real-app comparison¶

Use an unlocked Windows desktop reserved for the benchmark. Install Notepad, Word, PowerPoint, and Chrome. Build the server and authenticate the GitHub CLI or provide GITHUB_TOKEN; never put a token in source files or reports.

Run the no-model checks first:

dotnet build src\Sbroenne.WindowsMcp -c Release
uv run --project tests\usage_evals pytest tests\usage_evals\tests\unit -q
uv run --project tests\usage_evals pytest tests\usage_evals\tests\comparison --collect-only -q

The live command is in the evaluation guide. It requires approval for 16 model sessions and an exclusive desktop. Without that opt-in, the comparison is skipped.

Do not resume retired standalone runners or combine interrupted runs. Optional AI-generated report analysis is a separate paid operation and is not part of the comparison.

All 300 answers were correct. With every model, direct field reads used 51.4% fewer input tokens than low-detail screenshots and 70.5% fewer than high-detail screenshots.

This comparison measures GPT-6 Astra, GPT-6 Luna, GPT-5.6 Sol, and GPT-5.6 Luna reading the same Windows form. Each model receives 25 direct text reads, 25 low-detail screenshots, and 25 high-detail screenshots.

Live AI reading results¶

Each row contains 25 reads. Token counts and response times below are medians per read; seconds include network and service time.

Differences between the models¶

Accuracy and input usage were tied. All four models read every value correctly from text and both image settings. Each used the same input tokens for each matching observation. This task does not distinguish their accuracy on harder forms or other apps.

Response time differed. GPT-5.6 Sol had the lowest median time for direct text; GPT-6 Luna had the lowest for both image settings in these runs. Direct text was quicker than images for three models. GPT-6 Luna was slightly quicker with images, so fewer input tokens did not always mean a quicker answer.

GPT-6 Luna reported more output. Across its 75 reads it used 1,277 output tokens, including 514 reported reasoning tokens. Each other model reported 690 output tokens and zero reasoning tokens. These are service-reported counts, not a claim about how much unreported computation each model performs.

Sample totals¶

Every model completed five samples of five observations for each input type. Each sample used 851 direct-text input tokens, 1,750 low-detail image tokens, and 2,880 high-detail image tokens. The table totals all five samples; no first views or unfavorable results are omitted.

There were exactly 300 model requests: no failed requests, retries, incorrect answers, or malformed answers. None reported cached input or cache writes.

How to interpret the comparison¶

Input tokens are reported by the model service, including the question and system instruction. Direct text includes both the field-discovery reply and the value-reading reply. Savings compare the total input across all 25 reads, including each sample's first view, rather than rounding the median first.

Output tokens include any reasoning tokens reported by the service. They are shown separately from input, not silently added to the input-saving claim. Token counts are not currency costs: models and services can charge differently.

Response time runs from sending the observation to receiving the completed answer. It includes network and service time, but excludes session creation and Windows capture time. Requests run one at a time; models run in the order Astra, GPT-6 Luna, Sol, GPT-5.6 Luna. Input types rotate within each model. Times describe these batches, not a guaranteed model-speed ranking: service load and the time of each batch can affect them.

Method¶

The question and expected answers¶

Every read starts a fresh session with no tools, skills, repository instructions, or previous answers. The question is:

What is the username entered in this form? Reply with JSON only: {"username": "the exact value"}.

The system instruction is identical for every model and input type: Answer the user's question using only the supplied observation. The text route appends both saved JSON replies under Observed form controls:. The image routes attach the original JPEG as form.jpg, without the answer in the filename or prompt. Expected values stay in the scorer.

The five values are Baseline, Alice, Alice Smith, Bob, and Carol. There are five captures of each value: 25 images of one form, not 25 different apps or forms. Images are 700 by 550 pixels. The form also repeats the username in a status heading; the image route can read either visible occurrence. Scoring requires an exact string match in the requested JSON format.

Images, controls, and model settings¶

The runner sets the actual Responses API input_image.detail field to low or high through the Copilot SDK's supported request-handler hook. It checks that the submitted bytes match the saved screenshot's SHA-256 hash. It does not simulate low detail by resizing a file.

All four models use medium reasoning effort, the same question, instructions, and saved inputs. The comparison checks outgoing model IDs, image detail, reported response models, usage, and input hashes. Models are called explicitly, not selected through automatic routing.

The saved images come from Windows MCP's production ScreenshotService: window-only JPEG, quality 60, without cursor or annotations. The full control tree is read before and after each screenshot. Both reads must succeed, be nonempty, contain no truncation warning, and match. This checks control-state stability, not purely visual animations.

Direct reads discover the unique UsernameInput edit control and read its value. Both replies are included every time. Choosing this selector is outside the reading test, just as deciding where to inspect an image is.

Scope¶

This compares reading a field from an observation. It does not include choosing tools, navigating the app, entering values, or completing a whole user task. It is not a comparison of separate products. A full-window control list is not a focused field read and can contain much more information.

Run evidence¶

The live model requests ran on 2026-09-27 through GitHub Copilot, using Copilot SDK 1.0.14 and runtime 1.0.85.

The complete machine-readable evidence contains every answer, expected value, outgoing detail setting, usage count, duration, and image/text hash, plus a checked summary for each model. Each run records its source revision, dirty working-tree state, and SHA-256 of the exact runner. Wrong and malformed answers are retained. Transport failures stop the runner and preserve the incomplete trial rather than silently retrying.

The source captures are from screenshot-winforms-field-editing-9a1e0081953948b1ac39f960ac0932bc, recorded on Windows 10.0.26220.0 and .NET 10.0.12, Release build with SDK 10.0.401. Source was an uncommitted working tree based on 67ba78d6b3a99bdabee27898dab64b38aafb55ed, including the benchmark and the branch's automation changes, not a released package.

Raw screenshots and response JSON remain local; the commands below produce fresh captures. Control identifiers can vary between capture runs, so text counts can vary slightly. All four model runs here use the same saved captures.

Reproduce¶

Capture the form¶

Use an unlocked Windows desktop and the test prerequisites in the testing instructions. Do not use the desktop while this capture runs. These commands do not make paid model requests.

$env:MCP_SNAPSHOT_BENCHMARK_OUTPUT = Join-Path $env:TEMP "windows-mcp-screenshot-benchmark"
dotnet build tests\Sbroenne.WindowsMcp.Tests -c Release
dotnet test tests\Sbroenne.WindowsMcp.Tests -c Release --no-build --filter "FullyQualifiedName~Unit.ScreenshotComparisonBenchmarkTests|FullyQualifiedName~Unit.SnapshotBenchmarkRunnerTests"
dotnet test tests\Sbroenne.WindowsMcp.Tests -c Release --no-build --filter "FullyQualifiedName~Benchmark_ScreenshotVersusControls_WinForms" --logger "console;verbosity=detailed"

A completed capture writes report.md, observations.json, and a JPEG plus response JSON for each observation. Use its uniquely named folder below. The live comparison uses actual model usage, not the capture report's size estimates.

Model reads¶

The published 300 reads remain historical evidence. Their standalone SDK runner is retired; the retained Python helpers only read and validate saved evidence. Do not use that script to start more paid requests.

New model-driven comparisons must run through pytest-skill-engineering. The current live suite measures complete real-app tasks, not another set of single-field reads. Reintroducing live single-field reads requires a framework-based test and separate approval of its model-call budget. The summary command rejects incomplete runs, duplicate cases, mismatched inputs/settings, and inconsistent usage or scoring.

Offline checks make no model requests:

python -m unittest discover -s scripts\tests -p "*readability*.py" -v
python -m unittest discover -s gh-pages\tests -v

The changes-only measurement compares text updates with full text. It measures a separate capability, not another saving to add to the screenshot comparison.

A snapshot describes the controls in a window. Automatic snapshots can send just the changes since the last view, rather than repeating the whole window.

The server still reads the complete window before comparing it. It sends changes only when they are safe to apply and take less than 80% of a complete automatic response. Otherwise it sends a complete view. Explicit mode=full requests always return the full tree of controls.

Recorded measurements: 27 September 2026¶

These runs used source revision 67ba78d6b3a99bdabee27898dab64b38aafb55ed, Release mode, .NET SDK 10.0.401, and Windows 10.0.26220.0. The Word run includes the document-startup fix in this change: wait for the document window, not the temporary "Opening" window. The Chrome run used an earlier implementation that removed a selected address-completion suggestion inside the typing tool. Current typing reports that mismatch without changing it. The benchmark caller now explicitly checks the selected suggestion before rejecting it and verifies the final URL. The Chrome figures below are retained as earlier snapshot measurements, not validation or timing measurements of the current typing workflow. The tests ran one at a time on a Windows desktop, using temporary test content.

Chrome completed all 15 runs: five samples in each of the three modes, covering 60 page changes. All 20 measured automatic replies were complete views, not changes-only replies. Their 11.3% token saving came from the more compact automatic output. This page-to-page workload behaves differently from small edits within the same window.

What the percentages mean¶

Each workload runs five times in each of three modes:

1. Action-only: perform four actions without taking snapshots. This measures action time, not information savings.
2. Full: request a complete snapshot after each action.
3. Auto: take one initial snapshot, then request changes after each action. The initial snapshot is not included in the measured totals.

For each auto run, the test also saves the same captured controls as full responses. The savings compare those two versions of the same captures. The reported percentage is the median of the five run-level percentages. This avoids treating differences between separate live webpages as savings.

Tokens are the small pieces of text an AI model processes. Counts here use SharpToken's cl100k_base tokenizer as an approximation. They do not measure the cost of a whole agent conversation, and other models may count differently.

Timing and response sizes¶

These are median totals for four actions or snapshots, not per-call times.

Smaller responses do not necessarily mean faster capture. Timing depends on the application, its startup state, and the machine's other work.

Workloads¶

• Electron: move through Forms, Data, Settings, and Home in the project's test application. This does not measure all Electron applications.
• Chrome: visit Issues, Pull requests, Actions, and Code in the public microsoft/vscode GitHub repository, using a separate browser profile. The address bar is discovered during setup and its exact ID is reused. The current benchmark caller handles a verified selected URL suggestion explicitly; it does not repeat a failed navigation. Setup observations are outside the measured actions. Measured snapshots include the webpage, not just the browser's toolbar.
• Word: edit, append text, undo, and edit a temporary RTF document.
• Excel: enter four values in a temporary CSV file. This does not test formulas, ribbon commands, or every Excel feature.

The order of the three modes rotates between samples. Each starts with equivalent test content. Response sizes use the server's normal JSON output.

Chrome startup checks read the window once per attempt and require the webpage controls to appear. Popup handling uses buttons already found in that view, rather than searching repeatedly for popups that may not exist. Those searches could previously consume the entire startup waiting time. Startup failures now record the timing of each attempt and the last window view.

Raw samples¶

Each row covers four actions. The "Same-capture full" columns are the complete versions used to calculate auto savings.

Electron 44.2.0¶

Excel 16.0.20326.20158¶

Chrome 154.0.8037.57¶

Word 16.0.20326.20158¶

CLI continuity check¶

Separate wincli commands can share the same remembered view. Pass the returned snapshotToken as --since on the next automatic snapshot. A missing or outdated token returns a complete view.

On 27 September 2026, a Release-mode check against the project's Windows test app returned:

That unchanged-window response was 95.7% smaller. Missing and interleaved tokens returned complete views as required. Both continuity tests passed, including rejecting an old ID when a control with the same label replaced it. The test's CLI service was stopped afterward.

These are single-check measurements, not a multi-application benchmark or a cost estimate. The first call includes startup only if the service was not already running.

Reproduce¶

Run on a Windows desktop that the tests can use without interruption. They take control of the mouse and keyboard. Chrome, Word, and Excel must be installed for their workloads. Tests explicitly skip an unavailable application.

Run the workloads one at a time:

$project = '.\tests\Sbroenne.WindowsMcp.Tests\Sbroenne.WindowsMcp.Tests.csproj'
$env:MCP_TEST_CHROME = '1'
$env:MCP_SNAPSHOT_BENCHMARK_OUTPUT = "$env:TEMP\mcp-windows-snapshot-benchmark"

dotnet build $project -c Release
dotnet test $project -c Release --no-build --filter 'FullyQualifiedName~SnapshotBenchmarkRunnerTests'
dotnet test $project -c Release --no-build --filter 'FullyQualifiedName~ElectronSnapshotBenchmarkTests'
dotnet test $project -c Release --no-build --filter 'FullyQualifiedName~Benchmark_PublicGitHubRepositoryWorkflow_Chrome'
dotnet test $project -c Release --no-build --filter 'FullyQualifiedName~Benchmark_RealOfficeWorkflow&DisplayName~Word'
dotnet test $project -c Release --no-build --filter 'FullyQualifiedName~Benchmark_RealOfficeWorkflow&DisplayName~Excel'
dotnet test $project -c Release --no-build --filter 'FullyQualifiedName~CliSnapshotContinuityTests'

Each benchmark writes a Markdown report with medians and raw samples to MCP_SNAPSHOT_BENCHMARK_OUTPUT.

Limits¶

Live webpages, application versions, network conditions, and machine load vary. These results describe the listed runs, not a guaranteed saving for every task. Large changes may need a complete view. Smaller responses do not necessarily mean faster capture or lower total agent costs.