Skip to content

Architecture

Windows MCP Server is a local .NET 10 MCP server that exposes Windows desktop automation to AI agents. Its design follows one core idea: let Windows describe the UI semantically, and keep the tools thin.

Semantic first, fallback when needed

The server uses the Windows UI Automation API (UIA) as the primary interaction method — the same accessibility API screen readers use. This gives agents a semantic view of applications: elements are found by name, control type and automation ID rather than by parsing pixels.

Screenshot + mouse + keyboard remain available as a fallback for games, canvas/OpenGL surfaces and other custom-drawn UIs that expose no accessibility tree.

flowchart LR
    A[AI agent] -->|MCP over stdio| B[Windows MCP Server]
    B --> C[Tool layer]
    C --> D[UI Automation API]
    C --> E[Win32 input / window APIs]
    C --> F[Screen capture + OCR]
    D --> G[(Windows desktop)]
    E --> G
    F --> G

Layers

  • Tool layer — 10 focused tools (ui_find, ui_click, ui_type, ui_read, file_save, window_management, screenshot_control, mouse_control, keyboard_control, app). Following the project's "augmentation, not duplication" principle, tools are thin actuators: they expose Windows capabilities without embedding complex logic.
  • Automation — wraps UIA element discovery, matching and actions.
  • Input — keyboard and mouse simulation via the standard SendInput API, with configurable timing.
  • Window — find, activate, move, resize and wait-for window operations.
  • Capture — screen and element capture, annotated screenshots and OCR.
  • Native — P/Invoke bindings to the Win32 and UIA surfaces the tools build on.
  • Prompts — guidance prompts (such as browser automation) surfaced to the client.

Token optimization

Responses are designed for LLM efficiency: short property names, JPEG screenshots and automatic scaling. Annotated screenshots return element metadata with the image omitted by default. In practice this is roughly 60% fewer tokens than standard JSON, which keeps automation loops fast and affordable.

Transport

The server communicates with its client over stdio using the Model Context Protocol via the official .NET SDK. It is launched by the client (VS Code extension, plugin or a standalone MCP config entry) and runs as a local process — see Installation.

LLM-tested quality

Because tool descriptions that read clearly to humans can still confuse a model, every tool is validated with real AI models using pytest-aitest: 54 automated tests with a 100% pass rate required for release. When the AI misuses a tool, the fix is to the tool — not the test prompt. See Contributing for how the test suites are run.