Complete Feature Reference¶

🎯 The Approach: Semantic First, Fallback When Needed¶

Windows MCP uses the Windows UI Automation API as the primary interaction method. This gives AI agents semantic understanding of applications — finding elements by name, type, and state rather than parsing screenshots.

Token Optimization¶

All tool responses are designed for LLM efficiency, minimizing token usage while preserving information:

Example response comparison:

// Standard JSON (~180 tokens)
{ "success": true, "errorCode": "success", "message": "Clicked element", "element": { "name": "Save", "controlType": "Button", "handle": "123" } }

// Optimized JSON (~60 tokens)
{ "ok": true, "ec": "success", "msg": "Clicked", "el": { "n": "Save", "t": "Button", "h": "123" } }

This reduces LLM costs by ~60% and improves response times when processing tool results.

LLM Testing & Validation¶

Every tool is tested with a real AI model (GPT-5.5 via GitHub Copilot) using pytest-skill-engineering to ensure LLMs understand tool descriptions and use them correctly.

130+ LLM tests run against GPT-5.5 through the dedicated manual LLM Integration Tests workflow.

Why LLM testing matters:

• Tool descriptions must be LLM-friendly — If the AI misunderstands a parameter, it fails silently
• Response formats affect reasoning — Structured hints guide the LLM to correct next steps
• Edge cases surface quickly — Real models find ambiguities that unit tests miss

LLM tests are intentionally manual-only and never run as part of PR, CI, or release workflows. See CONTRIBUTING.md for how to run them.

When to Use Each Tool¶

Browser Automation¶

• Edge, Chrome, and other Chromium apps are auto-detected and searched with the deeper Chromium strategy.
• Launch with app(programPath='msedge.exe', arguments='https://example.com') or find an existing browser window first.
• Page links, buttons, and form fields usually surface visible text or ARIA labels as the UIA name, so start with ui_find, ui_click, and ui_type.
• For authenticated or SSO-only sites, prefer reusing an already-open signed-in Edge/Chrome window first. A Chromium launcher helper exiting immediately is often normal existing-session behavior, so check the browser window before retrying the launch.
• Keep discovery compact: screenshot_control already returns annotated element metadata without image bytes unless you opt in.
• For browser chrome like the address bar or tab switching, prefer shortcuts such as Ctrl+L, Ctrl+R, and Ctrl+Tab.
• Treat browser chrome and non-Chromium browsers as best-effort until dedicated browser coverage expands beyond the Electron/Chromium harnesses.
• The Chromium smoke slice now runs by default: deterministic local Edge/Chrome test pages plus a required public-web smoke check against https://demo.playwright.dev/todomvc/.
• Chromium browser coverage stays on the same semantic-first model as Electron: deep Chromium tree search, ARIA/visible-text discovery, and no separate browser-only tool family.
• The deterministic Chromium smoke harness launches Edge and Chrome app windows with isolated browser state (--user-data-dir), forces renderer accessibility, waits for page-owned readiness signals, and only uses narrow popup dismissal as a fallback so browser-owned UI does not mask real page interaction results.

Tools Overview¶

Two ways to call these tools¶

Every tool listed above is available through two equal entry points that share one implementation:

• MCP server — the tool schemas documented in this file (for MCP hosts).
• wincli CLI — the same tools as shell commands (for coding agents with terminal access). The CLI calls the exact same tool methods, so its JSON output is byte-for-byte identical to the MCP tools. It is the token-efficient path: discover everything via wincli --help / wincli tools / wincli guidance instead of loading every MCP schema. Command mapping mirrors the tool names, e.g. ui_click → wincli ui click, window_management → wincli window <action>, screenshot_control → wincli screenshot. See src/Sbroenne.WindowsMcp.Cli/README.md for the full command reference.

� App (app)¶

Launch applications and get their window handles for subsequent operations.

Parameters¶

Capabilities¶

• Launch applications by name or full path
• Automatic window detection after launch
• Returns window handle for use with other tools
• Configurable startup parameters

Example¶

app(programPath='notepad.exe') → handle='123456'
ui_type(windowHandle='123456', text='Hello World')

�🔍 UI Find (ui_find)¶

Find and discover UI elements by name, type, or automation ID.

Parameters¶

Capabilities¶

• Find elements by name, control type, or automation ID
• Partial name matching with nameContains
• Regex pattern matching with namePattern
• Sort results by prominence (largest first) for disambiguation
• Returns element IDs for use with other ui_* tools
• Electron app support (VS Code, Teams, Slack)

🖱️ UI Click (ui_click)¶

Click buttons, tabs, checkboxes, and other interactive elements.

Parameters¶

*Selectors are optional; without one, the first actionable match in the target window is used.

Capabilities¶

• Click buttons, tabs, menu items
• Toggle checkboxes and toggle buttons
• Handles various control patterns automatically
• Falls back to coordinate-based click if pattern fails
• doubleClick=true double-clicks an element by name/id - no coordinates needed for list/grid items that open on double-click. UI Automation has no double-click pattern, so this is always a physical double-click at the element's clickable point.

⌨️ UI Type (ui_type)¶

Type text into edit controls and text fields.

Parameters¶

Capabilities¶

• Type text into any editable control
• Clear existing content before typing with clearFirst=true
• Append text to existing content
• Unicode support for any language

📖 UI Read (ui_read)¶

Read text from elements using UI Automation or OCR.

Parameters¶

Capabilities¶

• Extract text from any UI element
• Automatic OCR fallback for custom-rendered text
• Windows.Media.Ocr for local text recognition
• Language support for international text
• Article mode (format: "article") for web pages in Edge/Chrome: returns the main content only — navigation chrome, breadcrumbs, and "in this article" rails are dropped, inline link URLs are stripped (visible link text is kept), and headings/lists are emitted as lightweight markdown. Because it reads the live signed-in browser window via UI Automation, it also works for authenticated/internal pages that an HTTP fetch cannot reach.

🧮 UI Read Table (ui_read_table)¶

Extract a grid, table, or details-view list into structured rows — no OCR, screenshot parsing, or cell-by-cell ui_read loops. Uses the native UIA Grid/Table patterns, so WinForms DataGridView, WPF DataGrid, and Win32 ListView (Details view) all return clean row/column data.

Parameters¶

If no selector matches an element that exposes the Grid pattern, the first grid-capable descendant of the target (or window root) is used automatically.

Response shape¶

{
  "success": true,
  "table": {
    "rowCount": 5,
    "columnCount": 5,
    "headers": ["ID", "Product Name", "Price", "Stock", "Available"],
    "rows": [["P001", "Laptop Pro 15", "1299.99", "12", "True"]],
    "truncated": true
  }
}

headers is omitted when the grid exposes no Table pattern; truncated is present only when rowCount exceeds the returned rows (raise maxRows to fetch the rest).

Capabilities¶

• One call instead of N×M ui_read calls to scrape a grid
• Column headers via the UIA Table pattern when available
• Row/column caps keep large grids within an agent's token budget
• Fails cleanly with a recovery hint when the target isn't a grid

🌳 UI Snapshot (ui_snapshot)¶

Capture a compact tree ("snapshot") of a window. This is the orient primitive: call it first on an unfamiliar window instead of guessing selectors or relying on screenshots. The response is hierarchical, depth-bounded, and token-optimized.

Parameters¶

Capabilities¶

• One call to see what's on screen with ids, names, types, click coordinates, and available value/toggle state
• Revisit a known part of a large window via parentElementId
• Prune noise with controlTypeFilter
• Feed returned ids straight into ui_click, ui_type, ui_read, ui_wait
• Use mode=full for a one-time inspection.
• Use mode=auto from the first check when the task will inspect the same window or known subtree again. full is not remembered. The first automatic response is complete; later responses contain only changes when that is clearly smaller.
• Use mode=reset when starting a new comparison. Separate wincli commands start fresh and safely return a complete view.

Savings depend on how stable an application's accessibility tree is. The four-workload benchmark measured 84-96% median byte/token savings for Electron, Word, and Excel changes. A Playwright-style semantic view improved realistic Chrome navigation to 13.1% fewer bytes and 13.4% fewer approximate tokens even though 18 of 20 responses were complete simplified views. Short live regressions still cover both Chrome and Edge because their Windows accessibility output is not identical. A later strict Chrome run measured the additional conservative display cleanup separately: 10.6% fewer bytes and 13.6% fewer tokens.

Snapshot response compatibility: Complete snapshots still return the compact tree, but no longer serialize the redundant full-detail elements copy. Consumers that read that former duplicate should migrate to tree, or call ui_find when they need a flat result.

Response examples¶

The first automatic view and a reset both return a complete tree:

{"success":true,"kind":"full","tree":[{"id":"1","name":"Window","type":"Window","click":[400,300,0],"enabled":true}]}

A useful update returns only the change:

{"success":true,"kind":"diff","changes":[{"op":"add","key":"root/Menu:File#0","node":{"id":"7","name":"File","type":"Menu","click":[50,20,0],"enabled":true}}]}

No change is explicit and very small:

{"success":true,"kind":"diff","changes":[]}

If a change list would be close to the complete response size, the server safely returns kind="full" instead.

🎚️ UI Select (ui_select)¶

Select a value in a combo box, drop-down, list box, or tab control using the proper UI Automation selection patterns (SelectionItem/ExpandCollapse) for cross-framework reliability.

Parameters¶

Capabilities¶

• Reliable selection without click-then-click guesswork
• Auto-expands drop-downs when needed
• Works across Win32, WinForms, WPF, WinUI, and browser controls

⏳ UI Wait (ui_wait)¶

Wait until a UI condition is met before continuing - no blind sleeps or screenshot polling. Uses efficient exponential-backoff polling internally.

Parameters¶

Capabilities¶

• Wait for dialogs/controls to appear before acting
• Wait for spinners/progress dialogs to disappear
• Wait for a specific element to become enabled/visible/toggled

🧩 UI Batch (ui_batch)¶

Run a sequence of UI automation steps against a window in a single call. Built for coding agents: a multi-field form fill + submit that would otherwise take many ui_type/ui_click round-trips becomes one request.

Parameters¶

Step actions¶

Each step is a JSON object with an action plus the fields that action needs:

• find - selectors; resolves an element and exposes its id to the next step as $prev
• click - selectors or elementId, optional doubleClick
• type - selectors or elementId, plus text (optional clearFirst)
• select - selectors, plus value (visible option text)
• wait - mode (appear/disappear/state), selectors or elementId+desiredState, optional timeoutMs
• read - selectors or elementId (or neither, to read the whole window), optional includeChildren
• snapshot - capture the window element tree (optional maxDepth)
• key - key (e.g. enter, tab, f5) with optional modifiers (ctrl,shift,alt,win) and repeat
• mouse - mouseAction (move/click/double_click/right_click/middle_click/drag/polyline/scroll/get_position) plus x,y (and endX,endY for drag), optional button, modifiers, direction, amount
• polyline - points as [[x1,y1],[x2,y2],...], drawn as ONE continuous stroke; optional button

Mouse steps¶

Mouse coordinates are window-relative by default (they inherit the batch windowHandle). Set target (primary_screen/secondary_screen) or monitorIndex on a step for screen-relative coordinates instead.

The target window is activated before each mouse step, so a batch cannot fail with "Could not retrieve foreground window information". Setting expectedProcessName or expectedWindowTitle on a step suppresses that auto-activation and verifies the foreground window instead - the step fails cleanly if it doesn't match, and stopOnError halts the batch. Set activate explicitly to force either behavior.

polyline presses once at the first point, traces every vertex, and releases at the last - a single continuous stroke. This is better than N segment-drags, which lift the pen at every vertex, and costs one round-trip instead of N.

[{"action":"click","name":"Pencil"},
 {"action":"polyline","points":[[300,200],[500,200],[500,400],[300,200]]},
 {"action":"mouse","mouseAction":"drag","x":600,"y":200,"endX":700,"endY":400}]

Capabilities¶

• One round-trip for multi-step workflows (fill username + password + submit) and for multi-stroke drawing
• Per-step results: { index, action, success, summary, error?, elementId?, text? }
• Chain steps by referencing the prior step's element with elementId: "$prev"
• stopOnError=false runs every step and reports each outcome
• Mouse steps delegate to the same engine as mouse_control, so monitor resolution, foreground guards, secure-desktop and elevation checks behave identically

Perceive/act fusion (withSnapshot)¶

ui_click, ui_type, ui_select, ui_batch, and ui_macro accept withSnapshot=true. By default they attach the complete window tree as postActionTree, preserving existing behavior. Set snapshotMode=auto to receive postActionChanges when a remembered update is clearly smaller.

💾 File Save (file_save)¶

Save files via Save As dialog. Handles the entire save workflow: triggers save, waits for dialog, fills path, confirms. English Windows only (detects English dialog titles and button text).

Parameters¶

Capabilities¶

• Trigger Ctrl+S to save
• Auto-detect Save As dialog appearance
• Fill in filename automatically
• Handle overwrite confirmation dialogs
• Works with Office apps, Notepad, and more

📂 File Open (file_open)¶

Open an existing file via the standard Windows Open dialog — the counterpart to file_save. Sends Ctrl+O, waits for the Open dialog, types the path into the File name field, and clicks Open. English Windows only; the file must already exist so the operation is deterministic and never hangs on a "file not found" prompt.

Parameters¶

Capabilities¶

• Trigger Ctrl+O to open
• Auto-detect Open dialog appearance (shares the Save-As dialog engine)
• Fill in the file path and click Open
• Validates the file exists up front for deterministic behavior

📋 Clipboard (clipboard)¶

Read and write the Windows text clipboard — often the fastest way to move bulk text in and out of desktop apps, far cheaper than typing character-by-character or OCR.

Parameters¶

Capabilities¶

• Pull text OUT of an app: focus it, keyboard_control(key='c', modifiers='ctrl'), then clipboard(action='get')
• Push text INTO an app: clipboard(action='set', text='...') then keyboard_control(key='v', modifiers='ctrl')
• get returns text, length, and hasText
• Uses the raw Win32 clipboard API on a dedicated STA thread (no message-pump dependency)

🔁 UI Macro (ui_macro)¶

Record and replay reusable UI workflows. A macro is a saved ui_batch steps array; running one replays it through the identical batch engine, so a macro run behaves exactly like the equivalent inline ui_batch call. Macros persist on disk across sessions.

Parameters¶

Capabilities¶

• save: build and verify a sequence with ui_batch, then persist it under a name
• run: replay a saved macro against any window (same $prev chaining and stopOnError semantics as ui_batch)
• list / get / delete: manage saved macros
• Turns a repeated multi-step task (open a form, fill fields, submit) into a single named call

⚙️ Process Management (process)¶

List and terminate running processes, task-manager style — useful for finding a hung application before automating it, or freeing resources after a workflow.

Parameters¶

Capabilities¶

• list returns pid, name, and memoryMb per process, ordered for token economy
• kill by pid (precise) or by name (all matches)
• Critical Windows processes (System, csrss, wininit, winlogon, services, lsass, …) and the automation server's own process are protected — the tool refuses to terminate them
• CPU usage is intentionally not reported (an accurate percentage requires sampling over time, which would add latency)

🖱️ Mouse Control (mouse_control)¶

Control mouse input on Windows with full multi-monitor and DPI awareness.

Actions¶

Parameters¶

Capabilities¶

• Click, double-click, right-click, middle-click
• Move cursor to absolute coordinates
• Drag operations with hold/release
• Continuous multi-point strokes via polyline - press once, trace every vertex, release once. Unlike N separate drags there is no pen lift at the vertices, so freehand shapes render as one stroke.
• Scroll up/down/left/right
• Multi-monitor support with DPI awareness
• Easy targeting with target='primary_screen' or 'secondary_screen'
• Modifier key support (Ctrl+click, Shift+click, etc.)
• Wrong window detection with expectedWindowTitle / expectedProcessName

⌨️ Keyboard Control (keyboard_control)¶

Control keyboard input on Windows with Unicode support.

Actions¶

Supported Keys¶

Function Keys¶

f1 through f24

Navigation¶

up, down, left, right, home, end, pageup, pagedown, insert, delete

Control¶

enter, tab, escape, space, backspace

Modifiers¶

ctrl, shift, alt, win

Media¶

volumemute, volumedown, volumeup, mediaplaypause, medianexttrack, mediaprevtrack, mediastop

Special¶

copilot (Windows 11 Copilot+ PCs)

Browser¶

browserback, browserforward, browserrefresh, browserstop, browsersearch, browserfavorites, browserhome

Capabilities¶

• Unicode text typing (layout-independent) - type any character in any language
• Virtual key presses - Enter, Tab, Escape, F1-F24, navigation keys
• Key combinations - Use press with modifiers parameter: press(key='s', modifiers='ctrl') for Ctrl+S
• Key sequences - multi-key macros with configurable timing
• Hold/release keys - for Shift-select and other hold operations
• Special keys - Copilot key (Windows 11), media controls, browser keys
• Layout detection - query current keyboard layout (BCP-47 format)
• Clear before typing - use clearFirst to select all (Ctrl+A) before typing new text
• Wait for idle - wait for keyboard input to be processed before continuing

🪟 Window Management (window_management)¶

Control windows on the Windows desktop. Use app tool to launch applications, then use this tool to manage the windows.

Actions¶

Close with discardChanges¶

Use discardChanges=true to automatically dismiss "Save?" dialogs when closing:

window_management(action='close', handle='123456', discardChanges=true)

English Windows only — detects English button text like "Don't Save".

Capabilities¶

• List all visible top-level windows with titles, handles, process info, and bounds
• Locate windows by title (substring or regex matching)
• Bring windows to foreground with focus
• Minimize, maximize, restore, and close windows
• Position and size windows with move, resize, or set_bounds
• Wait for a window to appear with configurable timeout
• Move windows between monitors
• Full multi-monitor support with DPI awareness
• Proper UWP/Store app detection and handling
• Cloaking detection to filter out virtual desktop and shell-managed windows

📸 Screenshot Capture (screenshot_control)¶

Capture screenshots on Windows with LLM-optimized defaults. By default, screenshots include annotated element overlays with numbered labels and structured element data — perfect for UI discovery.

Actions¶

Each list_monitors entry includes displayNumber, width/height, x/y, isPrimary, effectiveDpi, scale (e.g. 1.5 = 150%), orientation (landscape/portrait), and workArea (the desktop rectangle minus the taskbar — handy for placing windows where the taskbar won't cover them).

Capture Targets¶

Parameters¶

Annotated Screenshot Response¶

When annotate=true (default), the response includes structured element data. Image is omitted by default (includeImage=false) to save ~100K+ tokens:

{
  "success": true,
  "annotated_elements": [
    { "index": 1, "element_id": "...", "name": "File", "control_type": "MenuItem", "clickable_point": { "x": 50, "y": 30 } },
    { "index": 2, "element_id": "...", "name": "Edit", "control_type": "MenuItem", "clickable_point": { "x": 100, "y": 30 } }
  ],
  "element_count": 25
}

Use case: When you don't know element names, capture an annotated screenshot first. The numbered labels in the image correspond to the structured element data, making it easy to identify what to click.

Plain Screenshot (No Annotations)¶

For simple screenshots without element discovery:

{
  "action": "capture",
  "windowHandle": "123456",
  "annotate": false
}

Capabilities¶

• Annotated by Default - Screenshots include numbered element overlays and structured data for UI discovery
• LLM-Optimized - JPEG format, auto-scaling to 1568px, quality 60 for minimal token usage
• Easy targeting - Use window_management(action='find', title='...') to get a handle, then pass to screenshot_control
• Capture any monitor - Screenshot any connected display by index
• Capture windows - Screenshot a specific window (even if partially obscured)
• Capture regions - Screenshot an arbitrary rectangular area
• Capture all monitors - Composite screenshot of entire virtual desktop
• Format options - JPEG (default) or PNG with configurable quality (1-100)
• Auto-scaling - Large captures are scaled to the model-friendly output size
• Output modes - Inline base64 (default) or file path for zero-overhead file workflows
• Cursor inclusion - Optionally include mouse cursor in captures
• Multi-monitor aware - Supports extended desktop configurations
• DPI aware - Correct pixel dimensions on high-DPI displays

Error Handling¶

The server handles common Windows security scenarios:

Configuration¶

Environment Variables¶

Known Limitations¶

UAC & Elevated Processes¶

Windows security prevents any non-elevated process from interacting with UAC prompts or elevated (Administrator) windows. This is a fundamental Windows security boundary that no MCP server can bypass.

Why This Matters¶

Many common operations trigger elevation: - Installing software (winget, choco, MSI installers) - Modifying system settings - Apps that "Run as Administrator" - Antivirus, device manager, service management tools

When building automation workflows, plan for elevation boundaries: 1. Pre-install required software before running AI workflows 2. Use per-user installers when available (e.g., VS Code user installer) 3. Run the MCP server elevated if you need to automate admin tasks (with appropriate caution)

Security Considerations¶

• UIPI: Windows User Interface Privilege Isolation blocks input to elevated windows from non-elevated processes
• Secure Desktop: Input cannot be sent during UAC prompts or lock screen
• Input Simulation: The server uses SendInput which is the standard Windows API for simulating input