Complete Feature Reference¶

🎯 The Approach: Semantic First, Fallback When Needed¶

Windows MCP uses the Windows UI Automation API as the primary interaction method. This gives AI agents semantic understanding of applications — finding elements by name, type, and state rather than parsing screenshots.

Token Optimization¶

All tool responses are designed for LLM efficiency, minimizing token usage while preserving information:

Example response comparison:

// Standard JSON (~180 tokens)
{ "success": true, "errorCode": "success", "message": "Clicked element", "element": { "name": "Save", "controlType": "Button", "handle": "123" } }

// Optimized JSON (~60 tokens)
{ "ok": true, "ec": "success", "msg": "Clicked", "el": { "n": "Save", "t": "Button", "h": "123" } }

This reduces LLM costs by ~60% and improves response times when processing tool results.

LLM Testing & Validation¶

Every tool is tested with real AI models using pytest-aitest to ensure LLMs understand tool descriptions and use them correctly.

Why LLM testing matters:

• Tool descriptions must be LLM-friendly — If the AI misunderstands a parameter, it fails silently
• Response formats affect reasoning — Structured hints guide the LLM to correct next steps
• Edge cases surface quickly — Real models find ambiguities that unit tests miss

LLM tests run as part of every release. See CONTRIBUTING.md for how to run them yourself.

When to Use Each Tool¶

Browser Automation¶

• Edge, Chrome, and other Chromium apps are auto-detected and searched with the deeper Chromium strategy.
• Launch with app(programPath='msedge.exe', arguments='https://example.com') or find an existing browser window first.
• Page links, buttons, and form fields usually surface visible text or ARIA labels as the UIA name, so start with ui_find, ui_click, and ui_type.
• For authenticated or SSO-only sites, prefer reusing an already-open signed-in Edge/Chrome window first. A Chromium launcher helper exiting immediately is often normal existing-session behavior, so check the browser window before retrying the launch.
• Keep discovery compact: screenshot_control already returns annotated element metadata without image bytes unless you opt in.
• For browser chrome like the address bar or tab switching, prefer shortcuts such as Ctrl+L, Ctrl+R, and Ctrl+Tab.
• Treat browser chrome and non-Chromium browsers as best-effort until dedicated browser coverage expands beyond the Electron/Chromium harnesses.
• The Chromium smoke slice now runs by default: deterministic local Edge/Chrome test pages plus a required public-web smoke check against https://demo.playwright.dev/todomvc/.
• Chromium browser coverage stays on the same semantic-first model as Electron: deep Chromium tree search, ARIA/visible-text discovery, and no separate browser-only tool family.
• The deterministic Chromium smoke harness launches Edge and Chrome app windows with isolated browser state (--user-data-dir), forces renderer accessibility, waits for page-owned readiness signals, and only uses narrow popup dismissal as a fallback so browser-owned UI does not mask real page interaction results.

Tools Overview¶

� App (app)¶

Launch applications and get their window handles for subsequent operations.

Parameters¶

Capabilities¶

• Launch applications by name or full path
• Automatic window detection after launch
• Returns window handle for use with other tools
• Configurable startup parameters

Example¶

app(programPath='notepad.exe') → handle='123456'
ui_type(windowHandle='123456', text='Hello World')

�🔍 UI Find (ui_find)¶

Find and discover UI elements by name, type, or automation ID.

Parameters¶

Capabilities¶

• Find elements by name, control type, or automation ID
• Partial name matching with nameContains
• Regex pattern matching with namePattern
• Sort results by prominence (largest first) for disambiguation
• Returns element IDs for use with other ui_* tools
• Electron app support (VS Code, Teams, Slack)

🖱️ UI Click (ui_click)¶

Click buttons, tabs, checkboxes, and other interactive elements.

Parameters¶

*One of elementId, name, nameContains, or automationId required.

Capabilities¶

• Click buttons, tabs, menu items
• Toggle checkboxes and toggle buttons
• Handles various control patterns automatically
• Falls back to coordinate-based click if pattern fails

⌨️ UI Type (ui_type)¶

Type text into edit controls and text fields.

Parameters¶

Capabilities¶

• Type text into any editable control
• Clear existing content before typing (clearFirst)
• Append text to existing content
• Unicode support for any language

📖 UI Read (ui_read)¶

Read text from elements using UI Automation or OCR.

Parameters¶

Capabilities¶

• Extract text from any UI element
• Automatic OCR fallback for custom-rendered text
• Windows.Media.Ocr for local text recognition
• Language support for international text

💾 File Save (file_save)¶

Save files via Save As dialog. Handles the entire save workflow: triggers save, waits for dialog, fills path, confirms. English Windows only (detects English dialog titles and button text).

Parameters¶

Capabilities¶

• Trigger Ctrl+S to save
• Auto-detect Save As dialog appearance
• Fill in filename automatically
• Handle overwrite confirmation dialogs
• Works with Office apps, Notepad, and more

🖱️ Mouse Control¶

Control mouse input on Windows with full multi-monitor and DPI awareness.

Actions¶

Capabilities¶

• Click, double-click, right-click, middle-click
• Move cursor to absolute coordinates
• Drag operations with hold/release
• Scroll up/down/left/right
• Multi-monitor support with DPI awareness
• Easy targeting with target='primary_screen' or 'secondary_screen'
• Modifier key support (Ctrl+click, Shift+click, etc.)
• Wrong window detection with expectedWindowTitle / expectedProcessName

⌨️ Keyboard Control¶

Control keyboard input on Windows with Unicode support.

Actions¶

Supported Keys¶

Function Keys¶

f1 through f24

Navigation¶

up, down, left, right, home, end, pageup, pagedown, insert, delete

Control¶

enter, tab, escape, space, backspace

Modifiers¶

ctrl, shift, alt, win

Media¶

volumemute, volumedown, volumeup, mediaplaypause, medianexttrack, mediaprevtrack, mediastop

Special¶

copilot (Windows 11 Copilot+ PCs)

Browser¶

browserback, browserforward, browserrefresh, browserstop, browsersearch, browserfavorites, browserhome

Capabilities¶

• Unicode text typing (layout-independent) - type any character in any language
• Virtual key presses - Enter, Tab, Escape, F1-F24, navigation keys
• Key combinations - Use press with modifiers parameter: press(key='s', modifiers='ctrl') for Ctrl+S
• Key sequences - multi-key macros with configurable timing
• Hold/release keys - for Shift-select and other hold operations
• Special keys - Copilot key (Windows 11), media controls, browser keys
• Layout detection - query current keyboard layout (BCP-47 format)
• Clear before typing - use clearFirst to select all (Ctrl+A) before typing new text
• Wait for idle - wait for keyboard input to be processed before continuing

🪟 Window Management¶

Control windows on the Windows desktop. Use app tool to launch applications, then use this tool to manage the windows.

Actions¶

Close with discardChanges¶

Use discardChanges=true to automatically dismiss "Save?" dialogs when closing:

window_management(action='close', handle='123456', discardChanges=true)

English Windows only — detects English button text like "Don't Save".

Capabilities¶

• List all visible top-level windows with titles, handles, process info, and bounds
• Locate windows by title (substring or regex matching)
• Bring windows to foreground with focus
• Minimize, maximize, restore, and close windows
• Position and size windows with move, resize, or set_bounds
• Wait for a window to appear with configurable timeout
• Move windows between monitors
• Full multi-monitor support with DPI awareness
• Proper UWP/Store app detection and handling
• Cloaking detection to filter out virtual desktop and shell-managed windows

📸 Screenshot Capture¶

Capture screenshots on Windows with LLM-optimized defaults. By default, screenshots include annotated element overlays with numbered labels and structured element data — perfect for UI discovery.

Actions¶

Capture Targets¶

Parameters¶

Annotated Screenshot Response¶

When annotate=true (default), the response includes structured element data. Image is omitted by default (includeImage=false) to save ~100K+ tokens:

{
  "success": true,
  "annotated_elements": [
    { "index": 1, "element_id": "...", "name": "File", "control_type": "MenuItem", "clickable_point": { "x": 50, "y": 30 } },
    { "index": 2, "element_id": "...", "name": "Edit", "control_type": "MenuItem", "clickable_point": { "x": 100, "y": 30 } }
  ],
  "element_count": 25
}

Use case: When you don't know element names, capture an annotated screenshot first. The numbered labels in the image correspond to the structured element data, making it easy to identify what to click.

Plain Screenshot (No Annotations)¶

For simple screenshots without element discovery:

{
  "action": "capture",
  "windowHandle": "123456",
  "annotate": false
}

Capabilities¶

• Annotated by Default - Screenshots include numbered element overlays and structured data for UI discovery
• LLM-Optimized - JPEG format, auto-scaling to 1568px, quality 60 for minimal token usage
• Easy targeting - Use window_management(action='find', title='...') to get a handle, then pass to screenshot_control
• Capture any monitor - Screenshot any connected display by index
• Capture windows - Screenshot a specific window (even if partially obscured)
• Capture regions - Screenshot an arbitrary rectangular area
• Capture all monitors - Composite screenshot of entire virtual desktop
• Format options - JPEG (default) or PNG with configurable quality (1-100)
• Auto-scaling - Defaults to 1568px width (LLM vision model native limit); disable with maxWidth: 0
• Output modes - Inline base64 (default) or file path for zero-overhead file workflows
• Cursor inclusion - Optionally include mouse cursor in captures
• Multi-monitor aware - Supports extended desktop configurations
• DPI aware - Correct pixel dimensions on high-DPI displays

Error Handling¶

The server handles common Windows security scenarios:

Configuration¶

Environment Variables¶

Known Limitations¶

UAC & Elevated Processes¶

Windows security prevents any non-elevated process from interacting with UAC prompts or elevated (Administrator) windows. This is a fundamental Windows security boundary that no MCP server can bypass.

Why This Matters¶

Many common operations trigger elevation: - Installing software (winget, choco, MSI installers) - Modifying system settings - Apps that "Run as Administrator" - Antivirus, device manager, service management tools

When building automation workflows, plan for elevation boundaries: 1. Pre-install required software before running AI workflows 2. Use per-user installers when available (e.g., VS Code user installer) 3. Run the MCP server elevated if you need to automate admin tasks (with appropriate caution)

Security Considerations¶

• UIPI: Windows User Interface Privilege Isolation blocks input to elevated windows from non-elevated processes
• Secure Desktop: Input cannot be sent during UAC prompts or lock screen
• Input Simulation: The server uses SendInput which is the standard Windows API for simulating input