Skip to content

Changelog

[Unreleased]

Added

  • 5 Focused UI Tools - Split ui_automation into specialized tools for better LLM understanding:
  • ui_find — Find elements by name, type, or ID (with timeout/retry via timeoutMs)
  • ui_click — Click buttons, tabs, checkboxes
  • ui_type — Type text into edit controls
  • ui_read — Read text from elements (UIA + OCR fallback)
  • file_save — Save files via Save As dialog (English Windows only)
  • Dedicated app tool - Launch applications with app(programPath='notepad.exe')
  • Separated from window_management for clearer intent
  • Automatic window detection after launch
  • Returns window handle for subsequent operations
  • Close with discardChanges - window_management(action='close', handle='...', discardChanges=true)
  • Automatically dismisses "Save?" dialogs by clicking "Don't Save"
  • English Windows only (detects English button text)
  • Path auto-normalization - Forward slashes automatically converted to backslashes
  • ui_type(text='C:/Users/file.txt') → types C:\Users\file.txt
  • Works in ui_type and keyboard_control
  • LLM integration tests - Tools tested with real AI models using agent-benchmark
  • Verifies AI models understand tool descriptions and use them correctly
  • Catches usability issues before release

Changed

  • Optimized JSON output for token efficiency - Reduced token usage in tool responses
  • Short property names (e.g., s instead of success, h instead of handle)
  • Centralized JsonSerializerOptions for consistent formatting
  • Significantly reduces token consumption when LLMs process responses
  • Handle-based workflow - All tools now use explicit windowHandle for window targeting
  • LLM calls window_management(action='find') to get handles, then decides which to use
  • Removed implicit app parameter that made window selection decisions
  • Example: ui_click(windowHandle='123456', name='Save')
  • snake_case action names - All action parameters use snake_case (e.g., find, click, get_foreground)
  • screenshot_control returns inline images as MCP image content blocks - Base64 image data is now sent as a dedicated image/jpeg or image/png content block instead of a JSON imageData field, so clients render it as a real image
  • The JSON text block still carries dimensions, format, file size/path, and annotated elements, with the base64 payload stripped to avoid duplicating it
  • isError now reflects tool-level failures - Validation errors (invalid action/target/imageFormat/quality/outputMode, missing region params, output path errors) now set CallToolResult.IsError=true at the MCP protocol level, instead of only encoding the failure in the JSON body's success/errorCode/message fields
  • Annotated screenshots by default - screenshot_control now includes element overlays automatically
  • Element names, types, and coordinates embedded in the image
  • Simplified workflow - Most tasks now require fewer tool calls
  • Old: window_management(action='activate') → ui_automation(action='find') → ui_automation(action='click')
  • New: window_management(action='find') → ui_click(windowHandle='...', name='Save')

[1.1.4] - 2025-12-28

Changed

  • Ship self-contained exe releases (x64/arm64) — no .NET runtime required

[1.1.3] - 2025-12-27

Changed

  • Optimized MCP tool descriptions for LLM token efficiency

[1.1.2] - 2025-12-27

Added

  • Phase 3-4 LLM usability improvements
  • Enhanced error messages and guidance

[1.1.1] - 2025-12-26

Added

  • Phase 0-2 integration tests for LLM-driven improvements
  • Improved test coverage for UI automation scenarios

[1.1.0] - 2025-12-25

Changed

  • UIA3 Migration - Migrated UI Automation from UIA2 (managed wrapper) to UIA3 (COM API)
  • ~40% performance improvement for element discovery and tree traversal
  • Better compatibility with modern Windows applications
  • Improved stability and reduced memory usage

Added

  • CI workflow for automated testing on pull requests
  • Comprehensive UI Automation test infrastructure
  • WinForms test harness with TabControl, ListView, TreeView, DataGridView
  • Electron test harness for Chromium-based application testing
  • 45+ integration tests covering find, click, type, toggle, tree navigation