Google Open-Sources ARTEMIS: An AI Android Automation Agent That Automates End-to-End Workflows, Captures Logs, and Connects to AI IDEs via MCP

AIDeveloper44 Team

Google's Pixel-Test-Engineering (PTE) Fusion team has open-sourced ARTEMIS, an automation framework designed to execute end-to-end testing and workflows on Android devices using natural-language instructions. Released under the Apache 2.0 license, the project provides developers and QA teams with a way to drive Android emulators and physical hardware using multimodal language models and standard development environments.

Automating mobile applications traditionally relies on rigid scripting frameworks that interact with specific view hierarchies or test IDs. When UI elements change or dynamic content shifts, scripts often break. ARTEMIS addresses this issue by employing an agentic loop that dynamically observes screen states, identifies user interface elements, and executes corresponding actions directly on the device.

Core Architecture and Multimodal Targeting

ARTEMIS operates by interacting with Android devices through standard tooling, including the Android Debug Bridge (ADB), scrcpy for screen mirroring, and FFmpeg. The framework accepts high-level natural language instructions—such as checking specific settings, calculating routes across mapping apps, or navigating media players—and decomposes them into sequential UI actions.

To interact with user interfaces accurately, the framework implements a tiered targeting strategy. It prioritizes element indices derived from accessibility and view hierarchies when accessible. If an interface uses custom rendering engines or non-standard UI components that lack accessibility metadata, the system falls back to coordinate-based and visual targeting using vision-language models. Supported foundation models include Google's Gemini, Anthropic's Claude, OpenAI's GPT-4o, and Alibaba's Qwen-VL.

Execution is handled through what the team describes as a reactive observe-and-act loop. With asynchronous history summarization, action latencies typically range between 3 to 5 seconds per step. To manage edge cases, ARTEMIS validates interface targets prior to executing actions. If an action fails or encounters an unexpected modal, the framework routes the blocked state back to the operator agent for recovery, enabling sustained exploratory testing.

Integration via Model Context Protocol (MCP)

A central feature of the release is native support for Anthropic's Model Context Protocol (MCP). Through this integration, ARTEMIS exposes mobile testing tools directly to AI-assisted coding environments and desktop assistants, including Antigravity, Claude Code, Codex, Windsurf, Cursor, and VS Code.

When configured as an MCP server, ARTEMIS provides client assistants with specialized tools to drive devices and retrieve diagnostics. These tools include:

• mobile_run_task: Dispatches natural-language tasks directly to the connected phone or emulator.
• mobile_manage_task: Controls running execution threads and handles interruptions.
• mobile_get_device_state: Retrieves real-time screen captures, current foreground package data, and orientation.
• mobile_inspect_trace: Allows the assistant to review execution steps and visual action records.
• mobile_diagnose: Captures device Logcat output, stack traces, and system resource data to debug app failures.

The repository also includes standard behavioral guidelines (rules.md) designed to instruct coding assistants on systematic testing practices, guiding the models to verify device preconditions and avoid speculative interactions.

Benchmark Results and Setup

According to the project documentation, ARTEMIS achieves an evaluation score exceeding a 99% task completion rate on Google Research's AndroidWorld benchmark, which comprises more than 100 complex, multi-step mobile tasks across varied applications.

To run ARTEMIS, developers require Python 3.12 or newer and an Android device with USB Debugging enabled. The repository includes one-click setup scripts for macOS, Linux, and Windows PowerShell that configure dependencies via the uv package manager and launch a local web interface at http://localhost:8000. This dashboard includes a device connection wizard, real-time screen mirroring, an interactive prompt sandbox, and replays of previous runs. Commands can also be executed directly from the terminal or dispatched through AI IDEs.

Enjoyed this?

Get more posts like this delivered to your inbox.

Subscribe