# VLAA-GUI

VLAA-GUI is a multi-agent system for desktop GUI automation. VLAA-GUI decomposes complex desktop tasks into perception, planning, and action through a pipeline of specialized sub-agents.

## Architecture

```mermaid
flowchart TD
    A["Agent<br/>Entry point; manages episode lifecycle"]
    W["Worker<br/>Core executor: iterative planning + action generation"]

    G["GateAgent<br/>Proposes success criteria"]
    ACI["Grounding (ACI)"]
    V["Verifier Agent<br/>Checks task completion"]

    C["Code Agent<br/>Writes & runs code"]
    S["Searcher Agent<br/>LLM web search for tutorials"]

    A --> W

    G --> W
    W --> G

    W --> ACI
    ACI --> W

    V --> W
    W --> V

    ACI --> C
    ACI --> S
```

### Sub-agents

| Agent | Role |
|-------|------|
| **Worker** | Core executor. Receives the task instruction and screenshot, maintains an iterative plan, and selects the next action each step. |
| **Grounding (ACI)** | Maps natural-language references to screen coordinates via a visual grounding model (e.g. Qwen GUI Plus). Handles OCR fallback with Tesseract. |
| **GateAgent** | Runs once at step 1 to propose 1-3 UI-observable success criteria. The Worker checks these every step to decide when the task is done. |
| **VerifierAgent** | Called when the Worker signals completion. Independently judges whether the task is truly finished; can force a replan if not. |
| **CodeAgent** | Generates and executes multi-step pyautogui scripts for actions that are easier to express as code (file manipulation, terminal commands, etc.). |
| **SearcherAgent** | Searches the web for tutorials and hints when the Worker encounters an unfamiliar application or workflow. |


## Quick Start

### 1. Set up the environment

Follow the [OSWorld setup guide](https://github.com/xlang-ai/OSWorld) to install OSWorld and its dependencies.

> [!NOTE]
> You need to use `ami-0b505e9d0d99ba88c` as your base image if you run evaluation on AWS. See the [code](https://github.com/xlang-ai/OSWorld/blob/main/desktop_env/providers/aws/manager.py) for details.

### 2. Install additional dependencies

```bash
pip install pytesseract scikit-learn backoff boto3 google-genai
```

### 3. Configure API keys

Set the following environment variables depending on which providers you use:

```bash
# Main reasoning model (Anthropic Bedrock — uses default AWS credential chain)
export AWS_REGION=us-east-1

# Visual grounding (Qwen)
export DASHSCOPE_API_KEY=your_key

# Web search (Gemini)
export GEMINI_API_KEY=your_key

...
```

### 4. Run the evaluation

```bash
bash scripts/bash/run_vlaa_gui.sh
```

The script accepts environment variable overrides:

```bash
PROVIDER_NAME=aws \
REGION=us-east-1 \
NUM_ENVS=4 \
MAX_STEPS=100 \
RESULT_DIR=./results \
TEST_META_PATH=evaluation_examples/test_all.json \
  bash scripts/bash/run_vlaa_gui.sh
```

### Key flags

| Flag | Description |
|------|-------------|
| `--with_reflection` | Worker reflects on action outcomes before planning next step |
| `--use_verifier` | Enable the VerifierAgent to confirm task completion |
| `--enable_gate` | Enable the GateAgent to propose success criteria |
| `--loop_detection` | Detect and break out of repeated action loops |
| `--feasibility_check` | Skip tasks the agent determines are infeasible |
| `--action_space pyautogui_coding` | Use the CodeAgent for multi-step pyautogui scripts |

## Acknowledgements

This work builds on the [Agent-S project](https://github.com/simular-ai/Agent-S). We adapted and extended their agent architecture with additional sub-agents (gate, verifier, code, search), multi-provider LLM support, and Bedrock prompt caching.
