# Surfer H — OSWorld Benchmark Runner

Runs the OSWorld benchmark against Surfer H via H's Agent Platform (AGP).
Each task spawns an AWS EC2 VM, installs RDDS, submits the task to AGP, and evaluates locally.

## Prerequisites

- **AWS credentials** configured (`~/.aws/credentials` or env vars `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`). The runner provisions EC2 instances and needs `ec2:RunInstances`, `ec2:DescribeInstances`, `ec2:CreateTags`, and `ec2:TerminateInstances` permissions.
- **Python 3.11+** with the OSWorld repo dependencies installed (`pip install -e .` from repo root).

## Setup

1. Pull the git submodules (both are required):

```bash
git submodule update --init --recursive
curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync
uv pip uninstall opencv-python
uv pip install opencv-python-headless
```

This pulls:
- `agp_client/` — AGP Python client (required, used by `surfer_agent.py`)
- `rdds/` — Remote Desktop Driver Server source (required, deployed onto each VM)

2. Install the local `agp_client` package (handled automatically if you ran `pip install -e .` from repo root, see `pyproject.toml`).

## Usage

```bash
source .venv/bin/activate

python mm_agents/surferH/run_benchmark_with_retries.py --num_envs 50 --agent_identifier OSWorldSurfer-Holo3-35B-A3B-20260330-OSWorld-Verified
```

Logs go to `results/benchmark_*.log`.

### How it works

Tasks are placed in a shared queue. Each worker (`TaskWorker`) pulls from the queue and, for every task:

1. **Snapshot revert** — `env.reset()` reverts the VM to a clean AMI state.
2. **Flask readiness** — waits for the OSWorld Flask backend on `:5000` to respond (up to 300s).
3. **RDDS install + verify** — uploads and starts the Remote Desktop Driver Server, then explicitly checks both `/health` and `/screenshot` on `:8087`.
4. **Task setup** — `env.reset(task_config=...)` runs the task-specific setup scripts on the VM.
5. **Agent execution** — submits the task to AGP via `SurferAgent.predict()`.
6. **Evaluation** — runs the OSWorld evaluator to score the result.

If any of steps 1-4 fail, the worker tears down the VM and provisions a fresh one, then retries the same task. This happens transparently — no tasks are requeued to a separate pass.

### status.json lifecycle

Each task writes a `status.json` to its result directory that evolves as the task progresses:

- **Before agent starts**: `{"task_id", "instruction", "status": "running", "started_at"}`
- **On trajectory creation**: adds `"trajectory_id"` (written via callback before the agent finishes, so the ID is always recoverable even if the process crashes)
- **On completion**: adds `"score"`, `"elapsed_s"`, `"agp_message"`, `"agp_actions"`, sets `"status": "completed"`
- **On error**: sets `"status": "error"` with an `"error"` field

### Dead worker recovery

The main process monitors all workers. If a worker process dies (crash, OOM, etc.), it is automatically replaced with a new one that provisions a fresh VM and continues consuming from the queue.

## Security Group — Required Inbound Rules

The security group [`sg-07e906fcff24547b1`](https://211125782317-dmrkely5.us-west-2.console.aws.amazon.com/ec2/home?region=us-west-2#SecurityGroup:groupId=sg-07e906fcff24547b1) must allow the following inbound ports for OSWorld VMs to function. Use your VPC CIDR (e.g. `10.0.0.0/16`) as the source unless noted otherwise.

| Port | Protocol | Source | Purpose |
|------|----------|--------|---------|
| 22 | TCP | 0.0.0.0/0 | SSH access |
| 80 | TCP | VPC CIDR | HTTP traffic |
| 5000 | TCP | VPC CIDR | OSWorld Flask backend (screenshot, accessibility, command execution) |
| 5910 | TCP | 0.0.0.0/0 | noVNC web visualization |
| 8006 | TCP | VPC CIDR | VNC service |
| 8080 | TCP | VPC CIDR | VLC service (used by OSWorld evaluators) |
| 8081 | TCP | VPC CIDR | Additional service |
| 8087 | TCP | 0.0.0.0/0 | RDDS (Remote Desktop Driver Server) — must be reachable by AGP agents outside the VPC. Port defined by `RDDS_PORT` in `run_benchmark.py` |
| 9222 | TCP | VPC CIDR | Chrome DevTools Protocol |

Outbound: allow all traffic (`0.0.0.0/0`).



## UFW Firewall & Port 8087

The OSWorld AMIs ship with UFW enabled and a default `DROP` policy on inbound traffic. Only a fixed set of ports (22, 5000, 5900, 5901, 5910, 8006, 8080, 9222) are pre-allowed. Port 8087 (RDDS) was not included when the AMIs were built.

`install_rdds.py` handles this automatically — it runs `sudo ufw allow 8087/tcp` on the VM before starting the RDDS server. If you need to debug manually, SSH/SSM into the instance and run:

```bash
sudo ufw allow 8087/tcp
```

