mirror of
https://github.com/getpaseo/paseo.git
synced 2026-07-29 12:01:31 +00:00
docs(browser): document browser automation
This commit is contained in:
Mohamed Boudra
@@ -1,21 +1,26 @@
|
|||||||
# Changelog
|
# Changelog
|
||||||
|
|
||||||
## 0.1.104-beta.2 - 2026-07-03
|
## 0.1.104-beta.3 - 2026-07-04
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
|
||||||
- Agents can drive the in-app browser: open tabs, navigate, click, type, and take screenshots
|
- Agents can drive the in-app browser with page snapshots, trusted input, dialogs, and tab controls ([#1881](https://github.com/getpaseo/paseo/pull/1881))
|
||||||
- Inspect, annotate, and send page elements from a browser tab to the agent ([#1708](https://github.com/getpaseo/paseo/pull/1708) by [@huiliaoning](https://github.com/huiliaoning))
|
- Inspect, annotate, and send page elements from a browser tab to the agent ([#1708](https://github.com/getpaseo/paseo/pull/1708) by [@huiliaoning](https://github.com/huiliaoning))
|
||||||
- Schedules screen to create and manage recurring agents ([#1246](https://github.com/getpaseo/paseo/pull/1246))
|
- Schedules screen to create and manage recurring agents ([#1246](https://github.com/getpaseo/paseo/pull/1246))
|
||||||
- Open a project from anywhere with Cmd+O ([#1849](https://github.com/getpaseo/paseo/pull/1849))
|
- Open a project from anywhere with Cmd+O ([#1849](https://github.com/getpaseo/paseo/pull/1849))
|
||||||
|
- Agents can rename workspaces after they understand the task ([#1876](https://github.com/getpaseo/paseo/pull/1876))
|
||||||
|
- Claude Ultra Code is available for supported Claude models ([#1872](https://github.com/getpaseo/paseo/pull/1872))
|
||||||
- ByteDance TRAE CLI available as an agent provider ([#1831](https://github.com/getpaseo/paseo/pull/1831) by [@park0er](https://github.com/park0er))
|
- ByteDance TRAE CLI available as an agent provider ([#1831](https://github.com/getpaseo/paseo/pull/1831) by [@park0er](https://github.com/park0er))
|
||||||
|
|
||||||
### Improved
|
### Improved
|
||||||
|
|
||||||
|
- Workspaces created by agents now get readable generated names ([#1887](https://github.com/getpaseo/paseo/pull/1887))
|
||||||
|
- Browser tabs opened by agents stay in the background until you switch to them ([#1875](https://github.com/getpaseo/paseo/pull/1875))
|
||||||
- Clearer cards when an agent asks a question ([#1643](https://github.com/getpaseo/paseo/pull/1643) by [@cleiter](https://github.com/cleiter))
|
- Clearer cards when an agent asks a question ([#1643](https://github.com/getpaseo/paseo/pull/1643) by [@cleiter](https://github.com/cleiter))
|
||||||
|
|
||||||
### Fixed
|
### Fixed
|
||||||
|
|
||||||
|
- Docker images keep running during provider cleanup and diagnostics ([#1877](https://github.com/getpaseo/paseo/pull/1877))
|
||||||
- New Workspace drafts survive archiving a workspace ([#1838](https://github.com/getpaseo/paseo/pull/1838))
|
- New Workspace drafts survive archiving a workspace ([#1838](https://github.com/getpaseo/paseo/pull/1838))
|
||||||
- Composer autocomplete stays open after switching screens ([#1851](https://github.com/getpaseo/paseo/pull/1851))
|
- Composer autocomplete stays open after switching screens ([#1851](https://github.com/getpaseo/paseo/pull/1851))
|
||||||
- Claude usage appears when a quota window has no scheduled reset ([#1855](https://github.com/getpaseo/paseo/pull/1855))
|
- Claude usage appears when a quota window has no scheduled reset ([#1855](https://github.com/getpaseo/paseo/pull/1855))
|
||||||
|
|||||||
78
public-docs/browser-tools.md
Normal file
78
public-docs/browser-tools.md
Normal file
@@ -0,0 +1,78 @@
|
|||||||
|
---
|
||||||
|
title: Browser tools
|
||||||
|
description: The browser_* MCP tools agents use to drive Paseo browser tabs.
|
||||||
|
nav: Tools reference
|
||||||
|
order: 36
|
||||||
|
category: Browser
|
||||||
|
---
|
||||||
|
|
||||||
|
# Browser tools
|
||||||
|
|
||||||
|
The `browser_*` tools are injected into agents alongside the other [Paseo MCP tools](/docs/mcp) when [browser automation](/docs/browser) is enabled.
|
||||||
|
|
||||||
|
Shared concepts:
|
||||||
|
|
||||||
|
- **`browserId`** identifies a tab. It comes from `browser_new_tab` or `browser_list_tabs` and is required by every tab-scoped tool.
|
||||||
|
- **`ref`** identifies an element, e.g. `@e3`. Refs come from the latest `browser_snapshot` of the same tab and expire when the page changes — stale refs return an error instead of acting on the wrong element.
|
||||||
|
- Every result reports **dialogs** the page opened during the command (alerts accepted; confirm/prompt/beforeunload dismissed).
|
||||||
|
|
||||||
|
Arguments marked `?` are optional.
|
||||||
|
|
||||||
|
## Tabs
|
||||||
|
|
||||||
|
| Tool | Arguments | Purpose |
|
||||||
|
| ------------------- | -------------------------- | ------------------------------------------------------------------------- |
|
||||||
|
| `browser_list_tabs` | — | List open tabs in the agent's workspace across connected hosts. |
|
||||||
|
| `browser_new_tab` | `url?` | Open a tab in the background and return its `browserId`. |
|
||||||
|
| `browser_close_tab` | `browserId` | Close a tab and clean up its webview. |
|
||||||
|
| `browser_resize` | `browserId, width, height` | Resize the tab's viewport — check a layout at phone or tablet dimensions. |
|
||||||
|
|
||||||
|
## Reading the page
|
||||||
|
|
||||||
|
| Tool | Arguments | Purpose |
|
||||||
|
| -------------------- | -------------------------------------- | ------------------------------------------------------------------------------------- |
|
||||||
|
| `browser_snapshot` | `browserId` | Return the page as an accessibility tree with element refs. |
|
||||||
|
| `browser_screenshot` | `browserId, fullPage?` | Capture a PNG of the viewport, or the full page with `fullPage`. |
|
||||||
|
| `browser_logs` | `browserId, maxEntries?` | Read recent console messages and network timing entries. |
|
||||||
|
| `browser_wait` | `browserId, text? \| url?, timeoutMs?` | Wait until the page contains text or reaches a URL fragment (exactly one of the two). |
|
||||||
|
|
||||||
|
## Interacting
|
||||||
|
|
||||||
|
| Tool | Arguments | Purpose |
|
||||||
|
| ------------------ | --------------------------------------------------- | ------------------------------------------------------------------------------------ |
|
||||||
|
| `browser_click` | `browserId, ref, button?, doubleClick?, modifiers?` | Click an element — left/right/middle, double-click, keyboard modifiers. |
|
||||||
|
| `browser_fill` | `browserId, ref, value` | Set the value of an input-like element. |
|
||||||
|
| `browser_type` | `browserId, text, ref?` | Type text into an element, or into the focused element when `ref` is omitted. |
|
||||||
|
| `browser_keypress` | `browserId, key, ref?` | Press a key (`Enter`, `Escape`, `Tab`, `Space`, …) on an element or the focused one. |
|
||||||
|
| `browser_hover` | `browserId, ref` | Hover an element — triggers real CSS `:hover`. |
|
||||||
|
| `browser_select` | `browserId, ref, value` | Choose an option in a `<select>`. |
|
||||||
|
| `browser_drag` | `browserId, sourceRef, targetRef` | Drag one element onto another. |
|
||||||
|
| `browser_upload` | `browserId, ref, filePaths` | Set files on a file input. Paths must be inside the agent's workspace. |
|
||||||
|
| `browser_scroll` | `browserId, deltaX, deltaY, ref?` | Scroll the page, or center the wheel input over an element with `ref`. |
|
||||||
|
|
||||||
|
## Navigation
|
||||||
|
|
||||||
|
| Tool | Arguments | Purpose |
|
||||||
|
| ------------------ | ---------------- | --------------------------------------------------- |
|
||||||
|
| `browser_navigate` | `browserId, url` | Go to an `http(s)` URL. |
|
||||||
|
| `browser_back` | `browserId` | Go back — errors when there is no history to go to. |
|
||||||
|
| `browser_forward` | `browserId` | Go forward — errors when there is no forward entry. |
|
||||||
|
| `browser_reload` | `browserId` | Reload the page. |
|
||||||
|
|
||||||
|
## Scripting
|
||||||
|
|
||||||
|
| Tool | Arguments | Purpose |
|
||||||
|
| ------------------ | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
|
||||||
|
| `browser_evaluate` | `browserId, function, ref?` | Run a JavaScript function in the page. With `ref`, the resolved element is passed as the first argument. Results return as bounded JSON. |
|
||||||
|
|
||||||
|
## Errors
|
||||||
|
|
||||||
|
Tools return structured errors rather than failing silently. The ones agents see most:
|
||||||
|
|
||||||
|
| Code | Meaning |
|
||||||
|
| ------------------- | ------------------------------------------------------------------------------- |
|
||||||
|
| `browser_disabled` | Browser tools are turned off on this host. |
|
||||||
|
| `browser_no_host` | No browser host (desktop app) is connected. Retryable. |
|
||||||
|
| `browser_stale_ref` | The ref no longer matches the page — take a new snapshot. |
|
||||||
|
| `browser_timeout` | The element never became actionable, or the wait condition never held. |
|
||||||
|
| `browser_denied` | The action isn't allowed — e.g. a non-`http(s)` URL, or no history to navigate. |
|
||||||
82
public-docs/browser.md
Normal file
82
public-docs/browser.md
Normal file
@@ -0,0 +1,82 @@
|
|||||||
|
---
|
||||||
|
title: Browser automation
|
||||||
|
description: Give agents a real browser inside Paseo to open pages, click, type, and verify their work.
|
||||||
|
nav: Browser automation
|
||||||
|
order: 35
|
||||||
|
category: Browser
|
||||||
|
---
|
||||||
|
|
||||||
|
# Browser automation
|
||||||
|
|
||||||
|
Agents in Paseo can drive real browser tabs — the same tabs you see in the Paseo desktop app. An agent can open your dev server, read the page, click through a flow, fill a form, and take a screenshot, all without leaving your machine.
|
||||||
|
|
||||||
|
This closes the loop on frontend work: instead of telling you "the change should work," an agent opens the page and checks.
|
||||||
|
|
||||||
|
Typical uses:
|
||||||
|
|
||||||
|
- **Verify its own changes.** After editing a component, the agent opens the dev server, snapshots the page, and confirms the new text or layout is actually there.
|
||||||
|
- **Reproduce and diagnose bugs.** Click the exact sequence from a bug report, then read the console and network logs.
|
||||||
|
- **Exercise full flows.** Forms, multi-step wizards, hover menus, drag and drop, file uploads.
|
||||||
|
- **Work in logged-in sessions.** Tabs keep their session state. Log in once yourself, and the agent can work behind the login.
|
||||||
|
|
||||||
|
Because you share the browser with the agent, you can watch it work — and step in at any point.
|
||||||
|
|
||||||
|
## Enabling
|
||||||
|
|
||||||
|
Browser tools are off by default. Turn them on per host:
|
||||||
|
|
||||||
|
- **In the app:** open your host's settings and enable **Browser tools**.
|
||||||
|
- **In `config.json`** (`~/.paseo/config.json`):
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"daemon": {
|
||||||
|
"browserTools": {
|
||||||
|
"enabled": true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The tools are part of the [Paseo MCP toolset](/docs/mcp), so **Inject Paseo tools** (`daemon.mcp.injectIntoAgents`) must also be on for agents to receive them. Existing agents may need a reload to pick up new tools.
|
||||||
|
|
||||||
|
> Browser tools let agents access and control Paseo browser tabs, including logged-in browser state. Only enable this for agents you trust.
|
||||||
|
|
||||||
|
## Desktop only, for now
|
||||||
|
|
||||||
|
Browser tabs are hosted by the Paseo desktop app. The daemon itself doesn't run a browser — it routes tool calls to a connected desktop app, and returns an error when none is connected. The wire contract is host-neutral, so other hosts can carry the same tools later.
|
||||||
|
|
||||||
|
## How an agent sees a page
|
||||||
|
|
||||||
|
The primary tool is `browser_snapshot`, which returns the page as an accessibility tree — headings, text, form state, and hierarchy — instead of raw HTML:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
- document "Settings"
|
||||||
|
- heading "Workspace" [level=1]
|
||||||
|
- text: "Connected as Maya"
|
||||||
|
- textbox "Display name" [ref=@e2]
|
||||||
|
- button "Save changes" [ref=@e3]
|
||||||
|
```
|
||||||
|
|
||||||
|
Interactive elements carry refs like `@e3`. The agent passes a ref to `browser_click`, `browser_fill`, and the other action tools. Refs come from the latest snapshot of that tab and expire when the page changes — a stale ref returns an error instead of acting on the wrong element.
|
||||||
|
|
||||||
|
For anything the tree can't capture, agents fall back to `browser_screenshot`, and `browser_logs` exposes console messages and network timing.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
agent ──MCP──▶ daemon (broker) ──▶ browser host (desktop app) ──▶ webview
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Workspace-scoped tabs.** An agent only sees and controls tabs in its own workspace. New tabs open in the background without stealing your focus.
|
||||||
|
- **Tab-to-host routing.** The daemon remembers which host owns each tab and routes tab commands there. `browser_list_tabs` aggregates all connected hosts.
|
||||||
|
- **Trusted input.** Clicks, keys, hovers, and drags are dispatched as real browser input events — CSS `:hover` triggers, and pages can't tell an agent's click from a user's. Every action first waits for its target to be visible, enabled, and stable.
|
||||||
|
- **Dialogs never block.** `alert` is accepted; `confirm`, `prompt`, and `beforeunload` are dismissed. Every handled dialog is reported in the tool result so the agent knows the page flow changed.
|
||||||
|
|
||||||
|
## Security
|
||||||
|
|
||||||
|
- Navigation is restricted to `http(s)` URLs.
|
||||||
|
- File uploads can only reference files inside the agent's workspace.
|
||||||
|
- Tabs share the browser profile you use in Paseo, including cookies and logins — that's what makes logged-in testing work, and why the feature is opt-in per host.
|
||||||
|
|
||||||
|
See the [tools reference](/docs/browser-tools) for the full tool list.
|
||||||
Reference in New Issue
Block a user