Compare commits

...
Author SHA1 Message Date
Mohamed Boudra 1c479d7b3d Keep finished replies visible across idle notifications (#10135)
* fix(timeline): keep replies visible across idle notifications

Resolve notification membership at the manager's live event boundary. An idle notice gets a display turn without starting a run, and in-flight notices join the open run.

* test(timeline): keep notice paging assertions compatible with async history

* fix(timeline): reuse managed turn assignment for notices

Resolve notices before waiter matching with the same rule used by live persistence. Verify untagged foreground runtime notices through runAgent and idle Pi notices through timeline paging.
2026-11-19 13:15:13 +07:00
Mohamed Boudra 808766ed7f Acknowledge workspace taps promptly and reduce destination mount work (#9887)
* fix(app): acknowledge accepted workspace taps before navigation

Short taps finish before the delayed hold feedback paints, then navigation occupies JS. Paint accepted feedback on the native UI thread while retaining the existing press cancellation and delay rules.

* fix(app): avoid persistent tint on selected docked workspace rows

* perf(app): reduce workspace mount work and narrow panel subscriptions
2026-11-19 13:10:12 +07:00
Christian WuandClaude Fable 5.2 9faa3034f1 fix(app): keep a submenu open when its row is tapped on a touch screen (#10175)
A tap on the web arrives as pointer events: the finger enters the row as it
lands and leaves as it lifts. The submenu row and its flyout read both as
hover, so the leave queued a close just as the press opened the flyout, and
it disappeared about a quarter of a second later. A tap inside an open
flyout closed it the same way.

Hover intent now ignores pointer events whose pointerType is touch. Mouse
and pen behave as before, including on touch screens with a mouse attached.

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-19 06:02:06 +00:00
Mohamed Boudra 157abbbbda Let agents choose which desktop opens a browser tab (#10189)
* fix(browser): let agents choose which desktop opens a tab

Registration order sent new tabs to the wrong desktop window. Let the agent discover and select the intended window by name, and require a choice when several windows can open tabs.

* test(desktop): run browser host selection in a fresh app session

Closing the second window can leave the existing browser journey in a different renderer state. Give both journeys their own windows and daemon so the routing test does not change the old test setup.

* fix(browser): keep window names visible and stable across reconnects

Frameless desktop windows need an in-app place to see and copy their name. Give each window one browser host identity so a reconnect replaces a lingering connection without creating a duplicate choice.

* test(desktop): cover host selection in the browser journey

Open the second window after the existing browser checks finish so their setup remains isolated within one CI run.
2026-11-19 13:01:34 +07:00
Mohamed BoudraandBrandon Elliott 2c22e64644 Keep dictation working after closing another chat (#10185)
* fix(app): keep dictation working after closing another chat

Keep the shared native recorder alive while a client still uses it. Report microphone startup errors at the composer so the user can retry from the same control.

Co-authored-by: Brandon Elliott <923787+Brandon168@users.noreply.github.com>

* fix(app): explain how to recover from dictation startup failures

Classify known microphone failures at the capture boundary. Keep daemon stream errors available to the composer and clear stale capture feedback on the next attempt.

* fix(app): keep permission guidance when the native request fails

* fix(app): show dictation errors after a terminal prompt shortcut

---------

Co-authored-by: Brandon Elliott <923787+Brandon168@users.noreply.github.com>
2026-11-19 12:06:30 +07:00
paseo-bot[bot]andClaude Opus 5.5 b09b17e6ee fix(opencode): fail the turn when a shared OpenCode service stops (#10187)
When Paseo used an OpenCode v2 service it did not start (a discovered
`opencode serve --service`, or a configured server URL), nothing
reported that service going away. The connection's `exited` signal
resolved only when Paseo itself stopped using it. A turn that was
running when the service died was never failed, the event stream
retried the dead port every 100 ms, and Stop, send and reload were
refused with "active run cancellation was not acknowledged".

The shared connection now treats a refused connection as the service
exiting, the same signal an owned server's process exit gives. The
session fails the turn, stops its stream, and reconnects on the next
operation.

Co-authored-by: paseo-bot[bot] <266920839+paseo-bot[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 05:01:50 +00:00
84 changed files with 2933 additions and 607 deletions
+12 -5
View File
@@ -163,8 +163,16 @@ override a scroll away because they can arrive after the action that requested t
### Draw distance
`drawDistance` is 2000 px on each side of the viewport. A fling over rows that were never measured
blanks the viewport until LegendList renders and measures them. Twelve flings over a chat with a
The native timeline starts with one window-height of draw distance, up to 2000 px. This keeps
nearby rows ready for a fling during placement. Once the chat is placed, history is visible, and
the sidebar has settled, the timeline schedules expansion to 2000 px through two frame callbacks.
The callbacks leave time for the reveal to paint. Verify the actual reveal on the device.
A drag cancels a pending expansion and uses the existing buffer as the window moves. Expansion
resumes when the gesture settles, including momentum. In an Android emulator capture, expanding
the full buffer during the first fling changed the measured content height and returned the chat
to the end mid-fling.
A fling over rows that were never measured blanks the viewport
until LegendList renders and measures them. Twelve flings over a chat with a
15,000 px turn expanded, Android emulator, dev build:
| List | Blank viewport | Janky frames | p50 / p90 / p99 |
@@ -179,9 +187,8 @@ With every turn collapsed, flinging from a cold open into history did not blank
### LegendList patch
`patches/@legendapp+list+3.7.1.patch` carries twelve changes for the timeline. Its header lists
them by purpose, and
[legend-list #593](https://github.com/LegendApp/legend-list/issues/593) reports each upstream.
`patches/@legendapp+list+3.7.1.patch` lists its changes by purpose.
[legend-list #593](https://github.com/LegendApp/legend-list/issues/593) reports the first twelve.
Drop a hunk when the installed LegendList ships its fix.
### Scroll view corrections
@@ -16,6 +16,13 @@ import {
expectComposerFullscreenControls,
} from "../support/helpers/composer";
import { waitForSidebarHydration } from "../support/helpers/workspace-ui";
import {
seedTerminalProfiles,
selectLaunchOption,
terminalPromptInput,
} from "../support/helpers/new-workspace-launch";
test.use({ channel: "chromium", launchOptions: { args: ["--use-fake-device-for-media-stream"] } });
type WebSocketMessage = string | Buffer;
@@ -254,3 +261,82 @@ test("dictation shortcut leaves retained fullscreen and exposes recording contro
await agent.cleanup();
}
});
async function controlMicrophonePermission(page: Page) {
const cdp = await page.context().newCDPSession(page);
const { targetInfo } = await cdp.send("Target.getTargetInfo");
const origin = new URL(page.url()).origin;
return async (setting: "denied" | "granted") => {
await cdp.send("Browser.setPermission", {
permission: { name: "microphone" },
setting,
origin,
browserContextId: targetInfo.browserContextId,
});
};
}
function startDictationByMic(page: Page) {
return page.getByRole("button", { name: "Start dictation", exact: true }).click();
}
function expectDictationPermissionError(page: Page) {
return expect(page.getByTestId("message-input-root").getByRole("alert")).toHaveText(
"Allow microphone access and try again.",
);
}
test("keeps the draft and retries from the mic after access is restored", async ({ page }) => {
const agent = await seedMockAgentWorkspace({
repoPrefix: "dictation-permission-",
title: "Dictation permission",
});
const draft = "Keep my draft";
await installDictationFailureHarness(page);
try {
await openAgentRoute(page, agent);
await expectComposerVisible(page);
await fillComposerDraft(page, draft);
const setMicrophonePermission = await controlMicrophonePermission(page);
await setMicrophonePermission("denied");
await startDictationByMic(page);
await expectDictationPermissionError(page);
await expect(composerLocator(page)).toHaveValue(draft);
await setMicrophonePermission("granted");
await startDictationByMic(page);
await expect(page.getByRole("button", { name: "Cancel dictation", exact: true })).toBeVisible();
await expect(page.getByText("Allow microphone access and try again.")).toHaveCount(0);
await page.getByRole("button", { name: "Cancel dictation", exact: true }).click();
await expect(composerLocator(page)).toHaveValue(draft);
} finally {
await agent.cleanup();
}
});
test("shows dictation failure when the terminal prompt starts it by shortcut", async ({ page }) => {
const workspace = await seedWorkspace({ repoPrefix: "terminal-dictation-permission-" });
const profile = {
id: "echo-prompt",
name: "Echo prompt",
command: "echo",
args: ["{{{prompt}}}"],
};
const profiles = await seedTerminalProfiles([profile]);
await installDictationFailureHarness(page);
try {
await gotoAppShell(page);
await waitForSidebarHydration(page);
await openNewWorkspaceComposer(page, workspace);
await selectLaunchOption(page, profile.id);
const setMicrophonePermission = await controlMicrophonePermission(page);
await setMicrophonePermission("denied");
await terminalPromptInput(page).press("Control+d");
await expectDictationPermissionError(page);
} finally {
await profiles.restore();
await workspace.cleanup();
}
});
@@ -0,0 +1,19 @@
apply plugin: 'com.android.library'
group = 'sh.paseo.pressfeedback'
version = '0.1.0'
def expoModulesCorePlugin = new File(project(":expo-modules-core").projectDir.absolutePath, "ExpoModulesCorePlugin.gradle")
apply from: expoModulesCorePlugin
applyKotlinExpoModulesCorePlugin()
useCoreDependencies()
useExpoPublishing()
useDefaultAndroidSdkVersions()
android {
namespace "sh.paseo.pressfeedback"
defaultConfig {
testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
}
}
@@ -0,0 +1 @@
<manifest xmlns:android="http://schemas.android.com/apk/res/android" />
@@ -0,0 +1,39 @@
package sh.paseo.pressfeedback
import android.content.Context
import android.graphics.Canvas
import android.graphics.Color
import android.graphics.Paint
import expo.modules.kotlin.AppContext
import expo.modules.kotlin.modules.Module
import expo.modules.kotlin.modules.ModuleDefinition
import expo.modules.kotlin.types.ColorCompat
import expo.modules.kotlin.views.ExpoView
class PaseoPressFeedbackModule : Module() {
override fun definition() = ModuleDefinition {
Name("PaseoPressFeedback")
View(PaseoPressFeedbackView::class) {
Prop("tint") { view: PaseoPressFeedbackView, color: Color -> view.paint.color = ColorCompat.toArgb(color) }
Prop("radius") { view: PaseoPressFeedbackView, radius: Double -> view.radius = radius.toFloat() * view.resources.displayMetrics.density }
Prop("resetKey") { view: PaseoPressFeedbackView, key: String -> view.reset(key) }
AsyncFunction("acknowledge") { view: PaseoPressFeedbackView -> view.acknowledge() }
}
}
}
class PaseoPressFeedbackView(context: Context, appContext: AppContext) : ExpoView(context, appContext) {
val paint = Paint(Paint.ANTI_ALIAS_FLAG)
var radius = 0f
private var acknowledged = false
private var resetKey = ""
init { setWillNotDraw(false); importantForAccessibility = IMPORTANT_FOR_ACCESSIBILITY_NO }
fun acknowledge() { acknowledged = true; invalidate() }
fun reset(key: String) {
if (key != resetKey) { resetKey = key; acknowledged = false; invalidate() }
}
override fun onDraw(canvas: Canvas) {
super.onDraw(canvas)
if (acknowledged) canvas.drawRoundRect(0f, 0f, width.toFloat(), height.toFloat(), radius, radius, paint)
}
}
@@ -0,0 +1,5 @@
{
"platforms": ["android", "ios"],
"android": { "modules": ["sh.paseo.pressfeedback.PaseoPressFeedbackModule"] },
"ios": { "modules": ["PaseoPressFeedbackModule"] }
}
@@ -0,0 +1 @@
export { useWorkspaceRowPress } from "./src/ui";
@@ -0,0 +1,22 @@
Pod::Spec.new do |s|
s.name = 'PaseoPressFeedback'
s.version = '0.1.0'
s.summary = 'Native accepted press feedback for Paseo'
s.description = 'Paints accepted press feedback independently of React commits'
s.license = 'Apache-2.0'
s.author = 'Paseo'
s.homepage = 'https://paseo.sh'
s.platforms = { :ios => '15.1' }
s.swift_version = '5.4'
s.source = { :path => '.' }
s.static_framework = true
s.dependency 'ExpoModulesCore'
s.pod_target_xcconfig = {
'DEFINES_MODULE' => 'YES',
'SWIFT_COMPILATION_MODE' => 'wholemodule'
}
s.source_files = "**/*.{h,m,swift}"
end
@@ -0,0 +1,33 @@
import ExpoModulesCore
import UIKit
public class PaseoPressFeedbackModule: Module {
public func definition() -> ModuleDefinition {
Name("PaseoPressFeedback")
View(PaseoPressFeedbackView.self) {
Prop("tint") { (view: PaseoPressFeedbackView, color: UIColor) in view.tint = color }
Prop("radius") { (view: PaseoPressFeedbackView, radius: Double) in view.radius = radius }
Prop("resetKey") { (view: PaseoPressFeedbackView, key: String) in view.reset(key) }
AsyncFunction("acknowledge") { (view: PaseoPressFeedbackView) in view.acknowledge() }
}
}
}
final class PaseoPressFeedbackView: ExpoView {
var tint = UIColor.clear
var radius: Double = 0
private var acknowledged = false
private var resetKey = ""
required init(appContext: AppContext? = nil) {
super.init(appContext: appContext)
isOpaque = false
isUserInteractionEnabled = false
}
func acknowledge() { acknowledged = true; setNeedsDisplay() }
func reset(_ key: String) {
if key != resetKey { resetKey = key; acknowledged = false; setNeedsDisplay() }
}
override func draw(_ rect: CGRect) {
if acknowledged { tint.setFill(); UIBezierPath(roundedRect: bounds, cornerRadius: radius).fill() }
}
}
@@ -0,0 +1,5 @@
{
"name": "paseo-press-feedback",
"version": "0.1.0",
"private": true
}
@@ -0,0 +1,104 @@
import { expect, it } from "vitest";
import { acceptWorkspaceRowPress, type WorkspaceRowPress } from "./accept";
function rowPress(overrides: Partial<WorkspaceRowPress> = {}) {
const visible: string[] = [];
const press: WorkspaceRowPress = {
cancelled: false,
native: true,
selected: false,
sidebar: "overlay",
target: "agent-list",
acknowledge: () => visible.push("row"),
navigate: () => visible.push("workspace"),
...overrides,
};
return { press, visible };
}
it("acknowledges an accepted row before navigation starts", () => {
const { press, visible } = rowPress();
expect(acceptWorkspaceRowPress(press)).toBe(true);
expect(visible).toEqual(["row", "workspace"]);
});
it("does not acknowledge or navigate after scrolling or a long press claims the touch", () => {
const { press, visible } = rowPress({ cancelled: true });
expect(acceptWorkspaceRowPress(press)).toBe(false);
expect(visible).toEqual([]);
});
it("accepts only the first row while its selection closes the sidebar", () => {
const { press, visible } = rowPress();
press.navigate = () => {
press.target = "agent";
visible.push("workspace B");
};
expect(acceptWorkspaceRowPress(press)).toBe(true);
expect(acceptWorkspaceRowPress(press)).toBe(false);
expect(visible).toEqual(["row", "workspace B"]);
});
it.each([
{
name: "open phone or tablet overlay",
sidebar: "overlay",
target: "agent-list",
native: true,
accepted: true,
},
{ name: "closing overlay", sidebar: "overlay", target: "agent", native: true, accepted: false },
{
name: "explorer covering an overlay",
sidebar: "overlay",
target: "file-explorer",
native: true,
accepted: false,
},
{ name: "docked sidebar", sidebar: "dock", target: "agent", native: true, accepted: true },
{
name: "docked sidebar beside explorer",
sidebar: "dock",
target: "file-explorer",
native: true,
accepted: true,
},
{ name: "hidden sidebar", sidebar: "hidden", target: "agent", native: true, accepted: false },
{
name: "suspended open overlay",
sidebar: "suspended",
target: "agent-list",
native: true,
accepted: true,
},
{
name: "suspended closed overlay",
sidebar: "suspended",
target: "agent",
native: true,
accepted: false,
},
{
name: "web sidebar independent of native panel target",
sidebar: "overlay",
target: "agent",
native: false,
accepted: true,
},
] as const)("uses the panel state for $name", ({ accepted, name: _, ...state }) => {
const { press, visible } = rowPress(state);
expect(acceptWorkspaceRowPress(press)).toBe(accepted);
expect(visible).toEqual(accepted ? ["row", "workspace"] : []);
});
it("does not leave an acknowledgement over an already selected docked row", () => {
const { press, visible } = rowPress({ selected: true, sidebar: "dock", target: "agent" });
expect(acceptWorkspaceRowPress(press)).toBe(true);
expect(visible).toEqual(["workspace"]);
});
it("still acknowledges a selected overlay row before the sidebar leaves", () => {
const { press, visible } = rowPress({ selected: true });
expect(acceptWorkspaceRowPress(press)).toBe(true);
expect(visible).toEqual(["row", "workspace"]);
});
@@ -0,0 +1,20 @@
import type { LeftSidebarPresentation, MobilePanelView } from "@/stores/panel-store";
export interface WorkspaceRowPress {
cancelled: boolean;
native: boolean;
selected: boolean;
sidebar: LeftSidebarPresentation;
target: MobilePanelView;
acknowledge(): void;
navigate(): void;
}
export function acceptWorkspaceRowPress(press: WorkspaceRowPress): boolean {
if (press.cancelled) return false;
if (press.native && press.sidebar !== "dock" && press.target !== "agent-list") return false;
// A selected docked row neither changes selection nor leaves the screen to clear the fill.
if (!press.selected || press.sidebar !== "dock") press.acknowledge();
press.navigate();
return true;
}
@@ -0,0 +1,47 @@
import { requireNativeView } from "expo";
import { type Ref } from "react";
import { usePanelStore } from "@/stores/panel-store";
import { withUnistyles } from "react-native-unistyles";
import { View, StyleSheet, type ViewProps } from "react-native";
export interface RowFeedbackHandle {
acknowledge(): Promise<void>;
}
interface FeedbackProps extends ViewProps {
ref?: Ref<RowFeedbackHandle>;
resetKey: string;
tint?: string;
radius?: number;
}
const NativeFeedback = withUnistyles(requireNativeView<FeedbackProps>("PaseoPressFeedback"));
const mapTheme = (theme: import("@/styles/theme").Theme) => ({
tint: theme.colors.interactionHighlight,
radius: theme.borderRadius.lg,
});
export function RowFeedback({
ref,
selected,
}: {
ref?: Ref<RowFeedbackHandle>;
selected: boolean;
}) {
const revision = usePanelStore((state) => state.mobilePanel.revision);
// Selection takes over the fill; a new panel transition clears it before reopening.
const resetKey = `${selected}:${revision}`;
return (
<View pointerEvents="none" style={styles.overlay}>
<NativeFeedback
ref={ref}
resetKey={resetKey}
pointerEvents="none"
style={styles.fill}
uniProps={mapTheme}
/>
</View>
);
}
const styles = StyleSheet.create({
overlay: { position: "absolute", top: 0, right: 0, bottom: 0, left: 0 },
fill: { flex: 1 },
});
@@ -0,0 +1,7 @@
import type { Ref } from "react";
export interface RowFeedbackHandle {
acknowledge(): Promise<void>;
}
export function RowFeedback(_props: { ref?: Ref<RowFeedbackHandle>; selected: boolean }) {
return null;
}
@@ -0,0 +1,38 @@
import { useCallback, useRef, type RefObject } from "react";
import { isNative } from "@/constants/platform";
import { usePanelStore } from "@/stores/panel-store";
import { acceptWorkspaceRowPress } from "../accept";
import { RowFeedback, type RowFeedbackHandle } from "./feedback";
export function useWorkspaceRowPress({
didLongPressRef,
onPress,
selected,
}: {
didLongPressRef?: RefObject<boolean>;
onPress(): void;
selected: boolean;
}) {
const feedback = useRef<RowFeedbackHandle>(null);
const handlePress = useCallback(() => {
const cancelled = didLongPressRef?.current ?? false;
if (didLongPressRef) didLongPressRef.current = false;
const panel = usePanelStore.getState();
acceptWorkspaceRowPress({
cancelled,
native: isNative,
selected,
sidebar: panel.layout.leftSidebar,
target: panel.mobilePanel.target,
// Paint on the native UI thread before navigation occupies JS; never wait for paint.
acknowledge: () => {
void feedback.current?.acknowledge();
},
navigate: onPress,
});
}, [didLongPressRef, onPress, selected]);
return {
onPress: handlePress,
feedback: <RowFeedback ref={feedback} selected={selected} />,
};
}
@@ -0,0 +1,85 @@
import { expect, it } from "vitest";
import { createInitialDrawDistanceExpansion } from "./initial-draw-distance";
function mountInitialBuffer() {
let nextId = 0;
const pending = new Map<number, () => void>();
const expansions: string[] = [];
const buffer = createInitialDrawDistanceExpansion(
{
requestFrame(callback) {
const id = ++nextId;
pending.set(id, callback);
return id;
},
cancelFrame(id) {
pending.delete(id);
},
},
() => expansions.push("full draw distance"),
);
function advanceFrame() {
const callbacks = [...pending.values()];
pending.clear();
for (const callback of callbacks) callback();
}
return { buffer, expansions, advanceFrame };
}
it("requests the full draw distance after two frame callbacks once the reveal is eligible", () => {
const { buffer, expansions, advanceFrame } = mountInitialBuffer();
buffer.schedule();
advanceFrame();
expect(expansions).toEqual([]);
advanceFrame();
expect(expansions).toEqual(["full draw distance"]);
});
it.each([0, 1])(
"leaves initial draw distance in place during a gesture after %i frames",
(elapsed) => {
const { buffer, expansions, advanceFrame } = mountInitialBuffer();
buffer.schedule();
if (elapsed === 1) advanceFrame();
buffer.pauseForScroll();
advanceFrame();
advanceFrame();
expect(expansions).toEqual([]);
buffer.resumeAfterScroll();
advanceFrame();
expect(expansions).toEqual([]);
advanceFrame();
expect(expansions).toEqual(["full draw distance"]);
},
);
it("cancels a covered reveal and schedules again when it is eligible", () => {
const { buffer, expansions, advanceFrame } = mountInitialBuffer();
const cancel = buffer.schedule();
advanceFrame();
cancel();
advanceFrame();
expect(expansions).toEqual([]);
buffer.schedule();
advanceFrame();
advanceFrame();
expect(expansions).toEqual(["full draw distance"]);
});
it("keeps a covered list at its initial draw distance when a gesture ends", () => {
const { buffer, expansions, advanceFrame } = mountInitialBuffer();
buffer.pauseForScroll();
const cover = buffer.schedule();
advanceFrame();
advanceFrame();
expect(expansions).toEqual([]);
cover();
buffer.resumeAfterScroll();
advanceFrame();
advanceFrame();
expect(expansions).toEqual([]);
});
@@ -0,0 +1,50 @@
interface InitialDrawDistanceFrames {
requestFrame(callback: () => void): number;
cancelFrame(id: number): void;
}
export function createInitialDrawDistanceExpansion(
frames: InitialDrawDistanceFrames,
expand: () => void,
) {
let frame: number | null = null;
let eligible = false;
let scrolling = false;
let expanded = false;
function cancel() {
if (frame === null) return;
frames.cancelFrame(frame);
frame = null;
}
function scheduleIfIdle() {
if (!eligible || scrolling || expanded || frame !== null) return;
frame = frames.requestFrame(() => {
frame = frames.requestFrame(() => {
frame = null;
expanded = true;
expand();
});
});
}
return {
schedule() {
eligible = true;
scheduleIfIdle();
return () => {
eligible = false;
cancel();
};
},
pauseForScroll() {
scrolling = true;
cancel();
},
resumeAfterScroll() {
scrolling = false;
scheduleIfIdle();
},
};
}
@@ -40,6 +40,7 @@ function wait(ms: number): Promise<void> {
}
const LIST_STYLE = { flex: 1 };
const ESTIMATED_LIST_SIZE = { width: 300, height: VIEWPORT_HEIGHT };
const rowStyles = new Map<number, { height: number }>();
function rowStyle(height: number) {
@@ -89,6 +90,10 @@ function keyExtractor(item: Row) {
return item.id;
}
function estimateRowSize(item: Row) {
return item.height;
}
function mountList(heldRowId: string) {
const listRef = React.createRef<LegendListRef>();
const maintainVisibleContentPosition = holdOnly(heldRowId);
@@ -264,3 +269,49 @@ it("keeps measured row sizes while the list is not laid out", async () => {
expect(rows.map((row) => list.rowSize(row.id))).toEqual([40, 40, 40]);
});
it("expands the draw distance of a placed end-anchored list without moving its bottom row", async () => {
const rows: Row[] = Array.from({ length: 99 }, (_, index) => ({
id: `row-${index}`,
height: index % 2 === 0 ? 20 : 60,
}));
const lastRow = { id: "last-row", height: 180 };
rows.push(lastRow);
const listRef = React.createRef<LegendListRef>();
function render(drawDistance: number) {
act(() => {
root.render(
<LegendList
ref={listRef}
data={rows}
renderItem={renderRow}
keyExtractor={keyExtractor}
estimatedListSize={ESTIMATED_LIST_SIZE}
recycleItems={false}
getEstimatedItemSize={estimateRowSize}
drawDistance={drawDistance}
alignItemsAtEnd
initialScrollAtEnd
style={LIST_STYLE}
/>,
);
});
}
render(50);
await wait(300);
const initial = listRef.current?.getState();
const lastRowTop = initial?.positionByKey(lastRow.id);
expect(initial?.sizes.get(lastRow.id)).toBe(lastRow.height);
expect(initial?.sizes.size).toBeLessThan(8);
expect(initial?.scroll).toBe(Number(lastRowTop) + lastRow.height - VIEWPORT_HEIGHT);
render(1000);
await wait(150);
const expanded = listRef.current?.getState();
expect(expanded?.sizes.size).toBeGreaterThan(20);
expect(expanded?.scroll).toBe(initial?.scroll);
expect(expanded?.positionByKey(lastRow.id)).toBe(lastRowTop);
});
@@ -9,6 +9,7 @@ import {
useState,
} from "react";
import {
Dimensions,
View,
type LayoutChangeEvent,
type NativeScrollEvent,
@@ -31,6 +32,7 @@ import { useTranslation } from "react-i18next";
import type { StreamItem } from "@/types/stream";
import type { Theme } from "@/styles/theme";
import { useStableEvent } from "@/hooks/use-stable-event";
import { useIsMobilePanelActive } from "@/mobile-panels/provider";
import { shouldAnimateNativeScrollToBottom } from "./native-scroll-animation";
import { useRevisedHistoryRows } from "./history-row-revision";
import { useBottomAnchorController } from "./bottom-anchor-controller";
@@ -43,6 +45,7 @@ import {
} from "./native-disclosure-hold";
import { EndPinnedScrollView } from "./native-end-pin";
import { createNewRowEstimates } from "./new-row-estimates";
import { createInitialDrawDistanceExpansion } from "./initial-draw-distance";
import type {
DisclosureTriggerEdge,
StreamRenderInput,
@@ -132,6 +135,14 @@ function keyExtractor(item: { id: string }): string {
return item.id;
}
function InitialDrawDistanceExpansion({ schedule }: { schedule: () => () => void }) {
const isSidebarActive = useIsMobilePanelActive("agent-list");
useLayoutEffect(() => {
if (!isSidebarActive) return schedule();
}, [schedule, isSidebarActive]);
return null;
}
function NativeStreamViewport(props: StreamRenderInput & { strategy: StreamStrategy }) {
const {
agentId,
@@ -157,6 +168,16 @@ function NativeStreamViewport(props: StreamRenderInput & { strategy: StreamStrat
strategy,
} = props;
const isActive = useRetainedPanelActive();
// Keep a screenful in reserve for a fling that starts before the full buffer mounts.
const [drawDistance, setDrawDistance] = useState(() =>
Math.min(Dimensions.get("window").height, NATIVE_STREAM_DRAW_DISTANCE),
);
const [drawDistanceExpansion] = useState(() =>
createInitialDrawDistanceExpansion(
{ requestFrame: requestAnimationFrame, cancelFrame: cancelAnimationFrame },
() => setDrawDistance(NATIVE_STREAM_DRAW_DISTANCE),
),
);
const { t } = useTranslation();
const { renderHistoryMountedRow, renderLiveHeadRow, renderLiveAuxiliary } = renderers;
const listRef = useRef<LegendListRef>(null);
@@ -569,6 +590,9 @@ function NativeStreamViewport(props: StreamRenderInput & { strategy: StreamStrat
cancelDisclosureHold();
bottomAnchorController.beginUserScroll();
historyStartPagination.rearm();
// The reserve follows the gesture. Expanding the buffer here shifted the measured
// content and returned the viewport to the end during an Android fling.
drawDistanceExpansion.pauseForScroll();
});
useEffect(() => {
@@ -591,6 +615,7 @@ function NativeStreamViewport(props: StreamRenderInput & { strategy: StreamStrat
bottomAnchorController.endUserScroll({
isAtEnd: isReleasedAtEndRef.current || isAtEnd(),
});
drawDistanceExpansion.resumeAfterScroll();
});
});
@@ -610,6 +635,7 @@ function NativeStreamViewport(props: StreamRenderInput & { strategy: StreamStrat
bottomAnchorController.endUserScroll({
isAtEnd: isReleasedAtEndRef.current || isScrollEventAtEnd(event),
});
drawDistanceExpansion.resumeAfterScroll();
},
);
@@ -732,6 +758,11 @@ function NativeStreamViewport(props: StreamRenderInput & { strategy: StreamStrat
);
}, [historyStartPaginationState, hasOlderHistory, retryHistoryStart, t]);
const canExpandDrawDistance =
drawDistance < NATIVE_STREAM_DRAW_DISTANCE &&
isActive &&
isListPlaced &&
historyStartPaginationState.visible;
return (
<View style={viewportStyle}>
<LegendList<StreamItem>
@@ -764,11 +795,14 @@ function NativeStreamViewport(props: StreamRenderInput & { strategy: StreamStrat
recycleItems={false}
alignItemsAtEnd
initialScrollAtEnd
drawDistance={NATIVE_STREAM_DRAW_DISTANCE}
drawDistance={drawDistance}
getEstimatedItemSize={getEstimatedItemSize}
scrollEnabled={scrollEnabled}
showsVerticalScrollIndicator
/>
{canExpandDrawDistance ? (
<InitialDrawDistanceExpansion schedule={drawDistanceExpansion.schedule} />
) : null}
{!historyStartPaginationState.visible ? (
<View pointerEvents="none" testID="agent-initial-fill" style={initialFillStyle}>
<ThemedLoadingSpinner size="large" uniProps={foregroundMutedColorMapping} />
+18 -1
View File
@@ -3,7 +3,6 @@ export interface AudioEngineCallbacks {
onVolumeLevel(level: number): void;
onInterruption?(): void;
onCaptureAvailabilityChange?(available: boolean): void;
onError?(error: Error): void;
}
export interface AudioPlaybackSource {
@@ -31,3 +30,21 @@ export interface AudioEngineOptions {
traceLabel?: string;
captureMode?: "conversation" | "dictation";
}
export type AudioCaptureStartFailure =
| "permissionDenied"
| "insecureContext"
| "unsupported"
| "noMicrophone"
| "startFailed";
export class AudioCaptureStartError extends Error {
constructor(
readonly reason: AudioCaptureStartFailure,
message: string,
options?: ErrorOptions,
) {
super(message, options);
this.name = "AudioCaptureStartError";
}
}
+50 -229
View File
@@ -1,239 +1,60 @@
import type {
AudioEngine,
AudioEngineCallbacks,
AudioEngineOptions,
AudioPlaybackSource,
} from "./audio-engine-types";
import type { AudioEngine, AudioEngineCallbacks, AudioEngineOptions } from "./audio-engine-types";
import { createAudioPlayer, setAudioModeAsync } from "expo-audio";
import { File, Paths } from "expo-file-system";
import { createPlaybackQueue } from "./playback";
import { playFile } from "./file-playback";
import { playPcm16 } from "./pcm";
import { createNativeAudioEngine, type NativeAudioEngineDeps } from "./native-audio-engine";
// AVPlayer needs a file extension to recognize local encoded audio on iOS.
const FILE_EXTENSIONS: Record<string, string> = {
"audio/wav": "wav",
"audio/x-wav": "wav",
"audio/wave": "wav",
"audio/mpeg": "mp3",
"audio/mp3": "mp3",
"audio/mp4": "m4a",
"audio/aac": "aac",
"audio/ogg": "ogg",
"audio/flac": "flac",
};
let nextFileId = 0;
async function playEncodedAudio(
bytes: Uint8Array,
mimeType: string,
signal: AbortSignal,
context: { isCaptureActive(): boolean },
): Promise<number> {
// Capture owns its audio session while active. File playback alone must not
// initialize the microphone or the native two-way engine.
if (!context.isCaptureActive()) {
await setAudioModeAsync({
playsInSilentMode: true,
allowsRecording: false,
interruptionMode: "duckOthers",
interruptionModeAndroid: "duckOthers",
});
}
if (signal.aborted) throw new Error("Playback stopped");
const extension = FILE_EXTENSIONS[mimeType.split(";")[0].trim()] ?? "audio";
const file = new File(Paths.cache, `paseo-audio-${Date.now()}-${nextFileId++}.${extension}`);
try {
file.write(bytes);
const player = createAudioPlayer(file.uri, {
updateInterval: 100,
keepAudioSessionActive: context.isCaptureActive(),
});
return await playFile(player, signal);
} finally {
if (file.exists) file.delete();
}
}
export function createAudioEngine(
callbacks: AudioEngineCallbacks,
options?: AudioEngineOptions,
): AudioEngine {
const native = require("@getpaseo/expo-two-way-audio");
const refs: {
initialized: boolean;
captureActive: boolean;
muted: boolean;
destroyed: boolean;
} = {
initialized: false,
captureActive: false,
muted: false,
destroyed: false,
};
const microphoneSubscription = native.addExpoTwoWayAudioEventListener(
"onMicrophoneData",
(event: { data: Uint8Array }) => {
if (!refs.captureActive || refs.muted) {
return;
}
const pcm = event.data;
callbacks.onCaptureData(pcm);
},
);
const volumeSubscription = native.addExpoTwoWayAudioEventListener(
"onInputVolumeLevelData",
(event: { data: number }) => {
if (!refs.captureActive) {
return;
}
const level = refs.muted ? 0 : event.data;
callbacks.onVolumeLevel(level);
},
);
const interruptionSubscription = native.addExpoTwoWayAudioEventListener(
"onAudioInterruption",
(event: { data: string }) => {
if (event.data === "capture_paused" || event.data === "capture_resumed") {
if (refs.captureActive) {
callbacks.onCaptureAvailabilityChange?.(event.data === "capture_resumed");
}
return;
}
if (event.data !== "blocked") {
return;
}
const wasCaptureActive = refs.captureActive;
refs.captureActive = false;
refs.muted = false;
callbacks.onVolumeLevel(0);
if (wasCaptureActive) {
callbacks.onInterruption?.();
}
},
);
async function ensureInitialized(): Promise<void> {
if (refs.initialized) {
return;
}
const success = await native.initialize();
if (!success) {
throw new Error("expo-two-way-audio: native initialize() returned false");
}
refs.initialized = true;
}
/**
* Release the OS audio session as soon as we are neither capturing nor playing.
* Holding it keeps the user's background music paused — on iOS the non-mixing
* `.playAndRecord` category survives backgrounding and is re-asserted on every
* foreground, so an unreleased session means their music never comes back.
*/
function releaseSessionIfIdle(): void {
if (!refs.initialized || refs.destroyed) {
return;
}
if (refs.captureActive || playback.isPlaying()) {
return;
}
// The wrapper no-ops on binaries whose native module predates this function.
native.releaseAudioSession();
}
async function ensureMicrophonePermission(): Promise<void> {
let permission = await native.getMicrophonePermissionsAsync().catch(() => null);
if (!permission?.granted) {
permission = await native.requestMicrophonePermissionsAsync().catch(() => null);
}
if (!permission?.granted) {
throw new Error(
"Microphone permission is required to capture audio. Please enable microphone access in system settings.",
);
}
}
let nextFileId = 0;
async function playAudio(audio: AudioPlaybackSource, signal: AbortSignal): Promise<number> {
const bytes = new Uint8Array(await audio.arrayBuffer());
if (signal.aborted) throw new Error("Playback stopped");
if (audio.type.startsWith("audio/pcm")) {
await ensureInitialized();
return playPcm16(bytes, audio.type, signal, native);
}
// Capture owns its audio session while active. File playback alone must not
// initialize the microphone or the native two-way engine.
if (!refs.captureActive) {
await setAudioModeAsync({
playsInSilentMode: true,
allowsRecording: false,
interruptionMode: "duckOthers",
interruptionModeAndroid: "duckOthers",
});
}
if (signal.aborted) throw new Error("Playback stopped");
// AVPlayer needs a file extension to recognize local encoded audio on iOS.
const extension =
{
"audio/wav": "wav",
"audio/x-wav": "wav",
"audio/wave": "wav",
"audio/mpeg": "mp3",
"audio/mp3": "mp3",
"audio/mp4": "m4a",
"audio/aac": "aac",
"audio/ogg": "ogg",
"audio/flac": "flac",
}[audio.type.split(";")[0].trim()] ?? "audio";
const file = new File(Paths.cache, `paseo-audio-${Date.now()}-${nextFileId++}.${extension}`);
try {
file.write(bytes);
const player = createAudioPlayer(file.uri, {
updateInterval: 100,
keepAudioSessionActive: refs.captureActive,
});
return await playFile(player, signal);
} finally {
if (file.exists) file.delete();
}
}
const playback = createPlaybackQueue(playAudio, releaseSessionIfIdle);
return {
async initialize() {
await ensureInitialized();
},
async destroy() {
if (refs.destroyed) {
return;
}
refs.destroyed = true;
playback.destroy();
if (refs.captureActive) {
native.toggleRecording(false);
refs.captureActive = false;
}
refs.muted = false;
callbacks.onVolumeLevel(0);
if (refs.initialized) {
native.tearDown();
refs.initialized = false;
}
microphoneSubscription.remove();
volumeSubscription.remove();
interruptionSubscription.remove();
},
async startCapture() {
if (refs.captureActive) {
return;
}
try {
await ensureMicrophonePermission();
await ensureInitialized();
const isRecording =
options?.captureMode === "dictation"
? native.startDictation()
: native.toggleRecording(true);
if (!isRecording) {
throw new Error(
"Microphone capture could not start because Android audio focus is unavailable.",
);
}
refs.captureActive = true;
} catch (error) {
const wrapped = error instanceof Error ? error : new Error(String(error));
callbacks.onError?.(wrapped);
throw wrapped;
}
},
async stopCapture() {
if (refs.captureActive) {
native.toggleRecording(false);
}
refs.captureActive = false;
refs.muted = false;
callbacks.onVolumeLevel(0);
callbacks.onCaptureAvailabilityChange?.(true);
releaseSessionIfIdle();
},
toggleMute() {
refs.muted = !refs.muted;
if (refs.muted) {
callbacks.onVolumeLevel(0);
}
return refs.muted;
},
isMuted() {
return refs.muted;
},
play: playback.play,
stop: playback.stop,
clearQueue: playback.clearQueue,
isPlaying: playback.isPlaying,
};
const native: NativeAudioEngineDeps["native"] = require("@getpaseo/expo-two-way-audio");
return createNativeAudioEngine({ native, playEncodedAudio }, callbacks, options);
}
+1 -3
View File
@@ -292,9 +292,7 @@ export function createAudioEngine(
refs.gain = gain;
} catch (error) {
await stopCapture();
const wrapped = error instanceof Error ? error : new Error(String(error));
callbacks.onError?.(wrapped);
throw wrapped;
throw error;
}
},
+1
View File
@@ -1,3 +1,4 @@
export { createAudioEngine } from "./audio-engine";
export type { AudioEngine, AudioEngineCallbacks, AudioPlaybackSource } from "./audio-engine-types";
export { createPlayAudio } from "./play-audio";
export { AudioCaptureStartError, type AudioCaptureStartFailure } from "./audio-engine-types";
@@ -0,0 +1,188 @@
import { expect, test } from "vitest";
import type { AudioEngineCallbacks } from "./audio-engine-types";
import { createNativeAudioEngine, type NativeAudioEngineDeps } from "./native-audio-engine";
// In-memory stand-in for `@getpaseo/expo-two-way-audio`, matching its iOS and Android modules:
// one native engine per app, `initialize()` reuses it, `tearDown()` frees it, and
// `toggleRecording(true)` / `startDictation()` return false while no engine exists.
// Session handling follows iOS: recording activates the OS audio session, stopping does not
// deactivate it, and `releaseAudioSession()` deactivates it only while nothing is recording.
function fakeTwoWayAudio({
captureAvailable = true,
permissionRequestError,
}: { captureAvailable?: boolean; permissionRequestError?: Error } = {}) {
let engine: { recording: boolean; sessionActive: boolean } | null = null;
const calls = { initialize: 0, tearDown: 0 };
const native: NativeAudioEngineDeps["native"] = {
async initialize() {
calls.initialize += 1;
engine ??= { recording: false, sessionActive: false };
return true;
},
tearDown() {
calls.tearDown += 1;
engine = null;
},
toggleRecording(value: boolean) {
if (!engine) return false;
engine.recording = value;
if (value) engine.sessionActive = true;
return value;
},
startDictation() {
if (!engine || !captureAvailable) return false;
engine.recording = true;
engine.sessionActive = true;
return true;
},
releaseAudioSession() {
if (engine && !engine.recording) engine.sessionActive = false;
},
playPCMData() {},
resumePlayback() {},
stopPlayback() {},
async getMicrophonePermissionsAsync() {
return { granted: !permissionRequestError };
},
async requestMicrophonePermissionsAsync() {
if (permissionRequestError) throw permissionRequestError;
return { granted: true };
},
addExpoTwoWayAudioEventListener() {
return { remove() {} };
},
};
return {
native,
calls,
hasEngine: () => engine !== null,
sessionActive: () => engine?.sessionActive ?? false,
};
}
const recordingCallbacks: AudioEngineCallbacks = {
onCaptureData() {},
onVolumeLevel() {},
};
function deps(native: NativeAudioEngineDeps["native"]): NativeAudioEngineDeps {
return {
native,
playEncodedAudio: async () => 0,
};
}
// Issue #4147: on iPhone, voice mode failed with "Android audio focus is unavailable"
// after the dictation engine was destroyed.
test("voice capture starts after the dictation engine is destroyed", async () => {
const fake = fakeTwoWayAudio();
const voiceEngine = createNativeAudioEngine(deps(fake.native), recordingCallbacks);
await voiceEngine.initialize();
const dictationEngine = createNativeAudioEngine(deps(fake.native), recordingCallbacks, {
captureMode: "dictation",
});
await dictationEngine.startCapture();
await dictationEngine.stopCapture();
await dictationEngine.destroy();
await expect(voiceEngine.startCapture()).resolves.toBeUndefined();
});
test("destroying dictation while voice is capturing keeps the native engine", async () => {
const fake = fakeTwoWayAudio();
const voiceEngine = createNativeAudioEngine(deps(fake.native), recordingCallbacks);
await voiceEngine.startCapture();
const dictationEngine = createNativeAudioEngine(deps(fake.native), recordingCallbacks, {
captureMode: "dictation",
});
await dictationEngine.initialize();
await dictationEngine.destroy();
expect(fake.hasEngine()).toBe(true);
expect(fake.calls.tearDown).toBe(0);
});
// Review on #9051: with voice holding the engine, destroying an active dictation engine must
// still hand back the idle OS audio session so background music resumes (#2866).
test("destroying active dictation releases the idle audio session while voice keeps the engine", async () => {
const fake = fakeTwoWayAudio();
const voiceEngine = createNativeAudioEngine(deps(fake.native), recordingCallbacks);
await voiceEngine.initialize();
const dictationEngine = createNativeAudioEngine(deps(fake.native), recordingCallbacks, {
captureMode: "dictation",
});
await dictationEngine.startCapture();
await dictationEngine.destroy();
expect(fake.hasEngine()).toBe(true);
expect(fake.sessionActive()).toBe(false);
});
test("destroying dictation does not release the session while voice is recording", async () => {
const fake = fakeTwoWayAudio();
const voiceEngine = createNativeAudioEngine(deps(fake.native), recordingCallbacks);
await voiceEngine.startCapture();
const dictationEngine = createNativeAudioEngine(deps(fake.native), recordingCallbacks, {
captureMode: "dictation",
});
await dictationEngine.initialize();
await dictationEngine.destroy();
expect(fake.sessionActive()).toBe(true);
});
test("destroying the last engine tears the native engine down", async () => {
const fake = fakeTwoWayAudio();
const voiceEngine = createNativeAudioEngine(deps(fake.native), recordingCallbacks);
const dictationEngine = createNativeAudioEngine(deps(fake.native), recordingCallbacks, {
captureMode: "dictation",
});
await voiceEngine.initialize();
await dictationEngine.initialize();
await dictationEngine.destroy();
await voiceEngine.destroy();
expect(fake.calls.tearDown).toBe(1);
expect(fake.hasEngine()).toBe(false);
});
test("an engine destroyed while initializing does not keep the native engine", async () => {
const fake = fakeTwoWayAudio();
const engine = createNativeAudioEngine(deps(fake.native), recordingCallbacks);
const initializing = expect(engine.initialize()).rejects.toThrow("Audio engine destroyed");
await engine.destroy();
await initializing;
expect(fake.hasEngine()).toBe(false);
});
test("a failed native recording start rejects with a platform-neutral error", async () => {
const fake = fakeTwoWayAudio({ captureAvailable: false });
const engine = createNativeAudioEngine(deps(fake.native), recordingCallbacks, {
captureMode: "dictation",
});
await expect(engine.startCapture()).rejects.toThrow(/^Microphone capture could not start\.$/);
await engine.destroy();
});
test("a failed native permission request retains the permission reason and original error", async () => {
const permissionRequestError = new Error("The current activity is not available.");
const fake = fakeTwoWayAudio({ permissionRequestError });
const engine = createNativeAudioEngine(deps(fake.native), recordingCallbacks, {
captureMode: "dictation",
});
await expect(engine.startCapture()).rejects.toMatchObject({
reason: "permissionDenied",
cause: permissionRequestError,
});
expect(fake.hasEngine()).toBe(false);
await engine.destroy();
});
@@ -0,0 +1,292 @@
import type {
AudioEngine,
AudioEngineCallbacks,
AudioEngineOptions,
AudioPlaybackSource,
} from "./audio-engine-types";
import { AudioCaptureStartError } from "./audio-engine-types";
import { createNativeEngineLease, type NativeEngineLease } from "./native-engine-lease";
import { createPlaybackQueue } from "./playback";
import { playPcm16 } from "./pcm";
interface NativeSubscription {
remove(): void;
}
interface MicrophonePermission {
granted: boolean;
}
/** The surface of `@getpaseo/expo-two-way-audio` the engine drives. */
export interface TwoWayAudioModule {
initialize(): Promise<boolean>;
tearDown(): void;
toggleRecording(value: boolean): boolean;
startDictation(): boolean;
releaseAudioSession(): void;
playPCMData(data: Uint8Array): void;
getMicrophonePermissionsAsync(): Promise<MicrophonePermission>;
requestMicrophonePermissionsAsync(): Promise<MicrophonePermission>;
addExpoTwoWayAudioEventListener(
event: "onMicrophoneData",
listener: (event: { data: Uint8Array }) => void,
): NativeSubscription;
addExpoTwoWayAudioEventListener(
event: "onInputVolumeLevelData",
listener: (event: { data: number }) => void,
): NativeSubscription;
addExpoTwoWayAudioEventListener(
event: "onAudioInterruption",
listener: (event: { data: string }) => void,
): NativeSubscription;
}
export interface NativeAudioEngineDeps {
native: TwoWayAudioModule & Parameters<typeof playPcm16>[3];
/** Plays encoded audio (wav, mp3, ...) through the platform player. */
playEncodedAudio(
bytes: Uint8Array,
mimeType: string,
signal: AbortSignal,
context: { isCaptureActive(): boolean },
): Promise<number>;
}
// Voice mode and dictation each create an engine, but the native module holds a single engine
// per app. Every JS engine on the same module shares one lease, so one engine's teardown
// cannot free the native engine another still uses (#4147).
const leases = new WeakMap<TwoWayAudioModule, NativeEngineLease>();
function leaseFor(native: TwoWayAudioModule): NativeEngineLease {
let lease = leases.get(native);
if (!lease) {
lease = createNativeEngineLease(native);
leases.set(native, lease);
}
return lease;
}
export function createNativeAudioEngine(
deps: NativeAudioEngineDeps,
callbacks: AudioEngineCallbacks,
options?: AudioEngineOptions,
): AudioEngine {
const { native } = deps;
const lease = leaseFor(native);
const refs: {
initialized: boolean;
captureActive: boolean;
muted: boolean;
destroyed: boolean;
} = {
initialized: false,
captureActive: false,
muted: false,
destroyed: false,
};
const microphoneSubscription = native.addExpoTwoWayAudioEventListener(
"onMicrophoneData",
(event: { data: Uint8Array }) => {
if (!refs.captureActive || refs.muted) {
return;
}
const pcm = event.data;
callbacks.onCaptureData(pcm);
},
);
const volumeSubscription = native.addExpoTwoWayAudioEventListener(
"onInputVolumeLevelData",
(event: { data: number }) => {
if (!refs.captureActive) {
return;
}
const level = refs.muted ? 0 : event.data;
callbacks.onVolumeLevel(level);
},
);
const interruptionSubscription = native.addExpoTwoWayAudioEventListener(
"onAudioInterruption",
(event: { data: string }) => {
if (event.data === "capture_paused" || event.data === "capture_resumed") {
if (refs.captureActive) {
callbacks.onCaptureAvailabilityChange?.(event.data === "capture_resumed");
}
return;
}
if (event.data !== "blocked") {
return;
}
const wasCaptureActive = refs.captureActive;
refs.captureActive = false;
refs.muted = false;
callbacks.onVolumeLevel(0);
if (wasCaptureActive) {
callbacks.onInterruption?.();
}
},
);
let initializing: Promise<void> | null = null;
async function ensureInitialized(): Promise<void> {
if (refs.destroyed) {
throw new Error("Audio engine destroyed");
}
if (refs.initialized) {
return;
}
initializing ??= (async () => {
try {
await lease.acquire();
if (refs.destroyed) {
releaseNativeEngine();
throw new Error("Audio engine destroyed");
}
refs.initialized = true;
} finally {
initializing = null;
}
})();
await initializing;
}
/**
* Release the OS audio session as soon as we are neither capturing nor playing.
* Holding it keeps the user's background music paused — on iOS the non-mixing
* `.playAndRecord` category survives backgrounding and is re-asserted on every
* foreground, so an unreleased session means their music never comes back.
*/
function releaseSessionIfIdle(): void {
if (!refs.initialized || refs.destroyed) {
return;
}
if (refs.captureActive || playback.isPlaying()) {
return;
}
// The wrapper no-ops on binaries whose native module predates this function.
native.releaseAudioSession();
}
async function ensureMicrophonePermission(): Promise<void> {
const permission = await native.getMicrophonePermissionsAsync().catch(() => null);
if (permission?.granted) return;
let cause: unknown;
try {
if ((await native.requestMicrophonePermissionsAsync()).granted) return;
} catch (error) {
cause = error;
}
throw new AudioCaptureStartError(
"permissionDenied",
"Allow microphone access in system settings and try again.",
{ cause },
);
}
function releaseNativeEngine(): void {
try {
// Native keeps the session active while any wrapper is capturing or playing.
native.releaseAudioSession();
} finally {
lease.release();
}
}
async function playAudio(audio: AudioPlaybackSource, signal: AbortSignal): Promise<number> {
const bytes = new Uint8Array(await audio.arrayBuffer());
if (signal.aborted) throw new Error("Playback stopped");
if (audio.type.startsWith("audio/pcm")) {
await ensureInitialized();
if (signal.aborted) throw new Error("Playback stopped");
return playPcm16(bytes, audio.type, signal, native);
}
return deps.playEncodedAudio(bytes, audio.type, signal, {
isCaptureActive: () => refs.captureActive,
});
}
const playback = createPlaybackQueue(playAudio, releaseSessionIfIdle);
return {
async initialize() {
await ensureInitialized();
},
async destroy() {
if (refs.destroyed) {
return;
}
refs.destroyed = true;
try {
playback.destroy();
if (refs.captureActive) {
refs.captureActive = false;
native.toggleRecording(false);
}
refs.muted = false;
callbacks.onVolumeLevel(0);
} finally {
microphoneSubscription.remove();
volumeSubscription.remove();
interruptionSubscription.remove();
if (refs.initialized) {
refs.initialized = false;
releaseNativeEngine();
}
}
},
async startCapture() {
if (refs.captureActive) {
return;
}
await ensureMicrophonePermission();
await ensureInitialized();
if (refs.destroyed) {
throw new Error("Audio engine destroyed");
}
if (refs.captureActive) {
return;
}
const isRecording =
options?.captureMode === "dictation"
? native.startDictation()
: native.toggleRecording(true);
if (!isRecording) {
throw new Error("Microphone capture could not start.");
}
refs.captureActive = true;
},
async stopCapture() {
if (refs.captureActive) {
native.toggleRecording(false);
}
refs.captureActive = false;
refs.muted = false;
callbacks.onVolumeLevel(0);
callbacks.onCaptureAvailabilityChange?.(true);
releaseSessionIfIdle();
},
toggleMute() {
refs.muted = !refs.muted;
if (refs.muted) {
callbacks.onVolumeLevel(0);
}
return refs.muted;
},
isMuted() {
return refs.muted;
},
play: playback.play,
stop: playback.stop,
clearQueue: playback.clearQueue,
isPlaying: playback.isPlaying,
};
}
@@ -0,0 +1,116 @@
import { expect, test } from "vitest";
import { createNativeEngineLease, type NativeAudioEngineLifecycle } from "./native-engine-lease";
// Mirrors the iOS and Android modules: one engine per app, `initialize()` reuses it,
// `tearDown()` frees it, and `toggleRecording(true)` returns false once it is gone.
function fakeNativeModule(options: { initializeResult?: boolean } = {}) {
let engine: object | null = null;
const calls = { initialize: 0, tearDown: 0 };
let finishInitialize: (() => void) | null = null;
let deferInitialize = false;
const native: NativeAudioEngineLifecycle & {
toggleRecording(value: boolean): boolean;
} = {
initialize() {
calls.initialize += 1;
const complete = () => {
if (options.initializeResult === false) {
return false;
}
engine ??= {};
return true;
};
if (!deferInitialize) {
return Promise.resolve(complete());
}
return new Promise<boolean>((resolve) => {
finishInitialize = () => resolve(complete());
});
},
tearDown() {
calls.tearDown += 1;
engine = null;
},
toggleRecording(value: boolean) {
return engine !== null && value;
},
};
return {
native,
calls,
deferNextInitialize() {
deferInitialize = true;
},
finishInitialize() {
finishInitialize?.();
},
};
}
test("dictation releasing the engine keeps it for voice mode", async () => {
const fake = fakeNativeModule();
const lease = createNativeEngineLease(fake.native);
await lease.acquire(); // voice mode
await lease.acquire(); // dictation
lease.release(); // dictation unmounts
expect(fake.calls.tearDown).toBe(0);
expect(fake.native.toggleRecording(true)).toBe(true);
});
test("the last release tears the native engine down and the next acquire rebuilds it", async () => {
const fake = fakeNativeModule();
const lease = createNativeEngineLease(fake.native);
await lease.acquire();
lease.release();
expect(fake.calls.tearDown).toBe(1);
expect(fake.native.toggleRecording(true)).toBe(false);
await lease.acquire();
expect(fake.calls.initialize).toBe(2);
expect(fake.native.toggleRecording(true)).toBe(true);
});
test("concurrent acquires share one native initialize", async () => {
const fake = fakeNativeModule();
fake.deferNextInitialize();
const lease = createNativeEngineLease(fake.native);
const first = lease.acquire();
const second = lease.acquire();
fake.finishInitialize();
await Promise.all([first, second]);
expect(fake.calls.initialize).toBe(1);
lease.release();
expect(fake.calls.tearDown).toBe(0);
lease.release();
expect(fake.calls.tearDown).toBe(1);
});
test("a failed initialize is not counted as a holder and can be retried", async () => {
const fake = fakeNativeModule({ initializeResult: false });
const lease = createNativeEngineLease(fake.native);
await expect(lease.acquire()).rejects.toThrow("native initialize() returned false");
lease.release();
expect(fake.calls.tearDown).toBe(0);
await expect(lease.acquire()).rejects.toThrow();
expect(fake.calls.initialize).toBe(2);
});
test("a release with no holders is a no-op", async () => {
const fake = fakeNativeModule();
const lease = createNativeEngineLease(fake.native);
await lease.acquire();
lease.release();
lease.release();
await lease.acquire();
lease.release();
expect(fake.calls.tearDown).toBe(2);
});
@@ -0,0 +1,57 @@
/**
* The part of `@getpaseo/expo-two-way-audio` that owns the native engine's lifetime.
* Each platform keeps a single engine per app: `initialize()` reuses it, `tearDown()` frees it.
*/
export interface NativeAudioEngineLifecycle {
initialize(): Promise<boolean>;
tearDown(): void;
}
export interface NativeEngineLease {
/** Ensure the native engine exists and count the caller as a holder. */
acquire(): Promise<void>;
/** Drop one holder; the native engine is torn down when the last holder releases. */
release(): void;
}
/**
* Voice mode and dictation each create their own JS audio engine, but both drive the one
* native engine. Counting holders keeps one consumer's teardown from freeing the engine
* the other still uses (issue #4147: voice mode on iOS could not start after dictation).
*/
export function createNativeEngineLease(native: NativeAudioEngineLifecycle): NativeEngineLease {
let holders = 0;
let ready: Promise<void> | null = null;
async function initialize(): Promise<void> {
const success = await native.initialize();
if (!success) {
throw new Error("expo-two-way-audio: native initialize() returned false");
}
}
return {
async acquire() {
holders += 1;
ready ??= initialize();
try {
await ready;
} catch (error) {
holders -= 1;
ready = null;
throw error;
}
},
release() {
if (holders === 0) {
return;
}
holders -= 1;
if (holders === 0) {
ready = null;
native.tearDown();
}
},
};
}
@@ -5,6 +5,7 @@ import { useTranslation } from "react-i18next";
import {
CalendarClock,
CircleDashed,
Copy,
Folder,
FolderPlus,
History,
@@ -17,6 +18,8 @@ import {
} from "lucide-react-native";
import { withUnistyles } from "react-native-unistyles";
import { useIsCompactFormFactor } from "@/constants/layout";
import { useToast } from "@/contexts/toast-context";
import { getDesktopHost } from "@/desktop/host";
import { useKeyboardShortcutOverrides } from "@/hooks/use-keyboard-shortcut-overrides";
import { useOpenAddProject } from "@/hooks/use-open-add-project";
import { useImportSession } from "@/hooks/use-import-session";
@@ -27,6 +30,7 @@ import { useKeyboardShortcutsStore } from "@/stores/keyboard-shortcuts-store";
import { usePanelStore } from "@/stores/panel-store";
import { useSidebarViewStore } from "@/stores/sidebar-view-store";
import { clearCommandCenterFocusRestoreElement } from "@/utils/command-center-focus-restore";
import { copyToClipboard } from "@/utils/copy-to-clipboard";
import {
buildOpenProjectRoute,
buildSchedulesRoute,
@@ -35,6 +39,7 @@ import {
} from "@/utils/host-routes";
import { getShortcutOs } from "@/utils/shortcut-platform";
import type { CommandCenterContribution, CommandCenterIconProps } from "./contributions";
import { getCommandCenterIcon } from "./icon";
import { useCommandCenterActions } from "./provider";
import { buildGroupingContribution } from "./root-contributions";
@@ -63,6 +68,7 @@ const ThemedCircleDashed = withUnistyles(CircleDashed, (theme) => ({
const ThemedPanelLeft = withUnistyles(PanelLeft, (theme) => ({
color: theme.colors.foregroundMuted,
}));
const BrowserHostCopyIcon = getCommandCenterIcon(Copy);
function PlusIcon({ size }: CommandCenterIconProps) {
return <ThemedPlus size={size} strokeWidth={2.4} />;
@@ -111,6 +117,8 @@ function PanelLeftIcon({ size }: CommandCenterIconProps) {
export function CommandCenterRootActions() {
const keyboardActionDispatcher = useKeyboardActionDispatcher();
const { t } = useTranslation();
const toast = useToast();
const browserHostLabel = getDesktopHost()?.browserHostLabel;
const { overrides } = useKeyboardShortcutOverrides();
const shortcutsAvailable = useHardwareKeyboardPresent();
const openAddProject = useOpenAddProject();
@@ -296,6 +304,32 @@ export function CommandCenterRootActions() {
},
];
if (browserHostLabel) {
availableActions.push({
id: "copy-browser-host-name",
group: "actions",
groupRank: 0,
rank: 9,
keywords: ["browser", "host", "window", browserHostLabel],
visibility: "query",
run: async () => {
try {
await copyToClipboard(browserHostLabel);
toast.copied();
} catch {
toast.error(t("common.errors.unableToCopy"));
}
},
presentation: {
kind: "action",
title: t("shell.commandCenter.copyBrowserHostName"),
subtitle: browserHostLabel,
sectionTitle: t("shell.commandCenter.actions"),
icon: BrowserHostCopyIcon,
},
});
}
if (shortcutsAvailable) {
availableActions.push({
id: "keyboard-shortcuts",
@@ -332,6 +366,7 @@ export function CommandCenterRootActions() {
return availableActions;
}, [
browserHostLabel,
groupMode,
homeRoute,
keyboardActionDispatcher,
@@ -346,6 +381,7 @@ export function CommandCenterRootActions() {
shortcutPlatform,
shortcutsAvailable,
t,
toast,
toggleAgentList,
]);
@@ -1,3 +1,4 @@
import { useWorkspaceRowPress } from "../../modules/paseo-press-feedback";
import { PressHighlight as Pressable } from "@/components/ui/press-highlight";
import { LoadingSpinner } from "@/components/ui/loading-spinner";
import {
@@ -1091,13 +1092,11 @@ function WorkspaceRowInner({
...dragAttributes
} = dragHandleProps?.attributes ?? {};
const handlePress = useCallback(() => {
if (interaction.didLongPressRef.current) {
interaction.didLongPressRef.current = false;
return;
}
onPress();
}, [interaction.didLongPressRef, onPress]);
const { onPress: handlePress, feedback } = useWorkspaceRowPress({
didLongPressRef: interaction.didLongPressRef,
onPress,
selected,
});
const accessibilityState = useMemo(() => ({ selected }), [selected]);
const isEditingTitle = titleEditor !== null;
@@ -1192,6 +1191,7 @@ function WorkspaceRowInner({
onTogglePin={onTogglePin}
/>
</SidebarWorkspaceRowContent>
{feedback}
</SidebarWorkspaceContextMenu>
</View>
);
@@ -1,3 +1,4 @@
import { useWorkspaceRowPress } from "../../../modules/paseo-press-feedback";
import { PressHighlight as Pressable } from "@/components/ui/press-highlight";
import {
memo,
@@ -848,13 +849,11 @@ function StatusWorkspaceRowInnerContent({
const startDragPress = dragInteraction?.handlePressIn;
const moveDragPress = dragInteraction?.handleTouchMove;
const endDragPress = dragInteraction?.handlePressOut;
const handlePress = useCallback(() => {
if (didLongPressRef?.current) {
didLongPressRef.current = false;
return;
}
onPress();
}, [didLongPressRef, onPress]);
const { onPress: handlePress, feedback } = useWorkspaceRowPress({
didLongPressRef,
onPress,
selected,
});
return (
<SidebarWorkspaceRowFrame
@@ -967,6 +966,7 @@ function StatusWorkspaceRowInnerContent({
/>
) : null}
</SidebarWorkspaceRowContent>
{feedback}
</SidebarWorkspaceContextMenu>
</View>
);
@@ -3,7 +3,15 @@ import { createRoot, type Root } from "react-dom/client";
import { Text, View } from "react-native";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { page } from "vitest/browser";
import { MenuRoot, MenuSurface, MenuTrigger, type Size } from "./index";
import {
MenuItem,
MenuRoot,
MenuSubTrigger,
MenuSurface,
MenuTrigger,
type MenuPageDefinition,
type Size,
} from "./index";
// App sources compile against the classic JSX runtime, which expects React on the global.
beforeEach(() => vi.stubGlobal("React", React));
@@ -122,3 +130,65 @@ describe("anchored menu surface on web", () => {
expect(panel.width).toBeGreaterThanOrEqual(body.width);
});
});
const SUB_PAGES: MenuPageDefinition[] = [
{
id: "show",
title: "Show",
content: (
<MenuItem testID="show-option" closeOnSelect={false}>
Archived
</MenuItem>
),
},
];
/** Past the grace a flyout is given after the pointer leaves, so a queued close has landed. */
const HOVER_CLOSE_SETTLED_MS = 600;
/**
* A tap as the browser reports it: the finger enters the row as it lands and leaves it as it
* lifts, and only then does the click arrive.
*/
function tap(target: HTMLElement): void {
const pointer = (type: string) =>
target.dispatchEvent(
new PointerEvent(type, { bubbles: true, pointerType: "touch", isPrimary: true }),
);
act(() => {
pointer("pointerover");
pointer("pointerdown");
pointer("pointerup");
pointer("pointerout");
target.click();
});
}
describe("submenu flyout on a touch screen", () => {
beforeEach(async () => {
await page.viewport(1280, 900);
});
it("stays open after a tap on its row, and after a tap inside it", async () => {
mount(
<MenuRoot>
<MenuTrigger testID="pill" accessibilityLabel="Open menu">
<Text>Open menu</Text>
</MenuTrigger>
<MenuSurface pages={SUB_PAGES} testID="panel">
<MenuSubTrigger id="show" testID="show-row">
Show
</MenuSubTrigger>
</MenuSurface>
</MenuRoot>,
);
act(() => element("pill").click());
tap(element("show-row"));
await wait(HOVER_CLOSE_SETTLED_MS);
tap(element("show-option"));
await wait(HOVER_CLOSE_SETTLED_MS);
expect(document.querySelector('[data-testid="show-option"]')).not.toBeNull();
});
});
@@ -18,6 +18,7 @@ import {
Platform,
Pressable,
View,
type PointerEvent,
type StyleProp,
type ViewStyle,
} from "react-native";
@@ -217,8 +218,8 @@ export interface AnchoredSurfaceProps {
* Hover tracking for the surface box, per docs/hover.md: plain View, pointer events, never
* a Pressable. Web-only by nature, which is all a flyout needs.
*/
onPointerEnter?: () => void;
onPointerLeave?: () => void;
onPointerEnter?: (event: PointerEvent) => void;
onPointerLeave?: (event: PointerEvent) => void;
testID?: string;
keyboardFocusScope?: KeyboardFocusScope;
children: ReactElement;
@@ -0,0 +1,13 @@
import type { PointerEvent } from "react-native";
/**
* Whether a pointer event came from something that can rest on a row without pressing it.
*
* A finger cannot, but the browser still reports one entering a row as the tap lands and leaving
* as it lifts. Read as hover, that leave queues a close just as the tap's own press opens the
* submenu, and the flyout vanishes a moment after it appears. A touch screen can have a mouse
* attached too, so this has to be asked of each event, not of the device.
*/
export function isHoverPointer(event: PointerEvent): boolean {
return event.nativeEvent.pointerType !== "touch";
}
@@ -6,7 +6,7 @@ import {
type PropsWithChildren,
type ReactElement,
} from "react";
import { Text, View } from "react-native";
import { Text, View, type PointerEvent } from "react-native";
import { StyleSheet, withUnistyles } from "react-native-unistyles";
import { ChevronRight } from "lucide-react-native";
import type { Theme } from "@/styles/theme";
@@ -14,6 +14,7 @@ import { useMenuContext, useMenuDepth } from "./menu-context";
import { isSubPageOpen } from "./menu-navigation";
import { useMenuSurface } from "./menu-surface";
import { MenuItem } from "./menu-item";
import { isHoverPointer } from "./menu-pointer";
const ThemedChevronRight = withUnistyles(ChevronRight);
const mutedIconMapping = (theme: Theme) => ({ color: theme.colors.foregroundMuted });
@@ -25,7 +26,8 @@ const mutedIconMapping = (theme: Theme) => ({ color: theme.colors.foregroundMute
*
* Hover lives on a plain `View` wrapping the item's `Pressable`, per docs/hover.md — a
* `Pressable` tracking its own hover would fight the one inside it. Hover only fires on web,
* which is exactly where flyouts exist; every other platform opens the page by pressing.
* which is exactly where flyouts exist; every other platform opens the page by pressing. So does
* a finger on the web, whose taps arrive as pointer events but are not hover.
*/
export function MenuSubTrigger({
id,
@@ -68,14 +70,22 @@ export function MenuSubTrigger({
menu.openSub({ id, depth });
}, [menu, id, depth]);
const handlePointerEnter = useCallback(() => {
cancelHoverClose();
hoverOpen({ id, depth });
}, [cancelHoverClose, hoverOpen, id, depth]);
const handlePointerEnter = useCallback(
(event: PointerEvent) => {
if (!isHoverPointer(event)) return;
cancelHoverClose();
hoverOpen({ id, depth });
},
[cancelHoverClose, hoverOpen, id, depth],
);
const handlePointerLeave = useCallback(() => {
hoverClose(depth);
}, [hoverClose, depth]);
const handlePointerLeave = useCallback(
(event: PointerEvent) => {
if (!isHoverPointer(event)) return;
hoverClose(depth);
},
[hoverClose, depth],
);
const trailing = useMemo(
() => <MenuSubTrailing value={value} indicator={indicator} />,
@@ -9,7 +9,7 @@ import {
type ReactNode,
} from "react";
import { useTranslation } from "react-i18next";
import { Pressable, Text, View, useWindowDimensions } from "react-native";
import { Pressable, Text, View, useWindowDimensions, type PointerEvent } from "react-native";
import { StyleSheet, withUnistyles } from "react-native-unistyles";
import { BottomSheetScrollView } from "@gorhom/bottom-sheet";
import { ChevronLeft } from "lucide-react-native";
@@ -30,6 +30,7 @@ import { useMenuContext, MenuContextProvider } from "./menu-context";
import { MenuPage } from "./menu-item";
import { currentPageId, isSubPageOpen } from "./menu-navigation";
import { AnchoredSurface, MenuOverlay } from "./menu-overlay";
import { isHoverPointer } from "./menu-pointer";
import { getMenuSheetBottomPadding } from "./menu-sheet-layout";
import type { Alignment, Placement } from "../anchor";
import type { KeyboardFocusScope } from "@/keyboard/actions";
@@ -327,7 +328,19 @@ function MenuFlyout({
const menu = useMenuContext("MenuFlyout");
const { hoverClose, cancelHoverClose } = useMenuSurface("MenuFlyout");
const handleClose = useCallback(() => menu.setOpen(false), [menu]);
const handleHoverOut = useCallback(() => hoverClose(depth), [hoverClose, depth]);
const handleHoverIn = useCallback(
(event: PointerEvent) => {
if (isHoverPointer(event)) cancelHoverClose();
},
[cancelHoverClose],
);
// A tap inside the flyout ends with the finger leaving it, which is not the pointer walking off.
const handleHoverOut = useCallback(
(event: PointerEvent) => {
if (isHoverPointer(event)) hoverClose(depth);
},
[hoverClose, depth],
);
if (!anchorRef) return null;
@@ -344,7 +357,7 @@ function MenuFlyout({
maxHeight={maxHeight}
scrollable={scrollable}
backdrop={false}
onPointerEnter={cancelHoverClose}
onPointerEnter={handleHoverIn}
onPointerLeave={handleHoverOut}
testID={testID}
>
+76 -47
View File
@@ -1624,6 +1624,8 @@ export const MessageInput = forwardRef<MessageInputRef, MessageInputProps>(
volume: dictationVolume,
duration: dictationDuration,
error: dictationError,
startError: dictationStartError,
clearStartError: clearDictationStartError,
status: dictationStatus,
startDictation,
cancelDictation,
@@ -1660,17 +1662,29 @@ export const MessageInput = forwardRef<MessageInputRef, MessageInputProps>(
sendAfterTranscriptRef.current = false;
}, [dictationStatus, isDictating, isDictationProcessing]);
const startDictationIfAvailable = useCallback(
() =>
startDictationIfAvailableImpl({
dictationUnavailableMessage,
canStartDictation,
toast,
startDictation,
exitFullscreen,
}),
[canStartDictation, dictationUnavailableMessage, startDictation, toast, exitFullscreen],
);
useEffect(() => {
if (showRealtimeOverlay) {
clearDictationStartError();
}
}, [showRealtimeOverlay, clearDictationStartError]);
const startDictationIfAvailable = useCallback(() => {
clearDictationStartError();
return startDictationIfAvailableImpl({
dictationUnavailableMessage,
canStartDictation,
toast,
startDictation,
exitFullscreen,
});
}, [
canStartDictation,
clearDictationStartError,
dictationUnavailableMessage,
startDictation,
toast,
exitFullscreen,
]);
const handleVoicePress = useCallback(
() =>
@@ -2224,43 +2238,50 @@ export const MessageInput = forwardRef<MessageInputRef, MessageInputProps>(
);
const toolbar = useMemo(
() => (
<View style={isFullscreen ? styles.hiddenToolbar : buttonRowStyle}>
{/* Toolbar left: attachment button + agent controls */}
<View style={styles.leftButtonGroup}>
{!isFullscreen && (
<AttachmentDropdown
visible={mode.showAttachments}
isConnected={isConnected}
disabled={disabled}
attachButtonStyle={attachButtonStyle}
renderAttachButtonIcon={renderAttachButtonIcon}
attachmentMenuItems={attachmentMenuItems}
addAttachmentLabel={t("composer.input.addAttachment")}
/>
)}
{leftContent}
</View>
{/* Right: dictation, then the trailing action (voice mode, send or stop) */}
{!isFullscreen && (
<View style={styles.rightButtonGroup}>
{beforeVoiceContent}
<VoiceButtonTooltip
visible={mode.showVoice}
onVoicePress={handleVoicePress}
isDictationStartEnabled={isDictationStartEnabled}
voiceButtonAccessibilityLabel={voiceButtonAccessibilityLabel}
voiceButtonStyle={voiceButtonStyle}
renderVoiceButtonIcon={renderVoiceButtonIcon}
voiceTooltipText={voiceTooltipText}
isRealtimeVoiceForCurrentAgent={isRealtimeVoiceForCurrentAgent}
voiceMuteToggleKeys={voiceMuteToggleKeys}
dictationToggleKeys={dictationToggleKeys}
/>
<View style={styles.trailingDisc}>{sendButton}</View>
<>
{!isFullscreen && !showOverlay && dictationStartError ? (
<Text accessibilityRole="alert" style={styles.dictationStartError}>
{t(`message.dictation.${dictationStartError}`)}
</Text>
) : null}
<View style={isFullscreen ? styles.hiddenToolbar : buttonRowStyle}>
{/* Toolbar left: attachment button + agent controls */}
<View style={styles.leftButtonGroup}>
{!isFullscreen && (
<AttachmentDropdown
visible={mode.showAttachments}
isConnected={isConnected}
disabled={disabled}
attachButtonStyle={attachButtonStyle}
renderAttachButtonIcon={renderAttachButtonIcon}
attachmentMenuItems={attachmentMenuItems}
addAttachmentLabel={t("composer.input.addAttachment")}
/>
)}
{leftContent}
</View>
)}
</View>
{/* Right: dictation, then the trailing action (voice mode, send or stop) */}
{!isFullscreen && (
<View style={styles.rightButtonGroup}>
{beforeVoiceContent}
<VoiceButtonTooltip
visible={mode.showVoice}
onVoicePress={handleVoicePress}
isDictationStartEnabled={isDictationStartEnabled}
voiceButtonAccessibilityLabel={voiceButtonAccessibilityLabel}
voiceButtonStyle={voiceButtonStyle}
renderVoiceButtonIcon={renderVoiceButtonIcon}
voiceTooltipText={voiceTooltipText}
isRealtimeVoiceForCurrentAgent={isRealtimeVoiceForCurrentAgent}
voiceMuteToggleKeys={voiceMuteToggleKeys}
dictationToggleKeys={dictationToggleKeys}
/>
<View style={styles.trailingDisc}>{sendButton}</View>
</View>
)}
</View>
</>
),
[
buttonRowStyle,
@@ -2275,6 +2296,8 @@ export const MessageInput = forwardRef<MessageInputRef, MessageInputProps>(
leftContent,
beforeVoiceContent,
mode.showVoice,
showOverlay,
dictationStartError,
handleVoicePress,
isDictationStartEnabled,
voiceButtonAccessibilityLabel,
@@ -2477,6 +2500,12 @@ const styles = StyleSheet.create((theme: Theme) => ({
alignItems: "center",
justifyContent: "space-between",
},
dictationStartError: {
alignSelf: "flex-end",
textAlign: "right",
color: theme.colors.destructive,
fontSize: theme.fontSize.sm,
},
leftButtonGroup: {
minWidth: 0,
flexShrink: 1,
@@ -130,9 +130,6 @@ export function VoiceProvider({ children }: VoiceProviderProps) {
console.error("[VoiceEngine] Failed to stop after audio interruption:", error);
});
},
onError: (error) => {
console.error("[VoiceEngine] Capture error:", error);
},
});
runtime = createVoiceRuntime({
@@ -1,4 +1,5 @@
import { BROWSER_AUTOMATION_COMMAND_NAMES } from "@getpaseo/protocol/browser-automation/rpc-schemas";
import type { DaemonClient } from "@getpaseo/client/internal/daemon-client";
import type { SessionInboundMessage, SessionOutboundMessage } from "@getpaseo/protocol/messages";
import { getDesktopHost, type DesktopHostBridge } from "@/desktop/host";
import {
@@ -26,6 +27,7 @@ type BrowserAutomationExecuteResponse = Extract<
type BrowserAutomationResponsePayload = BrowserAutomationExecuteResponse["payload"];
type BrowserAutomationFailurePayload = Extract<BrowserAutomationResponsePayload, { ok: false }>;
type BrowserAutomationErrorCode = BrowserAutomationFailurePayload["error"]["code"];
type BrowserHostRegistration = Parameters<DaemonClient["registerBrowserHost"]>[0];
interface BrowserAutomationClient {
on(
@@ -69,14 +71,24 @@ export function mountBrowserAutomationHandler(
};
}
export function getDesktopBrowserHostRegistration(): BrowserHostRegistration | null {
const desktop = getDesktopHost();
if (typeof desktop?.browser?.executeAutomationCommand !== "function") return null;
return {
hostKind: "desktop app",
label: desktop.browserHostLabel,
instanceId: desktop.browserHostInstanceId,
supportedCommands: [...BROWSER_AUTOMATION_COMMAND_NAMES],
};
}
export function mountBrowserAutomationDaemonClientHandler(
client: import("@getpaseo/client/internal/daemon-client").DaemonClient,
client: DaemonClient,
registration: BrowserHostRegistration,
options?: { serverId?: string },
): () => void {
const observation = client.registerBrowserHost({
hostKind: "desktop app",
supportedCommands: [...BROWSER_AUTOMATION_COMMAND_NAMES],
});
const observation = client.registerBrowserHost(registration);
const unmount = mountBrowserAutomationHandler({
client: {
on: (_type, handler) =>
+2
View File
@@ -175,6 +175,8 @@ export interface DesktopInvokeBridge {
export interface DesktopHostBridge {
platform?: string;
windowChromeMode?: string;
browserHostLabel?: string;
browserHostInstanceId?: string;
invoke?: DesktopInvokeBridge["invoke"];
getPendingOpenProject?: () => Promise<string | null>;
agentNavigation?: DesktopAgentNavigationBridge;
@@ -11,7 +11,6 @@ import type {
export function useDictationAudioSource(config: DictationAudioSourceConfig): DictationAudioSource {
const onPcmSegmentRef = useRef(config.onPcmSegment);
const onErrorRef = useRef(config.onError);
const onInterruptionRef = useRef(config.onInterruption);
const [volume, setVolume] = useState(0);
const [isPaused, setIsPaused] = useState(false);
@@ -30,9 +29,6 @@ export function useDictationAudioSource(config: DictationAudioSourceConfig): Dic
onVolumeLevel: (level) => {
setVolume(level);
},
onError: (error) => {
onErrorRef.current?.(error);
},
onInterruption: () => {
onInterruptionRef.current?.();
},
@@ -45,14 +41,11 @@ export function useDictationAudioSource(config: DictationAudioSourceConfig): Dic
useEffect(() => {
onPcmSegmentRef.current = config.onPcmSegment;
onErrorRef.current = config.onError;
onInterruptionRef.current = config.onInterruption;
}, [config.onPcmSegment, config.onError, config.onInterruption]);
}, [config.onPcmSegment, config.onInterruption]);
const start = useCallback(async () => {
const engine = getOrCreateEngine();
await engine.initialize();
await engine.startCapture();
await getOrCreateEngine().startCapture();
}, [getOrCreateEngine]);
const stop = useCallback(async () => {
@@ -1,3 +1,4 @@
import { AudioCaptureStartError } from "@/audio";
import { resampleToPcm16 } from "@/audio/pcm";
import { useCallback, useEffect, useMemo, useRef, useState } from "react";
import { parsePcm16Wav } from "@/utils/pcm16-wav";
@@ -133,6 +134,32 @@ async function finalizeRecorderStoppedPromise(input: {
}
}
async function requestMicrophone(): Promise<MediaStream> {
try {
return await navigator.mediaDevices.getUserMedia({
audio: {
channelCount: 1,
noiseSuppression: true,
echoCancellation: true,
autoGainControl: true,
},
});
} catch (error) {
if (error instanceof DOMException) {
if (error.name === "NotAllowedError") {
throw new AudioCaptureStartError("permissionDenied", error.message, { cause: error });
}
if (error.name === "NotFoundError") {
throw new AudioCaptureStartError("noMicrophone", error.message, { cause: error });
}
if (error.name === "NotSupportedError") {
throw new AudioCaptureStartError("unsupported", error.message, { cause: error });
}
}
throw error;
}
}
export function useDictationAudioSource(config: DictationAudioSourceConfig): DictationAudioSource {
const [volume, setVolume] = useState(0);
@@ -200,50 +227,19 @@ export function useDictationAudioSource(config: DictationAudioSourceConfig): Dic
}, []);
const start = useCallback(async () => {
const missingNavigator =
typeof navigator === "undefined" ||
!navigator.mediaDevices ||
typeof navigator.mediaDevices.getUserMedia !== "function";
const secureContext =
typeof window !== "undefined" && typeof window.isSecureContext === "boolean"
? window.isSecureContext
: true;
const currentOrigin =
typeof window !== "undefined" && window.location ? window.location.origin : "unknown";
const isDesktopApp = isElectronRuntime();
if (missingNavigator) {
throw new Error("Microphone capture is not supported in this environment");
}
if (!secureContext && !isDesktopApp) {
throw new Error(
`Microphone access requires HTTPS or localhost. Current origin: ${currentOrigin}`,
);
}
if (!secureContext && isDesktopApp) {
console.warn(
"[DictationAudio][Web] Insecure context reported under Desktop; attempting getUserMedia anyway",
{ currentOrigin },
if (!window.isSecureContext && !isElectronRuntime()) {
throw new AudioCaptureStartError(
"insecureContext",
`Microphone access requires HTTPS or localhost. Current origin: ${window.location.origin}`,
);
}
const AudioContextCtor = getAudioContextCtor();
if (!AudioContextCtor) {
throw new Error("AudioContext unavailable");
if (!navigator.mediaDevices?.getUserMedia || !AudioContextCtor) {
throw new AudioCaptureStartError("unsupported", "Browser audio capture is unavailable.");
}
const rawStream = await navigator.mediaDevices.getUserMedia({
audio: {
channelCount: 1,
noiseSuppression: true,
echoCancellation: true,
autoGainControl: true,
},
});
const stream = rawStream;
const stream = await requestMicrophone();
const context = new AudioContextCtor();
try {
@@ -431,15 +427,7 @@ export function useDictationAudioSource(config: DictationAudioSourceConfig): Dic
return useMemo(
() => ({
start: async () => {
try {
await start();
} catch (err) {
const normalized = err instanceof Error ? err : new Error(String(err));
onErrorRef.current?.(normalized);
throw normalized;
}
},
start,
stop,
volume,
isPaused: false,
@@ -1,3 +1,4 @@
import type { AudioCaptureStartFailure } from "@/audio";
import { i18n } from "@/i18n/i18next";
export type DictationStatus = "idle" | "recording" | "paused" | "uploading" | "failed";
@@ -21,6 +22,8 @@ export interface UseDictationResult {
volume: number;
duration: number;
error: string | null;
startError: AudioCaptureStartFailure | null;
clearStartError: () => void;
status: DictationStatus;
startDictation: () => Promise<void>;
cancelDictation: () => Promise<void>;
+15 -1
View File
@@ -1,6 +1,7 @@
import { useCallback, useEffect, useRef, useState } from "react";
import { useTranslation } from "react-i18next";
import { AudioCaptureStartError, type AudioCaptureStartFailure } from "@/audio";
import { DictationStreamSender } from "@/dictation/dictation-stream-sender";
import { useDictationAudioSource } from "@/hooks/use-dictation-audio-source";
import { generateMessageId } from "@/types/stream";
@@ -32,6 +33,8 @@ export function useDictation(options: UseDictationOptions): UseDictationResult {
const [partialTranscript, setPartialTranscript] = useState("");
const [duration, setDuration] = useState(0);
const [error, setError] = useState<string | null>(null);
const [startError, setStartError] = useState<AudioCaptureStartFailure | null>(null);
const clearStartError = useCallback(() => setStartError(null), []);
const [status, setStatus] = useState<DictationStatus>("idle");
const latestPartialTranscriptRef = useRef("");
@@ -265,14 +268,17 @@ export function useDictation(options: UseDictationOptions): UseDictationResult {
actionGateRef.current.starting = true;
setError(null);
setStartError(null);
setPartialTranscript("");
setDuration(0);
setIsProcessing(false);
setStatus("recording");
clearStreamingState();
let captureStarted = false;
try {
await audio.start();
captureStarted = true;
isRecordingRef.current = true;
setIsRecording(true);
if (enableDuration) {
@@ -287,7 +293,12 @@ export function useDictation(options: UseDictationOptions): UseDictationResult {
isRecordingRef.current = false;
setIsRecording(false);
setStatus("idle");
reportError(err, "Failed to start dictation");
if (captureStarted) {
reportError(err, "Failed to start dictation stream");
} else {
console.error("[useDictation] Failed to start dictation capture", err);
setStartError(err instanceof AudioCaptureStartError ? err.reason : "startFailed");
}
} finally {
actionGateRef.current.starting = false;
}
@@ -438,6 +449,7 @@ export function useDictation(options: UseDictationOptions): UseDictationResult {
stopDurationTracking();
setDuration(0);
setError(null);
setStartError(null);
setStatus("idle");
clearStreamingState();
}, [clearStreamingState, stopDurationTracking]);
@@ -461,6 +473,8 @@ export function useDictation(options: UseDictationOptions): UseDictationResult {
volume: audio.volume,
duration,
error,
startError,
clearStartError,
status: status === "recording" && audio.isPaused ? "paused" : status,
startDictation,
cancelDictation,
+6
View File
@@ -115,6 +115,7 @@ export const ar: TranslationResources = {
home: "بيت",
groupByProject: "تجميع حسب المشروع",
groupByStatus: "تجميع حسب الحالة",
copyBrowserHostName: "نسخ اسم مضيف المتصفح",
modelGroupLabel: "النموذج",
modelSearchKeywords: "تبديل النموذج تغيير النموذج تعيين النموذج اختيار النموذج",
thinkingGroupLabel: "التفكير",
@@ -377,6 +378,11 @@ export const ar: TranslationResources = {
details: "تفاصيل",
},
dictation: {
insecureContext: "افتح Paseo عبر HTTPS أو localhost لاستخدام الميكروفون.",
unsupported: "الإملاء غير متاح في هذا المتصفح.",
noMicrophone: "صِل ميكروفونًا وحاول مرة أخرى.",
permissionDenied: "اسمح بالوصول إلى الميكروفون وحاول مرة أخرى.",
startFailed: "تعذر بدء الإملاء. حاول مرة أخرى.",
paused: "تمت مقاطعة الميكروفون. في انتظار الاستئناف…",
start: "بدء الإملاء الصوتي",
cancel: "إلغاء الإملاء",
+6
View File
@@ -113,6 +113,7 @@ export const en = {
home: "Home",
groupByProject: "Group by project",
groupByStatus: "Group by status",
copyBrowserHostName: "Copy browser host name",
modelGroupLabel: "Model",
modelSearchKeywords: "switch model change model set model select model",
thinkingGroupLabel: "Thinking",
@@ -376,6 +377,11 @@ export const en = {
details: "Details",
},
dictation: {
insecureContext: "Open Paseo over HTTPS or localhost to use the microphone.",
unsupported: "Dictation is unavailable in this browser.",
noMicrophone: "Connect a microphone and try again.",
permissionDenied: "Allow microphone access and try again.",
startFailed: "Could not start dictation. Try again.",
paused: "Microphone interrupted. Waiting to resume…",
start: "Start voice dictation",
cancel: "Cancel dictation",
+6
View File
@@ -116,6 +116,7 @@ export const es: TranslationResources = {
home: "Hogar",
groupByProject: "Agrupar por proyecto",
groupByStatus: "Agrupar por estado",
copyBrowserHostName: "Copiar el nombre del host del navegador",
modelGroupLabel: "Modelo",
modelSearchKeywords: "cambiar modelo modificar modelo establecer modelo seleccionar modelo",
thinkingGroupLabel: "Razonamiento",
@@ -382,6 +383,11 @@ export const es: TranslationResources = {
details: "Detalles",
},
dictation: {
insecureContext: "Abre Paseo por HTTPS o en localhost para usar el micrófono.",
unsupported: "El dictado no está disponible en este navegador.",
noMicrophone: "Conecta un micrófono e inténtalo de nuevo.",
permissionDenied: "Permite el acceso al micrófono e inténtalo de nuevo.",
startFailed: "No se pudo iniciar el dictado. Inténtalo de nuevo.",
paused: "Micrófono interrumpido. Esperando para reanudar…",
start: "Iniciar dictado de voz",
cancel: "Cancelar dictado",
+6
View File
@@ -116,6 +116,7 @@ export const fr: TranslationResources = {
home: "Accueil",
groupByProject: "Grouper par projet",
groupByStatus: "Grouper par statut",
copyBrowserHostName: "Copier le nom de l’hôte du navigateur",
modelGroupLabel: "Modèle",
modelSearchKeywords:
"changer de modèle modifier le modèle définir le modèle sélectionner le modèle model",
@@ -382,6 +383,11 @@ export const fr: TranslationResources = {
details: "Détails",
},
dictation: {
insecureContext: "Ouvrez Paseo via HTTPS ou sur localhost pour utiliser le microphone.",
unsupported: "La dictée est indisponible dans ce navigateur.",
noMicrophone: "Connectez un microphone, puis réessayez.",
permissionDenied: "Autorisez l’accès au microphone, puis réessayez.",
startFailed: "Impossible de lancer la dictée. Réessayez.",
paused: "Microphone interrompu. En attente de reprise…",
start: "Démarrer la dictée vocale",
cancel: "Annuler la dictée",
+6
View File
@@ -116,6 +116,7 @@ export const ja: TranslationResources = {
home: "ホーム",
groupByProject: "プロジェクトでグループ化",
groupByStatus: "ステータスでグループ化",
copyBrowserHostName: "ブラウザホスト名をコピー",
modelGroupLabel: "モデル",
modelSearchKeywords: "モデルを切り替え モデルを変更 モデルを設定 モデルを選択",
thinkingGroupLabel: "思考",
@@ -382,6 +383,11 @@ export const ja: TranslationResources = {
details: "詳細",
},
dictation: {
insecureContext: "マイクを使用するには、HTTPS または localhost で Paseo を開いてください。",
unsupported: "このブラウザーでは音声入力を利用できません。",
noMicrophone: "マイクを接続して、もう一度お試しください。",
permissionDenied: "マイクへのアクセスを許可して、もう一度お試しください。",
startFailed: "音声入力を開始できませんでした。もう一度お試しください。",
paused: "マイクが中断されました。再開を待っています…",
start: "音声入力を開始",
cancel: "音声入力をキャンセル",
+6
View File
@@ -115,6 +115,7 @@ export const ko: TranslationResources = {
home: "홈",
groupByProject: "프로젝트별 그룹화",
groupByStatus: "상태별 그룹화",
copyBrowserHostName: "브라우저 호스트 이름 복사",
modelGroupLabel: "모델",
modelSearchKeywords: "모델 전환 모델 변경 모델 설정 모델 선택",
thinkingGroupLabel: "추론",
@@ -378,6 +379,11 @@ export const ko: TranslationResources = {
details: "세부 정보",
},
dictation: {
insecureContext: "마이크를 사용하려면 HTTPS 또는 localhost에서 Paseo를 여세요.",
unsupported: "이 브라우저에서는 받아쓰기를 사용할 수 없습니다.",
noMicrophone: "마이크를 연결한 후 다시 시도하세요.",
permissionDenied: "마이크 접근을 허용한 후 다시 시도하세요.",
startFailed: "받아쓰기를 시작할 수 없습니다. 다시 시도하세요.",
paused: "마이크가 중단되었습니다. 재개를 기다리는 중…",
start: "음성 받아쓰기 시작",
cancel: "받아쓰기 취소",
+6
View File
@@ -116,6 +116,7 @@ export const ptBR: TranslationResources = {
home: "Início",
groupByProject: "Agrupar por projeto",
groupByStatus: "Agrupar por status",
copyBrowserHostName: "Copiar o nome do host do navegador",
modelGroupLabel: "Modelo",
modelSearchKeywords: "trocar modelo mudar modelo definir modelo selecionar modelo",
thinkingGroupLabel: "Raciocínio",
@@ -381,6 +382,11 @@ export const ptBR: TranslationResources = {
details: "Detalhes",
},
dictation: {
insecureContext: "Abra o Paseo por HTTPS ou em localhost para usar o microfone.",
unsupported: "O ditado não está disponível neste navegador.",
noMicrophone: "Conecte um microfone e tente novamente.",
permissionDenied: "Permita o acesso ao microfone e tente novamente.",
startFailed: "Não foi possível iniciar o ditado. Tente novamente.",
paused: "Microfone interrompido. Aguardando para retomar…",
start: "Iniciar ditado por voz",
cancel: "Cancelar ditado",
+6
View File
@@ -116,6 +116,7 @@ export const ru: TranslationResources = {
home: "Главная",
groupByProject: "Группировать по проекту",
groupByStatus: "Группировать по статусу",
copyBrowserHostName: "Скопировать имя хоста браузера",
modelGroupLabel: "Модель",
modelSearchKeywords: "Сменить, изменить, выбрать или установить модель",
thinkingGroupLabel: "Мышление",
@@ -381,6 +382,11 @@ export const ru: TranslationResources = {
details: "Подробности",
},
dictation: {
insecureContext: "Чтобы использовать микрофон, откройте Paseo по HTTPS или на localhost.",
unsupported: "Диктовка недоступна в этом браузере.",
noMicrophone: "Подключите микрофон и попробуйте снова.",
permissionDenied: "Разрешите доступ к микрофону и попробуйте снова.",
startFailed: "Не удалось начать диктовку. Попробуйте снова.",
paused: "Микрофон прерван. Ожидание возобновления…",
start: "Начать диктовку голосом",
cancel: "Отменить диктовку",
+6
View File
@@ -115,6 +115,7 @@ export const zhCN: TranslationResources = {
home: "首页",
groupByProject: "按项目分组",
groupByStatus: "按状态分组",
copyBrowserHostName: "复制浏览器宿主名称",
modelGroupLabel: "模型",
modelSearchKeywords: "切换模型 更改模型 设置模型 选择模型",
thinkingGroupLabel: "思考",
@@ -377,6 +378,11 @@ export const zhCN: TranslationResources = {
details: "详情",
},
dictation: {
insecureContext: "请通过 HTTPS 或 localhost 打开 Paseo,以使用麦克风。",
unsupported: "此浏览器无法使用听写。",
noMicrophone: "请连接麦克风,然后重试。",
permissionDenied: "请允许访问麦克风,然后重试。",
startFailed: "无法开始听写。请重试。",
paused: "麦克风已中断,正在等待恢复…",
start: "开始语音听写",
cancel: "取消听写",
@@ -0,0 +1,60 @@
import React, { act } from "react";
import { createRoot } from "react-dom/client";
import { expect, it } from "vitest";
import { usePanelStore } from "@/stores/panel-store";
import { MobilePanelsProvider, useIsMobilePanelActive } from "./provider";
Object.assign(globalThis, { IS_REACT_ACT_ENVIRONMENT: true });
it("renders a panel subscriber only when that panel settles or stops being settled", async () => {
const initial = usePanelStore.getState();
const host = document.createElement("div");
document.body.appendChild(host);
const root = createRoot(host);
const explorerStates: boolean[] = [];
function ExplorerSubscriber() {
explorerStates.push(useIsMobilePanelActive("file-explorer"));
return null;
}
function AgentSubscriber() {
const active = useIsMobilePanelActive("agent");
return <span>{active ? "Agent settled" : "Agent covered"}</span>;
}
async function showPanel(target: "agent" | "agent-list" | "file-explorer") {
await act(async () => {
usePanelStore.setState((state) => ({
mobilePanel: { target, revision: state.mobilePanel.revision + 1 },
}));
});
}
try {
await showPanel("agent-list");
await act(async () => {
root.render(
<MobilePanelsProvider>
<ExplorerSubscriber />
<AgentSubscriber />
</MobilePanelsProvider>,
);
});
await showPanel("agent");
expect(host.textContent).toBe("Agent covered");
await expect.poll(() => host.textContent).toBe("Agent settled");
expect(explorerStates).toEqual([false]);
await showPanel("file-explorer");
expect(explorerStates).toEqual([false]);
await expect.poll(() => explorerStates).toEqual([false, true]);
await showPanel("agent");
await expect.poll(() => explorerStates).toEqual([false, true, false]);
} finally {
await act(async () => root.unmount());
host.remove();
usePanelStore.setState(initial, true);
}
});
+22 -16
View File
@@ -13,6 +13,7 @@ import {
type ReactNode,
type RefObject,
} from "react";
import { createStore, useStore } from "zustand";
import { Keyboard, useWindowDimensions } from "react-native";
import type { GestureType } from "react-native-gesture-handler";
import {
@@ -73,7 +74,11 @@ interface FinishGestureInput {
}
const MobilePanelsContext = createContext<MobilePanelsRuntime | null>(null);
const MobilePanelActiveContext = createContext<MobilePanelView>("agent");
function createSettledPanelStore(panel: MobilePanelView) {
return createStore(() => panel);
}
const SettledPanelContext = createContext(createSettledPanelStore("agent"));
/**
* The left sidebar's width: the full window for the compact drawer, the saved sidebar width
@@ -113,7 +118,7 @@ export function MobilePanelsProvider({ children }: { children: ReactNode }) {
const leftOpenGestureRef = useRef<GestureType | undefined>(undefined);
const leftCloseGestureRef = useRef<GestureType | undefined>(undefined);
const rightOpenGestureRef = useRef<GestureType | undefined>(undefined);
const [activePanel, setActivePanel] = useState(initialSelection.target);
const [settledPanel] = useState(() => createSettledPanelStore(initialSelection.target));
const setOpenGestureBlocked = useCallback(
(owner: symbol, blocked: boolean) => {
@@ -127,16 +132,19 @@ export function MobilePanelsProvider({ children }: { children: ReactNode }) {
[openGesturesBlocked],
);
const publishActivePanel = useCallback((panel: MobilePanelView, revision: number) => {
const selection = usePanelStore.getState().mobilePanel;
if (selection.revision !== revision || selection.target !== panel) {
return;
}
if (isNative && panel !== "agent") {
Keyboard.dismiss();
}
setActivePanel(panel);
}, []);
const publishActivePanel = useCallback(
(panel: MobilePanelView, revision: number) => {
const selection = usePanelStore.getState().mobilePanel;
if (selection.revision !== revision || selection.target !== panel) {
return;
}
if (isNative && panel !== "agent") {
Keyboard.dismiss();
}
settledPanel.setState(panel);
},
[settledPanel],
);
useAnimatedReaction(
() => ({ motionState: motionState.value, position: position.value }),
@@ -279,9 +287,7 @@ export function MobilePanelsProvider({ children }: { children: ReactNode }) {
return (
<MobilePanelsContext.Provider value={value}>
<MobilePanelActiveContext.Provider value={activePanel}>
{children}
</MobilePanelActiveContext.Provider>
<SettledPanelContext.Provider value={settledPanel}>{children}</SettledPanelContext.Provider>
</MobilePanelsContext.Provider>
);
}
@@ -301,7 +307,7 @@ export function useLeftPanelWidth(): number {
}
export function useIsMobilePanelActive(panel: MobilePanelView): boolean {
return useContext(MobilePanelActiveContext) === panel;
return useStore(useContext(SettledPanelContext), (settledPanel) => settledPanel === panel);
}
export function useBlockMobilePanelOpenGestures(blocked: boolean): void {
+14 -20
View File
@@ -55,10 +55,8 @@ import {
buildDesktopDaemonTransportUrl,
createDesktopDaemonTransportFactory,
} from "@/desktop/daemon/desktop-daemon-transport";
import { getDesktopHost } from "@/desktop/host";
import { readDesktopManagedLocalCredential } from "@/desktop/daemon/local-credential";
import { CLIENT_CAPS } from "@getpaseo/protocol/client-capabilities";
import { BROWSER_AUTOMATION_COMMAND_NAMES } from "@getpaseo/protocol/browser-automation/rpc-schemas";
import {
useSessionStore,
toDaemonServerInfo,
@@ -74,7 +72,10 @@ import {
invalidateServerDataQueriesAfterReconnect,
mountServerDataPushRouter,
} from "@/data/push-router";
import { mountBrowserAutomationDaemonClientHandler } from "@/desktop/browser/automation/handler";
import {
getDesktopBrowserHostRegistration,
mountBrowserAutomationDaemonClientHandler,
} from "@/desktop/browser/automation/handler";
import { schedulesQueryBaseKey } from "@/schedules/aggregated-schedules";
import { dispatchComposerAgentMessage, sendQueuedComposerMessageNow } from "@/composer/actions";
import { createMessageSubmissionWriter } from "@/composer/submission/writer";
@@ -566,19 +567,10 @@ function probeIntervalForConnection(
function createDefaultDeps(
resolveOrigin: (serverId: string) => DaemonClient | null = () => null,
): HostRuntimeControllerDeps {
const browserHostAvailable =
typeof getDesktopHost()?.browser?.executeAutomationCommand === "function";
const browserAutomationCapabilities = browserHostAvailable
? {
[CLIENT_CAPS.browserHost]: {
supportedCommands: [...BROWSER_AUTOMATION_COMMAND_NAMES],
hostKind: "desktop app",
},
}
: undefined;
const appCapabilities = {
...browserAutomationCapabilities,
};
const browserHostRegistration = getDesktopBrowserHostRegistration();
const appCapabilities = browserHostRegistration
? { [CLIENT_CAPS.browserHost]: browserHostRegistration }
: {};
return {
createClient: ({ host, connection, clientId, runtimeGeneration }) => {
@@ -702,12 +694,14 @@ function createDefaultDeps(
queryClient,
serverId: host.serverId,
});
if (!browserAutomationCapabilities) {
if (!browserHostRegistration) {
return unmountServerData;
}
const unmountBrowserAutomation = mountBrowserAutomationDaemonClientHandler(client, {
serverId: host.serverId,
});
const unmountBrowserAutomation = mountBrowserAutomationDaemonClientHandler(
client,
browserHostRegistration,
{ serverId: host.serverId },
);
return () => {
unmountBrowserAutomation();
unmountServerData();
@@ -0,0 +1,94 @@
import React, { act } from "react";
import { createRoot } from "react-dom/client";
import { PortalProvider } from "@gorhom/portal";
import { afterEach, expect, it, vi } from "vitest";
import { page } from "vitest/browser";
import { WindowLayoutContext } from "@/window-layout/context";
import { WindowLayoutActionsHost } from "@/window-layout";
import { usePanelStore } from "@/stores/panel-store";
import { CompactWorkspaceHeader } from ".";
import { i18n } from "@/i18n/i18next";
import type { WindowLayout } from "@/window-layout";
void i18n;
vi.stubGlobal("IS_REACT_ACT_ENVIRONMENT", true);
const container = document.createElement("div");
document.body.appendChild(container);
const root = createRoot(container);
const initialPanel = usePanelStore.getState();
afterEach(async () => {
await act(async () => root.render(null));
usePanelStore.setState(initialPanel, true);
});
const noop = () => {};
const explorerState = { expanded: false };
const geometry: WindowLayout = {
bounds: { x: 0, y: 0, width: window.innerWidth, height: window.innerHeight },
verticalBarEdge: "trailing",
verticalBar: { x: 20, y: 20, width: 44, height: 400 },
divisions: [],
};
it("updates compact header actions on panel changes without rendering the workspace owner", async () => {
let ownerRenders = 0;
function WorkspaceOwner({ focused = true, compact = true }) {
ownerRenders++;
return (
<CompactWorkspaceHeader
isRouteFocused={focused}
isCompact={compact}
serverId="test"
isLoading={false}
title="Workspace"
subtitle="Project"
isSubtitleDistinct
onToggleExplorer={noop}
explorerToggleLabel="Explorer"
explorerToggleAccessibilityState={explorerState}
tabSwitcher={null}
headerVariant="full"
>
<button type="button">Workspace menu</button>
</CompactWorkspaceHeader>
);
}
async function renderWorkspace(focused = true, compact = true) {
await act(async () =>
root.render(
<PortalProvider>
<WindowLayoutContext.Provider value={geometry}>
<WindowLayoutActionsHost />
<WorkspaceOwner focused={focused} compact={compact} />
</WindowLayoutContext.Provider>
</PortalProvider>,
),
);
}
async function showPanel(target: "agent" | "agent-list" | "file-explorer") {
await act(async () =>
usePanelStore.setState((state) => ({
mobilePanel: { ...state.mobilePanel, target, revision: state.mobilePanel.revision + 1 },
})),
);
}
const menu = page.getByRole("button", {
name: "Workspace menu",
exact: true,
includeHidden: true,
});
await showPanel("agent");
await renderWorkspace();
await expect.element(menu).toBeVisible();
const rendersBeforePanelChanges = ownerRenders;
await showPanel("agent-list");
await expect.element(menu).not.toBeVisible();
await showPanel("agent");
await expect.element(menu).toBeVisible();
await showPanel("file-explorer");
await expect.element(menu).not.toBeVisible();
expect(ownerRenders).toBe(rendersBeforePanelChanges);
await renderWorkspace(true, false);
await expect.element(menu).toBeVisible();
await renderWorkspace(false, false);
await expect.element(menu).not.toBeVisible();
});
@@ -1,5 +1,6 @@
import { Fragment, useMemo, type ReactNode } from "react";
import { useTranslation } from "react-i18next";
import { usePanelStore } from "@/stores/panel-store";
import { CompactHeader } from "@/components/headers/compact-header";
import {
EXPLORER_TOGGLE_KEYS,
@@ -29,7 +30,8 @@ interface CompactWorkspaceHeaderProps {
/** Workspace overflow menu. */
children: ReactNode;
headerVariant: "compact" | "full";
actionsActive: boolean;
isRouteFocused: boolean;
isCompact: boolean;
}
/** The compact header with the workspace's project row and controls. */
@@ -45,7 +47,8 @@ export function CompactWorkspaceHeader({
tabSwitcher,
serverId,
headerVariant,
actionsActive,
isRouteFocused,
isCompact,
}: CompactWorkspaceHeaderProps) {
const { t } = useTranslation();
const actionsInBar = resolveWorkspaceActionPlacement(useWindowLayout()) === "vertical-bar";
@@ -89,7 +92,26 @@ export function CompactWorkspaceHeader({
actions={actionsInBar ? undefined : actions}
/>
) : null}
<WindowLayoutActions active={actionsActive}>{actions}</WindowLayoutActions>
<WorkspaceWindowActions isRouteFocused={isRouteFocused} isCompact={isCompact}>
{actions}
</WorkspaceWindowActions>
</IconButtonChromeProvider>
);
}
// Panel changes only affect the portaled controls. Keep their subscription below the
// workspace and header so opening a sidebar does not render the chat or rebuild its actions.
function WorkspaceWindowActions({
isRouteFocused,
isCompact,
children,
}: {
isRouteFocused: boolean;
isCompact: boolean;
children: ReactNode;
}) {
const active = usePanelStore(
(state) => isRouteFocused && (!isCompact || state.mobilePanel.target === "agent"),
);
return <WindowLayoutActions active={active}>{children}</WindowLayoutActions>;
}
@@ -3,7 +3,6 @@ export interface ExplorerFrameProps {
children: ReactNode;
presentation: "overlay" | "dock";
persistenceKey: string;
containerWidth: number;
isOpen: boolean;
isWorkspaceActive: boolean;
surfaceColor?: string;
@@ -0,0 +1,55 @@
import React, { act } from "react";
import { createRoot } from "react-dom/client";
import { View } from "react-native";
import { expect, it } from "vitest";
import { MobilePanelsProvider } from "@/mobile-panels/provider";
import { ExplorerFrame } from "./frame";
Object.assign(globalThis, { IS_REACT_ACT_ENVIRONMENT: true });
const WORKSPACE_STYLE = { flexDirection: "row", height: 200 } as const;
const FILL = { flex: 1 };
it("measures dock allocation independently of the workspace owner", async () => {
const host = document.createElement("div");
host.style.width = "900px";
document.body.appendChild(host);
const root = createRoot(host);
let workspaceRenders = 0;
function Workspace() {
workspaceRenders++;
return (
<View style={WORKSPACE_STYLE}>
<View role="main" style={FILL} />
<ExplorerFrame presentation="dock" isWorkspaceActive isOpen persistenceKey="workspace">
<View role="complementary" style={FILL} />
</ExplorerFrame>
</View>
);
}
function dockWidth() {
return host.querySelector('[role="complementary"]')?.getBoundingClientRect().width;
}
try {
await act(async () => {
root.render(
<MobilePanelsProvider>
<Workspace />
</MobilePanelsProvider>,
);
});
await expect.poll(dockWidth).toBe(320);
const initialRenderCount = workspaceRenders;
const dock = host.querySelector('[role="complementary"]');
host.style.width = "620px";
await expect.poll(dockWidth).toBe(260);
expect(workspaceRenders).toBe(initialRenderCount);
expect(host.querySelector('[role="complementary"]')).toBe(dock);
} finally {
await act(async () => root.unmount());
host.remove();
}
});
@@ -1,5 +1,5 @@
import { useCallback } from "react";
import { View } from "react-native";
import { useCallback, useState } from "react";
import { View, type LayoutChangeEvent } from "react-native";
import { StyleSheet, withUnistyles } from "react-native-unistyles";
import {
FileExplorerCloseGestureContext,
@@ -20,8 +20,12 @@ function ExplorerFrameBase({
isOpen,
isWorkspaceActive,
persistenceKey,
containerWidth,
}: ExplorerFrameProps) {
const [containerWidth, setContainerWidth] = useState(0);
const measureContainer = useCallback(({ nativeEvent }: LayoutChangeEvent) => {
const { width, height } = nativeEvent.layout;
if (width > 0 && height > 0) setContainerWidth(width);
}, []);
const overlay = presentation === "overlay";
const settledOpen = useIsMobilePanelActive("file-explorer");
const active = isWorkspaceActive && (overlay ? settledOpen : isOpen);
@@ -59,10 +63,15 @@ function ExplorerFrameBase({
</View>
</MobilePanelOverlay>
</FileExplorerCloseGestureContext>
<View pointerEvents="none" style={styles.measurement} onLayout={measureContainer} />
</RetainedPanelActivity>
);
}
export const ExplorerFrame = withUnistyles(ExplorerFrameBase, (theme) => ({
surfaceColor: theme.colors.surfaceSidebar,
}));
const styles = StyleSheet.create({ content: { flex: 1, minHeight: 0 } });
const styles = StyleSheet.create({
content: { flex: 1, minHeight: 0 },
// Measure the workspace row even when the dock is closed. The gesture host stays in its slot.
measurement: { position: "absolute", top: 0, right: 0, bottom: 0, left: 0 },
});
@@ -13,7 +13,6 @@ import {
import { useFileExplorerActions } from "@/hooks/use-file-explorer-actions";
import { useWindowLayout, resolveWorkspaceActionPlacement } from "@/window-layout";
import { revealWorkspaceTab } from "@/workspace-tabs/navigation";
import type { LayoutChangeEvent } from "react-native";
import { ExplorerSidebarDock } from "@/screens/workspace/explorer-sidebar";
import { ForgeShorthandProvider } from "@/git/forge-shorthand";
import { ExplorerFrame } from "./explorer-sidebar/frame";
@@ -805,9 +804,7 @@ function WorkspacePanelSurface({
children,
tabIds,
workspaceKey,
onLayout,
}: {
onLayout: (event: LayoutChangeEvent) => void;
workspaceKey: string;
launcher: NewTabLauncher;
children: ReactNode;
@@ -818,9 +815,7 @@ function WorkspacePanelSurface({
<WorkspaceComposerEditingSessionsProvider workspaceKey={workspaceKey} tabIds={tabIds}>
<NewTabLauncherProvider value={launcher}>
<SheetHostScope enabled={!isCompact}>
<View style={styles.threePaneRow} onLayout={onLayout}>
{children}
</View>
<View style={styles.threePaneRow}>{children}</View>
</SheetHostScope>
</NewTabLauncherProvider>
</WorkspaceComposerEditingSessionsProvider>
@@ -1061,7 +1056,6 @@ function WorkspaceScreenContent({
resolveWorkspaceActionPlacement(useWindowLayout()) === "vertical-bar";
const toast = useToast();
const isMobile = useIsCompactFormFactor();
const workspacePanelActive = usePanelStore((state) => state.mobilePanel.target === "agent");
const canRenderDesktopPaneSplits = supportsDesktopPaneSplits();
const layoutMode = useWorkspaceLayoutStore((state) => state.layoutMode);
const hasMacTrafficLights = useHasWindowChromeObstruction("top-left");
@@ -3463,7 +3457,8 @@ function WorkspaceScreenContent({
return (
<CompactWorkspaceHeader
headerVariant={actionsOnly ? "full" : "compact"}
actionsActive={isRouteFocused && (!isMobile || workspacePanelActive)}
isRouteFocused={isRouteFocused}
isCompact={isMobile}
tabSwitcher={tabSwitcher}
isLoading={isWorkspaceHeaderLoading}
title={workspaceHeaderTitle}
@@ -3524,7 +3519,6 @@ function WorkspaceScreenContent({
workspaceDirectory,
workspaceHeaderSubtitle,
workspaceHeaderTitle,
workspacePanelActive,
],
);
const workspaceCompactHeaderTabs = {
@@ -3710,11 +3704,6 @@ function WorkspaceScreenContent({
</DesktopCenterColumnChrome>
);
const [workspaceViewportWidth, setWorkspaceViewportWidth] = useState(0);
const measureWorkspaceWidth = useCallback(
(event: LayoutChangeEvent) => setWorkspaceViewportWidth(event.nativeEvent.layout.width),
[],
);
const sharedExplorer = useMemo(() => {
if (!workspaceLayout || !explorerSidebarPaneId || !persistenceKey) return null;
const explorerPane = findPaneById(workspaceLayout.root, explorerSidebarPaneId);
@@ -3753,7 +3742,6 @@ function WorkspaceScreenContent({
presentation={isMobile ? "overlay" : "dock"}
isWorkspaceActive={isRouteFocused}
persistenceKey={persistenceKey}
containerWidth={workspaceViewportWidth}
isOpen={isExplorerSidebarShowing && !desktopFocusModeEnabled}
>
{dock}
@@ -3788,7 +3776,6 @@ function WorkspaceScreenContent({
handleReorderTabsInPane,
buildDesktopPaneContentModel,
renderExplorerSidebarHeaderAction,
workspaceViewportWidth,
desktopFocusModeEnabled,
]);
@@ -3846,7 +3833,6 @@ function WorkspaceScreenContent({
isRouteFocused={isRouteFocused}
/>
<WorkspacePanelSurface
onLayout={measureWorkspaceWidth}
launcher={newTabLauncher}
workspaceKey={workspacePanelKey}
tabIds={workspacePanelTabIds}
@@ -74,6 +74,7 @@ export const MoreVertical = StubIcon;
export const MousePointer2 = StubIcon;
export const PackagePlus = StubIcon;
export const PanelLeft = StubIcon;
export const PanelRight = StubIcon;
export const PanelsTopLeft = StubIcon;
export const Paperclip = StubIcon;
export const Pencil = StubIcon;
@@ -132,3 +133,10 @@ export const MicVocal = StubIcon;
export const Scissors = StubIcon;
export const Sparkles = StubIcon;
export const TriangleAlertIcon = StubIcon;
export const ArrowRightToLine = StubIcon;
export const Columns2 = StubIcon;
export const CopyX = StubIcon;
export const Ellipsis = StubIcon;
export const Maximize = StubIcon;
export const Minimize = StubIcon;
export const Rows2 = StubIcon;
+93 -6
View File
@@ -338,6 +338,86 @@ async function callBrowserTool(client, name, args = {}) {
return mcpPayload(await client.callTool({ name, args }), name);
}
async function connectBrowserToolClient(daemonPort, callerAgentId) {
const transport = new StreamableHTTPClientTransport(
new URL(
`http://127.0.0.1:${daemonPort}/mcp/agents?callerAgentId=${encodeURIComponent(callerAgentId)}`,
),
);
return experimental_createMCPClient({ transport });
}
async function openBrowserWorkspace(page, workspaceKey) {
await page.getByTestId(`sidebar-workspace-row-${workspaceKey}`).click({ timeout: timeoutMs });
await expect(page.getByTestId(`workspace-deck-entry-${workspaceKey}`)).toBeVisible();
}
async function openSecondDesktopWindow(page) {
const nextWindow = page.context().waitForEvent("page", { timeout: timeoutMs });
await page.evaluate(() => window.paseoDesktop.window.openNew());
return nextWindow;
}
async function listBrowserHosts(client) {
const reply = await client.callTool({ name: "browser_list_hosts", args: {} });
assert(Array.isArray(reply.structuredContent?.browserHosts), JSON.stringify(reply));
return reply.structuredContent.browserHosts;
}
async function expectBrowserHostNameInCommandCenter(page, name) {
await page.getByTestId("sidebar-search").click();
const search = page.getByTestId("command-center-input");
await search.fill("Copy browser host name");
const action = page.getByRole("button", { name: /Copy browser host name/ });
await expect(action).toContainText(name);
await action.click();
await expect(search).toBeHidden();
await expect(page.getByText("Copied", { exact: true })).toBeVisible();
}
async function verifyBrowserHostRouting({ page, client, targetUrl, workspaceKey }) {
await openBrowserWorkspace(page, workspaceKey);
await expect.poll(async () => (await listBrowserHosts(client)).length).toBe(1);
const [firstHost] = await listBrowserHosts(client);
await expectBrowserHostNameInCommandCenter(page, firstHost.label);
const secondWindow = await openSecondDesktopWindow(page);
try {
await openBrowserWorkspace(secondWindow, workspaceKey);
await expect.poll(async () => (await listBrowserHosts(client)).length).toBe(2);
const secondHost = (await listBrowserHosts(client)).find(
(host) => host.browserHostId !== firstHost.browserHostId,
);
assert(secondHost?.label && secondHost.label !== firstHost.label, "Windows share a host label");
await expectBrowserHostNameInCommandCenter(secondWindow, secondHost.label);
const ambiguous = await client.callTool({ name: "browser_new_tab", args: { url: targetUrl } });
expect(ambiguous.structuredContent).toMatchObject({
ok: false,
error: { code: "browser_host_ambiguous", browserHosts: [firstHost, secondHost] },
});
expect((await callBrowserTool(client, "browser_list_tabs")).tabs).toEqual([]);
const { browserId, browserHost } = await callBrowserTool(client, "browser_new_tab", {
url: targetUrl,
browserHostId: firstHost.browserHostId,
});
expect(browserHost).toEqual(firstHost);
// Only the tab's test ID carries the exact ID returned by the agent tool.
await expect(page.getByTestId(`workspace-tab-browser_${browserId}`)).toBeVisible();
await expect(secondWindow.getByTestId(`workspace-tab-browser_${browserId}`)).toHaveCount(0);
expect((await callBrowserTool(client, "browser_list_tabs")).tabs).toMatchObject([
{ browserId, browserHost: firstHost, isActive: false },
]);
await callBrowserTool(client, "browser_wait", { browserId, text: "Bridge target" });
await callBrowserTool(client, "browser_close_tab", { browserId });
} finally {
await secondWindow.close();
}
await expect.poll(() => listBrowserHosts(client)).toEqual([firstHost]);
}
async function waitForGuestSelector(client, browserId) {
const deadline = Date.now() + 5_000;
while (Date.now() < deadline) {
@@ -1413,12 +1493,7 @@ async function main() {
workspaceId: workspaceIds.at(-1),
agentId: await createCallerAgent(daemonPort, workspaceIds.at(-1)),
};
const transport = new StreamableHTTPClientTransport(
new URL(
`http://127.0.0.1:${daemonPort}/mcp/agents?callerAgentId=${encodeURIComponent(callerAgentId)}`,
),
);
client = await experimental_createMCPClient({ transport });
client = await connectBrowserToolClient(daemonPort, callerAgentId);
const report = await runRegression({
page,
client,
@@ -1430,11 +1505,23 @@ async function main() {
artifactDir,
});
const pluginLinks = await checkPluginLinks();
const routingClient = await connectBrowserToolClient(daemonPort, typingAgent.agentId);
try {
await verifyBrowserHostRouting({
page,
client: routingClient,
targetUrl: target.url,
workspaceKey: `${status.serverId}:${typingAgent.workspaceId}`,
});
} finally {
await routingClient.close();
}
writeJson(path.join(artifactDir, "result.json"), {
...report,
settingsMemory,
pluginLinks,
adaptiveTheme,
browserHostRouting: "passed",
});
console.log(
`Browser desktop browser E2E passed: WebContents ${report.originalWebContentsId} remained ${report.finalWebContentsId}; viewport, inactive capture, focus continuity, list, snapshot, click, local-page selectors passed.`,
+16 -1
View File
@@ -10,6 +10,8 @@ log.initialize({ spyRendererConsole: true });
import { inheritLoginShellEnv } from "./login-shell-env.js";
import path from "node:path";
import { randomBytes, randomUUID } from "node:crypto";
import { hostname } from "node:os";
import { pathToFileURL } from "node:url";
import { existsSync } from "node:fs";
import { execFileSync } from "node:child_process";
@@ -668,6 +670,14 @@ function getWorkAreasPrimaryFirst(): Electron.Rectangle[] {
return [primary, ...others].map((display) => display.workArea);
}
const browserHostComputerName =
hostname()
.replace(/\.local$/i, "")
.trim()
.slice(0, 120) || "Desktop";
const browserHostSessionCode = randomBytes(3).toString("hex").toUpperCase();
let browserHostWindowNumber = 0;
async function createWindow(
options: {
initialRoute?: string | null;
@@ -693,6 +703,7 @@ async function createWindow(
: null;
const title = devWorktreeName ? `${APP_NAME} (${devWorktreeName})` : APP_NAME;
const browserHostLabel = `${browserHostComputerName} · Window ${++browserHostWindowNumber} (${browserHostSessionCode})`;
const mainWindow = new BrowserWindow({
title,
...resolveWindowBounds(restoredWindowState),
@@ -704,7 +715,11 @@ async function createWindow(
}),
webPreferences: {
preload: getPreloadPath(),
additionalArguments: [windowChromeModeArgument(DESKTOP_WINDOW_CHROME_MODE)],
additionalArguments: [
windowChromeModeArgument(DESKTOP_WINDOW_CHROME_MODE),
`--paseo-browser-host-label=${browserHostLabel}`,
`--paseo-browser-host-instance-id=${randomUUID()}`,
],
contextIsolation: true,
nodeIntegration: false,
webviewTag: true,
+6
View File
@@ -12,6 +12,10 @@ const PASEO_BROWSER_PROFILE_PARTITION = "persist:paseo-browser";
type EventHandler = (payload: unknown) => void;
function readArgValue(prefix: string): string | undefined {
return process.argv.find((argument) => argument.startsWith(prefix))?.slice(prefix.length);
}
function readWindowChromeMode(): DesktopWindowChromeMode {
const prefix = "--paseo-window-chrome-mode=";
const value = process.argv.find((argument) => argument.startsWith(prefix))?.slice(prefix.length);
@@ -32,6 +36,8 @@ interface AttachedBrowserRegistration {
contextBridge.exposeInMainWorld("paseoDesktop", {
platform: process.platform,
windowChromeMode: readWindowChromeMode(),
browserHostLabel: readArgValue("--paseo-browser-host-label="),
browserHostInstanceId: readArgValue("--paseo-browser-host-instance-id="),
invoke: (command: string, args?: Record<string, unknown>) =>
ipcRenderer.invoke("paseo:invoke", command, args),
getPendingOpenProject: () =>
@@ -31,6 +31,8 @@ export const BrowserAutomationHostCapabilitySchema = z
return supportedCommands;
}),
hostKind: z.string().min(1).default("browser host"),
label: z.string().min(1).max(160).optional(),
instanceId: z.uuid().optional(),
})
.passthrough();
@@ -13,7 +13,7 @@ import {
describe("browser automation protocol integration", () => {
const browserId = "11111111-1111-4111-8111-111111111111";
test("browser host capability parses supported commands in hello", () => {
test("browser host capability parses supported commands and a label in hello", () => {
expect(
WSHelloMessageSchema.parse({
type: "hello",
@@ -24,6 +24,7 @@ describe("browser automation protocol integration", () => {
[CLIENT_CAPS.browserHost]: {
supportedCommands: [...BROWSER_AUTOMATION_COMMAND_NAMES],
hostKind: "desktop app",
label: "MacBook Pro · Window 2",
},
},
}).capabilities,
@@ -31,6 +32,7 @@ describe("browser automation protocol integration", () => {
[CLIENT_CAPS.browserHost]: {
supportedCommands: [...BROWSER_AUTOMATION_COMMAND_NAMES],
hostKind: "desktop app",
label: "MacBook Pro · Window 2",
},
});
});
@@ -133,6 +135,23 @@ describe("browser automation protocol integration", () => {
).toBeUndefined();
});
test("browser host registration accepts window identity and remains valid without it", () => {
const request = {
type: "browser.host.register.request",
requestId: "register-browser",
hostKind: "desktop app",
supportedCommands: ["new_tab"],
};
const label = "MacBook Pro · Window 2";
const instanceId = "33333333-3333-4333-8333-333333333333";
expect(SessionInboundMessageSchema.parse(request)).toEqual(request);
expect(SessionInboundMessageSchema.parse({ ...request, label, instanceId })).toEqual({
...request,
label,
instanceId,
});
});
test("daemon to browser host execute request is an outbound session message", () => {
const parsed = SessionOutboundMessageSchema.parse({
type: "browser.automation.execute.request",
+2
View File
@@ -3493,6 +3493,8 @@ export const BrowserHostRegisterRequestSchema = z.object({
type: z.literal("browser.host.register.request"),
requestId: z.string(),
hostKind: z.string().min(1),
label: z.string().min(1).max(160).optional(),
instanceId: z.uuid().optional(),
supportedCommands: z.array(z.enum(BROWSER_AUTOMATION_COMMAND_NAMES)).min(1),
});
export const BrowserHostRegisterResponseSchema = z.object({
@@ -390,6 +390,7 @@ export const PASEO_TOOL_CATALOG = {
},
// Browser
browser_list_hosts: browser("List browser hosts", "Listed browser hosts"),
browser_list_tabs: browser("List browser tabs", "Listed browser tabs"),
browser_new_tab: browser("Open browser tab", "Opened browser tab", field("url")),
browser_close_tab: browser("Close browser tab", "Closed browser tab", field("browserId")),
@@ -8,6 +8,10 @@ import { AgentManager, type AgentManagerEvent } from "./agent-manager.js";
import { AGENT_STREAM_COALESCE_DEFAULT_WINDOW_MS } from "./agent-stream-coalescer.js";
import { limitAgentTimelineItemContent } from "./agent-timeline-content.js";
import type { AgentTimelineRow } from "./agent-timeline-store-types.js";
import { OmpAgentClient } from "./providers/omp/agent.js";
import { FakeOmp } from "./providers/omp/test-utils/fake-omp.js";
import { PiRpcAgentClient } from "./providers/pi/agent.js";
import { FakePi } from "./providers/pi/test-utils/fake-pi.js";
import { projectTimelineRows } from "./timeline-projection.js";
import type {
AgentCapabilityFlags,
@@ -394,6 +398,235 @@ afterEach(() => {
vi.useRealTimers();
});
describe("live notification turns", () => {
test.each([
{ sequence: "while idle", agentStarted: false, emptyCycle: false },
{ sequence: "between agent_start and turn_start", agentStarted: true, emptyCycle: false },
{ sequence: "after an empty cycle", agentStarted: true, emptyCycle: true },
])(
"keeps OMP replies in separate turns with a notice $sequence",
async ({ agentStarted, emptyCycle }) => {
const runtime = new FakeOmp();
const provider = "omp";
const logger = createTestLogger();
const manager = new AgentManager({
clients: {
omp: new OmpAgentClient({
runtime,
logger,
runtimeSettings: { command: { mode: "replace", argv: [process.execPath] } },
}),
},
logger,
});
const workdir = mkdtempSync(join(tmpdir(), "omp-live-turns-"));
const events: AgentManagerEvent[] = [];
manager.subscribe((event) => events.push(event), { replayState: false });
const { id } = await manager.createAgent({ provider, cwd: workdir }, undefined, {
workspaceId: undefined,
});
try {
const session = runtime.latestSession();
const finishReply = async (text: string) => {
const completed = () =>
getStreamEvents(events, id).filter(
(event) => event.type === "agent_stream" && event.event.type === "turn_completed",
).length;
const before = completed();
session.streamAssistantText(text, text);
session.finishTurn({ role: "assistant", content: [] });
await vi.waitFor(() => {
expect(completed()).toBe(before + 1);
});
};
const notice = (message: string) => {
session.emit({ type: "extension_ui_request", id: message, method: "notify", message });
};
session.beginTurn();
await finishReply("First reply");
if (agentStarted) session.emit({ type: "agent_start" });
if (emptyCycle) session.emit({ type: "agent_end", messages: [] });
notice("Idle notice");
session.beginTurn();
session.acceptCustomMessage("<system-notice>Completion</system-notice>");
notice("Running notice");
await finishReply("Second reply");
const page = await manager.fetchTimeline(id, { wholeTurns: true, limit: 0 });
const ids = page.rows.map((row) => row.turnId);
expect(
page.rows.map(({ item }) => {
if (item.type === "assistant_message") return item.text;
if (item.type === "notification") return item.message;
return item.type;
}),
).toEqual(["First reply", "Idle notice", "Completion", "Running notice", "Second reply"]);
const [first, idle, wake] = ids;
expect(ids).toEqual([
expect.any(String),
expect.any(String),
expect.any(String),
wake,
wake,
]);
expect(new Set([first, idle, wake]).size).toBe(3);
expect(
getTimelineStreamEvents(events, id).map((event) =>
event.type === "agent_stream" && event.event.type === "timeline"
? event.event.turnId
: null,
),
).toEqual(ids);
expect(
(await manager.fetchTimeline(id, { wholeTurns: true, limit: 1 })).rows.map(
(row) => row.turnId,
),
).toEqual([wake, wake, wake]);
} finally {
await manager.closeAgent(id);
rmSync(workdir, { recursive: true, force: true });
}
},
);
test("keeps a Claude completion with the run it wakes after an independent notice", async () => {
const harness = createHarness({ provider: "claude" });
try {
const { agentId, session } = await createManagedSession(harness);
session.pushEvent(assistant("First reply", "claude", "first"));
session.pushEvent(
timelineEvent({ type: "notification", level: "info", message: "Idle" }, "claude"),
);
const completion = { ...toolCall({ status: "completed" }), name: "task_notification" };
session.pushEvent(
timelineEvent({ ...completion, callId: "stopped-task" }, "claude", "stopped"),
);
session.pushEvent(timelineEvent(completion, "claude", "wake"));
session.pushEvent({ type: "turn_started", provider: "claude", turnId: "wake" });
session.pushEvent(assistant("Second reply", "claude", "wake"));
session.pushEvent({ type: "turn_completed", provider: "claude", turnId: "wake" });
await vi.waitFor(() => {
expect(getStreamEvents(harness.events, agentId).at(-1)).toMatchObject({
event: { type: "turn_completed", turnId: "wake" },
});
});
const page = await harness.manager.fetchTimeline(agentId, { wholeTurns: true, limit: 0 });
expect(page.rows.map((row) => row.turnId)).toEqual([
"first",
expect.any(String),
"stopped",
"wake",
"wake",
]);
expect(page.rows[1]?.turnId).not.toBe("first");
expect(page.rows[1]?.turnId).not.toBe("wake");
expect(
getStreamEvents(harness.events, agentId).filter(
(event) => event.type === "agent_stream" && event.event.type === "turn_started",
),
).toEqual([
expect.objectContaining({
event: { type: "turn_started", provider: "claude", turnId: "wake" },
}),
]);
expect(
(await harness.manager.fetchTimeline(agentId, { wholeTurns: true, limit: 1 })).rows.map(
(row) => row.item,
),
).toEqual([completion, { type: "assistant_message", text: "Second reply" }]);
} finally {
harness.cleanup();
}
});
test("an OMP runtime notice during a foreground turn reaches the run caller", async () => {
const runtime = new FakeOmp();
const logger = createTestLogger();
const manager = new AgentManager({
clients: {
omp: new OmpAgentClient({
runtime,
logger,
runtimeSettings: { command: { mode: "replace", argv: [process.execPath] } },
}),
},
logger,
});
const workdir = mkdtempSync(join(tmpdir(), "omp-foreground-notice-"));
const agent = await manager.createAgent({ provider: "omp", cwd: workdir }, undefined, {
workspaceId: undefined,
});
const session = runtime.latestSession();
try {
const result = manager.runAgent(agent.id, "Work");
await vi.waitFor(() => expect(session.prompts).toHaveLength(1));
session.beginTurn();
session.emit({
type: "notice",
level: "info",
message: "Working",
});
session.streamAssistantText("Done");
session.finishTurn({ role: "assistant", content: [] });
const item = { type: "notification", level: "info", message: "Working" };
expect((await result).timeline).toContainEqual(item);
const rows = (await manager.fetchTimeline(agent.id, { wholeTurns: true, limit: 0 })).rows;
expect(rows.find((row) => row.item.type === "notification")?.turnId).toEqual(
expect.any(String),
);
expect(rows.find((row) => row.item.type === "notification")?.turnId).toEqual(
rows.find((row) => row.item.type === "assistant_message")?.turnId,
);
} finally {
await manager.closeAgent(agent.id);
rmSync(workdir, { recursive: true, force: true });
}
});
test("an idle Pi UI notice gets its own turn without starting a run", async () => {
const runtime = new FakePi();
const logger = createTestLogger();
const manager = new AgentManager({
clients: {
pi: new PiRpcAgentClient({
runtime,
logger,
runtimeSettings: { command: { mode: "replace", argv: [process.execPath] } },
}),
},
logger,
});
const workdir = mkdtempSync(join(tmpdir(), "pi-idle-notice-"));
const events: AgentManagerEvent[] = [];
manager.subscribe((event) => events.push(event), { replayState: false });
const { id } = await manager.createAgent({ provider: "pi", cwd: workdir }, undefined, {
workspaceId: undefined,
});
try {
runtime.latestSession().emit({
type: "extension_ui_request",
id: "notice",
method: "notify",
message: "Idle",
});
await vi.waitFor(() => expect(getTimelineStreamEvents(events, id)).toHaveLength(1));
const { rows } = await manager.fetchTimeline(id, { wholeTurns: true, limit: 1 });
const item = { type: "notification", level: "info", message: "Idle" };
expect(rows).toEqual([expect.objectContaining({ item, turnId: expect.any(String) })]);
expect(getStreamEvents(events, id)).toEqual([
expect.objectContaining({
event: { type: "timeline", provider: "pi", item, turnId: rows[0]?.turnId },
}),
]);
expect(manager.getAgent(id)?.lifecycle).toBe("idle");
} finally {
await manager.closeAgent(id);
rmSync(workdir, { recursive: true, force: true });
}
});
});
describe("target coalesced behavior", () => {
test("bounds tool output before persisting and streaming it", async () => {
const harness = createHarness();
@@ -608,9 +608,11 @@ function attachManagedTurnIdentity(
return turnId ? { event: { ...event, turnId }, turnId } : { event, turnId };
}
case "timeline": {
// Live provider items belong to the foreground turn that owns their dispatch.
// Provider history deliberately keeps absent IDs because it has no daemon turn identity.
const turnId = agent.activeForegroundTurnId ?? agent.activeTurnId ?? undefined;
// A notice with no open run is a display turn of its own.
const turnId =
agent.activeForegroundTurnId ??
agent.activeTurnId ??
(event.item.type === "notification" ? randomUUID() : undefined);
return turnId ? { event: { ...event, turnId }, turnId } : { event, turnId };
}
default:
@@ -4825,6 +4827,10 @@ export class AgentManager {
this.publishBackgroundWork(agent.id, items);
return;
}
// Resolve notice membership before selecting the foreground run's waiters.
if (event.type === "timeline" && event.item.type === "notification") {
event = attachManagedTurnIdentity(agent, event, false).event;
}
const turnId = getAgentStreamEventTurnId(event);
const matchingWaiters = this.runs.getMatchingWaiters(agent, turnId);
this.logger.trace(
@@ -1583,7 +1583,6 @@ export class OmpAgentSession implements AgentSession {
this.emit({
type: "timeline",
provider: this.provider,
turnId: this.currentTurnIdForEvent(),
item: {
type: "notification",
level:
@@ -101,6 +101,33 @@ describe("V2Runtime against an external server", () => {
expect(requests).toHaveLength(1);
});
test("reports the external server as exited once it stops accepting connections", async () => {
server = createServer((request, response) => {
if (request.url !== "/api/info") {
response.writeHead(404);
response.end();
return;
}
response.writeHead(200, { "content-type": "application/json" });
response.end(JSON.stringify({ version: "2.0.10", pid: 1, urls: [], paths: { tmp: "/tmp" } }));
});
const port = await listen(server);
const runtime = new V2Runtime({
logger: createTestLogger(),
externalServer: { url: `http://127.0.0.1:${port}` },
});
const connection = await runtime.acquire();
const exited = connection.exited.then((error) => error.message);
await close(server);
server = undefined;
await expect(connection.client.server.info()).rejects.toThrow();
await expect(exited).resolves.toContain(`OpenCode server at http://127.0.0.1:${port} stopped`);
await connection.release();
await runtime.shutdown();
});
test("selects the external server from provider environment", async () => {
const paths: Array<string | undefined> = [];
server = createServer((request, response) => {
@@ -53,7 +53,14 @@ export async function connectSharedService(
const signal = init?.signal
? AbortSignal.any([init.signal, lifetime.signal])
: lifetime.signal;
const response = await fetch(request, { ...init, signal });
let response: Response;
try {
response = await fetch(request, { ...init, signal });
} catch (error) {
// Paseo does not own this process, so a refused connection is its only exit signal.
if (isConnectionRefused(error)) markExited(error);
throw error;
}
const html = response.headers.get("content-type")?.includes("text/html") ?? false;
if (!response.ok || html) {
const requestUrl = request instanceof Request ? request.url : String(request);
@@ -62,6 +69,15 @@ export async function connectSharedService(
return response;
},
});
const markExited = (cause: unknown) => {
if (lifetime.signal.aborted) return;
const error = new Error(`OpenCode server at ${server.url} stopped accepting connections`, {
cause,
});
logger.warn({ url: server.url }, "OpenCode v2 server stopped accepting connections");
lifetime.abort(error);
resolveExited(error);
};
let info: Awaited<ReturnType<typeof client.server.info>>;
try {
info = await client.server.info({
@@ -117,6 +133,13 @@ export async function connectSharedService(
return generation;
}
function isConnectionRefused(error: unknown): boolean {
const cause = error instanceof Error ? error.cause : undefined;
return Boolean(
cause && typeof cause === "object" && "code" in cause && cause.code === "ECONNREFUSED",
);
}
function validateBridgeAddress(
selection: Extract<OpenCodeServiceSelection, { kind: "explicit" | "discovered" }>,
bridge: OpenCodeBridge | undefined,
@@ -2074,7 +2074,6 @@ export class PiRpcAgentSession implements AgentSession {
this.emit({
type: "timeline",
provider: this.provider,
turnId: this.currentTurnIdForEvent(),
item: {
type: "notification",
level: toNotificationLevel(event.notifyType),
@@ -14,6 +14,8 @@ const SECOND_BROWSER_ID = "22222222-2222-4222-8222-222222222222";
class FakeBrowserHostClient implements BrowserHostClient {
public readonly receivedRequests: BrowserAutomationExecuteRequest[] = [];
public readonly hostKind: string;
public readonly label?: string;
public readonly instanceId?: string;
public readonly supportedCommands: readonly BrowserAutomationCommandName[];
public constructor(
@@ -21,9 +23,13 @@ class FakeBrowserHostClient implements BrowserHostClient {
options: {
hostKind?: string;
supportedCommands?: readonly BrowserAutomationCommandName[];
label?: string;
instanceId?: string;
} = {},
) {
this.hostKind = options.hostKind ?? "desktop app";
this.label = options.label;
this.instanceId = options.instanceId;
this.supportedCommands = options.supportedCommands ?? [...BROWSER_AUTOMATION_COMMAND_NAMES];
}
@@ -170,6 +176,7 @@ describe("BrowserToolsBroker", () => {
title: "Example",
isActive: false,
isLoading: false,
browserHost: broker.listBrowserHosts()[0],
},
],
},
@@ -229,20 +236,31 @@ describe("BrowserToolsBroker", () => {
});
});
test("new tabs target the most recently registered host and tab commands stay with that host", async () => {
test("new tabs require a choice with multiple hosts and tab commands stay with the chosen host", async () => {
const broker = createBroker();
const firstHost = new FakeBrowserHostClient("host-1");
const firstHost = new FakeBrowserHostClient("host-1", { label: "MacBook Pro · Window 1" });
const recentHost = new FakeBrowserHostClient("host-2");
broker.registerClient(firstHost);
broker.registerClient(recentHost);
const newTabPromise = broker.execute({
const input = {
command: { command: "new_tab", args: { url: "https://example.com" } },
workspaceId: "workspace-1",
} as const;
const [selectedHost, otherHost] = broker.listBrowserHosts();
expect(selectedHost.label).toBe("MacBook Pro · Window 1");
expect(otherHost.label).toBe("desktop app · Connection 2");
await expect(broker.execute(input)).resolves.toMatchObject({
ok: false,
error: { code: "browser_host_ambiguous", browserHosts: [selectedHost, otherHost] },
});
expect(firstHost.receivedRequests).toEqual([]);
expect(recentHost.receivedRequests).toEqual([
expect(recentHost.receivedRequests).toEqual([]);
const newTabPromise = broker.execute({ ...input, browserHostId: selectedHost.browserHostId });
expect(recentHost.receivedRequests).toEqual([]);
expect(firstHost.receivedRequests).toEqual([
{
type: "browser.automation.execute.request",
requestId: "req-1",
@@ -251,7 +269,7 @@ describe("BrowserToolsBroker", () => {
},
]);
recentHost.resolveLatestWith(broker, {
firstHost.resolveLatestWith(broker, {
requestId: "req-1",
ok: true,
result: {
@@ -264,7 +282,7 @@ describe("BrowserToolsBroker", () => {
await expect(newTabPromise).resolves.toMatchObject({
ok: true,
result: { command: "new_tab", browserId: BROWSER_ID },
result: { command: "new_tab", browserId: BROWSER_ID, browserHost: selectedHost },
});
const snapshotPromise = broker.execute({
@@ -272,15 +290,15 @@ describe("BrowserToolsBroker", () => {
workspaceId: "workspace-1",
});
expect(firstHost.receivedRequests).toEqual([]);
expect(recentHost.receivedRequests.at(-1)).toEqual({
expect(recentHost.receivedRequests).toEqual([]);
expect(firstHost.receivedRequests.at(-1)).toEqual({
type: "browser.automation.execute.request",
requestId: "req-1",
workspaceId: "workspace-1",
command: { command: "snapshot", args: { browserId: BROWSER_ID } },
});
recentHost.resolveLatestWith(broker, {
firstHost.resolveLatestWith(broker, {
requestId: "req-1",
ok: true,
result: {
@@ -302,6 +320,102 @@ describe("BrowserToolsBroker", () => {
});
});
test("a disconnected selection stays invalid and the sole eligible host is automatic", async () => {
const broker = createBroker();
const formerHost = new FakeBrowserHostClient("former", { label: "Mac Mini · Window 1" });
const remainingHost = new FakeBrowserHostClient("remaining");
const inventoryHost = new FakeBrowserHostClient("inventory", {
supportedCommands: ["list_tabs"],
});
const unregister = broker.registerClient(formerHost);
broker.registerClient(remainingHost);
broker.registerClient(inventoryHost);
const [previous, remaining, inventoryOnly] = broker.listBrowserHosts();
expect(inventoryOnly.canOpenTabs).toBe(false);
unregister();
expect(broker.listBrowserHosts()).toEqual([remaining, inventoryOnly]);
const input = {
command: { command: "new_tab", args: {} },
workspaceId: "workspace-1",
} as const;
await expect(
broker.execute({ ...input, browserHostId: previous.browserHostId }),
).resolves.toMatchObject({ ok: false, error: { code: "browser_no_host" } });
await expect(
broker.execute({ ...input, browserHostId: inventoryOnly.browserHostId }),
).resolves.toMatchObject({ ok: false, error: { code: "browser_unsupported" } });
expect(remainingHost.receivedRequests).toEqual([]);
expect(inventoryHost.receivedRequests).toEqual([]);
const creation = broker.execute(input);
remainingHost.resolveLatestWith(broker, {
requestId: "req-1",
ok: true,
result: {
command: "new_tab",
browserId: BROWSER_ID,
workspaceId: "workspace-1",
url: "https://example.com",
},
});
await expect(creation).resolves.toMatchObject({
ok: true,
result: { browserHost: remaining },
});
expect(inventoryHost.receivedRequests).toEqual([]);
broker.registerClient(new FakeBrowserHostClient("reconnected", { label: previous.label }));
const reconnected = broker.listBrowserHosts().find(({ label }) => label === previous.label);
expect(reconnected).toBeDefined();
expect(reconnected?.browserHostId).not.toBe(previous.browserHostId);
await expect(
broker.execute({ ...input, browserHostId: previous.browserHostId }),
).resolves.toMatchObject({ ok: false, error: { code: "browser_no_host" } });
expect(formerHost.receivedRequests).toEqual([]);
});
test("a reconnect replaces the old connection for the same browser host", async () => {
const broker = createBroker();
const identity = {
instanceId: "33333333-3333-4333-8333-333333333333",
label: "Desktop · Window 1",
};
const oldConnection = new FakeBrowserHostClient("old", identity);
const releaseOld = broker.registerClient(oldConnection);
const [target] = broker.listBrowserHosts();
const pending = broker.execute({ command: { command: "new_tab", args: {} } });
const newConnection = new FakeBrowserHostClient("new", identity);
broker.registerClient(newConnection);
expect(broker.listBrowserHosts()).toEqual([target]);
await expect(pending).resolves.toMatchObject({
ok: false,
error: { code: "browser_no_host" },
});
releaseOld();
const creation = broker.execute({
command: { command: "new_tab", args: {} },
browserHostId: target.browserHostId,
});
newConnection.resolveLatestWith(broker, {
requestId: "req-1",
ok: true,
result: {
command: "new_tab",
browserId: BROWSER_ID,
workspaceId: "workspace-1",
url: "https://example.com",
},
});
await expect(creation).resolves.toMatchObject({
ok: true,
result: { browserHost: target },
});
expect(oldConnection.receivedRequests).toHaveLength(1);
});
test("list tabs aggregates all hosts and seeds browser id affinity", async () => {
const broker = createBroker();
const firstHost = new FakeBrowserHostClient("host-1");
@@ -375,6 +489,7 @@ describe("BrowserToolsBroker", () => {
title: "One",
isActive: false,
isLoading: false,
browserHost: broker.listBrowserHosts()[0],
},
{
browserId: SECOND_BROWSER_ID,
@@ -383,6 +498,7 @@ describe("BrowserToolsBroker", () => {
title: "Two",
isActive: false,
isLoading: false,
browserHost: broker.listBrowserHosts()[1],
},
],
},
@@ -479,7 +595,11 @@ describe("BrowserToolsBroker", () => {
broker.registerClient(other);
broker.registerClient(owner);
const newTabPromise = broker.execute({ command: { command: "new_tab", args: {} } });
const [, selectedHost] = broker.listBrowserHosts();
const newTabPromise = broker.execute({
command: { command: "new_tab", args: {} },
browserHostId: selectedHost.browserHostId,
});
owner.resolveLatestWith(broker, {
requestId: "req-1",
ok: true,
@@ -521,7 +641,11 @@ describe("BrowserToolsBroker", () => {
broker.registerClient(other);
broker.registerClient(owner);
const newTabPromise = broker.execute({ command: { command: "new_tab", args: {} } });
const [, selectedHost] = broker.listBrowserHosts();
const newTabPromise = broker.execute({
command: { command: "new_tab", args: {} },
browserHostId: selectedHost.browserHostId,
});
owner.resolveLatestWith(broker, {
requestId: "req-1",
ok: true,
@@ -1,4 +1,4 @@
import { randomUUID } from "node:crypto";
import { createHmac, randomBytes, randomUUID } from "node:crypto";
import {
BrowserAutomationExecuteRequestSchema,
BrowserAutomationExecuteResponseSchema,
@@ -7,11 +7,17 @@ import {
type BrowserAutomationExecuteRequest,
type BrowserAutomationExecuteResponse,
} from "@getpaseo/protocol/browser-automation/rpc-schemas";
import { browserToolsFailure, type BrowserToolsResponsePayload } from "./errors.js";
import {
browserToolsFailure,
type BrowserHostTarget,
type BrowserToolsResponsePayload,
} from "./errors.js";
export interface BrowserHostClient {
id: string;
hostKind: string;
label?: string;
instanceId?: string;
supportedCommands: readonly BrowserAutomationCommandName[];
sendBrowserAutomationRequest(request: BrowserAutomationExecuteRequest): void | Promise<void>;
}
@@ -23,10 +29,11 @@ export interface BrowserToolsExecuteInput {
workspaceId?: string;
requestId?: string;
timeoutMs?: number;
browserHostId?: string;
}
interface PendingBrowserToolsRequest {
clientId: string;
host: RegisteredBrowserHost;
rememberAffinity: boolean;
timeout: ReturnType<typeof setTimeout>;
resolve: (payload: BrowserToolsResponsePayload) => void;
@@ -34,10 +41,15 @@ interface PendingBrowserToolsRequest {
interface RegisteredBrowserHost {
client: BrowserHostClient;
target: BrowserHostTarget;
registeredAt: number;
supportedCommands: ReadonlySet<BrowserAutomationCommandName>;
}
type HostSelection =
| { ok: true; value: RegisteredBrowserHost }
| { ok: false; payload: BrowserToolsResponsePayload };
export interface BrowserToolsBrokerOptions {
defaultTimeoutMs?: number;
createRequestId?: () => string;
@@ -51,7 +63,7 @@ export class BrowserToolsBroker {
private readonly clients = new Map<string, RegisteredBrowserHost>();
private readonly pending = new Map<string, PendingBrowserToolsRequest>();
private readonly browserHostByBrowserId = new Map<string, string>();
private readonly strandedBrowserHostByBrowserId = new Map<string, string>();
private readonly hostIdKey = randomBytes(32);
private registrationSequence = 0;
public constructor(options: BrowserToolsBrokerOptions) {
@@ -60,35 +72,36 @@ export class BrowserToolsBroker {
}
public registerClient(client: BrowserHostClient): () => void {
this.unregisterClient(client.id);
// The discovery ID must not allow another connection to claim this window.
const browserHostId = createHmac("sha256", this.hostIdKey)
.update(client.instanceId ? `instance:${client.instanceId}` : `connection:${client.id}`)
.digest("hex");
this.unregisterHost(browserHostId);
const registeredAt = ++this.registrationSequence;
this.clients.set(client.id, {
const label = client.label?.trim() || `${client.hostKind} · Connection ${registeredAt}`;
const supportedCommands = new Set(client.supportedCommands);
this.clients.set(browserHostId, {
client,
target: Object.freeze({
browserHostId,
label,
canOpenTabs: supportedCommands.has("new_tab"),
}),
registeredAt,
supportedCommands: new Set(client.supportedCommands),
supportedCommands,
});
return () => this.unregisterClient(client.id, registeredAt);
return () => this.unregisterHost(browserHostId, registeredAt);
}
public unregisterClient(clientId: string, registeredAt?: number): void {
const current = this.clients.get(clientId);
private unregisterHost(browserHostId: string, registeredAt?: number): void {
const current = this.clients.get(browserHostId);
if (!current || (registeredAt !== undefined && current.registeredAt !== registeredAt)) {
return;
}
this.clients.delete(clientId);
for (const [browserId, ownerClientId] of this.browserHostByBrowserId) {
if (ownerClientId !== clientId) {
continue;
}
this.browserHostByBrowserId.delete(browserId);
this.strandedBrowserHostByBrowserId.set(browserId, clientId);
}
this.clients.delete(browserHostId);
for (const [requestId, pending] of this.pending) {
if (pending.clientId !== clientId) {
continue;
}
if (pending.host !== current) continue;
this.pending.delete(requestId);
clearTimeout(pending.timeout);
pending.resolve(
@@ -110,6 +123,10 @@ export class BrowserToolsBroker {
return this.clients.size;
}
public listBrowserHosts(): BrowserHostTarget[] {
return Array.from(this.clients.values(), ({ target }) => target);
}
public async execute(input: BrowserToolsExecuteInput): Promise<BrowserToolsResponsePayload> {
const requestId = input.requestId ?? this.createRequestId();
@@ -137,7 +154,11 @@ export class BrowserToolsBroker {
});
}
const host = this.selectHostForCommand(request.data.command, requestId);
const command = request.data.command;
const host =
command.command === "new_tab"
? this.selectHostForNewTab(requestId, input.browserHostId)
: this.selectHostForTab(command.args.browserId, requestId);
if (!host.ok) {
return host.payload;
}
@@ -170,7 +191,7 @@ export class BrowserToolsBroker {
}
const pending = this.pending.get(requestId);
if (!pending || (clientIds && !clientIds.includes(pending.clientId))) {
if (!pending || (clientIds && !clientIds.includes(pending.host.client.id))) {
return false;
}
@@ -187,16 +208,16 @@ export class BrowserToolsBroker {
}
const pending = this.pending.get(parsed.data.payload.requestId);
if (!pending || (clientIds && !clientIds.includes(pending.clientId))) {
if (!pending || (clientIds && !clientIds.includes(pending.host.client.id))) {
return false;
}
this.pending.delete(parsed.data.payload.requestId);
clearTimeout(pending.timeout);
if (pending.rememberAffinity) {
this.rememberBrowserHostForPayload(pending.clientId, parsed.data.payload);
if (pending.rememberAffinity && parsed.data.payload.ok) {
this.rememberBrowserHostForPayload(pending.host.target.browserHostId, parsed.data.payload);
}
pending.resolve(parsed.data.payload);
pending.resolve(attachBrowserHost(parsed.data.payload, pending.host.target));
return true;
}
@@ -249,7 +270,9 @@ export class BrowserToolsBroker {
}
for (const { host, payload } of hostResponses) {
this.rememberBrowserHostForPayload(host.client.id, payload);
if (payload.ok && this.clients.get(host.target.browserHostId) === host) {
this.rememberBrowserHostForPayload(host.target.browserHostId, payload);
}
}
return {
@@ -264,63 +287,60 @@ export class BrowserToolsBroker {
};
}
private selectHostForCommand(
command: BrowserAutomationCommand,
requestId: string,
):
| { ok: true; value: RegisteredBrowserHost }
| { ok: false; payload: BrowserToolsResponsePayload } {
if (command.command === "new_tab") {
const host = this.selectMostRecentlyRegisteredHost();
return host
? { ok: true, value: host }
: { ok: false, payload: this.noBrowserHostFailure(requestId) };
}
const browserId = getBrowserIdForCommand(command);
if (!browserId) {
const host = this.selectMostRecentlyRegisteredHost();
return host
? { ok: true, value: host }
: { ok: false, payload: this.noBrowserHostFailure(requestId) };
}
const ownerClientId = this.browserHostByBrowserId.get(browserId);
if (ownerClientId) {
const host = this.clients.get(ownerClientId);
if (host) {
return { ok: true, value: host };
}
private selectHostForNewTab(requestId: string, browserHostId: string | undefined): HostSelection {
if (browserHostId !== undefined) {
const host = this.clients.get(browserHostId);
if (host) return { ok: true, value: host };
return {
ok: false,
payload: this.strandedBrowserTabFailure({ requestId, browserId }),
payload: browserToolsFailure({
requestId,
code: "browser_no_host",
message:
"The selected browser host is not connected. Call browser_list_hosts to choose a connected host.",
}),
};
}
const strandedOwnerClientId = this.strandedBrowserHostByBrowserId.get(browserId);
if (strandedOwnerClientId) {
const reconnectedHost = this.clients.get(strandedOwnerClientId);
if (reconnectedHost) {
this.strandedBrowserHostByBrowserId.delete(browserId);
this.browserHostByBrowserId.set(browserId, strandedOwnerClientId);
return { ok: true, value: reconnectedHost };
}
if (this.clients.size === 0)
return { ok: false, payload: this.noBrowserHostFailure(requestId) };
const eligible = Array.from(this.clients.values()).filter(({ target }) => target.canOpenTabs);
if (eligible.length === 1) return { ok: true, value: eligible[0] };
if (eligible.length === 0) {
return {
ok: false,
payload: this.strandedBrowserTabFailure({ requestId, browserId }),
payload: browserToolsFailure({
requestId,
code: "browser_unsupported",
message: "None of the connected browser hosts support opening tabs.",
}),
};
}
return {
ok: false,
payload: browserToolsFailure({
requestId,
code: "browser_host_ambiguous",
message: "Choose a browser host and retry with its browserHostId.",
browserHosts: eligible.map(({ target }) => target),
}),
};
}
if (this.clients.size === 1) {
const host = this.selectMostRecentlyRegisteredHost();
if (host) {
return { ok: true, value: host };
}
private selectHostForTab(browserId: string, requestId: string): HostSelection {
const browserHostId = this.browserHostByBrowserId.get(browserId);
if (browserHostId) {
const host = this.clients.get(browserHostId);
return host
? { ok: true, value: host }
: { ok: false, payload: this.strandedBrowserTabFailure({ requestId, browserId }) };
}
if (this.clients.size === 0) {
const firstHost = this.clients.values().next().value;
if (!firstHost) {
return { ok: false, payload: this.noBrowserHostFailure(requestId) };
}
if (this.clients.size === 1) return { ok: true, value: firstHost };
return {
ok: false,
@@ -332,14 +352,6 @@ export class BrowserToolsBroker {
};
}
private selectMostRecentlyRegisteredHost(): RegisteredBrowserHost | null {
let selected: RegisteredBrowserHost | null = null;
for (const host of this.clients.values()) {
selected = host;
}
return selected;
}
private unsupportedCommandFailure(params: {
host: RegisteredBrowserHost;
commandName: BrowserAutomationCommandName;
@@ -377,30 +389,23 @@ export class BrowserToolsBroker {
}
private rememberBrowserHostForPayload(
clientId: string,
payload: BrowserToolsResponsePayload,
browserHostId: string,
payload: Extract<BrowserAutomationExecuteResponse["payload"], { ok: true }>,
): void {
if (!payload.ok) {
return;
}
if (payload.result.command === "list_tabs") {
for (const tab of payload.result.tabs) {
this.browserHostByBrowserId.set(tab.browserId, clientId);
this.strandedBrowserHostByBrowserId.delete(tab.browserId);
this.browserHostByBrowserId.set(tab.browserId, browserHostId);
}
return;
}
if (payload.result.command === "close_tab") {
this.browserHostByBrowserId.delete(payload.result.browserId);
this.strandedBrowserHostByBrowserId.delete(payload.result.browserId);
return;
}
if ("browserId" in payload.result) {
this.browserHostByBrowserId.set(payload.result.browserId, clientId);
this.strandedBrowserHostByBrowserId.delete(payload.result.browserId);
this.browserHostByBrowserId.set(payload.result.browserId, browserHostId);
}
}
@@ -429,7 +434,7 @@ export class BrowserToolsBroker {
}, timeoutMs);
this.pending.set(request.requestId, {
clientId: client.id,
host,
rememberAffinity: params.rememberAffinity ?? true,
timeout,
resolve,
@@ -458,11 +463,20 @@ export class BrowserToolsBroker {
}
}
function getBrowserIdForCommand(command: BrowserAutomationCommand): string | null {
if (command.command === "list_tabs" || command.command === "new_tab") {
return null;
function attachBrowserHost(
payload: BrowserAutomationExecuteResponse["payload"],
target: BrowserHostTarget,
): BrowserToolsResponsePayload {
if (!payload.ok) return payload;
const result = payload.result;
if (result.command === "new_tab") {
return { ...payload, result: { ...result, browserHost: target } };
}
return command.args.browserId;
if (result.command === "list_tabs") {
const tabs = result.tabs.map((tab) => ({ ...tab, browserHost: target }));
return { ...payload, result: { ...result, tabs } };
}
return { ...payload, result };
}
function describeBrowserHost(host: RegisteredBrowserHost): string {
@@ -3,14 +3,40 @@ import type {
BrowserAutomationExecuteResponse,
} from "@getpaseo/protocol/browser-automation/rpc-schemas";
export type BrowserToolsResponsePayload = BrowserAutomationExecuteResponse["payload"];
export type BrowserToolsErrorPayload = Extract<BrowserToolsResponsePayload, { ok: false }>["error"];
export interface BrowserHostTarget {
readonly browserHostId: string;
readonly label: string;
readonly canOpenTabs: boolean;
}
type BrowserHostResponse = BrowserAutomationExecuteResponse["payload"];
type BrowserHostSuccess = Extract<BrowserHostResponse, { ok: true }>;
type BrowserHostResult = BrowserHostSuccess["result"];
type NewTabResult = Extract<BrowserHostResult, { command: "new_tab" }>;
type ListTabsResult = Extract<BrowserHostResult, { command: "list_tabs" }>;
type BrowserHostFailure = Extract<BrowserHostResponse, { ok: false }>;
export type BrowserToolsErrorPayload = Omit<BrowserHostFailure["error"], "code"> & {
code: BrowserAutomationErrorCode | "browser_host_ambiguous";
browserHosts?: BrowserHostTarget[];
};
export type BrowserToolsResponsePayload =
| (Omit<BrowserHostSuccess, "result"> & {
result:
| Exclude<BrowserHostResult, NewTabResult | ListTabsResult>
| (NewTabResult & { browserHost: BrowserHostTarget })
| (Omit<ListTabsResult, "tabs"> & {
tabs: (ListTabsResult["tabs"][number] & { browserHost: BrowserHostTarget })[];
});
})
| (Omit<BrowserHostFailure, "error"> & { error: BrowserToolsErrorPayload });
export class BrowserToolsRequestError extends Error {
public readonly code: BrowserAutomationErrorCode;
public readonly retryable: boolean;
public constructor(error: BrowserToolsErrorPayload) {
public constructor(error: BrowserHostFailure["error"]) {
super(error.message);
this.name = "BrowserToolsRequestError";
this.code = error.code;
@@ -20,9 +46,10 @@ export class BrowserToolsRequestError extends Error {
export function browserToolsFailure(params: {
requestId: string;
code: BrowserAutomationErrorCode;
code: BrowserToolsErrorPayload["code"];
message: string;
retryable?: boolean;
browserHosts?: BrowserHostTarget[];
}): BrowserToolsResponsePayload {
return {
requestId: params.requestId,
@@ -31,6 +58,7 @@ export function browserToolsFailure(params: {
code: params.code,
message: params.message,
retryable: params.retryable ?? false,
...(params.browserHosts ? { browserHosts: params.browserHosts } : {}),
},
};
}
@@ -1,6 +1,6 @@
import { describe, expect, test } from "vitest";
import { z } from "zod";
import type { BrowserToolsBroker, BrowserToolsExecuteInput } from "./broker.js";
import type { BrowserToolsExecuteInput } from "./broker.js";
import type { BrowserToolsResponsePayload } from "./errors.js";
import { registerBrowserTools, type RegisterBrowserToolsOptions } from "./tools.js";
import type {
@@ -10,6 +10,11 @@ import type {
} from "../agent/tools/types.js";
const BROWSER_ID = "11111111-1111-4111-8111-111111111111";
const BROWSER_HOST = {
browserHostId: "22222222-2222-4222-8222-222222222222",
label: "MacBook Pro · Window 1",
canOpenTabs: true,
};
const BROWSER_ID_MESSAGE =
"browserId must be a real id returned by browser_new_tab or browser_list_tabs";
const WAIT_CONDITION_MESSAGE = "browser_wait requires exactly one of text or url";
@@ -27,6 +32,10 @@ class FakeBrowserBroker {
public constructor(private response: BrowserToolsResponsePayload = listTabsPayload()) {}
public listBrowserHosts() {
return [BROWSER_HOST];
}
public setResponse(response: BrowserToolsResponsePayload): void {
this.response = response;
}
@@ -53,7 +62,7 @@ class BrowserToolHarness {
registerTool: (name, config, handler) => {
this.tools.set(name, { config, handler });
},
broker: this.broker as Pick<BrowserToolsBroker, "execute">,
broker: this.broker,
...(this.callerAgentId ? { callerAgentId: this.callerAgentId } : {}),
resolveCallerAgent: () => this.callerAgent,
});
@@ -100,6 +109,7 @@ function listTabsPayload(): Extract<BrowserToolsResponsePayload, { ok: true }> {
tabs: [
{
browserId: BROWSER_ID,
browserHost: BROWSER_HOST,
url: "https://example.com",
title: "Example",
isActive: true,
@@ -117,6 +127,7 @@ function newTabPayload(): Extract<BrowserToolsResponsePayload, { ok: true }> {
result: {
command: "new_tab",
browserId: BROWSER_ID,
browserHost: BROWSER_HOST,
workspaceId: "wks_workspace_a",
url: "https://example.com",
},
@@ -565,6 +576,7 @@ describe("registerBrowserTools", () => {
const harness = new BrowserToolHarness();
expect(harness.toolNames()).toEqual([
"browser_list_hosts",
"browser_list_tabs",
"browser_new_tab",
"browser_snapshot",
@@ -606,7 +618,7 @@ describe("registerBrowserTools", () => {
expect(response.content).toEqual([
{
type: "text",
text: `Found 1 Paseo browser tab. Use these browserId values for tab-scoped browser tools.\n- browserId=${BROWSER_ID} active title="Example" url=https://example.com`,
text: `Found 1 Paseo browser tab. Use these browserId values for tab-scoped browser tools.\n- browserId=${BROWSER_ID} active title="Example" url=https://example.com host="${BROWSER_HOST.label}"`,
},
]);
});
@@ -628,11 +640,48 @@ describe("registerBrowserTools", () => {
expect(response.content).toEqual([
{
type: "text",
text: `Created browser tab browserId=${BROWSER_ID} url=https://example.com. Use this browserId for tab-scoped browser tools.`,
text: `Created browser tab browserId=${BROWSER_ID} url=https://example.com on ${BROWSER_HOST.label}. Use this browserId for tab-scoped browser tools.`,
},
]);
});
test("lists labeled browser hosts and passes the selected ID for new tabs", async () => {
const harness = new BrowserToolHarness();
const [host] = harness.broker.listBrowserHosts();
const hosts = await harness.execute("browser_list_hosts", {});
expect(hosts.structuredContent).toEqual({ browserHosts: [host] });
expect(hosts.content).toEqual([
{ type: "text", text: `Connected browser hosts\n- ${host.label} (${host.browserHostId})` },
]);
expect(harness.broker.calls).toEqual([]);
harness.broker.setResponse({
requestId: "ambiguous",
ok: false,
error: {
code: "browser_host_ambiguous",
message: "Choose a browser host.",
retryable: false,
browserHosts: [host],
},
});
const ambiguous = await harness.execute("browser_new_tab", {});
expect(ambiguous.structuredContent).toMatchObject({
ok: false,
error: { code: "browser_host_ambiguous", browserHosts: [host] },
});
expect(ambiguous.content).toEqual([
{ type: "text", text: `Choose a browser host.\n- ${host.label} (${host.browserHostId})` },
]);
harness.broker.setResponse(newTabPayload());
await harness.execute("browser_new_tab", { browserHostId: host.browserHostId });
expect(harness.broker.calls.at(-1)).toMatchObject({
browserHostId: host.browserHostId,
command: { command: "new_tab", args: {} },
});
});
test.each([
{
name: "navigate accepts localhost without a scheme as http",
@@ -1,7 +1,7 @@
import { z } from "zod";
import { BrowserAutomationBrowserIdSchema } from "@getpaseo/protocol/browser-automation/rpc-schemas";
import type { BrowserToolsBroker } from "./broker.js";
import type { BrowserToolsResponsePayload } from "./errors.js";
import type { BrowserHostTarget, BrowserToolsResponsePayload } from "./errors.js";
import type {
PaseoToolConfig,
PaseoToolExecutionContext,
@@ -24,7 +24,7 @@ export interface RegisterBrowserToolsOptions {
context: PaseoToolExecutionContext,
) => Promise<PaseoToolResult>,
) => void;
broker: Pick<BrowserToolsBroker, "execute">;
broker: Pick<BrowserToolsBroker, "execute" | "listBrowserHosts">;
callerAgentId?: string;
resolveCallerAgent: () => CallerAgentContext | null;
}
@@ -64,6 +64,24 @@ const BrowserWaitInputSchema = z
});
export function registerBrowserTools(options: RegisterBrowserToolsOptions): void {
options.registerTool(
"browser_list_hosts",
{
title: "List browser hosts",
description:
"List connected Paseo browser hosts, including hosts with no tabs. Use a browserHostId from this list to choose where browser_new_tab opens.",
inputSchema: {},
},
async () => {
const browserHosts = options.broker.listBrowserHosts();
const text =
browserHosts.length > 0
? ["Connected browser hosts", ...browserHosts.map(formatBrowserHost)].join("\n")
: "No browser hosts are connected.";
return { content: [{ type: "text", text }], structuredContent: { browserHosts } };
},
);
options.registerTool(
"browser_list_tabs",
{
@@ -96,12 +114,20 @@ export function registerBrowserTools(options: RegisterBrowserToolsOptions): void
{
title: "Create browser tab",
description:
"Create a new Paseo browser tab in this agent's workspace on the most recently connected browser automation host, opened in the background without switching the user's view. Pass an http(s) URL or a scheme-less host URL, which is treated as http; the returned browserId is used by tab-scoped tools.",
"Open a background browser tab in this agent's workspace. Choose a browserHostId when multiple browser hosts can open tabs. Use the returned browserId for tab commands.",
inputSchema: {
url: BrowserHttpUrlInputSchema.optional(),
url: BrowserHttpUrlInputSchema.optional().describe(
"An HTTP or HTTPS URL. A URL without a scheme uses HTTP.",
),
browserHostId: z
.string()
.min(1)
.max(128)
.optional()
.describe("Destination ID from browser_list_hosts."),
},
},
async ({ url }) => {
async ({ url, browserHostId }) => {
const context = resolveBrowserToolContext(options);
const missingWorkspace = requireWorkspaceContext(context);
if (missingWorkspace) {
@@ -111,6 +137,7 @@ export function registerBrowserTools(options: RegisterBrowserToolsOptions): void
agentId: context.agentId,
cwd: context.cwd,
...(context.workspaceId ? { workspaceId: context.workspaceId } : {}),
...(browserHostId ? { browserHostId } : {}),
command: {
command: "new_tab",
args: url ? { url } : {},
@@ -875,7 +902,7 @@ function summarizeBrowserSuccess(
}
const tabLines = payload.result.tabs.map((tab) => {
const active = tab.isActive ? " active" : "";
return `- browserId=${tab.browserId}${active} title=${JSON.stringify(tab.title || "Untitled")} url=${tab.url}`;
return `- browserId=${tab.browserId}${active} title=${JSON.stringify(tab.title || "Untitled")} url=${tab.url} host=${JSON.stringify(tab.browserHost.label)}`;
});
return withDialogs(
[
@@ -887,7 +914,7 @@ function summarizeBrowserSuccess(
if (payload.result.command === "new_tab") {
return withDialogs(
`Created browser tab browserId=${payload.result.browserId} url=${payload.result.url}. Use this browserId for tab-scoped browser tools.`,
`Created browser tab browserId=${payload.result.browserId} url=${payload.result.url} on ${payload.result.browserHost.label}. Use this browserId for tab-scoped browser tools.`,
);
}
@@ -1041,7 +1068,14 @@ function summarizeBrowserError(
return "Browser tools are disabled. Enable browser tools on the host, then try again.";
case "browser_stale_ref":
return "That browser element reference is stale. Take a new browser snapshot and try again.";
case "browser_host_ambiguous":
return [error.message, ...(error.browserHosts ?? []).map(formatBrowserHost)].join("\n");
default:
return error.message;
}
}
function formatBrowserHost(host: BrowserHostTarget): string {
const capability = host.canOpenTabs ? "" : " (cannot open tabs)";
return `- ${host.label} (${host.browserHostId})${capability}`;
}
+4
View File
@@ -1407,6 +1407,8 @@ export class Session {
type: "browser.host.register.request" as const,
requestId: uuidv4(),
hostKind: host.data.hostKind,
label: host.data.label,
instanceId: host.data.instanceId,
supportedCommands: host.data.supportedCommands,
};
void this.delivery
@@ -2571,6 +2573,8 @@ export class Session {
unregister = this.browserToolsBroker.registerClient({
id: owner.id,
hostKind: request.hostKind,
label: request.label,
instanceId: request.instanceId,
supportedCommands: request.supportedCommands,
sendBrowserAutomationRequest: (message) => owner.emit(message),
});
+37 -3
View File
@@ -1,5 +1,5 @@
# LegendList 3.7.1 for the native chat timeline (docs/agent-stream-disclosure.md).
# Upstream: https://github.com/LegendApp/legend-list/issues/593 covers every change below.
# Upstream: https://github.com/LegendApp/legend-list/issues/593 covers changes 1–12 below.
# react-native.js and react-native.mjs carry the same hunks; each has a "Paseo patch" comment.
#
# 1. getEstimatedItemSize (react-native.d.ts, getItemSize, LegendListInner props): a per-item
@@ -34,6 +34,8 @@
# of a shrink past the end from the new end, assuming the scroll view clamped first. The app's
# scroll view clamps after the adjustment on both platforms (modules/paseo-scroll), so a
# bottom-edge disclosure collapsing near the end landed at the end instead of in place.
# 13. Recalculate the rendered range when drawDistance changes (LegendListInner). Chat uses
# a small window for initial placement and schedules expansion after initial reveal.
#
# Upgrading: drop each hunk whose fix the new version ships, regenerate with
# `npx patch-package @legendapp/list`, and copy this header back on top.
@@ -54,7 +56,7 @@ index 983e937..970ba51 100644
* Returns a stable item type used for pooling and size estimation.
*/
diff --git a/node_modules/@legendapp/list/react-native.js b/node_modules/@legendapp/list/react-native.js
index eeb3fb0..5ebd8e9 100644
index eeb3fb0..33d6241 100644
--- a/node_modules/@legendapp/list/react-native.js
+++ b/node_modules/@legendapp/list/react-native.js
@@ -1497,7 +1497,15 @@ function getItemSize(ctx, key, index, data, useAverageSize, preferCachedSize, no
@@ -344,8 +346,24 @@ index eeb3fb0..5ebd8e9 100644
getFixedItemSize: useWrapIfItem(getFixedItemSize),
getItemType: useWrapIfItem(getItemType),
hasExternalScroll,
@@ -8549,6 +8618,15 @@ var LegendListInner = typedForwardRef(function LegendListInner2(props, forwarded
doMaintainScrollAtEnd(ctx);
}
}, [!!maintainScrollAtEndConfig]);
+ // Paseo patch: the chat fills its offscreen buffer after its initial reveal.
+ const previousDrawDistance = React2.useRef(drawDistance);
+ React2.useLayoutEffect(() => {
+ if (previousDrawDistance.current !== drawDistance) {
+ previousDrawDistance.current = drawDistance;
+ state.scrollForNextCalculateItemsInView = void 0;
+ calculateItemsInView(ctx);
+ }
+ }, [drawDistance]);
const previousAlwaysRenderIndices = React2.useRef(alwaysRenderIndices);
React2.useLayoutEffect(() => {
const previousPins = previousAlwaysRenderIndices.current;
diff --git a/node_modules/@legendapp/list/react-native.mjs b/node_modules/@legendapp/list/react-native.mjs
index fe5f415..9788ce6 100644
index fe5f415..8219839 100644
--- a/node_modules/@legendapp/list/react-native.mjs
+++ b/node_modules/@legendapp/list/react-native.mjs
@@ -1476,7 +1476,15 @@ function getItemSize(ctx, key, index, data, useAverageSize, preferCachedSize, no
@@ -635,3 +653,19 @@ index fe5f415..9788ce6 100644
getFixedItemSize: useWrapIfItem(getFixedItemSize),
getItemType: useWrapIfItem(getItemType),
hasExternalScroll,
@@ -8528,6 +8597,15 @@ var LegendListInner = typedForwardRef(function LegendListInner2(props, forwarded
doMaintainScrollAtEnd(ctx);
}
}, [!!maintainScrollAtEndConfig]);
+ // Paseo patch: the chat fills its offscreen buffer after its initial reveal.
+ const previousDrawDistance = useRef(drawDistance);
+ useLayoutEffect(() => {
+ if (previousDrawDistance.current !== drawDistance) {
+ previousDrawDistance.current = drawDistance;
+ state.scrollForNextCalculateItemsInView = void 0;
+ calculateItemsInView(ctx);
+ }
+ }, [drawDistance]);
const previousAlwaysRenderIndices = useRef(alwaysRenderIndices);
useLayoutEffect(() => {
const previousPins = previousAlwaysRenderIndices.current;
+17 -13
View File
@@ -20,12 +20,15 @@ Arguments marked `?` are optional.
## Tabs
| Tool | Arguments | Purpose |
| ------------------- | -------------------------- | ------------------------------------------------------------------------- |
| `browser_list_tabs` | — | List open tabs in the agent's workspace across connected hosts. |
| `browser_new_tab` | `url?` | Open a tab in the background and return its `browserId`. |
| `browser_close_tab` | `browserId` | Close a tab and clean up its webview. |
| `browser_resize` | `browserId, width, height` | Resize the tab's viewport — check a layout at phone or tablet dimensions. |
| Tool | Arguments | Purpose |
| -------------------- | -------------------------- | -------------------------------------------------------------------------- |
| `browser_list_hosts` | — | List connected browser hosts by name and ID, including hosts with no tabs. |
| `browser_list_tabs` | — | List workspace tabs and their browser hosts. |
| `browser_new_tab` | `url?, browserHostId?` | Open a background tab and return its `browserId` and browser host. |
| `browser_close_tab` | `browserId` | Close a tab and clean up its webview. |
| `browser_resize` | `browserId, width, height` | Resize the tab's viewport — check a layout at phone or tablet dimensions. |
With one host that can open tabs, omit `browserHostId`. With several, pass an ID from `browser_list_hosts`. In Paseo Desktop, search for **Copy browser host name** in the Command Center. The result shows the same name the agent sees. Select it to copy the name. If the selected host is unavailable, list hosts again.
## Reading the page
@@ -69,10 +72,11 @@ Arguments marked `?` are optional.
Tools return structured errors rather than failing silently. The ones agents see most:
| Code | Meaning |
| ------------------- | ------------------------------------------------------------------------------- |
| `browser_disabled` | Browser tools are turned off on this host. |
| `browser_no_host` | No browser host (desktop app) is connected. Retryable. |
| `browser_stale_ref` | The ref no longer matches the page — take a new snapshot. |
| `browser_timeout` | The element never became actionable, or the wait condition never held. |
| `browser_denied` | The action isn't allowed — e.g. a non-`http(s)` URL, or no history to navigate. |
| Code | Meaning |
| ------------------------ | ------------------------------------------------------------------------------- |
| `browser_disabled` | Browser tools are turned off on this host. |
| `browser_no_host` | There is no connected browser host, or the selected host disconnected. |
| `browser_host_ambiguous` | Several hosts can open tabs. Choose one from the error and retry with its ID. |
| `browser_stale_ref` | The ref no longer matches the page — take a new snapshot. |
| `browser_timeout` | The element never became actionable, or the wait condition never held. |
| `browser_denied` | The action isn't allowed — e.g. a non-`http(s)` URL, or no history to navigate. |
+17 -13
View File
@@ -18,12 +18,15 @@ nav: 工具参考
## 标签页 {#tabs}
| 工具 | 参数 | 用途 |
| ------------------- | -------------------------- | ------------------------------------------------- |
| `browser_list_tabs` | — | 列出 Agent 工作区在所有已连接宿主上打开的标签页。 |
| `browser_new_tab` | `url?` | 在后台打开一个标签页并返回它的 `browserId`。 |
| `browser_close_tab` | `browserId` | 关闭标签页并清理它的 webview。 |
| `browser_resize` | `browserId, width, height` | 调整标签页视口大小,用手机或平板的尺寸检查布局。 |
| 工具 | 参数 | 用途 |
| -------------------- | -------------------------- | -------------------------------------------------------------- |
| `browser_list_hosts` | — | 按名称和 ID 列出已连接的浏览器宿主,包括尚未打开标签页的宿主。 |
| `browser_list_tabs` | — | 列出工作区中的标签页及其浏览器宿主。 |
| `browser_new_tab` | `url?, browserHostId?` | 在后台打开标签页,并返回它的 `browserId` 和浏览器宿主。 |
| `browser_close_tab` | `browserId` | 关闭标签页并清理它的 webview。 |
| `browser_resize` | `browserId, width, height` | 调整标签页视口大小,用手机或平板的尺寸检查布局。 |
如果只有一个宿主支持打开标签页,可以省略 `browserHostId`。如果有多个,请传入 `browser_list_hosts` 返回的 ID。在 Paseo Desktop 的命令中心搜索 **复制浏览器宿主名称**。结果会显示 Agent 看到的名称,选择该结果即可复制名称。如果选定的宿主不可用,请重新列出宿主。
## 读取页面 {#reading-the-page}
@@ -67,10 +70,11 @@ nav: 工具参考
工具会返回结构化的错误,而不是静默失败。Agent 最常遇到的错误:
| 错误码 | 含义 |
| ------------------- | --------------------------------------------------------------- |
| `browser_disabled` | 该主机上的浏览器工具已关闭。 |
| `browser_no_host` | 没有已连接的浏览器宿主(桌面应用)。可以重试。 |
| `browser_stale_ref` | 引用与页面不再匹配,请重新获取快照。 |
| `browser_timeout` | 元素始终无法操作,或等待条件始终未满足。 |
| `browser_denied` | 不允许该操作,例如非 `http(s)` 的 URL,或没有可导航的历史记录。 |
| 错误码 | 含义 |
| ------------------------ | ------------------------------------------------------------------------------ |
| `browser_disabled` | 该主机上的浏览器工具已关闭。 |
| `browser_no_host` | 没有已连接的浏览器宿主,或选定的宿主已断开连接。 |
| `browser_host_ambiguous` | 多个宿主支持打开标签页。请从错误返回的列表中选择一个宿主,并使用它的 ID 重试。 |
| `browser_stale_ref` | 引用与页面不再匹配,请重新获取快照。 |
| `browser_timeout` | 元素始终无法操作,或等待条件始终未满足。 |
| `browser_denied` | 不允许该操作,例如非 `http(s)` 的 URL,或没有可导航的历史记录。 |