Agents

UiAgent lets an AI agent read and operate the components on screen, without screenshots. It is off until the app installs it. A snapshot lists each component with a ref, its kind, label, value, items, and the actions it allows now. Each action carries a JSON Schema for its value: a select's select action lists the option labels as an enum.

import wvlet.uni.dom.all.*
import wvlet.uni.dom.all.given
import wvlet.uni.rx.Rx
import wvlet.uni.ui.*
import scala.language.implicitConversions
import wvlet.uni.ui.agent.{AgentPolicy, UiAgent}

val agentRegion = Rx.variable("tokyo")

val form = div(
  Select(agentRegion, Seq("tokyo", "oregon"))(identity).withLabel("Region"),
  Button("Delete project").danger(UiAgent.confirm("Delete the project?")),
  Input(Rx.variable("sk-123")).withLabel("API key")(UiAgent.secret),
  div(UiAgent.hidden, Button("Billing"))
)

val agent = UiAgent.install(AgentPolicy.allowAll)
agent.snapshot()                                      // Seq[UiNode]: ref, kind, label, ...
agent.act("select:region", "select", Some("oregon")) // Rx[ActResult]
agent.changes(since = 0)                              // what people changed, and a cursor

• Actions. press (button, menu item), setText (text field), select (select, tabs, list, tree), sort (table), move (a list or table with .withReorder, as "Review to 1"), open/close (menu), and close (dialog). The agent's change goes through the component's onChange with ChangeSource.Agent, so a handler can cancel it, and act then reports canceled.
• Policy. AgentPolicy.allowAll (the default), AgentPolicy.readOnly, or AgentPolicy.allow(t => t.kind != "button"). Snapshots list only allowed actions, and a disabled component lists none.
• Marks. UiAgent.ref("name") sets a stable ref (otherwise kind:label). UiAgent.hidden removes a subtree; so do hidden, inert, and aria-hidden. UiAgent.secret shows the value as <redacted>, as every password field does. UiAgent.confirm(message) makes the agent wait while the human allows or denies the action in a dialog. Every other agent action is refused until they answer.
• Item labels. A list or tree names its items by .withTypeahead(label), and a table its rows by .withRowLabel(label); without one, an item is named by its key's toString.
• Changes. agent.changes(cursor) returns what people and app code changed since the cursor, without the agent's own changes, so an agent can catch up at the start of its turn. agent.subscribe(c => ...) sees every change as it happens.
• Transports. agent.api is the same API as JSON text (wvlet.uni.agent.UiAgentApi). MCPServer().withTools[UiAgentApi](agent.api) serves it as MCP tools, and agent.installWebMcp() registers it with the browser's WebMCP (document.modelContext) where the browser offers it. In Electron, ElectronRenderer.serve(RPCRouter.of[UiAgentApi](agent.api)) serves it to the main process, which can expose it as an MCP endpoint (see Calling the renderer from the main process).

Driving an Electron app over MCP

examples/uni-harness shows the whole path. pnpm dev:agent starts the app with --agent-bridge: the renderer installs UiAgent and serves agent.api to the main process with ElectronRenderer.serve, and the main process serves it as MCP tools (snapshot, query, act, changes) on http://127.0.0.1:<port>/mcp. It prints the URL, a bearer token, and a claude mcp add line to register the endpoint with an agent. The endpoint listens only on 127.0.0.1, rejects requests without the token, and picks a new port and token on each launch; plain pnpm dev starts no endpoint.

Testing with UiAgent

Tests can drive a view the same way an agent does: mount it with DomTestSupport, then call snapshot() and act.