Skip to navigation

Client tools and safety

How the coach acts in your page, exactly what it may do, and the guarantees that hold even if the model is tricked.

When the Onboarding coach points at a button or fills a field, it isn’t controlling the page from a server. It asks your page to do it, through the browser SDK, and the SDK decides. This page describes that protocol and the rules the SDK enforces. It’s written to be read by a security reviewer.

How a tool call travels

"Show me where to invite people" (voice) RPC rydhm.tool {callId, name: "highlight", args: {target: "Invite button"}} resolve the target, run the safety checks {ok: true, result: {...}} or {ok: false, error: "..."} "It's this button, top right." (only after ok) User Your page (@rydhm/web) Rydhm voice worker

The conversation runs in a LiveKit room. The voice worker calls the page with LiveKit RPC method rydhm.tool, waiting up to 10 seconds (30 seconds for click, because a risky click waits for the person’s answer):

{ "callId": "call_8f2…", "name": "fill", "args": { "target": "Workspace name", "value": "Acme" } }

The page answers:

{ "ok": true, "result": { "filled": true } }

or

{ "ok": false, "error": "That is a password field; I can't type there." }
  • Payloads are JSON, at most 15 KB each way. A larger result is shortened to { "truncated": true, "text": "…" }.
  • Each call answers inside the caller’s timeout, less a second of headroom; no answer means it didn’t happen.
  • Only the expert (an agent participant in the room) can call tools. Calls from anyone else are refused.
  • The coach never claims an action it didn’t get ok for. A tool.called event (name, ok, duration; never the arguments) is recorded for each call.

Built-in tools

Which ones an expert has is a setting of the expert: Page tools, in a coach’s Settings → Actions & tools and in other experts’ Advanced tab (toolPolicy.clientTools in the API). All of them except look are on for an Onboarding coach and off for other experts. look (screen vision, See charts and icons) is off for every expert until you turn it on (toolPolicy.screenVision: "on"), because it sends a picture of part of the person’s screen to a vision model.

ToolArgumentsWhat it does
navigate{ to }Opens a page of this site with your router, else pushState, else a page load. Other origins and non-http(s) links are refused.
highlight{ target, message? }Dims the page around the element, pulses a ring on it and shows message beside it. Goes away on the next highlight, a click, Escape, or after 15 seconds. Respects reduced motion.
scroll_to{ target } or { direction }Scrolls the element into view, or scrolls the page itself down, up, to the top or the bottom, in whatever area of it scrolls.
click{ target }Clicks it, asking the person first when it’s risky (see below).
fill{ target, value }Types into a field the way a person would, so React, Vue and Angular see the change. Never touches secret or payment fields.
read_screen{}What’s visible: headings, tabs, buttons, links, fields and lists, each with a target for the other tools. At most 4 KB, labelled as page content.
show_message{ text }Shows a short note in the bubble, for things better read than heard: a link, a code, a name.
press_key{ key, target? }Presses Enter, Escape, Tab, an arrow, Backspace or Space, on the target or on what has focus. A key that would submit a payment or delete asks first.
hover{ target }Moves the pointer over something to show what only appears on hover (a menu, a tooltip, row actions). Nothing is clicked.
drag{ target, to } or { target, dx, dy }Drags something to another place (reorders a list, moves a card to a column). A risky drop asks first.
look{ target?, question? }Screen vision, off unless switched on: a small picture (at most 768 px and 150 KB) of a chart, an image, a section, what’s in view, or the buttons that show only an icon (target: "icons"), drawn in the browser with no screen-sharing prompt. Typed values, password and payment fields, private areas and other sites’ content are blanked. A vision model describes it to the expert, as page data.

An Onboarding coach also has complete_step, which runs on the worker, not in your page: it marks a guide step done, only after it saw the step happen. Rydhm’s API decides whether a guide is complete; the page’s word is never enough.

Targets

target is resolved in this order:

  1. A page-map name from Rydhm (“Invite button”): its CSS selector, then its visible-text hint.
  2. A CSS selector (#invite, [data-test=invite]).
  3. text:<visible text>: the best match among visible, enabled, interactive elements. An exact match beats a prefix, which beats a substring. Hidden elements never match.

A target that doesn’t resolve is answered ok: false and reported as a target.missing event, so the page map shows it as Missing.

Custom tools

Your page adds its own with registerTool(name, { description, parameters }, handler). They’re offered to the model next to the built-ins:

  • Names: lowercase letters, digits and _, starting with a letter, 2 to 41 characters, and not a built-in’s name.
  • Up to 20 per page; each description up to 1,000 characters and each JSON Schema up to 4 KB.
  • The handler gets (args, { callId, signal }) and returns any JSON. Throwing answers ok: false with your error’s message (up to 500 characters).

See Give it your own tools.

WebMCP

WebMCP lets a page declare tools to the browser’s own AI agents. It’s a W3C Community Group draft that still changes, so the SDK feature-detects it and does nothing in a browser without it.

  • Your registerTool tools are WebMCP tools too. When the browser has document.modelContext (or navigator.modelContext), registerTool also registers the tool there, and the function it returns unregisters both.
  • Your WebMCP tools are the coach’s. During a conversation the SDK offers the page’s WebMCP tools to the expert as custom tools, run through the browser’s executeTool: up to 12, each with a description and an object schema of at most 4 KB. A tool marked destructive asks the person first, like a risky click.

Page context

The browser keeps two participant attributes current, and the worker reads them on every turn:

AttributeValueLimit
rydhm.page{ "url", "path", "title" }. The URL is sent without its query string; a #hash is kept only when it’s a hash route (#/settings).2 KB
rydhm.contextYour setContext JSON. Larger objects are cut and marked _truncated.4 KB
rydhm.toolsYour custom tools’ names, descriptions and schemas.8 KB
rydhm.guideThe active guide as the browser sees it, { id, stepsDone }. A hint only: Rydhm’s progress is the record.

Safety guarantees

These live in the SDK, in the browser, so text in your page or a prompt injection that reaches the model can’t turn them off.

  • Page text is data, never instructions. What read_screen returns is marked as page content and reaches the model inside a clearly marked block.
  • fill refuses password, hidden and file inputs; every payment autocomplete token (cc-*, transaction-*); current-password, new-password and one-time-code; fields whose name, id, label or placeholder say card, CVV or CVC, expiry, IBAN, routing or account number, SSN, password, PIN, OTP, API key or secret; every field in a form that collects card details; and disabled or read-only fields.
  • Risky clicks ask first. A click on an element whose text, aria-label, title, value, name, id, test id, link or form action says delete, remove, pay, buy, purchase, checkout, place order, cancel subscription, close account, transfer, refund, withdraw, unsubscribe, upgrade, downgrade, deactivate, disable, reset, revoke, disconnect or uninstall (and a few more), on a link to another website, or on the submit button of a payment or password form, shows a confirmation. It clicks only on Yes; no answer before the coach’s timeout means no click. Add data-rydhm-confirm to any element to always ask.
  • read_screen never reads the value of password, payment, one-time-code, hidden or private fields (it says protected: true).
  • data-rydhm-private on any element hides it and everything inside it from every tool: never read, clicked, filled or highlighted.
  • navigate stays on your origin.
  • Nothing is parsed as HTML. The bubble and the spotlight live in Shadow DOM and set every string with textContent.
  • Network traffic goes only to the API address, the LiveKit address the API returns, and the expert’s photo (served from Rydhm’s own hosts). No cookies, no external fonts, no third-party requests.
  • The publishable key works only from your allowed websites, and only for its one expert. It can start a conversation and report events; it can’t read your data or change any setting.

A good rule for your own tools: anything that spends money, deletes or sends something should ask the person in your UI before it acts, the same way the built-in click does.