Trailblaze Tools¶
Trailblaze ships a catalog of framework tools (tap, inputText, pressKey,
assertVisible, …) that cover most UI automation. When your app or test surface needs
something the catalog doesn’t have, you add custom tools to a workspace trailmap.
There are two flavors of custom tool:
- Scripted (TypeScript) tool — the recommended path for almost everyone. Declare
inputs and result shape as TypeScript interfaces and write the handler against the
typed
ctx.tools.<name>(args)surface. One.tsfile per tool; no per-tool YAML descriptor. This covers everything from a thin wrapper (“press back twice”) to a command with branching, retries, or async coordination. Every tool in the trailmap’stools/folder is on its target — notrailmap.yamlentry needed; add a name to the trailmap’sexports:to share it with trailmaps that depend on it.
Start with Scripted Tools (TypeScript) for the canonical authoring shape and the worked iOS Contacts + Wikipedia examples; see Trailmaps for the manifest wiring and the per-suffix file conventions.
- Kotlin-backed tool (
@TrailblazeToolClass-annotated class +*.tool.yamlinclass:mode) — advanced / host-side state only. Use when you need to read host-side state that the scripting SDK doesn’t expose today (a host process handle, a JVM-only library, a private internal API). Two files: the Kotlin class lives in your own Kotlin source tree; a sibling*.tool.yamldescriptor registers the FQN under the framework’s tool registry.
@Serializable
@TrailblazeToolClass("web_navigate")
@LLMDescription("Navigate the browser to a URL, or go back/forward in browser history.")
class PlaywrightNativeNavigateTool(
@param:LLMDescription("GOTO navigates to a URL, BACK/FORWARD moves through browser history.")
val action: NavigationAction = NavigationAction.GOTO,
@param:LLMDescription("The URL to navigate to. Required when action is GOTO.")
val url: String = "",
override val reasoning: String? = null,
) : PlaywrightExecutableTool, ReasoningTrailblazeTool {
override suspend fun executeWithPlaywright(
page: Page,
context: TrailblazeToolExecutionContext,
): TrailblazeToolResult { /* ... */ }
}
# trails/config/trailmaps/<id>/tools/web_navigate.tool.yaml (on the classpath alongside the class)
id: web_navigate
class: xyz.block.trailblaze.playwright.tools.PlaywrightNativeNavigateTool
id: must match the @TrailblazeToolClass name — that’s the dispatch identifier the
agent and the trailmap loader use. The @LLMDescription annotations on the class and
each parameter become the agent-facing schema (description + parameter docs); the
executable interface (PlaywrightExecutableTool, an Android equivalent, etc.) is
where the host-side work happens. Framework tools live under their owning
trails/config/trailmaps/<id>/tools/ directory — e.g. the PlaywrightNative*Tool
classes under trailblaze-playwright/src/main/java/xyz/block/trailblaze/playwright/tools/
ship their *.tool.yaml descriptors at trails/config/trailmaps/web/tools/. Workspace
authors can also drop a *.tool.yaml at <workspace>/trails/config/tools/ to override
or extend the framework set by id.
If you’re not sure which flavor fits, default to the scripted TypeScript path. Falling back to Kotlin is straightforward once you’ve hit a case where the typed authoring surface doesn’t reach.
A *.tool.yaml with a tools: list of other tool calls (the retired YAML-composed
flavor) no longer loads; write it as a TypeScript tool instead.
Branching in a recorded trail: runIf¶
A recording replays a fixed list of tool calls, so a step like “dismiss the welcome popup if it
appears” would replay as an unconditional tap, which fails on runs where the popup doesn’t show.
Wrap such steps in runIf. It is recorded as written and re-checks its condition on every replay:
- step: Dismiss the welcome popup if it appears
recording:
android-phone:
- runIf:
condition:
selector:
androidAccessibility:
textRegex: Not now
then:
- tapOnElementBySelector:
nodeSelector:
androidAccessibility:
textRegex: Not now
conditionis either{ selector: ... }, which is true when the selector matches something on screen, or{ tool: { <toolName>: <args> } }, which is true when that tool call succeeds. Conditions are pass or fail, not values: whatever the tool returns is ignored. A tool that runs and fails makes the condition false. A tool that cannot run at all (an unknown name, a misspelled argument) fails the step instead, so a typo never reads as “no”.negate: trueflips the condition.thenruns when the condition is true andelse, which is optional, when it is false. Both take a list of tool calls in the same shape as a recording, and either can contain anotherrunIf.- The whole
runIfis checked before anything runs, branches not taken included. An unknown key (els:), a malformed condition, or arunIfnested more than two deep fails the step up front.
Use runIf for one-off conditionals, so a check that only one trail needs doesn’t become a tool
in the shared catalog. Nesting a runIf inside another’s branch is allowed, two deep. Once the
same branch appears in a second trail, write a scripted tool
instead: it can be named, tested, and reused. runIf has no else if or switch.
runIf is available on every target for Android, iOS and web sessions. It is not available on
Compose or Revyl sessions, and a selector condition needs a view hierarchy, so drivers that only
capture screenshots must use a tool condition. The agent never sees runIf, so trailblaze
toolbox lists it only with --all. Write it into trail steps yourself, or call it with
trailblaze tool runIf. Scripted tools can call it too, via ctx.tools.runIf(...).
Checking what an element looks like: assertRegionColor¶
The accessibility tree says what is on screen, not what it looks like. “Dark is selected” and “the
app draws dark” are separate claims. assertRegionColor checks the second from the screen’s pixels,
with no model call, so it works in AI-off replays:
- verify: The settings screen renders dark
recording:
ios-ipad:
- assertRegionColor:
name: settings_main_pane
selector:
iosMaestro:
resourceIdRegex: ^settings_detail$
color: "#1C1C1E"
maxColorDifference: 6
minMatchPercent: 70
selectornames the region: the element’s on-screen rectangle. Pick a container, not a label inside it. It must match exactly one element.- It passes when at least
minMatchPercent(default 60) of the region’s pixels are withinmaxColorDifference(default 3) ofcolor. Every threshold is yours: a theme check is a colour check with your app’s own background colour. maxColorDifferenceis how far a pixel’s colour may be fromcolorand still match, on the CIEDE2000 colour-difference scale where 0 is identical. About 2 is the smallest difference a person notices, 3 looks like the same colour, and about 6 also absorbs the drift in a compressed screenshot. Black against a dark grey like#2A2A2Ais about 17, and black against white is 100. The result says when a read was compressed.maxMatchPercentturns it into an absence check: at most that share of the region may match.maxMatchPercent: 0with nominMatchPercentmeans none of the pixels read is that colour, e.g. a sticker that was dragged to the trash is gone. A region is read at most 1024 pixels on its longer side, so a mark thinner than that sampling step can fall between samples. Set both for a band.- It re-reads the screen until it passes or
timeoutMs(default 10000) runs out, so a repaint needs no sleep. - A region it can’t read fails the step as “can’t check”, never as a pass: no match or several, an element with no bounds, or a screen that is black everywhere.
- Every run saves a screenshot named after the tool and
namewhen the driver can take one (the result says when it could not). The result lists the region’s bounds, the measured percentage against its threshold, and the most common colours found, which is where to read your app’s colour from the first time.
The agent never sees this tool; write it into trail steps yourself, or call it with trailblaze tool. Scripted tools build their own
checks from the same building blocks: ctx.screen.image, the SDK’s colour helpers and checkRegion
(see checking pixels).
Launch arguments¶
ios_openApp treats values inside arguments as confidential and masks them in tool logs. The
real values still reach the app at launch; execution and transport serialization retain them.
Native iOS launch failures also mask arguments if the device transport repeats them in its error
output.
Use whole-value memory references for launch arguments that need to survive recording. Literal
and mixed-template arguments are masked in logs; recording generation refuses the whole session
recording because it cannot replay a redacted placeholder or safely omit the launch from a later
action sequence. A matching value elsewhere in authored text is masked only when it was also
authored literally in the launch argument.
For direct MCP step(tools: ...) on native iOS, the outer log cannot verify that a launch
reference resolved, so saving the interactive step may refuse the recording. Put the launch in
a trail or scripted tool when it must be replayed.
Nested framework calls are excluded from recordings while their tool logs remain available. Recordability alone does not hide data: keep credentials inside the sign-in tool and avoid printing them, returning them to the model, or storing them as ordinary memory.
Browsing the live tool catalog¶
Once you’ve added a tool — or you’re trying to figure out what’s already available for a given target — there are two ways to inspect the live tool catalog the agent sees:
- From the CLI:
trailblaze toolbox -d <platform> --target <trailmap-id>lists every tool the agent sees for that target, with descriptions. Add--search <substring>to filter (substring match on tool name and description). Add--allto also list tools the agent is not offered (surfaceToLlm: false, or dropped byexcluded_tools:), which hand-written trail steps andctx.tools.<name>can still call. - In your IDE: open any
.tsfile under<trailmap>/tools/and start typingctx.tools.— autocomplete pulls from the per-trailmap declaration rollup at<trailmap>/tools/trailblaze-client.d.ts. That file is emitted bytrailblaze check(and re-emitted on every daemon-aware command); if it’s missing, runtrailblaze checkonce.
Either surface includes framework built-ins resolved from the trailmap’s
platforms.<p>.tool_sets:, its own scripted tools, and any tools its dependencies
publish via their exports: field. There’s no global cross-trailmap aggregator —
per-trailmap slicing keeps autocomplete pollution out of your IDE and matches the
runtime dispatch surface one-to-one.