Skip to content

TrailblazeNode — Type-Safe Driver-Specific View Hierarchy

Creating a type-safe abstraction over platform-specific UI trees.

Background

Trailblaze interacts with UI across four driver backends: Android via Maestro, Android via the native Accessibility Service, Web via Playwright, and Desktop via Compose. Each driver captures fundamentally different information about the UI.

The original shared model, ViewHierarchyTreeNode, was designed as a lowest-common-denominator representation that mirrors Maestro’s TreeNode. Every driver was forced to map its native data into this single shape, losing platform-specific information in the process:

  • Android Accessibility captures ~30 properties that ViewHierarchyTreeNode drops entirely: className, inputType, collectionItemInfo (list position), labeledByText, stateDescription, isEditable, isHeading, isCheckable, and more. These are the properties that would make element disambiguation dramatically more reliable.
  • Playwright naturally identifies elements by ARIA descriptor + occurrence index, not by integer node IDs. Forcing it through ViewHierarchyTreeNode requires an artificial mapping that doesn’t represent how Playwright actually works.
  • Compose uses semantic roles, test tags, and toggleable state — none of which map cleanly to ViewHierarchyTreeNode’s fields.

This forced normalization had a direct impact on selector quality. The TapSelectorV2 selector generator could only use properties available in ViewHierarchyTreeNode: text, resource ID, and a few boolean state flags. When five list items share the same text, it had to fall back to brittle strategies like index or complex hierarchy traversals, even though the native accessibility tree contained collectionItemInfo.rowIndex that would disambiguate instantly.

The key insight is that Maestro’s YAML-based selectors were designed for humans to write by hand, so they use a small, simple set of properties. Trailblaze’s selectors are generated by AI and computed programmatically — we can leverage arbitrarily rich property sets because the complexity is managed by the system, not the user.

What we decided

1. Introduce TrailblazeNode as the universal tree model

TrailblazeNode is a minimal data class with only truly universal properties:

data class TrailblazeNode(
  val nodeId: Long,
  val children: List<TrailblazeNode>,
  val bounds: Bounds?,
  val driverDetail: DriverNodeDetail,
)

There is no shared text, role, or isEnabled field. Those concepts mean different things on different platforms (Android’s className vs ARIA roles vs Compose semantic roles), and normalizing them loses information. All meaningful properties live in driverDetail.

2. DriverNodeDetail sealed interface with strongly-typed driver variants

Each driver has its own data class with full native properties:

sealed interface DriverNodeDetail {
  data class AndroidAccessibility(...) : DriverNodeDetail  // ~30 properties
  data class AndroidMaestro(...)       : DriverNodeDetail  // Maestro-compatible subset
  data class Web(...)                  : DriverNodeDetail  // ARIA + CSS selectors
  data class Compose(...)              : DriverNodeDetail  // Semantics + testTag
}

This is explicitly NOT a Map<String, Any>. Strongly-typed data classes provide: - Compile-time safety — no typos in property names, no wrong types - IDE autocompletion when writing matchers and generators - Exhaustive when matching ensures new drivers are handled everywhere - Serialization via kotlinx.serialization for recording persistence

3. Property matchability annotations

Each property in DriverNodeDetail is documented as either matchable or display-only:

  • Matchable properties are stable across runs and safe for recorded selectors (e.g., className, resourceId, text, labeledByText, collectionItemInfo).
  • Display-only properties are transient and useful for the LLM to recognize elements but must not appear in recordings (e.g., error, isVisibleToUser, isShowingHintText, drawingOrder).

Each variant exposes a matchablePropertyNames: Set<String> for programmatic access, ensuring selector generators only use stable properties.

4. TrailblazeNodeSelector with driver-specific matchers

A new selector model parallels the DriverNodeDetail sealed hierarchy:

data class TrailblazeNodeSelector(
  val driverMatch: DriverNodeMatch?,     // Driver-specific property matching
  val above: TrailblazeNodeSelector?,    // Spatial relationships
  val below: TrailblazeNodeSelector?,
  val leftOf: TrailblazeNodeSelector?,
  val rightOf: TrailblazeNodeSelector?,
  val childOf: TrailblazeNodeSelector?,  // Hierarchy
  val containsChild: TrailblazeNodeSelector?,
  val containsDescendants: List<TrailblazeNodeSelector>?,
  val index: Int?,                       // Last resort
)

DriverNodeMatch mirrors the sealed hierarchy — each driver has its own matcher that can match on all matchable properties:

sealed interface DriverNodeMatch {
  data class AndroidAccessibility(
    val classNameRegex: String?,
    val textRegex: String?,
    val labeledByTextRegex: String?,
    val collectionItemRowIndex: Int?,
    val inputType: Int?,
    // ... all matchable properties
  ) : DriverNodeMatch
  // ... Web, Compose, AndroidMaestro
}

5. Selector generation via cascading strategies

TrailblazeNodeSelectorGenerator computes the simplest selector that uniquely identifies a target node, trying strategies from most stable to most brittle:

For Android Accessibility (11 strategies): 1. Unique stable ID (uniqueId or resourceId) 2. Text alone 3. Text + className (e.g., “Fries” in a TextView vs EditText) 4. labeledByText (form fields: “the input labeled Email”) 5. labeledByText + className 6. className + state flags (editable, checkable, heading) 7. childOf unique parent 8. collectionItemInfo (semantic list position) 9. containsChild (unique child content) 10. Text + childOf parent 11. Index (positional fallback)

The generator verifies each candidate selector against the resolver to confirm it produces exactly one match before returning it.

6. Coexistence with existing Maestro path

The existing production pipeline is untouched: - ViewHierarchyTreeNode stays as-is for Maestro-based paths - TrailblazeElementSelector stays as-is for Maestro selector matching - TapSelectorV2 stays as-is for Maestro selector generation - AccessibilityElementResolver stays as-is for existing accessibility playback

TrailblazeNode and TrailblazeNodeSelector are new, parallel systems. Recordings can store either selector type, with the driver determining which to use.

File Layout

trailblaze-models/src/commonMain/.../api/
  TrailblazeNode.kt                  — Universal tree node
  DriverNodeDetail.kt                — Sealed interface (4 driver variants)
  TrailblazeNodeSelector.kt          — Rich selector model
  TrailblazeNodeSelectorResolver.kt  — Matches selectors against trees
  TrailblazeNodeSelectorGenerator.kt — Computes selectors for target nodes

trailblaze-accessibility/.../accessibility/
  TrailblazeNodeMapper.kt            — AccessibilityNode -> TrailblazeNode

What changed

Positive:

  • Selectors can now use className, inputType, collectionItemInfo, labeledByText, and 20+ other properties that were previously invisible — dramatically improving disambiguation for duplicate elements in lists, forms, and complex UIs.
  • Each driver keeps its native richness intact — no lossy normalization into a shared model that doesn’t fit any platform well.
  • Adding a new driver (e.g., iOS UIAutomation) means adding one DriverNodeDetail data class and one DriverNodeMatch data class. The resolver and selector model handle it automatically via sealed class exhaustive matching.
  • The matchability annotation system ensures recordings only capture stable properties, preventing flaky tests from transient state.
  • Type-safe sealed interfaces prevent the “string key” bugs that come with HashMap-based property bags.

Negative:

  • Two parallel view hierarchy models exist during migration (ViewHierarchyTreeNode for Maestro, TrailblazeNode for everything else). This is intentional — forcing migration would risk regressions in the Maestro path.
  • Each driver-specific matcher has its own matching function in the resolver, leading to some code duplication. This is the trade-off for type safety over a generic property-matching system.
  • Selector generation strategies are driver-specific, meaning each new driver needs its own strategy list. However, the spatial and hierarchy strategies are shared.