feat(perception): surface accessibility interaction state on UI-tree elements

SceneElement gains enabled/clickable/selected/checked/focused (bool | None),
populated from the literal attributes Appium's XCUITest and UiAutomator2
page_source already emit (iOS: enabled only; Android: all five). None means
"not reported by this platform", not false. to_dict() omits unset fields to
keep the LLM-facing scene JSON compact; planner_prompts.py documents the new
fields so the AI planner knows how to use them (e.g. don't tap disabled
elements, use selected/checked to judge whether a toggle already matches the
goal).
This commit is contained in:
2026-07-15 18:10:53 +08:00
parent 7c6cdc5b67
commit 361dada276
4 changed files with 142 additions and 8 deletions
+15
View File
@@ -10,6 +10,15 @@ list of UI elements with id, type, text, and pixel bounds), and — when
available — a screenshot of the same screen and a short history of recent
actions and their outcomes.
Some elements also carry accessibility state fields when the platform
reports them: `enabled`, `clickable`, `selected`, `checked`, `focused`.
A field is omitted entirely when the platform does not report it for that
element — omitted does NOT mean false, treat it as unknown. When present,
`enabled: false` or `clickable: false` means the element cannot currently be
interacted with (do not tap it); `selected`/`checked`/`focused` describe its
current toggle/focus state and are useful for deciding whether an action is
already done or still needed.
Before calling a tool, output a short text block (1-2 sentences):
1. If this is the first step, state what you intend to do and why.
2. Otherwise, first assess whether the previous action achieved its intended
@@ -22,6 +31,12 @@ You must then call exactly one tool:
- `finish_task` when the goal has been reached, or when it cannot be reached
and no further action would help.
For every device-action tool call, you must provide both required structured
fields in addition to the physical-action arguments:
- `purpose`: one concise sentence describing why this action advances the goal.
- `expected_outcome`: one concise, observable screen state expected after it.
These fields are used to verify and reuse successful actions; do not omit them.
Ground every coordinate you choose in the Scene element bounds (and the
screenshot, if provided) for the current turn only — never reuse coordinates
from history, since the screen may have changed. Only call `finish_task` with