feat(perception): surface accessibility interaction state on UI-tree elements
SceneElement gains enabled/clickable/selected/checked/focused (bool | None), populated from the literal attributes Appium's XCUITest and UiAutomator2 page_source already emit (iOS: enabled only; Android: all five). None means "not reported by this platform", not false. to_dict() omits unset fields to keep the LLM-facing scene JSON compact; planner_prompts.py documents the new fields so the AI planner knows how to use them (e.g. don't tap disabled elements, use selected/checked to judge whether a toggle already matches the goal).
This commit is contained in:
@@ -10,6 +10,15 @@ list of UI elements with id, type, text, and pixel bounds), and — when
|
||||
available — a screenshot of the same screen and a short history of recent
|
||||
actions and their outcomes.
|
||||
|
||||
Some elements also carry accessibility state fields when the platform
|
||||
reports them: `enabled`, `clickable`, `selected`, `checked`, `focused`.
|
||||
A field is omitted entirely when the platform does not report it for that
|
||||
element — omitted does NOT mean false, treat it as unknown. When present,
|
||||
`enabled: false` or `clickable: false` means the element cannot currently be
|
||||
interacted with (do not tap it); `selected`/`checked`/`focused` describe its
|
||||
current toggle/focus state and are useful for deciding whether an action is
|
||||
already done or still needed.
|
||||
|
||||
Before calling a tool, output a short text block (1-2 sentences):
|
||||
1. If this is the first step, state what you intend to do and why.
|
||||
2. Otherwise, first assess whether the previous action achieved its intended
|
||||
@@ -22,6 +31,12 @@ You must then call exactly one tool:
|
||||
- `finish_task` when the goal has been reached, or when it cannot be reached
|
||||
and no further action would help.
|
||||
|
||||
For every device-action tool call, you must provide both required structured
|
||||
fields in addition to the physical-action arguments:
|
||||
- `purpose`: one concise sentence describing why this action advances the goal.
|
||||
- `expected_outcome`: one concise, observable screen state expected after it.
|
||||
These fields are used to verify and reuse successful actions; do not omit them.
|
||||
|
||||
Ground every coordinate you choose in the Scene element bounds (and the
|
||||
screenshot, if provided) for the current turn only — never reuse coordinates
|
||||
from history, since the screen may have changed. Only call `finish_task` with
|
||||
|
||||
Reference in New Issue
Block a user