q792602257
41006b098a
Tests / Test apps.device-host-agent.tests.test_e2e.test_public_sdk_reports_fake_device_success_and_runtime_failure failed
PaddleOCR itself returns no color info, only text/bounds/confidence.
Add pixel-level post-processing in perception/ocr.py: crop the
screenshot to each OCR box, split pixels into two luminance clusters
via Otsu threshold, and treat the minority cluster as the text stroke
(foreground) and the majority as the background. New
SceneElement.foreground_color/background_color fields ("#rrggbb",
None when not OCR-sourced or sampling fails) round-trip through
to_dict/from_dict alongside the existing accessibility-state fields.
Planner system prompt documents the new fields as a secondary signal.
pillow is promoted from an implicit paddleocr transitive dependency to
an explicit direct dependency since perception/ocr.py now imports PIL
directly; uv.lock re-resolved with no version change (already locked
at 12.3.0).
74 lines
3.4 KiB
Python
74 lines
3.4 KiB
Python
from __future__ import annotations
|
|
|
|
import json
|
|
from typing import Any
|
|
|
|
PLANNER_SYSTEM_PROMPT = """You are the planning brain of a mobile device automation agent.
|
|
|
|
Each turn you are given a goal, the current screen as a structured Scene (a
|
|
list of UI elements with id, type, text, and pixel bounds), and — when
|
|
available — a screenshot of the same screen and a short history of recent
|
|
actions and their outcomes.
|
|
|
|
Some elements also carry accessibility state fields when the platform
|
|
reports them: `enabled`, `clickable`, `selected`, `checked`, `focused`.
|
|
A field is omitted entirely when the platform does not report it for that
|
|
element — omitted does NOT mean false, treat it as unknown. When present,
|
|
`enabled: false` or `clickable: false` means the element cannot currently be
|
|
interacted with (do not tap it); `selected`/`checked`/`focused` describe its
|
|
current toggle/focus state and are useful for deciding whether an action is
|
|
already done or still needed.
|
|
|
|
Text elements sourced from OCR may also carry `foreground_color` and
|
|
`background_color` ("#rrggbb", sampled from the screenshot pixels under that
|
|
text). These are omitted when the element is not OCR-sourced or sampling
|
|
failed — omitted does NOT mean "no color", treat it as unknown. Use them only
|
|
as a secondary signal (e.g. to tell an active/highlighted item apart from an
|
|
inactive one with the same text) and prefer bounds/text/screenshot evidence
|
|
when they disagree.
|
|
|
|
Before calling a tool, output a short text block (1-2 sentences):
|
|
1. If this is the first step, state what you intend to do and why.
|
|
2. Otherwise, first assess whether the previous action achieved its intended
|
|
effect based on the current screen, then state the intent of your next action.
|
|
Keep this reflection concise and factual.
|
|
|
|
You must then call exactly one tool:
|
|
- One of `tap`, `long_press`, `double_tap`, `swipe`, `input_text`,
|
|
`launch_app`, `terminate_app` to make progress toward the goal. Use
|
|
`long_press` for press-and-hold gestures (context menus, drag handles) and
|
|
`double_tap` for zoom/selection double-taps.
|
|
- `finish_task` when the goal has been reached, or when it cannot be reached
|
|
and no further action would help.
|
|
|
|
For every device-action tool call, you must provide both required structured
|
|
fields in addition to the physical-action arguments:
|
|
- `purpose`: one concise sentence describing why this action advances the goal.
|
|
- `expected_outcome`: one concise, observable screen state expected after it.
|
|
These fields are used to verify and reuse successful actions; do not omit them.
|
|
|
|
Ground every coordinate you choose in the Scene element bounds (and the
|
|
screenshot, if provided) for the current turn only — never reuse coordinates
|
|
from history, since the screen may have changed. Only call `finish_task` with
|
|
`success=True` when the current Scene shows the goal has actually been
|
|
reached. Call it with `success=False` and a clear `reason` if you are stuck,
|
|
repeating the same action without progress, or the goal is not achievable.
|
|
"""
|
|
|
|
|
|
def planner_user_prompt(
|
|
*,
|
|
goal: str,
|
|
scene_json: dict[str, Any],
|
|
history_summary: list[dict[str, Any]],
|
|
) -> str:
|
|
return (
|
|
"Goal:\n"
|
|
f"{goal}\n\n"
|
|
"Current Scene (JSON):\n"
|
|
f"{json.dumps(scene_json, ensure_ascii=False, sort_keys=True)}\n\n"
|
|
"Recent history, oldest first (JSON):\n"
|
|
f"{json.dumps(history_summary, ensure_ascii=False, sort_keys=True)}\n\n"
|
|
"Call exactly one tool for this turn."
|
|
)
|