feat(perception): sample OCR text foreground/background colors
Tests / Test apps.device-host-agent.tests.test_e2e.test_public_sdk_reports_fake_device_success_and_runtime_failure failed

PaddleOCR itself returns no color info, only text/bounds/confidence.
Add pixel-level post-processing in perception/ocr.py: crop the
screenshot to each OCR box, split pixels into two luminance clusters
via Otsu threshold, and treat the minority cluster as the text stroke
(foreground) and the majority as the background. New
SceneElement.foreground_color/background_color fields ("#rrggbb",
None when not OCR-sourced or sampling fails) round-trip through
to_dict/from_dict alongside the existing accessibility-state fields.
Planner system prompt documents the new fields as a secondary signal.

pillow is promoted from an implicit paddleocr transitive dependency to
an explicit direct dependency since perception/ocr.py now imports PIL
directly; uv.lock re-resolved with no version change (already locked
at 12.3.0).
This commit is contained in:
2026-07-15 20:58:47 +08:00
parent f64f98834f
commit 41006b098a
6 changed files with 197 additions and 5 deletions
+8
View File
@@ -19,6 +19,14 @@ interacted with (do not tap it); `selected`/`checked`/`focused` describe its
current toggle/focus state and are useful for deciding whether an action is
already done or still needed.
Text elements sourced from OCR may also carry `foreground_color` and
`background_color` ("#rrggbb", sampled from the screenshot pixels under that
text). These are omitted when the element is not OCR-sourced or sampling
failed — omitted does NOT mean "no color", treat it as unknown. Use them only
as a secondary signal (e.g. to tell an active/highlighted item apart from an
inactive one with the same text) and prefer bounds/text/screenshot evidence
when they disagree.
Before calling a tool, output a short text block (1-2 sentences):
1. If this is the first step, state what you intend to do and why.
2. Otherwise, first assess whether the previous action achieved its intended