6 Commits

Include renames

Author	SHA1	Message	Date
q792602257	17a709c92f	fix(perception): disable PaddleOCR doc-unwarping for screenshots ... PaddleOCR's OCR.yaml pipeline defaults to use_doc_orientation_classify and use_doc_unwarping enabled, which are meant for photographed paper documents. Applied to a flat, upright device screenshot, UVDoc geometrically warps the image before detection, and returns box coordinates in that warped space with no inverse mapping back to the original image. Verified on a real screenshot: with unwarping on, the same detected element ("新项目") shifts from y=158 to y=71 versus the original image, and 2 boxes near the top edge (status bar time/battery) are dropped entirely. Disabling both flags by default (still overridable via explicit kwargs) makes detected boxes match the original screenshot.	2026-07-15 16:51:36 +08:00
q792602257	7f439f0db5	fix(perception): reconcile points/pixels scale and stale overlay screenshot ... Host-agent console showed OCR/UI-tree overlay boxes misaligned with the displayed screenshot. Two independent causes, both confirmed with real task data and pixel-level measurement of a user-provided screenshot: 1. perception/ui_parser.py parses XCUITest UI-tree bounds as iOS logical points, while scene_builder.py's Scene.width/height (via infer_png_size) and OCR bounds are in screenshot pixels, never reconciled (2.0x on Retina devices). build_scene() now detects the scale from the first x==0,y==0 UI element and rescales OCR bounds down to points-space, reporting Scene.width/height in points too. No-op for Android, where UiAutomator2 bounds already match pixels 1:1. This also fixes tap() landing at the wrong location for OCR-matched text, and lets the IOU fusion between UI-tree and OCR elements actually fire on iOS. 2. runtime/task.py captured `scene` (OCR/UI-tree data) before the LLM planning call, but re-captured `before_screenshot` for each step afterward - a real time gap during which on-screen content (e.g. a keyboard) could shift, producing a directional drift between the overlay and the displayed image. The first step of each plan batch now reuses the screenshot already taken for planning instead of capturing a new one; later steps in a multi-step batch still take a fresh capture (left unresolved, scoped out by request). Regression tests added for both the scale reconciliation (using real 828x1792 vs 414x896 numbers) and the screenshot reuse behavior.	2026-07-15 16:12:53 +08:00
q792602257	ccde30e378	feat(runtime): capture step evidence in console	2026-07-15 10:12:09 +08:00
q792602257	8d5b02e37f	fix(perception): degrade to OCR-only when UI tree is unavailable ... driver.tree() failures (WDA/Appium session errors) previously raised uncaught, killing describe_screen() before OCR ever ran. Malformed tree content (invalid XML) had the same problem inside parse_ui_tree(). Both are now caught and logged, falling back to an empty ui_elements list so the scene degrades to OCR-only, mirroring the existing OCR-failure fallback in run_ocr().	2026-07-15 10:03:30 +08:00
q792602257	6e511111c4	fix(perception): harden PaddleOCR result handling	2026-07-14 20:07:49 +08:00
q792602257	5658735bca	feat: checkpoint device agent runtime milestones	2026-07-06 17:24:03 +08:00