q792602257
8162509158
Tests / Test passed: 863
Fixes issue 3: the host-agent console showed OCR results but never real UI-tree data, because _ui_tree_nodes() checked for a get_ui_tree/ui_tree tool action that has never existed anywhere in the codebase. - storage/timeline.py: add a ui_tree_results field to TimelineRecord and Timeline.append(), mirroring the existing ocr_results field. - runtime/task.py: _append_timeline() now extracts scene.elements with source == "ui" into ui_tree_results (scene_builder.build_scene() already preserved these; they were just never persisted). - host_agent/web/app.py: _ui_tree_nodes() reads the new field directly instead of the dead tool-action check. New _overlay_payload() exposes each step's scene dimensions and fused element list for client-side rendering. - task_detail.html: adds a toggle to overlay OCR (orange) and UI-tree (blue) bounding boxes on the before-action screenshot, plus a visual marker for the actually executed action (tap circle, or an animated swipe path) using an SVG viewBox so no manual coordinate-scaling JS is needed. Legacy/incomplete records degrade to no overlay, never an error. Also corrects openspec/specs/runtime-task-evidence and host-agent-console-task-pages, which had encoded the same nonexistent-tool assumption, via the new host-agent-console-visual-evidence change. 600 tests passing; ruff/compileall/openspec validate all clean.
45 lines
5.4 KiB
Markdown
45 lines
5.4 KiB
Markdown
## 1. Timeline — persist UI-tree elements per step
|
|
|
|
- [x] 1.1 Add `ui_tree_results: list[dict[str, Any]] = field(default_factory=list)` to `TimelineRecord` in `storage/timeline.py`, mirroring `ocr_results`
|
|
- [x] 1.2 Add `ui_tree_results: list[dict[str, Any]] | None = None` parameter to `Timeline.append()`, stored the same way as `ocr_results`
|
|
- [x] 1.3 Add unit tests: `Timeline.append()` with `ui_tree_results` → record round-trips it; without it → record defaults to `[]`; a record serialized before this field existed → reads as `[]` without error
|
|
|
|
## 2. Runtime — extract UI-tree elements into the Timeline
|
|
|
|
- [x] 2.1 In `runtime/task.py::_append_timeline()`, compute `ui_tree_results = [element.to_dict() for element in scene.elements if element.source == "ui"]` and pass it to `self.timeline.append(..., ui_tree_results=ui_tree_results)`
|
|
- [x] 2.2 Add unit tests: a scene with `source == "ui"` elements → `ui_tree_results` populated in the appended record; a scene with no UI elements → `ui_tree_results` is `[]`; a scene with mixed `ui`/`ocr`/merged elements → only `source == "ui"` elements appear in `ui_tree_results` and only `source == "ocr"` (or fallback) elements appear in `ocr_results`
|
|
|
|
## 3. Host Agent console — render real UI-tree data
|
|
|
|
- [x] 3.1 Rewrite `_ui_tree_nodes()` in `apps/device-host-agent/host_agent/web/app.py` to read `record.get("ui_tree_results", [])` directly (list of dicts), removing the dead `tool_call.get("action") in {"get_ui_tree", "ui_tree"}` check
|
|
- [x] 3.2 Confirm `_timeline_step_context()` passes the corrected `ui_tree_nodes` through unchanged; no `task_detail.html` markup changes needed for the list view itself (existing `step.ui_tree_nodes` block already renders it)
|
|
- [x] 3.3 Add/update unit tests for `_ui_tree_nodes()`: a record with `ui_tree_results` → returns those dicts; a record without the field (legacy) → returns `[]`; a record with non-dict entries → those entries are filtered out
|
|
|
|
## 4. Overlay toggle — OCR/UI-tree bounding boxes on the before-screenshot
|
|
|
|
- [x] 4.1 Add `_overlay_payload(record)` to `app.py` exposing each step's `scene` screen dimensions (`width`/`height`) and its fused element list (`bounds`/`text`/`source`, straight from `scene.elements` — already the combined OCR+UI-tree view) as `step.overlay`; embedded per-step as an inline `<script type="application/json">` blob in `task_detail.html`
|
|
- [x] 4.2 Add a single page-level toggle control (checkbox `#overlay-boxes-toggle`) in `task_detail.html` that adds/removes a `show-boxes` class on every step's overlay `<svg>`
|
|
- [x] 4.3 Render each step's overlay as an inline `<svg viewBox="0 0 {scene.width} {scene.height}">` absolutely positioned over the before-screenshot `<img>`; boxes are drawn as `<rect>` elements directly in scene-pixel coordinates, so the SVG viewBox scaling handles image-size scaling natively — no manual JS scale computation or resize listener needed (simpler than the originally planned separate static JS file with manual scaling)
|
|
- [x] 4.4 OCR boxes (`source="ocr"`) and UI-tree boxes (`source="ui"`) get distinct stroke colors (orange vs. blue) via CSS classes
|
|
- [x] 4.5 Verified: `_overlay_payload()` degrades to `{"width": 0, "height": 0, "elements": []}` for a record with no/malformed `scene`; the template only emits the `<svg>` when `overlay.width`/`overlay.height` are truthy, so legacy records render the screenshot with no overlay markup at all
|
|
|
|
## 5. Action-effect visualization — tap marker / swipe path
|
|
|
|
- [x] 5.1 Inline JS in `task_detail.html` reads each step's already-embedded `tool_call.action`/`args`; draws an SVG `<circle>` marker for `tap`, or an SVG `<line>` plus an `<animateMotion>`-animated dot for `swipe`; any other action draws nothing
|
|
- [x] 5.2 Action markers are drawn in the same `<svg viewBox>` as the overlay boxes (task 4.3), so no separate scaling logic was needed
|
|
- [x] 5.3 Verified: `buildActionGroup()` checks `isFinite()` on every required coordinate before drawing; missing/non-numeric args (or no `<svg>` at all, for a legacy record) render nothing, no error
|
|
- [x] 5.4 Rendered a real task-detail page (via the FastAPI test client with fabricated `tap`/`swipe` Timeline records) and inspected the generated HTML/JSON: `viewBox`, box coordinates, tap marker coordinates, and swipe line/path coordinates all match the source data; opened the page in a browser for visual confirmation
|
|
|
|
## 6. Spec corrections
|
|
|
|
- [x] 6.1 `openspec/specs/runtime-task-evidence/spec.md` (via this change's spec delta): UI-tree requirement no longer references a `get_ui_tree`/`ui_tree` tool call
|
|
- [x] 6.2 `openspec/specs/host-agent-console-task-pages/spec.md` (via this change's spec delta): UI-tree scenario corrected; overlay toggle and action-effect requirements added
|
|
|
|
## 7. End-to-end verification
|
|
|
|
- [x] 7.1 Run full non-integration test suite (`uv run --all-packages pytest -m "not integration"`) and confirm no regressions
|
|
- [x] 7.2 Run `ruff check` and `ruff format --check` on all modified Python files
|
|
- [x] 7.3 Run `python -m compileall` on modified packages
|
|
- [x] 7.4 Run `openspec validate --strict host-agent-console-visual-evidence` and confirm all artifacts pass validation
|
|
- [ ] 7.5 Manual smoke test (requires Host Agent + Appium + a device): run a live task, open its task-detail page, confirm UI-tree elements render, the overlay toggle shows/hides boxes correctly on the before-screenshot, and the tap/swipe action-effect marker/path renders correctly
|