Host-agent console showed OCR/UI-tree overlay boxes misaligned with the
displayed screenshot. Two independent causes, both confirmed with real
task data and pixel-level measurement of a user-provided screenshot:
1. perception/ui_parser.py parses XCUITest UI-tree bounds as iOS logical
points, while scene_builder.py's Scene.width/height (via infer_png_size)
and OCR bounds are in screenshot pixels, never reconciled (2.0x on
Retina devices). build_scene() now detects the scale from the first
x==0,y==0 UI element and rescales OCR bounds down to points-space,
reporting Scene.width/height in points too. No-op for Android, where
UiAutomator2 bounds already match pixels 1:1. This also fixes tap()
landing at the wrong location for OCR-matched text, and lets the IOU
fusion between UI-tree and OCR elements actually fire on iOS.
2. runtime/task.py captured `scene` (OCR/UI-tree data) before the LLM
planning call, but re-captured `before_screenshot` for each step
afterward - a real time gap during which on-screen content (e.g. a
keyboard) could shift, producing a directional drift between the
overlay and the displayed image. The first step of each plan batch
now reuses the screenshot already taken for planning instead of
capturing a new one; later steps in a multi-step batch still take a
fresh capture (left unresolved, scoped out by request).
Regression tests added for both the scale reconciliation (using real
828x1792 vs 414x896 numbers) and the screenshot reuse behavior.
Fixes issue 3: the host-agent console showed OCR results but never real
UI-tree data, because _ui_tree_nodes() checked for a get_ui_tree/ui_tree
tool action that has never existed anywhere in the codebase.
- storage/timeline.py: add a ui_tree_results field to TimelineRecord and
Timeline.append(), mirroring the existing ocr_results field.
- runtime/task.py: _append_timeline() now extracts scene.elements with
source == "ui" into ui_tree_results (scene_builder.build_scene() already
preserved these; they were just never persisted).
- host_agent/web/app.py: _ui_tree_nodes() reads the new field directly
instead of the dead tool-action check. New _overlay_payload() exposes
each step's scene dimensions and fused element list for client-side
rendering.
- task_detail.html: adds a toggle to overlay OCR (orange) and UI-tree
(blue) bounding boxes on the before-action screenshot, plus a visual
marker for the actually executed action (tap circle, or an animated
swipe path) using an SVG viewBox so no manual coordinate-scaling JS is
needed. Legacy/incomplete records degrade to no overlay, never an error.
Also corrects openspec/specs/runtime-task-evidence and
host-agent-console-task-pages, which had encoded the same nonexistent-tool
assumption, via the new host-agent-console-visual-evidence change.
600 tests passing; ruff/compileall/openspec validate all clean.
Forced tool_choice ("any"/"required") makes both Anthropic and OpenAI
skip any text/thinking block before the tool call, which silently made
rationale and thinking always None despite the planner-reflection-history
change's capture code being correct. Switch the primary call to
tool_choice="auto" (Anthropic: type=auto, disable_parallel_tool_use=true;
OpenAI: "auto") so the model can emit its reflection text, and add a
one-time forced retry (Anthropic "any", OpenAI "required", thinking
disabled) if the model responds without a tool call, guaranteeing a step
never stalls. Also add OpenAI text_output capture from message.content,
which was never extracted before (Anthropic-only gap).
Update planner-reflection-history design.md/tasks.md to document the bug
found during the pending manual smoke test (task 8.5) and the fix (new
section 9).
driver.tree() failures (WDA/Appium session errors) previously raised
uncaught, killing describe_screen() before OCR ever ran. Malformed
tree content (invalid XML) had the same problem inside
parse_ui_tree(). Both are now caught and logged, falling back to an
empty ui_elements list so the scene degrades to OCR-only, mirroring
the existing OCR-failure fallback in run_ocr().
Replaces the separate Vue/Vite `console/` SPA with a same-origin,
server-rendered console built on a module-level Jinja2 Environment
with select_autoescape(["html","xml"]).
- Add api/console_web.py with /ui/ routes (dashboard, tasks, task
detail/timeline, config) and a _status_fragment polled every 10s.
- Refactor api/console.py into a typed ConsoleService shared by the
JSON and HTML routers so validation/persistence cannot drift.
- Remove RUNTIME_CONSOLE_STATIC_DIR, SpaStaticFiles, and the wildcard
CORS middleware from api/rest.py; GET / now redirects to /ui/.
- Delete the top-level console/ project; add jinja2 and python-multipart
as direct dependencies and ship templates/CSS/JS via package-data.
- Add 31 tests (XSS probes, PRG flows, fragment refresh, no-static-dir
and no-CORS regressions, wheel-packaging smoke test).
/console/* JSON endpoints remain unchanged. The console keeps the
trusted-network-only boundary; auth/CSRF is intentionally deferred.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds CloudApiSkillClient (Cloud API per-host sync endpoint + inventory
report), forwards since_version for incremental sync (full-replace on
first/stale), and forks a local override into a standalone local skill
when its cloud skill is revoked (design D9). Skill-side tests green (87
passed).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds the merged read surface (api/skill_catalog_view.py) over synced +
local stores with origin discrimination and override precedence, and
extends the skill MCP tools with create_skill/update_skill/delete_skill
that dispatch by origin (edit local skills; create/update/remove local
overrides for cloud skills). Wired into api.mcp.create_mcp_server.
Full non-integration suite green (564 passed).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Opens the skill-management-console openspec change (cloud/local skill split
with local override) with proposal, design (D1-D11), four delta specs, and
tasks. Implements the agent-side persistent local skill store
(storage/local_skills.py): authored local skills + cloud-skill overrides in
a physically separate SQLite file, with fork-on-revocation. 10 tests pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Host Agent now persists step-level execution detail locally (via a real
TaskMetadataStore/Timeline wired into TaskRunner) and reports a bounded
in-progress snapshot piggybacked on lease renewal. Cloud persists that
snapshot per active assignment and exposes it through the existing task
list/detail query path; Cloud Console renders it as a live badge. Host
Agent's local console gains authenticated, read-only task list and
detail/timeline pages (same-origin, server-rendered) with inlined
screenshots.
Also fixes a pre-existing gap in the shared Timeline: the actual
per-step LLM prompt is now recorded instead of the task goal, benefiting
both Runtime and Host Agent consoles. When a host uses the cloud planner
transport, each decide call's prompt and resulting tool decision are
durably logged in a new planner_decision_log table (with bounded
retention) and browsable from Cloud Console; direct-transport hosts
explicitly surface a "not reported" state.
Includes Alembic migrations 0008 (progress columns on scheduled_tasks)
and 0009 (planner_decision_log), bounded Host-Agent-local retention,
dual-backend repository parity, and Vitest + pytest coverage. Task 6.5
(manual end-to-end device verification) remains.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements all 19 tasks of the cloud-planner-proxy OpenSpec change:
- Cloud API: cloud.planner_config (CloudPlannerConfig, load/build helpers)
reusing runtime.tool_calling_client provider clients (no new dependency
needed -- device-cloud-platform already depends on device-agent-runtime).
- Cloud API: new host-scoped POST /internal/v1/hosts/{host_id}/planner/decide
internal endpoint, reusing existing bearer auth; logs only metadata
(host id, tool name, latency, error class), never prompt/screenshot
content.
- Host Agent: new AI_PLANNER_TRANSPORT config (direct default | cloud) and
host_agent/cloud_planner_client.py::CloudProxyToolCallingClient, a
synchronous ToolCallingClient implementation (structural, not importing
runtime) that calls the new endpoint via its own httpx.Client -- avoids
bridging the async HostAgentClient across the worker-thread boundary
that AIPlanner.plan() runs in (asyncio.to_thread in lease.py).
- Host Agent wiring: create_execution_factories()/_host_agent_planner()
select the cloud-proxy client only when AI_PLANNER_TRANSPORT=cloud;
direct/unset transport is unchanged (still the default).
- Tests: 22 new tests across Cloud API config, the new endpoint, the new
client, and transport-selection wiring; full non-integration suite
(492 tests) passes with no regressions.
- Docs: docs/CLOUD_DEPLOYMENT.md documents the cloud transport, its
trade-offs, and the credential split between Host Agent and Cloud API.
proposal.md/design.md were corrected during implementation to reflect two
findings: no new anthropic/openai dependency is actually needed, and
CloudProxyToolCallingClient uses its own sync httpx.Client rather than a
new HostAgentClient method, per the thread-boundary reasoning above.
Host Agent:
- One-time local operator account bootstrap (PBKDF2-HMAC-SHA256, atomic
0600-permission write) gating the daemon's first unattended start via a
new `setup` CLI subcommand.
- Default control-plane URL now https://amcp.home.jerryyan.top (env var
override unchanged).
- Enrollment no longer requires a pre-issued token; falls back to
zero-token self-service enrollment when none is configured.
Cloud control plane:
- CLOUD_SELF_SERVICE_ENROLLMENT_ENABLED (default false) opt-in flag.
- SelfServiceEnrollmentAuthProvider + ChainedEnrollmentAuthProvider:
configured tokens still take priority; self-service only applies when
no token matches, preserving edge-host-enrollment's token-bound path.
- Fixed a latent bug in sql_repository.py::enroll_host: the token-conflict
lookup used `== enrollment_token_digest`, which SQLAlchemy compiles to
`IS NULL` when the value is None, so every self-service enrollment after
the first would have falsely collided with an existing NULL-digest host.
Skipped that lookup entirely when the digest is None.
Docs/deploy: .env.example, compose.yaml, compose.deploy.yaml,
CLOUD_DEPLOYMENT.md, MACOS_IPHONE_SETUP.md updated for the new flag,
URL default, and required `device-host-agent setup` step.
Verification: 494 non-integration tests pass; openspec validate --strict
passes. PostgreSQL-backed contract tests and full manual end-to-end
verification were not run (no Postgres/Docker or reachable cloud-api in
this environment); noted as unchecked in tasks.md 7.2/7.4.
Register `SUPPORTED_DRIVER_TYPES["uiautomator2"]` backed by the new
`AndroidDriver`, mirroring `WDADriver` method-for-method. tap/swipe/home use
the confirmed UiAutomator2 mobile commands (`clickGesture`, `dragGesture`,
`pressKey`); swipe converts `duration_ms` to a drag speed with a zero-distance
guard. Adds 34 mocked unit tests, an env-var-gated integration test, and
corrects the outdated "Android not registered" note in the iPhone setup doc.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements the cloud-console OpenSpec change: adds GET /v1/tasks (filterable,
bounded pagination, tasks:read) and GET /v1/tasks/{id}/attempts (404 on unknown
task) to the platform SDK, with matching CloudClient methods and a closed-by-
default CLOUD_CONSOLE_CORS_ORIGINS allow-list wired through CloudControlConfig.
Ships an independent Vue 3 + Vite SPA at cloud-console/ that authenticates with
an operator-supplied bearer token held in sessionStorage, renders tasks with
attempt history, device pool, host registry, and the plugin registry with a
registration form.
Backend test suite: 438 passed (-m "not integration"); cloud-console typecheck
and production build both succeed. PostgreSQL-backed repository tests and
manual end-to-end verification remain pending external infrastructure.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
q792602257
