PaddleOCR itself returns no color info, only text/bounds/confidence.
Add pixel-level post-processing in perception/ocr.py: crop the
screenshot to each OCR box, split pixels into two luminance clusters
via Otsu threshold, and treat the minority cluster as the text stroke
(foreground) and the majority as the background. New
SceneElement.foreground_color/background_color fields ("#rrggbb",
None when not OCR-sourced or sampling fails) round-trip through
to_dict/from_dict alongside the existing accessibility-state fields.
Planner system prompt documents the new fields as a secondary signal.
pillow is promoted from an implicit paddleocr transitive dependency to
an explicit direct dependency since perception/ocr.py now imports PIL
directly; uv.lock re-resolved with no version change (already locked
at 12.3.0).
Adds long_press/double_tap atomic gestures, a centralized humanize layer
(coordinate jitter, curved W3C-Actions swipe, timing jitter) gated by
APEX_HUMANIZE_ENABLED, and planner integration. 651 non-integration tests
pass on the branch.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- test_cancellation_full_path_queued_immediate_and_dispatched_collaborative
exercises the full public-API cancellation path: immediate cancel of a
queued task, collaborative cancel of a dispatched task surfaced through
lease renewal and a cancelled terminal report, and visibility of the
cancelled status via both the get and list endpoints.
- Full backend suite (869 passed, 50 skipped) and cloud-console frontend
suite (27 passed) + typecheck show no regressions; the only failures are
4 pre-existing live-LLM integration tests unrelated to this change.
- New internal API route POST /internal/v1/hosts/{host_id}/tasks/{task_id}/cancel,
authenticated via the host's own bearer credential (authorize_host) with an
ownership check, since host tokens carry no scopes and cannot reach the
public SDK's tasks:submit-scoped cancel endpoint.
- HostAgentClient.cancel_task() calls the new internal route directly.
- create_console_app() gains a cancel_task callable with automatic default
wiring from host_client, so production app.py needs no changes.
- Local console: POST /tasks/{task_id}/cancel route resolves the local
execution id to its Cloud source_task_id before cancelling, and the task
detail page/template show a Cancel button plus notice/error banners.
- Tests across all three layers: internal API route, Jinja2 template
rendering, and FastAPI console route behavior.
- POST /v1/tasks/{task_id}/cancel: tasks:submit scoped, 200 for
immediate/idempotent cancellation, 202 for newly recorded pending
cancellation, 404 for unknown task, 409 for terminal task.
- TaskCancellationResponse{task_id, status} model.
- Widen list_tasks status_filter Literal to include "cancelled".
- CloudClient.cancel_task(task_id).
- SDK-level tests covering queued/assigned/idempotent/404/409/scope
cases for both the router and CloudClient.
Task 5/9 of task-cancellation change.
- LeaseGuard gains an is_cancellation convenience property
- ActiveAssignmentRunner marks the lease lost with a cancellation
reason when a renewal response reports cancel_requested
- AssignmentExecutor threads stop_reason through to TaskRunner/
WorkflowRunner and maps a cancellation-flavored stop to
AssignmentExecutionResult.status = "cancelled" instead of "failed"
- AssignmentProcessor forwards a three-way done/cancelled/failed
status when reporting the terminal result
- Add/extend tests across lease, assignment, processor, and client
- LeaseRenewalResponse gains cancel_requested (populated from the
repository's renew_lease result)
- TerminalResultRequest.status widened to accept "cancelled"
- Add internal API tests for a renewal surfacing cancel_requested=True
and a cancelled terminal report being accepted/idempotent
SceneElement gains enabled/clickable/selected/checked/focused (bool | None),
populated from the literal attributes Appium's XCUITest and UiAutomator2
page_source already emit (iOS: enabled only; Android: all five). None means
"not reported by this platform", not false. to_dict() omits unset fields to
keep the LLM-facing scene JSON compact; planner_prompts.py documents the new
fields so the AI planner knows how to use them (e.g. don't tap disabled
elements, use selected/checked to judge whether a toggle already matches the
goal).
- Add nullable cancel_requested_at column (migration 0012)
- Widen ScheduledTaskStatus/TerminalTaskStatus to include cancelled
- Add CancellationRequestStatus + request_task_cancellation() to
CloudRepository protocol and SQLAlchemy implementation
- renew_lease() now returns LeaseRenewalResult, surfacing whether
cancellation is pending, instead of a bare status string
- reap_expired_leases() resolves pending-cancellation tasks to
cancelled instead of requeuing/failing them
- record_task_result() accepts cancelled and clears
cancel_requested_at on any terminal write
Note: internal_api/api.py's renew_assignment route still compares
renew_lease()'s return value against a bare string; it will be
updated in the next task (Internal Host<->Cloud protocol) to consume
LeaseRenewalResult and populate the new cancel_requested wire field.
- TaskRunner.run() and WorkflowRunner.run()/resume() accept an optional
stop_reason callable alongside should_stop, distinguishing a genuine
cancellation from other stop conditions (e.g. lost lease).
- is_cancellation_reason() shared helper added to runtime/task.py.
- WorkflowRunner._stop_status() now branches cancelled/failed based on
stop_reason, correcting a prior blanket cancelled-on-any-stop behavior
that conflicted with the host-agent-protocol spec's requirement to
distinguish cancellation from lease-loss stops.
- Default behavior (stop_reason=None) is preserved exactly for both
runners so existing callers/tests are unaffected.
- Task 1 of openspec change task-cancellation.
PaddleOCR's OCR.yaml pipeline defaults to use_doc_orientation_classify
and use_doc_unwarping enabled, which are meant for photographed paper
documents. Applied to a flat, upright device screenshot, UVDoc
geometrically warps the image before detection, and returns box
coordinates in that warped space with no inverse mapping back to the
original image.
Verified on a real screenshot: with unwarping on, the same detected
element ("新项目") shifts from y=158 to y=71 versus the original image,
and 2 boxes near the top edge (status bar time/battery) are dropped
entirely. Disabling both flags by default (still overridable via
explicit kwargs) makes detected boxes match the original screenshot.
Host-agent console showed OCR/UI-tree overlay boxes misaligned with the
displayed screenshot. Two independent causes, both confirmed with real
task data and pixel-level measurement of a user-provided screenshot:
1. perception/ui_parser.py parses XCUITest UI-tree bounds as iOS logical
points, while scene_builder.py's Scene.width/height (via infer_png_size)
and OCR bounds are in screenshot pixels, never reconciled (2.0x on
Retina devices). build_scene() now detects the scale from the first
x==0,y==0 UI element and rescales OCR bounds down to points-space,
reporting Scene.width/height in points too. No-op for Android, where
UiAutomator2 bounds already match pixels 1:1. This also fixes tap()
landing at the wrong location for OCR-matched text, and lets the IOU
fusion between UI-tree and OCR elements actually fire on iOS.
2. runtime/task.py captured `scene` (OCR/UI-tree data) before the LLM
planning call, but re-captured `before_screenshot` for each step
afterward - a real time gap during which on-screen content (e.g. a
keyboard) could shift, producing a directional drift between the
overlay and the displayed image. The first step of each plan batch
now reuses the screenshot already taken for planning instead of
capturing a new one; later steps in a multi-step batch still take a
fresh capture (left unresolved, scoped out by request).
Regression tests added for both the scale reconciliation (using real
828x1792 vs 414x896 numbers) and the screenshot reuse behavior.
Fixes issue 3: the host-agent console showed OCR results but never real
UI-tree data, because _ui_tree_nodes() checked for a get_ui_tree/ui_tree
tool action that has never existed anywhere in the codebase.
- storage/timeline.py: add a ui_tree_results field to TimelineRecord and
Timeline.append(), mirroring the existing ocr_results field.
- runtime/task.py: _append_timeline() now extracts scene.elements with
source == "ui" into ui_tree_results (scene_builder.build_scene() already
preserved these; they were just never persisted).
- host_agent/web/app.py: _ui_tree_nodes() reads the new field directly
instead of the dead tool-action check. New _overlay_payload() exposes
each step's scene dimensions and fused element list for client-side
rendering.
- task_detail.html: adds a toggle to overlay OCR (orange) and UI-tree
(blue) bounding boxes on the before-action screenshot, plus a visual
marker for the actually executed action (tap circle, or an animated
swipe path) using an SVG viewBox so no manual coordinate-scaling JS is
needed. Legacy/incomplete records degrade to no overlay, never an error.
Also corrects openspec/specs/runtime-task-evidence and
host-agent-console-task-pages, which had encoded the same nonexistent-tool
assumption, via the new host-agent-console-visual-evidence change.
600 tests passing; ruff/compileall/openspec validate all clean.
Forced tool_choice ("any"/"required") makes both Anthropic and OpenAI
skip any text/thinking block before the tool call, which silently made
rationale and thinking always None despite the planner-reflection-history
change's capture code being correct. Switch the primary call to
tool_choice="auto" (Anthropic: type=auto, disable_parallel_tool_use=true;
OpenAI: "auto") so the model can emit its reflection text, and add a
one-time forced retry (Anthropic "any", OpenAI "required", thinking
disabled) if the model responds without a tool call, guaranteeing a step
never stalls. Also add OpenAI text_output capture from message.content,
which was never extracted before (Anthropic-only gap).
Update planner-reflection-history design.md/tasks.md to document the bug
found during the pending manual smoke test (task 8.5) and the fix (new
section 9).
driver.tree() failures (WDA/Appium session errors) previously raised
uncaught, killing describe_screen() before OCR ever ran. Malformed
tree content (invalid XML) had the same problem inside
parse_ui_tree(). Both are now caught and logged, falling back to an
empty ui_elements list so the scene degrades to OCR-only, mirroring
the existing OCR-failure fallback in run_ocr().
Replaces the separate Vue/Vite `console/` SPA with a same-origin,
server-rendered console built on a module-level Jinja2 Environment
with select_autoescape(["html","xml"]).
- Add api/console_web.py with /ui/ routes (dashboard, tasks, task
detail/timeline, config) and a _status_fragment polled every 10s.
- Refactor api/console.py into a typed ConsoleService shared by the
JSON and HTML routers so validation/persistence cannot drift.
- Remove RUNTIME_CONSOLE_STATIC_DIR, SpaStaticFiles, and the wildcard
CORS middleware from api/rest.py; GET / now redirects to /ui/.
- Delete the top-level console/ project; add jinja2 and python-multipart
as direct dependencies and ship templates/CSS/JS via package-data.
- Add 31 tests (XSS probes, PRG flows, fragment refresh, no-static-dir
and no-CORS regressions, wheel-packaging smoke test).
/console/* JSON endpoints remain unchanged. The console keeps the
trusted-network-only boundary; auth/CSRF is intentionally deferred.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds CloudApiSkillClient (Cloud API per-host sync endpoint + inventory
report), forwards since_version for incremental sync (full-replace on
first/stale), and forks a local override into a standalone local skill
when its cloud skill is revoked (design D9). Skill-side tests green (87
passed).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds the merged read surface (api/skill_catalog_view.py) over synced +
local stores with origin discrimination and override precedence, and
extends the skill MCP tools with create_skill/update_skill/delete_skill
that dispatch by origin (edit local skills; create/update/remove local
overrides for cloud skills). Wired into api.mcp.create_mcp_server.
Full non-integration suite green (564 passed).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Opens the skill-management-console openspec change (cloud/local skill split
with local override) with proposal, design (D1-D11), four delta specs, and
tasks. Implements the agent-side persistent local skill store
(storage/local_skills.py): authored local skills + cloud-skill overrides in
a physically separate SQLite file, with fork-on-revocation. 10 tests pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Host Agent now persists step-level execution detail locally (via a real
TaskMetadataStore/Timeline wired into TaskRunner) and reports a bounded
in-progress snapshot piggybacked on lease renewal. Cloud persists that
snapshot per active assignment and exposes it through the existing task
list/detail query path; Cloud Console renders it as a live badge. Host
Agent's local console gains authenticated, read-only task list and
detail/timeline pages (same-origin, server-rendered) with inlined
screenshots.
Also fixes a pre-existing gap in the shared Timeline: the actual
per-step LLM prompt is now recorded instead of the task goal, benefiting
both Runtime and Host Agent consoles. When a host uses the cloud planner
transport, each decide call's prompt and resulting tool decision are
durably logged in a new planner_decision_log table (with bounded
retention) and browsable from Cloud Console; direct-transport hosts
explicitly surface a "not reported" state.
Includes Alembic migrations 0008 (progress columns on scheduled_tasks)
and 0009 (planner_decision_log), bounded Host-Agent-local retention,
dual-backend repository parity, and Vitest + pytest coverage. Task 6.5
(manual end-to-end device verification) remains.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements all 19 tasks of the cloud-planner-proxy OpenSpec change:
- Cloud API: cloud.planner_config (CloudPlannerConfig, load/build helpers)
reusing runtime.tool_calling_client provider clients (no new dependency
needed -- device-cloud-platform already depends on device-agent-runtime).
- Cloud API: new host-scoped POST /internal/v1/hosts/{host_id}/planner/decide
internal endpoint, reusing existing bearer auth; logs only metadata
(host id, tool name, latency, error class), never prompt/screenshot
content.
- Host Agent: new AI_PLANNER_TRANSPORT config (direct default | cloud) and
host_agent/cloud_planner_client.py::CloudProxyToolCallingClient, a
synchronous ToolCallingClient implementation (structural, not importing
runtime) that calls the new endpoint via its own httpx.Client -- avoids
bridging the async HostAgentClient across the worker-thread boundary
that AIPlanner.plan() runs in (asyncio.to_thread in lease.py).
- Host Agent wiring: create_execution_factories()/_host_agent_planner()
select the cloud-proxy client only when AI_PLANNER_TRANSPORT=cloud;
direct/unset transport is unchanged (still the default).
- Tests: 22 new tests across Cloud API config, the new endpoint, the new
client, and transport-selection wiring; full non-integration suite
(492 tests) passes with no regressions.
- Docs: docs/CLOUD_DEPLOYMENT.md documents the cloud transport, its
trade-offs, and the credential split between Host Agent and Cloud API.
proposal.md/design.md were corrected during implementation to reflect two
findings: no new anthropic/openai dependency is actually needed, and
CloudProxyToolCallingClient uses its own sync httpx.Client rather than a
new HostAgentClient method, per the thread-boundary reasoning above.
q792602257
