Commit Graph
224 Commits
Author SHA1 Message Date
q792602257 ab15218b27 feat(host-agent): include mcp_busy_device_ids in heartbeat payload 2026-07-21 14:56:21 +08:00
q792602257 b0932dd398 feat(cloud): skip MCP-busy devices in scheduler 2026-07-21 14:37:21 +08:00
q792602257 9e3007e7f6 feat(cloud): accept mcp_busy_device_ids in heartbeat payload 2026-07-21 14:32:44 +08:00
q792602257andClaude Opus 4.6 98089b6748 fix(host-agent): use stable ServerSession id for MCP lock identity
The previous _current_session_id() implementation tried to import a
non-existent get_context() helper, so the production code path always
fell through to the empty _TEST_SESSION_ID ContextVar — meaning every
MCP client shared the empty-string identity and there was no per-session
isolation in production.

Use Context.session (the long-lived ServerSession object) as the source
of identity. id(ctx.session) is stable across every tool call the same
client makes within a Streamable HTTP session, which is exactly what the
busy tracker needs to renew leases.

Wire FastMCP to inject the Context into the wrapper by setting
tool.context_kwarg = "ctx" after swapping tool.fn; wrap the swap in a
defensive try/except that surfaces a FastMcpSdkIncompatibilityError on
future SDK layout drift.

Add 4 tests covering the production path: stability across calls in the
same session, isolation between sessions, fallback to _TEST_SESSION_ID
when no Context is supplied, and verification that the registered tool
declares context_kwarg="ctx".

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-21 14:28:50 +08:00
q792602257 b73db01626 feat(host-agent): wrap tool_handlers with busy check + status mapping 2026-07-21 14:09:42 +08:00
q792602257 cf8affe4d7 feat(host-agent): add BearerAuthMiddleware for MCP server 2026-07-21 14:02:20 +08:00
q792602257 61c923b92b feat(host-agent): add McpBusyTracker for per-device session locks 2026-07-21 13:58:37 +08:00
q792602257 c7faee8da3 feat(host-agent): add McpTokenStore for MCP bearer token 2026-07-21 13:55:24 +08:00
q792602257 29b9a8c39a refactor(api): make tool_handlers require a DeviceManager
Eliminates the silent fallback to DEFAULT_MANAGER that produced the
DeviceNotFoundError incident. All existing callers already pass
manager explicitly.
2026-07-21 13:52:01 +08:00
q792602257 d1b0fffabb build(host-agent): add mcp as direct dependency 2026-07-21 13:49:17 +08:00
q792602257andClaude Opus 4.6 47eac0f2a7 docs(superpowers): add host-agent MCP server implementation plan
15-task TDD plan implementing the spec committed in 3b62195. Covers
the four new host-agent modules (mcp_token, mcp_lock, web/mcp_auth,
web/mcp), cloud heartbeat + scheduler coordination, console mount
wiring, CLI subcommand, and documentation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-21 13:47:15 +08:00
q792602257andClaude Opus 4.6 2f8f4a36c6 docs(superpowers): add host-agent MCP server design spec
Design for mounting a Streamable HTTP MCP server inside the host-agent
process so Hermes Agent (or any MCP client) can drive devices directly.
Reuses the existing console FastAPI + uvicorn on port 8765, adds bearer-
token auth, per-device session-level locks with 60s TTL, and cloud
coordination via a new heartbeat field. Cloud scheduler skips devices
reported as MCP-busy.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-21 13:47:15 +08:00
q792602257 358f4623ba feat(planner): add execution context to prompts
Tests / Test passed: 977
2026-07-16 10:25:20 +08:00
q792602257 240b7be7b8 fix(cloud-console): refresh task planner history 2026-07-16 09:35:31 +08:00
q792602257 9a17297f1e feat(perception): expose active app metadata in UI tree
Tests / Test passed: 971
2026-07-16 08:13:04 +08:00
q792602257 059fb272bb test(device-host-agent): add conftest.py to disable humanize in tests
Tests / Test passed: 966
The root tests/conftest.py fixture disables APEX_HUMANIZE_ENABLED by default for deterministic assertions, but apps/device-host-agent/tests/ was outside its scope. test_e2e.py::test_public_sdk_reports_fake_device_success_and_runtime_failure failed because tap coordinates were jittered (2.67..., 3.37...) instead of exact (2, 3).

Add the same autouse fixture to apps/device-host-agent/tests/conftest.py so all tests in that directory inherit the deterministic behavior.
2026-07-15 21:19:20 +08:00
q792602257 41006b098a feat(perception): sample OCR text foreground/background colors
Tests / Test apps.device-host-agent.tests.test_e2e.test_public_sdk_reports_fake_device_success_and_runtime_failure failed
PaddleOCR itself returns no color info, only text/bounds/confidence.
Add pixel-level post-processing in perception/ocr.py: crop the
screenshot to each OCR box, split pixels into two luminance clusters
via Otsu threshold, and treat the minority cluster as the text stroke
(foreground) and the majority as the background. New
SceneElement.foreground_color/background_color fields ("#rrggbb",
None when not OCR-sourced or sampling fails) round-trip through
to_dict/from_dict alongside the existing accessibility-state fields.
Planner system prompt documents the new fields as a secondary signal.

pillow is promoted from an implicit paddleocr transitive dependency to
an explicit direct dependency since perception/ocr.py now imports PIL
directly; uv.lock re-resolved with no version change (already locked
at 12.3.0).
2026-07-15 20:58:47 +08:00
q792602257andClaude Opus 4.6 f64f98834f Merge branch 'worktree-gesture-humanize': gesture primitives + humanize
Tests / Test apps.device-host-agent.tests.test_e2e.test_public_sdk_reports_fake_device_success_and_runtime_failure failed
Adds long_press/double_tap atomic gestures, a centralized humanize layer
(coordinate jitter, curved W3C-Actions swipe, timing jitter) gated by
APEX_HUMANIZE_ENABLED, and planner integration. 651 non-integration tests
pass on the branch.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-15 20:11:05 +08:00
q792602257 9da73cc6e3 fix(humanize): preserve gaussian magnitude in jitter_point; unify swipe return shape 2026-07-15 19:58:02 +08:00
q792602257 5a93651db7 feat(planner): expose long_press/double_tap to the AI planner 2026-07-15 19:45:06 +08:00
q792602257 bd6b7e64e2 Merge branch 'worktree-task-cancellation': task cancellation feature
Tests / Test passed: 926
# Conflicts:
#	packages/cloud-platform/cloud/schema.py
2026-07-15 19:39:28 +08:00
q792602257 dda70940c0 feat(tools): humanize swipe into curved W3C path when enabled 2026-07-15 19:38:07 +08:00
q792602257 865c163683 test(tools): assert tap humanize actually jitters coords 2026-07-15 19:33:46 +08:00
q792602257 85f0d6e188 feat(tools): humanize tap coordinates; default off in tests 2026-07-15 19:29:57 +08:00
q792602257 fd6365cf6e Add end-to-end cancellation path test and verify no regressions (task-cancellation 9.1-9.2)
- test_cancellation_full_path_queued_immediate_and_dispatched_collaborative
  exercises the full public-API cancellation path: immediate cancel of a
  queued task, collaborative cancel of a dispatched task surfaced through
  lease renewal and a cancelled terminal report, and visibility of the
  cancelled status via both the get and list endpoints.
- Full backend suite (869 passed, 50 skipped) and cloud-console frontend
  suite (27 passed) + typecheck show no regressions; the only failures are
  4 pre-existing live-LLM integration tests unrelated to this change.
2026-07-15 19:23:35 +08:00
q792602257andClaude Opus 4.6 d8e7be4ccb feat(tools): add double_tap tool with humanize hook
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-15 19:20:35 +08:00
q792602257andClaude Opus 4.6 c4ee4279ef feat(tools): add long_press tool with humanize hook
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-15 19:16:25 +08:00
q792602257 1dd24825ca Document task cancellation latency in CLOUD_DEPLOYMENT.md (task-cancellation 8.1) 2026-07-15 19:14:52 +08:00
q792602257 18f053e64b Add Host Agent local console task cancellation (task-cancellation 7.1-7.3)
- New internal API route POST /internal/v1/hosts/{host_id}/tasks/{task_id}/cancel,
  authenticated via the host's own bearer credential (authorize_host) with an
  ownership check, since host tokens carry no scopes and cannot reach the
  public SDK's tasks:submit-scoped cancel endpoint.
- HostAgentClient.cancel_task() calls the new internal route directly.
- create_console_app() gains a cancel_task callable with automatic default
  wiring from host_client, so production app.py needs no changes.
- Local console: POST /tasks/{task_id}/cancel route resolves the local
  execution id to its Cloud source_task_id before cancelling, and the task
  detail page/template show a Cancel button plus notice/error banners.
- Tests across all three layers: internal API route, Jinja2 template
  rendering, and FastAPI console route behavior.
2026-07-15 19:13:40 +08:00
q792602257 8a73edf4db feat(driver): add Driver.double_tap via W3C actions 2026-07-15 19:12:10 +08:00
q792602257 c25ccb491d feat(driver): add W3C actions helper and Driver.swipe_path 2026-07-15 19:04:57 +08:00
q792602257 ff91bd4f70 feat(driver): add Driver.long_press on WDA and Android 2026-07-15 18:57:53 +08:00
q792602257 7d79f677fe feat(humanize): add coordinate/duration/swipe-path jitter module 2026-07-15 18:51:15 +08:00
q792602257 4d04d7ac83 Add Cloud Console cancel action and cancelled status
- Widen TaskStatus to include "cancelled"; add it to TasksView's
  STATUSES filter dropdown.
- Add TaskCancellationResponse type and cancelTask(taskId) to api.ts.
- Add a Cancel button to TasksView's task detail panel, gated on
  tasks:submit and a non-terminal task status; updates the displayed
  status on success and surfaces errors via the existing error path.
- Extract the cancellability rule into a pure taskCancellation.ts
  module (mirroring taskProgress.ts/plannerHistory.ts) with unit
  tests, since the project has no Vue component-mounting test setup.

Task 6/9 of task-cancellation change.
2026-07-15 18:43:55 +08:00
q792602257 6776ac2f2d Add public SDK cancel endpoint and CloudClient method
- POST /v1/tasks/{task_id}/cancel: tasks:submit scoped, 200 for
  immediate/idempotent cancellation, 202 for newly recorded pending
  cancellation, 404 for unknown task, 409 for terminal task.
- TaskCancellationResponse{task_id, status} model.
- Widen list_tasks status_filter Literal to include "cancelled".
- CloudClient.cancel_task(task_id).
- SDK-level tests covering queued/assigned/idempotent/404/409/scope
  cases for both the router and CloudClient.

Task 5/9 of task-cancellation change.
2026-07-15 18:39:00 +08:00
q792602257 d3024b4810 feat(host-agent): stop assignment execution collaboratively on cancellation
- LeaseGuard gains an is_cancellation convenience property
- ActiveAssignmentRunner marks the lease lost with a cancellation
  reason when a renewal response reports cancel_requested
- AssignmentExecutor threads stop_reason through to TaskRunner/
  WorkflowRunner and maps a cancellation-flavored stop to
  AssignmentExecutionResult.status = "cancelled" instead of "failed"
- AssignmentProcessor forwards a three-way done/cancelled/failed
  status when reporting the terminal result
- Add/extend tests across lease, assignment, processor, and client
2026-07-15 18:28:21 +08:00
q792602257andClaude Opus 4.6 a58ded055e docs(superpowers): add gesture primitives + humanize implementation plan
9-task TDD plan: humanize module, Driver.long_press/swipe_path/double_tap
on WDA+Android (+ all Fake subclasses), tool wrappers, tap/swipe hooks,
and planner integration.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-15 18:26:34 +08:00
q792602257andClaude Opus 4.6 557c8a25ba docs(superpowers): add gesture primitives + humanize design spec
Design for long_press/double_tap atomic gestures and a centralized
humanize layer (coordinate jitter, curved W3C-Actions swipe, timing
jitter) gated by APEX_HUMANIZE_ENABLED.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-15 18:15:43 +08:00
q792602257 d69be48f96 feat(planner): persist reusable action semantics
Tests / Test passed: 879
2026-07-15 18:14:28 +08:00
q792602257 8a0d48eada feat(cloud): surface cancellation over the internal Host<->Cloud protocol
- LeaseRenewalResponse gains cancel_requested (populated from the
  repository's renew_lease result)
- TerminalResultRequest.status widened to accept "cancelled"
- Add internal API tests for a renewal surfacing cancel_requested=True
  and a cancelled terminal report being accepted/idempotent
2026-07-15 18:13:43 +08:00
q792602257 361dada276 feat(perception): surface accessibility interaction state on UI-tree elements
SceneElement gains enabled/clickable/selected/checked/focused (bool | None),
populated from the literal attributes Appium's XCUITest and UiAutomator2
page_source already emit (iOS: enabled only; Android: all five). None means
"not reported by this platform", not false. to_dict() omits unset fields to
keep the LLM-facing scene JSON compact; planner_prompts.py documents the new
fields so the AI planner knows how to use them (e.g. don't tap disabled
elements, use selected/checked to judge whether a toggle already matches the
goal).
2026-07-15 18:10:53 +08:00
q792602257 19c6669800 feat(cloud): add durable cancellation support to task repository
- Add nullable cancel_requested_at column (migration 0012)
- Widen ScheduledTaskStatus/TerminalTaskStatus to include cancelled
- Add CancellationRequestStatus + request_task_cancellation() to
  CloudRepository protocol and SQLAlchemy implementation
- renew_lease() now returns LeaseRenewalResult, surfacing whether
  cancellation is pending, instead of a bare status string
- reap_expired_leases() resolves pending-cancellation tasks to
  cancelled instead of requeuing/failing them
- record_task_result() accepts cancelled and clears
  cancel_requested_at on any terminal write

Note: internal_api/api.py's renew_assignment route still compares
renew_lease()'s return value against a bare string; it will be
updated in the next task (Internal Host<->Cloud protocol) to consume
LeaseRenewalResult and populate the new cancel_requested wire field.
2026-07-15 18:08:48 +08:00
q792602257 7c6cdc5b67 feat(host-agent): persist OCR/UI overlay toggle preference
Tests / Test passed: 868
Save the "Show OCR/UI-tree bounding boxes" checkbox state to localStorage so it persists across page refreshes in the task detail view.
2026-07-15 18:01:10 +08:00
q792602257 88189770ff feat(runtime): add cancellation-aware stop_reason to TaskRunner and WorkflowRunner
- TaskRunner.run() and WorkflowRunner.run()/resume() accept an optional
  stop_reason callable alongside should_stop, distinguishing a genuine
  cancellation from other stop conditions (e.g. lost lease).
- is_cancellation_reason() shared helper added to runtime/task.py.
- WorkflowRunner._stop_status() now branches cancelled/failed based on
  stop_reason, correcting a prior blanket cancelled-on-any-stop behavior
  that conflicted with the host-agent-protocol spec's requirement to
  distinguish cancellation from lease-loss stops.
- Default behavior (stop_reason=None) is preserved exactly for both
  runners so existing callers/tests are unaffected.
- Task 1 of openspec change task-cancellation.
2026-07-15 17:52:33 +08:00
q792602257 947434b65a docs(openspec): add task-cancellation proposal, design, specs, tasks 2026-07-15 17:38:02 +08:00
q792602257 a25542694d fix test
Tests / Test passed: 868
2026-07-15 17:04:59 +08:00
q792602257 17a709c92f fix(perception): disable PaddleOCR doc-unwarping for screenshots
Tests / Test tests.test_device_config.test_device_config_store_settings_get_set_and_defaults failed
PaddleOCR's OCR.yaml pipeline defaults to use_doc_orientation_classify
and use_doc_unwarping enabled, which are meant for photographed paper
documents. Applied to a flat, upright device screenshot, UVDoc
geometrically warps the image before detection, and returns box
coordinates in that warped space with no inverse mapping back to the
original image.

Verified on a real screenshot: with unwarping on, the same detected
element ("新项目") shifts from y=158 to y=71 versus the original image,
and 2 boxes near the top edge (status bar time/battery) are dropped
entirely. Disabling both flags by default (still overridable via
explicit kwargs) makes detected boxes match the original screenshot.
2026-07-15 16:51:36 +08:00
q792602257 4046c9452d fix(deps): pin paddlepaddle below 3.3.0 to avoid OCR text corruption
paddlepaddle 3.3.1 silently corrupts non-ASCII (CJK) recognized text
into literal U+FFFD replacement characters during rec postprocessing,
while leaving confidence scores high and ASCII/digit text unaffected.
The same release also breaks CPU oneDNN inference on Windows entirely
(NotImplementedError in onednn_instruction.cc). Verified on a real
task screenshot that downgrading to 3.2.x eliminates the corruption
with no other environment changes (same GBK-locale machine).
2026-07-15 16:41:58 +08:00
q792602257 7f439f0db5 fix(perception): reconcile points/pixels scale and stale overlay screenshot
Tests / Test tests.test_device_config.test_device_config_store_settings_get_set_and_defaults failed
Host-agent console showed OCR/UI-tree overlay boxes misaligned with the
displayed screenshot. Two independent causes, both confirmed with real
task data and pixel-level measurement of a user-provided screenshot:

1. perception/ui_parser.py parses XCUITest UI-tree bounds as iOS logical
   points, while scene_builder.py's Scene.width/height (via infer_png_size)
   and OCR bounds are in screenshot pixels, never reconciled (2.0x on
   Retina devices). build_scene() now detects the scale from the first
   x==0,y==0 UI element and rescales OCR bounds down to points-space,
   reporting Scene.width/height in points too. No-op for Android, where
   UiAutomator2 bounds already match pixels 1:1. This also fixes tap()
   landing at the wrong location for OCR-matched text, and lets the IOU
   fusion between UI-tree and OCR elements actually fire on iOS.

2. runtime/task.py captured `scene` (OCR/UI-tree data) before the LLM
   planning call, but re-captured `before_screenshot` for each step
   afterward - a real time gap during which on-screen content (e.g. a
   keyboard) could shift, producing a directional drift between the
   overlay and the displayed image. The first step of each plan batch
   now reuses the screenshot already taken for planning instead of
   capturing a new one; later steps in a multi-step batch still take a
   fresh capture (left unresolved, scoped out by request).

Regression tests added for both the scale reconciliation (using real
828x1792 vs 414x896 numbers) and the screenshot reuse behavior.
2026-07-15 16:12:53 +08:00
q792602257 c50ce1faec OCR不展开
Tests / Test tests.test_device_config.test_device_config_store_settings_get_set_and_defaults failed
2026-07-15 15:06:06 +08:00