Commit Graph
86 Commits
Author SHA1 Message Date
showtan001 71fd182f50 Support multimodal images in local chat agent 2026-08-29 21:17:20 +08:00
showtan001 a315c62f3a Improve local agent chat and device health detection 2026-08-29 16:03:15 +08:00
showtan001 050d1329c4 Add local host mode and configurable LLM providers 2026-08-24 08:49:18 +08:00
q792602257andClaude Opus 4.6 70e0624a47 fix(host-agent): align MCP integration with mcp SDK 1.28.1 realities
Three final-review deviations closed:

I1 (session-end release): mcp SDK 1.28.1 exposes no per-session
shutdown callback (only a server-level lifespan). Lower the
McpBusyTracker default TTL from 60s to 20s and update spec §6.5,
Q5/R3, D9, and docs/MCP_INTEGRATION.md concurrency section to
document the TTL-only recovery path. 20s is short enough to recover
within one 30s heartbeat interval but long enough that an active
session does not lose its lease during normal operator pauses.

I2 (JSON-RPC error shape): FastMCP Tool.run wraps every non-
UrlElicitationRequiredError exception (including McpError with typed
ErrorData) into ToolError, which the lowlevel call_tool handler
serializes as CallToolResult(isError=true, content=[TextContent(...)]).
There is no public path that surfaces JSON-RPC -32000 with structured
data.busy_owner from a tool call site. Update spec §7 error matrix
and docs/MCP_INTEGRATION.md error table to document the actual wire
shape; busy_owner now lives in the text content.

I3 (typing): mcp_server: Any = None -> FastMCP | None = None via
TYPE_CHECKING, keeping the mcp import lazy (matches precedent
elsewhere in the codebase) while adding static type checking at the
create_console_app boundary.

Tests added (4):
- test_default_ttl_is_20_seconds — locks I1's new default TTL
- test_default_ttl_recovers_dead_session_within_one_window — locks
  I1's recovery semantics (lease sweeped on next read after 20s)
- test_busy_error_wire_shape_is_calltoolresult_iserror — pins I2's
  wire envelope via Tool.run + lowlevel Server._make_error_result
- test_busy_error_text_includes_cloud_assignment_owner — same for
  the cloud_assignment busy_owner branch

Full non-integration suite: 697 passed / 54 deselected (was 693 / 54).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-21 16:25:35 +08:00
q792602257andClaude Opus 4.6 e69cea0245 style: ruff format after MCP server integration
Reformat the files touched by Tasks 1-14 of the host-agent MCP server
plan. No semantic changes; pre-existing format issues in unrelated
files (test_templates, test_skill_sync_wiring, 0010_skill_management,
test_skill_catalog_mcp) left untouched for a separate housekeeping
pass.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-21 15:52:32 +08:00
q792602257andClaude Opus 4.6 2d0c740c88 feat(host-agent): add mcp-token CLI subcommand
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-21 15:21:32 +08:00
q792602257 ce64c4eb47 feat(host-agent): wire MCP server into create_application 2026-07-21 15:17:28 +08:00
q792602257 ce2469616e feat(host-agent): mount /mcp + surface MCP status in console 2026-07-21 15:09:41 +08:00
q792602257andClaude Opus 4.6 dcb4798408 feat(host-agent): fail-fast cloud assignment when MCP holds device
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-21 15:00:54 +08:00
q792602257 ab15218b27 feat(host-agent): include mcp_busy_device_ids in heartbeat payload 2026-07-21 14:56:21 +08:00
q792602257andClaude Opus 4.6 98089b6748 fix(host-agent): use stable ServerSession id for MCP lock identity
The previous _current_session_id() implementation tried to import a
non-existent get_context() helper, so the production code path always
fell through to the empty _TEST_SESSION_ID ContextVar — meaning every
MCP client shared the empty-string identity and there was no per-session
isolation in production.

Use Context.session (the long-lived ServerSession object) as the source
of identity. id(ctx.session) is stable across every tool call the same
client makes within a Streamable HTTP session, which is exactly what the
busy tracker needs to renew leases.

Wire FastMCP to inject the Context into the wrapper by setting
tool.context_kwarg = "ctx" after swapping tool.fn; wrap the swap in a
defensive try/except that surfaces a FastMcpSdkIncompatibilityError on
future SDK layout drift.

Add 4 tests covering the production path: stability across calls in the
same session, isolation between sessions, fallback to _TEST_SESSION_ID
when no Context is supplied, and verification that the registered tool
declares context_kwarg="ctx".

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-21 14:28:50 +08:00
q792602257 b73db01626 feat(host-agent): wrap tool_handlers with busy check + status mapping 2026-07-21 14:09:42 +08:00
q792602257 cf8affe4d7 feat(host-agent): add BearerAuthMiddleware for MCP server 2026-07-21 14:02:20 +08:00
q792602257 61c923b92b feat(host-agent): add McpBusyTracker for per-device session locks 2026-07-21 13:58:37 +08:00
q792602257 c7faee8da3 feat(host-agent): add McpTokenStore for MCP bearer token 2026-07-21 13:55:24 +08:00
q792602257 d1b0fffabb build(host-agent): add mcp as direct dependency 2026-07-21 13:49:17 +08:00
q792602257 358f4623ba feat(planner): add execution context to prompts
Tests / Test passed: 977
2026-07-16 10:25:20 +08:00
q792602257 059fb272bb test(device-host-agent): add conftest.py to disable humanize in tests
Tests / Test passed: 966
The root tests/conftest.py fixture disables APEX_HUMANIZE_ENABLED by default for deterministic assertions, but apps/device-host-agent/tests/ was outside its scope. test_e2e.py::test_public_sdk_reports_fake_device_success_and_runtime_failure failed because tap coordinates were jittered (2.67..., 3.37...) instead of exact (2, 3).

Add the same autouse fixture to apps/device-host-agent/tests/conftest.py so all tests in that directory inherit the deterministic behavior.
2026-07-15 21:19:20 +08:00
q792602257andClaude Opus 4.6 f64f98834f Merge branch 'worktree-gesture-humanize': gesture primitives + humanize
Tests / Test apps.device-host-agent.tests.test_e2e.test_public_sdk_reports_fake_device_success_and_runtime_failure failed
Adds long_press/double_tap atomic gestures, a centralized humanize layer
(coordinate jitter, curved W3C-Actions swipe, timing jitter) gated by
APEX_HUMANIZE_ENABLED, and planner integration. 651 non-integration tests
pass on the branch.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-15 20:11:05 +08:00
q792602257 bd6b7e64e2 Merge branch 'worktree-task-cancellation': task cancellation feature
Tests / Test passed: 926
# Conflicts:
#	packages/cloud-platform/cloud/schema.py
2026-07-15 19:39:28 +08:00
q792602257 18f053e64b Add Host Agent local console task cancellation (task-cancellation 7.1-7.3)
- New internal API route POST /internal/v1/hosts/{host_id}/tasks/{task_id}/cancel,
  authenticated via the host's own bearer credential (authorize_host) with an
  ownership check, since host tokens carry no scopes and cannot reach the
  public SDK's tasks:submit-scoped cancel endpoint.
- HostAgentClient.cancel_task() calls the new internal route directly.
- create_console_app() gains a cancel_task callable with automatic default
  wiring from host_client, so production app.py needs no changes.
- Local console: POST /tasks/{task_id}/cancel route resolves the local
  execution id to its Cloud source_task_id before cancelling, and the task
  detail page/template show a Cancel button plus notice/error banners.
- Tests across all three layers: internal API route, Jinja2 template
  rendering, and FastAPI console route behavior.
2026-07-15 19:13:40 +08:00
q792602257 8a73edf4db feat(driver): add Driver.double_tap via W3C actions 2026-07-15 19:12:10 +08:00
q792602257 c25ccb491d feat(driver): add W3C actions helper and Driver.swipe_path 2026-07-15 19:04:57 +08:00
q792602257 ff91bd4f70 feat(driver): add Driver.long_press on WDA and Android 2026-07-15 18:57:53 +08:00
q792602257 d3024b4810 feat(host-agent): stop assignment execution collaboratively on cancellation
- LeaseGuard gains an is_cancellation convenience property
- ActiveAssignmentRunner marks the lease lost with a cancellation
  reason when a renewal response reports cancel_requested
- AssignmentExecutor threads stop_reason through to TaskRunner/
  WorkflowRunner and maps a cancellation-flavored stop to
  AssignmentExecutionResult.status = "cancelled" instead of "failed"
- AssignmentProcessor forwards a three-way done/cancelled/failed
  status when reporting the terminal result
- Add/extend tests across lease, assignment, processor, and client
2026-07-15 18:28:21 +08:00
q792602257 d69be48f96 feat(planner): persist reusable action semantics
Tests / Test passed: 879
2026-07-15 18:14:28 +08:00
q792602257 7c6cdc5b67 feat(host-agent): persist OCR/UI overlay toggle preference
Tests / Test passed: 868
Save the "Show OCR/UI-tree bounding boxes" checkbox state to localStorage so it persists across page refreshes in the task detail view.
2026-07-15 18:01:10 +08:00
q792602257 c50ce1faec OCR不展开
Tests / Test tests.test_device_config.test_device_config_store_settings_get_set_and_defaults failed
2026-07-15 15:06:06 +08:00
q792602257 8162509158 feat(host-agent): persist UI-tree evidence and add overlay/action visualization
Tests / Test passed: 863
Fixes issue 3: the host-agent console showed OCR results but never real
UI-tree data, because _ui_tree_nodes() checked for a get_ui_tree/ui_tree
tool action that has never existed anywhere in the codebase.

- storage/timeline.py: add a ui_tree_results field to TimelineRecord and
  Timeline.append(), mirroring the existing ocr_results field.
- runtime/task.py: _append_timeline() now extracts scene.elements with
  source == "ui" into ui_tree_results (scene_builder.build_scene() already
  preserved these; they were just never persisted).
- host_agent/web/app.py: _ui_tree_nodes() reads the new field directly
  instead of the dead tool-action check. New _overlay_payload() exposes
  each step's scene dimensions and fused element list for client-side
  rendering.
- task_detail.html: adds a toggle to overlay OCR (orange) and UI-tree
  (blue) bounding boxes on the before-action screenshot, plus a visual
  marker for the actually executed action (tap circle, or an animated
  swipe path) using an SVG viewBox so no manual coordinate-scaling JS is
  needed. Legacy/incomplete records degrade to no overlay, never an error.

Also corrects openspec/specs/runtime-task-evidence and
host-agent-console-task-pages, which had encoded the same nonexistent-tool
assumption, via the new host-agent-console-visual-evidence change.

600 tests passing; ruff/compileall/openspec validate all clean.
2026-07-15 14:39:28 +08:00
q792602257 24992fc9fb fix test
Tests / Test passed: 851
2026-07-15 12:53:50 +08:00
q792602257 77d4813bb2 feat(host-agent): make execution history authoritative
Tests / Test failed: 2, passed: 830
2026-07-15 11:46:27 +08:00
q792602257 778af2da53 fix(cloud): align planner configuration and records
Tests / Test passed: 856
2026-07-15 09:43:14 +08:00
q792602257andClaude Opus 4.6 f8054cb58c chore(skills): docs + ruff format for skill-management-console
Tests / Test passed: 855
Documents Skill Management in CLOUD_DEPLOYMENT.md (cloud-skill store,
per-host entitlement, incremental sync, local authoring/override,
inventory report, skills:admin scope) and applies ruff check/format to
all touched modules. All tasks complete; full non-integration suite
green (593 passed) and openspec validate --strict passes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-15 08:10:20 +08:00
q792602257andClaude Opus 4.6 0d944ec97d feat(host-agent): wire skill sync + inventory report into app lifecycle
Adds host_agent/skill_sync.py (HostAgentSkillSync) which constructs the
synced + local skill stores, the Cloud API sync client, and the runner,
then drives them on the host-agent lifecycle: incremental per-host pull
into the synced catalog, fork-on-revocation, and a best-effort local-
skill inventory report to the Cloud (design D7). Wired into
create_application + run_async start/stop. Host-agent suite green
(219 passed).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-15 08:02:31 +08:00
q792602257andClaude Opus 4.6 cbfdb2ae39 feat(cloud): skill management + host-scoped sync REST endpoints
Adds the skills:admin router (cloud/sdk/skill_api.py) for cloud-skill CRUD
and per-host entitlement grant/revoke with CSRF/scope/audit, and a
host-scoped router serving incremental per-host sync deltas plus the
agent local-skill inventory report/readback. Both composed into the
Cloud API app. cloud-api suite green (46 passed).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-15 07:51:27 +08:00
q792602257andClaude Opus 4.6 52e442790a feat(cloud): cloud-managed skill store + per-host entitlement + sync versioning
Tests / Test passed: 819
Adds cloud/skills.py (domain + service), SQLAlchemy models and Alembic
migration 0010_skill_management (cloud_skills, cloud_skill_entitlements,
cloud_skill_sync_state, a per-host changelog, and host_skill_inventory),
and repository methods with a monotonic per-host entitlement_version that
drives correct incremental fetch_host_delta. cloud-api suite green (41
passed); HEAD_REVISION bumped to 0010.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-15 07:45:22 +08:00
q792602257 30f09b6268 feat(host-agent): default planner transport to cloud
Tests / Test passed: 794
2026-07-14 20:42:27 +08:00
q792602257 25ebc10a8a feat: downgrade Python baseline to 3.13 for PaddleOCR compatibility
Tests / Test passed: 789
paddlepaddle has no Python 3.14 (cp314) wheel on PyPI, so host-agent
deployments on 3.14 can never install it, causing OCR to fail at
runtime with RuntimeError. Pin the workspace to Python 3.13 across
all pyproject.toml files, the Docker base image, and the Jenkins CI
image; regenerate uv.lock against 3.13.

Also fixes a pre-existing Python-2-style `except X, Y:` syntax error
(invalid in all Python 3.x) in runtime/task.py and
packages/cloud-platform/cloud/{sql_repository,internal_api/api}.py,
introduced in 22d37ca9 and unrelated to this change's scope, which
blocked the full test suite from collecting on any interpreter
version.

openspec change: downgrade-python-3-13-paddleocr
2026-07-14 18:05:49 +08:00
q792602257 a883903b66 style(host-agent): format Console task submission files 2026-07-14 17:52:35 +08:00
q792602257 a46f7d02a1 Merge branch 'opsx/host-agent-console-task-submission' into master
Tests / Test passed: 789
Adds a CSRF-protected task submission form to the local Console Tasks page

# Conflicts:
#	apps/device-host-agent/host_agent/app.py
2026-07-14 17:05:37 +08:00
q792602257 fb09924835 feat(host-agent): add Console task submission with Host self-submission client
Adds a CSRF-protected task submission form to the local Console Tasks page
2026-07-14 17:02:41 +08:00
q792602257andClaude Sonnet 5 08cef7ca3c fix(host-agent): thread device manager into task runner observer/screenshot
create_task_runner() built TaskRunner's observer/screenshot_provider by
calling describe_screen(device_id)/take_screenshot(device_id) without
manager=, so both silently fell back to the process-global DEFAULT_MANAGER
singleton instead of the Host Agent's real, device-populated DeviceManager.
DEFAULT_MANAGER never has any device registered, so every task's first step
raised DeviceNotFoundError even though the console (which does pass
manager=) showed the same device as connected. Deterministic on every task,
independent of process count.

Add regression tests confirming both lambdas now resolve devices via the
configured manager; verified each fails with the original DeviceNotFoundError
symptom when the fix is reverted.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-14 16:35:48 +08:00
q792602257andClaude Opus 4.6 d00ada67a5 feat(host-agent): add single-instance lock to prevent duplicate-process dispatch races
Acquire an exclusive, non-blocking filelock on the identity state directory
as the first action of create_application(), before resolve_host_identity()
or any enrollment/heartbeat side effect. A second process against the same
identity_path exits immediately with InstanceAlreadyRunningError naming the
lock path; the lock releases automatically on any process exit (including
SIGKILL) via OS-level advisory locking, and explicitly during run_async()'s
shutdown finally block. filelock is promoted from transitive to direct
dependency (version unchanged at 3.29.7).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-14 15:49:30 +08:00
q792602257 99bde4febb test: align deployment and lease contracts
Tests / Test passed: 748
2026-07-14 13:53:00 +08:00
q792602257andClaude Opus 4.6 8381b3068a feat(host-agent): migrate local console to Jinja2 templates with autoescape
Tests / Test failed: 4, passed: 744
Replace hand-written f-string + html.escape() rendering in the Host Agent
local console with a module-level Jinja2 Environment configured with
select_autoescape(["html","xml"]). XSS safety now holds by mechanism
rather than per-call discipline — every operator-controlled field
(device name, connection_info, task summary, etc.) is escaped by the
engine uniformly.

Eight templates under host_agent/web/templates/ replace the former
_chrome(), _CSS, escape(), and per-page _xxx_body() helpers: base.html
(header/nav/CSS + {% block body %}), login, dashboard (with the polling
<script> preserved byte-identically inside {% raw %}), devices, account,
history, tasks_list, and task_detail. The task-list and task-detail
templates — added by the just-landed task-execution-progress-visibility
change — were also migrated here rather than left in f-string form,
since this change removes the shared helpers they depended on.

URLs, auth/session/CSRF semantics, redirects, and /api/status JSON are
unchanged. 15 new template tests cover render-smoke, XSS probing, script
byte-identity, and no-autoescape-bypass guards. Tasks 8.1-8.6 (manual
browser verification) remain.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-14 13:05:19 +08:00
q792602257andClaude Opus 4.6 ec261d57c2 feat: surface task execution progress across Host Agent and Cloud
Host Agent now persists step-level execution detail locally (via a real
TaskMetadataStore/Timeline wired into TaskRunner) and reports a bounded
in-progress snapshot piggybacked on lease renewal. Cloud persists that
snapshot per active assignment and exposes it through the existing task
list/detail query path; Cloud Console renders it as a live badge. Host
Agent's local console gains authenticated, read-only task list and
detail/timeline pages (same-origin, server-rendered) with inlined
screenshots.

Also fixes a pre-existing gap in the shared Timeline: the actual
per-step LLM prompt is now recorded instead of the task goal, benefiting
both Runtime and Host Agent consoles. When a host uses the cloud planner
transport, each decide call's prompt and resulting tool decision are
durably logged in a new planner_decision_log table (with bounded
retention) and browsable from Cloud Console; direct-transport hosts
explicitly surface a "not reported" state.

Includes Alembic migrations 0008 (progress columns on scheduled_tasks)
and 0009 (planner_decision_log), bounded Host-Agent-local retention,
dual-backend repository parity, and Vitest + pytest coverage. Task 6.5
(manual end-to-end device verification) remains.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-14 12:47:49 +08:00
q792602257 c049c3c1b1 fix(host-agent): distinguish connected devices from busy-with-task in console
Tests / Test failed: 2, passed: 693
DeviceManager marks a device "busy" as soon as an Appium/WDA session is
connected, which is unrelated to whether a task is currently executing on
it. The Host Agent's local console displayed this raw status, making
connected-but-idle devices look permanently busy. Cross-reference the
device id against AgentStatusTracker's current_assignment (already
tracked via mark_assignment_started/finished) to show "connected" unless
a task is actually running on that device.
2026-07-14 11:48:33 +08:00
q792602257andClaude Opus 4.6 75879c8a52 feat(host-agent): add optional Appium/Runtime supervisor
Tests / Test failed: 2, passed: 691
Adds an opt-in dependency supervisor inside the Host Agent that probes,
spawns, and restarts the two local processes the macOS single-machine
real-device workflow depends on: the Appium server (gates Driver.connect())
and the local Runtime API (local inspection). Default-off; gated by
HOST_AGENT_DEPENDENCY_SUPERVISOR_ENABLED plus per-dependency *_SUPERVISED
flags.

Mitigates the live-incident failure mode where forgetting to start Appium
silently keeps devices offline and tasks queued forever with no error
surfaced in Host Agent logs.

Behavior (per openspec change):
- Adopt-don't-fight: probe (TCP + dependency-specific HTTP health check)
  before spawn. Healthy listener → adopted (never killed/restarted).
  Unhealthy listener → port-conflict error, skip. No listener → spawn.
- Only supervisor-spawned processes are restarted on crash, with capped
  exponential backoff (1s/2s/4s/8s, capped at 30s) and a per-process-lifetime
  attempt ceiling (HOST_AGENT_DEPENDENCY_RESTART_MAX_ATTEMPTS, default 5).
- Spawn failures (e.g. missing executable) logged distinctly from crashes.
- Graceful stop terminates only spawned children; adopted processes untouched.
- Supervisor starts before the heartbeat loop's first connect_devices() pass
  and stops alongside existing heartbeat/console teardown.

Validation: ruff check + format clean, compileall clean, openspec validate
--strict valid. Non-integration suite 503 passed / 44 deselected / 2 failed
(both failures pre-existing from unrelated 03c7c30 LLM_PROVIDER_ENC_KEY;
verified by stashing this change). macOS real-device manual verification
(task 6.4) deferred to a macOS host.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-14 09:28:57 +08:00
q792602257 9250254dec feat(cloud): support Anthropic provider base URL
Tests / Test passed: 665
2026-07-14 07:58:38 +08:00
q792602257 8b7e5a2800 feat(host-agent): start local console by default
Tests / Test passed: 662
2026-07-14 07:31:31 +08:00