SceneElement gains enabled/clickable/selected/checked/focused (bool | None),
populated from the literal attributes Appium's XCUITest and UiAutomator2
page_source already emit (iOS: enabled only; Android: all five). None means
"not reported by this platform", not false. to_dict() omits unset fields to
keep the LLM-facing scene JSON compact; planner_prompts.py documents the new
fields so the AI planner knows how to use them (e.g. don't tap disabled
elements, use selected/checked to judge whether a toggle already matches the
goal).
PaddleOCR's OCR.yaml pipeline defaults to use_doc_orientation_classify
and use_doc_unwarping enabled, which are meant for photographed paper
documents. Applied to a flat, upright device screenshot, UVDoc
geometrically warps the image before detection, and returns box
coordinates in that warped space with no inverse mapping back to the
original image.
Verified on a real screenshot: with unwarping on, the same detected
element ("新项目") shifts from y=158 to y=71 versus the original image,
and 2 boxes near the top edge (status bar time/battery) are dropped
entirely. Disabling both flags by default (still overridable via
explicit kwargs) makes detected boxes match the original screenshot.
paddlepaddle 3.3.1 silently corrupts non-ASCII (CJK) recognized text
into literal U+FFFD replacement characters during rec postprocessing,
while leaving confidence scores high and ASCII/digit text unaffected.
The same release also breaks CPU oneDNN inference on Windows entirely
(NotImplementedError in onednn_instruction.cc). Verified on a real
task screenshot that downgrading to 3.2.x eliminates the corruption
with no other environment changes (same GBK-locale machine).
Host-agent console showed OCR/UI-tree overlay boxes misaligned with the
displayed screenshot. Two independent causes, both confirmed with real
task data and pixel-level measurement of a user-provided screenshot:
1. perception/ui_parser.py parses XCUITest UI-tree bounds as iOS logical
points, while scene_builder.py's Scene.width/height (via infer_png_size)
and OCR bounds are in screenshot pixels, never reconciled (2.0x on
Retina devices). build_scene() now detects the scale from the first
x==0,y==0 UI element and rescales OCR bounds down to points-space,
reporting Scene.width/height in points too. No-op for Android, where
UiAutomator2 bounds already match pixels 1:1. This also fixes tap()
landing at the wrong location for OCR-matched text, and lets the IOU
fusion between UI-tree and OCR elements actually fire on iOS.
2. runtime/task.py captured `scene` (OCR/UI-tree data) before the LLM
planning call, but re-captured `before_screenshot` for each step
afterward - a real time gap during which on-screen content (e.g. a
keyboard) could shift, producing a directional drift between the
overlay and the displayed image. The first step of each plan batch
now reuses the screenshot already taken for planning instead of
capturing a new one; later steps in a multi-step batch still take a
fresh capture (left unresolved, scoped out by request).
Regression tests added for both the scale reconciliation (using real
828x1792 vs 414x896 numbers) and the screenshot reuse behavior.
Remove the 20-step execution limit that was causing "max steps exceeded" errors for long-running tasks. Increase the default max_steps to 999999 in all configurations, effectively removing the practical limit while maintaining the safety mechanism.
Changes:
- runtime/task.py: TaskRunnerConfig.max_steps 20 → 999999
- agents/collab_runner.py: CollaborativeTaskRunnerConfig.max_steps 20 → 999999
- storage/device_config.py: DEFAULT_MAX_STEPS 20 → 999999
Fixes issue 3: the host-agent console showed OCR results but never real
UI-tree data, because _ui_tree_nodes() checked for a get_ui_tree/ui_tree
tool action that has never existed anywhere in the codebase.
- storage/timeline.py: add a ui_tree_results field to TimelineRecord and
Timeline.append(), mirroring the existing ocr_results field.
- runtime/task.py: _append_timeline() now extracts scene.elements with
source == "ui" into ui_tree_results (scene_builder.build_scene() already
preserved these; they were just never persisted).
- host_agent/web/app.py: _ui_tree_nodes() reads the new field directly
instead of the dead tool-action check. New _overlay_payload() exposes
each step's scene dimensions and fused element list for client-side
rendering.
- task_detail.html: adds a toggle to overlay OCR (orange) and UI-tree
(blue) bounding boxes on the before-action screenshot, plus a visual
marker for the actually executed action (tap circle, or an animated
swipe path) using an SVG viewBox so no manual coordinate-scaling JS is
needed. Legacy/incomplete records degrade to no overlay, never an error.
Also corrects openspec/specs/runtime-task-evidence and
host-agent-console-task-pages, which had encoded the same nonexistent-tool
assumption, via the new host-agent-console-visual-evidence change.
600 tests passing; ruff/compileall/openspec validate all clean.
Forced tool_choice ("any"/"required") makes both Anthropic and OpenAI
skip any text/thinking block before the tool call, which silently made
rationale and thinking always None despite the planner-reflection-history
change's capture code being correct. Switch the primary call to
tool_choice="auto" (Anthropic: type=auto, disable_parallel_tool_use=true;
OpenAI: "auto") so the model can emit its reflection text, and add a
one-time forced retry (Anthropic "any", OpenAI "required", thinking
disabled) if the model responds without a tool call, guaranteeing a step
never stalls. Also add OpenAI text_output capture from message.content,
which was never extracted before (Anthropic-only gap).
Update planner-reflection-history design.md/tasks.md to document the bug
found during the pending manual smoke test (task 8.5) and the fix (new
section 9).
driver.tree() failures (WDA/Appium session errors) previously raised
uncaught, killing describe_screen() before OCR ever ran. Malformed
tree content (invalid XML) had the same problem inside
parse_ui_tree(). Both are now caught and logged, falling back to an
empty ui_elements list so the scene degrades to OCR-only, mirroring
the existing OCR-failure fallback in run_ocr().
Documents Skill Management in CLOUD_DEPLOYMENT.md (cloud-skill store,
per-host entitlement, incremental sync, local authoring/override,
inventory report, skills:admin scope) and applies ruff check/format to
all touched modules. All tasks complete; full non-integration suite
green (593 passed) and openspec validate --strict passes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds SkillsView.vue (cloud-skill CRUD, per-host entitlement grant/revoke,
read-only host local-skill inventory), skill API client methods + types,
and wires it into App.vue behind the skills:admin scope. Console
typecheck/build/tests green (20 passed).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replaces the separate Vue/Vite `console/` SPA with a same-origin,
server-rendered console built on a module-level Jinja2 Environment
with select_autoescape(["html","xml"]).
- Add api/console_web.py with /ui/ routes (dashboard, tasks, task
detail/timeline, config) and a _status_fragment polled every 10s.
- Refactor api/console.py into a typed ConsoleService shared by the
JSON and HTML routers so validation/persistence cannot drift.
- Remove RUNTIME_CONSOLE_STATIC_DIR, SpaStaticFiles, and the wildcard
CORS middleware from api/rest.py; GET / now redirects to /ui/.
- Delete the top-level console/ project; add jinja2 and python-multipart
as direct dependencies and ship templates/CSS/JS via package-data.
- Add 31 tests (XSS probes, PRG flows, fragment refresh, no-static-dir
and no-CORS regressions, wheel-packaging smoke test).
/console/* JSON endpoints remain unchanged. The console keeps the
trusted-network-only boundary; auth/CSRF is intentionally deferred.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds host_agent/skill_sync.py (HostAgentSkillSync) which constructs the
synced + local skill stores, the Cloud API sync client, and the runner,
then drives them on the host-agent lifecycle: incremental per-host pull
into the synced catalog, fork-on-revocation, and a best-effort local-
skill inventory report to the Cloud (design D7). Wired into
create_application + run_async start/stop. Host-agent suite green
(219 passed).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds CloudApiSkillClient (Cloud API per-host sync endpoint + inventory
report), forwards since_version for incremental sync (full-replace on
first/stale), and forks a local override into a standalone local skill
when its cloud skill is revoked (design D9). Skill-side tests green (87
passed).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds the skills:admin router (cloud/sdk/skill_api.py) for cloud-skill CRUD
and per-host entitlement grant/revoke with CSRF/scope/audit, and a
host-scoped router serving incremental per-host sync deltas plus the
agent local-skill inventory report/readback. Both composed into the
Cloud API app. cloud-api suite green (46 passed).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds cloud/skills.py (domain + service), SQLAlchemy models and Alembic
migration 0010_skill_management (cloud_skills, cloud_skill_entitlements,
cloud_skill_sync_state, a per-host changelog, and host_skill_inventory),
and repository methods with a monotonic per-host entitlement_version that
drives correct incremental fetch_host_delta. cloud-api suite green (41
passed); HEAD_REVISION bumped to 0010.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds the merged read surface (api/skill_catalog_view.py) over synced +
local stores with origin discrimination and override precedence, and
extends the skill MCP tools with create_skill/update_skill/delete_skill
that dispatch by origin (edit local skills; create/update/remove local
overrides for cloud skills). Wired into api.mcp.create_mcp_server.
Full non-integration suite green (564 passed).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Opens the skill-management-console openspec change (cloud/local skill split
with local override) with proposal, design (D1-D11), four delta specs, and
tasks. Implements the agent-side persistent local skill store
(storage/local_skills.py): authored local skills + cloud-skill overrides in
a physically separate SQLite file, with fork-on-revocation. 10 tests pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change is complete (24/24 tasks) per its declared scope (read-only local
catalog + MCP tools + sync client contract). Management UI and the
upstream Subscription Platform were explicitly out of scope. Deltas
synced into three new main specs: skill-catalog, skill-mcp-tools,
skill-subscription-sync.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change is complete (17/17 tasks). Deltas synced: MODIFIED the
cloud-planner-proxy "Endpoint resolves exactly one tool-call decision"
requirement to resolve provider config from the active database profile,
and created a new main spec openspec/specs/llm-provider-management/spec.md.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change is complete (20/20 tasks). Deltas synced: MODIFIED the
agent-runtime "Pluggable dual-provider tool-calling abstraction"
requirement (added transport selection), and created a new main spec
openspec/specs/cloud-planner-proxy/spec.md.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change is complete (22/22 tasks) and its delta spec has been synced
into a new main spec openspec/specs/agent-runtime/spec.md.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change is complete (17/17 tasks) and its delta spec has been synced
into a new main spec openspec/specs/workspace-packaging/spec.md.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change is complete (19/19 tasks) and its delta spec has been synced
into a new main spec openspec/specs/python-runtime-baseline/spec.md.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sync delta specs into main specs before archiving: modified
cloud-control-plane, device-pool, and host-agent-protocol; created
new edge-host-enrollment capability spec. openspec validate --specs
reports 18/18 passing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
paddlepaddle has no Python 3.14 (cp314) wheel on PyPI, so host-agent
deployments on 3.14 can never install it, causing OCR to fail at
runtime with RuntimeError. Pin the workspace to Python 3.13 across
all pyproject.toml files, the Docker base image, and the Jenkins CI
image; regenerate uv.lock against 3.13.
Also fixes a pre-existing Python-2-style `except X, Y:` syntax error
(invalid in all Python 3.x) in runtime/task.py and
packages/cloud-platform/cloud/{sql_repository,internal_api/api}.py,
introduced in 22d37ca9 and unrelated to this change's scope, which
blocked the full test suite from collecting on any interpreter
version.
openspec change: downgrade-python-3-13-paddleocr
This change was originally proposed as the root-cause fix for the
2026-07-14 DeviceNotFoundError incident. That diagnosis was wrong: it was
subsequently confirmed only one Host Agent process was running at the time,
ruling out the duplicate-process precondition this change addresses. The
actual root cause was execution.py's create_task_runner() omitting manager=
when wiring TaskRunner (see 08cef7c). Reframe the Why section: this change
stands on its own as independent duplicate-process hardening, not as a fix
for an incident it turned out not to have caused.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
create_task_runner() built TaskRunner's observer/screenshot_provider by
calling describe_screen(device_id)/take_screenshot(device_id) without
manager=, so both silently fell back to the process-global DEFAULT_MANAGER
singleton instead of the Host Agent's real, device-populated DeviceManager.
DEFAULT_MANAGER never has any device registered, so every task's first step
raised DeviceNotFoundError even though the console (which does pass
manager=) showed the same device as connected. Deterministic on every task,
independent of process count.
Add regression tests confirming both lambdas now resolve devices via the
configured manager; verified each fails with the original DeviceNotFoundError
symptom when the fix is reverted.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Proposal, design, spec, and tasks for the per-installation exclusive
instance lock. 15/16 tasks complete; only manual real-environment
verification (5.4) remains, with semantics covered by unit tests in
test_app.py and test_instance_lock.py.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Acquire an exclusive, non-blocking filelock on the identity state directory
as the first action of create_application(), before resolve_host_identity()
or any enrollment/heartbeat side effect. A second process against the same
identity_path exits immediately with InstanceAlreadyRunningError naming the
lock path; the lock releases automatically on any process exit (including
SIGKILL) via OS-level advisory locking, and explicitly during run_async()'s
shutdown finally block. filelock is promoted from transitive to direct
dependency (version unchanged at 3.29.7).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Apply consistent line-length formatting across governance, plugins, the
SDK routers (governance_api, user_api), user_auth, and migrations
0003/0005/0006.
Also restore the parentheses on two except clauses that had been dropped
into invalid Python 3 `except A, B:` syntax: plugins._coerce_to_manifest
(KeyError, ValueError) and PasswordHasher.verify (InvalidHashError,
VerificationError). Both modules now import cleanly again.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add an optional single-process mode where the backend serves the built
console bundle itself, so operators don't need a separate `npm run dev`
for edge/dev setups. When RUNTIME_CONSOLE_STATIC_DIR points at the
console dist directory, the app mounts a SpaStaticFiles handler at /ui/
(with 404 fallback to index.html for client-side routing) and redirects
/ to /ui/. The console build uses an empty VITE_API_BASE_URL for relative
API paths (same-origin, no CORS), and Vite's base is set to /ui/ so
assets resolve under the mount. /console/* JSON API is unchanged and is
shared by both serve modes.
api.ts now treats an explicitly-empty VITE_API_BASE_URL as "use relative
paths" instead of falling back to the dev default, which previously
forced absolute URLs even in same-origin builds.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace hand-written f-string + html.escape() rendering in the Host Agent
local console with a module-level Jinja2 Environment configured with
select_autoescape(["html","xml"]). XSS safety now holds by mechanism
rather than per-call discipline — every operator-controlled field
(device name, connection_info, task summary, etc.) is escaped by the
engine uniformly.
Eight templates under host_agent/web/templates/ replace the former
_chrome(), _CSS, escape(), and per-page _xxx_body() helpers: base.html
(header/nav/CSS + {% block body %}), login, dashboard (with the polling
<script> preserved byte-identically inside {% raw %}), devices, account,
history, tasks_list, and task_detail. The task-list and task-detail
templates — added by the just-landed task-execution-progress-visibility
change — were also migrated here rather than left in f-string form,
since this change removes the shared helpers they depended on.
URLs, auth/session/CSRF semantics, redirects, and /api/status JSON are
unchanged. 15 new template tests cover render-smoke, XSS probing, script
byte-identity, and no-autoescape-bypass guards. Tasks 8.1-8.6 (manual
browser verification) remain.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Host Agent now persists step-level execution detail locally (via a real
TaskMetadataStore/Timeline wired into TaskRunner) and reports a bounded
in-progress snapshot piggybacked on lease renewal. Cloud persists that
snapshot per active assignment and exposes it through the existing task
list/detail query path; Cloud Console renders it as a live badge. Host
Agent's local console gains authenticated, read-only task list and
detail/timeline pages (same-origin, server-rendered) with inlined
screenshots.
Also fixes a pre-existing gap in the shared Timeline: the actual
per-step LLM prompt is now recorded instead of the task goal, benefiting
both Runtime and Host Agent consoles. When a host uses the cloud planner
transport, each decide call's prompt and resulting tool decision are
durably logged in a new planner_decision_log table (with bounded
retention) and browsable from Cloud Console; direct-transport hosts
explicitly surface a "not reported" state.
Includes Alembic migrations 0008 (progress columns on scheduled_tasks)
and 0009 (planner_decision_log), bounded Host-Agent-local retention,
dual-backend repository parity, and Vitest + pytest coverage. Task 6.5
(manual end-to-end device verification) remains.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
q792602257
