This commit is contained in:
@@ -1,137 +1,176 @@
|
||||
## Context
|
||||
|
||||
Three surfaces currently cannot show a task's in-progress status:
|
||||
Host Agent executes Cloud assignments in-process through the shared
|
||||
`runtime.TaskRunner`. Its `HostAgentApplication` already constructs a
|
||||
Host-local `TaskMetadataStore` and `Timeline` and injects them into the
|
||||
runner factory. The missing link is task creation: `AssignmentExecutor`
|
||||
constructs a `Task`, calls `runner.run(task)`, and `TaskRunner` only
|
||||
issues updates. SQLite therefore receives updates for a row that does not
|
||||
exist, so the Host Agent `/tasks` UI is empty.
|
||||
|
||||
1. **Host Agent local console** (`apps/device-host-agent/host_agent/web/app.py`): `HostAgentApplication`'s builder calls `create_execution_factories(resolved_manager, host_agent_config=resolved_config)` without `metadata_store`/`timeline` (`app.py:218-223`), so the in-process `TaskRunner` (`execution.py:39-46`) runs with `metadata_store=None, timeline=None`. `runtime/task.py::TaskRunner._update_task()` only persists when `metadata_store` is truthy, so step transitions vanish. The console's only signal is `AgentStatusTracker.snapshot()` (`status.py:14-88`), a single opaque span: `task_id/device_id/goal/started_at`, cleared on `mark_assignment_finished()`.
|
||||
2. **Cloud Control Plane / Cloud Console**: the internal protocol (`packages/cloud-platform/cloud/internal_api/models.py`) only has claim, heartbeat, lease-renewal, and terminal-result messages. Lease renewal (`ActiveAssignmentRunner._renew_while_running`, `apps/device-host-agent/host_agent/lease.py:79-106`) already fires periodically (~1/3 of remaining lease) for every in-flight assignment, but carries no task-progress payload.
|
||||
3. **Runtime `console/` SPA**: `api/rest.py::create_app()` builds its own `TaskMetadataStore`/`Timeline`/`TaskRunner` (`api/rest.py:24-41`), queried by `api/console.py::create_console_router()`'s `/console/tasks*` routes, which `console/src/api.ts` calls against `API_BASE_URL` (default `http://127.0.0.1:8000`). This is a wholly separate process from Host Agent, with no shared store or IPC. `runtime/`-owned packages are forbidden from importing host/cloud concerns (`test_runtime_owned_packages_do_not_import_host_or_cloud_concerns`), and `host-agent-local-console/design.md` already treats Host Agent's local history as "not a system of record," distinct from Cloud's.
|
||||
The separate `api.rest` process was never on this execution path. It owned a
|
||||
different database and artifact root, so its REST endpoints and UI could only
|
||||
show tasks submitted directly to that unrelated process. Running it alongside
|
||||
a Host Agent created two competing operator entry points without transferring
|
||||
any history between them.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Host Agent's local console shows live step-level progress for its current assignment (step index, step status, short summary), not just a static started-at snapshot.
|
||||
- Cloud Control Plane learns step-level progress near-real-time (piggybacked on existing periodic traffic, not a new polling loop) and Cloud Console renders it for an operator watching a remote task.
|
||||
- An operator can inspect a Host-Agent-executed task's progress/history from a web console, without coupling `runtime/` to host/cloud concerns.
|
||||
- Reuse existing, already-tested building blocks (`TaskMetadataStore`, `Timeline`, `api/console.py`'s query shape, the lease-renewal cadence) instead of inventing new storage or transport primitives.
|
||||
- Both the Host Agent's local task history and the Cloud Control Plane persist the *actual* per-step content sent to and received from the LLM (the real prompt text, not just the task's overall goal; the model's resulting decision, not just the parsed tool call) — so a step's record answers "what did we actually ask the model and what did it say," not just "what call did we make." Cloud's copy is the durable, centrally-searchable record for troubleshooting; the Host Agent's local copy remains the on-device record of what actually ran.
|
||||
|
||||
- Make durable task-row creation an invariant of `TaskRunner.run()` whenever
|
||||
a metadata store is configured.
|
||||
- Correlate a Host-local goal execution with the Cloud task ID and attempt that
|
||||
caused it, without making shared Runtime or storage packages import Cloud
|
||||
models.
|
||||
- Make the authenticated Host Agent console at port `8765` the authoritative
|
||||
web view for actual task execution: task list, detail, live status, before
|
||||
screenshot, operation, after screenshot, OCR observations, and normalized
|
||||
UI-tree results.
|
||||
- Preserve the shared Runtime Timeline as the evidence model and keep its
|
||||
before/after screenshot, OCR, and UI-tree capture behavior.
|
||||
- Retire the standalone Runtime REST service/UI and its Host Agent supervisor
|
||||
configuration while retaining Runtime, storage, MCP, and skill-sync library
|
||||
modules.
|
||||
- Retain the existing Cloud latest-progress and Cloud-proxy planner-decision
|
||||
history behavior. Cloud remains a fleet-level view and never stores
|
||||
screenshots.
|
||||
|
||||
**Non-Goals:**
|
||||
- No SSE/WebSocket push. All three surfaces keep polling (Host Agent console 5s, Cloud Console/`cloud-console` and `console/` at their existing intervals). Nothing here requires push infra, and adding it would be a bigger, separate change.
|
||||
- The coarse live step index/status/summary reported via lease renewal (D4/D5) stays *latest-snapshot-only* on the Cloud side (overwrite semantics) — it is a "what's happening right now" indicator, not a history mechanism, and is unaffected by the LLM-content decisions below.
|
||||
- Full step-by-step **LLM interaction** history *is* now synced to and durably persisted by Cloud (see D7/D8) — this is a deliberate reversal of this change's earlier draft, which had scoped Cloud to latest-only. The motivation: Host Agent deployments already route every AI Planner decision through Cloud API's existing `cloud-planner-proxy` endpoint when configured for the `cloud` transport, so Cloud is already on the natural path for this data and centralizing it there (rather than only on whichever edge host happens to still be running) is the more useful place for an operator doing centralized troubleshooting. This durable log is scoped strictly to LLM prompt/response text — it is not a general-purpose duplicate of the Host Agent's full `Timeline` (no screenshots, no scene JSON dumps beyond what's embedded in the prompt text itself).
|
||||
- No multi-backend Runtime console or cross-origin Host Agent integration. Runtime changes remain local to its existing same-origin API/UI and shared timeline evidence model.
|
||||
- No change to `driver/`/`device/`/`core/` device-control internals.
|
||||
- No screenshot or full-scene data leaves the Host Agent process as part of progress reporting, and no screenshot bytes are ever persisted by the Cloud Control Plane. (Full LLM prompt/response *text* is, by contrast, an explicit Goal below — see D7/D8 — which partially supersedes the original `cloud-planner-proxy` proposal's "does not durably persist ... full prompt text" statement. Screenshots remain excluded; prompts no longer are.)
|
||||
|
||||
- No SSE/WebSocket push; Host and Cloud consoles keep their existing polling.
|
||||
- No duplicate full evidence upload to Cloud. Screenshots and Timeline
|
||||
artifacts remain Host-local.
|
||||
- No new Host/Cloud dependencies in `runtime/`, `storage/`, `driver/`, or
|
||||
`device/`.
|
||||
- No replacement general-purpose device-control REST API. Operators use Host
|
||||
Agent device management and task pages for the managed execution workflow.
|
||||
|
||||
## Decisions
|
||||
|
||||
### D1: Wire a real `metadata_store`/`timeline` into Host Agent's `TaskRunner`, reusing the existing classes
|
||||
### D1: TaskRunner owns idempotent task metadata creation
|
||||
|
||||
`create_execution_factories()` already accepts `metadata_store: TaskMetadataStore | None` and `timeline: Timeline | None` (`execution.py:28-34`) — these are the same classes `api/rest.py` uses for the Runtime's own console. Reuse them as-is rather than inventing a Host-Agent-specific store: `HostAgentApplication`'s builder constructs a `TaskMetadataStore(db_path=<host-agent-local path>)` and `Timeline(ArtifactStore(<host-agent-local path>))` and passes them into `create_execution_factories()`.
|
||||
`TaskRunner.run()` will call `TaskMetadataStore.create_task(task)` before
|
||||
the first running-state update. `create_task()` will use idempotent insert
|
||||
semantics so callers that already created a direct Runtime task remain
|
||||
compatible and workflow-created tasks are captured automatically.
|
||||
|
||||
**Alternative considered**: a bespoke in-memory-only step recorder (cheaper, no disk I/O). Rejected because `TaskMetadataStore`/`Timeline` are already proven in production (Runtime's own console has run on them since the original `web-console` change) and reusing them means Host Agent's task/timeline data has the *exact* same shape as `api/console.py` already serializes — a prerequisite for D3 below. A bespoke format would need its own (de)serialization and its own console rendering code for no real benefit.
|
||||
This places the invariant at the shared execution boundary rather than relying
|
||||
on every caller to remember an out-of-band persistence call. It fixes goal
|
||||
assignments and prevents the same failure for `WorkflowRunner` planned-goal
|
||||
steps.
|
||||
|
||||
New config: `HOST_AGENT_TASK_PROGRESS_DB_PATH` (default e.g. `host_agent_data/task_progress.sqlite3`) and a matching artifact directory, kept separate from any local Runtime `tasks/` directory that might exist on the same machine, to avoid two unrelated processes silently sharing or colliding on a path.
|
||||
### D2: Host assignment correlation stays in the Host adapter
|
||||
|
||||
### D2: Bounded retention for the Host-Agent-local store
|
||||
Before executing a Cloud goal assignment, `AssignmentExecutor` creates the
|
||||
local task record with optional generic `source_task_id` and
|
||||
`source_attempt` metadata. Shared storage uses generic names and does not
|
||||
import Cloud types. The Host Agent task list/detail renders those fields as the
|
||||
Cloud task ID and attempt.
|
||||
|
||||
Unlike a Runtime dev session (short-lived, manually cleared), a Host Agent process runs indefinitely and executes many assignments over its lifetime. Unbounded `TaskMetadataStore` rows and `Timeline` screenshot artifacts would grow without limit.
|
||||
The local `Task.id` remains generated by the Runtime. Reusing the Cloud task
|
||||
ID as an artifact directory name would make retries overwrite each other and
|
||||
would admit unsafe path characters from an external identifier.
|
||||
|
||||
Decision: add a lightweight retention pass (e.g. on a timer, or opportunistically after each assignment finishes) that prunes tasks older than a configurable window or beyond a configurable count, deleting both the `tasks` row and its `Timeline`/`ArtifactStore` files. This is new logic — neither `TaskMetadataStore` nor `Timeline` currently supports deletion — scoped as a small addition to `storage/task_metadata.py` / `storage/timeline.py` (or a Host-Agent-side wrapper if adding delete methods to the shared `storage` package feels too broad; prefer extending `storage` since both Runtime and Host Agent benefit from bounded retention).
|
||||
### D3: Host Agent console is the authoritative evidence UI
|
||||
|
||||
### D3: Give Host Agent's own console read-only task/timeline pages instead of making `console/` multi-backend
|
||||
The Host Agent already owns the device manager, assignment executor,
|
||||
metadata store, Timeline, local account, session, and CSRF boundary. Its
|
||||
same-origin `/tasks` and `/tasks/{task_id}` pages therefore render the
|
||||
shared Timeline directly. The task list is named for Host executions rather
|
||||
than "Local Runtime tasks", and submission feedback tells the operator that
|
||||
the submitted Cloud task appears there when this Host begins execution.
|
||||
|
||||
Two options were evaluated for capability `host-agent-console-task-pages`:
|
||||
No browser needs to point a separate frontend at the Host Agent. This keeps
|
||||
the conservative local-account/session model and does not add CORS.
|
||||
|
||||
- **(a) Multi-backend Runtime `console/` SPA**: let the existing Vue SPA point at a Host Agent's console origin (a saved/selectable base URL) in addition to the local Runtime. Since D1 makes Host Agent's data shape identical to what `api/console.py` already serializes, the SPA's existing fetch/render code would work unmodified against a Host Agent origin.
|
||||
- **(b) Extend Host Agent's own server-rendered console** (`host_agent/web/app.py`) with new read-only pages for task list/detail/timeline, reusing `TaskMetadataStore`/`Timeline` query calls directly (the same calls `api/console.py`'s handlers make), rendered as server-side HTML via the existing f-string + `html.escape()` convention (no SPA, no frontend build) established by `host-agent-local-console`.
|
||||
### D4: Complete per-step evidence is rendered by the Host Agent
|
||||
|
||||
**Decision: (b).** (a) requires Host Agent's console to serve cross-origin, credentialed requests from whatever origin `console/` is running on (Vite dev server, or a different deployed origin) — meaning either relaxing Host Agent's CORS posture for its session-cookie+CSRF-protected endpoints, or reworking its auth model to tolerate cross-origin fetches. That is real new attack surface on a console whose entire local-only threat model (`console_bind_host` defaulting to loopback, explicit opt-in for non-loopback) was deliberately conservative. (b) reuses an already-accepted pattern (server-rendered, same-origin, same auth) and fully satisfies the actual operator need — a web page showing what a Host Agent is doing — without touching `runtime/`, `console/`, or Host Agent's CORS/auth posture at all. The cost is a small amount of view duplication (Host Agent's console and Runtime's `console/` render similar-shaped data with different templates/frameworks), judged acceptable given they serve different operational contexts (local direct-connect Runtime vs. remote Host Agent fleet).
|
||||
Timeline records retain distinct pre-action and post-action screenshot paths,
|
||||
the action description and arguments, the execution result, raw OCR
|
||||
observations, and the existing normalized UI-tree result. The Host Agent page
|
||||
creates data URIs only for available local artifacts and supports legacy
|
||||
records where the single `screenshot_path` is the post-action image.
|
||||
|
||||
Any future desire for a unified SPA across both surfaces is left as a follow-up, not blocked by this decision (D1's shared data shape keeps that door open).
|
||||
OCR is rendered when present. UI-tree output is rendered only for
|
||||
`get_ui_tree` and `ui_tree` records that contain normalized nodes; it uses a
|
||||
collapsible structured view while retaining the JSON result. No tool contract
|
||||
or duplicate persistence field is introduced.
|
||||
|
||||
### D4: Piggyback progress reporting on the existing lease-renewal call
|
||||
### D5: Host-local retention remains bounded
|
||||
|
||||
`ActiveAssignmentRunner._renew_while_running` (`lease.py:79-106`) already makes an authenticated, per-assignment, periodic call (`client.renew(assignment)`, roughly every 1/3 of remaining lease) while a task executes, validated server-side against `host_id`/`task_id`/`attempt`/`lease_id` (`internal_api/api.py:291-320`). This is the natural piggyback point for progress: extend `LeaseRenewalRequest` with an optional `progress: TaskProgressModel | None` field (new small model: `step_index: int`, `step_status: Literal[...]`, `summary: str` bounded length — no screenshot/scene payload) in `cloud/internal_api/models.py`, the single schema both sides already import directly (no parallel schema, matching existing convention noted for this protocol).
|
||||
The existing Host retention pass continues to remove metadata rows, Timeline
|
||||
records, and artifacts according to the configured count and age thresholds.
|
||||
The new task creation invariant must use that same store so it cannot create
|
||||
an unbounded second history source.
|
||||
|
||||
A small thread-safe "latest progress" holder (similar in spirit to `LeaseGuard`) is written by the execution thread (via a hook on `TaskRunner`/`AssignmentExecutor`, updated once per step) and read by `_renew_while_running` just before each renewal call, so no new timer/cadence is introduced.
|
||||
### D6: Cloud progress remains a latest snapshot on lease renewal
|
||||
|
||||
**Alternative considered**: a dedicated `POST /internal/v1/hosts/{host_id}/assignments/{task_id}/progress` endpoint called on its own cadence. Rejected — it would duplicate the auth/validation `renew_assignment` already does, and introduce a second periodic call where one already exists and fires at a reasonable frequency for this purpose.
|
||||
The Host execution thread writes a bounded latest-progress holder. The lease
|
||||
renewal path optionally carries its step index, status, and summary; Cloud
|
||||
stores only the latest snapshot for an active assignment and stops exposing it
|
||||
after terminal completion. No screenshot or scene payload enters this
|
||||
protocol.
|
||||
|
||||
### D5: Cloud persists only the latest progress snapshot, overwritten alongside lease renewal
|
||||
### D7: Cloud-proxy planner decisions remain durable Cloud history
|
||||
|
||||
`renew_lease()` (`repository.py:407`, `sql_repository.py:1480`) already takes a row lock and writes a lease-expiry update on every renewal; extend it to also accept and store the optional progress fields (few scalar columns — `progress_step_index`, `progress_step_status`, `progress_summary`, `progress_updated_at` — rather than a schema-flexible JSON blob, keeping it queryable and consistent with the rest of the row's typed columns). This requires a new Alembic migration (head is currently `0007_llm_provider_management`) plus the equivalent SQLite schema change, applied to both `repository.py` and `sql_repository.py` to keep the dual-backend contract (`cloud-control-plane` spec's "Deployment and local persistence modes share one contract" requirement).
|
||||
For `AI_PLANNER_TRANSPORT=cloud`, Cloud's existing planner-decision endpoint
|
||||
persists successful system/user prompts, tool calls, arguments, and a
|
||||
task/attempt-scoped step index. The direct transport intentionally produces no
|
||||
such Cloud history. This log has bounded terminal-task retention and never
|
||||
persists request screenshot bytes.
|
||||
|
||||
Progress columns are cleared (or simply superseded and ignored) once a terminal result is recorded — they represent "what's happening right now," not history; Cloud's durable attempt/result history is unaffected and unduplicated.
|
||||
### D8: Retire the standalone Runtime REST service and UI
|
||||
|
||||
### D6: Cloud Console reads progress from the existing task/attempt query path, not a new endpoint
|
||||
Remove `api/rest.py`, `api/console.py`, `api/console_web.py`, their
|
||||
templates/static assets, their package-data declarations, and their
|
||||
service/UI tests. Preserve `api/mcp.py`, `api/errors.py`, skill-sync, and
|
||||
skill-catalog modules because they are independent library integrations.
|
||||
|
||||
Extend whatever response model Cloud Console's task list/detail already uses (Cloud API public router) with the optional latest-progress fields from D5, rather than adding a new endpoint. Cloud Console renders it as a small inline badge/line ("step 4: tapping login button") next to the existing status, refreshed on the SPA's existing polling interval.
|
||||
Remove `HOST_AGENT_RUNTIME_SUPERVISED`,
|
||||
`HOST_AGENT_RUNTIME_HOST`, and `HOST_AGENT_RUNTIME_PORT`. The optional
|
||||
dependency supervisor continues to support Appium only. Configuration with a
|
||||
removed Runtime-supervision variable fails with an actionable migration error
|
||||
instead of silently doing nothing.
|
||||
|
||||
### D7: Capture full LLM interaction history for free via the existing `cloud-planner-proxy` decide endpoint, not a new reporting channel
|
||||
### D9: Documentation points operators to Host Agent
|
||||
|
||||
Investigated the Host Agent → Cloud call path in detail: when a Host Agent is configured with `AI_PLANNER_TRANSPORT=cloud`, `AIPlanner.plan()` (`runtime/ai_planner.py`) calls `CloudProxyToolCallingClient.decide()` (`apps/device-host-agent/host_agent/cloud_planner_client.py:46-95`), which `POST`s the *complete* `system_prompt`/`user_prompt` (plus `task_id`/`attempt`/`lease_id` from `current_planner_execution_context()`) to Cloud API's `/hosts/{host_id}/planner/decide` (`packages/cloud-platform/cloud/internal_api/api.py::decide_planner_call`, line ~367). That handler already resolves and returns a `ToolCallDecision` (`tool_name`/`arguments`/`usage`) and already does per-call bookkeeping (`pool.store.settle_host_token_reservation(...)`, line ~463) using the same repository object this change's D5 already touches for lease renewal.
|
||||
|
||||
This means **every planning step's actual prompt and resulting decision already flows through a Cloud-owned request handler** when the `cloud` transport is used — no new endpoint, no new protocol field, no queue/batching scheme is needed to get full LLM content to Cloud. The only change needed is to make that handler *persist* what it currently discards.
|
||||
|
||||
**Decision**: extend `decide_planner_call` to, immediately after computing `decision` (success path only — a `ToolCallUnavailable`/502 path persists nothing), insert one row into a new log table (D8) keyed by `(task_id, attempt, step_index)`, where `step_index` is assigned by the Cloud side itself (an auto-incrementing counter scoped to `task_id`+`attempt`, e.g. `select count(*) + 1` under the same row lock, or a DB sequence/identity column) — Host Agent does not need to track or send a step counter for this.
|
||||
|
||||
**Explicit limitation, called out rather than papered over**: this only captures LLM content for hosts using the `cloud` transport. A host on the (still-default) `direct` transport never sends its prompts to Cloud at all — Cloud has zero LLM content for that host's tasks, and only ever sees the coarse index/status/summary from D4/D5's lease-renewal piggyback (which is transport-agnostic, since it's driven by `TaskRunner` step completion, not by the planner's transport choice). This is a real operational dependency: centralized LLM-interaction troubleshooting via Cloud Console requires the fleet (or the hosts an operator cares about) to run with `AI_PLANNER_TRANSPORT=cloud`. This change does not make `cloud` the new default transport — that remains a separate, already-existing configuration decision outside this change's scope.
|
||||
|
||||
**Alternative considered**: extend the D4 lease-renewal piggyback to also carry full prompt/response text (queued, not overwritten, so no step is lost between renewals). Rejected: it would duplicate a transport that already exists for exactly this payload (the decide call itself) whenever `cloud` transport is active, and would still need a *separate* new channel for the `direct`-transport case where Cloud never sees the prompt anyway — i.e., it does not actually solve the `direct`-transport gap, so it only adds complexity without expanding coverage.
|
||||
|
||||
### D8: New bounded-retention table for the full per-step LLM decision log, extended on both repository backends
|
||||
|
||||
Add a new table (e.g. `planner_decision_log`): `id`, `host_id`, `task_id`, `attempt`, `step_index`, `system_prompt` (text), `user_prompt` (text), `tool_name`, `arguments_json` (text), `created_at`. No screenshot column — screenshots are never sent to this endpoint's persistence path (the request's `screenshot_base64` is used only to call the LLM provider and is never written to this log, consistent with the Non-Goals screenshot exclusion).
|
||||
|
||||
Like D5, this needs a new Alembic migration and the equivalent SQLite path, implemented on both `repository.py`'s SQLite-backed implementation and `sql_repository.py::SQLAlchemyCloudRepository` to preserve the existing dual-backend contract. Unlike D5's few-nullable-columns-on-an-existing-row approach, this is an independent append-only table (one row per decide call, not an overwrite), since the whole point is durable per-step history rather than a live snapshot.
|
||||
|
||||
**Retention**: this table grows once per planning step across the whole fleet, indefinitely, on a shared multi-tenant Cloud database — unbounded growth is a real risk here in a way D5's single-row-per-assignment overwrite never was. Decision: a scheduled prune job (mirrors D2's Host-Agent-local retention) deletes rows whose owning task reached a terminal state more than a configurable window ago (default: prune 7 days after task terminal, or once the task itself is pruned/archived by whatever existing Cloud task-retention policy applies — reuse that cadence rather than inventing a second one if `cloud-control-plane` already has one; otherwise default to a simple time-based prune).
|
||||
|
||||
### D9: Fix the shared `Timeline`/`TaskRunner`/`AIPlanner` path so the *actual* per-step prompt and response are recorded locally, not just the task goal and the parsed tool call
|
||||
|
||||
Independent of Cloud persistence, the existing local recording is itself wrong today: `TaskRunner._append_timeline()` (`runtime/task.py:296-319`) calls `Timeline.append(prompt=task.goal, tool_call={"action": ..., "description": ..., "args": ...}, ...)`. `task.goal` is the overall task goal, not the per-step prompt actually sent to the LLM — the real per-step prompt (`planner_user_prompt(goal, scene_json, history_summary)`, built in `ai_planner.py::plan()`) is constructed, sent, and discarded entirely within `AIPlanner.plan()`, never reaching `TaskRunner`. Likewise `ToolCallDecision` (`runtime/tool_calling_client.py:23-27`) carries only the parsed `tool_name`/`arguments`/`usage` — any raw response text/content the model returned is discarded during parsing (`_decision_from_anthropic_response`/`_decision_from_openai_response`).
|
||||
|
||||
This is a pre-existing gap in a component shared by Runtime and Host Agent alike (not new to this change), and it undermines the very "step-level detail" goal D1 already committed to — a persisted step whose "prompt" field is just the task's goal repeated on every row is not useful for troubleshooting.
|
||||
|
||||
**Decision**: extend `ToolCallDecision` with the actual `user_prompt`/`system_prompt` it was given (or have `AIPlanner.plan()` return a small side-channel result instead of changing the `Planner` interface's return type) so `TaskRunner._append_timeline()` can pass the real per-step prompt into `Timeline.append()`. Rename `Timeline`/`TimelineRecord`'s `prompt` field's meaning (or add a new field) to unambiguously mean "the prompt actually sent to the LLM for this step." This fix lands in the shared `runtime`/`storage` packages, so both Host Agent's local console (D3) and the existing Runtime `console/` automatically benefit — it is not Host-Agent-specific plumbing.
|
||||
|
||||
### D10: Persist and render before/after action evidence in the Runtime timeline
|
||||
|
||||
The existing Timeline captures only one screenshot after an action, while the planning screenshot used before the action is transient. Extend each Timeline record with distinct before/after screenshot paths, preserving the existing `screenshot_path` as a backward-compatible alias for the after screenshot. `TaskRunner` captures each image immediately before and after invoking the executor, including failed action attempts. The Runtime console API inlines both images and the existing task-detail page renders them beside the recorded action and result.
|
||||
|
||||
OCR is already available during perception but fused into the normalized Scene, where an OCR value that overlaps a UI-tree node can lose its raw provenance. Preserve raw OCR observations on `Scene` for local timeline capture only; keep them out of `Scene.to_dict()` so the LLM-facing planner payload does not grow with duplicated text. The Runtime task-detail page renders the persisted OCR list when it is available and handles legacy records without it.
|
||||
|
||||
The existing `get_ui_tree`/`ui_tree` tool returns a normalized flat list of UI nodes, not the driver-specific raw XML hierarchy. Its StepResult is already persisted in the Timeline. Detect those action names in the Runtime UI and render the returned nodes in a collapsible structured view while retaining the full JSON result below it. This makes the inspection result readable without changing the tool response contract or storing a second tree copy.
|
||||
Operator documentation no longer instructs users to start `uvicorn
|
||||
api.rest:create_app` or browse port `8000`. It identifies the Host Agent
|
||||
console at `http://127.0.0.1:8765/tasks` as the execution-history authority,
|
||||
explains its local authentication, and documents that the Cloud Console is a
|
||||
fleet/progress and Cloud-proxy LLM-history surface rather than a screenshot
|
||||
store.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- [Risk] Per-step SQLite writes in a long-lived Host Agent process add I/O overhead → Mitigation: this is the same write pattern Runtime's own console has always used per step; no new proof of acceptability needed. If profiling later shows it matters for very high step-rate tasks, batching/debouncing is a follow-up, not a blocker here.
|
||||
- [Risk] Unbounded local disk growth from `Timeline` screenshots on an indefinitely-running Host Agent → Mitigation: D2's bounded retention pass; must ship in the same change as D1, not deferred, since D1 alone would otherwise introduce an unbounded-growth regression.
|
||||
- [Risk] New DB columns/migration touch both `repository.py` (SQLite) and `sql_repository.py` (Postgres) — drift between the two has been a real defect category in this codebase (see `cloud-control-plane-integration` archive notes) → Mitigation: contract/parity tests already exist for this dual-backend boundary; extend them to cover the new progress columns.
|
||||
- [Risk] View duplication between Host Agent's server-rendered task pages (D3) and Runtime `console/`'s task pages (different frameworks, same data shape) → Mitigation: accepted trade-off (see D3 rationale); shared data shape keeps future unification possible without rework.
|
||||
- [Risk] Progress payload could accidentally grow to include sensitive data (scene dumps, prompts) if a future change casually extends `TaskProgressModel` → Mitigation: `summary` field is explicitly bounded/plain-text only; code review for this change and future extensions should treat this the same as the existing `cloud-planner-proxy` no-screenshot-persistence rule (screenshots specifically remain excluded from every Cloud-side table this change introduces).
|
||||
- [Risk] `planner_decision_log` (D8) grows unboundedly across the whole fleet, unlike D5's single-row-per-assignment overwrite → Mitigation: D8's retention/prune job must ship in the same change as D7/D8, not deferred, for the same reason D2 must ship alongside D1.
|
||||
- [Risk] Full prompt text can itself contain sensitive on-screen content (whatever text was visible in the scene description embedded in the prompt) — persisting it centrally is a deliberate trade-off the user has explicitly requested for centralized troubleshooting, but it is a real expansion of what Cloud stores → Mitigation: no additional mitigation beyond what's already decided (screenshots still excluded); flagged here so it is a visible, intentional decision rather than a silent scope creep.
|
||||
- [Risk] Full LLM history in Cloud is silently absent for any host on the `direct` transport, which could read as "it's broken" rather than "expected" → Mitigation: Cloud Console should visibly distinguish "no progress reported yet" from "this host does not report LLM content" (e.g. by also surfacing the host's configured transport), rather than just showing an empty history with no explanation.
|
||||
- Per-step metadata writes add small SQLite I/O. This is the same local store
|
||||
already selected for Host history, and bounded retention limits growth.
|
||||
- A Host-local evidence record is only available while retained on that Host.
|
||||
This is intentional: it reflects the actual device execution and avoids
|
||||
sending screenshots to Cloud.
|
||||
- Removing the unauthenticated Runtime REST service is a breaking operator
|
||||
change. Clear configuration errors and documentation avoid a silent
|
||||
fallback to a nonexistent inspection surface.
|
||||
- Full Cloud-proxy prompts can contain visible screen text. This is the
|
||||
previously accepted Cloud troubleshooting trade-off; screenshot bytes remain
|
||||
excluded.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
1. Add `TaskProgressModel` and the optional `progress` field to `LeaseRenewalRequest`/response in `cloud/internal_api/models.py`. Backward compatible: `progress` is optional, older Host Agents omit it.
|
||||
2. Add progress columns + Alembic migration `0008_task_progress_columns`; extend `renew_lease()` in both repository implementations to accept/store them.
|
||||
3. Extend Cloud Console's task read model and UI to render the new fields (no-op if absent, keeping rollback trivial).
|
||||
4. Wire `metadata_store`/`timeline` into Host Agent's `create_execution_factories()` call site, add the retention pass (D2), and add the progress-holder hook feeding D4's renewal piggyback.
|
||||
5. Extend Host Agent's local console with the read-only task/timeline pages (D3).
|
||||
6. Fix `Timeline`/`TaskRunner`/`AIPlanner`/`ToolCallingClient` to record the real per-step prompt and response locally (D9) — independent of Cloud, benefits both Host Agent and Runtime consoles immediately.
|
||||
7. Add `planner_decision_log` + Alembic migration for it (D8) on both repository backends, with its retention/prune job.
|
||||
8. Extend `decide_planner_call` (`internal_api/api.py`) to persist each resolved decision into `planner_decision_log` (D7).
|
||||
9. Extend Cloud Console with a per-task LLM interaction history view reading the new table, including the "host uses `direct` transport, no LLM content available" distinction from the Risks section.
|
||||
10. Rollback: each step is independently revertible (optional field, additive columns, additive tables, additive UI, additive Host Agent wiring) — no destructive migration is required; both `0008` and the new decision-log migration's down-revisions drop only what they added.
|
||||
1. Upgrade the Host Agent code. Existing task databases gain nullable source
|
||||
correlation columns on startup; legacy Timeline records remain readable.
|
||||
2. Remove any `HOST_AGENT_RUNTIME_*` environment variables and stop any
|
||||
`api.rest` process. Start or browse only the Host Agent console for local
|
||||
execution evidence.
|
||||
3. Confirm a completed Host assignment appears at `/tasks` with its Cloud
|
||||
task/attempt correlation and complete Timeline evidence.
|
||||
4. Roll back only by restoring the prior release. The retired REST/UI routes
|
||||
are deliberately not kept as a compatibility alias because their storage
|
||||
was not connected to Host execution.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Exact retention window/count defaults for D2 (time-based vs. count-based, or both) — left for tasks.md to pick a concrete, documented default (e.g. keep last 50 tasks or 7 days, whichever is smaller) rather than block design on it.
|
||||
- Whether Cloud Console's existing task list/detail component can absorb the new fields with a small edit or needs a new sub-component — an implementation detail, not an architectural fork.
|
||||
- Whether a future change should unify Host Agent's server-rendered task pages and Runtime `console/`'s SPA into one shared frontend, now that D1 gives them an identical underlying data shape — explicitly deferred, not part of this change.
|
||||
- Exact retention default for D8's `planner_decision_log` (prune-after-terminal window, or reuse an existing Cloud task-retention cadence if one already exists) — left for tasks.md to pick a concrete default rather than block design on it.
|
||||
- Whether `step_index` in D8 should be assigned via a `SELECT count(*) + 1` under a row lock or a dedicated per-`(task_id, attempt)` counter/sequence — an implementation detail to resolve in tasks.md, not an architectural fork.
|
||||
- Manual verification still requires a real Host Agent, Appium, and device.
|
||||
Automated coverage verifies persistence, correlation, rendering, and
|
||||
service removal; real hardware validates screenshots and OCR availability.
|
||||
|
||||
@@ -1,26 +1,28 @@
|
||||
## Why
|
||||
|
||||
Nobody can see a task while it is running. Host Agent's local console only shows a coarse "current assignment" snapshot (task id, device id, goal, started-at) because the `TaskRunner` it drives is wired with `metadata_store=None, timeline=None` — every step transition happens in memory and is discarded the instant the assignment finishes. Cloud Control Plane only learns about an assignment at claim, heartbeat, and terminal-result time, so Cloud Console has nothing better to show. The local Runtime (`api/rest.py` + `console/`) is a separate process with its own independent `TaskMetadataStore`/`Timeline`, so it never sees a task that a Host Agent executed at all. An operator debugging a stuck or misbehaving task currently has no live signal anywhere in the system until the task finishes or times out.
|
||||
Host Agent now wires a local `TaskMetadataStore` and `Timeline` into its `TaskRunner`, but its Cloud-assignment path still creates a `Task` and immediately calls `runner.run(task)`. `TaskRunner` only updates an existing metadata row, so every update affects zero rows and the Host Agent task page remains empty. The standalone Runtime REST service/UI is a separate process with unrelated storage, so it cannot be the inspection surface for Host Agent executions. An operator who submits or receives a task therefore has no authoritative web view of the actions that actually ran on that Host.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Wire Host Agent's `AssignmentExecutor` / `create_execution_factories()` with a real `metadata_store` and `timeline` (or Host-Agent-local equivalents) so the in-process `TaskRunner` actually records step-by-step status instead of discarding it.
|
||||
- Make `TaskRunner.run()` create its metadata row idempotently before its first status update, so every execution path, including workflow-owned tasks, persists history when a metadata store is configured. Have Host Agent register Cloud task/attempt correlation before goal execution so operators can match a submitted Cloud task to its local execution record.
|
||||
- Expose that step-level detail through Host Agent's local console: extend `AgentStatusTracker`/`/api/status` (or add a focused endpoint) with current step index, step status, and a short in-progress step log; render it in the dashboard's "Current assignment" section instead of the static started-at-only view.
|
||||
- Add a progress-reporting path from Host Agent to Cloud Control Plane so the control plane learns step-level state near-real-time rather than only at claim/heartbeat/terminal-result. Extend the existing `cloud.internal_api.models` Pydantic schema (the single source of truth Host Agent already imports directly) rather than introducing a parallel schema.
|
||||
- Persist and expose the latest per-assignment progress on the Cloud Control Plane side, and surface it in Cloud Console so an operator watching a remote task sees live step progress, not just "dispatched" / "succeeded" / "failed".
|
||||
- Give an operator a web console view of Host-Agent-executed task progress without making the `runtime/`-owned packages import host or cloud concerns (enforced by `test_runtime_owned_packages_do_not_import_host_or_cloud_concerns`). After evaluating the alternative of making the Runtime `console/` SPA multi-backend (pointing its existing JS bundle at a Host Agent's console origin), this change instead extends Host Agent's own server-rendered local console with read-only task list/detail/timeline pages, reusing the same query shape `api/console.py` already exposes to the Runtime SPA. This avoids new cross-origin/session-cookie surface between the SPA and Host Agent, and keeps `runtime/` untouched. See design.md for the full trade-off analysis.
|
||||
- Make Host Agent's own authenticated, server-rendered `:8765/tasks` pages the authoritative web entry point for actual execution history. They render the shared Timeline records directly and retain the existing same-origin session/CSRF boundary.
|
||||
- All three surfaces continue to use polling (matching current behavior); this change does not introduce SSE/WebSocket infrastructure unless design.md finds a compelling reason to.
|
||||
- Fix the shared `Timeline`/`TaskRunner`/`AIPlanner` recording path so a persisted step's "prompt" is the *actual* prompt sent to the LLM for that step (not the task's overall goal) and the model's resulting decision is captured too — this pre-existing gap affects Runtime and Host Agent alike and undermines the step-level detail this change otherwise adds.
|
||||
- Persist a durable, per-step log of full LLM prompt/response content on the Cloud Control Plane, for centralized troubleshooting — reusing the already-existing `cloud-planner-proxy` decide endpoint as the capture point (no new protocol/endpoint) rather than the coarse lease-renewal piggyback used for live index/status. This durable log is populated only for hosts using the `cloud` planner transport; hosts on the `direct` transport still get only the coarse index/status via lease renewal. Cloud Console gains a view to browse a task's full LLM interaction history.
|
||||
- Extend the local Runtime timeline so every executed action retains a before screenshot, operation detail, after screenshot, and any available raw OCR observations; render that evidence in the Runtime task-detail UI and expose it from the existing Runtime console API. When a step uses the existing UI-tree tool, render its normalized node result as a structured, collapsible view as well.
|
||||
- Keep before/after screenshots, operation detail, raw OCR observations, and normalized UI-tree results in the shared Runtime Timeline; render all of that evidence in the Host Agent task-detail page.
|
||||
- Retire the standalone Runtime REST service, its unauthenticated console/UI, and Host Agent Runtime-supervision settings. Preserve the shared `runtime/`, `storage/`, and non-REST `api/` library modules used by the Host Agent and MCP integrations.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `host-agent-task-progress`: Host Agent captures step-level execution progress for its in-flight assignment (via a wired `metadata_store`/`timeline`) and exposes it through its local console/API.
|
||||
- `host-agent-task-progress`: Host Agent captures step-level execution progress for every in-process task, correlates Cloud assignments with local records, and exposes it through its local console/API.
|
||||
- `cloud-task-progress-visibility`: Cloud Control Plane receives, persists, and exposes near-real-time step-level progress for assignments it has dispatched to a Host Agent, and Cloud Console renders it.
|
||||
- `host-agent-console-task-pages`: Host Agent's local server-rendered console gains read-only task list/detail/timeline pages (mirroring `api/console.py`'s task query shape) so an operator can inspect a Host-Agent-executed task's progress and history without needing the separate Runtime `console/` SPA or violating the `runtime/`-package host/cloud isolation boundary.
|
||||
- `runtime-task-evidence`: Runtime task history retains and renders pre/post action evidence, raw OCR observations, and existing UI-tree inspection results.
|
||||
- `host-agent-console-task-pages`: Host Agent's local server-rendered console gains the complete before/after evidence, OCR, and UI-tree views for Host-Agent-executed tasks and is the only web inspection surface for those executions.
|
||||
- `runtime-task-evidence`: Runtime task history retains complete pre/post action evidence, raw OCR observations, and existing UI-tree inspection results for Host Agent rendering.
|
||||
- `runtime-standalone-service`: the standalone Runtime REST service and its UI are removed; Runtime remains an execution library rather than a second operational console.
|
||||
|
||||
### Modified Capabilities
|
||||
- `host-agent-protocol`: add a requirement that the Host Agent reports in-progress step-level status updates to the control plane (in addition to the existing heartbeat/claim/renewal/result operations), and that the control plane accepts and stores them per active assignment.
|
||||
@@ -32,5 +34,6 @@ Nobody can see a task while it is running. Host Agent's local console only shows
|
||||
- `runtime/task.py`, `runtime/ai_planner.py`, `runtime/tool_calling_client.py`, `storage/timeline.py`, `storage/artifact_store.py`, `core/models.py`, `perception/scene_builder.py` — fix the shared step-recording path so the real per-step prompt and the model's response are captured, retain pre/post action screenshots and raw OCR observations, not just the task goal and parsed tool call.
|
||||
- `packages/cloud-platform/cloud/internal_api/models.py`, `packages/cloud-platform/cloud/internal_api/api.py` (`decide_planner_call`), `repository.py`/`sql_repository.py`, a new Alembic migration — new progress-reporting request/response models and a persistence + query path for latest per-assignment coarse progress, *and* a new durable per-step `planner_decision_log` table (with its own retention job) populated from the existing planner-decision endpoint.
|
||||
- `cloud-console/` (Vue3 SPA) — new UI to render live per-assignment progress, and a new view to browse a task's full LLM interaction history.
|
||||
- `api/console.py`, `api/console_web.py`, `api/templates/runtime_console/*` — expose and render Runtime-local step evidence; Host Agent task pages receive the data through the shared timeline shape.
|
||||
- No changes anticipated to `driver/`, `device/`, `core/` device-control internals.
|
||||
- `api/rest.py`, `api/console.py`, `api/console_web.py`, `api/templates/runtime_console/*`, `api/static/runtime_console/*`, their tests, package data, and Runtime-supervision configuration — removed.
|
||||
- `apps/device-host-agent/host_agent/assignment.py`, `execution.py`, `web/app.py`, and its task templates — create correlated execution records and render the complete shared Timeline evidence.
|
||||
- `runtime/task.py`, `storage/task_metadata.py`, and `storage/timeline.py` — make durable task creation an execution invariant while retaining generic storage boundaries.
|
||||
|
||||
+56
-23
@@ -1,34 +1,67 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Host Agent local console exposes step-level status for the current assignment
|
||||
The Host Agent's local console SHALL display, for its currently executing assignment, the current step index, step status, and a short summary, sourced from the Host Agent's local task metadata store, refreshed on the console's existing polling interval.
|
||||
|
||||
#### Scenario: An assignment is currently executing
|
||||
- **WHEN** an operator views the Host Agent local console dashboard while an assignment is executing
|
||||
- **THEN** the dashboard shows the current step index, step status, and a short summary for that assignment, updating on subsequent polls
|
||||
|
||||
#### Scenario: No assignment is currently executing
|
||||
- **WHEN** an operator views the dashboard while the Host Agent is idle
|
||||
- **THEN** the dashboard shows no in-progress step information
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Host Agent local console exposes read-only task history with per-step detail and screenshots
|
||||
The Host Agent's local console SHALL provide authenticated, read-only pages listing recently executed tasks and, for a selected task, its full per-step history including any captured screenshots, sourced from the Host Agent's local task metadata store and timeline.
|
||||
The Host Agent's local console SHALL provide authenticated, read-only pages
|
||||
listing recently executed local tasks and, for a selected task, its full
|
||||
per-step history from the Host-local metadata store and Timeline. The task
|
||||
detail SHALL show available before and after screenshots, operation details and
|
||||
arguments, execution result, OCR observations, and normalized UI-tree results.
|
||||
It SHALL render legacy Timeline records that only have a single screenshot as a
|
||||
post-action image.
|
||||
|
||||
#### Scenario: Operator lists recent tasks
|
||||
- **WHEN** an authenticated operator opens the Host Agent local console's task list page
|
||||
- **THEN** it shows tasks from the local task metadata store, most recent first, including tasks that have already reached a terminal state
|
||||
#### Scenario: Operator lists recent Host executions
|
||||
- **WHEN** an authenticated operator opens the Host Agent local console's task
|
||||
list page
|
||||
- **THEN** it shows local executions most recent first, including terminal
|
||||
tasks and any available Cloud task ID and attempt correlation
|
||||
|
||||
#### Scenario: Operator inspects a completed task's step history
|
||||
- **WHEN** an authenticated operator opens the detail page for a specific completed task
|
||||
- **THEN** the page shows each recorded step in order, including its tool call, result, and any captured screenshot
|
||||
- **WHEN** an authenticated operator opens the detail page for a completed
|
||||
Host execution
|
||||
- **THEN** the page shows each recorded step in order with its tool call,
|
||||
result, and available before/after screenshots
|
||||
|
||||
#### Scenario: OCR was captured for a step
|
||||
- **WHEN** the selected Timeline record contains OCR observations
|
||||
- **THEN** the detail page shows each observation's text, confidence, and
|
||||
bounds
|
||||
|
||||
#### Scenario: A UI-tree tool returned normalized nodes
|
||||
- **WHEN** the selected Timeline record invoked `get_ui_tree` or `ui_tree`
|
||||
and its result contains normalized nodes
|
||||
- **THEN** the detail page exposes a structured, collapsible node view while
|
||||
retaining the persisted result JSON
|
||||
|
||||
#### Scenario: A legacy timeline record is displayed
|
||||
- **WHEN** a Timeline record has only `screenshot_path`
|
||||
- **THEN** the detail page renders it as the post-action image without failing
|
||||
|
||||
#### Scenario: Unauthenticated request
|
||||
- **WHEN** a request to the task list or task detail pages is made without a valid Host Agent console session
|
||||
- **THEN** the Host Agent rejects the request the same way it rejects unauthenticated requests to its other console pages
|
||||
- **WHEN** a request to the task list or task detail pages is made without a
|
||||
valid Host Agent console session
|
||||
- **THEN** the Host Agent rejects the request the same way it rejects
|
||||
unauthenticated requests to its other console pages
|
||||
|
||||
### Requirement: Host Agent local console task pages require no new cross-origin surface
|
||||
The Host Agent local console's task pages SHALL be served same-origin from the Host Agent's existing web application, without introducing new CORS allowances or a dependency on the separate Runtime `console/` frontend.
|
||||
The Host Agent local console's task pages SHALL be served same-origin from the
|
||||
Host Agent's existing web application, without introducing new CORS allowances
|
||||
or a dependency on a separate Runtime frontend.
|
||||
|
||||
#### Scenario: Task pages are requested
|
||||
- **WHEN** an operator's browser requests the Host Agent local console's task pages
|
||||
- **THEN** the pages are served by the Host Agent's own application using its existing session/CSRF protections, with no additional cross-origin configuration required
|
||||
- **WHEN** an operator's browser requests the Host Agent local console's task
|
||||
pages
|
||||
- **THEN** the pages are served by the Host Agent's own application using its
|
||||
existing session/CSRF protections, with no additional cross-origin
|
||||
configuration required
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Host Agent console is the authority for actual execution evidence
|
||||
The Host Agent local console SHALL be the web authority for task evidence
|
||||
produced by that Host's in-process execution path. A standalone Runtime
|
||||
service/UI SHALL NOT be required or consulted to inspect a Host execution.
|
||||
|
||||
#### Scenario: A Cloud task is executed by a Host Agent
|
||||
- **WHEN** an operator opens that Host Agent's task page after execution starts
|
||||
- **THEN** the page reads the same Host-local metadata and Timeline that the
|
||||
executing `TaskRunner` writes
|
||||
|
||||
+51
@@ -0,0 +1,51 @@
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Supervisor is opt-in and disabled by default
|
||||
The Host Agent SHALL NOT start, adopt-check, or supervise Appium unless
|
||||
`HOST_AGENT_DEPENDENCY_SUPERVISOR_ENABLED` is explicitly set to true. Appium
|
||||
supervision SHALL additionally require `HOST_AGENT_APPIUM_SUPERVISED=true`
|
||||
and SHALL default to false.
|
||||
|
||||
#### Scenario: Default configuration behaves exactly as before
|
||||
- **WHEN** a Host Agent starts with no
|
||||
`HOST_AGENT_DEPENDENCY_SUPERVISOR_ENABLED` or related Appium environment
|
||||
variable set
|
||||
- **THEN** the Host Agent does not attempt to connect to, probe, or spawn
|
||||
Appium, and its heartbeat/claim behavior is unchanged
|
||||
|
||||
#### Scenario: Top-level flag on and Appium flag off
|
||||
- **WHEN** `HOST_AGENT_DEPENDENCY_SUPERVISOR_ENABLED=true` and
|
||||
`HOST_AGENT_APPIUM_SUPERVISED=false`
|
||||
- **THEN** the Host Agent does not probe, adopt, or spawn Appium
|
||||
|
||||
### Requirement: Spawn supervised dependencies that are not already running
|
||||
The Host Agent SHALL spawn Appium as a child process when Appium supervision is
|
||||
enabled and no healthy Appium instance is adopted, via
|
||||
`appium --address <host> --port <port>` and SHALL forward the child
|
||||
process's stdout/stderr into the Host Agent's own logging, tagged by dependency
|
||||
name.
|
||||
|
||||
#### Scenario: Appium is not running at Host Agent startup
|
||||
- **WHEN** Appium supervision is enabled and no healthy Appium instance is
|
||||
already listening
|
||||
- **THEN** the Host Agent spawns Appium before proceeding to its first
|
||||
device-connect attempt, and its output is visible in Host Agent logs
|
||||
|
||||
#### Scenario: Spawn fails because the executable is missing
|
||||
- **WHEN** the Host Agent attempts to spawn Appium but `appium` is not found
|
||||
on `PATH`
|
||||
- **THEN** the Host Agent logs a dependency-supervisor-specific startup error
|
||||
naming the missing dependency, distinct from a runtime crash of an
|
||||
already-started process
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Runtime supervision settings are retired
|
||||
The Host Agent SHALL reject `HOST_AGENT_RUNTIME_SUPERVISED`,
|
||||
`HOST_AGENT_RUNTIME_HOST`, and `HOST_AGENT_RUNTIME_PORT` because the
|
||||
standalone Runtime service no longer exists.
|
||||
|
||||
#### Scenario: A legacy Runtime supervision variable is set
|
||||
- **WHEN** startup configuration includes any removed Runtime supervision
|
||||
variable
|
||||
- **THEN** configuration fails with an actionable migration error
|
||||
+58
-14
@@ -1,30 +1,74 @@
|
||||
## ADDED Requirements
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Host Agent records step-level execution detail for its in-process TaskRunner
|
||||
The Host Agent SHALL construct its in-process `TaskRunner` with a durable metadata store and timeline so that every step transition (status, index, the actual prompt submitted to the LLM for that step, the model's resulting decision, result, and screenshot when captured) is persisted as it happens, rather than discarded when the assignment completes. The persisted prompt SHALL be the prompt actually sent to the LLM for that specific step, not the task's overall goal.
|
||||
The Host Agent SHALL construct its in-process `TaskRunner` with a durable
|
||||
metadata store and Timeline. `TaskRunner.run()` SHALL create the task's
|
||||
metadata row idempotently before its first status update, so every execution
|
||||
path persists its task status and evidence rather than discarding updates for a
|
||||
missing row. Every completed step SHALL retain its index, actual per-step LLM
|
||||
prompt and decision when available, tool call, result, distinct before/after
|
||||
screenshots when captured, raw OCR observations when available, and normalized
|
||||
UI-tree result when the invoked tool returned one.
|
||||
|
||||
#### Scenario: A goal assignment starts execution
|
||||
- **WHEN** the Host Agent's `AssignmentExecutor` invokes its `TaskRunner`
|
||||
- **THEN** the task metadata row exists before the runner records its running
|
||||
status
|
||||
|
||||
#### Scenario: A step completes during goal execution
|
||||
- **WHEN** the Host Agent's `TaskRunner` completes a step while executing an assigned goal
|
||||
- **THEN** the step's status, index, the actual per-step LLM prompt and response, tool call, result, and any captured screenshot are persisted to the Host Agent's local task metadata store and timeline before the next step begins
|
||||
- **WHEN** the Host Agent's `TaskRunner` completes a step while executing an
|
||||
assigned goal
|
||||
- **THEN** the step's status, index, actual per-step LLM prompt and response,
|
||||
tool call, result, and available evidence are persisted before the next step
|
||||
begins
|
||||
|
||||
#### Scenario: A workflow creates a planned-goal task
|
||||
- **WHEN** a `WorkflowRunner` invokes a Host Agent-configured
|
||||
`TaskRunner` for a planned-goal step
|
||||
- **THEN** that task is persisted without requiring the workflow caller to
|
||||
create a metadata row separately
|
||||
|
||||
#### Scenario: An assignment finishes
|
||||
- **WHEN** an assignment reaches a terminal state (succeeded or failed)
|
||||
- **THEN** its full step history remains queryable from the Host Agent's local store after the in-memory `Task` object is discarded
|
||||
- **THEN** its full step history remains queryable from the Host Agent's local
|
||||
store after the in-memory `Task` object is discarded
|
||||
|
||||
### Requirement: Host-Agent-local task history is retained within a bounded window
|
||||
The Host Agent SHALL prune persisted task metadata, timeline records, and associated screenshot artifacts once they exceed a configurable retention window or count, so that indefinite process uptime does not cause unbounded local disk growth.
|
||||
The Host Agent SHALL prune persisted task metadata, Timeline records, and
|
||||
associated screenshot artifacts once they exceed a configurable retention
|
||||
window or count, so that indefinite process uptime does not cause unbounded
|
||||
local disk growth.
|
||||
|
||||
#### Scenario: Retention window is exceeded
|
||||
- **WHEN** a persisted task's age or position exceeds the configured retention threshold
|
||||
- **THEN** the Host Agent removes that task's metadata row, timeline records, and screenshot artifacts from local storage
|
||||
- **WHEN** a persisted task's age or position exceeds the configured retention
|
||||
threshold
|
||||
- **THEN** the Host Agent removes that task's metadata row, Timeline records,
|
||||
and screenshot artifacts from local storage
|
||||
|
||||
#### Scenario: Retention has not been exceeded
|
||||
- **WHEN** a persisted task is within the configured retention threshold
|
||||
- **THEN** its metadata, timeline records, and screenshot artifacts remain available for query
|
||||
- **THEN** its metadata, Timeline records, and screenshot artifacts remain
|
||||
available for query
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Host Agent correlates local execution records with Cloud assignments
|
||||
For a Cloud-dispatched goal assignment, the Host Agent SHALL persist the Cloud
|
||||
task ID and attempt alongside its generated local Runtime task ID before
|
||||
execution starts. The correlation fields SHALL remain optional and generic in
|
||||
the shared storage layer.
|
||||
|
||||
#### Scenario: A Cloud goal assignment begins
|
||||
- **WHEN** the Host Agent begins executing a Cloud goal assignment
|
||||
- **THEN** the local task row records that assignment's Cloud task ID and
|
||||
attempt
|
||||
|
||||
#### Scenario: A task is not Cloud-dispatched
|
||||
- **WHEN** a shared Runtime caller executes a task without Host/Cloud
|
||||
assignment context
|
||||
- **THEN** the task metadata row is created and the optional source
|
||||
correlation fields remain empty
|
||||
|
||||
## REMOVED Requirements
|
||||
|
||||
### Requirement: Host Agent local task storage is isolated from an unrelated local Runtime
|
||||
The Host Agent SHALL use a configurable, Host-Agent-specific database and artifact path for its task metadata store and timeline, distinct from any local Runtime API's own task storage path, so that the two processes cannot silently collide or share state when run on the same machine.
|
||||
|
||||
#### Scenario: Host Agent and local Runtime run on the same machine
|
||||
- **WHEN** both a Host Agent process and a local Runtime API process run on the same machine with their default configurations
|
||||
- **THEN** each process reads and writes its own task metadata store and timeline without observing or modifying the other's data
|
||||
|
||||
+33
@@ -0,0 +1,33 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Runtime is not exposed as a standalone REST service or web console
|
||||
The repository SHALL not ship a standalone Runtime REST application, its
|
||||
unauthenticated web console, or JSON console routes. The shared Runtime and
|
||||
storage packages SHALL remain reusable execution libraries for the Host Agent
|
||||
and other in-process callers.
|
||||
|
||||
#### Scenario: An operator needs to inspect a Host-executed task
|
||||
- **WHEN** an operator needs task evidence for a Host Agent execution
|
||||
- **THEN** the operator uses the authenticated Host Agent console rather than
|
||||
starting or querying a separate Runtime service
|
||||
|
||||
#### Scenario: A package uses shared Runtime execution
|
||||
- **WHEN** the Host Agent or another in-process caller creates a
|
||||
`TaskRunner`
|
||||
- **THEN** it continues to use the shared Runtime and storage packages without
|
||||
importing a REST or UI adapter
|
||||
|
||||
### Requirement: Host Agent does not supervise a retired Runtime service
|
||||
The Host Agent SHALL not expose Runtime-supervision configuration or spawn a
|
||||
Runtime REST subprocess. It MAY continue to optionally supervise Appium.
|
||||
|
||||
#### Scenario: Host Agent dependency supervision is enabled
|
||||
- **WHEN** `HOST_AGENT_DEPENDENCY_SUPERVISOR_ENABLED=true` and Appium
|
||||
supervision is enabled
|
||||
- **THEN** the Host Agent probes and supervises Appium only
|
||||
|
||||
#### Scenario: A removed Runtime-supervision variable is configured
|
||||
- **WHEN** a Host Agent configuration includes a removed
|
||||
`HOST_AGENT_RUNTIME_*` variable
|
||||
- **THEN** startup fails with a message directing the operator to the Host
|
||||
Agent console and Appium-only supervision
|
||||
+39
-16
@@ -1,38 +1,61 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Runtime persists complete evidence for each executed action
|
||||
The Runtime SHALL persist, for each action it attempts, a screenshot captured immediately before the executor call, the action description and arguments, the execution result, and a screenshot captured immediately after the executor call. Existing timeline records that contain only the legacy single screenshot SHALL remain readable, with that screenshot treated as the post-action image.
|
||||
The shared Runtime SHALL persist, for each action it attempts, a screenshot
|
||||
captured immediately before the executor call, the action description and
|
||||
arguments, the execution result, and a screenshot captured immediately after
|
||||
the executor call. Existing Timeline records that contain only the legacy
|
||||
single screenshot SHALL remain readable, with that screenshot treated as the
|
||||
post-action image.
|
||||
|
||||
#### Scenario: An action succeeds
|
||||
- **WHEN** the Runtime executes an action for a task
|
||||
- **THEN** its timeline record includes distinct before and after screenshots, the action detail, and the execution result
|
||||
- **THEN** its Timeline record includes distinct before and after screenshots,
|
||||
action detail, and execution result
|
||||
|
||||
#### Scenario: An action fails
|
||||
- **WHEN** the Runtime executor exhausts its retries for an action
|
||||
- **THEN** the action's timeline record still includes any screenshots that were captured and the failure result before the task is marked failed
|
||||
- **THEN** the Timeline record still includes any captured screenshots and the
|
||||
failure result before the task is marked failed
|
||||
|
||||
#### Scenario: A legacy timeline record is read
|
||||
- **WHEN** a timeline record has only the prior `screenshot_path` field
|
||||
- **THEN** the Runtime exposes it as the post-action screenshot without failing to render the record
|
||||
#### Scenario: A legacy Timeline record is read
|
||||
- **WHEN** a Timeline record has only the prior `screenshot_path` field
|
||||
- **THEN** the Runtime exposes it as the post-action screenshot without
|
||||
failing to render the record
|
||||
|
||||
### Requirement: Runtime task evidence exposes available OCR observations
|
||||
The Runtime SHALL persist raw OCR observations associated with the scene used to plan an action when available, without adding duplicate OCR data to the LLM-facing normalized Scene payload. The Runtime task-detail UI SHALL render available OCR text, confidence, and bounds, and SHALL render normally when no OCR result exists.
|
||||
### Requirement: Runtime task evidence retains available OCR observations
|
||||
The shared Runtime SHALL persist raw OCR observations associated with the scene
|
||||
used to plan an action when available, without adding duplicate OCR data to the
|
||||
LLM-facing normalized Scene payload. The Host Agent task-detail UI SHALL render
|
||||
available OCR text, confidence, and bounds, and SHALL render normally when no
|
||||
OCR result exists.
|
||||
|
||||
#### Scenario: OCR found text while planning an action
|
||||
- **WHEN** perception produced one or more OCR observations for the action's planning scene
|
||||
- **THEN** the corresponding timeline record includes those observations and the Runtime task-detail UI displays them
|
||||
- **WHEN** perception produced one or more OCR observations for the action's
|
||||
planning scene
|
||||
- **THEN** the corresponding Timeline record includes those observations and
|
||||
the Host Agent task-detail page displays them
|
||||
|
||||
#### Scenario: OCR was unavailable or found no text
|
||||
- **WHEN** perception yields no OCR observations
|
||||
- **THEN** the Runtime records the action evidence and renders the task detail without an OCR result list
|
||||
- **THEN** the Runtime records the action evidence and the Host Agent task
|
||||
detail renders without an OCR result list
|
||||
|
||||
### Requirement: Runtime task evidence renders UI-tree inspection results
|
||||
When a Runtime step invokes the existing `get_ui_tree` or `ui_tree` tool and the persisted result contains normalized UI nodes, the Runtime task-detail UI SHALL render those nodes in a structured, collapsible view while retaining the recorded JSON result. The Runtime SHALL NOT change the tool's response contract or duplicate the result in a separate persistence field.
|
||||
### Requirement: Runtime task evidence retains UI-tree inspection results
|
||||
The Runtime SHALL retain a UI-tree inspection result when a step invokes the
|
||||
existing `get_ui_tree` or `ui_tree` tool and the result contains normalized
|
||||
nodes. The Host Agent task-detail UI SHALL render those nodes in a structured,
|
||||
collapsible view while retaining the recorded JSON result. The Runtime SHALL
|
||||
NOT change the tool response contract or duplicate the result in a separate
|
||||
persistence field.
|
||||
|
||||
#### Scenario: UI-tree inspection succeeds
|
||||
- **WHEN** a task step uses `get_ui_tree` or `ui_tree` and returns one or more normalized nodes
|
||||
- **THEN** the task-detail UI displays each node's type, visible text or identifier, bounds, and available confidence
|
||||
- **WHEN** a task step uses `get_ui_tree` or `ui_tree` and returns one or
|
||||
more normalized nodes
|
||||
- **THEN** the Host Agent task-detail page displays each node's type, visible
|
||||
text or identifier, bounds, and available confidence
|
||||
|
||||
#### Scenario: A non-UI-tree step is displayed
|
||||
- **WHEN** a task step did not invoke a UI-tree tool
|
||||
- **THEN** the task-detail UI does not render an empty UI-tree section
|
||||
- **THEN** the Host Agent task-detail page does not render an empty UI-tree
|
||||
section
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
## 1. Host Agent local task storage (D1, D2)
|
||||
|
||||
- [x] 1.1 Add `HOST_AGENT_TASK_PROGRESS_DB_PATH` (and matching artifact directory config) to `HostAgentConfig`/`load_host_agent_config()`, with a Host-Agent-specific default distinct from any local Runtime `tasks/` path.
|
||||
- [x] 1.1 Add `HOST_AGENT_TASK_PROGRESS_DB_PATH` (and matching artifact directory config) to `HostAgentConfig`/`load_host_agent_config()`, with Host-Agent-specific defaults for durable execution history.
|
||||
- [x] 1.2 In `HostAgentApplication`'s builder (`app.py:218-223`), construct a `TaskMetadataStore`/`Timeline` from that config and pass them into `create_execution_factories(..., metadata_store=..., timeline=...)`.
|
||||
- [x] 1.3 Add delete/prune methods to `storage/task_metadata.py::TaskMetadataStore` and `storage/timeline.py::Timeline` (remove a task's row, timeline records, and screenshot artifacts).
|
||||
- [x] 1.4 Implement a bounded retention pass in the Host Agent (default: keep the newer of "last 50 tasks" or "7 days", whichever keeps fewer rows) that runs after each assignment finishes, calling the new prune methods.
|
||||
- [x] 1.5 Add unit tests: step transitions persist during execution; a task's history is queryable after the assignment completes and the in-memory `Task` is discarded; retention prunes tasks beyond the configured threshold; Host Agent and a local Runtime process using default paths on the same machine do not collide.
|
||||
- [x] 1.5 Add unit tests: step transitions persist during execution; a task's history is queryable after the assignment completes and the in-memory `Task` is discarded; retention prunes tasks beyond the configured threshold; Host Agent task metadata and artifacts use their configured local paths.
|
||||
|
||||
## 2. Host Agent reports progress during lease renewal (D4)
|
||||
|
||||
@@ -31,14 +31,14 @@
|
||||
## 5. Host Agent local console task pages (D3)
|
||||
|
||||
- [x] 5.1 Extend `AgentStatusTracker`/`/api/status` (or a small addition alongside it) to surface the current step index/status/summary for the in-progress assignment, sourced from the same progress holder built in section 2, and update the dashboard's "Current assignment" rendering to show it.
|
||||
- [x] 5.2 Add authenticated, read-only task list and task detail/timeline routes to `host_agent/web/app.py`, querying the Host-Agent-local `TaskMetadataStore`/`Timeline` (reusing the same query calls `api/console.py` makes) and rendering server-side HTML consistent with the existing dashboard's f-string + `html.escape()` convention, including inlined screenshots on the detail/timeline page.
|
||||
- [x] 5.2 Add authenticated, read-only task list and task detail/timeline routes to `host_agent/web/app.py`, querying the Host-Agent-local `TaskMetadataStore`/`Timeline` and rendering server-side Jinja HTML consistent with the existing console, including inlined screenshots on the detail/timeline page.
|
||||
- [x] 5.3 Gate the new routes behind the existing Host Agent console session/CSRF protection; verify no new CORS configuration is introduced.
|
||||
- [x] 5.4 Add tests: unauthenticated requests to the new routes are rejected the same way as other console routes; task list/detail/timeline pages render expected data including screenshots for a completed task.
|
||||
|
||||
## 6. Documentation and verification
|
||||
|
||||
- [x] 6.1 Update `docs/MACOS_IPHONE_SETUP.md` and/or `docs/CLOUD_DEPLOYMENT.md` with the new Host Agent config vars (`HOST_AGENT_TASK_PROGRESS_DB_PATH` and retention settings) and a short note on where to view live/historical task progress in each console.
|
||||
- [x] 6.2 Run `uv run --all-packages pytest -m "not integration"` and targeted Cloud API / Host Agent test suites; run `cloud-console/` and `console/`-equivalent Vitest suites for the touched frontend.
|
||||
- [x] 6.2 Run `uv run --all-packages pytest -m "not integration"` and targeted Cloud API / Host Agent test suites; run the Cloud Console Vitest suite for its touched frontend.
|
||||
- [x] 6.3 Run Ruff check/format and `compileall` across touched packages.
|
||||
- [x] 6.4 Run `openspec validate --strict` for this change.
|
||||
- [ ] 6.5 Manual verification (requires a real Host Agent + Appium/device setup per `docs/MACOS_IPHONE_SETUP.md`): run a real task end-to-end and confirm step progress appears live in the Host Agent console and Cloud Console, and that full step history with screenshots is browsable afterward in the Host Agent console.
|
||||
@@ -48,7 +48,7 @@
|
||||
- [x] 7.1 Extend `ToolCallDecision` (`runtime/tool_calling_client.py:23-27`) with the actual prompt content given to that call (or return it via a small side-channel from `AIPlanner.plan()`, not by changing the `Planner.plan()` return type used by other planners).
|
||||
- [x] 7.2 Update `TaskRunner._append_timeline()` (`runtime/task.py:296-319`) to pass the real per-step prompt and the model's resulting decision into `Timeline.append()`, instead of `task.goal`.
|
||||
- [x] 7.3 Update `storage/timeline.py`'s `Timeline`/`TimelineRecord` field(s) so the persisted meaning is unambiguously "the prompt actually sent to the LLM for this step" (rename or add a field; keep backward-compatible read of any already-persisted rows if a Runtime dev DB might already have old-shaped rows).
|
||||
- [x] 7.4 Update any renderer of this data (`api/console.py`, Runtime `console/`, and the new Host Agent console task pages from section 5) to show the corrected field.
|
||||
- [x] 7.4 Update the Host Agent console task pages to show the corrected field without requiring a separate Runtime renderer.
|
||||
- [x] 7.5 Add unit tests: a persisted step's prompt matches what `AIPlanner.plan()` actually sent for that step (not the task goal), across at least one multi-step task.
|
||||
|
||||
## 8. Cloud persists full per-step LLM decision history via the existing cloud-planner-proxy endpoint (D7, D8)
|
||||
@@ -67,10 +67,20 @@
|
||||
- [x] 9.3 When a task's host used the `direct` transport (no persisted decisions and the host's configured transport is known to be `direct`), show an explicit "not reported by this host's transport" state rather than an empty list.
|
||||
- [x] 9.4 Add Vitest coverage for populated history, empty-but-cloud-transport (task hasn't produced any decisions yet), and direct-transport-hidden cases.
|
||||
|
||||
## 10. Runtime per-step evidence (D10)
|
||||
## 10. Shared Runtime per-step evidence (D4)
|
||||
|
||||
- [x] 10.1 Extend the shared Scene/Timeline/ArtifactStore model to retain raw OCR observations and separate before/after screenshots while preserving compatibility with existing single-screenshot records.
|
||||
- [x] 10.2 Capture before and after screenshots around every TaskRunner executor call, then persist the action, result, and available OCR observations in the same timeline record.
|
||||
- [x] 10.3 Extend the Runtime console JSON API and task-detail UI to render before/after screenshots, operation details, and available OCR results.
|
||||
- [x] 10.4 Render persisted normalized UI-tree tool results in the Runtime task-detail UI without changing the tool contract or duplicating stored data.
|
||||
- [x] 10.5 Add focused Timeline, TaskRunner, perception, JSON API, Runtime UI, and UI-tree regression coverage; run the relevant format, lint, test, and strict OpenSpec validation commands.
|
||||
- [x] 10.3 Preserve the shared Timeline representation needed to render before/after screenshots, operation details, and available OCR results without changing Runtime execution contracts.
|
||||
- [x] 10.4 Preserve persisted normalized UI-tree tool results without changing the tool contract or duplicating stored data.
|
||||
- [x] 10.5 Add focused Timeline, TaskRunner, perception, and UI-tree regression coverage; run the relevant format, lint, test, and strict OpenSpec validation commands.
|
||||
|
||||
## 11. Host Agent execution authority and Runtime service retirement (D1-D9)
|
||||
|
||||
- [x] 11.1 Make `TaskRunner.run()` create a task metadata row idempotently before status updates; add optional generic source task/attempt fields to task metadata and record Cloud assignment correlation in the Host adapter.
|
||||
- [x] 11.2 Add regression coverage for Host-dispatched goal execution, workflow-owned TaskRunner execution, retry-safe metadata creation, and Cloud task/attempt correlation.
|
||||
- [x] 11.3 Extend Host Agent `/tasks` and task-detail rendering so the actual execution list and detail page show correlation, before/after screenshots, operation detail, OCR observations, and structured normalized UI-tree results, including legacy screenshot compatibility.
|
||||
- [x] 11.4 Remove the standalone Runtime REST service/UI, its package data and dedicated tests, and remove Runtime supervision from Host Agent configuration/supervision with an actionable legacy-config error.
|
||||
- [x] 11.5 Update operator documentation and OpenSpec artifacts to direct execution inspection to Host Agent `:8765/tasks` and remove port `8000` instructions.
|
||||
- [x] 11.6 Run focused Host Agent and shared Runtime tests, workspace non-integration tests, Ruff, compileall, and strict OpenSpec validation.
|
||||
- [ ] 11.7 Manual verification (requires a real Host Agent + Appium/device setup): submit or dispatch a task, then confirm the Host Agent console is the only local execution-history UI and shows the complete retained evidence.
|
||||
|
||||
Reference in New Issue
Block a user