This commit is contained in:
@@ -1,137 +1,176 @@
|
||||
## Context
|
||||
|
||||
Three surfaces currently cannot show a task's in-progress status:
|
||||
Host Agent executes Cloud assignments in-process through the shared
|
||||
`runtime.TaskRunner`. Its `HostAgentApplication` already constructs a
|
||||
Host-local `TaskMetadataStore` and `Timeline` and injects them into the
|
||||
runner factory. The missing link is task creation: `AssignmentExecutor`
|
||||
constructs a `Task`, calls `runner.run(task)`, and `TaskRunner` only
|
||||
issues updates. SQLite therefore receives updates for a row that does not
|
||||
exist, so the Host Agent `/tasks` UI is empty.
|
||||
|
||||
1. **Host Agent local console** (`apps/device-host-agent/host_agent/web/app.py`): `HostAgentApplication`'s builder calls `create_execution_factories(resolved_manager, host_agent_config=resolved_config)` without `metadata_store`/`timeline` (`app.py:218-223`), so the in-process `TaskRunner` (`execution.py:39-46`) runs with `metadata_store=None, timeline=None`. `runtime/task.py::TaskRunner._update_task()` only persists when `metadata_store` is truthy, so step transitions vanish. The console's only signal is `AgentStatusTracker.snapshot()` (`status.py:14-88`), a single opaque span: `task_id/device_id/goal/started_at`, cleared on `mark_assignment_finished()`.
|
||||
2. **Cloud Control Plane / Cloud Console**: the internal protocol (`packages/cloud-platform/cloud/internal_api/models.py`) only has claim, heartbeat, lease-renewal, and terminal-result messages. Lease renewal (`ActiveAssignmentRunner._renew_while_running`, `apps/device-host-agent/host_agent/lease.py:79-106`) already fires periodically (~1/3 of remaining lease) for every in-flight assignment, but carries no task-progress payload.
|
||||
3. **Runtime `console/` SPA**: `api/rest.py::create_app()` builds its own `TaskMetadataStore`/`Timeline`/`TaskRunner` (`api/rest.py:24-41`), queried by `api/console.py::create_console_router()`'s `/console/tasks*` routes, which `console/src/api.ts` calls against `API_BASE_URL` (default `http://127.0.0.1:8000`). This is a wholly separate process from Host Agent, with no shared store or IPC. `runtime/`-owned packages are forbidden from importing host/cloud concerns (`test_runtime_owned_packages_do_not_import_host_or_cloud_concerns`), and `host-agent-local-console/design.md` already treats Host Agent's local history as "not a system of record," distinct from Cloud's.
|
||||
The separate `api.rest` process was never on this execution path. It owned a
|
||||
different database and artifact root, so its REST endpoints and UI could only
|
||||
show tasks submitted directly to that unrelated process. Running it alongside
|
||||
a Host Agent created two competing operator entry points without transferring
|
||||
any history between them.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Host Agent's local console shows live step-level progress for its current assignment (step index, step status, short summary), not just a static started-at snapshot.
|
||||
- Cloud Control Plane learns step-level progress near-real-time (piggybacked on existing periodic traffic, not a new polling loop) and Cloud Console renders it for an operator watching a remote task.
|
||||
- An operator can inspect a Host-Agent-executed task's progress/history from a web console, without coupling `runtime/` to host/cloud concerns.
|
||||
- Reuse existing, already-tested building blocks (`TaskMetadataStore`, `Timeline`, `api/console.py`'s query shape, the lease-renewal cadence) instead of inventing new storage or transport primitives.
|
||||
- Both the Host Agent's local task history and the Cloud Control Plane persist the *actual* per-step content sent to and received from the LLM (the real prompt text, not just the task's overall goal; the model's resulting decision, not just the parsed tool call) — so a step's record answers "what did we actually ask the model and what did it say," not just "what call did we make." Cloud's copy is the durable, centrally-searchable record for troubleshooting; the Host Agent's local copy remains the on-device record of what actually ran.
|
||||
|
||||
- Make durable task-row creation an invariant of `TaskRunner.run()` whenever
|
||||
a metadata store is configured.
|
||||
- Correlate a Host-local goal execution with the Cloud task ID and attempt that
|
||||
caused it, without making shared Runtime or storage packages import Cloud
|
||||
models.
|
||||
- Make the authenticated Host Agent console at port `8765` the authoritative
|
||||
web view for actual task execution: task list, detail, live status, before
|
||||
screenshot, operation, after screenshot, OCR observations, and normalized
|
||||
UI-tree results.
|
||||
- Preserve the shared Runtime Timeline as the evidence model and keep its
|
||||
before/after screenshot, OCR, and UI-tree capture behavior.
|
||||
- Retire the standalone Runtime REST service/UI and its Host Agent supervisor
|
||||
configuration while retaining Runtime, storage, MCP, and skill-sync library
|
||||
modules.
|
||||
- Retain the existing Cloud latest-progress and Cloud-proxy planner-decision
|
||||
history behavior. Cloud remains a fleet-level view and never stores
|
||||
screenshots.
|
||||
|
||||
**Non-Goals:**
|
||||
- No SSE/WebSocket push. All three surfaces keep polling (Host Agent console 5s, Cloud Console/`cloud-console` and `console/` at their existing intervals). Nothing here requires push infra, and adding it would be a bigger, separate change.
|
||||
- The coarse live step index/status/summary reported via lease renewal (D4/D5) stays *latest-snapshot-only* on the Cloud side (overwrite semantics) — it is a "what's happening right now" indicator, not a history mechanism, and is unaffected by the LLM-content decisions below.
|
||||
- Full step-by-step **LLM interaction** history *is* now synced to and durably persisted by Cloud (see D7/D8) — this is a deliberate reversal of this change's earlier draft, which had scoped Cloud to latest-only. The motivation: Host Agent deployments already route every AI Planner decision through Cloud API's existing `cloud-planner-proxy` endpoint when configured for the `cloud` transport, so Cloud is already on the natural path for this data and centralizing it there (rather than only on whichever edge host happens to still be running) is the more useful place for an operator doing centralized troubleshooting. This durable log is scoped strictly to LLM prompt/response text — it is not a general-purpose duplicate of the Host Agent's full `Timeline` (no screenshots, no scene JSON dumps beyond what's embedded in the prompt text itself).
|
||||
- No multi-backend Runtime console or cross-origin Host Agent integration. Runtime changes remain local to its existing same-origin API/UI and shared timeline evidence model.
|
||||
- No change to `driver/`/`device/`/`core/` device-control internals.
|
||||
- No screenshot or full-scene data leaves the Host Agent process as part of progress reporting, and no screenshot bytes are ever persisted by the Cloud Control Plane. (Full LLM prompt/response *text* is, by contrast, an explicit Goal below — see D7/D8 — which partially supersedes the original `cloud-planner-proxy` proposal's "does not durably persist ... full prompt text" statement. Screenshots remain excluded; prompts no longer are.)
|
||||
|
||||
- No SSE/WebSocket push; Host and Cloud consoles keep their existing polling.
|
||||
- No duplicate full evidence upload to Cloud. Screenshots and Timeline
|
||||
artifacts remain Host-local.
|
||||
- No new Host/Cloud dependencies in `runtime/`, `storage/`, `driver/`, or
|
||||
`device/`.
|
||||
- No replacement general-purpose device-control REST API. Operators use Host
|
||||
Agent device management and task pages for the managed execution workflow.
|
||||
|
||||
## Decisions
|
||||
|
||||
### D1: Wire a real `metadata_store`/`timeline` into Host Agent's `TaskRunner`, reusing the existing classes
|
||||
### D1: TaskRunner owns idempotent task metadata creation
|
||||
|
||||
`create_execution_factories()` already accepts `metadata_store: TaskMetadataStore | None` and `timeline: Timeline | None` (`execution.py:28-34`) — these are the same classes `api/rest.py` uses for the Runtime's own console. Reuse them as-is rather than inventing a Host-Agent-specific store: `HostAgentApplication`'s builder constructs a `TaskMetadataStore(db_path=<host-agent-local path>)` and `Timeline(ArtifactStore(<host-agent-local path>))` and passes them into `create_execution_factories()`.
|
||||
`TaskRunner.run()` will call `TaskMetadataStore.create_task(task)` before
|
||||
the first running-state update. `create_task()` will use idempotent insert
|
||||
semantics so callers that already created a direct Runtime task remain
|
||||
compatible and workflow-created tasks are captured automatically.
|
||||
|
||||
**Alternative considered**: a bespoke in-memory-only step recorder (cheaper, no disk I/O). Rejected because `TaskMetadataStore`/`Timeline` are already proven in production (Runtime's own console has run on them since the original `web-console` change) and reusing them means Host Agent's task/timeline data has the *exact* same shape as `api/console.py` already serializes — a prerequisite for D3 below. A bespoke format would need its own (de)serialization and its own console rendering code for no real benefit.
|
||||
This places the invariant at the shared execution boundary rather than relying
|
||||
on every caller to remember an out-of-band persistence call. It fixes goal
|
||||
assignments and prevents the same failure for `WorkflowRunner` planned-goal
|
||||
steps.
|
||||
|
||||
New config: `HOST_AGENT_TASK_PROGRESS_DB_PATH` (default e.g. `host_agent_data/task_progress.sqlite3`) and a matching artifact directory, kept separate from any local Runtime `tasks/` directory that might exist on the same machine, to avoid two unrelated processes silently sharing or colliding on a path.
|
||||
### D2: Host assignment correlation stays in the Host adapter
|
||||
|
||||
### D2: Bounded retention for the Host-Agent-local store
|
||||
Before executing a Cloud goal assignment, `AssignmentExecutor` creates the
|
||||
local task record with optional generic `source_task_id` and
|
||||
`source_attempt` metadata. Shared storage uses generic names and does not
|
||||
import Cloud types. The Host Agent task list/detail renders those fields as the
|
||||
Cloud task ID and attempt.
|
||||
|
||||
Unlike a Runtime dev session (short-lived, manually cleared), a Host Agent process runs indefinitely and executes many assignments over its lifetime. Unbounded `TaskMetadataStore` rows and `Timeline` screenshot artifacts would grow without limit.
|
||||
The local `Task.id` remains generated by the Runtime. Reusing the Cloud task
|
||||
ID as an artifact directory name would make retries overwrite each other and
|
||||
would admit unsafe path characters from an external identifier.
|
||||
|
||||
Decision: add a lightweight retention pass (e.g. on a timer, or opportunistically after each assignment finishes) that prunes tasks older than a configurable window or beyond a configurable count, deleting both the `tasks` row and its `Timeline`/`ArtifactStore` files. This is new logic — neither `TaskMetadataStore` nor `Timeline` currently supports deletion — scoped as a small addition to `storage/task_metadata.py` / `storage/timeline.py` (or a Host-Agent-side wrapper if adding delete methods to the shared `storage` package feels too broad; prefer extending `storage` since both Runtime and Host Agent benefit from bounded retention).
|
||||
### D3: Host Agent console is the authoritative evidence UI
|
||||
|
||||
### D3: Give Host Agent's own console read-only task/timeline pages instead of making `console/` multi-backend
|
||||
The Host Agent already owns the device manager, assignment executor,
|
||||
metadata store, Timeline, local account, session, and CSRF boundary. Its
|
||||
same-origin `/tasks` and `/tasks/{task_id}` pages therefore render the
|
||||
shared Timeline directly. The task list is named for Host executions rather
|
||||
than "Local Runtime tasks", and submission feedback tells the operator that
|
||||
the submitted Cloud task appears there when this Host begins execution.
|
||||
|
||||
Two options were evaluated for capability `host-agent-console-task-pages`:
|
||||
No browser needs to point a separate frontend at the Host Agent. This keeps
|
||||
the conservative local-account/session model and does not add CORS.
|
||||
|
||||
- **(a) Multi-backend Runtime `console/` SPA**: let the existing Vue SPA point at a Host Agent's console origin (a saved/selectable base URL) in addition to the local Runtime. Since D1 makes Host Agent's data shape identical to what `api/console.py` already serializes, the SPA's existing fetch/render code would work unmodified against a Host Agent origin.
|
||||
- **(b) Extend Host Agent's own server-rendered console** (`host_agent/web/app.py`) with new read-only pages for task list/detail/timeline, reusing `TaskMetadataStore`/`Timeline` query calls directly (the same calls `api/console.py`'s handlers make), rendered as server-side HTML via the existing f-string + `html.escape()` convention (no SPA, no frontend build) established by `host-agent-local-console`.
|
||||
### D4: Complete per-step evidence is rendered by the Host Agent
|
||||
|
||||
**Decision: (b).** (a) requires Host Agent's console to serve cross-origin, credentialed requests from whatever origin `console/` is running on (Vite dev server, or a different deployed origin) — meaning either relaxing Host Agent's CORS posture for its session-cookie+CSRF-protected endpoints, or reworking its auth model to tolerate cross-origin fetches. That is real new attack surface on a console whose entire local-only threat model (`console_bind_host` defaulting to loopback, explicit opt-in for non-loopback) was deliberately conservative. (b) reuses an already-accepted pattern (server-rendered, same-origin, same auth) and fully satisfies the actual operator need — a web page showing what a Host Agent is doing — without touching `runtime/`, `console/`, or Host Agent's CORS/auth posture at all. The cost is a small amount of view duplication (Host Agent's console and Runtime's `console/` render similar-shaped data with different templates/frameworks), judged acceptable given they serve different operational contexts (local direct-connect Runtime vs. remote Host Agent fleet).
|
||||
Timeline records retain distinct pre-action and post-action screenshot paths,
|
||||
the action description and arguments, the execution result, raw OCR
|
||||
observations, and the existing normalized UI-tree result. The Host Agent page
|
||||
creates data URIs only for available local artifacts and supports legacy
|
||||
records where the single `screenshot_path` is the post-action image.
|
||||
|
||||
Any future desire for a unified SPA across both surfaces is left as a follow-up, not blocked by this decision (D1's shared data shape keeps that door open).
|
||||
OCR is rendered when present. UI-tree output is rendered only for
|
||||
`get_ui_tree` and `ui_tree` records that contain normalized nodes; it uses a
|
||||
collapsible structured view while retaining the JSON result. No tool contract
|
||||
or duplicate persistence field is introduced.
|
||||
|
||||
### D4: Piggyback progress reporting on the existing lease-renewal call
|
||||
### D5: Host-local retention remains bounded
|
||||
|
||||
`ActiveAssignmentRunner._renew_while_running` (`lease.py:79-106`) already makes an authenticated, per-assignment, periodic call (`client.renew(assignment)`, roughly every 1/3 of remaining lease) while a task executes, validated server-side against `host_id`/`task_id`/`attempt`/`lease_id` (`internal_api/api.py:291-320`). This is the natural piggyback point for progress: extend `LeaseRenewalRequest` with an optional `progress: TaskProgressModel | None` field (new small model: `step_index: int`, `step_status: Literal[...]`, `summary: str` bounded length — no screenshot/scene payload) in `cloud/internal_api/models.py`, the single schema both sides already import directly (no parallel schema, matching existing convention noted for this protocol).
|
||||
The existing Host retention pass continues to remove metadata rows, Timeline
|
||||
records, and artifacts according to the configured count and age thresholds.
|
||||
The new task creation invariant must use that same store so it cannot create
|
||||
an unbounded second history source.
|
||||
|
||||
A small thread-safe "latest progress" holder (similar in spirit to `LeaseGuard`) is written by the execution thread (via a hook on `TaskRunner`/`AssignmentExecutor`, updated once per step) and read by `_renew_while_running` just before each renewal call, so no new timer/cadence is introduced.
|
||||
### D6: Cloud progress remains a latest snapshot on lease renewal
|
||||
|
||||
**Alternative considered**: a dedicated `POST /internal/v1/hosts/{host_id}/assignments/{task_id}/progress` endpoint called on its own cadence. Rejected — it would duplicate the auth/validation `renew_assignment` already does, and introduce a second periodic call where one already exists and fires at a reasonable frequency for this purpose.
|
||||
The Host execution thread writes a bounded latest-progress holder. The lease
|
||||
renewal path optionally carries its step index, status, and summary; Cloud
|
||||
stores only the latest snapshot for an active assignment and stops exposing it
|
||||
after terminal completion. No screenshot or scene payload enters this
|
||||
protocol.
|
||||
|
||||
### D5: Cloud persists only the latest progress snapshot, overwritten alongside lease renewal
|
||||
### D7: Cloud-proxy planner decisions remain durable Cloud history
|
||||
|
||||
`renew_lease()` (`repository.py:407`, `sql_repository.py:1480`) already takes a row lock and writes a lease-expiry update on every renewal; extend it to also accept and store the optional progress fields (few scalar columns — `progress_step_index`, `progress_step_status`, `progress_summary`, `progress_updated_at` — rather than a schema-flexible JSON blob, keeping it queryable and consistent with the rest of the row's typed columns). This requires a new Alembic migration (head is currently `0007_llm_provider_management`) plus the equivalent SQLite schema change, applied to both `repository.py` and `sql_repository.py` to keep the dual-backend contract (`cloud-control-plane` spec's "Deployment and local persistence modes share one contract" requirement).
|
||||
For `AI_PLANNER_TRANSPORT=cloud`, Cloud's existing planner-decision endpoint
|
||||
persists successful system/user prompts, tool calls, arguments, and a
|
||||
task/attempt-scoped step index. The direct transport intentionally produces no
|
||||
such Cloud history. This log has bounded terminal-task retention and never
|
||||
persists request screenshot bytes.
|
||||
|
||||
Progress columns are cleared (or simply superseded and ignored) once a terminal result is recorded — they represent "what's happening right now," not history; Cloud's durable attempt/result history is unaffected and unduplicated.
|
||||
### D8: Retire the standalone Runtime REST service and UI
|
||||
|
||||
### D6: Cloud Console reads progress from the existing task/attempt query path, not a new endpoint
|
||||
Remove `api/rest.py`, `api/console.py`, `api/console_web.py`, their
|
||||
templates/static assets, their package-data declarations, and their
|
||||
service/UI tests. Preserve `api/mcp.py`, `api/errors.py`, skill-sync, and
|
||||
skill-catalog modules because they are independent library integrations.
|
||||
|
||||
Extend whatever response model Cloud Console's task list/detail already uses (Cloud API public router) with the optional latest-progress fields from D5, rather than adding a new endpoint. Cloud Console renders it as a small inline badge/line ("step 4: tapping login button") next to the existing status, refreshed on the SPA's existing polling interval.
|
||||
Remove `HOST_AGENT_RUNTIME_SUPERVISED`,
|
||||
`HOST_AGENT_RUNTIME_HOST`, and `HOST_AGENT_RUNTIME_PORT`. The optional
|
||||
dependency supervisor continues to support Appium only. Configuration with a
|
||||
removed Runtime-supervision variable fails with an actionable migration error
|
||||
instead of silently doing nothing.
|
||||
|
||||
### D7: Capture full LLM interaction history for free via the existing `cloud-planner-proxy` decide endpoint, not a new reporting channel
|
||||
### D9: Documentation points operators to Host Agent
|
||||
|
||||
Investigated the Host Agent → Cloud call path in detail: when a Host Agent is configured with `AI_PLANNER_TRANSPORT=cloud`, `AIPlanner.plan()` (`runtime/ai_planner.py`) calls `CloudProxyToolCallingClient.decide()` (`apps/device-host-agent/host_agent/cloud_planner_client.py:46-95`), which `POST`s the *complete* `system_prompt`/`user_prompt` (plus `task_id`/`attempt`/`lease_id` from `current_planner_execution_context()`) to Cloud API's `/hosts/{host_id}/planner/decide` (`packages/cloud-platform/cloud/internal_api/api.py::decide_planner_call`, line ~367). That handler already resolves and returns a `ToolCallDecision` (`tool_name`/`arguments`/`usage`) and already does per-call bookkeeping (`pool.store.settle_host_token_reservation(...)`, line ~463) using the same repository object this change's D5 already touches for lease renewal.
|
||||
|
||||
This means **every planning step's actual prompt and resulting decision already flows through a Cloud-owned request handler** when the `cloud` transport is used — no new endpoint, no new protocol field, no queue/batching scheme is needed to get full LLM content to Cloud. The only change needed is to make that handler *persist* what it currently discards.
|
||||
|
||||
**Decision**: extend `decide_planner_call` to, immediately after computing `decision` (success path only — a `ToolCallUnavailable`/502 path persists nothing), insert one row into a new log table (D8) keyed by `(task_id, attempt, step_index)`, where `step_index` is assigned by the Cloud side itself (an auto-incrementing counter scoped to `task_id`+`attempt`, e.g. `select count(*) + 1` under the same row lock, or a DB sequence/identity column) — Host Agent does not need to track or send a step counter for this.
|
||||
|
||||
**Explicit limitation, called out rather than papered over**: this only captures LLM content for hosts using the `cloud` transport. A host on the (still-default) `direct` transport never sends its prompts to Cloud at all — Cloud has zero LLM content for that host's tasks, and only ever sees the coarse index/status/summary from D4/D5's lease-renewal piggyback (which is transport-agnostic, since it's driven by `TaskRunner` step completion, not by the planner's transport choice). This is a real operational dependency: centralized LLM-interaction troubleshooting via Cloud Console requires the fleet (or the hosts an operator cares about) to run with `AI_PLANNER_TRANSPORT=cloud`. This change does not make `cloud` the new default transport — that remains a separate, already-existing configuration decision outside this change's scope.
|
||||
|
||||
**Alternative considered**: extend the D4 lease-renewal piggyback to also carry full prompt/response text (queued, not overwritten, so no step is lost between renewals). Rejected: it would duplicate a transport that already exists for exactly this payload (the decide call itself) whenever `cloud` transport is active, and would still need a *separate* new channel for the `direct`-transport case where Cloud never sees the prompt anyway — i.e., it does not actually solve the `direct`-transport gap, so it only adds complexity without expanding coverage.
|
||||
|
||||
### D8: New bounded-retention table for the full per-step LLM decision log, extended on both repository backends
|
||||
|
||||
Add a new table (e.g. `planner_decision_log`): `id`, `host_id`, `task_id`, `attempt`, `step_index`, `system_prompt` (text), `user_prompt` (text), `tool_name`, `arguments_json` (text), `created_at`. No screenshot column — screenshots are never sent to this endpoint's persistence path (the request's `screenshot_base64` is used only to call the LLM provider and is never written to this log, consistent with the Non-Goals screenshot exclusion).
|
||||
|
||||
Like D5, this needs a new Alembic migration and the equivalent SQLite path, implemented on both `repository.py`'s SQLite-backed implementation and `sql_repository.py::SQLAlchemyCloudRepository` to preserve the existing dual-backend contract. Unlike D5's few-nullable-columns-on-an-existing-row approach, this is an independent append-only table (one row per decide call, not an overwrite), since the whole point is durable per-step history rather than a live snapshot.
|
||||
|
||||
**Retention**: this table grows once per planning step across the whole fleet, indefinitely, on a shared multi-tenant Cloud database — unbounded growth is a real risk here in a way D5's single-row-per-assignment overwrite never was. Decision: a scheduled prune job (mirrors D2's Host-Agent-local retention) deletes rows whose owning task reached a terminal state more than a configurable window ago (default: prune 7 days after task terminal, or once the task itself is pruned/archived by whatever existing Cloud task-retention policy applies — reuse that cadence rather than inventing a second one if `cloud-control-plane` already has one; otherwise default to a simple time-based prune).
|
||||
|
||||
### D9: Fix the shared `Timeline`/`TaskRunner`/`AIPlanner` path so the *actual* per-step prompt and response are recorded locally, not just the task goal and the parsed tool call
|
||||
|
||||
Independent of Cloud persistence, the existing local recording is itself wrong today: `TaskRunner._append_timeline()` (`runtime/task.py:296-319`) calls `Timeline.append(prompt=task.goal, tool_call={"action": ..., "description": ..., "args": ...}, ...)`. `task.goal` is the overall task goal, not the per-step prompt actually sent to the LLM — the real per-step prompt (`planner_user_prompt(goal, scene_json, history_summary)`, built in `ai_planner.py::plan()`) is constructed, sent, and discarded entirely within `AIPlanner.plan()`, never reaching `TaskRunner`. Likewise `ToolCallDecision` (`runtime/tool_calling_client.py:23-27`) carries only the parsed `tool_name`/`arguments`/`usage` — any raw response text/content the model returned is discarded during parsing (`_decision_from_anthropic_response`/`_decision_from_openai_response`).
|
||||
|
||||
This is a pre-existing gap in a component shared by Runtime and Host Agent alike (not new to this change), and it undermines the very "step-level detail" goal D1 already committed to — a persisted step whose "prompt" field is just the task's goal repeated on every row is not useful for troubleshooting.
|
||||
|
||||
**Decision**: extend `ToolCallDecision` with the actual `user_prompt`/`system_prompt` it was given (or have `AIPlanner.plan()` return a small side-channel result instead of changing the `Planner` interface's return type) so `TaskRunner._append_timeline()` can pass the real per-step prompt into `Timeline.append()`. Rename `Timeline`/`TimelineRecord`'s `prompt` field's meaning (or add a new field) to unambiguously mean "the prompt actually sent to the LLM for this step." This fix lands in the shared `runtime`/`storage` packages, so both Host Agent's local console (D3) and the existing Runtime `console/` automatically benefit — it is not Host-Agent-specific plumbing.
|
||||
|
||||
### D10: Persist and render before/after action evidence in the Runtime timeline
|
||||
|
||||
The existing Timeline captures only one screenshot after an action, while the planning screenshot used before the action is transient. Extend each Timeline record with distinct before/after screenshot paths, preserving the existing `screenshot_path` as a backward-compatible alias for the after screenshot. `TaskRunner` captures each image immediately before and after invoking the executor, including failed action attempts. The Runtime console API inlines both images and the existing task-detail page renders them beside the recorded action and result.
|
||||
|
||||
OCR is already available during perception but fused into the normalized Scene, where an OCR value that overlaps a UI-tree node can lose its raw provenance. Preserve raw OCR observations on `Scene` for local timeline capture only; keep them out of `Scene.to_dict()` so the LLM-facing planner payload does not grow with duplicated text. The Runtime task-detail page renders the persisted OCR list when it is available and handles legacy records without it.
|
||||
|
||||
The existing `get_ui_tree`/`ui_tree` tool returns a normalized flat list of UI nodes, not the driver-specific raw XML hierarchy. Its StepResult is already persisted in the Timeline. Detect those action names in the Runtime UI and render the returned nodes in a collapsible structured view while retaining the full JSON result below it. This makes the inspection result readable without changing the tool response contract or storing a second tree copy.
|
||||
Operator documentation no longer instructs users to start `uvicorn
|
||||
api.rest:create_app` or browse port `8000`. It identifies the Host Agent
|
||||
console at `http://127.0.0.1:8765/tasks` as the execution-history authority,
|
||||
explains its local authentication, and documents that the Cloud Console is a
|
||||
fleet/progress and Cloud-proxy LLM-history surface rather than a screenshot
|
||||
store.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- [Risk] Per-step SQLite writes in a long-lived Host Agent process add I/O overhead → Mitigation: this is the same write pattern Runtime's own console has always used per step; no new proof of acceptability needed. If profiling later shows it matters for very high step-rate tasks, batching/debouncing is a follow-up, not a blocker here.
|
||||
- [Risk] Unbounded local disk growth from `Timeline` screenshots on an indefinitely-running Host Agent → Mitigation: D2's bounded retention pass; must ship in the same change as D1, not deferred, since D1 alone would otherwise introduce an unbounded-growth regression.
|
||||
- [Risk] New DB columns/migration touch both `repository.py` (SQLite) and `sql_repository.py` (Postgres) — drift between the two has been a real defect category in this codebase (see `cloud-control-plane-integration` archive notes) → Mitigation: contract/parity tests already exist for this dual-backend boundary; extend them to cover the new progress columns.
|
||||
- [Risk] View duplication between Host Agent's server-rendered task pages (D3) and Runtime `console/`'s task pages (different frameworks, same data shape) → Mitigation: accepted trade-off (see D3 rationale); shared data shape keeps future unification possible without rework.
|
||||
- [Risk] Progress payload could accidentally grow to include sensitive data (scene dumps, prompts) if a future change casually extends `TaskProgressModel` → Mitigation: `summary` field is explicitly bounded/plain-text only; code review for this change and future extensions should treat this the same as the existing `cloud-planner-proxy` no-screenshot-persistence rule (screenshots specifically remain excluded from every Cloud-side table this change introduces).
|
||||
- [Risk] `planner_decision_log` (D8) grows unboundedly across the whole fleet, unlike D5's single-row-per-assignment overwrite → Mitigation: D8's retention/prune job must ship in the same change as D7/D8, not deferred, for the same reason D2 must ship alongside D1.
|
||||
- [Risk] Full prompt text can itself contain sensitive on-screen content (whatever text was visible in the scene description embedded in the prompt) — persisting it centrally is a deliberate trade-off the user has explicitly requested for centralized troubleshooting, but it is a real expansion of what Cloud stores → Mitigation: no additional mitigation beyond what's already decided (screenshots still excluded); flagged here so it is a visible, intentional decision rather than a silent scope creep.
|
||||
- [Risk] Full LLM history in Cloud is silently absent for any host on the `direct` transport, which could read as "it's broken" rather than "expected" → Mitigation: Cloud Console should visibly distinguish "no progress reported yet" from "this host does not report LLM content" (e.g. by also surfacing the host's configured transport), rather than just showing an empty history with no explanation.
|
||||
- Per-step metadata writes add small SQLite I/O. This is the same local store
|
||||
already selected for Host history, and bounded retention limits growth.
|
||||
- A Host-local evidence record is only available while retained on that Host.
|
||||
This is intentional: it reflects the actual device execution and avoids
|
||||
sending screenshots to Cloud.
|
||||
- Removing the unauthenticated Runtime REST service is a breaking operator
|
||||
change. Clear configuration errors and documentation avoid a silent
|
||||
fallback to a nonexistent inspection surface.
|
||||
- Full Cloud-proxy prompts can contain visible screen text. This is the
|
||||
previously accepted Cloud troubleshooting trade-off; screenshot bytes remain
|
||||
excluded.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
1. Add `TaskProgressModel` and the optional `progress` field to `LeaseRenewalRequest`/response in `cloud/internal_api/models.py`. Backward compatible: `progress` is optional, older Host Agents omit it.
|
||||
2. Add progress columns + Alembic migration `0008_task_progress_columns`; extend `renew_lease()` in both repository implementations to accept/store them.
|
||||
3. Extend Cloud Console's task read model and UI to render the new fields (no-op if absent, keeping rollback trivial).
|
||||
4. Wire `metadata_store`/`timeline` into Host Agent's `create_execution_factories()` call site, add the retention pass (D2), and add the progress-holder hook feeding D4's renewal piggyback.
|
||||
5. Extend Host Agent's local console with the read-only task/timeline pages (D3).
|
||||
6. Fix `Timeline`/`TaskRunner`/`AIPlanner`/`ToolCallingClient` to record the real per-step prompt and response locally (D9) — independent of Cloud, benefits both Host Agent and Runtime consoles immediately.
|
||||
7. Add `planner_decision_log` + Alembic migration for it (D8) on both repository backends, with its retention/prune job.
|
||||
8. Extend `decide_planner_call` (`internal_api/api.py`) to persist each resolved decision into `planner_decision_log` (D7).
|
||||
9. Extend Cloud Console with a per-task LLM interaction history view reading the new table, including the "host uses `direct` transport, no LLM content available" distinction from the Risks section.
|
||||
10. Rollback: each step is independently revertible (optional field, additive columns, additive tables, additive UI, additive Host Agent wiring) — no destructive migration is required; both `0008` and the new decision-log migration's down-revisions drop only what they added.
|
||||
1. Upgrade the Host Agent code. Existing task databases gain nullable source
|
||||
correlation columns on startup; legacy Timeline records remain readable.
|
||||
2. Remove any `HOST_AGENT_RUNTIME_*` environment variables and stop any
|
||||
`api.rest` process. Start or browse only the Host Agent console for local
|
||||
execution evidence.
|
||||
3. Confirm a completed Host assignment appears at `/tasks` with its Cloud
|
||||
task/attempt correlation and complete Timeline evidence.
|
||||
4. Roll back only by restoring the prior release. The retired REST/UI routes
|
||||
are deliberately not kept as a compatibility alias because their storage
|
||||
was not connected to Host execution.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Exact retention window/count defaults for D2 (time-based vs. count-based, or both) — left for tasks.md to pick a concrete, documented default (e.g. keep last 50 tasks or 7 days, whichever is smaller) rather than block design on it.
|
||||
- Whether Cloud Console's existing task list/detail component can absorb the new fields with a small edit or needs a new sub-component — an implementation detail, not an architectural fork.
|
||||
- Whether a future change should unify Host Agent's server-rendered task pages and Runtime `console/`'s SPA into one shared frontend, now that D1 gives them an identical underlying data shape — explicitly deferred, not part of this change.
|
||||
- Exact retention default for D8's `planner_decision_log` (prune-after-terminal window, or reuse an existing Cloud task-retention cadence if one already exists) — left for tasks.md to pick a concrete default rather than block design on it.
|
||||
- Whether `step_index` in D8 should be assigned via a `SELECT count(*) + 1` under a row lock or a dedicated per-`(task_id, attempt)` counter/sequence — an implementation detail to resolve in tasks.md, not an architectural fork.
|
||||
- Manual verification still requires a real Host Agent, Appium, and device.
|
||||
Automated coverage verifies persistence, correlation, rendering, and
|
||||
service removal; real hardware validates screenshots and OCR availability.
|
||||
|
||||
Reference in New Issue
Block a user