feat: surface task execution progress across Host Agent and Cloud
Host Agent now persists step-level execution detail locally (via a real TaskMetadataStore/Timeline wired into TaskRunner) and reports a bounded in-progress snapshot piggybacked on lease renewal. Cloud persists that snapshot per active assignment and exposes it through the existing task list/detail query path; Cloud Console renders it as a live badge. Host Agent's local console gains authenticated, read-only task list and detail/timeline pages (same-origin, server-rendered) with inlined screenshots. Also fixes a pre-existing gap in the shared Timeline: the actual per-step LLM prompt is now recorded instead of the task goal, benefiting both Runtime and Host Agent consoles. When a host uses the cloud planner transport, each decide call's prompt and resulting tool decision are durably logged in a new planner_decision_log table (with bounded retention) and browsable from Cloud Console; direct-transport hosts explicitly surface a "not reported" state. Includes Alembic migrations 0008 (progress columns on scheduled_tasks) and 0009 (planner_decision_log), bounded Host-Agent-local retention, dual-backend repository parity, and Vitest + pytest coverage. Task 6.5 (manual end-to-end device verification) remains. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-07-14
|
||||
@@ -0,0 +1,129 @@
|
||||
## Context
|
||||
|
||||
Three surfaces currently cannot show a task's in-progress status:
|
||||
|
||||
1. **Host Agent local console** (`apps/device-host-agent/host_agent/web/app.py`): `HostAgentApplication`'s builder calls `create_execution_factories(resolved_manager, host_agent_config=resolved_config)` without `metadata_store`/`timeline` (`app.py:218-223`), so the in-process `TaskRunner` (`execution.py:39-46`) runs with `metadata_store=None, timeline=None`. `runtime/task.py::TaskRunner._update_task()` only persists when `metadata_store` is truthy, so step transitions vanish. The console's only signal is `AgentStatusTracker.snapshot()` (`status.py:14-88`), a single opaque span: `task_id/device_id/goal/started_at`, cleared on `mark_assignment_finished()`.
|
||||
2. **Cloud Control Plane / Cloud Console**: the internal protocol (`packages/cloud-platform/cloud/internal_api/models.py`) only has claim, heartbeat, lease-renewal, and terminal-result messages. Lease renewal (`ActiveAssignmentRunner._renew_while_running`, `apps/device-host-agent/host_agent/lease.py:79-106`) already fires periodically (~1/3 of remaining lease) for every in-flight assignment, but carries no task-progress payload.
|
||||
3. **Runtime `console/` SPA**: `api/rest.py::create_app()` builds its own `TaskMetadataStore`/`Timeline`/`TaskRunner` (`api/rest.py:24-41`), queried by `api/console.py::create_console_router()`'s `/console/tasks*` routes, which `console/src/api.ts` calls against `API_BASE_URL` (default `http://127.0.0.1:8000`). This is a wholly separate process from Host Agent, with no shared store or IPC. `runtime/`-owned packages are forbidden from importing host/cloud concerns (`test_runtime_owned_packages_do_not_import_host_or_cloud_concerns`), and `host-agent-local-console/design.md` already treats Host Agent's local history as "not a system of record," distinct from Cloud's.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Host Agent's local console shows live step-level progress for its current assignment (step index, step status, short summary), not just a static started-at snapshot.
|
||||
- Cloud Control Plane learns step-level progress near-real-time (piggybacked on existing periodic traffic, not a new polling loop) and Cloud Console renders it for an operator watching a remote task.
|
||||
- An operator can inspect a Host-Agent-executed task's progress/history from a web console, without coupling `runtime/` to host/cloud concerns.
|
||||
- Reuse existing, already-tested building blocks (`TaskMetadataStore`, `Timeline`, `api/console.py`'s query shape, the lease-renewal cadence) instead of inventing new storage or transport primitives.
|
||||
- Both the Host Agent's local task history and the Cloud Control Plane persist the *actual* per-step content sent to and received from the LLM (the real prompt text, not just the task's overall goal; the model's resulting decision, not just the parsed tool call) — so a step's record answers "what did we actually ask the model and what did it say," not just "what call did we make." Cloud's copy is the durable, centrally-searchable record for troubleshooting; the Host Agent's local copy remains the on-device record of what actually ran.
|
||||
|
||||
**Non-Goals:**
|
||||
- No SSE/WebSocket push. All three surfaces keep polling (Host Agent console 5s, Cloud Console/`cloud-console` and `console/` at their existing intervals). Nothing here requires push infra, and adding it would be a bigger, separate change.
|
||||
- The coarse live step index/status/summary reported via lease renewal (D4/D5) stays *latest-snapshot-only* on the Cloud side (overwrite semantics) — it is a "what's happening right now" indicator, not a history mechanism, and is unaffected by the LLM-content decisions below.
|
||||
- Full step-by-step **LLM interaction** history *is* now synced to and durably persisted by Cloud (see D7/D8) — this is a deliberate reversal of this change's earlier draft, which had scoped Cloud to latest-only. The motivation: Host Agent deployments already route every AI Planner decision through Cloud API's existing `cloud-planner-proxy` endpoint when configured for the `cloud` transport, so Cloud is already on the natural path for this data and centralizing it there (rather than only on whichever edge host happens to still be running) is the more useful place for an operator doing centralized troubleshooting. This durable log is scoped strictly to LLM prompt/response text — it is not a general-purpose duplicate of the Host Agent's full `Timeline` (no screenshots, no scene JSON dumps beyond what's embedded in the prompt text itself).
|
||||
- No change to the Runtime `console/` SPA or `runtime/`-owned packages. The multi-backend-SPA alternative was considered and rejected (see Decisions).
|
||||
- No change to `driver/`/`device/`/`core/` device-control internals.
|
||||
- No screenshot or full-scene data leaves the Host Agent process as part of progress reporting, and no screenshot bytes are ever persisted by the Cloud Control Plane. (Full LLM prompt/response *text* is, by contrast, an explicit Goal below — see D7/D8 — which partially supersedes the original `cloud-planner-proxy` proposal's "does not durably persist ... full prompt text" statement. Screenshots remain excluded; prompts no longer are.)
|
||||
|
||||
## Decisions
|
||||
|
||||
### D1: Wire a real `metadata_store`/`timeline` into Host Agent's `TaskRunner`, reusing the existing classes
|
||||
|
||||
`create_execution_factories()` already accepts `metadata_store: TaskMetadataStore | None` and `timeline: Timeline | None` (`execution.py:28-34`) — these are the same classes `api/rest.py` uses for the Runtime's own console. Reuse them as-is rather than inventing a Host-Agent-specific store: `HostAgentApplication`'s builder constructs a `TaskMetadataStore(db_path=<host-agent-local path>)` and `Timeline(ArtifactStore(<host-agent-local path>))` and passes them into `create_execution_factories()`.
|
||||
|
||||
**Alternative considered**: a bespoke in-memory-only step recorder (cheaper, no disk I/O). Rejected because `TaskMetadataStore`/`Timeline` are already proven in production (Runtime's own console has run on them since the original `web-console` change) and reusing them means Host Agent's task/timeline data has the *exact* same shape as `api/console.py` already serializes — a prerequisite for D3 below. A bespoke format would need its own (de)serialization and its own console rendering code for no real benefit.
|
||||
|
||||
New config: `HOST_AGENT_TASK_PROGRESS_DB_PATH` (default e.g. `host_agent_data/task_progress.sqlite3`) and a matching artifact directory, kept separate from any local Runtime `tasks/` directory that might exist on the same machine, to avoid two unrelated processes silently sharing or colliding on a path.
|
||||
|
||||
### D2: Bounded retention for the Host-Agent-local store
|
||||
|
||||
Unlike a Runtime dev session (short-lived, manually cleared), a Host Agent process runs indefinitely and executes many assignments over its lifetime. Unbounded `TaskMetadataStore` rows and `Timeline` screenshot artifacts would grow without limit.
|
||||
|
||||
Decision: add a lightweight retention pass (e.g. on a timer, or opportunistically after each assignment finishes) that prunes tasks older than a configurable window or beyond a configurable count, deleting both the `tasks` row and its `Timeline`/`ArtifactStore` files. This is new logic — neither `TaskMetadataStore` nor `Timeline` currently supports deletion — scoped as a small addition to `storage/task_metadata.py` / `storage/timeline.py` (or a Host-Agent-side wrapper if adding delete methods to the shared `storage` package feels too broad; prefer extending `storage` since both Runtime and Host Agent benefit from bounded retention).
|
||||
|
||||
### D3: Give Host Agent's own console read-only task/timeline pages instead of making `console/` multi-backend
|
||||
|
||||
Two options were evaluated for capability `host-agent-console-task-pages`:
|
||||
|
||||
- **(a) Multi-backend Runtime `console/` SPA**: let the existing Vue SPA point at a Host Agent's console origin (a saved/selectable base URL) in addition to the local Runtime. Since D1 makes Host Agent's data shape identical to what `api/console.py` already serializes, the SPA's existing fetch/render code would work unmodified against a Host Agent origin.
|
||||
- **(b) Extend Host Agent's own server-rendered console** (`host_agent/web/app.py`) with new read-only pages for task list/detail/timeline, reusing `TaskMetadataStore`/`Timeline` query calls directly (the same calls `api/console.py`'s handlers make), rendered as server-side HTML via the existing f-string + `html.escape()` convention (no SPA, no frontend build) established by `host-agent-local-console`.
|
||||
|
||||
**Decision: (b).** (a) requires Host Agent's console to serve cross-origin, credentialed requests from whatever origin `console/` is running on (Vite dev server, or a different deployed origin) — meaning either relaxing Host Agent's CORS posture for its session-cookie+CSRF-protected endpoints, or reworking its auth model to tolerate cross-origin fetches. That is real new attack surface on a console whose entire local-only threat model (`console_bind_host` defaulting to loopback, explicit opt-in for non-loopback) was deliberately conservative. (b) reuses an already-accepted pattern (server-rendered, same-origin, same auth) and fully satisfies the actual operator need — a web page showing what a Host Agent is doing — without touching `runtime/`, `console/`, or Host Agent's CORS/auth posture at all. The cost is a small amount of view duplication (Host Agent's console and Runtime's `console/` render similar-shaped data with different templates/frameworks), judged acceptable given they serve different operational contexts (local direct-connect Runtime vs. remote Host Agent fleet).
|
||||
|
||||
Any future desire for a unified SPA across both surfaces is left as a follow-up, not blocked by this decision (D1's shared data shape keeps that door open).
|
||||
|
||||
### D4: Piggyback progress reporting on the existing lease-renewal call
|
||||
|
||||
`ActiveAssignmentRunner._renew_while_running` (`lease.py:79-106`) already makes an authenticated, per-assignment, periodic call (`client.renew(assignment)`, roughly every 1/3 of remaining lease) while a task executes, validated server-side against `host_id`/`task_id`/`attempt`/`lease_id` (`internal_api/api.py:291-320`). This is the natural piggyback point for progress: extend `LeaseRenewalRequest` with an optional `progress: TaskProgressModel | None` field (new small model: `step_index: int`, `step_status: Literal[...]`, `summary: str` bounded length — no screenshot/scene payload) in `cloud/internal_api/models.py`, the single schema both sides already import directly (no parallel schema, matching existing convention noted for this protocol).
|
||||
|
||||
A small thread-safe "latest progress" holder (similar in spirit to `LeaseGuard`) is written by the execution thread (via a hook on `TaskRunner`/`AssignmentExecutor`, updated once per step) and read by `_renew_while_running` just before each renewal call, so no new timer/cadence is introduced.
|
||||
|
||||
**Alternative considered**: a dedicated `POST /internal/v1/hosts/{host_id}/assignments/{task_id}/progress` endpoint called on its own cadence. Rejected — it would duplicate the auth/validation `renew_assignment` already does, and introduce a second periodic call where one already exists and fires at a reasonable frequency for this purpose.
|
||||
|
||||
### D5: Cloud persists only the latest progress snapshot, overwritten alongside lease renewal
|
||||
|
||||
`renew_lease()` (`repository.py:407`, `sql_repository.py:1480`) already takes a row lock and writes a lease-expiry update on every renewal; extend it to also accept and store the optional progress fields (few scalar columns — `progress_step_index`, `progress_step_status`, `progress_summary`, `progress_updated_at` — rather than a schema-flexible JSON blob, keeping it queryable and consistent with the rest of the row's typed columns). This requires a new Alembic migration (head is currently `0007_llm_provider_management`) plus the equivalent SQLite schema change, applied to both `repository.py` and `sql_repository.py` to keep the dual-backend contract (`cloud-control-plane` spec's "Deployment and local persistence modes share one contract" requirement).
|
||||
|
||||
Progress columns are cleared (or simply superseded and ignored) once a terminal result is recorded — they represent "what's happening right now," not history; Cloud's durable attempt/result history is unaffected and unduplicated.
|
||||
|
||||
### D6: Cloud Console reads progress from the existing task/attempt query path, not a new endpoint
|
||||
|
||||
Extend whatever response model Cloud Console's task list/detail already uses (Cloud API public router) with the optional latest-progress fields from D5, rather than adding a new endpoint. Cloud Console renders it as a small inline badge/line ("step 4: tapping login button") next to the existing status, refreshed on the SPA's existing polling interval.
|
||||
|
||||
### D7: Capture full LLM interaction history for free via the existing `cloud-planner-proxy` decide endpoint, not a new reporting channel
|
||||
|
||||
Investigated the Host Agent → Cloud call path in detail: when a Host Agent is configured with `AI_PLANNER_TRANSPORT=cloud`, `AIPlanner.plan()` (`runtime/ai_planner.py`) calls `CloudProxyToolCallingClient.decide()` (`apps/device-host-agent/host_agent/cloud_planner_client.py:46-95`), which `POST`s the *complete* `system_prompt`/`user_prompt` (plus `task_id`/`attempt`/`lease_id` from `current_planner_execution_context()`) to Cloud API's `/hosts/{host_id}/planner/decide` (`packages/cloud-platform/cloud/internal_api/api.py::decide_planner_call`, line ~367). That handler already resolves and returns a `ToolCallDecision` (`tool_name`/`arguments`/`usage`) and already does per-call bookkeeping (`pool.store.settle_host_token_reservation(...)`, line ~463) using the same repository object this change's D5 already touches for lease renewal.
|
||||
|
||||
This means **every planning step's actual prompt and resulting decision already flows through a Cloud-owned request handler** when the `cloud` transport is used — no new endpoint, no new protocol field, no queue/batching scheme is needed to get full LLM content to Cloud. The only change needed is to make that handler *persist* what it currently discards.
|
||||
|
||||
**Decision**: extend `decide_planner_call` to, immediately after computing `decision` (success path only — a `ToolCallUnavailable`/502 path persists nothing), insert one row into a new log table (D8) keyed by `(task_id, attempt, step_index)`, where `step_index` is assigned by the Cloud side itself (an auto-incrementing counter scoped to `task_id`+`attempt`, e.g. `select count(*) + 1` under the same row lock, or a DB sequence/identity column) — Host Agent does not need to track or send a step counter for this.
|
||||
|
||||
**Explicit limitation, called out rather than papered over**: this only captures LLM content for hosts using the `cloud` transport. A host on the (still-default) `direct` transport never sends its prompts to Cloud at all — Cloud has zero LLM content for that host's tasks, and only ever sees the coarse index/status/summary from D4/D5's lease-renewal piggyback (which is transport-agnostic, since it's driven by `TaskRunner` step completion, not by the planner's transport choice). This is a real operational dependency: centralized LLM-interaction troubleshooting via Cloud Console requires the fleet (or the hosts an operator cares about) to run with `AI_PLANNER_TRANSPORT=cloud`. This change does not make `cloud` the new default transport — that remains a separate, already-existing configuration decision outside this change's scope.
|
||||
|
||||
**Alternative considered**: extend the D4 lease-renewal piggyback to also carry full prompt/response text (queued, not overwritten, so no step is lost between renewals). Rejected: it would duplicate a transport that already exists for exactly this payload (the decide call itself) whenever `cloud` transport is active, and would still need a *separate* new channel for the `direct`-transport case where Cloud never sees the prompt anyway — i.e., it does not actually solve the `direct`-transport gap, so it only adds complexity without expanding coverage.
|
||||
|
||||
### D8: New bounded-retention table for the full per-step LLM decision log, extended on both repository backends
|
||||
|
||||
Add a new table (e.g. `planner_decision_log`): `id`, `host_id`, `task_id`, `attempt`, `step_index`, `system_prompt` (text), `user_prompt` (text), `tool_name`, `arguments_json` (text), `created_at`. No screenshot column — screenshots are never sent to this endpoint's persistence path (the request's `screenshot_base64` is used only to call the LLM provider and is never written to this log, consistent with the Non-Goals screenshot exclusion).
|
||||
|
||||
Like D5, this needs a new Alembic migration and the equivalent SQLite path, implemented on both `repository.py`'s SQLite-backed implementation and `sql_repository.py::SQLAlchemyCloudRepository` to preserve the existing dual-backend contract. Unlike D5's few-nullable-columns-on-an-existing-row approach, this is an independent append-only table (one row per decide call, not an overwrite), since the whole point is durable per-step history rather than a live snapshot.
|
||||
|
||||
**Retention**: this table grows once per planning step across the whole fleet, indefinitely, on a shared multi-tenant Cloud database — unbounded growth is a real risk here in a way D5's single-row-per-assignment overwrite never was. Decision: a scheduled prune job (mirrors D2's Host-Agent-local retention) deletes rows whose owning task reached a terminal state more than a configurable window ago (default: prune 7 days after task terminal, or once the task itself is pruned/archived by whatever existing Cloud task-retention policy applies — reuse that cadence rather than inventing a second one if `cloud-control-plane` already has one; otherwise default to a simple time-based prune).
|
||||
|
||||
### D9: Fix the shared `Timeline`/`TaskRunner`/`AIPlanner` path so the *actual* per-step prompt and response are recorded locally, not just the task goal and the parsed tool call
|
||||
|
||||
Independent of Cloud persistence, the existing local recording is itself wrong today: `TaskRunner._append_timeline()` (`runtime/task.py:296-319`) calls `Timeline.append(prompt=task.goal, tool_call={"action": ..., "description": ..., "args": ...}, ...)`. `task.goal` is the overall task goal, not the per-step prompt actually sent to the LLM — the real per-step prompt (`planner_user_prompt(goal, scene_json, history_summary)`, built in `ai_planner.py::plan()`) is constructed, sent, and discarded entirely within `AIPlanner.plan()`, never reaching `TaskRunner`. Likewise `ToolCallDecision` (`runtime/tool_calling_client.py:23-27`) carries only the parsed `tool_name`/`arguments`/`usage` — any raw response text/content the model returned is discarded during parsing (`_decision_from_anthropic_response`/`_decision_from_openai_response`).
|
||||
|
||||
This is a pre-existing gap in a component shared by Runtime and Host Agent alike (not new to this change), and it undermines the very "step-level detail" goal D1 already committed to — a persisted step whose "prompt" field is just the task's goal repeated on every row is not useful for troubleshooting.
|
||||
|
||||
**Decision**: extend `ToolCallDecision` with the actual `user_prompt`/`system_prompt` it was given (or have `AIPlanner.plan()` return a small side-channel result instead of changing the `Planner` interface's return type) so `TaskRunner._append_timeline()` can pass the real per-step prompt into `Timeline.append()`. Rename `Timeline`/`TimelineRecord`'s `prompt` field's meaning (or add a new field) to unambiguously mean "the prompt actually sent to the LLM for this step." This fix lands in the shared `runtime`/`storage` packages, so both Host Agent's local console (D3) and the existing Runtime `console/` automatically benefit — it is not Host-Agent-specific plumbing.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- [Risk] Per-step SQLite writes in a long-lived Host Agent process add I/O overhead → Mitigation: this is the same write pattern Runtime's own console has always used per step; no new proof of acceptability needed. If profiling later shows it matters for very high step-rate tasks, batching/debouncing is a follow-up, not a blocker here.
|
||||
- [Risk] Unbounded local disk growth from `Timeline` screenshots on an indefinitely-running Host Agent → Mitigation: D2's bounded retention pass; must ship in the same change as D1, not deferred, since D1 alone would otherwise introduce an unbounded-growth regression.
|
||||
- [Risk] New DB columns/migration touch both `repository.py` (SQLite) and `sql_repository.py` (Postgres) — drift between the two has been a real defect category in this codebase (see `cloud-control-plane-integration` archive notes) → Mitigation: contract/parity tests already exist for this dual-backend boundary; extend them to cover the new progress columns.
|
||||
- [Risk] View duplication between Host Agent's server-rendered task pages (D3) and Runtime `console/`'s task pages (different frameworks, same data shape) → Mitigation: accepted trade-off (see D3 rationale); shared data shape keeps future unification possible without rework.
|
||||
- [Risk] Progress payload could accidentally grow to include sensitive data (scene dumps, prompts) if a future change casually extends `TaskProgressModel` → Mitigation: `summary` field is explicitly bounded/plain-text only; code review for this change and future extensions should treat this the same as the existing `cloud-planner-proxy` no-screenshot-persistence rule (screenshots specifically remain excluded from every Cloud-side table this change introduces).
|
||||
- [Risk] `planner_decision_log` (D8) grows unboundedly across the whole fleet, unlike D5's single-row-per-assignment overwrite → Mitigation: D8's retention/prune job must ship in the same change as D7/D8, not deferred, for the same reason D2 must ship alongside D1.
|
||||
- [Risk] Full prompt text can itself contain sensitive on-screen content (whatever text was visible in the scene description embedded in the prompt) — persisting it centrally is a deliberate trade-off the user has explicitly requested for centralized troubleshooting, but it is a real expansion of what Cloud stores → Mitigation: no additional mitigation beyond what's already decided (screenshots still excluded); flagged here so it is a visible, intentional decision rather than a silent scope creep.
|
||||
- [Risk] Full LLM history in Cloud is silently absent for any host on the `direct` transport, which could read as "it's broken" rather than "expected" → Mitigation: Cloud Console should visibly distinguish "no progress reported yet" from "this host does not report LLM content" (e.g. by also surfacing the host's configured transport), rather than just showing an empty history with no explanation.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
1. Add `TaskProgressModel` and the optional `progress` field to `LeaseRenewalRequest`/response in `cloud/internal_api/models.py`. Backward compatible: `progress` is optional, older Host Agents omit it.
|
||||
2. Add progress columns + Alembic migration `0008_task_progress_columns`; extend `renew_lease()` in both repository implementations to accept/store them.
|
||||
3. Extend Cloud Console's task read model and UI to render the new fields (no-op if absent, keeping rollback trivial).
|
||||
4. Wire `metadata_store`/`timeline` into Host Agent's `create_execution_factories()` call site, add the retention pass (D2), and add the progress-holder hook feeding D4's renewal piggyback.
|
||||
5. Extend Host Agent's local console with the read-only task/timeline pages (D3).
|
||||
6. Fix `Timeline`/`TaskRunner`/`AIPlanner`/`ToolCallingClient` to record the real per-step prompt and response locally (D9) — independent of Cloud, benefits both Host Agent and Runtime consoles immediately.
|
||||
7. Add `planner_decision_log` + Alembic migration for it (D8) on both repository backends, with its retention/prune job.
|
||||
8. Extend `decide_planner_call` (`internal_api/api.py`) to persist each resolved decision into `planner_decision_log` (D7).
|
||||
9. Extend Cloud Console with a per-task LLM interaction history view reading the new table, including the "host uses `direct` transport, no LLM content available" distinction from the Risks section.
|
||||
10. Rollback: each step is independently revertible (optional field, additive columns, additive tables, additive UI, additive Host Agent wiring) — no destructive migration is required; both `0008` and the new decision-log migration's down-revisions drop only what they added.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Exact retention window/count defaults for D2 (time-based vs. count-based, or both) — left for tasks.md to pick a concrete, documented default (e.g. keep last 50 tasks or 7 days, whichever is smaller) rather than block design on it.
|
||||
- Whether Cloud Console's existing task list/detail component can absorb the new fields with a small edit or needs a new sub-component — an implementation detail, not an architectural fork.
|
||||
- Whether a future change should unify Host Agent's server-rendered task pages and Runtime `console/`'s SPA into one shared frontend, now that D1 gives them an identical underlying data shape — explicitly deferred, not part of this change.
|
||||
- Exact retention default for D8's `planner_decision_log` (prune-after-terminal window, or reuse an existing Cloud task-retention cadence if one already exists) — left for tasks.md to pick a concrete default rather than block design on it.
|
||||
- Whether `step_index` in D8 should be assigned via a `SELECT count(*) + 1` under a row lock or a dedicated per-`(task_id, attempt)` counter/sequence — an implementation detail to resolve in tasks.md, not an architectural fork.
|
||||
@@ -0,0 +1,34 @@
|
||||
## Why
|
||||
|
||||
Nobody can see a task while it is running. Host Agent's local console only shows a coarse "current assignment" snapshot (task id, device id, goal, started-at) because the `TaskRunner` it drives is wired with `metadata_store=None, timeline=None` — every step transition happens in memory and is discarded the instant the assignment finishes. Cloud Control Plane only learns about an assignment at claim, heartbeat, and terminal-result time, so Cloud Console has nothing better to show. The local Runtime (`api/rest.py` + `console/`) is a separate process with its own independent `TaskMetadataStore`/`Timeline`, so it never sees a task that a Host Agent executed at all. An operator debugging a stuck or misbehaving task currently has no live signal anywhere in the system until the task finishes or times out.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Wire Host Agent's `AssignmentExecutor` / `create_execution_factories()` with a real `metadata_store` and `timeline` (or Host-Agent-local equivalents) so the in-process `TaskRunner` actually records step-by-step status instead of discarding it.
|
||||
- Expose that step-level detail through Host Agent's local console: extend `AgentStatusTracker`/`/api/status` (or add a focused endpoint) with current step index, step status, and a short in-progress step log; render it in the dashboard's "Current assignment" section instead of the static started-at-only view.
|
||||
- Add a progress-reporting path from Host Agent to Cloud Control Plane so the control plane learns step-level state near-real-time rather than only at claim/heartbeat/terminal-result. Extend the existing `cloud.internal_api.models` Pydantic schema (the single source of truth Host Agent already imports directly) rather than introducing a parallel schema.
|
||||
- Persist and expose the latest per-assignment progress on the Cloud Control Plane side, and surface it in Cloud Console so an operator watching a remote task sees live step progress, not just "dispatched" / "succeeded" / "failed".
|
||||
- Give an operator a web console view of Host-Agent-executed task progress without making the `runtime/`-owned packages import host or cloud concerns (enforced by `test_runtime_owned_packages_do_not_import_host_or_cloud_concerns`). After evaluating the alternative of making the Runtime `console/` SPA multi-backend (pointing its existing JS bundle at a Host Agent's console origin), this change instead extends Host Agent's own server-rendered local console with read-only task list/detail/timeline pages, reusing the same query shape `api/console.py` already exposes to the Runtime SPA. This avoids new cross-origin/session-cookie surface between the SPA and Host Agent, and keeps `runtime/` untouched. See design.md for the full trade-off analysis.
|
||||
- All three surfaces continue to use polling (matching current behavior); this change does not introduce SSE/WebSocket infrastructure unless design.md finds a compelling reason to.
|
||||
- Fix the shared `Timeline`/`TaskRunner`/`AIPlanner` recording path so a persisted step's "prompt" is the *actual* prompt sent to the LLM for that step (not the task's overall goal) and the model's resulting decision is captured too — this pre-existing gap affects Runtime and Host Agent alike and undermines the step-level detail this change otherwise adds.
|
||||
- Persist a durable, per-step log of full LLM prompt/response content on the Cloud Control Plane, for centralized troubleshooting — reusing the already-existing `cloud-planner-proxy` decide endpoint as the capture point (no new protocol/endpoint) rather than the coarse lease-renewal piggyback used for live index/status. This durable log is populated only for hosts using the `cloud` planner transport; hosts on the `direct` transport still get only the coarse index/status via lease renewal. Cloud Console gains a view to browse a task's full LLM interaction history.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `host-agent-task-progress`: Host Agent captures step-level execution progress for its in-flight assignment (via a wired `metadata_store`/`timeline`) and exposes it through its local console/API.
|
||||
- `cloud-task-progress-visibility`: Cloud Control Plane receives, persists, and exposes near-real-time step-level progress for assignments it has dispatched to a Host Agent, and Cloud Console renders it.
|
||||
- `host-agent-console-task-pages`: Host Agent's local server-rendered console gains read-only task list/detail/timeline pages (mirroring `api/console.py`'s task query shape) so an operator can inspect a Host-Agent-executed task's progress and history without needing the separate Runtime `console/` SPA or violating the `runtime/`-package host/cloud isolation boundary.
|
||||
|
||||
### Modified Capabilities
|
||||
- `host-agent-protocol`: add a requirement that the Host Agent reports in-progress step-level status updates to the control plane (in addition to the existing heartbeat/claim/renewal/result operations), and that the control plane accepts and stores them per active assignment.
|
||||
- `cloud-planner-proxy`: the existing planner-decision endpoint additionally persists each resolved decision's prompt/response into a durable, bounded-retention per-step log, instead of discarding it after the response is returned.
|
||||
|
||||
## Impact
|
||||
|
||||
- `apps/device-host-agent/host_agent/app.py`, `execution.py`, `assignment.py`, `status.py`, `web/app.py` — wire a real metadata/timeline store into the in-process `TaskRunner`, extend status tracking and the local console UI/API.
|
||||
- `runtime/task.py`, `runtime/ai_planner.py`, `runtime/tool_calling_client.py`, `storage/timeline.py` — fix the shared step-recording path so the real per-step prompt and the model's response are captured, not just the task goal and parsed tool call (a correctness fix in already-existing, shared code, not new behavior scope).
|
||||
- `packages/cloud-platform/cloud/internal_api/models.py`, `packages/cloud-platform/cloud/internal_api/api.py` (`decide_planner_call`), `repository.py`/`sql_repository.py`, a new Alembic migration — new progress-reporting request/response models and a persistence + query path for latest per-assignment coarse progress, *and* a new durable per-step `planner_decision_log` table (with its own retention job) populated from the existing planner-decision endpoint.
|
||||
- `cloud-console/` (Vue3 SPA) — new UI to render live per-assignment progress, and a new view to browse a task's full LLM interaction history.
|
||||
- `apps/device-host-agent/host_agent/web/app.py` — new read-only task list/detail/timeline pages backed by the Host-Agent-local `TaskMetadataStore`/`Timeline`; no changes anticipated to `console/` (Vue3 SPA).
|
||||
- No changes anticipated to `driver/`, `device/`, `core/` device-control internals.
|
||||
@@ -0,0 +1,34 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Cloud Control Plane persists each resolved planner decision for later retrieval
|
||||
The Cloud Control Plane's planner-decision endpoint SHALL, for each request it successfully resolves to a tool-call decision, persist the request's system prompt, user prompt, the resolved tool name and arguments, and an assigned step index (scoped to the request's `task_id`/`attempt`) to a durable, bounded-retention store, in addition to returning the decision to the requesting Host Agent. The Cloud Control Plane SHALL NOT persist screenshot bytes from these requests.
|
||||
|
||||
#### Scenario: A planner-decision request resolves successfully
|
||||
- **WHEN** the Cloud Control Plane resolves a planner-decision request to a tool-call decision
|
||||
- **THEN** it persists the request's system prompt, user prompt, the resolved tool name and arguments, and a step index for that `task_id`/`attempt`, before returning the decision to the Host Agent
|
||||
|
||||
#### Scenario: A planner-decision request fails
|
||||
- **WHEN** the configured provider call fails and the endpoint returns a structured failure response
|
||||
- **THEN** no row is persisted for that request
|
||||
|
||||
#### Scenario: A screenshot was included in the request
|
||||
- **WHEN** a planner-decision request includes a screenshot
|
||||
- **THEN** the screenshot bytes are used only to call the LLM provider and are not written to the persisted decision log
|
||||
|
||||
### Requirement: Persisted planner decisions are retained within a bounded window
|
||||
The Cloud Control Plane SHALL prune persisted planner decisions once their owning task has been in a terminal state for longer than a configurable retention window, so that indefinite operation does not cause unbounded growth of the decision log.
|
||||
|
||||
#### Scenario: A task's retention window has elapsed since reaching a terminal state
|
||||
- **WHEN** a task reached a terminal state more than the configured retention window ago
|
||||
- **THEN** the Cloud Control Plane removes that task's persisted planner decisions
|
||||
|
||||
#### Scenario: A task is still active or within its retention window
|
||||
- **WHEN** a task is still active, or reached a terminal state less than the configured retention window ago
|
||||
- **THEN** its persisted planner decisions remain available for query
|
||||
|
||||
### Requirement: Cloud-side planner decision history is scoped to hosts using the cloud-proxy transport
|
||||
The Cloud Control Plane's persisted planner decision log SHALL only ever contain entries for hosts whose planner calls were routed through the cloud-proxy transport; it SHALL NOT contain entries, synthesized or otherwise, for hosts using the direct-to-provider transport.
|
||||
|
||||
#### Scenario: A host uses the direct-to-provider transport
|
||||
- **WHEN** a Host Agent configured for the direct-to-provider transport executes an assignment
|
||||
- **THEN** no planner decision entries for that assignment appear in the Cloud Control Plane's persisted decision log
|
||||
+38
@@ -0,0 +1,38 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Cloud Control Plane exposes the latest in-progress step status for an active assignment
|
||||
The Cloud Control Plane's existing task query surface SHALL include the latest reported step index, step status, and summary for any assignment that has an in-progress Host Agent execution, alongside the task's existing status fields.
|
||||
|
||||
#### Scenario: An assignment has reported progress
|
||||
- **WHEN** an operator queries a task that has an active, in-progress assignment with previously reported step progress
|
||||
- **THEN** the response includes that step's index, status, and summary alongside the task's existing fields
|
||||
|
||||
#### Scenario: An assignment has no reported progress yet
|
||||
- **WHEN** an operator queries a task whose active assignment has not yet reported any progress
|
||||
- **THEN** the response omits progress fields rather than showing stale or default values
|
||||
|
||||
#### Scenario: An assignment has reached a terminal state
|
||||
- **WHEN** an operator queries a task whose assignment has already completed (succeeded or failed)
|
||||
- **THEN** the response does not present the last in-progress step as current status; the task's terminal status and result take precedence
|
||||
|
||||
### Requirement: Cloud Console renders live step progress for in-progress tasks
|
||||
Cloud Console SHALL display the current step index, status, and summary for a task with an active, in-progress Host Agent execution, refreshed on its existing polling interval, without requiring a new push channel.
|
||||
|
||||
#### Scenario: Operator views an in-progress task
|
||||
- **WHEN** an operator opens a task detail view for a task with an active in-progress assignment
|
||||
- **THEN** Cloud Console shows the latest known step index, status, and summary, updating on subsequent polls as new progress is reported
|
||||
|
||||
#### Scenario: Operator views a task with no in-progress execution
|
||||
- **WHEN** an operator opens a task detail view for a queued, terminal, or otherwise not-currently-executing task
|
||||
- **THEN** Cloud Console does not display stale in-progress step information
|
||||
|
||||
### Requirement: Cloud Console displays a task's full LLM interaction history
|
||||
Cloud Console SHALL provide a view, for a given task, listing each persisted planner decision in step order, including its full prompt and resulting decision, sourced from the Cloud Control Plane's persisted planner decision log.
|
||||
|
||||
#### Scenario: Task has persisted planner decisions
|
||||
- **WHEN** an operator opens the LLM interaction history view for a task that has one or more persisted planner decisions
|
||||
- **THEN** Cloud Console shows each decision in step order with its prompt and resulting tool call
|
||||
|
||||
#### Scenario: Task's host used the direct-to-provider transport
|
||||
- **WHEN** an operator opens the LLM interaction history view for a task whose host used the direct-to-provider transport
|
||||
- **THEN** Cloud Console indicates that no LLM interaction history is available because the host does not report it, rather than showing an empty history with no explanation
|
||||
+34
@@ -0,0 +1,34 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Host Agent local console exposes step-level status for the current assignment
|
||||
The Host Agent's local console SHALL display, for its currently executing assignment, the current step index, step status, and a short summary, sourced from the Host Agent's local task metadata store, refreshed on the console's existing polling interval.
|
||||
|
||||
#### Scenario: An assignment is currently executing
|
||||
- **WHEN** an operator views the Host Agent local console dashboard while an assignment is executing
|
||||
- **THEN** the dashboard shows the current step index, step status, and a short summary for that assignment, updating on subsequent polls
|
||||
|
||||
#### Scenario: No assignment is currently executing
|
||||
- **WHEN** an operator views the dashboard while the Host Agent is idle
|
||||
- **THEN** the dashboard shows no in-progress step information
|
||||
|
||||
### Requirement: Host Agent local console exposes read-only task history with per-step detail and screenshots
|
||||
The Host Agent's local console SHALL provide authenticated, read-only pages listing recently executed tasks and, for a selected task, its full per-step history including any captured screenshots, sourced from the Host Agent's local task metadata store and timeline.
|
||||
|
||||
#### Scenario: Operator lists recent tasks
|
||||
- **WHEN** an authenticated operator opens the Host Agent local console's task list page
|
||||
- **THEN** it shows tasks from the local task metadata store, most recent first, including tasks that have already reached a terminal state
|
||||
|
||||
#### Scenario: Operator inspects a completed task's step history
|
||||
- **WHEN** an authenticated operator opens the detail page for a specific completed task
|
||||
- **THEN** the page shows each recorded step in order, including its tool call, result, and any captured screenshot
|
||||
|
||||
#### Scenario: Unauthenticated request
|
||||
- **WHEN** a request to the task list or task detail pages is made without a valid Host Agent console session
|
||||
- **THEN** the Host Agent rejects the request the same way it rejects unauthenticated requests to its other console pages
|
||||
|
||||
### Requirement: Host Agent local console task pages require no new cross-origin surface
|
||||
The Host Agent local console's task pages SHALL be served same-origin from the Host Agent's existing web application, without introducing new CORS allowances or a dependency on the separate Runtime `console/` frontend.
|
||||
|
||||
#### Scenario: Task pages are requested
|
||||
- **WHEN** an operator's browser requests the Host Agent local console's task pages
|
||||
- **THEN** the pages are served by the Host Agent's own application using its existing session/CSRF protections, with no additional cross-origin configuration required
|
||||
@@ -0,0 +1,23 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Host Agent reports execution progress alongside lease renewal
|
||||
The Host Agent SHALL optionally include a bounded, screenshot-free progress summary (current step index, step status, and a short plain-text summary) in its periodic lease-renewal request for an active assignment, and the control plane SHALL accept and store only the most recent such summary per active assignment.
|
||||
|
||||
#### Scenario: Progress is available at renewal time
|
||||
- **WHEN** the Host Agent renews the lease for an in-progress assignment and has a current step index, status, and summary available
|
||||
- **THEN** the renewal request includes that progress summary and the control plane overwrites any previously stored progress for that assignment with it
|
||||
|
||||
#### Scenario: Progress is not available at renewal time
|
||||
- **WHEN** the Host Agent renews a lease without a progress summary available (e.g. before the first step completes)
|
||||
- **THEN** the renewal request omits the progress field and any previously stored progress for that assignment is left unchanged
|
||||
|
||||
#### Scenario: Assignment reaches a terminal state
|
||||
- **WHEN** an assignment's terminal result is recorded
|
||||
- **THEN** the control plane's stored progress for that assignment is no longer treated as current and is not exposed as an in-progress status
|
||||
|
||||
### Requirement: Progress reports exclude screenshot and scene payloads
|
||||
The control plane SHALL reject or ignore any progress field on a renewal request that includes screenshot, scene, or other bulk payload data beyond the bounded step index, status, and short text summary.
|
||||
|
||||
#### Scenario: Renewal request includes an oversized or non-text summary
|
||||
- **WHEN** a Host Agent submits a progress summary exceeding the configured length bound
|
||||
- **THEN** the control plane truncates or rejects the oversized field without failing the underlying lease renewal
|
||||
+30
@@ -0,0 +1,30 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Host Agent records step-level execution detail for its in-process TaskRunner
|
||||
The Host Agent SHALL construct its in-process `TaskRunner` with a durable metadata store and timeline so that every step transition (status, index, the actual prompt submitted to the LLM for that step, the model's resulting decision, result, and screenshot when captured) is persisted as it happens, rather than discarded when the assignment completes. The persisted prompt SHALL be the prompt actually sent to the LLM for that specific step, not the task's overall goal.
|
||||
|
||||
#### Scenario: A step completes during goal execution
|
||||
- **WHEN** the Host Agent's `TaskRunner` completes a step while executing an assigned goal
|
||||
- **THEN** the step's status, index, the actual per-step LLM prompt and response, tool call, result, and any captured screenshot are persisted to the Host Agent's local task metadata store and timeline before the next step begins
|
||||
|
||||
#### Scenario: An assignment finishes
|
||||
- **WHEN** an assignment reaches a terminal state (succeeded or failed)
|
||||
- **THEN** its full step history remains queryable from the Host Agent's local store after the in-memory `Task` object is discarded
|
||||
|
||||
### Requirement: Host-Agent-local task history is retained within a bounded window
|
||||
The Host Agent SHALL prune persisted task metadata, timeline records, and associated screenshot artifacts once they exceed a configurable retention window or count, so that indefinite process uptime does not cause unbounded local disk growth.
|
||||
|
||||
#### Scenario: Retention window is exceeded
|
||||
- **WHEN** a persisted task's age or position exceeds the configured retention threshold
|
||||
- **THEN** the Host Agent removes that task's metadata row, timeline records, and screenshot artifacts from local storage
|
||||
|
||||
#### Scenario: Retention has not been exceeded
|
||||
- **WHEN** a persisted task is within the configured retention threshold
|
||||
- **THEN** its metadata, timeline records, and screenshot artifacts remain available for query
|
||||
|
||||
### Requirement: Host Agent local task storage is isolated from an unrelated local Runtime
|
||||
The Host Agent SHALL use a configurable, Host-Agent-specific database and artifact path for its task metadata store and timeline, distinct from any local Runtime API's own task storage path, so that the two processes cannot silently collide or share state when run on the same machine.
|
||||
|
||||
#### Scenario: Host Agent and local Runtime run on the same machine
|
||||
- **WHEN** both a Host Agent process and a local Runtime API process run on the same machine with their default configurations
|
||||
- **THEN** each process reads and writes its own task metadata store and timeline without observing or modifying the other's data
|
||||
@@ -0,0 +1,68 @@
|
||||
## 1. Host Agent local task storage (D1, D2)
|
||||
|
||||
- [x] 1.1 Add `HOST_AGENT_TASK_PROGRESS_DB_PATH` (and matching artifact directory config) to `HostAgentConfig`/`load_host_agent_config()`, with a Host-Agent-specific default distinct from any local Runtime `tasks/` path.
|
||||
- [x] 1.2 In `HostAgentApplication`'s builder (`app.py:218-223`), construct a `TaskMetadataStore`/`Timeline` from that config and pass them into `create_execution_factories(..., metadata_store=..., timeline=...)`.
|
||||
- [x] 1.3 Add delete/prune methods to `storage/task_metadata.py::TaskMetadataStore` and `storage/timeline.py::Timeline` (remove a task's row, timeline records, and screenshot artifacts).
|
||||
- [x] 1.4 Implement a bounded retention pass in the Host Agent (default: keep the newer of "last 50 tasks" or "7 days", whichever keeps fewer rows) that runs after each assignment finishes, calling the new prune methods.
|
||||
- [x] 1.5 Add unit tests: step transitions persist during execution; a task's history is queryable after the assignment completes and the in-memory `Task` is discarded; retention prunes tasks beyond the configured threshold; Host Agent and a local Runtime process using default paths on the same machine do not collide.
|
||||
|
||||
## 2. Host Agent reports progress during lease renewal (D4)
|
||||
|
||||
- [x] 2.1 Add `TaskProgressModel` (`step_index: int`, `step_status`, `summary: str` with a bounded max length) to `packages/cloud-platform/cloud/internal_api/models.py`, and an optional `progress: TaskProgressModel | None` field on `LeaseRenewalRequest`.
|
||||
- [x] 2.2 Add a thread-safe "latest progress" holder written by the Host Agent's execution thread (hook into `TaskRunner`/`AssignmentExecutor` per-step completion) and read by `ActiveAssignmentRunner._renew_while_running` (`lease.py`) just before each renewal call.
|
||||
- [x] 2.3 Update `HostAgentClient.renew()` to include the current progress snapshot (if any) in the renewal request.
|
||||
- [x] 2.4 Add unit tests: renewal includes progress once a step has completed; renewal omits progress before any step completes; an oversized summary is truncated/rejected client-side before sending.
|
||||
|
||||
## 3. Cloud Control Plane persists latest progress (D5)
|
||||
|
||||
- [x] 3.1 Add progress columns (`progress_step_index`, `progress_step_status`, `progress_summary`, `progress_updated_at`, all nullable) via a new Alembic migration (`0008_task_progress_columns`) with a down-revision that drops them.
|
||||
- [x] 3.2 Add the equivalent columns/handling to the SQLite schema path.
|
||||
- [x] 3.3 Extend `renew_lease()` in both `repository.py` and `sql_repository.py` to accept and overwrite the optional progress fields under the same row lock already taken for the lease-expiry update.
|
||||
- [x] 3.4 Update `renew_assignment` in `internal_api/api.py` to pass the optional `payload.progress` fields through to `renew_lease()`, rejecting/truncating an oversized `summary` without failing the underlying renewal.
|
||||
- [x] 3.5 Ensure progress fields are treated as stale/ignored once `record_task_result` records a terminal outcome for that task/attempt.
|
||||
- [x] 3.6 Add/extend repository contract tests (both SQLite and PostgreSQL where the existing test setup covers it) so the two backends stay in parity for the new columns.
|
||||
|
||||
## 4. Cloud Console displays live progress (D6)
|
||||
|
||||
- [x] 4.1 Extend the Cloud API's existing task list/detail response model with the optional latest-progress fields from section 3.
|
||||
- [x] 4.2 Extend `cloud-console/`'s task list/detail view to render step index, status, and summary for a task with an active in-progress assignment, refreshed on its existing polling interval; show nothing when no progress is present or the task is terminal.
|
||||
- [x] 4.3 Add Vitest coverage for the new progress rendering (present, absent, and terminal-task-hides-stale-progress cases).
|
||||
|
||||
## 5. Host Agent local console task pages (D3)
|
||||
|
||||
- [x] 5.1 Extend `AgentStatusTracker`/`/api/status` (or a small addition alongside it) to surface the current step index/status/summary for the in-progress assignment, sourced from the same progress holder built in section 2, and update the dashboard's "Current assignment" rendering to show it.
|
||||
- [x] 5.2 Add authenticated, read-only task list and task detail/timeline routes to `host_agent/web/app.py`, querying the Host-Agent-local `TaskMetadataStore`/`Timeline` (reusing the same query calls `api/console.py` makes) and rendering server-side HTML consistent with the existing dashboard's f-string + `html.escape()` convention, including inlined screenshots on the detail/timeline page.
|
||||
- [x] 5.3 Gate the new routes behind the existing Host Agent console session/CSRF protection; verify no new CORS configuration is introduced.
|
||||
- [x] 5.4 Add tests: unauthenticated requests to the new routes are rejected the same way as other console routes; task list/detail/timeline pages render expected data including screenshots for a completed task.
|
||||
|
||||
## 6. Documentation and verification
|
||||
|
||||
- [x] 6.1 Update `docs/MACOS_IPHONE_SETUP.md` and/or `docs/CLOUD_DEPLOYMENT.md` with the new Host Agent config vars (`HOST_AGENT_TASK_PROGRESS_DB_PATH` and retention settings) and a short note on where to view live/historical task progress in each console.
|
||||
- [x] 6.2 Run `uv run --all-packages pytest -m "not integration"` and targeted Cloud API / Host Agent test suites; run `cloud-console/` and `console/`-equivalent Vitest suites for the touched frontend.
|
||||
- [x] 6.3 Run Ruff check/format and `compileall` across touched packages.
|
||||
- [x] 6.4 Run `openspec validate --strict` for this change.
|
||||
- [ ] 6.5 Manual verification (requires a real Host Agent + Appium/device setup per `docs/MACOS_IPHONE_SETUP.md`): run a real task end-to-end and confirm step progress appears live in the Host Agent console and Cloud Console, and that full step history with screenshots is browsable afterward in the Host Agent console.
|
||||
|
||||
## 7. Real per-step LLM prompt/response recorded locally (D9)
|
||||
|
||||
- [x] 7.1 Extend `ToolCallDecision` (`runtime/tool_calling_client.py:23-27`) with the actual prompt content given to that call (or return it via a small side-channel from `AIPlanner.plan()`, not by changing the `Planner.plan()` return type used by other planners).
|
||||
- [x] 7.2 Update `TaskRunner._append_timeline()` (`runtime/task.py:296-319`) to pass the real per-step prompt and the model's resulting decision into `Timeline.append()`, instead of `task.goal`.
|
||||
- [x] 7.3 Update `storage/timeline.py`'s `Timeline`/`TimelineRecord` field(s) so the persisted meaning is unambiguously "the prompt actually sent to the LLM for this step" (rename or add a field; keep backward-compatible read of any already-persisted rows if a Runtime dev DB might already have old-shaped rows).
|
||||
- [x] 7.4 Update any renderer of this data (`api/console.py`, Runtime `console/`, and the new Host Agent console task pages from section 5) to show the corrected field.
|
||||
- [x] 7.5 Add unit tests: a persisted step's prompt matches what `AIPlanner.plan()` actually sent for that step (not the task goal), across at least one multi-step task.
|
||||
|
||||
## 8. Cloud persists full per-step LLM decision history via the existing cloud-planner-proxy endpoint (D7, D8)
|
||||
|
||||
- [x] 8.1 Add a new `planner_decision_log` table (`id`, `host_id`, `task_id`, `attempt`, `step_index`, `system_prompt`, `user_prompt`, `tool_name`, `arguments_json`, `created_at`) via a new Alembic migration, plus the equivalent SQLite schema/table, on both `repository.py` and `sql_repository.py`.
|
||||
- [x] 8.2 Add a repository method (e.g. `record_planner_decision(...)`) on both backends that inserts one row, assigning `step_index` as the next value scoped to `(task_id, attempt)`.
|
||||
- [x] 8.3 Extend `decide_planner_call` (`packages/cloud-platform/cloud/internal_api/api.py:367-488`) to call the new repository method with the resolved decision, immediately after `settle_host_token_reservation`, on the success path only (never on the `ToolCallUnavailable`/502 path).
|
||||
- [x] 8.4 Add a retention/prune job for `planner_decision_log` (default: prune a task's rows once it has been terminal for longer than a configurable window), mirroring section 1's Host Agent retention approach.
|
||||
- [x] 8.5 Add/extend repository contract tests so SQLite and PostgreSQL stay in parity for the new table and prune job.
|
||||
- [x] 8.6 Add unit tests: a successful decision is persisted with the correct prompt/tool_name/arguments and an incrementing step_index; a failed decision persists nothing; screenshot bytes are never written to the log even when the request included one; retention prunes rows for long-terminal tasks.
|
||||
|
||||
## 9. Cloud Console browses full LLM interaction history (D7/D8 UI)
|
||||
|
||||
- [x] 9.1 Add a Cloud API query endpoint (or extend an existing task-detail endpoint) to list `planner_decision_log` rows for a task in step order.
|
||||
- [x] 9.2 Add a `cloud-console/` view rendering a task's full LLM interaction history (prompt + resulting decision per step).
|
||||
- [x] 9.3 When a task's host used the `direct` transport (no persisted decisions and the host's configured transport is known to be `direct`), show an explicit "not reported by this host's transport" state rather than an empty list.
|
||||
- [x] 9.4 Add Vitest coverage for populated history, empty-but-cloud-transport (task hasn't produced any decisions yet), and direct-transport-hidden cases.
|
||||
Reference in New Issue
Block a user