74 lines
18 KiB
Markdown
74 lines
18 KiB
Markdown
## Context
|
|
|
|
`runtime/task.py`'s `TaskRunner.run()` is the Observe→Think→Act→Observe loop (`agent-runtime` capability, `apex-agent-mvp`, code-complete but unapplied): each iteration calls `self.observer(device_id)` to get a `Scene`, appends it to `TaskContext.scenes`, calls `self.planner.plan(goal=..., scene=..., context=...)`, executes each `PlannedStep` via `Executor.execute()`, and appends the `StepResult` to `TaskContext.step_results`. `TaskContext` (`runtime/context.py`) is a flat dataclass — `scenes: list[Scene]`, `step_results: list[StepResult]` — with no derived/summarized state; `context.latest_scene` is the only convenience accessor today. `semantic-scene-runtime` (Milestone 5, drafted alongside this change) adds `enrich_scene(scene) -> SemanticScene | None`, a same-step, non-persisted artifact — its own design.md explicitly leaves open "should `SemanticScene` be recorded in `TaskContext`/the timeline" as a question for this milestone to answer. Today's `Planner` is a two-step stub (`runtime/planner.py`) that emits one hardcoded `describe_screen` step and declares the goal reached once any step has succeeded — it does not yet reason about page identity at all, so nothing currently consumes anything like "have I already navigated here." A future LLM-driven Planner is the actual reason this capability needs to exist: without a persisted, bounded summary of "where am I / what have I already established," a real Planner has no way to skip re-navigating to a screen it visited two steps ago except by re-scanning the entire `scenes`/`step_results` history and re-deriving semantics itself, every single step.
|
|
|
|
Two stakeholders: `runtime/task.py`'s `TaskRunner` (must gain a well-defined, optional integration point without changing behavior when `WorldModel` is not configured), and a future LLM-driven Planner (not built in this change) that will be the actual consumer reading `WorldState` to make cheaper plans.
|
|
|
|
## Goals / Non-Goals
|
|
|
|
**Goals:**
|
|
- Maintain one `WorldState` per task run — `current_app`, `current_page`, a `variables` dict, and a bounded `history` of recent semantic-scene/action pairs — updated incrementally after each executed step.
|
|
- Make the update purely a **derivation over data the loop already produces** (the just-observed `Scene`, the optional `SemanticScene`, the `PlannedStep`, and its `StepResult`) — no new I/O, no new LLM call, no new external dependency introduced by this change.
|
|
- Keep `history` genuinely bounded (a fixed-size ring buffer) so a long-running task (up to `TaskRunnerConfig.max_steps`, currently 20, but not contractually capped) cannot grow `WorldState` without limit.
|
|
- Expose `WorldState` to the Planner strictly **read-only** and **additively** — `Planner.plan()` gains an optional keyword argument with a safe default, so every existing caller and test keeps working unmodified.
|
|
- Make `WorldModel` degrade sensibly when `SemanticScene` is absent (semantic enrichment disabled, per `semantic-scene-runtime`'s default-disabled config) — page/app tracking falls back to a raw-`Scene`/action heuristic rather than going stale or raising.
|
|
|
|
**Non-Goals:**
|
|
- No cross-task or cross-device world sharing — `WorldState` is scoped to exactly one `Task`/`TaskContext` and is discarded when the task ends; a follow-on "world persistence" or "world sharing across a device's tasks" capability is a separate future decision, not made here.
|
|
- No UI for viewing or editing world state — that belongs to `web-console`'s `console-status-api`/`console-config-api`/`web-console-ui` capabilities (a separate pending change), not touched here.
|
|
- No real LLM-driven Planner — this change only makes `WorldState` available as a Planner input; teaching a Planner to actually *use* it (e.g. skip a navigation step because `current_page` already matches) is future Planner-capability work, out of scope here.
|
|
- No second LLM call to summarize/update world state — updates are deterministic rules over already-available per-step data, not a new enrichment call (see D2).
|
|
- No persistence of `WorldState` into `storage/timeline.py` or `task-memory`'s timeline format — `WorldState` lives only in the in-memory `TaskContext` for the duration of a run (see Open Questions).
|
|
|
|
## Decisions
|
|
|
|
### D1: New `world/` package, sibling to `semantic/`, not folded into `runtime/context.py`
|
|
`WorldState`/`WorldModel` get their own package (`world/models.py`, `world/model.py`, `world/config.py`) rather than growing `runtime/context.py` in place. `TaskContext` gains only a thin `world: WorldState | None` field pointing at an object `world/` owns and updates; the update *rules* (how `current_page` is derived from a `SemanticScene`, how `history` eviction works) live in `world/model.py`, not in `runtime/`. This mirrors `semantic-scene-runtime`'s own D1 (a sibling package rather than folding into the layer below) and keeps `runtime/` focused on orchestration (loop shape, retries, tool dispatch) rather than accumulating every future kind of derived state inline.
|
|
**Alternative considered**: add `current_app`/`current_page`/`variables`/`history` fields directly onto `TaskContext` and put the update logic in `runtime/task.py`'s `TaskRunner.run()`. Rejected — `TaskContext` would become a grab-bag mixing raw per-step history (`scenes`, `step_results`, owned by `agent-runtime`) with derived cross-step summarization (owned by this new capability); a future World-related change (persistence, cross-task sharing) would then have to reach into `runtime/` internals instead of a self-contained `world/` package, repeating the exact "grab-bag package" problem `device-agent-runtime-foundation` already diagnosed and fixed for `core/`.
|
|
|
|
### D2: Deterministic rule-based update, not a second LLM call
|
|
`WorldModel.observe(scene, semantic_scene, step, result)` is pure Python: if `semantic_scene` is present, `current_page = semantic_scene.page`; `current_app` is refreshed whenever `step.action in {"launch_app", "terminate_app"}` succeeds, using `step.args["bundle_id"]`/`step.args["app_id"]` (falling back to leaving `current_app` unchanged if the action fails); `variables` are updated only when `step.args` contains an explicit `"remember"` mapping (`{key: value}`) a Planner opted into; `history` appends one `WorldEvent(semantic_scene_or_scene, step.action, result.success)` per call and evicts the oldest entry once past the configured bound. No network call, no additional latency, no new failure mode beyond "field stays stale if inputs are absent."
|
|
**Alternative considered**: a second structured-output LLM call (reusing `semantic-scene-runtime`'s `llm_client.py` pattern) that looks at the accumulated history and produces a compact world-state summary each step, similar to how `SemanticScene` itself is produced. Rejected for this milestone — doubling the per-step LLM call count (one for `SemanticScene`, one for `WorldState`) doubles latency/cost for a problem (bookkeeping already-known fields) that does not need model reasoning to solve; a rule-based derivation over the same `SemanticScene`/`PlannedStep`/`StepResult` data that already exists is sufficient and strictly cheaper. This can be revisited if a future milestone's Planner needs richer world summarization than simple field-tracking + bounded history provides.
|
|
|
|
### D3: `WorldState` exposed via `TaskContext.world`, not a second context object threaded separately
|
|
`TaskRunner` sets `context.world = world_model.state` once (or updates it in place) rather than introducing a `WorldContext` parameter threaded alongside `TaskContext` through `Planner.plan()`/`Executor.execute()`. `Planner.plan()`'s new `world: WorldState | None = None` keyword is a convenience mirror of `context.world` (some future Planner implementations may prefer an explicit parameter over reaching into `context`), but the source of truth is always `TaskContext.world`.
|
|
**Alternative considered**: keep `WorldState` entirely outside `TaskContext`, passed as a wholly separate argument to `Planner.plan()`/`Executor.execute()`/`TaskRunner.run()`. Rejected — `agent-runtime`'s existing requirement is "task context/memory available during a run" as one accessible object; splitting per-run state across two parallel objects (`TaskContext` for raw history, a bare `WorldState` for derived summary) makes every future call site that needs "everything about this run" thread two parameters rather than one, for no benefit since `WorldState` is inherently `TaskContext`-scoped (one task, one world) anyway.
|
|
|
|
### D4: Bounded `history` via a fixed-size ring buffer, not unbounded list or time-based eviction
|
|
`WorldState.history: deque[WorldEvent]` is a `collections.deque(maxlen=N)` with `N` from `world/config.py` (default 10), so the oldest event is dropped automatically once the buffer is full — no separate cleanup pass, no unbounded memory growth even if a task runs far beyond `TaskRunnerConfig.max_steps`'s current default of 20 (e.g. if a future change raises that ceiling).
|
|
**Alternative considered**: keep an unbounded `list[WorldEvent]` (simplest, matches `TaskContext.scenes`/`step_results` which are also unbounded lists today). Rejected — `scenes`/`step_results` are raw historical record (their unboundedness is intentional, mirroring the timeline), whereas `history` here exists specifically to give the Planner a *recent* window without asking it to reason about a growing list; an explicit bound keeps the read-only Planner-facing contract ("recent" is a stated, fixed size) rather than an implicit "whatever the task length happens to be."
|
|
|
|
### D5: World Runtime tracking defaults to enabled (unlike Semantic Scene's default-disabled enrichment)
|
|
`world/config.py`'s enable flag defaults to `True`, and `TaskRunner`'s optional `world_model` argument, when left `None`, still gets a default `WorldModel` constructed internally (not skipped) unless the config flag is explicitly off. This differs from `semantic-scene-runtime`'s enrichment, which defaults to **disabled** specifically because it makes a paid network call every step.
|
|
**Alternative considered**: default World Runtime to disabled as well, for consistency with the Semantic Scene precedent and to minimize behavior change on apply. Rejected — `WorldModel.observe()` has no network call and no meaningful cost/latency (it is a few dict/deque operations over data the loop already holds), so the reason Semantic Scene defaults off (protect cost/latency-sensitive environments from a silent new LLM bill) does not apply here; defaulting on means the capability's read-only benefit is available to a Planner immediately, and an explicit off-switch remains for anyone who wants to opt out (e.g. a test asserting exact `TaskContext` shape pre-this-change).
|
|
|
|
### D6: `Planner.plan()` gains an optional `world` kwarg with a safe default; the existing stub `Planner` ignores it
|
|
`runtime/planner.py`'s `Planner.plan(self, *, goal, scene, context, world=None)` accepts but does not use `world` — the existing stub planner's behavior (one hardcoded `describe_screen` step, then done) is unchanged. This is intentionally a no-op integration point in this change: making a Planner *smart enough* to skip steps using `WorldState` is real LLM-Planner work reserved for a future milestone/change, not implied by adding the parameter.
|
|
**Alternative considered**: leave `Planner.plan()`'s signature untouched and require future Planner implementations to read `context.world` directly instead of a dedicated parameter. Rejected — an explicit `world` parameter documents, at the call boundary, that world-state read access is a first-class, expected part of planning (matching how `scene`/`context` are already explicit parameters rather than everything being reached off one context blob), which matters for future callers/tests constructing a `Planner` subclass against a stable, self-documenting signature.
|
|
|
|
## Risks / Trade-offs
|
|
|
|
- **[Risk]** A rule-based `current_app`/`current_page` derivation can drift from reality faster than an LLM-based one would (e.g. a page transition not captured by any `launch_app`/`terminate_app` action, or `SemanticScene.page` phrased inconsistently step to step since `semantic-scene-runtime`'s Open Questions leave intents/page labels open-vocabulary) → **Mitigation**: `WorldState` is documented and enforced as **read-only, advisory** context for the Planner, never a source of truth the Executor trusts blindly; a future Planner consuming it is expected to still verify against the current `Scene`/`SemanticScene` before skipping a step, not skip purely on stale `WorldState` says-so. This change does not build that Planner logic, only the state it would read.
|
|
- **[Risk]** Enabling World Runtime by default (D5) means every existing `TaskRunner` caller/test that doesn't explicitly configure `world_model=None` starts accumulating `WorldState` it previously didn't have → **Mitigation**: this is additive-only (`TaskContext.world` is a new field with no effect on `scenes`/`step_results`/existing assertions), the update hook cannot raise (all derivation is defensive: missing `SemanticScene`, missing `step.args` keys, etc. are all "no-op, leave field unchanged," never an exception), and `world/config.py` provides an explicit off-switch for any caller/test that wants exact pre-this-change `TaskContext` shape.
|
|
- **[Risk]** `variables`'s `step.args["remember"]` convention has no schema/validation — a Planner could stuff arbitrary or unbounded data into `variables` over many steps → **Mitigation**: out of scope to solve generally in this change (no Planner exists yet that writes to it); `world/config.py` can later gain a max-`variables`-size guard if a real Planner's usage pattern demands it, but there is no real caller to validate against yet, so adding a limit now would be speculative.
|
|
- **[Trade-off]** `WorldState` is not persisted (Non-Goal) — if a task fails partway and is retried, or if `task-memory`'s timeline is later replayed, the accumulated world state from the failed run is not recoverable, only re-derivable from scratch on the retry → acceptable per this milestone's explicit scope (single task's world only); persistence is a `task-memory`-capability change to propose separately if needed, matching how `semantic-scene-runtime` deferred the same question for its own artifact.
|
|
- **[Trade-off]** Choosing rule-based derivation (D2) over LLM-based world summarization (rejected alternative) means `WorldState`'s quality is bounded by how good `SemanticScene.page`/action-name heuristics are — if a future milestone finds this insufficient (e.g. genuinely needs to reason "the user probably navigated back," not just track the last known page), upgrading to an LLM-based summarizer is a `world/model.py`-internal change, not a `WorldState` shape change, since the shape (`current_app`/`current_page`/`variables`/`history`) is intentionally derivation-method-agnostic.
|
|
|
|
## Migration Plan
|
|
|
|
This is purely additive with optional, defensively-defaulted integration points:
|
|
1. Add the `world/` package (`models.py`: `WorldState`, `WorldEvent`; `model.py`: `WorldModel.observe()`; `config.py`: enable flag + history size).
|
|
2. Add `TaskContext.world: WorldState | None = None` to `runtime/context.py`.
|
|
3. Add an optional `world_model: WorldModel | None` constructor argument to `TaskRunner` (`runtime/task.py`); when `None` and World Runtime is enabled in config, `TaskRunner` constructs a default `WorldModel` internally; call `world_model.observe(scene, semantic_scene, step, result)` once per executed step, immediately after the existing `context.add_step_result(result)` line, and set/refresh `context.world` from the model's current state.
|
|
4. Add the optional `world: WorldState | None = None` keyword argument to `Planner.plan()` (`runtime/planner.py`); `TaskRunner` passes `context.world` through; the existing stub `Planner` implementation ignores it (no behavior change).
|
|
5. Add `world*` to `pyproject.toml`'s `[tool.setuptools.packages.find].include` list; no new third-party dependency (no LLM SDK, no new I/O library — `collections.deque` is stdlib).
|
|
6. Run `pytest` — the full existing suite must remain green with zero test-content changes (only additive assertions in new `tests/test_world_model.py`-style tests are expected, not edits to existing tests), confirming the default-enabled World Runtime tracking does not alter any existing task-completion/failure behavior.
|
|
7. Rollback: since this is a net-new package plus two small additive fields/kwargs with safe defaults, reverting is `git revert` of the commit(s); no data migration, no persisted format touched, no external system involved.
|
|
|
|
## Open Questions
|
|
|
|
- Whether `WorldState` should eventually be persisted into `storage/timeline.py`'s entries (so a replayed/inspected task shows what the runtime "believed" at each step, not just the raw `Scene`/tool-call/result) — left open per this change's Non-Goals; would be a `task-memory`-capability change to propose separately once there is a concrete consumer (e.g. `web-console`'s status API wanting to show "current believed page").
|
|
- Whether `variables` needs any validation, size bound, or namespacing convention once a real Planner starts writing to it via the `"remember"` args convention — deferred until a real Planner (a future milestone) exists to observe actual usage patterns against.
|
|
- Whether a future Skill-synthesis milestone would want `WorldState.history`'s bounded window to be configurable per-task rather than one global default — left as a global-default-only setting for this milestone, matching `semantic-scene-runtime`'s equivalent "leaning global-default-only" stance on its own model-selection config.
|
|
- Whether `current_page` derivation should fall back to something richer than "last known value, unchanged" when `SemanticScene` is absent (e.g. a raw-`Scene`-shape heuristic keyed on element text/layout signature) — this change only specifies the fallback as "leave unchanged," which is safe but potentially stale; worth revisiting once semantic enrichment's real-world enable rate is known.
|