feat: checkpoint device agent runtime milestones
This commit is contained in:
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-07-06
|
||||
@@ -0,0 +1,69 @@
|
||||
## Context
|
||||
|
||||
`agent-runtime` (`apex-agent-mvp`, code-complete but unapplied) runs a single Observe→Think→Act loop: `TaskRunner.run()` (`runtime/task.py`) calls `Planner.plan()` once per iteration and hands every produced `PlannedStep` to `Executor.execute()`, whose only failure signal is "did the tool call raise" — retried with bounded backoff (`max_retries`). Nothing in that loop ever asks "did the intended effect actually happen," using anything richer than the tool call's own return value: a `tap` on a send button can return `success=True` from the driver while the message never actually sent. `semantic-scene-runtime` (Milestone 5) and `world-model-runtime` (Milestone 6) are adding richer, semantically-informed state (`SemanticScene`, `WorldState`) that a single Planner+Executor pair has no natural place to use for verification or recovery — today that state is planning input only. This change adds three new collaborating roles — Observer, Verifier, Reflector — around the existing, unmodified Planner/Executor, so that semantic verification and bounded recovery become first-class steps in the loop, layered above (not replacing) the Executor's existing low-level retry/backoff safety net.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Define a handoff protocol — `Observation`, `VerificationVerdict`, `ReflectionOutcome` — as plain dataclasses passed between roles, so the protocol itself, not any one role's internal implementation, is the stable, spec'd contract.
|
||||
- Add a `CollaborativeTaskRunner` that composes the existing `Planner`, `Executor`, and `TaskRunner` (`agent-runtime`) strictly by import, in a defined loop: Observer → Planner → Executor → Verifier → (Reflector only on a Verifier-flagged failure) → back to Observer.
|
||||
- Make multi-agent collaboration opt-in per task run, defaulting to **disabled**, so applying this change never silently changes any existing task's cost, latency, or step count.
|
||||
- Bound Reflector-triggered replanning with an explicit ceiling, separate from and on top of the Executor's own `max_retries`, so a persistently-failing step cannot loop forever between Verifier and Reflector.
|
||||
- Reuse the existing `semantic/llm_client.py` LLM client abstraction (added by `semantic-scene-runtime`) for Verifier/Reflector reasoning, rather than introducing a second LLM client abstraction.
|
||||
|
||||
**Non-Goals:**
|
||||
- No change to `runtime/planner.py`'s `Planner`, `runtime/executor.py`'s `Executor`, or `runtime/task.py`'s `TaskRunner` — multi-agent collaboration is an opt-in alternate driver of the same Task/Planner/Executor contracts, not a replacement for them.
|
||||
- No new LLM provider/vendor integration — Verifier/Reflector reasoning reuses the existing client abstraction.
|
||||
- No persistence of `Observation`/`VerificationVerdict`/`ReflectionOutcome` into `storage/timeline.py` in this change (see Open Questions).
|
||||
- No change to `workflow/runner.py`'s step-handler contract — `CollaborativeTaskRunner` is usable as an alternate driver for a Workflow's planned-goal step, but wiring that composition is left to whichever change adopts it, not authored here.
|
||||
|
||||
## Decisions
|
||||
|
||||
### D1: New `agents/` package, not folded into `runtime/`
|
||||
The Observer/Verifier/Reflector roles and the handoff protocol are conceptually "one layer above" the existing Observe→Think→Act loop, with different characteristics (LLM calls, optionality, per-task opt-in). A sibling `agents/` package (`models.py`, `observer.py`, `verifier.py`, `reflector.py`, `collab_runner.py`, `config.py`) keeps `runtime/`'s existing contract unchanged and makes the new roles' composition explicit at the package boundary, mirroring `semantic-scene-runtime`'s `semantic/` package precedent.
|
||||
**Alternative considered**: add verification/reflection hooks directly onto `TaskRunner`. Rejected — it would force every `TaskRunner` caller to reason about the new roles' failure modes and config even when not using them.
|
||||
|
||||
### D2: Handoff protocol as plain dataclasses with no role-specific side effects
|
||||
`Observation`, `VerificationVerdict`, and `ReflectionOutcome` (plus `ReflectionAction`) are plain dataclasses with `to_dict()`/`from_dict()`, mirroring `core/models.py`'s style. No role reaches into another role's internals; each role's public surface is "takes typed input, returns typed output."
|
||||
**Alternative considered**: pass a single mutable shared context object between roles (similar to `TaskContext`). Rejected — a shared mutable object makes it harder to reason about which role produced which piece of state, and the whole point of this change is making each handoff an explicit, testable data contract.
|
||||
|
||||
### D3: `CollaborativeTaskRunner` composes `TaskRunner`/`Planner`/`Executor` by import, not fork
|
||||
`agents/collab_runner.py` imports and calls into `runtime/task.py`'s `TaskRunner` as the underlying step-execution engine, reusing its existing planning/execution/timeline-append logic rather than reimplementing it. This is the same composition pattern `workflow-orchestration-runtime`'s `WorkflowRunner` already uses for its planned-goal steps.
|
||||
**Alternative considered**: fork `TaskRunner`'s loop logic into `agents/` to have full control over each iteration. Rejected — forking creates two divergent copies of step-execution logic that must be kept in sync; composing by import guarantees `CollaborativeTaskRunner` and plain `TaskRunner` runs stay behaviorally identical wherever the new roles don't intervene.
|
||||
|
||||
### D4: Verifier/Reflector reuse `semantic/llm_client.py`, not a new client abstraction
|
||||
Both roles need an LLM call (Verifier to judge whether a step's intended effect occurred; Reflector to propose a recovery action or replan request). Both reuse the existing `semantic/llm_client.py` abstraction added by `semantic-scene-runtime`, rather than introducing a second wrapper around the same underlying SDK.
|
||||
**Alternative considered**: give each role its own bespoke client. Rejected — would duplicate timeout/retry/structured-output handling already solved once in `semantic/llm_client.py`, with no benefit specific to Verifier/Reflector's use case.
|
||||
|
||||
### D5: Reflector proposes a bounded recovery action or a replan request, never a blind re-issue
|
||||
`Reflector.reflect(...)` is invoked only when the Verifier flags a step as not-achieved. It analyzes the `Observation`/`PlannedStep`/`StepResult`/verdict and returns a `ReflectionOutcome` carrying either a `ReflectionAction` (a bounded, different corrective action) or a replan request back to the Planner — never simply re-issuing the identical failed step, since that is already the Executor's own retry's job and re-issuing an already-failed-at-the-semantic-level step is unlikely to succeed differently.
|
||||
**Alternative considered**: let Reflector re-issue the same `PlannedStep` up to N times. Rejected — the Executor already owns mechanical (tool-call-level) retry; the Reflector's value is specifically in proposing something *different* when the mechanical retry already reported success but the semantic effect didn't happen.
|
||||
|
||||
### D6: Collaboration is opt-in per task run, defaulting to disabled
|
||||
A configuration flag (`agents/config.py`) enables `CollaborativeTaskRunner` per task run, defaulting to **disabled**, mirroring `semantic-scene-runtime`'s default-off precedent. When disabled, a task runs exactly as `runtime/task.py`'s existing `TaskRunner.run()` today.
|
||||
**Alternative considered**: default to enabled for all new tasks. Rejected — this change must not silently change any existing task's cost, latency, or step count; requiring an explicit opt-in keeps the blast radius at zero for callers who don't ask for it.
|
||||
|
||||
### D7: Reflection-recovery ceiling is a separate, explicit counter from the Executor's `max_retries`
|
||||
A configurable max-reflection-recovery-attempts ceiling (per task) is tracked independently in `agents/collab_runner.py`, distinct from and layered on top of `Executor`'s own `max_retries` (which bounds *mechanical* retries of a single tool call). Once the ceiling is reached, the loop stops attempting further reflection-driven recovery for that task and surfaces the failure instead of looping indefinitely.
|
||||
**Alternative considered**: reuse/extend `Executor.max_retries` to also cover reflection attempts. Rejected — conflating the two counters would make it impossible to reason independently about "how many times did the tool call itself get retried" versus "how many times did we try a semantically different recovery," which are different failure classes with different appropriate bounds.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- **[Risk] Verifier/Reflector LLM calls add latency and cost to every collaboratively-run step** → Mitigation: opt-in default-off (D6) means only callers who explicitly enable collaboration pay this cost; reuse of the existing cached/structured-output client (D4) keeps per-call overhead in line with `semantic-scene-runtime`'s established pattern.
|
||||
- **[Risk] Verifier/Reflector depend on `SemanticScene`/`WorldState`, both of which may be absent (disabled config, degrade path, or not-yet-applied milestones)** → Mitigation: Observer must work when `SemanticScene`/`WorldState` are `None`, falling back to the raw `Scene`/`PlannedStep`/`StepResult`, matching the existing skip/fallback degrade path those capabilities define.
|
||||
- **[Risk] Verifier/Reflector could loop indefinitely on a step that never semantically succeeds** → Mitigation: the explicit reflection-recovery ceiling (D7), independent of and on top of `Executor.max_retries`.
|
||||
- **[Risk] Verifier judgment quality (false positive/negative on "did the effect happen") is not directly testable against real model output in unit tests** → Mitigation: the LLM client is injected/mockable (following `semantic/llm_client.py`'s existing pattern), so unit tests exercise the verdict/reflection parsing and loop-control logic against canned responses; only a small, explicitly-marked integration test exercises the real API.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
This is a purely additive change with no rollback complexity beyond removing the new package and config:
|
||||
1. Add the `agents/` package (`models.py`, `observer.py`, `verifier.py`, `reflector.py`, `collab_runner.py`, `config.py`); no change to `runtime/`, `core/models.py`, or `storage/timeline.py`.
|
||||
2. Add an `agents*` entry to `pyproject.toml`'s `[tool.setuptools.packages.find].include`; no new third-party dependency (reuses `semantic/llm_client.py`).
|
||||
3. Add collaboration config (enabled/disabled per task run, defaulting to disabled; max-reflection-recovery-attempts ceiling), sourced from the same single config location pattern `semantic-scene-runtime` established.
|
||||
4. Rollback is simply not enabling collaboration in config / not constructing a `CollaborativeTaskRunner`; no existing capability needs to be reverted.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Should `Observation`/`VerificationVerdict`/`ReflectionOutcome` be recorded in the timeline (`storage/timeline.py`) alongside `Task`/`StepResult`, or are they purely in-loop, non-persisted artifacts for this milestone? Leaning toward "not persisted in this change," mirroring `world-model-runtime`'s deferred `WorldState` persistence question, to keep the boundary with future milestones clean.
|
||||
- Is the reflection-recovery ceiling scoped per task or per step? Leaning toward per-task for this milestone (simpler to reason about and configure); revisit if real usage shows a single problematic step exhausting the budget for an otherwise-healthy task.
|
||||
- Should `CollaborativeTaskRunner`'s composition with `workflow-orchestration`'s `WorkflowRunner` (using it as the driver for a planned-goal step) be wired in this change or left to a follow-up change once both capabilities are applied? Leaning toward follow-up, since `workflow-orchestration-runtime`'s step-handler contract (`run(task) -> Task`) already accommodates either runner without modification.
|
||||
@@ -0,0 +1,29 @@
|
||||
## Why
|
||||
|
||||
`agent-runtime` (`apex-agent-mvp`, code-complete but unapplied) runs a single Observe→Think→Act loop: `TaskRunner.run()` calls `Planner.plan()` once per iteration and hands every produced `PlannedStep` to `Executor.execute()`, whose only failure signal is "did the tool call raise" — retried with bounded backoff. That is a *mechanical* success signal, not a *semantic* one: a `tap` on the send button can return `success=True` from the driver while the message never actually sent (wrong element, stale screen, silent app-level rejection), and nothing in the current loop would ever notice, because no role ever asks "did the intended effect actually happen" using anything richer than the tool call's own return value. As the runtime grows richer state to reason with — `SemanticScene` (Milestone 5, page identity/intents/widget purposes) and `WorldState` (Milestone 6, current app/page/variables/bounded history) — a single Planner+Executor pair has no natural place to *use* that richer state for verification or recovery; today it is planning input only. This change splits the loop's responsibilities across five collaborating roles — Observer, Planner, Executor, Verifier, Reflector — so that "did this step actually work" and "what should we do about it if not" become first-class, semantically-informed steps in the loop, layered above (not replacing) the Executor's existing low-level retry/backoff safety net.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Add a new `agents/` package implementing three new roles — **Observer** (perceives and summarizes current device state via `SemanticScene`/`WorldState`, producing an `Observation`), **Verifier** (checks whether an already-*executed* step actually achieved its intended effect, comparing pre-step and post-step `Observation`s against the `PlannedStep`'s stated intent, distinct from and running after the Executor's own low-level retry), and **Reflector** (invoked only when the Verifier flags a step as not-achieved; analyzes the likely failure cause from the `Observation`/`PlannedStep`/`StepResult`/Verifier verdict and proposes either a bounded recovery action or a replan request, never a blind re-issue of the same step).
|
||||
- Add a `CollaborativeTaskRunner` (`agents/collab_runner.py`) that composes the existing, **unmodified** `Planner` and `Executor` (`agent-runtime`) with the three new roles in a defined handoff loop: `Observer → Planner → Executor → Verifier → (Reflector only on a Verifier-flagged failure) → back to Observer`. It reuses `runtime/task.py`'s `TaskRunner` as the underlying step-execution engine (composed by import, not forked) rather than reimplementing planning/execution/timeline-append logic.
|
||||
- Define the **handoff protocol** as typed data passed between roles (`Observation`, `VerificationVerdict`, `ReflectionOutcome`), each a plain dataclass with no role-specific side effects, so the protocol itself — not any one role's internal implementation — is the stable, spec'd contract.
|
||||
- Add configuration to enable/disable multi-agent collaboration per task run, defaulting to **disabled** (mirrors `semantic-scene-runtime`'s default-off precedent) so that applying this change does not silently change any existing task's cost, latency, or step count; when disabled, a task runs exactly as `runtime/task.py`'s existing `TaskRunner.run()` today.
|
||||
- Bound Reflector-triggered replanning with an explicit ceiling (a configurable max reflection-recovery attempts per task, separate from and on top of the Executor's own `max_retries`), so a persistently-failing step cannot loop forever between Verifier and Reflector.
|
||||
- **BREAKING**: none. `runtime/planner.py`'s `Planner`, `runtime/executor.py`'s `Executor`, and `runtime/task.py`'s `TaskRunner` are not modified by this change; multi-agent collaboration is an opt-in alternate driver of the same underlying Task/Planner/Executor contracts, not a replacement.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `multi-agent-collaboration`: Defines the Observer/Verifier/Reflector roles, their `Observation`/`VerificationVerdict`/`ReflectionOutcome` data contracts, and the Observer→Planner→Executor→Verifier→(Reflector)→Observer handoff protocol that composes with the existing Planner/Executor (`agent-runtime`) as a higher semantic-verification layer above (not a replacement for) the Executor's own bounded low-level retries, usable both for a single-goal task and for one step of a multi-step Workflow (`workflow-orchestration`).
|
||||
|
||||
### Modified Capabilities
|
||||
(none — `agent-runtime`'s `Planner`, `Executor`, and `TaskRunner` (`apex-agent-mvp`) keep their existing requirements unchanged; `CollaborativeTaskRunner` composes them strictly by import as an opt-in alternate driver, and neither `semantic-scene` nor `world-model` has their specified behavior altered by this change, see Impact.)
|
||||
|
||||
## Impact
|
||||
|
||||
- **New package**: `agents/` — `models.py` (`Observation`, `VerificationVerdict`, `ReflectionOutcome`, `ReflectionAction` dataclasses), `observer.py` (`Observer.observe(...) -> Observation`), `verifier.py` (`Verifier.verify(...) -> VerificationVerdict`), `reflector.py` (`Reflector.reflect(...) -> ReflectionOutcome`), `collab_runner.py` (`CollaborativeTaskRunner`, the handoff loop), `config.py` (enable flag, max-reflection-attempts ceiling).
|
||||
- **No change** to `runtime/planner.py`, `runtime/executor.py`, `runtime/task.py`, `core/models.py`, `storage/timeline.py`, or any existing tool in `tools/` — all existing tests and callers keep working unmodified; `CollaborativeTaskRunner` composes `TaskRunner`/`Planner`/`Executor` strictly by import, as `workflow-orchestration-runtime`'s `WorkflowRunner` already does for its planned-goal steps.
|
||||
- **Reads from pending capabilities (read-only composition, no spec changes to them)**: `semantic-scene` (Milestone 5) for `SemanticScene` as Observer/Verifier input (with the existing skip/fallback degrade path honored — Observer must work when `SemanticScene` is `None`), `world-model` (Milestone 6) for `WorldState` as additional Observer/Verifier context, and `agent-runtime` (`apex-agent-mvp`) for `Task`/`PlannedStep`/`StepResult`/`TaskContext` as the underlying vocabulary every new role's inputs/outputs are built from.
|
||||
- **Composable with `workflow-orchestration`** (Milestone 8): a `WorkflowDefinition`'s planned-goal step may be driven by `CollaborativeTaskRunner` instead of the plain `TaskRunner`, without any change to `workflow/runner.py`'s step-handler contract (`run(task) -> Task` in, `Task` out) — this change does not itself modify `workflow/`.
|
||||
- **Config**: `pyproject.toml` gains an `agents*` entry in `[tool.setuptools.packages.find].include`; no new third-party dependency (roles reuse `semantic/llm_client.py`'s existing LLM client abstraction for Verifier/Reflector reasoning, added in Milestone 5, not a new SDK).
|
||||
- **Out of scope**: no new LLM provider/vendor integration; no removal or modification of the Executor's existing retry/backoff logic; no persistence of `Observation`/`VerificationVerdict`/`ReflectionOutcome` into `storage/timeline.py` (left as an open question, mirroring `world-model-runtime`'s deferred `WorldState` persistence question); no change to `skill-catalog-subscription`, `web-console`, or `skill-learning-runtime`.
|
||||
@@ -0,0 +1,59 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Handoff protocol data contracts
|
||||
The system SHALL define `Observation`, `VerificationVerdict`, `ReflectionOutcome`, and `ReflectionAction` as plain dataclasses with `to_dict()`/`from_dict()` methods, forming the stable, spec'd contract passed between the Observer, Verifier, and Reflector roles, independent of any single role's internal implementation.
|
||||
|
||||
#### Scenario: Handoff dataclasses round-trip through serialization
|
||||
- **WHEN** an `Observation`, `VerificationVerdict`, or `ReflectionOutcome` instance is serialized via `to_dict()` and then reconstructed via `from_dict()`
|
||||
- **THEN** the reconstructed instance is equal to the original instance
|
||||
|
||||
### Requirement: Observer produces an Observation from available device state
|
||||
The system SHALL provide an Observer role that produces an `Observation` summarizing current device state, using `SemanticScene` and `WorldState` when available and falling back to the raw `Scene`/`PlannedStep`/`StepResult` when either or both are absent.
|
||||
|
||||
#### Scenario: Observation succeeds with SemanticScene and WorldState present
|
||||
- **WHEN** `Observer.observe(...)` is called and both `SemanticScene` and `WorldState` are available
|
||||
- **THEN** it returns an `Observation` incorporating both as context
|
||||
|
||||
#### Scenario: Observation degrades gracefully when semantic state is absent
|
||||
- **WHEN** `Observer.observe(...)` is called and `SemanticScene` and/or `WorldState` is `None`
|
||||
- **THEN** it returns an `Observation` built from the raw `Scene`/`PlannedStep`/`StepResult` instead of raising an exception
|
||||
|
||||
### Requirement: Verifier checks whether an executed step achieved its intended effect
|
||||
The system SHALL provide a Verifier role that, after a `PlannedStep` has already been executed by the existing `Executor`, compares a pre-step and post-step `Observation` against the step's stated intent and produces a `VerificationVerdict` indicating whether the intended effect was actually achieved, distinct from and running after the Executor's own low-level tool-call retry.
|
||||
|
||||
#### Scenario: Verifier confirms an achieved effect
|
||||
- **WHEN** `Verifier.verify(...)` is called with a pre-step `Observation`, a post-step `Observation`, the executed `PlannedStep`, and its `StepResult`, and the post-step `Observation` reflects the step's stated intent
|
||||
- **THEN** it returns a `VerificationVerdict` marking the step as achieved
|
||||
|
||||
#### Scenario: Verifier flags a mechanically-successful but semantically-failed step
|
||||
- **WHEN** the `StepResult` reports mechanical success but the post-step `Observation` does not reflect the step's stated intent
|
||||
- **THEN** `Verifier.verify(...)` returns a `VerificationVerdict` marking the step as not achieved
|
||||
|
||||
### Requirement: Reflector proposes bounded recovery, never a blind re-issue
|
||||
The system SHALL invoke a Reflector role only when the Verifier produces a not-achieved `VerificationVerdict`, and the Reflector SHALL analyze the `Observation`/`PlannedStep`/`StepResult`/verdict to produce a `ReflectionOutcome` carrying either a bounded, distinct recovery `ReflectionAction` or a replan request back to the Planner, never an outcome that simply re-issues the identical failed step.
|
||||
|
||||
#### Scenario: Reflector proposes a distinct recovery action
|
||||
- **WHEN** `Reflector.reflect(...)` is invoked following a not-achieved `VerificationVerdict` and a distinct corrective action is identifiable
|
||||
- **THEN** it returns a `ReflectionOutcome` carrying a `ReflectionAction` that differs from the originally failed `PlannedStep`
|
||||
|
||||
#### Scenario: Reflector requests a replan when no bounded recovery action applies
|
||||
- **WHEN** `Reflector.reflect(...)` is invoked and no bounded corrective action is identifiable from the available `Observation`/`PlannedStep`/`StepResult`/verdict
|
||||
- **THEN** it returns a `ReflectionOutcome` carrying a replan request rather than re-issuing the failed step
|
||||
|
||||
### Requirement: Reflection-driven recovery is bounded by an explicit ceiling
|
||||
The system SHALL enforce a configurable maximum number of Reflector-triggered recovery attempts per task, tracked independently of and in addition to the Executor's own `max_retries`, so that a persistently-failing step cannot loop indefinitely between the Verifier and Reflector.
|
||||
|
||||
#### Scenario: Reflection loop stops once the ceiling is reached
|
||||
- **WHEN** a task's Verifier-Reflector loop reaches the configured maximum reflection-recovery attempts without a step being verified as achieved
|
||||
- **THEN** the `CollaborativeTaskRunner` stops attempting further reflection-driven recovery for that task and surfaces the failure instead of continuing the loop
|
||||
|
||||
### Requirement: Multi-agent collaboration composes existing Planner/Executor without modifying them
|
||||
The system SHALL provide a `CollaborativeTaskRunner` that composes the existing `Planner`, `Executor`, and `TaskRunner` (`agent-runtime`) by import, without modifying `runtime/planner.py`, `runtime/executor.py`, or `runtime/task.py`, and SHALL default to disabled so that applying this capability does not alter any existing task's behavior, latency, or cost unless explicitly enabled.
|
||||
|
||||
#### Scenario: Collaboration disabled by default leaves existing task behavior unchanged
|
||||
- **WHEN** a task is run without explicitly enabling multi-agent collaboration
|
||||
- **THEN** it executes exactly as `runtime/task.py`'s existing `TaskRunner.run()` today, with no Observer/Verifier/Reflector invoked
|
||||
|
||||
#### Scenario: Enabling collaboration does not require changes to Planner or Executor
|
||||
- **WHEN** a task is run with multi-agent collaboration explicitly enabled via `CollaborativeTaskRunner`
|
||||
- **THEN** the existing `Planner.plan()` and `Executor.execute()` are invoked unchanged, with the Observer/Verifier/Reflector roles composed around them
|
||||
@@ -0,0 +1,48 @@
|
||||
## 1. Package scaffolding
|
||||
|
||||
- [ ] 1.1 Create the `agents/` package (`__init__.py`, `models.py`, `observer.py`, `verifier.py`, `reflector.py`, `collab_runner.py`, `config.py`)
|
||||
- [ ] 1.2 Add an `agents*` entry to `[tool.setuptools.packages.find].include` in `pyproject.toml` (no new third-party dependency)
|
||||
- [ ] 1.3 Add collaboration configuration: enabled/disabled flag (default disabled) and max-reflection-recovery-attempts ceiling, sourced from a single place `agents/config.py` reads from
|
||||
- [ ] 1.4 Extend the project's smoke test (that imports every package) to import `agents`
|
||||
|
||||
## 2. Handoff protocol data models (capability: multi-agent-collaboration)
|
||||
|
||||
- [ ] 2.1 Implement `agents/models.py`: `Observation`, `VerificationVerdict`, `ReflectionOutcome`, and `ReflectionAction` dataclasses with `to_dict()`/`from_dict()`, mirroring the style of `core/models.py`
|
||||
- [ ] 2.2 Write unit tests for each dataclass round-tripping through `to_dict()`/`from_dict()`
|
||||
|
||||
## 3. Observer role (capability: multi-agent-collaboration)
|
||||
|
||||
- [ ] 3.1 Implement `agents/observer.py`: `Observer.observe(...) -> Observation`, perceiving current device state via `SemanticScene`/`WorldState` when available
|
||||
- [ ] 3.2 Make `Observer.observe(...)` work when `SemanticScene` and/or `WorldState` are `None`, falling back to the raw `Scene`/`PlannedStep`/`StepResult`
|
||||
- [ ] 3.3 Write unit tests for `Observer.observe(...)` covering: both `SemanticScene`/`WorldState` present, both absent, and each present individually
|
||||
|
||||
## 4. Verifier role (capability: multi-agent-collaboration)
|
||||
|
||||
- [ ] 4.1 Implement `agents/verifier.py`: `Verifier.verify(pre_observation, post_observation, planned_step, step_result) -> VerificationVerdict`, comparing pre-step and post-step `Observation`s against the `PlannedStep`'s stated intent
|
||||
- [ ] 4.2 Wire `Verifier` to reuse `semantic/llm_client.py`'s existing LLM client abstraction (injectable/mockable) rather than a new client
|
||||
- [ ] 4.3 Write unit tests for `Verifier.verify(...)` against a fake client covering: verified-achieved, verified-not-achieved, and client-failure degrade cases
|
||||
|
||||
## 5. Reflector role (capability: multi-agent-collaboration)
|
||||
|
||||
- [ ] 5.1 Implement `agents/reflector.py`: `Reflector.reflect(observation, planned_step, step_result, verdict) -> ReflectionOutcome`, invoked only on a Verifier-flagged not-achieved verdict
|
||||
- [ ] 5.2 Ensure `Reflector.reflect(...)` never returns an outcome that simply re-issues the identical failed `PlannedStep`; it returns either a distinct `ReflectionAction` or a replan request
|
||||
- [ ] 5.3 Write unit tests for `Reflector.reflect(...)` covering: recovery-action outcome, replan-request outcome, and client-failure degrade case
|
||||
|
||||
## 6. CollaborativeTaskRunner and handoff loop (capability: multi-agent-collaboration)
|
||||
|
||||
- [ ] 6.1 Implement `agents/collab_runner.py`: `CollaborativeTaskRunner`, composing the existing `Planner`, `Executor`, and `runtime/task.py`'s `TaskRunner` strictly by import
|
||||
- [ ] 6.2 Implement the handoff loop: Observer → Planner → Executor → Verifier → (Reflector only on a Verifier-flagged failure) → back to Observer
|
||||
- [ ] 6.3 Implement the reflection-recovery ceiling: a configurable max-attempts counter, independent of and layered on top of `Executor.max_retries`, that stops further reflection-driven recovery once exhausted and surfaces the failure
|
||||
- [ ] 6.4 Write unit tests for `CollaborativeTaskRunner` covering: a task that completes without any Verifier-flagged failure, a task recovered via one Reflector-proposed action, and a task that exhausts the reflection-recovery ceiling
|
||||
|
||||
## 7. Config wiring and opt-in behavior (capability: multi-agent-collaboration)
|
||||
|
||||
- [ ] 7.1 Confirm collaboration is disabled by default: constructing/running a task without explicitly enabling it behaves exactly as plain `TaskRunner.run()` today
|
||||
- [ ] 7.2 Write a unit test asserting `runtime/planner.py`'s `Planner`, `runtime/executor.py`'s `Executor`, and `runtime/task.py`'s `TaskRunner` are unmodified/unaffected by this change (no accidental coupling introduced from `agents/`)
|
||||
|
||||
## 8. End-to-end validation
|
||||
|
||||
- [ ] 8.1 Write an end-to-end test simulating a full collaborative task run against a mocked `Driver`/`Scene`/`SemanticScene`/`WorldState` and a mocked LLM client, asserting the loop completes normally whether verification succeeds or triggers reflection-driven recovery
|
||||
- [ ] 8.2 Write an integration test (skippable without network/API credentials, following the existing `apex-agent-mvp` skippable-integration-test pattern) exercising Verifier/Reflector against the real LLM client
|
||||
- [ ] 8.3 Confirm multi-agent collaboration stays disabled by default after applying this change (no existing task's behavior, latency, or cost changes unless a caller explicitly opts in)
|
||||
- [ ] 8.4 Run the full test suite (`pytest`) and confirm no existing test in `tests/` needed a behavior change, only additive new tests
|
||||
Reference in New Issue
Block a user