q792602257andClaude Sonnet 5
61ff3b425d
Replaces the stub Planner's fixed describe_screen/[] behavior with a real decision-maker: AIPlanner uses native tool/function calling (Anthropic or OpenAI, pluggable via AI_PLANNER_PROVIDER) to select exactly one grounded action per turn, with an explicit finish_task(success, reason) tool for completion/failure instead of an ambiguous "no tool call" signal. Default disabled (AI_PLANNER_ENABLED=false) and additive; TaskRunner falls back to the existing stub Planner unchanged when disabled. Amends CONSTITUTION.md's Perception Boundary with one narrow exception: only the AI Planner may receive the current step's raw screenshot bytes alongside Scene, for vision-grounded coordinate grounding. Also fixes a latent gap in TaskRunner.run(): observe/plan exceptions are now caught per iteration and turned into a failed task with a failure_reason, instead of propagating uncaught. openspec change: ai-planner-runtime. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
94 lines
4.7 KiB
Markdown
94 lines
4.7 KiB
Markdown
## 1. Configuration and tool face
|
|
|
|
- [x] 1.1 Create `runtime/planner_config.py`: `PlannerConfig` (`enabled`,
|
|
`provider`, `model`, `timeout`), `load_config()` reading
|
|
`AI_PLANNER_ENABLED`/`AI_PLANNER_PROVIDER`/`AI_PLANNER_MODEL`/
|
|
`AI_PLANNER_TIMEOUT_SECONDS`, `enabled` defaulting `False`
|
|
- [x] 1.2 Create `runtime/tool_specs.py`: vendor-neutral `ToolSpec` dataclass
|
|
plus `TAP_SPEC`/`SWIPE_SPEC`/`INPUT_TEXT_SPEC`/`LAUNCH_APP_SPEC`/
|
|
`TERMINATE_APP_SPEC`/`FINISH_TASK_SPEC`, matching real tool signatures,
|
|
exported as `ACTION_TOOL_SPECS` (5) and `ALL_TOOL_SPECS` (6)
|
|
- [x] 1.3 Create `runtime/planner_prompts.py`: `PLANNER_SYSTEM_PROMPT` and
|
|
`planner_user_prompt(*, goal, scene_json, history_summary)`
|
|
|
|
## 2. Dual-provider tool-calling client
|
|
|
|
- [x] 2.1 Create `runtime/tool_calling_client.py`: `ToolCallDecision`,
|
|
`ToolCallUnavailable`, `ToolCallingClient` Protocol
|
|
- [x] 2.2 Implement `AnthropicToolCallingClient` (lazy SDK import, injectable
|
|
transport, forced single tool call via
|
|
`tool_choice={"type": "any", "disable_parallel_tool_use": True}`,
|
|
image content block when a screenshot is present)
|
|
- [x] 2.3 Implement `OpenAIToolCallingClient` (lazy SDK import, injectable
|
|
transport, `max_completion_tokens`, forced single tool call via
|
|
`tool_choice="required"` + `parallel_tool_calls=False`, `image_url`
|
|
data-URI block when a screenshot is present)
|
|
- [x] 2.4 Implement `build_client(config)` provider selection
|
|
|
|
## 3. AI Planner and runtime wiring
|
|
|
|
- [x] 3.1 Create `runtime/ai_planner.py::AIPlanner` (single-step decision,
|
|
`finish_task` → empty plan / `TaskFailedError`, `goal_reached()`
|
|
permanently `False`)
|
|
- [x] 3.2 Add `screenshot: bytes | None = None` to base
|
|
`runtime/planner.py::Planner.plan()`
|
|
- [x] 3.3 Wire `runtime/task.py::TaskRunner`: `planner_config` constructor
|
|
param, `_default_planner()` (AI Planner when enabled, stub otherwise),
|
|
generalize `_planner_accepts_world()` into `_planner_accepts(name)`,
|
|
add `_planning_screenshot()`, conditionally inject `screenshot=` in
|
|
`_plan()`
|
|
- [x] 3.4 Wrap `TaskRunner.run()`'s per-iteration observe+plan in try/except,
|
|
marking the task `status="failed"` with a `failure_reason` on any
|
|
exception instead of propagating it uncaught
|
|
|
|
## 4. Constitution amendment
|
|
|
|
- [x] 4.1 Amend `docs/CONSTITUTION.md`'s Perception Boundary section with the
|
|
narrow, Planner-only screenshot exception
|
|
|
|
## 5. openspec change artifacts
|
|
|
|
- [x] 5.1 Write `proposal.md`, `design.md` (with explicit Constitution
|
|
Compliance section), `tasks.md`
|
|
- [x] 5.2 Write `specs/agent-runtime/spec.md` (`## ADDED Requirements` only)
|
|
|
|
## 6. Tests
|
|
|
|
- [x] 6.1 `tests/test_planner_config.py`: env var parsing, defaults,
|
|
`enabled` defaults `False`, invalid/negative timeout falls back to
|
|
default
|
|
- [x] 6.2 `tests/test_tool_specs.py`: each `ToolSpec.parameters` schema
|
|
(required fields, `additionalProperties: False`), `ALL_TOOL_SPECS` has
|
|
exactly 6 entries
|
|
- [x] 6.3 `tests/test_tool_calling_client.py`: fake-transport tests per
|
|
provider — forced single-tool-call fields present, image block
|
|
present/absent based on screenshot, successful response parses to
|
|
`ToolCallDecision`, malformed/error response raises
|
|
`ToolCallUnavailable`
|
|
- [x] 6.4 `tests/test_ai_planner.py`: fake `ToolCallingClient` — action
|
|
decision → single `PlannedStep`; `finish_task(success=True)` → `[]`;
|
|
`finish_task(success=False, reason=...)` → raises `TaskFailedError`;
|
|
`goal_reached()` always `False`
|
|
- [x] 6.5 Added `tests/test_ai_planner_task_runner.py` (kept
|
|
`tests/test_task_loop.py` untouched rather than extending it): planner
|
|
exception mid-task → task ends `status="failed"` with a
|
|
`failure_reason` (not stuck `running`); observer exception is caught
|
|
the same way; a narrow-signature Planner does not receive an
|
|
unexpected `screenshot` kwarg while one that declares it does;
|
|
`PlannerConfig(enabled=False)` (default) still yields the stub
|
|
`Planner`; `PlannerConfig(enabled=True)` without an explicit
|
|
`planner=` yields an `AIPlanner`
|
|
- [x] 6.6 `tests/test_ai_planner_integration.py` (`@pytest.mark.integration`,
|
|
skipped without a real API key): one real call per provider
|
|
|
|
## 7. Verification
|
|
|
|
- [x] 7.1 Ran `pytest -m "not integration"`: 314 passed, 6 deselected (up
|
|
from the pre-existing 269-test baseline plus the 45 new tests added by
|
|
this change)
|
|
- [x] 7.2 Confirmed default-off behavior: no `AI_PLANNER_*` environment
|
|
variables set in the shell → `TaskRunner()` constructs
|
|
`planner_config=PlannerConfig(enabled=False, ...)` and
|
|
`type(runner.planner) is Planner`; `uvicorn api.rest:create_app
|
|
--factory` starts cleanly with no API key configured
|