feat(agent-runtime): add LLM-driven AI Planner with dual-provider tool calling
Replaces the stub Planner's fixed describe_screen/[] behavior with a real decision-maker: AIPlanner uses native tool/function calling (Anthropic or OpenAI, pluggable via AI_PLANNER_PROVIDER) to select exactly one grounded action per turn, with an explicit finish_task(success, reason) tool for completion/failure instead of an ambiguous "no tool call" signal. Default disabled (AI_PLANNER_ENABLED=false) and additive; TaskRunner falls back to the existing stub Planner unchanged when disabled. Amends CONSTITUTION.md's Perception Boundary with one narrow exception: only the AI Planner may receive the current step's raw screenshot bytes alongside Scene, for vision-grounded coordinate grounding. Also fixes a latent gap in TaskRunner.run(): observe/plan exceptions are now caught per iteration and turned into a failed task with a failure_reason, instead of propagating uncaught. openspec change: ai-planner-runtime. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,42 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
PLANNER_SYSTEM_PROMPT = """You are the planning brain of a mobile device automation agent.
|
||||
|
||||
Each turn you are given a goal, the current screen as a structured Scene (a
|
||||
list of UI elements with id, type, text, and pixel bounds), and — when
|
||||
available — a screenshot of the same screen and a short history of recent
|
||||
actions and their outcomes.
|
||||
|
||||
You must call exactly one tool per turn:
|
||||
- One of `tap`, `swipe`, `input_text`, `launch_app`, `terminate_app` to make
|
||||
progress toward the goal.
|
||||
- `finish_task` when the goal has been reached, or when it cannot be reached
|
||||
and no further action would help.
|
||||
|
||||
Ground every coordinate you choose in the Scene element bounds (and the
|
||||
screenshot, if provided) for the current turn only — never reuse coordinates
|
||||
from history, since the screen may have changed. Only call `finish_task` with
|
||||
`success=True` when the current Scene shows the goal has actually been
|
||||
reached. Call it with `success=False` and a clear `reason` if you are stuck,
|
||||
repeating the same action without progress, or the goal is not achievable.
|
||||
"""
|
||||
|
||||
|
||||
def planner_user_prompt(
|
||||
*,
|
||||
goal: str,
|
||||
scene_json: dict[str, Any],
|
||||
history_summary: list[dict[str, Any]],
|
||||
) -> str:
|
||||
return (
|
||||
"Goal:\n"
|
||||
f"{goal}\n\n"
|
||||
"Current Scene (JSON):\n"
|
||||
f"{json.dumps(scene_json, ensure_ascii=False, sort_keys=True)}\n\n"
|
||||
"Recent history, oldest first (JSON):\n"
|
||||
f"{json.dumps(history_summary, ensure_ascii=False, sort_keys=True)}\n\n"
|
||||
"Call exactly one tool for this turn."
|
||||
)
|
||||
Reference in New Issue
Block a user