feat(agent-runtime): add LLM-driven AI Planner with dual-provider tool calling

Replaces the stub Planner's fixed describe_screen/[] behavior with a real
decision-maker: AIPlanner uses native tool/function calling (Anthropic or
OpenAI, pluggable via AI_PLANNER_PROVIDER) to select exactly one grounded
action per turn, with an explicit finish_task(success, reason) tool for
completion/failure instead of an ambiguous "no tool call" signal. Default
disabled (AI_PLANNER_ENABLED=false) and additive; TaskRunner falls back to
the existing stub Planner unchanged when disabled.

Amends CONSTITUTION.md's Perception Boundary with one narrow exception:
only the AI Planner may receive the current step's raw screenshot bytes
alongside Scene, for vision-grounded coordinate grounding. Also fixes a
latent gap in TaskRunner.run(): observe/plan exceptions are now caught per
iteration and turned into a failed task with a failure_reason, instead of
propagating uncaught.

openspec change: ai-planner-runtime.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-12 13:48:50 +08:00
co-authored by Claude Sonnet 5
parent b94abde92a
commit 61ff3b425d
19 changed files with 1977 additions and 19 deletions
+42
View File
@@ -0,0 +1,42 @@
from __future__ import annotations
import json
from typing import Any
PLANNER_SYSTEM_PROMPT = """You are the planning brain of a mobile device automation agent.
Each turn you are given a goal, the current screen as a structured Scene (a
list of UI elements with id, type, text, and pixel bounds), and — when
available — a screenshot of the same screen and a short history of recent
actions and their outcomes.
You must call exactly one tool per turn:
- One of `tap`, `swipe`, `input_text`, `launch_app`, `terminate_app` to make
progress toward the goal.
- `finish_task` when the goal has been reached, or when it cannot be reached
and no further action would help.
Ground every coordinate you choose in the Scene element bounds (and the
screenshot, if provided) for the current turn only — never reuse coordinates
from history, since the screen may have changed. Only call `finish_task` with
`success=True` when the current Scene shows the goal has actually been
reached. Call it with `success=False` and a clear `reason` if you are stuck,
repeating the same action without progress, or the goal is not achievable.
"""
def planner_user_prompt(
*,
goal: str,
scene_json: dict[str, Any],
history_summary: list[dict[str, Any]],
) -> str:
return (
"Goal:\n"
f"{goal}\n\n"
"Current Scene (JSON):\n"
f"{json.dumps(scene_json, ensure_ascii=False, sort_keys=True)}\n\n"
"Recent history, oldest first (JSON):\n"
f"{json.dumps(history_summary, ensure_ascii=False, sort_keys=True)}\n\n"
"Call exactly one tool for this turn."
)