Files

T

q792602257andClaude Sonnet 5 61ff3b425d feat(agent-runtime): add LLM-driven AI Planner with dual-provider tool calling

Replaces the stub Planner's fixed describe_screen/[] behavior with a real
decision-maker: AIPlanner uses native tool/function calling (Anthropic or
OpenAI, pluggable via AI_PLANNER_PROVIDER) to select exactly one grounded
action per turn, with an explicit finish_task(success, reason) tool for
completion/failure instead of an ambiguous "no tool call" signal. Default
disabled (AI_PLANNER_ENABLED=false) and additive; TaskRunner falls back to
the existing stub Planner unchanged when disabled.

Amends CONSTITUTION.md's Perception Boundary with one narrow exception:
only the AI Planner may receive the current step's raw screenshot bytes
alongside Scene, for vision-grounded coordinate grounding. Also fixes a
latent gap in TaskRunner.run(): observe/plan exceptions are now caught per
iteration and turned into a failed task with a failure_reason, instead of
propagating uncaught.

openspec change: ai-planner-runtime.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

2026-07-12 13:48:50 +08:00

1.9 KiB

Raw Blame History

Constitution

These invariants are durable project rules. Every future milestone change must state in its design.md how it preserves them.

Device Boundary

Driver is the only device-capability contract. Concrete drivers translate external SDKs into that contract and do not own task, workflow, or planner state.

Drivers are stateless with respect to agent tasks. A driver may hold a live connection handle, but task memory, retries, plans, and workflow state belong above the driver layer.

Tool Boundary

tools/ exposes device capabilities to the runtime. Tools call into device/driver capabilities through abstractions and never import a concrete driver such as WDADriver.

Perception Boundary

Scene is the only perception artifact the LLM sees, with one narrow, explicit exception: the runtime-layer AI Planner (and only that Planner) may additionally receive the raw screenshot bytes for the current step, alongside Scene, to support vision-grounded decision-making. No other layer — api, tools, perception, storage, or any other LLM consumer — may receive raw screenshot bytes; every other perception consumer still receives Scene only. Scene itself is still produced exclusively through PerceptionProvider, not by direct calls to OCR, UI tree parsing, or scene_builder from runtime and API layers.

Runtime Boundary

The Planner produces a plan. The Executor is the only component that calls tools and handles retries for tool execution.

LLM dependencies enter at runtime through Planner behavior. HTTP and MCP dependencies enter at api. No LLM, HTTP, or MCP dependency may appear in core, driver, device, or tools.

Change Discipline

Future changes must keep the dependency direction:

core -> driver/device -> tools -> perception -> storage -> runtime -> api.

When a change needs a new external integration, add it at the adapter layer that owns that concern, not at the domain or device boundary.

1.9 KiB Raw Blame History