q792602257/agentic-mobile-control
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
09cfe54dd97adc59dc36e4aa353ccc44e682fd98
agentic-mobile-control/openspec/changes/ai-planner-runtime/tasks.md
T
q792602257andClaude Sonnet 5 61ff3b425d feat(agent-runtime): add LLM-driven AI Planner with dual-provider tool calling 
Replaces the stub Planner's fixed describe_screen/[] behavior with a real
decision-maker: AIPlanner uses native tool/function calling (Anthropic or
OpenAI, pluggable via AI_PLANNER_PROVIDER) to select exactly one grounded
action per turn, with an explicit finish_task(success, reason) tool for
completion/failure instead of an ambiguous "no tool call" signal. Default
disabled (AI_PLANNER_ENABLED=false) and additive; TaskRunner falls back to
the existing stub Planner unchanged when disabled.

Amends CONSTITUTION.md's Perception Boundary with one narrow exception:
only the AI Planner may receive the current step's raw screenshot bytes
alongside Scene, for vision-grounded coordinate grounding. Also fixes a
latent gap in TaskRunner.run(): observe/plan exceptions are now caught per
iteration and turned into a failed task with a failure_reason, instead of
propagating uncaught.

openspec change: ai-planner-runtime.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-12 13:48:50 +08:00

4.7 KiB
Raw Blame History

1. Configuration and tool face

  • 1.1 Create runtime/planner_config.py: PlannerConfig (enabled, provider, model, timeout), load_config() reading AI_PLANNER_ENABLED/AI_PLANNER_PROVIDER/AI_PLANNER_MODEL/ AI_PLANNER_TIMEOUT_SECONDS, enabled defaulting False
  • 1.2 Create runtime/tool_specs.py: vendor-neutral ToolSpec dataclass plus TAP_SPEC/SWIPE_SPEC/INPUT_TEXT_SPEC/LAUNCH_APP_SPEC/ TERMINATE_APP_SPEC/FINISH_TASK_SPEC, matching real tool signatures, exported as ACTION_TOOL_SPECS (5) and ALL_TOOL_SPECS (6)
  • 1.3 Create runtime/planner_prompts.py: PLANNER_SYSTEM_PROMPT and planner_user_prompt(*, goal, scene_json, history_summary)

2. Dual-provider tool-calling client

  • 2.1 Create runtime/tool_calling_client.py: ToolCallDecision, ToolCallUnavailable, ToolCallingClient Protocol
  • 2.2 Implement AnthropicToolCallingClient (lazy SDK import, injectable transport, forced single tool call via tool_choice={"type": "any", "disable_parallel_tool_use": True}, image content block when a screenshot is present)
  • 2.3 Implement OpenAIToolCallingClient (lazy SDK import, injectable transport, max_completion_tokens, forced single tool call via tool_choice="required" + parallel_tool_calls=False, image_url data-URI block when a screenshot is present)
  • 2.4 Implement build_client(config) provider selection

3. AI Planner and runtime wiring

  • 3.1 Create runtime/ai_planner.py::AIPlanner (single-step decision, finish_task → empty plan / TaskFailedError, goal_reached() permanently False)
  • 3.2 Add screenshot: bytes | None = None to base runtime/planner.py::Planner.plan()
  • 3.3 Wire runtime/task.py::TaskRunner: planner_config constructor param, _default_planner() (AI Planner when enabled, stub otherwise), generalize _planner_accepts_world() into _planner_accepts(name), add _planning_screenshot(), conditionally inject screenshot= in _plan()
  • 3.4 Wrap TaskRunner.run()'s per-iteration observe+plan in try/except, marking the task status="failed" with a failure_reason on any exception instead of propagating it uncaught

4. Constitution amendment

  • 4.1 Amend docs/CONSTITUTION.md's Perception Boundary section with the narrow, Planner-only screenshot exception

5. openspec change artifacts

  • 5.1 Write proposal.md, design.md (with explicit Constitution Compliance section), tasks.md
  • 5.2 Write specs/agent-runtime/spec.md (## ADDED Requirements only)

6. Tests

  • 6.1 tests/test_planner_config.py: env var parsing, defaults, enabled defaults False, invalid/negative timeout falls back to default
  • 6.2 tests/test_tool_specs.py: each ToolSpec.parameters schema (required fields, additionalProperties: False), ALL_TOOL_SPECS has exactly 6 entries
  • 6.3 tests/test_tool_calling_client.py: fake-transport tests per provider — forced single-tool-call fields present, image block present/absent based on screenshot, successful response parses to ToolCallDecision, malformed/error response raises ToolCallUnavailable
  • 6.4 tests/test_ai_planner.py: fake ToolCallingClient — action decision → single PlannedStep; finish_task(success=True) → []; finish_task(success=False, reason=...) → raises TaskFailedError; goal_reached() always False
  • 6.5 Added tests/test_ai_planner_task_runner.py (kept tests/test_task_loop.py untouched rather than extending it): planner exception mid-task → task ends status="failed" with a failure_reason (not stuck running); observer exception is caught the same way; a narrow-signature Planner does not receive an unexpected screenshot kwarg while one that declares it does; PlannerConfig(enabled=False) (default) still yields the stub Planner; PlannerConfig(enabled=True) without an explicit planner= yields an AIPlanner
  • 6.6 tests/test_ai_planner_integration.py (@pytest.mark.integration, skipped without a real API key): one real call per provider

7. Verification

  • 7.1 Ran pytest -m "not integration": 314 passed, 6 deselected (up from the pre-existing 269-test baseline plus the 45 new tests added by this change)
  • 7.2 Confirmed default-off behavior: no AI_PLANNER_* environment variables set in the shell → TaskRunner() constructs planner_config=PlannerConfig(enabled=False, ...) and type(runner.planner) is Planner; uvicorn api.rest:create_app --factory starts cleanly with no API key configured
Reference in New Issue View Git Blame Copy Permalink