5 Commits

Include renames

Author	SHA1	Message	Date
q792602257	41006b098a	feat(perception): sample OCR text foreground/background colors ... PaddleOCR itself returns no color info, only text/bounds/confidence. Add pixel-level post-processing in perception/ocr.py: crop the screenshot to each OCR box, split pixels into two luminance clusters via Otsu threshold, and treat the minority cluster as the text stroke (foreground) and the majority as the background. New SceneElement.foreground_color/background_color fields ("#rrggbb", None when not OCR-sourced or sampling fails) round-trip through to_dict/from_dict alongside the existing accessibility-state fields. Planner system prompt documents the new fields as a secondary signal. pillow is promoted from an implicit paddleocr transitive dependency to an explicit direct dependency since perception/ocr.py now imports PIL directly; uv.lock re-resolved with no version change (already locked at 12.3.0).	2026-07-15 20:58:47 +08:00
q792602257	5a93651db7	feat(planner): expose long_press/double_tap to the AI planner	2026-07-15 19:45:06 +08:00
q792602257	361dada276	feat(perception): surface accessibility interaction state on UI-tree elements ... SceneElement gains enabled/clickable/selected/checked/focused (bool | None), populated from the literal attributes Appium's XCUITest and UiAutomator2 page_source already emit (iOS: enabled only; Android: all five). None means "not reported by this platform", not false. to_dict() omits unset fields to keep the LLM-facing scene JSON compact; planner_prompts.py documents the new fields so the AI planner knows how to use them (e.g. don't tap disabled elements, use selected/checked to judge whether a toggle already matches the goal).	2026-07-15 18:10:53 +08:00
q792602257	a5aeb8889c	feat(runtime): add planner reflection history with rationale and thinking ... - ToolCallDecision captures thinking blocks and pre-tool text output - AnthropicToolCallingClient supports optional extended thinking (budget_tokens + beta header) - PlannedStep carries rationale and thinking from each LLM decision - WorldEvent replaces scene_summary with rationale/thinking/page fields (backward-compatible) - AI planner system prompt instructs reflection before each tool call - _history_summary() emits compact {page, rationale, action, success} dicts - Cloud DB migration 0011 adds nullable rationale/thinking columns to planner_decision_log - OpenAI client extracts reasoning_content into thinking field	2026-07-15 12:43:22 +08:00
q792602257andClaude Sonnet 5	61ff3b425d	feat(agent-runtime): add LLM-driven AI Planner with dual-provider tool calling ... Replaces the stub Planner's fixed describe_screen/[] behavior with a real decision-maker: AIPlanner uses native tool/function calling (Anthropic or OpenAI, pluggable via AI_PLANNER_PROVIDER) to select exactly one grounded action per turn, with an explicit finish_task(success, reason) tool for completion/failure instead of an ambiguous "no tool call" signal. Default disabled (AI_PLANNER_ENABLED=false) and additive; TaskRunner falls back to the existing stub Planner unchanged when disabled. Amends CONSTITUTION.md's Perception Boundary with one narrow exception: only the AI Planner may receive the current step's raw screenshot bytes alongside Scene, for vision-grounded coordinate grounding. Also fixes a latent gap in TaskRunner.run(): observe/plan exceptions are now caught per iteration and turned into a failed task with a failure_reason, instead of propagating uncaught. openspec change: ai-planner-runtime. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-12 13:48:50 +08:00