Support multimodal planner scenes without OCR

2026-08-30 21:56:11 +08:00
parent 5b8daab457
commit 44e1a6651a
3 changed files with 23 additions and 1 deletions
@@ -72,6 +72,7 @@ export AI_PLANNER_PROVIDER=openai-compatible
 export AI_PLANNER_MODEL=qwen2.5
 export AI_PLANNER_API_KEY=local-key
 export AI_PLANNER_BASE_URL=http://127.0.0.1:11434/v1
+export AI_PLANNER_MULTIMODAL=true
 uv run --package device-host-agent device-host-agent setup
 uv run --package device-host-agent device-host-agent
 ```
@@ -115,6 +116,10 @@ The compact form `{ "role": "user", "text": "...", "image_base64": "..." }`
 is also accepted. The model can inspect the supplied image and then call a
 phone tool such as `tap` in the same conversation.

+Set `AI_PLANNER_MULTIMODAL=true` for a vision model. The Planner sends the
+screenshot and omits OCR-only elements and OCR metadata from the structured
+scene payload, avoiding duplicate OCR text.
+
 Local mode records LLM responses, reasoning fields, tool calls, tool results,
 and final replies in a local SQLite database. View them at
 `http://127.0.0.1:8765/conversations`; image bytes are excluded. Set