Support multimodal images in local chat agent

2026-08-29 21:17:20 +08:00
parent a315c62f3a
commit 71fd182f50
3 changed files with 56 additions and 4 deletions
@@ -92,6 +92,25 @@ used by the Runtime; each tool result is fed back to the model before the final
 reply is returned. The endpoint uses the Host console session cookie, so it is
 not an unauthenticated device-control endpoint.

+For vision-capable OpenAI models, a user message may contain standard OpenAI
+multimodal blocks:
+
+```json
+{
+  "messages": [{
+    "role": "user",
+    "content": [
+      {"type": "text", "text": "点击图片中的登录按钮"},
+      {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}
+    ]
+  }]
+}
+```
+
+The compact form `{ "role": "user", "text": "...", "image_base64": "..." }`
+is also accepted. The model can inspect the supplied image and then call a
+phone tool such as `tap` in the same conversation.
+
 In local mode, Appium supervision is enabled by default. Host Agent probes
 `/status`, adopts a healthy existing Appium instance, starts Appium when no
 listener exists, restarts only processes it started if they crash, and stops