# 原子手势 + 拟人化抖动 Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** 为 Runtime 新增 `long_press` / `double_tap` 两个原子手势,并引入一个集中的拟人化层,对所有指针动作注入坐标抖动、曲线滑动、时序抖动(`APEX_HUMANIZE_ENABLED` 默认开)。 **Architecture:** 纯数学集中在 `tools/humanize.py`;W3C Actions 序列化集中在 `driver/_w3c_actions.py`;driver 仍是薄适配器,只把点序列/单点命令翻译成 Appium 调用;工具层(`tools/*.py`)在调 driver 前套一层 humanize。`swipe` 在 humanize 开时走 `Driver.swipe_path`(多点 W3C),关时走原 `Driver.swipe`(2 点命令,零回归)。 **Tech Stack:** Python 3.13+、appium-python-client(W3C Actions / mobile commands)、pytest。 ## Global Constraints - `APEX_HUMANIZE_ENABLED` 默认 `true`;env 前缀统一 `APEX_HUMANIZE_*`。humanize 关闭时 `tap`/`swipe` 必须走与今天完全相同的 driver 方法,现有精确坐标断言零改动。 - Driver ABC 新增抽象方法后,**所有** `Driver` 子类必须同步实现,否则无法实例化。全仓库子类清单(每个 driver 任务都要改全): - `driver/wda_driver.py::WDADriver` - `driver/android_driver.py::AndroidDriver` - `tests/fakes.py::FakeDriver` - `tests/test_describe_screen.py::TreeFailingDriver` - `apps/device-host-agent/tests/test_execution.py::FakeDriver` - `apps/device-host-agent/tests/test_e2e.py::FakeDriver` - mobile 命令名(`mobile: touchAndHold` / `mobile: longClickGesture` / W3C `pause`)实现期对照已安装的 appium-python-client 源码核实,不得凭记忆。真机执行无法在本机完成,仅做源码核实 + mock 单测。 - 所有新文件首行 `from __future__ import annotations`,遵循仓库既有风格。 - 验证命令:`uv run --all-packages pytest -m "not integration" -q`;lint:`uv run --with ruff ruff check `(本机无独立 ruff)。 --- ## File Structure | 文件 | 职责 | 任务 | |---|---|---| | `tools/humanize.py`(新) | 拟人化纯函数 + config(env) | Task 1 | | `tests/test_humanize.py`(新) | humanize 单测 | Task 1 | | `driver/base.py`(改) | Driver ABC:+`long_press`/+`swipe_path`/+`double_tap` 抽象方法 | Task 2/3/4 | | `driver/_w3c_actions.py`(新) | W3C Actions payload 构造 + 发送 seam | Task 3 | | `driver/wda_driver.py`(改) | WDA 实现 3 个新方法 | Task 2/3/4 | | `driver/android_driver.py`(改) | Android 实现 3 个新方法 | Task 2/3/4 | | `tests/fakes.py`(改) | FakeDriver 补 3 方法 | Task 2/3/4 | | `tests/test_describe_screen.py`(改) | TreeFailingDriver 补 3 方法 | Task 2/3/4 | | `apps/device-host-agent/tests/test_execution.py`(改) | FakeDriver 补 3 方法 | Task 2/3/4 | | `apps/device-host-agent/tests/test_e2e.py`(改) | FakeDriver 补 3 方法 | Task 2/3/4 | | `tests/test_android_driver.py`(改) | 命令 shape + 预连接守卫 + 错误包装 | Task 2/3/4 | | `tests/test_w3c_actions.py`(新) | W3C payload 纯函数单测 | Task 3 | | `tools/long_press.py`(新) | long_press 工具 | Task 5 | | `tools/double_tap.py`(新) | double_tap 工具 | Task 6 | | `tools/tap.py`(改) | 加 humanize 钩子 | Task 7 | | `tools/swipe.py`(改) | 加 humanize 钩子(开→swipe_path) | Task 8 | | `tests/conftest.py`(改/新) | autouse fixture:测试默认关 humanize | Task 7 | | `runtime/tool_specs.py`(改) | +LONG_PRESS_SPEC/+DOUBLE_TAP_SPEC | Task 9 | | `runtime/executor.py`(改) | registry 注册 2 个新工具 | Task 9 | | `runtime/planner_prompts.py`(改) | prompt 工具枚举补 2 个 | Task 9 | 依赖顺序:1 → (2,3,4 独立,但 4 依赖 3 的 helper) → 5 依赖 1+2;6 依赖 1+4;7 依赖 1;8 依赖 1+3;9 依赖 5+6。 --- ### Task 1: `tools/humanize.py` 拟人化模块 **Files:** - Create: `tools/humanize.py` - Test: `tests/test_humanize.py` **Interfaces:** - Produces: `HumanizeConfig`、`load_humanize_config(env=None) -> HumanizeConfig`、`get_rng() -> random.Random`、`set_rng(rng|None) -> None`、`jitter_point(x, y, *, radius, rng) -> tuple[float,float]`、`jitter_duration(value_ms, *, spread, rng) -> int`、`swipe_waypoints(start, end, *, curvature, n, rng) -> list[tuple[float,float]]` - [ ] **Step 1: 写失败测试 `tests/test_humanize.py`** ```python from __future__ import annotations import math import random import pytest from tools.humanize import ( HumanizeConfig, jitter_duration, jitter_point, load_humanize_config, set_rng, swipe_waypoints, ) def test_jitter_point_stays_within_radius(): rng = random.Random(42) for _ in range(500): x, y = jitter_point(100, 200, radius=5.0, rng=rng) assert math.hypot(x - 100, y - 200) <= 5.0 def test_jitter_point_centered_on_input(): rng = random.Random(0) xs = [jitter_point(0, 0, radius=4.0, rng=rng)[0] for _ in range(4000)] assert abs(sum(xs) / len(xs)) < 0.3 def test_jitter_duration_within_spread(): rng = random.Random(3) for _ in range(200): v = jitter_duration(1000, spread=0.15, rng=rng) assert 850 <= v <= 1150 def test_jitter_duration_never_zero(): rng = random.Random(9) for _ in range(100): assert jitter_duration(1, spread=0.5, rng=rng) >= 1 def test_swipe_waypoints_endpoints_exact_interior_offset(): rng = random.Random(7) pts = swipe_waypoints((0, 0), (100, 0), curvature=0.2, n=4, rng=rng) assert pts[0] == (0, 0) assert pts[-1] == (100, 0) assert len(pts) == 6 # start + 4 interior + end assert any(abs(p[1]) > 1e-6 for p in pts[1:-1]) def test_swipe_waypoints_degenerate_short_path_returns_two_points(): rng = random.Random(1) pts = swipe_waypoints((5, 5), (5, 5), curvature=0.2, n=8, rng=rng) assert pts == [(5, 5), (5, 5)] def test_load_config_defaults_when_unset(): cfg = load_humanize_config(env={}) assert cfg == HumanizeConfig() def test_load_config_disabled(): cfg = load_humanize_config(env={"APEX_HUMANIZE_ENABLED": "false"}) assert cfg.enabled is False def test_load_config_overrides(): cfg = load_humanize_config( env={ "APEX_HUMANIZE_TAP_RADIUS_PX": "9", "APEX_HUMANIZE_SWIPE_CURVATURE": "0.3", "APEX_HUMANIZE_SWIPE_WAYPOINTS": "12", "APEX_HUMANIZE_DURATION_SPREAD": "0.2", } ) assert cfg.tap_radius_px == 9.0 assert cfg.swipe_curvature == 0.3 assert cfg.swipe_waypoints == 12 assert cfg.duration_spread == 0.2 def test_set_rng_makes_output_reproducible(): set_rng(random.Random(99)) a = jitter_point(0, 0, radius=5.0, rng=random.Random(99)) set_rng(None) # reset to default rng assert isinstance(a, tuple) and len(a) == 2 ``` - [ ] **Step 2: 运行测试确认失败** Run: `uv run --all-packages pytest tests/test_humanize.py -q` Expected: FAIL — `ModuleNotFoundError: No module named 'tools.humanize'` - [ ] **Step 3: 写实现 `tools/humanize.py`** ```python from __future__ import annotations import math import os import random from collections.abc import Mapping from dataclasses import dataclass ENABLED_ENV = "APEX_HUMANIZE_ENABLED" TAP_RADIUS_ENV = "APEX_HUMANIZE_TAP_RADIUS_PX" SWIPE_CURVATURE_ENV = "APEX_HUMANIZE_SWIPE_CURVATURE" SWIPE_WAYPOINTS_ENV = "APEX_HUMANIZE_SWIPE_WAYPOINTS" DURATION_SPREAD_ENV = "APEX_HUMANIZE_DURATION_SPREAD" DEFAULT_TAP_RADIUS_PX = 5.0 DEFAULT_SWIPE_CURVATURE = 0.15 DEFAULT_SWIPE_WAYPOINTS = 8 DEFAULT_DURATION_SPREAD = 0.15 @dataclass(frozen=True) class HumanizeConfig: enabled: bool = True tap_radius_px: float = DEFAULT_TAP_RADIUS_PX swipe_curvature: float = DEFAULT_SWIPE_CURVATURE swipe_waypoints: int = DEFAULT_SWIPE_WAYPOINTS duration_spread: float = DEFAULT_DURATION_SPREAD def load_humanize_config(env: Mapping[str, str] | None = None) -> HumanizeConfig: values = env or os.environ return HumanizeConfig( enabled=_parse_bool(values.get(ENABLED_ENV), default=True), tap_radius_px=_parse_float(values.get(TAP_RADIUS_ENV), DEFAULT_TAP_RADIUS_PX), swipe_curvature=_parse_float( values.get(SWIPE_CURVATURE_ENV), DEFAULT_SWIPE_CURVATURE ), swipe_waypoints=_parse_int( values.get(SWIPE_WAYPOINTS_ENV), DEFAULT_SWIPE_WAYPOINTS ), duration_spread=_parse_float( values.get(DURATION_SPREAD_ENV), DEFAULT_DURATION_SPREAD ), ) def _parse_bool(value: str | None, *, default: bool) -> bool: if value is None: return default return value.strip().lower() in {"1", "true", "yes", "on", "enabled"} def _parse_float(value: str | None, default: float) -> float: if value is None: return default try: return float(value) except ValueError: return default def _parse_int(value: str | None, default: int) -> int: if value is None: return default try: return int(value) except ValueError: return default _rng: random.Random | None = None def get_rng() -> random.Random: global _rng if _rng is None: _rng = random.Random() return _rng def set_rng(rng: random.Random | None) -> None: """Inject a seeded rng (tests). Pass ``None`` to reset to the default.""" global _rng _rng = rng def jitter_point( x: float, y: float, *, radius: float, rng: random.Random ) -> tuple[float, float]: """Gaussian offset clamped to ``[-radius, radius]`` so the tap never leaves a ``radius``-px box around the target (keeps it inside the button).""" dx = max(-radius, min(radius, rng.gauss(0.0, radius / 2.0))) dy = max(-radius, min(radius, rng.gauss(0.0, radius / 2.0))) return (x + dx, y + dy) def jitter_duration( value_ms: int, *, spread: float, rng: random.Random ) -> int: """Uniform jitter within ``value_ms * spread``; floored at 1ms.""" delta = value_ms * spread return max(1, int(value_ms + rng.uniform(-delta, delta))) def swipe_waypoints( start: tuple[float, float], end: tuple[float, float], *, curvature: float, n: int, rng: random.Random, ) -> list[tuple[float, float]]: """Return start + ``n`` interior + end points. Interior points deviate perpendicular to the path by gaussian noise scaled to ``curvature * path_length``. Degenerate (zero-length) path returns ``[start, end]``.""" sx, sy = start ex, ey = end dx = ex - sx dy = ey - sy length = math.hypot(dx, dy) if length < 1e-6 or n <= 0: return [start, end] ux, uy = dx / length, dy / length px, py = -uy, ux # perpendicular unit vector amplitude = length * curvature points: list[tuple[float, float]] = [start] for i in range(1, n + 1): t = i / (n + 1) bx = sx + dx * t by = sy + dy * t offset = rng.gauss(0.0, amplitude / 2.0) points.append((bx + px * offset, by + py * offset)) points.append(end) return points ``` - [ ] **Step 4: 运行测试确认通过** Run: `uv run --all-packages pytest tests/test_humanize.py -q` Expected: PASS (10 passed) - [ ] **Step 5: 提交** ```bash git add tools/humanize.py tests/test_humanize.py git commit -m "feat(humanize): add coordinate/duration/swipe-path jitter module" ``` --- ### Task 2: `Driver.long_press`(抽象 + WDA + Android + 全部 Fake) **Files:** - Modify: `driver/base.py`(新增抽象方法) - Modify: `driver/wda_driver.py` - Modify: `driver/android_driver.py` - Modify: `tests/fakes.py`、`tests/test_describe_screen.py`、`apps/device-host-agent/tests/test_execution.py`、`apps/device-host-agent/tests/test_e2e.py` - Test: `tests/test_android_driver.py` **Interfaces:** - Produces: `Driver.long_press(self, x: float, y: float, duration_ms: int = 1200) -> None`(抽象,所有子类必须实现) - WDA 命令:`mobile: touchAndHold` `{"x", "y", "duration"}`(duration 秒,float) - Android 命令:`mobile: longClickGesture` `{"x", "y"}`(系统固定时长,`duration_ms` 被忽略) - [ ] **Step 1: 核实命令名(源码,不猜测)** Run: `uv run python -c "import appium, os; print(os.path.dirname(appium.__file__))"` 定位 appium 安装目录,然后在该目录下 `grep -rn "touchAndHold\|longClickGesture" ` 确认两个命令的参数键。把核实到的精确参数键填入下面的实现(默认按 spec:WDA `{x,y,duration}`,Android `{x,y}`)。若 appium 未安装导致无法核实,在 commit message 与代码注释里标 `# verify on real Appium`。 - [ ] **Step 2: 写失败测试(追加到 `tests/test_android_driver.py`)** ```python def test_long_press_uses_long_click_gesture() -> None: driver = _connected_driver() driver.long_press(30, 40, duration_ms=1500) driver._client.execute_script.assert_called_once_with( "mobile: longClickGesture", {"x": 30, "y": 40} ) def test_long_press_wraps_exception_into_driver_error() -> None: driver = _connected_driver() driver._client.execute_script.side_effect = RuntimeError("boom") with pytest.raises(DriverError): driver.long_press(1, 2) ``` 并把 `test_operation_before_connect_raises_device_offline` 的 parametrize 列表追加: ```python ("long_press", {"x": 1, "y": 2}), ``` - [ ] **Step 3: 运行确认失败** Run: `uv run --all-packages pytest tests/test_android_driver.py -q` Expected: FAIL — `AttributeError: 'AndroidDriver' object has no attribute 'long_press'` - [ ] **Step 4: 加抽象方法(`driver/base.py`,接在 `tap` 之后)** ```python @abstractmethod def long_press(self, x: float, y: float, duration_ms: int = 1200) -> None: """Press and hold at the given coordinates for ``duration_ms``. Some platforms ignore ``duration_ms`` (fixed system hold duration). """ ``` - [ ] **Step 5: WDA 实现(`driver/wda_driver.py`,接在 `tap` 之后)** ```python def long_press(self, x: float, y: float, duration_ms: int = 1200) -> None: client = self._require_client() try: client.execute_script( "mobile: touchAndHold", {"x": x, "y": y, "duration": duration_ms / 1000}, ) except Exception as exc: raise DriverError("long press failed") from exc ``` - [ ] **Step 6: Android 实现(`driver/android_driver.py`,接在 `tap` 之后)** ```python def long_press(self, x: float, y: float, duration_ms: int = 1200) -> None: # Android ``longClickGesture`` uses a system-fixed hold duration; the # caller's ``duration_ms`` is intentionally ignored (see design R2). client = self._require_client() try: client.execute_script("mobile: longClickGesture", {"x": x, "y": y}) except Exception as exc: raise DriverError("long press failed") from exc ``` - [ ] **Step 7: 同步全部 Fake/Stub 子类** `tests/fakes.py::FakeDriver`(在 `tap` 之后加,记录调用): ```python def long_press(self, x: float, y: float, duration_ms: int = 1200) -> None: self.calls.append(("long_press", (x, y, duration_ms))) ``` 其余三处(`tests/test_describe_screen.py::TreeFailingDriver`、`apps/device-host-agent/tests/test_execution.py::FakeDriver`、`apps/device-host-agent/tests/test_e2e.py::FakeDriver`)各加一个 no-op: ```python def long_press(self, x: float, y: float, duration_ms: int = 1200) -> None: return None ``` - [ ] **Step 8: 运行确认通过** Run: `uv run --all-packages pytest tests/test_android_driver.py tests/test_describe_screen.py apps/device-host-agent/tests/test_execution.py apps/device-host-agent/tests/test_e2e.py -q` Expected: PASS(含预连接守卫 parametrize 新增项) - [ ] **Step 9: 提交** ```bash git add driver/base.py driver/wda_driver.py driver/android_driver.py tests/fakes.py tests/test_describe_screen.py tests/test_android_driver.py apps/device-host-agent/tests/test_execution.py apps/device-host-agent/tests/test_e2e.py git commit -m "feat(driver): add Driver.long_press on WDA and Android" ``` --- ### Task 3: W3C Actions helper + `Driver.swipe_path` **Files:** - Create: `driver/_w3c_actions.py` - Modify: `driver/base.py`、`driver/wda_driver.py`、`driver/android_driver.py`、4 个 Fake/Stub、`tests/test_android_driver.py` - Test: `tests/test_w3c_actions.py`(新) **Interfaces:** - Produces: `driver/_w3c_actions.py` 的 `build_swipe_actions(waypoints, total_duration_ms) -> list[dict]`、`build_double_tap_actions(x, y, interval_ms) -> list[dict]`、`perform_actions(client, actions) -> None`;`Driver.swipe_path(self, waypoints: list[tuple[float,float]], duration_ms: int) -> None`(抽象) - [ ] **Step 1: 写失败测试 `tests/test_w3c_actions.py`** ```python from __future__ import annotations import pytest from driver._w3c_actions import ( build_double_tap_actions, build_swipe_actions, ) def test_build_swipe_actions_shape_and_duration_split(): actions = build_swipe_actions([(0, 0), (50, 10), (100, 0)], 600) assert len(actions) == 1 pointer = actions[0] assert pointer["type"] == "pointer" assert pointer["id"] == "finger1" assert pointer["parameters"]["pointerType"] == "touch" seq = pointer["actions"] types = [a["type"] for a in seq] assert types == [ "pointerMove", "pointerDown", "pointerMove", "pointerMove", "pointerUp", ] # 600ms over 2 segments => 300ms per move after pointerDown moves_after_down = [a for a in seq if a["type"] == "pointerMove"][1:] assert all(m["duration"] == 300 for m in moves_after_down) assert seq[0]["x"] == 0 and seq[0]["y"] == 0 assert seq[-1]["x"] == 100 and seq[-1]["y"] == 0 def test_build_swipe_actions_requires_two_waypoints(): with pytest.raises(ValueError): build_swipe_actions([(1, 1)], 100) def test_build_double_tap_actions_shape(): actions = build_double_tap_actions(10, 20, interval_ms=80) seq = actions[0]["actions"] types = [a["type"] for a in seq] assert types == [ "pointerMove", "pointerDown", "pointerUp", "pause", "pointerDown", "pointerUp", ] pause = next(a for a in seq if a["type"] == "pause") assert pause["duration"] == 80 assert seq[0]["x"] == 10 and seq[0]["y"] == 20 ``` - [ ] **Step 2: 运行确认失败** Run: `uv run --all-packages pytest tests/test_w3c_actions.py -q` Expected: FAIL — `ModuleNotFoundError: No module named 'driver._w3c_actions'` - [ ] **Step 3: 写实现 `driver/_w3c_actions.py`** ```python from __future__ import annotations from typing import Any POINTER_ID = "finger1" def _pointer(*actions: dict[str, Any]) -> list[dict[str, Any]]: return [ { "type": "pointer", "id": POINTER_ID, "parameters": {"pointerType": "touch"}, "actions": list(actions), } ] def build_swipe_actions( waypoints: list[tuple[float, float]], total_duration_ms: int ) -> list[dict[str, Any]]: """W3C actions moving through ``waypoints`` in one continuous touch.""" if len(waypoints) < 2: raise ValueError("swipe path requires at least two waypoints") segments = len(waypoints) - 1 per_segment = max(1, total_duration_ms // segments) sx, sy = waypoints[0] actions: list[dict[str, Any]] = [ {"type": "pointerMove", "duration": 0, "x": int(sx), "y": int(sy)}, {"type": "pointerDown", "button": 0}, ] for point in waypoints[1:]: actions.append( { "type": "pointerMove", "duration": per_segment, "x": int(point[0]), "y": int(point[1]), } ) actions.append({"type": "pointerUp", "button": 0}) return _pointer(*actions) def build_double_tap_actions( x: float, y: float, interval_ms: int ) -> list[dict[str, Any]]: actions = [ {"type": "pointerMove", "duration": 0, "x": int(x), "y": int(y)}, {"type": "pointerDown", "button": 0}, {"type": "pointerUp", "button": 0}, {"type": "pause", "duration": max(1, interval_ms)}, {"type": "pointerDown", "button": 0}, {"type": "pointerUp", "button": 0}, ] return _pointer(*actions) def perform_actions(client: Any, actions: list[dict[str, Any]]) -> None: """Send a W3C actions payload via the Appium/Selenium command seam. Import is lazy so the pure payload builders stay importable without selenium on the path (used by unit tests). """ from selenium.webdriver.remote.command import Command client.execute(Command.W3C_ACTIONS, {"actions": actions}) ``` - [ ] **Step 4: 运行确认通过** Run: `uv run --all-packages pytest tests/test_w3c_actions.py -q` Expected: PASS (3 passed) - [ ] **Step 5: 写 swipe_path 失败测试(追加到 `tests/test_android_driver.py`)** ```python def test_swipe_path_sends_w3c_actions() -> None: from driver._w3c_actions import build_swipe_actions driver = _connected_driver() waypoints = [(0, 0), (50, 5), (100, 0)] driver.swipe_path(waypoints, 600) args = driver._client.execute.call_args.args assert args[1] == {"actions": build_swipe_actions(waypoints, 600)} def test_swipe_path_wraps_exception_into_driver_error() -> None: driver = _connected_driver() driver._client.execute.side_effect = RuntimeError("boom") with pytest.raises(DriverError): driver.swipe_path([(0, 0), (1, 1)], 100) ``` 并把 `test_operation_before_connect_raises_device_offline` 的 parametrize 追加: ```python ("swipe_path", {"waypoints": [(0, 0), (1, 1)], "duration_ms": 100}), ``` - [ ] **Step 6: 运行确认失败** Run: `uv run --all-packages pytest tests/test_android_driver.py -q` Expected: FAIL — `AttributeError: 'AndroidDriver' object has no attribute 'swipe_path'` - [ ] **Step 7: 加抽象方法(`driver/base.py`,接在 `swipe` 之后)** ```python @abstractmethod def swipe_path( self, waypoints: list[tuple[float, float]], duration_ms: int ) -> None: """Swipe through a sequence of waypoints in one continuous touch.""" ``` - [ ] **Step 8: WDA + Android 实现(两处相同,各接在 `swipe` 之后)** ```python def swipe_path( self, waypoints: list[tuple[float, float]], duration_ms: int ) -> None: from driver._w3c_actions import build_swipe_actions, perform_actions client = self._require_client() try: perform_actions(client, build_swipe_actions(waypoints, duration_ms)) except Exception as exc: raise DriverError("swipe_path failed") from exc ``` - [ ] **Step 9: 同步全部 Fake/Stub 子类** `tests/fakes.py::FakeDriver`: ```python def swipe_path( self, waypoints: list[tuple[float, float]], duration_ms: int ) -> None: self.calls.append(("swipe_path", (tuple(waypoints), duration_ms))) ``` 其余三处各加 no-op: ```python def swipe_path( self, waypoints: list[tuple[float, float]], duration_ms: int ) -> None: return None ``` - [ ] **Step 10: 运行确认通过** Run: `uv run --all-packages pytest tests/test_w3c_actions.py tests/test_android_driver.py tests/test_describe_screen.py apps/device-host-agent/tests/test_execution.py apps/device-host-agent/tests/test_e2e.py -q` Expected: PASS - [ ] **Step 11: 提交** ```bash git add driver/_w3c_actions.py driver/base.py driver/wda_driver.py driver/android_driver.py tests/test_w3c_actions.py tests/test_android_driver.py tests/fakes.py tests/test_describe_screen.py apps/device-host-agent/tests/test_execution.py apps/device-host-agent/tests/test_e2e.py git commit -m "feat(driver): add W3C actions helper and Driver.swipe_path" ``` --- ### Task 4: `Driver.double_tap`(用 Task 3 的 helper) **Files:** - Modify: `driver/base.py`、`driver/wda_driver.py`、`driver/android_driver.py`、4 个 Fake/Stub、`tests/test_android_driver.py` **Interfaces:** - Produces: `Driver.double_tap(self, x: float, y: float, interval_ms: int = 80) -> None`(抽象)。两平台都用 W3C Actions(`pointerDown→up→pause(interval)→down→up`),单次往返。 - [ ] **Step 1: 写失败测试(追加到 `tests/test_android_driver.py`)** ```python def test_double_tap_sends_w3c_actions() -> None: from driver._w3c_actions import build_double_tap_actions driver = _connected_driver() driver.double_tap(15, 25, interval_ms=90) args = driver._client.execute.call_args.args assert args[1] == {"actions": build_double_tap_actions(15, 25, 90)} def test_double_tap_wraps_exception_into_driver_error() -> None: driver = _connected_driver() driver._client.execute.side_effect = RuntimeError("boom") with pytest.raises(DriverError): driver.double_tap(1, 2) ``` 并把 parametrize 追加: ```python ("double_tap", {"x": 1, "y": 2}), ``` - [ ] **Step 2: 运行确认失败** Run: `uv run --all-packages pytest tests/test_android_driver.py -q` Expected: FAIL — `AttributeError: 'AndroidDriver' object has no attribute 'double_tap'` - [ ] **Step 3: 加抽象方法(`driver/base.py`,接在 `long_press` 之后)** ```python @abstractmethod def double_tap(self, x: float, y: float, interval_ms: int = 80) -> None: """Tap twice at the given coordinates with ``interval_ms`` between.""" ``` - [ ] **Step 4: WDA + Android 实现(两处相同,各接在 `long_press` 之后)** ```python def double_tap(self, x: float, y: float, interval_ms: int = 80) -> None: from driver._w3c_actions import build_double_tap_actions, perform_actions client = self._require_client() try: perform_actions(client, build_double_tap_actions(x, y, interval_ms)) except Exception as exc: raise DriverError("double tap failed") from exc ``` - [ ] **Step 5: 同步全部 Fake/Stub 子类** `tests/fakes.py::FakeDriver`: ```python def double_tap(self, x: float, y: float, interval_ms: int = 80) -> None: self.calls.append(("double_tap", (x, y, interval_ms))) ``` 其余三处各加 no-op: ```python def double_tap(self, x: float, y: float, interval_ms: int = 80) -> None: return None ``` - [ ] **Step 6: 运行确认通过** Run: `uv run --all-packages pytest tests/test_android_driver.py tests/test_describe_screen.py apps/device-host-agent/tests/test_execution.py apps/device-host-agent/tests/test_e2e.py -q` Expected: PASS - [ ] **Step 7: 提交** ```bash git add driver/base.py driver/wda_driver.py driver/android_driver.py tests/test_android_driver.py tests/fakes.py tests/test_describe_screen.py apps/device-host-agent/tests/test_execution.py apps/device-host-agent/tests/test_e2e.py git commit -m "feat(driver): add Driver.double_tap via W3C actions" ``` --- ### Task 5: `tools/long_press.py` **Files:** - Create: `tools/long_press.py` - Test: `tests/test_long_press.py` **Interfaces:** - Consumes: `tools.humanize`(Task 1)、`Driver.long_press`(Task 2) - Produces: `long_press(x, y, *, duration_ms=1200, device_id=None, manager=None) -> dict` - [ ] **Step 1: 写失败测试 `tests/test_long_press.py`** ```python from __future__ import annotations from device.manager import DeviceManager from tests.fakes import FakeDriver def _connected(driver: FakeDriver) -> DeviceManager: manager = DeviceManager() manager.register_device("phone", lambda: driver) manager.connect("phone", max_retries=1) return manager def test_long_press_disabled_passes_exact_coords(monkeypatch): monkeypatch.setenv("APEX_HUMANIZE_ENABLED", "false") from tools.long_press import long_press driver = FakeDriver() res = long_press(100, 200, duration_ms=1500, manager=_connected(driver)) assert driver.calls[-1] == ("long_press", (100, 200, 1500)) assert res["action"] == "long_press" assert res["duration_ms"] == 1500 def test_long_press_enabled_jitters_within_radius(monkeypatch): import random monkeypatch.setenv("APEX_HUMANIZE_ENABLED", "true") from tools.humanize import set_rng from tools.long_press import long_press set_rng(random.Random(5)) try: driver = FakeDriver() long_press(100, 200, duration_ms=1500, manager=_connected(driver)) finally: set_rng(None) name, args = driver.calls[-1] assert name == "long_press" px, py, _ = args assert abs(px - 100) <= 5.0 and abs(py - 200) <= 5.0 ``` - [ ] **Step 2: 运行确认失败** Run: `uv run --all-packages pytest tests/test_long_press.py -q` Expected: FAIL — `ModuleNotFoundError: No module named 'tools.long_press'` - [ ] **Step 3: 写实现 `tools/long_press.py`** ```python from __future__ import annotations from device.manager import DeviceManager from tools._device import get_driver from tools.humanize import ( get_rng, jitter_duration, jitter_point, load_humanize_config, ) def long_press( x: float, y: float, *, duration_ms: int = 1200, device_id: str | None = None, manager: DeviceManager | None = None, ) -> dict[str, object]: cfg = load_humanize_config() px, py = x, y dms = duration_ms if cfg.enabled: rng = get_rng() px, py = jitter_point(x, y, radius=cfg.tap_radius_px, rng=rng) dms = jitter_duration(duration_ms, spread=cfg.duration_spread, rng=rng) get_driver(device_id, manager=manager).long_press(px, py, dms) return {"ok": True, "action": "long_press", "x": px, "y": py, "duration_ms": dms} ``` - [ ] **Step 4: 运行确认通过** Run: `uv run --all-packages pytest tests/test_long_press.py -q` Expected: PASS (2 passed) - [ ] **Step 5: 提交** ```bash git add tools/long_press.py tests/test_long_press.py git commit -m "feat(tools): add long_press tool with humanize hook" ``` --- ### Task 6: `tools/double_tap.py` **Files:** - Create: `tools/double_tap.py` - Test: `tests/test_double_tap.py` **Interfaces:** - Consumes: `tools.humanize`(Task 1)、`Driver.double_tap`(Task 4) - Produces: `double_tap(x, y, *, interval_ms=80, device_id=None, manager=None) -> dict` - [ ] **Step 1: 写失败测试 `tests/test_double_tap.py`** ```python from __future__ import annotations from device.manager import DeviceManager from tests.fakes import FakeDriver def _connected(driver: FakeDriver) -> DeviceManager: manager = DeviceManager() manager.register_device("phone", lambda: driver) manager.connect("phone", max_retries=1) return manager def test_double_tap_disabled_passes_exact(monkeypatch): monkeypatch.setenv("APEX_HUMANIZE_ENABLED", "false") from tools.double_tap import double_tap driver = FakeDriver() res = double_tap(50, 60, interval_ms=90, manager=_connected(driver)) assert driver.calls[-1] == ("double_tap", (50, 60, 90)) assert res["action"] == "double_tap" def test_double_tap_enabled_jitters_within_radius(monkeypatch): import random monkeypatch.setenv("APEX_HUMANIZE_ENABLED", "true") from tools.humanize import set_rng from tools.double_tap import double_tap set_rng(random.Random(8)) try: driver = FakeDriver() double_tap(50, 60, manager=_connected(driver)) finally: set_rng(None) px, py, _ = driver.calls[-1][1] assert abs(px - 50) <= 5.0 and abs(py - 60) <= 5.0 ``` - [ ] **Step 2: 运行确认失败** Run: `uv run --all-packages pytest tests/test_double_tap.py -q` Expected: FAIL — `ModuleNotFoundError` - [ ] **Step 3: 写实现 `tools/double_tap.py`** ```python from __future__ import annotations from device.manager import DeviceManager from tools._device import get_driver from tools.humanize import get_rng, jitter_duration, jitter_point, load_humanize_config def double_tap( x: float, y: float, *, interval_ms: int = 80, device_id: str | None = None, manager: DeviceManager | None = None, ) -> dict[str, object]: cfg = load_humanize_config() px, py = x, y interval = interval_ms if cfg.enabled: rng = get_rng() px, py = jitter_point(x, y, radius=cfg.tap_radius_px, rng=rng) interval = jitter_duration(interval_ms, spread=cfg.duration_spread, rng=rng) get_driver(device_id, manager=manager).double_tap(px, py, interval) return { "ok": True, "action": "double_tap", "x": px, "y": py, "interval_ms": interval, } ``` - [ ] **Step 4: 运行确认通过** Run: `uv run --all-packages pytest tests/test_double_tap.py -q` Expected: PASS (2 passed) - [ ] **Step 5: 提交** ```bash git add tools/double_tap.py tests/test_double_tap.py git commit -m "feat(tools): add double_tap tool with humanize hook" ``` --- ### Task 7: `tools/tap.py` 加 humanize 钩子 + 测试默认关 humanize **Files:** - Modify: `tools/tap.py` - Modify/Create: `tests/conftest.py`(autouse fixture) - Test: `tests/test_tap_humanize.py` **Interfaces:** - Consumes: `tools.humanize`(Task 1) - 改动后 `tap(x, y, *, device_id=None, manager=None)`:humanize 开则坐标抖动,关则原样 - [ ] **Step 1: 写失败测试 `tests/test_tap_humanize.py`** ```python from __future__ import annotations from device.manager import DeviceManager from tests.fakes import FakeDriver def _connected(driver: FakeDriver) -> DeviceManager: manager = DeviceManager() manager.register_device("phone", lambda: driver) manager.connect("phone", max_retries=1) return manager def test_tap_default_off_in_tests_passes_exact(monkeypatch): # Autouse conftest fixture sets APEX_HUMANIZE_ENABLED=false; reaffirm here. monkeypatch.setenv("APEX_HUMANIZE_ENABLED", "false") from tools.tap import tap driver = FakeDriver() tap(10, 20, manager=_connected(driver)) assert driver.calls[-1] == ("tap", (10, 20)) def test_tap_enabled_jitters_within_radius(monkeypatch): import random monkeypatch.setenv("APEX_HUMANIZE_ENABLED", "true") from tools.humanize import set_rng from tools.tap import tap set_rng(random.Random(11)) try: driver = FakeDriver() tap(10, 20, manager=_connected(driver)) finally: set_rng(None) px, py = driver.calls[-1][1] assert abs(px - 10) <= 5.0 and abs(py - 20) <= 5.0 ``` - [ ] **Step 2: 加 autouse fixture(`tests/conftest.py`,文件不存在则创建)** 读取现有 `tests/conftest.py`(若存在);在文件末尾追加(不破坏既有内容): ```python import pytest @pytest.fixture(autouse=True) def _humanize_disabled_by_default_in_tests(monkeypatch): """Humanize defaults ON in production; tests default it OFF so existing exact-coordinate assertions stay deterministic. Tests that want to exercise humanize call ``monkeypatch.setenv("APEX_HUMANIZE_ENABLED", "true")`` in their own body, which overrides this fixture (test body runs after fixture setup).""" monkeypatch.setenv("APEX_HUMANIZE_ENABLED", "false") ``` 若 conftest.py 已有 `import pytest`,不要重复 import。 - [ ] **Step 3: 运行确认失败(实现未改前 tap 不抖动,第二条测试会 FAIL)** Run: `uv run --all-packages pytest tests/test_tap_humanize.py -q` Expected: `test_tap_default_off_in_tests_passes_exact` PASS、`test_tap_enabled_jitters_within_radius` FAIL(坐标未偏移) - [ ] **Step 4: 改 `tools/tap.py`** ```python from __future__ import annotations from device.manager import DeviceManager from tools._device import get_driver from tools.humanize import get_rng, jitter_point, load_humanize_config def tap( x: float, y: float, *, device_id: str | None = None, manager: DeviceManager | None = None, ) -> dict[str, object]: cfg = load_humanize_config() px, py = x, y if cfg.enabled: px, py = jitter_point(x, y, radius=cfg.tap_radius_px, rng=get_rng()) get_driver(device_id, manager=manager).tap(px, py) return {"ok": True, "action": "tap", "x": px, "y": py} ``` - [ ] **Step 5: 运行新测试 + 回归 tap 相关测试** Run: `uv run --all-packages pytest tests/test_tap_humanize.py tests/test_skill_catalog_e2e.py tests/test_semantic_task_loop.py tests/test_mcp.py -q` Expected: PASS(autouse fixture 保护了既有精确坐标断言) - [ ] **Step 6: 提交** ```bash git add tools/tap.py tests/conftest.py tests/test_tap_humanize.py git commit -m "feat(tools): humanize tap coordinates; default off in tests" ``` --- ### Task 8: `tools/swipe.py` 加 humanize 钩子(开→swipe_path) **Files:** - Modify: `tools/swipe.py` - Test: `tests/test_swipe_humanize.py` **Interfaces:** - Consumes: `tools.humanize`(Task 1)、`Driver.swipe_path`(Task 3) - 改动后 `swipe(...)`:humanize 开→`driver.swipe_path(waypoints, jittered_duration)`;关→原 `driver.swipe(...)` - [ ] **Step 1: 写失败测试 `tests/test_swipe_humanize.py`** ```python from __future__ import annotations from device.manager import DeviceManager from tests.fakes import FakeDriver def _connected(driver: FakeDriver) -> DeviceManager: manager = DeviceManager() manager.register_device("phone", lambda: driver) manager.connect("phone", max_retries=1) return manager def test_swipe_disabled_uses_two_point_swipe(monkeypatch): monkeypatch.setenv("APEX_HUMANIZE_ENABLED", "false") from tools.swipe import swipe driver = FakeDriver() swipe(0, 0, 100, 0, duration_ms=500, manager=_connected(driver)) assert driver.calls[-1] == ("swipe", (0, 0, 100, 0, 500)) def test_swipe_enabled_uses_curved_swipe_path(monkeypatch): import random monkeypatch.setenv("APEX_HUMANIZE_ENABLED", "true") from tools.humanize import set_rng from tools.swipe import swipe set_rng(random.Random(21)) try: driver = FakeDriver() swipe(0, 0, 100, 0, duration_ms=500, manager=_connected(driver)) finally: set_rng(None) name, args = driver.calls[-1] assert name == "swipe_path" waypoints, _dms = args assert waypoints[0] == (0, 0) assert waypoints[-1] == (100, 0) assert len(waypoints) > 2 # curved: more than just endpoints ``` - [ ] **Step 2: 运行确认失败** Run: `uv run --all-packages pytest tests/test_swipe_humanize.py -q` Expected: `test_swipe_disabled_uses_two_point_swipe` PASS、`test_swipe_enabled_uses_curved_swipe_path` FAIL - [ ] **Step 3: 改 `tools/swipe.py`** ```python from __future__ import annotations from device.manager import DeviceManager from tools._device import get_driver from tools.humanize import ( get_rng, jitter_duration, load_humanize_config, swipe_waypoints, ) def swipe( start_x: float, start_y: float, end_x: float, end_y: float, *, duration_ms: int = 500, device_id: str | None = None, manager: DeviceManager | None = None, ) -> dict[str, object]: cfg = load_humanize_config() if cfg.enabled: rng = get_rng() dms = jitter_duration(duration_ms, spread=cfg.duration_spread, rng=rng) waypoints = swipe_waypoints( (start_x, start_y), (end_x, end_y), curvature=cfg.swipe_curvature, n=cfg.swipe_waypoints, rng=rng, ) get_driver(device_id, manager=manager).swipe_path(waypoints, dms) return { "ok": True, "action": "swipe", "waypoints": waypoints, "duration_ms": dms, } get_driver(device_id, manager=manager).swipe( start_x, start_y, end_x, end_y, duration_ms ) return { "ok": True, "action": "swipe", "start": {"x": start_x, "y": start_y}, "end": {"x": end_x, "y": end_y}, "duration_ms": duration_ms, } ``` - [ ] **Step 4: 运行确认通过 + swipe 回归** Run: `uv run --all-packages pytest tests/test_swipe_humanize.py tests/test_android_driver.py -q` Expected: PASS - [ ] **Step 5: 提交** ```bash git add tools/swipe.py tests/test_swipe_humanize.py git commit -m "feat(tools): humanize swipe into curved W3C path when enabled" ``` --- ### Task 9: Planner 集成(tool_specs + executor registry + prompt) **Files:** - Modify: `runtime/tool_specs.py` - Modify: `runtime/executor.py` - Modify: `runtime/planner_prompts.py` - Test: `tests/test_tool_specs.py`、`tests/test_executor.py` **Interfaces:** - Consumes: `tools.long_press`(Task 5)、`tools.double_tap`(Task 6) - Produces: `LONG_PRESS_SPEC`、`DOUBLE_TAP_SPEC` 加入 `ACTION_TOOL_SPECS`;executor registry 多 2 个键;prompt 工具枚举更新 - [ ] **Step 1: 写失败测试(追加到 `tests/test_tool_specs.py`)** 先读 `tests/test_tool_specs.py` 既有断言风格,再追加: ```python def test_long_press_and_double_tap_are_action_tools(): from runtime.tool_specs import ACTION_TOOL_NAMES assert "long_press" in ACTION_TOOL_NAMES assert "double_tap" in ACTION_TOOL_NAMES def test_long_press_spec_has_duration_with_default(): from runtime.tool_specs import LONG_PRESS_SPEC props = LONG_PRESS_SPEC.parameters["properties"] assert props["x"]["type"] == "number" assert props["y"]["type"] == "number" assert props["duration_ms"]["default"] == 1200 # purpose/expected_outcome are required metadata assert "purpose" in LONG_PRESS_SPEC.parameters["required"] def test_double_tap_spec_has_interval_with_default(): from runtime.tool_specs import DOUBLE_TAP_SPEC props = DOUBLE_TAP_SPEC.parameters["properties"] assert props["interval_ms"]["default"] == 80 ``` 并在 `tests/test_executor.py` 追加: ```python def test_default_registry_dispatches_long_press_and_double_tap(): from runtime.executor import default_tool_registry registry = default_tool_registry() assert callable(registry["long_press"]) assert callable(registry["double_tap"]) ``` - [ ] **Step 2: 运行确认失败** Run: `uv run --all-packages pytest tests/test_tool_specs.py tests/test_executor.py -q` Expected: FAIL — `ImportError: cannot import name 'LONG_PRESS_SPEC'` - [ ] **Step 3: 改 `runtime/tool_specs.py`(在 `TAP_SPEC` 之后插入两个 spec,并把它们加进 `ACTION_TOOL_SPECS`)** 在 `TAP_SPEC` 定义之后插入: ```python LONG_PRESS_SPEC = ToolSpec( name="long_press", description="Press and hold a point on the screen, given in Scene pixel coordinates.", parameters=_action_parameters( required=["x", "y"], properties={ "x": {"type": "number", "description": "X coordinate in Scene pixel space."}, "y": {"type": "number", "description": "Y coordinate in Scene pixel space."}, "duration_ms": { "type": "integer", "description": "Hold duration in milliseconds (ignored on some platforms).", "default": 1200, }, }, ), ) DOUBLE_TAP_SPEC = ToolSpec( name="double_tap", description="Tap a point twice quickly, given in Scene pixel coordinates.", parameters=_action_parameters( required=["x", "y"], properties={ "x": {"type": "number", "description": "X coordinate in Scene pixel space."}, "y": {"type": "number", "description": "Y coordinate in Scene pixel space."}, "interval_ms": { "type": "integer", "description": "Milliseconds between the two taps.", "default": 80, }, }, ), ) ``` 并把 `ACTION_TOOL_SPECS` 列表改为: ```python ACTION_TOOL_SPECS: list[ToolSpec] = [ TAP_SPEC, LONG_PRESS_SPEC, DOUBLE_TAP_SPEC, SWIPE_SPEC, INPUT_TEXT_SPEC, LAUNCH_APP_SPEC, TERMINATE_APP_SPEC, ] ``` - [ ] **Step 4: 改 `runtime/executor.py::default_tool_registry`** 在 import 段加: ```python from tools.double_tap import double_tap from tools.long_press import long_press ``` 在返回 dict 内(`"tap": ...` 之后)加: ```python "long_press": _bind_manager(long_press, manager), "double_tap": _bind_manager(double_tap, manager), ``` - [ ] **Step 5: 改 `runtime/planner_prompts.py` 的工具枚举** 把 `PLANNER_SYSTEM_PROMPT` 里: ``` - One of `tap`, `swipe`, `input_text`, `launch_app`, `terminate_app` to make progress toward the goal. ``` 改为: ``` - One of `tap`, `long_press`, `double_tap`, `swipe`, `input_text`, `launch_app`, `terminate_app` to make progress toward the goal. Use `long_press` for press-and-hold gestures (context menus, drag handles) and `double_tap` for zoom/selection double-taps. ``` - [ ] **Step 6: 运行确认通过** Run: `uv run --all-packages pytest tests/test_tool_specs.py tests/test_executor.py tests/test_humanize.py tests/test_long_press.py tests/test_double_tap.py tests/test_tap_humanize.py tests/test_swipe_humanize.py tests/test_w3c_actions.py tests/test_android_driver.py -q` Expected: PASS - [ ] **Step 7: 全量回归 + lint** Run: `uv run --all-packages pytest -m "not integration" -q` Expected: 全绿(既有非集成测试无回归;留意 `test_deployment_config.py` 的预存失败是否仍为预存) Run: `uv run --with ruff ruff check tools/humanize.py tools/long_press.py tools/double_tap.py tools/tap.py tools/swipe.py driver/_w3c_actions.py driver/wda_driver.py driver/android_driver.py driver/base.py runtime/tool_specs.py runtime/executor.py runtime/planner_prompts.py` Expected: 无错误 - [ ] **Step 8: 提交** ```bash git add runtime/tool_specs.py runtime/executor.py runtime/planner_prompts.py tests/test_tool_specs.py tests/test_executor.py git commit -m "feat(planner): expose long_press/double_tap to the AI planner" ``` --- ## Self-Review(plan 完成后自查,已执行) 1. **Spec coverage**:spec §4.1 两个新原语 → Task 5/6;§4.2 humanize 模块 → Task 1;§4.3 W3C 共享 + swipe_path/double_tap → Task 3/4;§4.4 开关矩阵 → Task 7/8(tap/swipe)+ Task 5/6(新工具自带);§4.5 tool_specs+registry → Task 9;§4.6 prompt → Task 9;§3 命令矩阵 → Task 2(long_press)+ Task 3(swipe W3C)+ Task 4(double_tap W3C);§5 env → Task 1;§7 测试 → 每个任务 TDD。无遗漏。 2. **Placeholder scan**:无 TBD/TODO;每步含完整代码或确切命令。命令名核实步骤(Task 2 Step 1)有可执行指令。 3. **Type consistency**:`Driver.long_press(x,y,duration_ms=1200)`、`swipe_path(waypoints, duration_ms)`、`double_tap(x,y,interval_ms=80)` 在 ABC/两驱动/Fake/工具/spec 间签名一致;`build_swipe_actions`/`build_double_tap_actions`/`perform_actions` 在 Task 3 定义、Task 4 复用,名称一致。 ## Execution Handoff Plan complete and saved to `docs/superpowers/plans/2026-07-15-gesture-primitives-humanize.md`.(未提交,下一步先提交此 plan 再执行。)