feat(trading): 浏览器掉线兜底——任务边界自愈重建，中途掉线转 needs_human
SiteInteractor 的 Chromium 是进程级单例，此前启动后即假定永远活着：全仓唯一的 is_connected() 探活在 scraping 侧，交易侧既不探活也不重启。容器里 Chromium 崩溃 是有真实前提的（/dev/shm 不足、OOM kill、seccomp 挡 sandbox，docker-compose.yml 里已有相关注释），一旦发生，进程还活着但之后每一单都会失败，且 /health 恒返回 ok，restart: unless-stopped 永远不会被触发。 更隐蔽的一条：clear_cart / enter_checkout / pay 等处的 new_page()、context.request 写在 try 之外，掉线抛的 TargetClosedError 不是 AppError，会穿过 runner 的 except AppError 落到主循环那个只记日志的兜底里——任务一次都不上报，网关侧要干等 整个 lease_ttl（默认 300s）才被 sweep 置 stale。clear_cart 是 execute() 的 step 0， 浏览器死时最先撞上的正是它。 分三层处理： - 任务边界自愈。_launch() 从 start() 抽出，_context_options() 统一 context 参数 （重建必须与启动完全一致，指纹漂移就是一次风控事件）；_refresh_context_if_stale 改名 _ensure_context_ready()，先探活重建再做原有的 storage_state mtime 检查 （顺序不可换，mtime 重建要用 self._browser）。重建前丢弃 _checkout_pages 里的 残留确认页并记 warning——那些 Page 已随浏览器一起没了。 - 中途掉线不重建，抛 BrowserDeadError（新增，5006）。新增 _new_page() 与 _request() 两个壳收口裸异常；_request() 只在确认浏览器真死了时才改写异常， 站点 5xx 这类正常业务失败原样抛出。submit_order / pay 入口用 _require_live_browser() 直接拒绝：这两步复用 enter_checkout 留存的 Page， 重建救不回服务端订单草稿，而 pay 跑的时候订单已经真的提交了。顺带修掉一个 误诊——浏览器死时 submit_order 原先报「未找到确认按钮」，把「浏览器崩了」 说成「站点改版了」，两者的处置方式完全不同。 - runner 把 BrowserDeadError 转 needs_human 而非 failed，except 分支排在 except AppError 之前（子类，顺序反了就报 failed）。掉线发生在动作中途， 站点侧生效与否无从判断，不能给上游「明确失败」的结论。 /health 暴露 browser 状态，掉线时 degraded + HTTP 503 + code 5006，让 Dockerfile.trading 的 HEALTHCHECK 探到并重启容器。重启不会导致重复下单：网关侧 任务绝不自动重投，租约过期只置 stale 等人工 reclaim（docs/order-gateway.md §5）， 重启只是恢复领新任务的能力。启动窗口期 started=False 不算掉线。 README 错误码表补 5005（此前遗漏）与 5006。 新增 15 个用例覆盖探活三态、is_connected() 自身抛错、边界重建/不重建/丢弃残留页/ 重建失败、两个包装壳的分支、submit/pay 拒绝、runner 转 needs_human、/health 503。 真实 Chromium 崩溃无法在离线测试里制造，用例模拟的是 is_connected() 返回 False 这个唯一可观测信号，覆盖的是代码对该信号的反应而非崩溃本身；容器 HEALTHCHECK 真的触发重启这条链路尚未实跑验证。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
@@ -168,6 +168,11 @@ USER rakuten
|
||||
RUN /app/.venv/bin/playwright install chromium
|
||||
|
||||
# /health 只读缓存登录态，不触发站点请求，适合高频探活。
|
||||
# 交易服务的 /health 在 Playwright 浏览器掉线时返回 503（curl -f 视为失败），
|
||||
# 于是这里的 retries 用尽后容器被 restart 掉——进程还活着但浏览器已死时，所有站点
|
||||
# 操作都做不了，重启是最后一道兜底。重启不会导致重复下单：网关侧任务绝不自动重投
|
||||
# （租约过期只置 stale 等人工 reclaim，见 docs/order-gateway.md §5）。
|
||||
# start-period 覆盖启动期，那时 browser 还没 start()，不算掉线。
|
||||
# 端口取 RAKUTEN_HEALTH_PORT，未设时用交易端口；跑网关的容器把它设成 31109。
|
||||
HEALTHCHECK --interval=30s --timeout=5s --start-period=45s --retries=3 \
|
||||
CMD curl -fsS http://127.0.0.1:${RAKUTEN_HEALTH_PORT:-${RAKUTEN_TRADING_PORT}}/health || exit 1
|
||||
|
||||
@@ -125,7 +125,7 @@ PC UA 在搜索页、详情页、店铺页上都能拿到完整模板。因此
|
||||
|
||||
| 接口 | 说明 |
|
||||
| --- | --- |
|
||||
| `GET /health` | 健康检查，含乐天账号登录态（只读缓存，不打站点） |
|
||||
| `GET /health` | 健康检查，含乐天账号登录态（只读缓存，不打站点）与浏览器连接状态；浏览器掉线时返回 **503** |
|
||||
| `POST /api/auth/status` | 查询登录态，默认真实探测一次 |
|
||||
| `POST /api/auth/login` | 按 `account.yaml` 自动登录（已登录则跳过；撞验证码要人工接管） |
|
||||
| `POST /api/auth/reload` | 人工重新登录后免重启换上新 cookie |
|
||||
@@ -639,6 +639,8 @@ trading 加购时的字段选择策略：多规格挑第一个非售罄的 varia
|
||||
| 5002 | 加购失败（交易服务） | 400 |
|
||||
| 5003 | 下单失败（交易服务） | 400 |
|
||||
| 5004 | 下单安全闸门未通过：未显式确认或金额超上限（交易服务） | 400 |
|
||||
| 5005 | 结算被站点风控拦截：session upgrade / 3DS 等需人工验证（交易服务） | 400 |
|
||||
| 5006 | Playwright 浏览器已掉线（交易服务；也是 `/health` degraded 时的 code） | 400 / 503 |
|
||||
| 6001 | 任务不存在（网关） | 404 |
|
||||
| 6002 | 租约无效：不是持有者、已过期或任务已终结（网关） | 409 |
|
||||
| 6003 | 任务状态不允许该操作（如对已终结任务 reclaim）（网关） | 409 |
|
||||
@@ -649,6 +651,10 @@ trading 加购时的字段选择策略：多规格挑第一个非售罄的 varia
|
||||
错误码在两站、三个服务之间通用。ラクマ 链路不会出现 `3002`（无反爬拦截行为）
|
||||
与 `4002`（无子站跳转）；`5xxx` 只会来自交易服务——抓取服务全程匿名，不会有登录态问题。
|
||||
`5001` 与 `5004` 都标记为不可重试：前者要人工重新登录，后者要调用方改入参。
|
||||
`5005` 与 `5006` 同样不可重试，且都转 `needs_human`：前者是站点主动要求人工验证，
|
||||
后者是浏览器在动作中途没了、站点侧生效与否无从判断——下单不可逆，这种时候必须
|
||||
停下来等人核对，不能赌。浏览器在**任务边界**掉线不会产生 `5006`，交易服务会就地
|
||||
重建一套继续跑；重建失败或掉线发生在一次调用途中，才抛这个码。
|
||||
`6xxx` 只会来自网关：`6001`–`6004`（下单任务通道）全部标记为不可重试——任务编排侧
|
||||
重试无意义，部分场景（如租约过期）重试可能变成重复下单；`6005`–`6006`
|
||||
（账号只读查询通道）**可以重试**——查询只读，重发没有副作用
|
||||
|
||||
@@ -172,6 +172,30 @@ class OrderGuardError(AppError):
|
||||
super().__init__(message=message, code="ORDER_GUARD", err_code=5004, retryable=False)
|
||||
|
||||
|
||||
class BrowserDeadError(AppError):
|
||||
"""Playwright 浏览器已掉线（Chromium 崩溃 / 被 OOM kill / 驱动连接断开）
|
||||
|
||||
单独成一类而不是复用 OrderOperationError，有两个理由：
|
||||
|
||||
1. **必须是 AppError**。掉线时 Playwright 抛的是 `TargetClosedError` 这类
|
||||
非 AppError 异常，会直接穿过 `runner._execute_with_renewal` 的
|
||||
`except AppError`，落到主循环那个只记日志的兜底里——任务一次都不上报，
|
||||
网关侧要干等整个 lease_ttl（默认 300s）才被 sweep 置 stale。包成 AppError
|
||||
是为了让 worker 能**立刻**回报。
|
||||
2. **结论必须是 needs_human 而不是 failed**。浏览器是在动作中途没的，站点侧
|
||||
到底生效没有无从判断（尤其 submit_order / pay 之后），按「明确失败」上报
|
||||
会误导上游。`runner` 为此单独接这一类，见 _execute_with_renewal。
|
||||
|
||||
掉线本身能自愈的部分在 `SiteInteractor._ensure_browser_alive()`：**任务边界**
|
||||
上探到浏览器没了会就地重建，那条路径不抛本异常。抛到这里的都是重建也救不回来
|
||||
的场景（重建失败、掉线发生在一次调用中途、submit/pay 复用的确认页已随浏览器
|
||||
一起消失）。
|
||||
"""
|
||||
|
||||
def __init__(self, message: str = "浏览器已掉线"):
|
||||
super().__init__(message=message, code="BROWSER_DEAD", err_code=5006, retryable=False)
|
||||
|
||||
|
||||
class CheckoutBlockedError(AppError):
|
||||
"""结算流程被站点风控拦截（session upgrade 二次验证 / 3DS / 短信验证等）
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
"""交易服务健康检查路由"""
|
||||
from fastapi import APIRouter, Depends
|
||||
from fastapi import APIRouter, Depends, Response
|
||||
|
||||
from app.shared.api import ApiResponse, get_container
|
||||
from app.trading.container import TradingContainer
|
||||
@@ -10,6 +10,7 @@ router = APIRouter(tags=["health"])
|
||||
|
||||
@router.get("/health", response_model=ApiResponse[TradingHealthData])
|
||||
async def health(
|
||||
response: Response,
|
||||
container: TradingContainer = Depends(get_container),
|
||||
) -> ApiResponse[TradingHealthData]:
|
||||
"""交易服务健康状态
|
||||
@@ -18,10 +19,34 @@ async def health(
|
||||
上游高频轮询时反复打站点；要实时结果请用 POST /api/auth/status。
|
||||
|
||||
注意 `logged_in=null` 表示服务启动后还没探测过，不等于未登录。
|
||||
|
||||
browser 反映 Playwright 浏览器的连接状态。**浏览器已掉线时本接口返回 503**：
|
||||
这个进程的所有站点操作都要靠那一个浏览器，它没了以后进程虽然还活着、端口还
|
||||
通，却已经什么都干不了（每单都会失败）。返回 503 是为了让容器 HEALTHCHECK
|
||||
（Dockerfile.trading）能探到并触发重启——这是掉线自愈的最后一道兜底，前面还有
|
||||
SiteInteractor 在任务边界的就地重建。
|
||||
|
||||
重启是安全的：网关侧的下单任务绝不自动重投（租约过期只置 stale 等人工
|
||||
reclaim，见 docs/order-gateway.md §5），所以重启只是让 worker 恢复领**新**
|
||||
任务的能力，不会让任何一笔已经在跑的订单被重复下单。
|
||||
|
||||
尚未 start() 的启动窗口期不算掉线（HEALTHCHECK 有 start-period 兜着），
|
||||
只有「起过浏览器且现在连不上」才转 degraded。
|
||||
"""
|
||||
# site 恒非空（lifespan 必建），留 None 分支只是不为一个健康检查赌这一点
|
||||
browser = container.site.browser_status() if container.site is not None else {}
|
||||
degraded = bool(browser.get("started")) and not browser.get("alive")
|
||||
if degraded:
|
||||
response.status_code = 503
|
||||
|
||||
return ApiResponse[TradingHealthData](
|
||||
success=True,
|
||||
msg="success",
|
||||
data=TradingHealthData(status="ok", auth=container.auth_session.status_all()),
|
||||
code=0,
|
||||
success=not degraded,
|
||||
msg="浏览器已掉线" if degraded else "success",
|
||||
data=TradingHealthData(
|
||||
status="degraded" if degraded else "ok",
|
||||
auth=container.auth_session.status_all(),
|
||||
browser=browser,
|
||||
),
|
||||
# 与 BrowserDeadError 用同一个错误码，上游按同一张错误码表分支
|
||||
code=5006 if degraded else 0,
|
||||
)
|
||||
|
||||
+11
-1
@@ -85,10 +85,20 @@ class TradingHealthData(BaseModel):
|
||||
|
||||
只读缓存的登录态，不触发网络探测——健康检查会被高频轮询，实时结果请用
|
||||
POST /api/auth/status。
|
||||
|
||||
browser 是唯一一个「实时」字段，但它只读 Playwright 的本地连接状态
|
||||
（`browser.is_connected()`），不打站点也不进事件循环，高频探活没有代价。
|
||||
"""
|
||||
|
||||
status: str = Field(description="服务状态，健康为 ok")
|
||||
status: str = Field(description="服务状态：ok 正常，degraded 表示浏览器已掉线（HTTP 503）")
|
||||
auth: dict[str, Any] = Field(default_factory=dict, description="缓存的各站点登录态（不触发网络探测）")
|
||||
browser: dict[str, Any] = Field(
|
||||
default_factory=dict,
|
||||
description=(
|
||||
"Playwright 浏览器状态：started 是否已跑过 start()，alive 连接是否正常，"
|
||||
"pending_checkout_tasks 仍留有下单确认页的 task_id，detail 文字说明"
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
# ---- 购物车接口（/api/cart/*）----
|
||||
|
||||
@@ -28,7 +28,12 @@ import contextlib
|
||||
import logging
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from app.shared.errors import AppError, CheckoutBlockedError, OrderGuardError
|
||||
from app.shared.errors import (
|
||||
AppError,
|
||||
BrowserDeadError,
|
||||
CheckoutBlockedError,
|
||||
OrderGuardError,
|
||||
)
|
||||
from app.shared.task_state import OrderState, TaskStatus
|
||||
from app.trading.worker import verify
|
||||
from app.trading.worker.client import GatewayClient
|
||||
@@ -240,6 +245,23 @@ class WorkerRunner:
|
||||
terminal=True,
|
||||
terminal_status=TaskStatus.NEEDS_HUMAN,
|
||||
)
|
||||
except BrowserDeadError as exc:
|
||||
# 浏览器在执行途中没了：站点侧到底生效没有无从判断（尤其掉线发生在
|
||||
# submit_order / pay 前后），按 needs_human 交人工核对订单列表。
|
||||
# 必须排在 except AppError 前面——BrowserDeadError 是 AppError 的
|
||||
# 子类，顺序反了就会被当成普通失败报 failed。
|
||||
logger.error(
|
||||
"浏览器掉线导致任务中断，转 needs_human：task_id=%s msg=%s",
|
||||
task.task_id,
|
||||
exc.message,
|
||||
)
|
||||
await self._report_safe(
|
||||
task,
|
||||
state=_coerce_state(task.known_state),
|
||||
detail=exc.message,
|
||||
terminal=True,
|
||||
terminal_status=TaskStatus.NEEDS_HUMAN,
|
||||
)
|
||||
except AppError as exc:
|
||||
logger.warning(
|
||||
"执行失败：task_id=%s code=%s msg=%s",
|
||||
|
||||
@@ -97,6 +97,7 @@ from datetime import datetime
|
||||
from typing import TYPE_CHECKING, TypeVar
|
||||
|
||||
from app.shared.errors import (
|
||||
BrowserDeadError,
|
||||
CartOperationError,
|
||||
CheckoutBlockedError,
|
||||
InvalidRequestError,
|
||||
@@ -117,6 +118,7 @@ from app.trading.worker.models import LeaseTask
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Awaitable, Callable
|
||||
from pathlib import Path
|
||||
|
||||
from playwright.async_api import Page
|
||||
|
||||
@@ -620,6 +622,20 @@ class SiteInteractor:
|
||||
浏览器 context 复用同一份登录态，所有任务串行（self._lock + worker 主循环本就串行），
|
||||
不需要为每个任务开新 context——开销大且 cookie 状态会乱。
|
||||
|
||||
**掉线兜底**：Chromium 可能崩溃（容器里 /dev/shm 不足、被 OOM kill、
|
||||
seccomp 挡住 sandbox 等，见 docker-compose.yml 注释），而这个浏览器是进程级
|
||||
单例，一旦没了后续每一单都会失败。处理分两层：
|
||||
|
||||
- 任务边界（各公开方法入口的 `_ensure_context_ready()`）探到掉线就**就地重建**
|
||||
一套完全一样的 playwright/browser/context，调用方无感知。
|
||||
- 一次调用**中途**掉线不重建：站点侧生效与否无从判断，统一抛 `BrowserDeadError`
|
||||
（AppError 的一种），由 runner 转 needs_human 交人工核对。submit_order / pay
|
||||
因为复用 enter_checkout 留存的 Page，入口处直接用 `_require_live_browser()`
|
||||
拒绝，不走重建。
|
||||
|
||||
`browser_status()` 把这份状态暴露给交易服务 /health，容器 HEALTHCHECK 据此
|
||||
在进程还活着但浏览器已死时触发重启。
|
||||
|
||||
调用方：
|
||||
- worker runner：传入 LeaseTask，调 add_to_cart(task) / verify_cart(task)
|
||||
- HTTP 路由 /api/cart/*：调 add_to_cart_payload(...) / cart_status() /
|
||||
@@ -654,28 +670,56 @@ class SiteInteractor:
|
||||
|
||||
# ---- 生命周期 ----
|
||||
|
||||
def _state_path(self) -> "Path":
|
||||
"""rakuten 的 storage_state 文件路径"""
|
||||
return self._settings.auth_state_path / auth_site.profile("rakuten").state_filename
|
||||
|
||||
@staticmethod
|
||||
def _context_options(storage_state: str | None) -> dict:
|
||||
"""new_context 的参数。启动与掉线重建必须用同一套，指纹漂移会触发风控"""
|
||||
return {
|
||||
"storage_state": storage_state,
|
||||
"user_agent": auth_site.RAKUTEN_USER_AGENT,
|
||||
"locale": "ja-JP",
|
||||
"timezone_id": "Asia/Tokyo",
|
||||
"viewport": {"width": 390, "height": 844},
|
||||
"is_mobile": True,
|
||||
"has_touch": True,
|
||||
}
|
||||
|
||||
async def start(self) -> None:
|
||||
"""启动 Playwright 与带 cookie 的浏览器 context
|
||||
|
||||
登录态文件不存在时同样启动（context 没 cookie），后续 add_to_cart 会
|
||||
在 require_logged_in 里报错。这样保持启动路径一致。
|
||||
"""
|
||||
await self._launch()
|
||||
logger.info("SiteInteractor 已就绪：storage_state=%s", self._state_path())
|
||||
|
||||
async def _launch(self) -> None:
|
||||
"""真正拉起 playwright + browser + context。start() 与掉线重建共用
|
||||
|
||||
抽出来的唯一目的就是让 `_ensure_browser_alive()` 能原样重放一遍启动流程——
|
||||
重建出来的浏览器必须和启动时**完全一致**（同样的 UA / viewport / proxy /
|
||||
launch args），否则 Akamai 那边指纹一变就是一次风控事件。
|
||||
"""
|
||||
from playwright.async_api import async_playwright
|
||||
|
||||
state_path = self._settings.auth_state_path / auth_site.profile("rakuten").state_filename
|
||||
state_path = self._state_path()
|
||||
storage_state = str(state_path) if state_path.exists() else None
|
||||
if storage_state is None:
|
||||
logger.warning(
|
||||
"登录态文件不存在：site_interactor 以无 cookie 状态启动，"
|
||||
"加购请求会被站点拒认"
|
||||
)
|
||||
self._state_mtime = None
|
||||
else:
|
||||
self._state_mtime = state_path.stat().st_mtime
|
||||
|
||||
self._playwright = await async_playwright().start()
|
||||
self._browser = await self._playwright.chromium.launch(
|
||||
# 2026-08-14 实测确认：headless=True 会让站点的购物车/结算 SPA 表现
|
||||
# 异常（购入手続き点了不跳转、shopUrlList 渲染不出来），换 headless=False
|
||||
# 异常（購入手続き点了不跳转、shopUrlList 渲染不出来），换 headless=False
|
||||
# 后行为与真实下单一致（能正常触发 session upgrade）——所有会改动站点
|
||||
# 状态的操作（加购/结算/支付）都必须走非无头浏览器
|
||||
headless=False,
|
||||
@@ -683,26 +727,150 @@ class SiteInteractor:
|
||||
proxy=playwright_launch_proxy(self._settings),
|
||||
args=["--no-first-run", "--disable-blink-features=AutomationControlled"],
|
||||
)
|
||||
self._context = await self._browser.new_context(
|
||||
storage_state=storage_state,
|
||||
user_agent=auth_site.RAKUTEN_USER_AGENT,
|
||||
locale="ja-JP",
|
||||
timezone_id="Asia/Tokyo",
|
||||
viewport={"width": 390, "height": 844},
|
||||
is_mobile=True,
|
||||
has_touch=True,
|
||||
)
|
||||
logger.info("SiteInteractor 已就绪：storage_state=%s", storage_state or "(none)")
|
||||
self._context = await self._browser.new_context(**self._context_options(storage_state))
|
||||
|
||||
async def _refresh_context_if_stale(self) -> None:
|
||||
"""检查 storage_state 文件 mtime，变化则重建 context
|
||||
# ---- 掉线探活与重建 ----
|
||||
|
||||
AuthSession.try_relogin 成功后会重写 storage_state 文件。本 context 启动时
|
||||
用快照式 storage_state 创建，cookie 不会自动同步——必须关掉旧 context、
|
||||
用新文件重建。在 add_to_cart / verify_cart 开头各调一次，开销可接受
|
||||
（只在 mtime 变了才重建）。
|
||||
@property
|
||||
def browser_alive(self) -> bool:
|
||||
"""浏览器进程当前是否还连着
|
||||
|
||||
供 /health 与 `_ensure_browser_alive()` 判断。`is_connected()` 是同步调用，
|
||||
不打站点、不进事件循环，高频探活没有代价。
|
||||
"""
|
||||
state_path = self._settings.auth_state_path / auth_site.profile("rakuten").state_filename
|
||||
browser = self._browser
|
||||
if browser is None:
|
||||
return False
|
||||
try:
|
||||
return bool(browser.is_connected())
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
def browser_status(self) -> dict:
|
||||
"""浏览器状态快照，供交易服务 /health 暴露
|
||||
|
||||
`started` 与 `alive` 要分开看：`started=False` 是「还没跑 start()」（正常的
|
||||
启动窗口期），`started=True, alive=False` 才是「浏览器死了」——只有后者该
|
||||
让健康检查转 degraded。
|
||||
"""
|
||||
started = self._browser is not None
|
||||
alive = self.browser_alive
|
||||
if not started:
|
||||
detail = "尚未启动"
|
||||
elif alive:
|
||||
detail = "连接正常"
|
||||
else:
|
||||
detail = "浏览器已掉线（Chromium 崩溃 / 被 kill），需要重建或重启进程"
|
||||
return {
|
||||
"started": started,
|
||||
"alive": alive,
|
||||
"pending_checkout_tasks": sorted(self._checkout_pages),
|
||||
"detail": detail,
|
||||
}
|
||||
|
||||
async def _ensure_browser_alive(self) -> None:
|
||||
"""任务边界上的探活：浏览器没了就地重建一套
|
||||
|
||||
**只在任务边界重建**（各公开方法入口，动作还没发出去的时候）。中途掉线不
|
||||
走这里——那时站点侧生不生效已经无从判断，只能抛 BrowserDeadError 交人工，
|
||||
见 `_new_page()` 与 `_require_live_browser()`。
|
||||
|
||||
重建前会把 `_checkout_pages` 里残留的确认页丢掉：那些 Page 已经随浏览器
|
||||
一起没了，留着只会让后续 submit_order / pay 拿到一个必然报错的死对象。
|
||||
丢弃前记一条 warning——留了确认页说明上一单卡在「已进确认页、可能已提交」
|
||||
的中间态，这条日志是人工核对的线索。
|
||||
|
||||
Raises:
|
||||
BrowserDeadError: 重建失败（Chromium 起不来），下一次调用会再试一次
|
||||
"""
|
||||
if self.browser_alive:
|
||||
return
|
||||
|
||||
orphaned = sorted(self._checkout_pages)
|
||||
if orphaned:
|
||||
logger.warning(
|
||||
"浏览器掉线时仍留有未收尾的下单确认页，随浏览器一起丢弃，"
|
||||
"这些任务的站点侧状态需人工核对：task_ids=%s",
|
||||
orphaned,
|
||||
)
|
||||
self._checkout_pages.clear()
|
||||
|
||||
logger.warning("检测到浏览器已掉线，就地重建 Playwright / 浏览器 / context")
|
||||
await self._teardown()
|
||||
try:
|
||||
await self._launch()
|
||||
except Exception as exc:
|
||||
# 重建失败时把半成品收干净，让下一次调用从 _browser=None 重新试
|
||||
await self._teardown()
|
||||
raise BrowserDeadError(
|
||||
f"浏览器掉线后重建失败：{type(exc).__name__}: {exc}"
|
||||
) from exc
|
||||
logger.info("浏览器已重建完成")
|
||||
|
||||
def _require_live_browser(self, where: str) -> None:
|
||||
"""确认浏览器还活着，否则抛 BrowserDeadError（**不重建**）
|
||||
|
||||
给 submit_order / pay 这种「复用 enter_checkout 留存的 Page」的步骤用：
|
||||
重建一套新浏览器救不回那份服务端订单草稿，而且这两步正好卡在提交前后，
|
||||
站点侧生效与否不能猜——直接抛出去让 runner 转 needs_human。
|
||||
"""
|
||||
if self.browser_alive:
|
||||
return
|
||||
raise BrowserDeadError(
|
||||
f"{where}：浏览器已掉线，enter_checkout 留存的下单确认页已随之失效，"
|
||||
"本单站点侧是否已提交无法判断，需人工核对订单列表（不要按失败重试）"
|
||||
)
|
||||
|
||||
async def _new_page(self) -> "Page":
|
||||
"""开一个新页面，掉线时抛 BrowserDeadError 而不是裸的 TargetClosedError
|
||||
|
||||
各方法里 `new_page()` 都写在 try 之外（页面还没拿到，没有要 close 的东西），
|
||||
裸异常会穿过 runner 的 `except AppError` 一路飘到主循环，任务一次都不上报。
|
||||
统一走这个壳把它包成 AppError。
|
||||
"""
|
||||
try:
|
||||
return await self._context.new_page()
|
||||
except Exception as exc:
|
||||
raise BrowserDeadError(
|
||||
f"打开新页面失败，浏览器可能已掉线：{type(exc).__name__}: {exc}"
|
||||
) from exc
|
||||
|
||||
async def _request(self, method: str, url: str, **kwargs):
|
||||
"""走 context.request 发请求（共享 cookie），掉线时转成 BrowserDeadError
|
||||
|
||||
与 `_new_page()` 同样的理由：裸的 TargetClosedError 会绕过 runner 的
|
||||
`except AppError`。区别是这里的请求本来就可能因为站点/网络原因失败，那些
|
||||
是正常的业务失败——所以**只在确认浏览器真的没了时**才改写异常，其余原样
|
||||
抛出，交给各调用点既有的判据处理。
|
||||
"""
|
||||
try:
|
||||
return await getattr(self._context.request, method)(url, **kwargs)
|
||||
except Exception as exc:
|
||||
if self.browser_alive:
|
||||
raise
|
||||
raise BrowserDeadError(
|
||||
f"{method.upper()} {url} 失败且浏览器已掉线：{type(exc).__name__}: {exc}"
|
||||
) from exc
|
||||
|
||||
async def _ensure_context_ready(self) -> None:
|
||||
"""每个站点动作前的统一前置：浏览器还活着 + context 拿的是最新 cookie
|
||||
|
||||
两件事按顺序做，顺序不能换——mtime 重建要用 `self._browser`，浏览器已经
|
||||
死了的话得先重建出来：
|
||||
|
||||
1. `_ensure_browser_alive()`：掉线就地重建（任务边界）
|
||||
2. storage_state mtime 变了就重建 context：AuthSession.try_relogin 成功后会
|
||||
重写该文件，而本 context 启动时拿的是快照式 storage_state，cookie 不会
|
||||
自动同步，必须关掉旧 context 用新文件重建
|
||||
|
||||
只在 mtime 真的变了才重建，每个动作前调一次的开销可接受。
|
||||
|
||||
Raises:
|
||||
BrowserDeadError: 浏览器掉线且重建失败
|
||||
"""
|
||||
await self._ensure_browser_alive()
|
||||
|
||||
state_path = self._state_path()
|
||||
if not state_path.exists():
|
||||
return
|
||||
mtime = state_path.stat().st_mtime
|
||||
@@ -718,15 +886,14 @@ class SiteInteractor:
|
||||
await self._context.close()
|
||||
except Exception:
|
||||
logger.debug("关闭旧 context 失败", exc_info=True)
|
||||
self._context = await self._browser.new_context(
|
||||
storage_state=str(state_path),
|
||||
user_agent=auth_site.RAKUTEN_USER_AGENT,
|
||||
locale="ja-JP",
|
||||
timezone_id="Asia/Tokyo",
|
||||
viewport={"width": 390, "height": 844},
|
||||
is_mobile=True,
|
||||
has_touch=True,
|
||||
)
|
||||
try:
|
||||
self._context = await self._browser.new_context(
|
||||
**self._context_options(str(state_path))
|
||||
)
|
||||
except Exception as exc:
|
||||
raise BrowserDeadError(
|
||||
f"重建 context 失败，浏览器可能已掉线：{type(exc).__name__}: {exc}"
|
||||
) from exc
|
||||
self._state_mtime = mtime
|
||||
|
||||
async def _read_with_relogin_retry(
|
||||
@@ -751,7 +918,7 @@ class SiteInteractor:
|
||||
await self._auth_session.require_logged_in("rakuten")
|
||||
# 重登会重写 storage_state，本 context 的 cookie 是启动时的快照，
|
||||
# 必须在每次 attempt 前都刷一遍，否则重登后仍拿旧 cookie 去读。
|
||||
await self._refresh_context_if_stale()
|
||||
await self._ensure_context_ready()
|
||||
try:
|
||||
return await read()
|
||||
except _LoggedOutMidRead as exc:
|
||||
@@ -778,6 +945,16 @@ class SiteInteractor:
|
||||
"""关闭 context、browser、playwright，吞掉单个 close 异常"""
|
||||
for task_id in list(self._checkout_pages):
|
||||
await self._discard_checkout_page(task_id)
|
||||
await self._teardown()
|
||||
|
||||
async def _teardown(self) -> None:
|
||||
"""释放 context / browser / playwright 三件套并置空
|
||||
|
||||
与 close() 分开是因为掉线重建（`_ensure_browser_alive`）也要走一遍：那条
|
||||
路径不能碰 `_checkout_pages`（那些 Page 已经死了，`page.close()` 只会再抛
|
||||
一次异常），它自己负责丢弃。每个 close 独立 try——已经死掉的浏览器上
|
||||
close 本来就会抛，不能因此漏掉后面两个资源的释放。
|
||||
"""
|
||||
for resource, name in (
|
||||
(self._context, "context"),
|
||||
(self._browser, "browser"),
|
||||
@@ -880,7 +1057,7 @@ class SiteInteractor:
|
||||
response_html / screenshot 供 worker 入口包成 PageSnapshot 落证据。
|
||||
"""
|
||||
await self._auth_session.require_logged_in("rakuten")
|
||||
await self._refresh_context_if_stale()
|
||||
await self._ensure_context_ready()
|
||||
|
||||
intent_for_extract: dict = {}
|
||||
if variant_id is not None:
|
||||
@@ -888,7 +1065,7 @@ class SiteInteractor:
|
||||
if choice is not None:
|
||||
intent_for_extract["choice"] = choice
|
||||
|
||||
page = await self._context.new_page()
|
||||
page = await self._new_page()
|
||||
try:
|
||||
try:
|
||||
await page.goto(item_url, wait_until="domcontentloaded", timeout=30_000)
|
||||
@@ -932,7 +1109,8 @@ class SiteInteractor:
|
||||
"加购请求：basket=%s payload=%s", fields["basket_domain"], payload,
|
||||
)
|
||||
|
||||
resp = await self._context.request.post(
|
||||
resp = await self._request(
|
||||
"post",
|
||||
fields["basket_domain"],
|
||||
form=payload,
|
||||
max_redirects=5,
|
||||
@@ -1000,7 +1178,7 @@ class SiteInteractor:
|
||||
"""
|
||||
async with self._lock:
|
||||
await self._auth_session.require_logged_in("rakuten")
|
||||
await self._refresh_context_if_stale()
|
||||
await self._ensure_context_ready()
|
||||
|
||||
per_task = self._per_task_state.get(task.task_id, {})
|
||||
item_id = (task.intent or {}).get("item_id") or per_task.get("item_id")
|
||||
@@ -1028,7 +1206,7 @@ class SiteInteractor:
|
||||
"""
|
||||
async with self._lock:
|
||||
await self._auth_session.require_logged_in("rakuten")
|
||||
await self._refresh_context_if_stale()
|
||||
await self._ensure_context_ready()
|
||||
raw_status, count = await self._query_cart_count()
|
||||
return {
|
||||
"logged_in": True,
|
||||
@@ -1057,9 +1235,9 @@ class SiteInteractor:
|
||||
"""
|
||||
async with self._lock:
|
||||
await self._auth_session.require_logged_in("rakuten")
|
||||
await self._refresh_context_if_stale()
|
||||
await self._ensure_context_ready()
|
||||
|
||||
page = await self._context.new_page()
|
||||
page = await self._new_page()
|
||||
removed = 0
|
||||
html = ""
|
||||
screenshot = b""
|
||||
@@ -1148,9 +1326,9 @@ class SiteInteractor:
|
||||
raise InvalidRequestError("item_id 必填")
|
||||
async with self._lock:
|
||||
await self._auth_session.require_logged_in("rakuten")
|
||||
await self._refresh_context_if_stale()
|
||||
await self._ensure_context_ready()
|
||||
|
||||
page = await self._context.new_page()
|
||||
page = await self._new_page()
|
||||
try:
|
||||
await page.goto(_CART_PAGE, wait_until="domcontentloaded", timeout=30_000)
|
||||
await self._wait_cart_rendered(page, label=f"remove_item {item_id}")
|
||||
@@ -1212,7 +1390,8 @@ class SiteInteractor:
|
||||
才是「获取失败」，抛 CartOperationError——常见原因是 Referer 错或
|
||||
cookie 失效
|
||||
"""
|
||||
resp = await self._context.request.get(
|
||||
resp = await self._request(
|
||||
"get",
|
||||
_CART_COUNT_API + "?sid=1010",
|
||||
headers={"Referer": _CART_PAGE},
|
||||
)
|
||||
@@ -1236,7 +1415,7 @@ class SiteInteractor:
|
||||
|
||||
返回渲染后的 cart 页 HTML + 整页截图（校验通过时），供调用方落证据。
|
||||
"""
|
||||
page = await self._context.new_page()
|
||||
page = await self._new_page()
|
||||
try:
|
||||
await page.goto(_CART_PAGE, wait_until="domcontentloaded", timeout=30_000)
|
||||
await self._wait_cart_rendered(page, label=label)
|
||||
@@ -1367,9 +1546,9 @@ class SiteInteractor:
|
||||
"""
|
||||
async with self._lock:
|
||||
await self._auth_session.require_logged_in("rakuten")
|
||||
await self._refresh_context_if_stale()
|
||||
await self._ensure_context_ready()
|
||||
|
||||
page = await self._context.new_page()
|
||||
page = await self._new_page()
|
||||
success = False
|
||||
try:
|
||||
await page.goto(_CART_PAGE, wait_until="domcontentloaded", timeout=30_000)
|
||||
@@ -1856,6 +2035,10 @@ class SiteInteractor:
|
||||
供人工核对，详见 _discard_checkout_page 的调用取舍）
|
||||
"""
|
||||
async with self._lock:
|
||||
# 掉线检查必须在取 page 之前：浏览器没了的话 _checkout_pages 里那个
|
||||
# Page 是个死对象（不是 None），下面的 selector 轮询会全部超时，最后
|
||||
# 报「未找到确认按钮」——把「浏览器崩了」误诊成「站点改版了」
|
||||
self._require_live_browser("submit_order")
|
||||
page = self._checkout_pages.get(task.task_id)
|
||||
if page is None:
|
||||
raise OrderOperationError(
|
||||
@@ -1952,6 +2135,10 @@ class SiteInteractor:
|
||||
整页截图，供 runner step 5 落证据。
|
||||
"""
|
||||
async with self._lock:
|
||||
# 这一步最危险：submit_order 已经真的提交过订单了。浏览器在这时候没了，
|
||||
# 「付款到底完成没有」完全无从判断，必须当场抛出去转人工，不能让裸的
|
||||
# TargetClosedError 飘到主循环变成「一次都不上报」。
|
||||
self._require_live_browser("pay")
|
||||
page = self._checkout_pages.pop(task.task_id, None)
|
||||
if page is None:
|
||||
raise OrderOperationError(
|
||||
@@ -1987,7 +2174,12 @@ class SiteInteractor:
|
||||
logger.warning("pay 页面截图失败（不影响付款判定）", exc_info=True)
|
||||
return PageSnapshot(html=html, screenshot=screenshot)
|
||||
finally:
|
||||
await page.close()
|
||||
# 关页面失败不能掩盖 try 里的原始异常——浏览器要是在本方法执行途中
|
||||
# 没的，close() 自己也会抛，那条异常没有任何诊断价值
|
||||
try:
|
||||
await page.close()
|
||||
except Exception:
|
||||
logger.debug("pay 关闭确认页失败", exc_info=True)
|
||||
|
||||
async def check_order_status(self, site_order_id: str) -> OrderStatusSnapshot:
|
||||
"""付款后监控的单次探测：查一次订单详情页的配送阶段，不循环
|
||||
@@ -2034,7 +2226,7 @@ class SiteInteractor:
|
||||
url = _ORDER_DETAIL_URL_TEMPLATE.format(order_number=site_order_id, shop_id=shop_id)
|
||||
|
||||
async def read() -> OrderDetailSnapshot:
|
||||
page = await self._context.new_page()
|
||||
page = await self._new_page()
|
||||
try:
|
||||
try:
|
||||
await page.goto(url, wait_until="domcontentloaded", timeout=30_000)
|
||||
@@ -2112,7 +2304,7 @@ class SiteInteractor:
|
||||
acc = _OrderListAccumulator()
|
||||
stop = False
|
||||
|
||||
page = await self._context.new_page()
|
||||
page = await self._new_page()
|
||||
try:
|
||||
for page_num in range(1, page_limit + 1):
|
||||
url = _ORDER_LIST_URL if page_num == 1 else f"{_ORDER_LIST_URL}?page={page_num}"
|
||||
|
||||
+230
-2
@@ -20,6 +20,7 @@ import pytest
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from app.shared.errors import (
|
||||
BrowserDeadError,
|
||||
CartOperationError,
|
||||
InvalidRequestError,
|
||||
NotLoggedInError,
|
||||
@@ -576,10 +577,24 @@ _ORDER_DETAIL_LANDED_URL = "https://order.my.rakuten.co.jp/purchase-history/?ord
|
||||
_ORDER_LIST_LANDED_URL = "https://order.my.rakuten.co.jp/purchase-history/order-list"
|
||||
|
||||
|
||||
class _FakeBrowser:
|
||||
"""Browser 替身：只实现 _ensure_browser_alive 用到的 is_connected()
|
||||
|
||||
没有它，`_ensure_context_ready` 会判定「浏览器没起来」并尝试真的 launch 一次
|
||||
Chromium。connected 可以翻成 False 来模拟掉线。
|
||||
"""
|
||||
|
||||
def __init__(self, connected: bool = True):
|
||||
self.connected = connected
|
||||
|
||||
def is_connected(self) -> bool:
|
||||
return self.connected
|
||||
|
||||
|
||||
def _build_site(tmp_path: Path, context: _FakeContext, auth: _FakeAuthSession) -> SiteInteractor:
|
||||
"""装配一个不依赖 Playwright 的 SiteInteractor
|
||||
|
||||
auth_state_dir 指向空目录：_refresh_context_if_stale 见不到 storage_state
|
||||
auth_state_dir 指向空目录：_ensure_context_ready 见不到 storage_state
|
||||
文件就直接返回，不会试图重建 context（重建需要真实 browser）。
|
||||
"""
|
||||
from app.shared.config import Settings
|
||||
@@ -588,6 +603,7 @@ def _build_site(tmp_path: Path, context: _FakeContext, auth: _FakeAuthSession) -
|
||||
auth_session=auth, # type: ignore[arg-type]
|
||||
settings=Settings(auth_state_dir=str(tmp_path / "auth"), evidence_dir=str(tmp_path)),
|
||||
)
|
||||
site._browser = _FakeBrowser() # type: ignore[assignment]
|
||||
site._context = context # type: ignore[assignment]
|
||||
return site
|
||||
|
||||
@@ -980,13 +996,16 @@ async def test_check_order_status_still_returns_only_status(tmp_path):
|
||||
|
||||
|
||||
async def test_submit_order_without_checkout_page_raises():
|
||||
"""浏览器活着但没有留存确认页 → OrderOperationError（不是掉线那类错误）"""
|
||||
site = SiteInteractor(auth_session=None, settings=None) # type: ignore[arg-type]
|
||||
site._browser = _FakeBrowser() # type: ignore[assignment]
|
||||
with pytest.raises(OrderOperationError):
|
||||
await site.submit_order(_make_task())
|
||||
|
||||
|
||||
async def test_pay_without_checkout_page_raises():
|
||||
site = SiteInteractor(auth_session=None, settings=None) # type: ignore[arg-type]
|
||||
site._browser = _FakeBrowser() # type: ignore[assignment]
|
||||
with pytest.raises(OrderOperationError):
|
||||
await site.pay(_make_task(), "ord-1")
|
||||
|
||||
@@ -1314,15 +1333,17 @@ class _FakeLoggedInAuth:
|
||||
def _build_clear_cart_site(*, delete_buttons: int, count_body: str, html: str) -> SiteInteractor:
|
||||
"""构造 clear_cart 可离线跑起来的 SiteInteractor（fake page + fake count API）"""
|
||||
site = SiteInteractor(auth_session=_FakeLoggedInAuth(), settings=None) # type: ignore[arg-type]
|
||||
site._browser = _FakeBrowser() # type: ignore[assignment]
|
||||
site._context = _FakeClearCartContext(
|
||||
page=_FakeClearCartPage(delete_buttons=delete_buttons, html=html),
|
||||
count_body=count_body,
|
||||
) # type: ignore[assignment]
|
||||
|
||||
# settings=None，走不了真实的 mtime 检查；探活那半段由 _FakeBrowser 覆盖
|
||||
async def _noop_refresh() -> None:
|
||||
return None
|
||||
|
||||
site._refresh_context_if_stale = _noop_refresh # type: ignore[assignment]
|
||||
site._ensure_context_ready = _noop_refresh # type: ignore[assignment]
|
||||
return site
|
||||
|
||||
|
||||
@@ -1390,3 +1411,210 @@ async def test_clear_cart_html_capture_failure_keeps_clear_result():
|
||||
|
||||
|
||||
# ---- helper ----
|
||||
|
||||
|
||||
# ---- 浏览器掉线：任务边界就地重建，中途掉线转 BrowserDeadError ----
|
||||
#
|
||||
# 真实的 Chromium 崩溃（容器 /dev/shm 不足、OOM kill、seccomp 挡 sandbox）没法在
|
||||
# 离线测试里制造，这里用 _FakeBrowser.connected 翻成 False 模拟「is_connected()
|
||||
# 返回 False」这个唯一的可观测信号，覆盖的是 SiteInteractor 对该信号的反应。
|
||||
|
||||
|
||||
def _build_dead_browser_site(tmp_path: Path) -> SiteInteractor:
|
||||
"""浏览器已掉线的 SiteInteractor：_browser 在位但 is_connected() 为 False"""
|
||||
from app.shared.config import Settings
|
||||
|
||||
site = SiteInteractor(
|
||||
auth_session=_FakeLoggedInAuth(), # type: ignore[arg-type]
|
||||
settings=Settings(auth_state_dir=str(tmp_path / "auth"), evidence_dir=str(tmp_path)),
|
||||
)
|
||||
site._browser = _FakeBrowser(connected=False) # type: ignore[assignment]
|
||||
return site
|
||||
|
||||
|
||||
def test_browser_alive_reflects_is_connected(tmp_path):
|
||||
"""browser_alive：没起过 / 已掉线 / 正常，三种情况分别为 False/False/True"""
|
||||
site = SiteInteractor(auth_session=None, settings=None) # type: ignore[arg-type]
|
||||
assert site.browser_alive is False # 还没 start()
|
||||
|
||||
site._browser = _FakeBrowser(connected=False) # type: ignore[assignment]
|
||||
assert site.browser_alive is False
|
||||
|
||||
site._browser = _FakeBrowser(connected=True) # type: ignore[assignment]
|
||||
assert site.browser_alive is True
|
||||
|
||||
|
||||
def test_browser_alive_survives_is_connected_raising():
|
||||
"""is_connected() 自己抛错（驱动已经没了）也要当掉线处理，不能把异常漏出去"""
|
||||
|
||||
class _ExplodingBrowser:
|
||||
def is_connected(self) -> bool:
|
||||
raise RuntimeError("driver connection closed")
|
||||
|
||||
site = SiteInteractor(auth_session=None, settings=None) # type: ignore[arg-type]
|
||||
site._browser = _ExplodingBrowser() # type: ignore[assignment]
|
||||
|
||||
assert site.browser_alive is False
|
||||
|
||||
|
||||
def test_browser_status_distinguishes_not_started_from_dead():
|
||||
"""/health 要能分清「还没启动」和「起过但死了」——只有后者算 degraded"""
|
||||
site = SiteInteractor(auth_session=None, settings=None) # type: ignore[arg-type]
|
||||
|
||||
not_started = site.browser_status()
|
||||
assert not_started["started"] is False
|
||||
assert not_started["alive"] is False
|
||||
|
||||
site._browser = _FakeBrowser(connected=False) # type: ignore[assignment]
|
||||
dead = site.browser_status()
|
||||
assert dead["started"] is True
|
||||
assert dead["alive"] is False
|
||||
|
||||
site._browser = _FakeBrowser(connected=True) # type: ignore[assignment]
|
||||
assert site.browser_status() == {
|
||||
"started": True,
|
||||
"alive": True,
|
||||
"pending_checkout_tasks": [],
|
||||
"detail": "连接正常",
|
||||
}
|
||||
|
||||
|
||||
async def test_ensure_browser_alive_relaunches_on_task_boundary(tmp_path):
|
||||
"""任务边界探到掉线 → 重放一遍启动流程，调用方无感知"""
|
||||
site = _build_dead_browser_site(tmp_path)
|
||||
relaunched = []
|
||||
|
||||
async def _fake_launch() -> None:
|
||||
relaunched.append(True)
|
||||
site._browser = _FakeBrowser(connected=True) # type: ignore[assignment]
|
||||
|
||||
site._launch = _fake_launch # type: ignore[assignment]
|
||||
|
||||
await site._ensure_browser_alive()
|
||||
|
||||
assert relaunched == [True]
|
||||
assert site.browser_alive is True
|
||||
|
||||
|
||||
async def test_ensure_browser_alive_is_noop_when_connected(tmp_path):
|
||||
"""浏览器好好的就不能重建——重建一次要几秒，还会丢掉当前 context 的 cookie"""
|
||||
site = _build_dead_browser_site(tmp_path)
|
||||
site._browser = _FakeBrowser(connected=True) # type: ignore[assignment]
|
||||
|
||||
async def _fail_launch() -> None:
|
||||
raise AssertionError("浏览器连接正常时不应重建")
|
||||
|
||||
site._launch = _fail_launch # type: ignore[assignment]
|
||||
|
||||
await site._ensure_browser_alive()
|
||||
|
||||
|
||||
async def test_ensure_browser_alive_drops_orphaned_checkout_pages(tmp_path):
|
||||
"""重建前丢掉残留的确认页：那些 Page 已随浏览器一起没了，留着必然报错"""
|
||||
site = _build_dead_browser_site(tmp_path)
|
||||
site._checkout_pages["t-stuck"] = object() # type: ignore[assignment]
|
||||
|
||||
async def _fake_launch() -> None:
|
||||
site._browser = _FakeBrowser(connected=True) # type: ignore[assignment]
|
||||
|
||||
site._launch = _fake_launch # type: ignore[assignment]
|
||||
|
||||
await site._ensure_browser_alive()
|
||||
|
||||
assert site._checkout_pages == {}
|
||||
|
||||
|
||||
async def test_ensure_browser_alive_raises_browser_dead_when_relaunch_fails(tmp_path):
|
||||
"""重建也起不来 → BrowserDeadError（AppError，runner 接得住），并把半成品收干净"""
|
||||
site = _build_dead_browser_site(tmp_path)
|
||||
|
||||
async def _broken_launch() -> None:
|
||||
raise RuntimeError("chromium 起不来")
|
||||
|
||||
site._launch = _broken_launch # type: ignore[assignment]
|
||||
|
||||
with pytest.raises(BrowserDeadError):
|
||||
await site._ensure_browser_alive()
|
||||
|
||||
assert site._browser is None # 已 teardown，下次调用从头再试
|
||||
|
||||
|
||||
async def test_new_page_wraps_target_closed_into_app_error():
|
||||
"""中途掉线的裸 TargetClosedError 必须被包成 AppError
|
||||
|
||||
这是整条链最关键的一环：不包的话异常会穿过 runner 的 except AppError，
|
||||
落到主循环那个只记日志的兜底里——任务一次都不上报，网关要干等租约过期。
|
||||
"""
|
||||
|
||||
class _DeadContext:
|
||||
async def new_page(self):
|
||||
raise RuntimeError("Target page, context or browser has been closed")
|
||||
|
||||
site = SiteInteractor(auth_session=None, settings=None) # type: ignore[arg-type]
|
||||
site._context = _DeadContext() # type: ignore[assignment]
|
||||
|
||||
with pytest.raises(BrowserDeadError) as excinfo:
|
||||
await site._new_page()
|
||||
|
||||
from app.shared.errors import AppError
|
||||
|
||||
assert isinstance(excinfo.value, AppError)
|
||||
assert excinfo.value.err_code == 5006
|
||||
|
||||
|
||||
async def test_request_keeps_original_error_when_browser_is_alive():
|
||||
"""浏览器活着时的请求失败是正常业务失败，不能被误标成掉线"""
|
||||
|
||||
class _FlakyRequest:
|
||||
async def get(self, url: str, **kwargs):
|
||||
raise RuntimeError("站点 502")
|
||||
|
||||
class _Context:
|
||||
request = _FlakyRequest()
|
||||
|
||||
site = SiteInteractor(auth_session=None, settings=None) # type: ignore[arg-type]
|
||||
site._browser = _FakeBrowser(connected=True) # type: ignore[assignment]
|
||||
site._context = _Context() # type: ignore[assignment]
|
||||
|
||||
with pytest.raises(RuntimeError, match="站点 502"):
|
||||
await site._request("get", "https://example.test/")
|
||||
|
||||
|
||||
async def test_request_converts_to_browser_dead_when_browser_is_gone():
|
||||
"""同样的请求失败，浏览器确实没了时才改写成 BrowserDeadError"""
|
||||
|
||||
class _FlakyRequest:
|
||||
async def get(self, url: str, **kwargs):
|
||||
raise RuntimeError("Target closed")
|
||||
|
||||
class _Context:
|
||||
request = _FlakyRequest()
|
||||
|
||||
site = SiteInteractor(auth_session=None, settings=None) # type: ignore[arg-type]
|
||||
site._browser = _FakeBrowser(connected=False) # type: ignore[assignment]
|
||||
site._context = _Context() # type: ignore[assignment]
|
||||
|
||||
with pytest.raises(BrowserDeadError):
|
||||
await site._request("get", "https://example.test/")
|
||||
|
||||
|
||||
async def test_submit_order_reports_browser_dead_before_blaming_selectors(tmp_path):
|
||||
"""浏览器死时 submit_order 要报掉线，而不是「找不到确认按钮」
|
||||
|
||||
留存的 Page 是个死对象（不是 None），不先探活的话 selector 轮询会全部超时，
|
||||
最后把「浏览器崩了」误诊成「站点改版了」——这两个结论的处置方式完全不同。
|
||||
"""
|
||||
site = _build_dead_browser_site(tmp_path)
|
||||
site._checkout_pages["t1"] = object() # type: ignore[assignment]
|
||||
|
||||
with pytest.raises(BrowserDeadError):
|
||||
await site.submit_order(_make_task(task_id="t1"))
|
||||
|
||||
|
||||
async def test_pay_reports_browser_dead(tmp_path):
|
||||
"""pay 之前 submit_order 已经真的下过单了，掉线必须当场抛出去交人工"""
|
||||
site = _build_dead_browser_site(tmp_path)
|
||||
site._checkout_pages["t1"] = object() # type: ignore[assignment]
|
||||
|
||||
with pytest.raises(BrowserDeadError):
|
||||
await site.pay(_make_task(task_id="t1"), "ord-1")
|
||||
|
||||
@@ -85,6 +85,7 @@ class StubSiteInteractor:
|
||||
self.clear_calls = 0
|
||||
self.remove_calls: list[str] = []
|
||||
self.fail_with: Exception | None = None
|
||||
self.browser_alive = True
|
||||
|
||||
async def add_to_cart_payload(
|
||||
self,
|
||||
@@ -129,6 +130,15 @@ class StubSiteInteractor:
|
||||
raise self.fail_with
|
||||
return {"removed": True, "item_id": item_id}
|
||||
|
||||
def browser_status(self) -> dict:
|
||||
"""/health 读的浏览器状态；browser_alive 可翻成 False 模拟掉线"""
|
||||
return {
|
||||
"started": True,
|
||||
"alive": self.browser_alive,
|
||||
"pending_checkout_tasks": [],
|
||||
"detail": "连接正常" if self.browser_alive else "浏览器已掉线",
|
||||
}
|
||||
|
||||
async def close(self) -> None:
|
||||
"""lifespan 收尾会调用；桩没有真实浏览器要关"""
|
||||
|
||||
@@ -168,6 +178,26 @@ def test_health_needs_no_token(client):
|
||||
body = response.json()
|
||||
assert body["data"]["status"] == "ok"
|
||||
assert set(body["data"]["auth"]) == {"rakuten"}
|
||||
assert body["data"]["browser"]["alive"] is True
|
||||
|
||||
|
||||
def test_health_returns_503_when_browser_is_dead(client, stub_site):
|
||||
"""浏览器掉线 → 503，让容器 HEALTHCHECK 探到并触发重启
|
||||
|
||||
进程还活着、端口还通，但这个服务的所有站点操作都要靠那一个浏览器，它没了以后
|
||||
每一单都会失败。返回 200 的话 HEALTHCHECK 永远绿灯，缺口就一直挂在那儿。
|
||||
重启是安全的：网关侧任务绝不自动重投（docs/order-gateway.md §5）。
|
||||
"""
|
||||
stub_site.browser_alive = False
|
||||
|
||||
response = client.get("/health")
|
||||
|
||||
assert response.status_code == 503
|
||||
body = response.json()
|
||||
assert body["success"] is False
|
||||
assert body["code"] == 5006 # 与 BrowserDeadError 同码
|
||||
assert body["data"]["status"] == "degraded"
|
||||
assert body["data"]["browser"]["alive"] is False
|
||||
|
||||
|
||||
def test_health_does_not_probe_the_site(client, stub):
|
||||
|
||||
@@ -20,7 +20,7 @@ from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from app.shared.errors import CheckoutBlockedError, OrderGuardError
|
||||
from app.shared.errors import BrowserDeadError, CheckoutBlockedError, OrderGuardError
|
||||
from app.shared.task_state import OrderState, TaskStatus
|
||||
from app.trading.worker import verify
|
||||
from app.trading.worker.evidence import EvidenceStore
|
||||
@@ -352,6 +352,41 @@ async def test_checkout_blocked_becomes_needs_human(
|
||||
assert "风控" in terminal["detail"]
|
||||
|
||||
|
||||
# ---- 浏览器掉线 → needs_human（不是 failed）----
|
||||
|
||||
|
||||
async def test_browser_dead_becomes_needs_human(runner: WorkerRunner):
|
||||
"""执行途中浏览器掉线 → BrowserDeadError → 转 needs_human
|
||||
|
||||
两件事同时被这个用例守着：
|
||||
|
||||
1. BrowserDeadError 是 AppError 的子类，能被 _execute_with_renewal 接住并**上报**。
|
||||
不是 AppError 的话它会一路飘到主循环的兜底日志里，任务一次都不上报，网关侧
|
||||
要干等整个 lease_ttl 才被 sweep 置 stale。
|
||||
2. 结论是 needs_human 而不是 failed——`except BrowserDeadError` 必须排在
|
||||
`except AppError` 前面，顺序反了这个断言就会挂。浏览器是在动作中途没的，
|
||||
站点侧生效与否无从判断，不能给上游一个「明确失败」的结论。
|
||||
"""
|
||||
|
||||
async def _noop(task): # noqa: ANN001
|
||||
return None
|
||||
|
||||
async def _browser_died(task): # noqa: ANN001
|
||||
raise BrowserDeadError("pay：浏览器已掉线，本单是否已提交无法判断，需人工核对订单列表")
|
||||
|
||||
runner._site.add_to_cart = _noop # type: ignore[assignment]
|
||||
runner._site.verify_cart = _noop # type: ignore[assignment]
|
||||
runner._site.enter_checkout = _browser_died # type: ignore[assignment]
|
||||
_set_site_clear(runner._site, {"removed_count": 0, "cart_count": 0})
|
||||
|
||||
await runner.handle(_make_task(task_id="t1"))
|
||||
|
||||
gateway: FakeGateway = runner._gateway_for_test # type: ignore[attr-defined]
|
||||
terminal = gateway.last_terminal_report()
|
||||
assert terminal["terminal_status"] == TaskStatus.NEEDS_HUMAN
|
||||
assert "掉线" in terminal["detail"]
|
||||
|
||||
|
||||
# ---- ラクマ 不在交易范围 → needs_human ----
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user