feat(observability): 抓取去掉首页预热,交易补齐链路埋点

两个问题一起处理,都与「出站请求与可观测性」有关。

## 抓取:正常路径不再多打一次首页

site_session 原先每条通道每 30 分钟打一次 www.rakuten.co.jp/ 做预热,而且预热
返回非 2xx 时 warmed_at 不置位——那种情况下每个请求前都会再打一次首页。

Akamai 的 cookie 随任意页面响应下发,目标页自己就会带回来,专门先打一次首页除了
多一个出站请求(以及多一次被风控计数的机会)之外没有额外收益:首个请求无论打哪个
URL 都是冷的 ~11s,之后都复用 cookie。

改为 cookie 由目标页响应建立(_note_cookies)、超 TTL 主动清空
(_drop_expired_cookies)。首页只保留在失败修复路径上(_rewarm_on_home):目标页
已经吃了挑战页时,拿首页换一套干净 cookie 比继续撞同一个 URL 更安全。happy path
的出站请求数 2 → 1。

_note_cookies 刻意不在每次响应时刷新时刻:TTL 要从「这套 cookie 第一次出现」算起,
每次都刷新会让一套 cookie 被无限续命,反而绕过了 session_ttl_seconds 的本意。

profile_status() 的 warmed 字段名保留(上游健康检查看板在用),语义改为「当前有
可复用的 Akamai cookie」,不再代表「已专门预热过首页」。

## 交易:此前没有任何有意义的链路数据

根因是 trading 的实际工作两类自动埋点都覆盖不到:站点交互走 Playwright(不经
httpx),worker 主循环是后台 asyncio 任务(没有 HTTP 入口,因此没有根 span)。
于是发给网关的每次 httpx 调用各自成为孤立 trace——观测后台上只剩一堆请求记录。

新增手工埋点:

- order.task:一笔下单的根 span,一个 task_id 一条 trace,带 order.route
  (execute / recovery / already_finished)与终态 order.terminal_status
- order.step.*:清车 → 加购 → 校验 → 确认 → 提交 → 付款,每步一个子 span,
  带 order.evidence_ref,可从 span 直接定位落盘证据
- site.*:12 个 Playwright 交互方法(用 traced 装饰器而非 with 块——这些方法的
  函数体本就很长,再加一层缩进不利于阅读)
- account_query:只读查询单的根 span,带 query.outcome

空转的长轮询(30 秒一次、绝大多数返回空)用 suppressed() 屏蔽:量大且没有信息量,
把观测后台刷满的正是它们。领到任务后的网关调用都在任务根 span 底下,不受影响。

闸门 / 风控拦截会被 _execute_with_renewal 吞掉转 needs_human,异常冒不到根 span,
被拦下的单在 trace 里跟成功下单一模一样。加 _execute_recording_errors 一层统一
记录,比每个 except 分支各写一遍省事,也不会漏掉后续新增的分支。

_report_safe 写 span 属性前判断 is_recording():付款后监控是 create_task 起的,
asyncio 在创建时就把 context 复制了进去,等它真正跑起来根 span 早已结束——
get_current_span() 拿到的仍是那个已结束的 span(不是 INVALID_SPAN),写属性会打
"Setting attribute on ended span"。当前监控路径不传 terminal_status 走不到那里,
这道判断是防以后。

## 顺带修掉:instrument_app 从未生效

instrument_app 用 _provider is None 做前置判断,但三个服务都在模块导入时执行
app = create_app(),而 setup_telemetry 要等 lifespan 才跑——那时 _provider 还是
None,照着判断直接 return。**FastAPI 从来没被打桩过,三个服务一条 server span
都没有。**

实测确认两件事:导入期打桩能出 span,lifespan 内打桩出不来(instrument_app 是加
中间件,应用开始服务后加进去不生效);provider 后设也不影响 ProxyTracer 委托到
真实 provider。所以只能在导入期装,判断条件改为 otel_enabled。

app/gateway/main.py 此前完全没接 telemetry,worker 出站请求带过来的 traceparent
没人接上,一条下单链路在网关这里断掉,只看得到 worker 侧那半截。补上
setup_telemetry(service_name="rakuten-gateway") 与 instrument_app / shutdown。

## 验证

新增 8 个用例:首页零请求、cookie 复用与过期清空、失败后用首页换 cookie、一任务
一 trace 的父子结构、闸门失败标 ERROR、空转不埋点,以及 instrument_app 调用顺序
的回归测试。全量 526 passed。

Playwright 那些 site.* 埋点只做了静态验证(测试用桩替换站点方法),没有跑真实
浏览器下单确认 span 真的落地。

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-08-28 14:56:49 +08:00
co-authored by Claude Opus 5
parent bba45d7f7c
commit 3c7618a1d6
12 changed files with 727 additions and 113 deletions
+2 -1
View File
@@ -41,7 +41,8 @@ RAKUTEN_REQUEST_TIMEOUT_SECONDS=30
RAKUTEN_MAX_SITE_CONCURRENCY=8
# 单次抓取的最大尝试次数(含首次):1 次直发,2 次换 cookie 重试,3 次起动用浏览器兜底
RAKUTEN_HTTP_MAX_ATTEMPTS=3
# Akamai cookie 最长复用时长(秒),超时后重新预热
# Akamai cookie 最长复用时长(秒),超时后清空、由下一次请求的响应重新建立。
# cookie 是随目标页响应下发的,正常路径不会额外访问站点首页。
RAKUTEN_SESSION_TTL_SECONDS=1800
# 浏览器兜底:纯 HTTP 被反爬拦截时用 Playwright 取回 cookie 再回灌重试。
+27 -2
View File
@@ -93,8 +93,8 @@ PC UA 在搜索页、详情页、店铺页上都能拿到完整模板。因此
- FastAPI 提供 HTTP API
- Bearer Token 接口鉴权(`Authorization: Bearer <token>`
- Akamai cookie 自动预热与复用,过期自动重新预热(仅乐天链路)
- 失败逐级升级:重新预热 → Playwright 兜底取 cookie → 结构化报错
- Akamai cookie 随目标页响应自动建立与复用,过期自动清空重建(仅乐天链路;正常路径不额外访问首页
- 失败逐级升级:首页换一套 cookie → Playwright 兜底取 cookie → 结构化报错
- 搜索结果自动剔除混入的 CPC 广告位(乐天)
- 识别站点深翻页「静默回绕到第 1 页」的行为,避免上游重复入库(乐天)
- 商品详情按落地域名分派解析,覆盖楽天ブックス / Rakuten Fashion / ビックカメラ 等官方旗舰子站
@@ -660,6 +660,30 @@ trading 加购时的字段选择策略:多规格挑第一个非售罄的 varia
(账号只读查询通道)**可以重试**——查询只读,重发没有副作用
(见 [docs/order-gateway.md §11](docs/order-gateway.md#11-账号只读查询通道))。
## 链路追踪
可选,默认关闭。`RAKUTEN_OTEL_ENABLED=true` + `RAKUTEN_OTEL_ENDPOINT` 指向 OTLP/HTTP
端点后,三个服务分别以 `rakuten-scraping` / `rakuten-trading` / `rakuten-gateway` 上报。
FastAPI 与 httpx 走自动埋点,但那只覆盖「收到 HTTP 请求」与「发出 httpx 请求」两类
边界。交易侧的实际工作两者都不是——站点交互走 Playwright,worker 主循环是后台任务,
所以这一侧手工埋点:
- `order.task`:一笔下单任务的**根 span**(一个 `task_id` 一条 trace),带
`order.route``execute` / `recovery` / `already_finished`)与终态
`order.terminal_status`;被闸门或风控拦下时置 ERROR 并带 `error.code`
- `order.step.*`:清车 → 加购 → 校验 → 确认页 → 提交 → 付款,每步一个子 span,
`order.evidence_ref`(可直接定位落盘证据)
- `site.*`:Playwright 站点交互(`site.add_to_cart``site.enter_checkout`
`site.submit_order``site.pay` 等)
- `account_query`:一张只读查询单的根 span,带 `query.outcome`
worker 空转的长轮询(每 30 秒问一次网关有没有活干)刻意不埋点——它们没有信息量,
量却极大,会把观测后台刷满。
> 解析失败时页面 HTML 会作为 span event 上报(`RAKUTEN_OTEL_SNAPSHOT_MAX_BYTES`
> 控制上限,默认 2MB),用于事后复现「抓到的内容为什么解析不出预期字段」。
## 常用环境变量
完整列表见 [.env.example](.env.example)。配置项前缀统一为 `RAKUTEN_`,三个服务共用同一份
@@ -675,6 +699,7 @@ trading 加购时的字段选择策略:多规格挑第一个非售罄的 varia
- 抓取:`RAKUTEN_MAX_SITE_CONCURRENCY`(默认 `8`,两站各自独立计数)、`RAKUTEN_HTTP_MAX_ATTEMPTS`(默认 `3`)、`RAKUTEN_SESSION_TTL_SECONDS`(默认 `1800`,仅乐天)
- 浏览器兜底(仅乐天):`RAKUTEN_BROWSER_FALLBACK_ENABLED``RAKUTEN_BROWSER_HEADLESS``RAKUTEN_BROWSER_CHANNEL`
- 代理(需日本 IP 时):`RAKUTEN_PROXY_SERVER``RAKUTEN_PROXY_USERNAME``RAKUTEN_PROXY_PASSWORD``RAKUTEN_PROXY_BYPASS`
- 链路追踪(可选,默认关闭;三个服务共用):`RAKUTEN_OTEL_ENABLED``RAKUTEN_OTEL_ENDPOINT``RAKUTEN_OTEL_HEADERS``RAKUTEN_OTEL_SNAPSHOT_MAX_BYTES`
> 从中国大陆直连实测可用、无需代理;`RAKUTEN_PROXY_SERVER` 留空即可。设置后,所有面向
> 外部站点的 HTTPX 与 Playwright 流量都会经代理;本机和 Docker 服务间地址由
+8
View File
@@ -27,6 +27,7 @@ from app.gateway.task_queue import TaskQueue
from app.shared.api import register_exception_handlers
from app.shared.config import get_settings
from app.shared.logging_setup import configure_logging
from app.shared.telemetry import instrument_app, setup_telemetry, shutdown_telemetry
logger = logging.getLogger(__name__)
@@ -96,6 +97,11 @@ async def lifespan(app: FastAPI):
app.state.container = container
configure_logging(container.settings)
# 网关这一侧原先完全没接 telemetry,导致 worker 出站请求带过来的 traceparent
# 没人接上:一条下单链路在网关这里断掉,观测后台上只看得到 worker 侧那半截。
# CallbackNotifier 的 httpx 客户端是每次投递时才建的(不是 __init__ 里),
# 所以放在 build_container() 之后仍然赶在客户端创建之前。
setup_telemetry(container.settings, service_name="rakuten-gateway")
logger.info(
"网关启动:%s:%s", container.settings.gateway_host, container.settings.gateway_port
)
@@ -141,6 +147,7 @@ async def lifespan(app: FastAPI):
# 等在途回调发完(各次发送有超时兜底),避免关停时静默丢通知
await container.notifier.aclose()
await container.db.close()
shutdown_telemetry()
def create_app() -> FastAPI:
@@ -151,6 +158,7 @@ def create_app() -> FastAPI:
app.include_router(queries_router)
app.include_router(account_router)
register_exception_handlers(app)
instrument_app(app)
return app
+78 -45
View File
@@ -8,7 +8,13 @@
画像(详情页只有手机 UA 才返回带 __INITIAL_STATE__ 的统一模板)。两条通道
各自持有独立 cookie 罐,避免把 PC 指纹拿到的 cookie 混用到手机请求上。
抓取失败时的升级路径:重新预热 → 浏览器兜底取 cookie → 放弃。
**不预热首页**:Akamai 的 cookie 是随任意一个页面响应下发的,目标页自己就会带回
来,专门先打一次 `www.rakuten.co.jp/` 除了多一个出站请求(以及多一次被风控计数
的机会)之外没有额外收益——首个请求无论打哪个 URL 都是冷的 ~11s,之后都复用
cookie。首页只在**失败修复**路径上使用:目标页已经被挑战时,拿首页换一套干净
cookie 比继续拿目标页去撞更安全(见 `_rewarm_on_home`)。
抓取失败时的升级路径:换 cookie(首页重新预热)→ 浏览器兜底取 cookie → 放弃。
"""
from __future__ import annotations
@@ -62,12 +68,18 @@ class _Profile:
mobile: bool
client: httpx.AsyncClient
lock: asyncio.Lock = field(default_factory=asyncio.Lock)
warmed_at: float = 0.0
# Akamai cookie 罐的建立时刻(monotonic)。0 表示当前没有可复用的 cookie。
# 超过 session_ttl_seconds 就主动清空:拿着过期 cookie 去撞反而更容易被挑战。
cookies_at: float = 0.0
@property
def cookie_names(self) -> set[str]:
return {cookie.name for cookie in self.client.cookies.jar}
@property
def akamai_cookies(self) -> set[str]:
return self.cookie_names & set(site.AKAMAI_COOKIE_NAMES)
class SiteSession:
"""乐天站点抓取会话,管理 cookie 预热、并发限流与失败升级"""
@@ -115,13 +127,17 @@ class SiteSession:
# ---- 状态 ----
def profile_status(self) -> dict[str, dict[str, Any]]:
"""各通道的预热状态,供健康检查展示"""
"""各通道的 cookie 状态,供健康检查展示
`warmed` 保留原字段名(上游健康检查看板在用),语义是「当前有可复用的
Akamai cookie」——不再代表「已专门预热过首页」,因为正常路径不打首页了。
"""
now = time.monotonic()
return {
name: {
"warmed": profile.warmed_at > 0,
"age_seconds": round(now - profile.warmed_at, 1) if profile.warmed_at else None,
"cookies": sorted(profile.cookie_names & set(site.AKAMAI_COOKIE_NAMES)),
"warmed": profile.cookies_at > 0,
"age_seconds": round(now - profile.cookies_at, 1) if profile.cookies_at else None,
"cookies": sorted(profile.akamai_cookies),
}
for name, profile in self._profiles.items()
}
@@ -142,7 +158,8 @@ class SiteSession:
) -> FetchedPage:
"""抓取页面,返回 HTML 与最终落地地址
失败时按 重新预热 → 浏览器兜底 的顺序逐级升级重试。
直接打目标页(不先访问首页),失败时按 换 cookie → 浏览器兜底 的顺序
逐级升级重试。
Args:
validator: 页面校验器,默认要求页面含 __INITIAL_STATE__。跨站抓取时
@@ -181,7 +198,8 @@ class SiteSession:
try:
for attempt in range(1, max_attempts + 1):
span.set_attribute("scrape.attempts", attempt)
await self._ensure_warm(profile)
# 过期 cookie 主动丢掉:带着它去撞比裸请求更容易吃挑战页
await self._drop_expired_cookies(profile)
try:
response = await profile.client.get(url)
except httpx.HTTPError as exc:
@@ -192,6 +210,10 @@ class SiteSession:
)
continue
# 任何响应都可能带 set-cookie(Akamai 不保证每次下发),拿到就
# 记下时刻,后续请求复用到 TTL 为止。
self._note_cookies(profile)
if response.status_code == 404:
err = ItemNotFoundError(f"Page not found: {url}")
span.record_exception(err)
@@ -223,9 +245,11 @@ class SiteSession:
if attempt >= max_attempts:
break
# 第一次失败先便宜地换一套 cookie;仍失败才动用浏览器
# 第一次失败先便宜地换一套 cookie(丢掉旧的,用首页换新的);
# 仍失败才动用浏览器
if attempt == 1:
await self._invalidate(profile)
await self._rewarm_on_home(profile)
span.set_attribute("scrape.rewarmed_on_home", True)
else:
page = await self._escalate_to_browser(profile, url, validate)
if page is not None:
@@ -265,44 +289,53 @@ class SiteSession:
def _is_server_error(reason: str) -> bool:
return reason.startswith("upstream status") or reason.startswith("httpx") or "Error:" in reason
async def _ensure_warm(self, profile: _Profile) -> None:
"""确保通道有一次新鲜的首页预热;过期时重新访问首页"""
if self._is_warm(profile):
def _note_cookies(self, profile: _Profile) -> None:
"""目标页响应带回 Akamai cookie 时记下时刻,作为 TTL 起点
已经在计时的不重置:TTL 要从「这套 cookie 第一次出现」算起,每次响应都
刷新会让一套 cookie 被无限续命,反而绕过了 session_ttl_seconds 的本意。
"""
if profile.cookies_at:
return
if profile.akamai_cookies:
profile.cookies_at = time.monotonic()
async with profile.lock:
if self._is_warm(profile):
return
try:
response = await profile.client.get(self._settings.home_url)
# Akamai 不保证每次都下发 cookie;首页探测成功本身就是可复用的
# 预热结果,cookie 只用于观测和失败升级时的回灌。
if response.status_code < 400:
profile.warmed_at = time.monotonic()
logger.info(
"会话预热完成:profile=%s status=%s cookies=%s",
profile.name,
response.status_code,
sorted(profile.cookie_names & set(site.AKAMAI_COOKIE_NAMES)),
)
except httpx.HTTPError as exc:
# 预热失败不阻断本次抓取:直连目标页仍可能成功,只是慢
logger.warning("会话预热失败:profile=%s err=%s", profile.name, exc)
profile.warmed_at = 0.0
def _is_warm(self, profile: _Profile) -> bool:
if not profile.warmed_at:
return False
if time.monotonic() - profile.warmed_at > self._settings.session_ttl_seconds:
return False
return True
async def _invalidate(self, profile: _Profile) -> None:
"""清空通道 cookie 并强制下次重新预热"""
async def _drop_expired_cookies(self, profile: _Profile) -> None:
"""cookie 罐超过 TTL 时清空,让下一次请求裸奔换一套新的"""
if not profile.cookies_at:
return
if time.monotonic() - profile.cookies_at <= self._settings.session_ttl_seconds:
return
async with profile.lock:
profile.client.cookies.clear()
profile.warmed_at = 0.0
logger.info("已清空会话 cookie,将重新预热:profile=%s", profile.name)
profile.cookies_at = 0.0
logger.info("会话 cookie 已过期,已清空:profile=%s", profile.name)
async def _rewarm_on_home(self, profile: _Profile) -> None:
"""首次失败后的修复:丢掉旧 cookie,用首页换一套新的
这是首页 URL 唯一的用途。目标页已经吃了挑战页,继续拿同一个 URL 去撞
只会把挑战坐实;首页是站点最"无害"的入口,换 cookie 的成功率更高。
失败不阻断本次抓取(下一次 attempt 会裸请求目标页,只是慢),所以这里
只记日志。
"""
async with profile.lock:
profile.client.cookies.clear()
profile.cookies_at = 0.0
try:
response = await profile.client.get(self._settings.home_url)
except httpx.HTTPError as exc:
logger.warning("首页换 cookie 失败:profile=%s err=%s", profile.name, exc)
return
if profile.akamai_cookies:
profile.cookies_at = time.monotonic()
logger.info(
"已用首页换一套新 cookie:profile=%s status=%s cookies=%s",
profile.name,
response.status_code,
sorted(profile.akamai_cookies),
)
async def _escalate_to_browser(
self, profile: _Profile, url: str, validate: PageValidator
@@ -332,7 +365,7 @@ class SiteSession:
domain=cookie.get("domain") or "",
path=cookie.get("path") or "/",
)
profile.warmed_at = time.monotonic()
profile.cookies_at = time.monotonic()
# 浏览器不回报最终 URL,这里以请求地址为准;跨站跳转场景下 HTTP 通道已先行
# 报错,走不到这一步。
+3 -1
View File
@@ -73,7 +73,9 @@ class Settings(BaseSettings):
request_timeout_seconds: float = 30.0
max_site_concurrency: int = 8 # 对站点的最大并发请求数
http_max_attempts: int = 3 # 单次抓取的最大尝试次数(含首次)
session_ttl_seconds: float = 1800.0 # Akamai cookie 会话最长复用时长超时后重新预热
# Akamai cookie 最长复用时长超时后清空 cookie 罐,由下一次响应重新建立。
# cookie 随目标页响应下发,正常路径不额外访问站点首页(见 site_session.py)。
session_ttl_seconds: float = 1800.0
# ---- 浏览器兜底配置(仅抓取服务)----
# 纯 HTTP 被 Akamai 拦截时,用 Playwright 打开页面取回 cookie 再回灌给
+106 -5
View File
@@ -11,28 +11,45 @@ instrumentation(FastAPI、httpx)。失败时(如 endpoint 不可达)不
OTel SDK 默认的 ProxyTracerProvider 在 setup 之前就能用(noop span),所以
其它代码里直接 `trace.get_tracer(__name__)` + `start_as_current_span` 即可,
不必关心 telemetry 是否启用——禁用时 span 不会真正产生与上报。
自动 instrumentation(FastAPI + httpx)只覆盖「进程收到 HTTP 请求」与「进程发出
httpx 请求」两类边界。交易侧的实际工作两者都不是:站点交互走 Playwright(不经
httpx),worker 主循环是后台 asyncio 任务(没有 HTTP 入口)。所以那一侧必须手工
埋点,否则 trace 里只剩 worker 与网关之间的往返记录,看不到任何业务链路。本模块
为此提供三件东西:
- `traced`:给 async 方法套一层 span,异常自动记录(站点交互各步骤在用)
- `set_attributes` / `record_error`:批量写属性、统一记异常
- `suppressed`:屏蔽空转长轮询产生的孤立 trace
"""
from __future__ import annotations
import functools
import logging
from typing import TYPE_CHECKING
from collections.abc import Awaitable, Callable, Iterator, Mapping
from contextlib import contextmanager
from typing import TYPE_CHECKING, ParamSpec, TypeVar
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor
from opentelemetry.instrumentation.utils import suppress_instrumentation
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.trace.sampling import ALWAYS_ON
from opentelemetry.trace import Span
from opentelemetry.trace import Span, SpanKind, Status, StatusCode
from opentelemetry.util.types import AttributeValue
from app.shared.config import Settings
from app.shared.config import Settings, get_settings
if TYPE_CHECKING:
from fastapi import FastAPI
P = ParamSpec("P")
R = TypeVar("R")
logger = logging.getLogger(__name__)
# 全局 provider 引用,用于 instrument_app / shutdown 时判断当前是否已初始化。
@@ -88,8 +105,24 @@ def setup_telemetry(settings: Settings, *, service_name: str) -> None:
def instrument_app(app: "FastAPI") -> None:
"""FastAPI 应用打桩;未初始化时 noop,调用顺序无要求。"""
if _provider is None:
"""FastAPI 应用打桩。必须在应用开始服务之前调用,与 setup_telemetry 的先后无关。
**不能用 `_provider is None` 做前置判断**:三个服务都在模块导入时执行
`app = create_app()`,而 `setup_telemetry` 要等 lifespan 启动才跑,那时
`_provider` 还是 None——照着判断就会直接 return,FastAPI 永远没被打桩,
观测后台里一条 server span 都不会有。
反过来「等 lifespan 里再打桩」也不行:instrument_app 是往应用上加中间件,
应用一旦开始服务,加进去的中间件不生效(实测 lifespan 内调用后 server span
为空)。所以只能在这里、在导入期就装上。
provider 尚未设置时拿到的是 ProxyTracer,它在 `set_tracer_provider` 之后会
自动委托到真实 provider(实测:导入期打桩 + lifespan 内设 provider,请求
照样产生 span),所以顺序不构成问题。
otel 关闭时跳过:省掉一层用不上的中间件。
"""
if not get_settings().otel_enabled:
return
FastAPIInstrumentor.instrument_app(app)
@@ -127,6 +160,74 @@ def snapshot(span: Span, name: str, html: str | None, max_bytes: int) -> None:
span.add_event(name, attributes=attributes)
@contextmanager
def suppressed() -> Iterator[None]:
"""在这个上下文里不产生任何自动 instrumentation span。
给「空转的长轮询」用:worker 每 30 秒问一次网关有没有活干,绝大多数时候
返回空。这些请求各自成为一条孤立 trace,量大且没有信息量——把观测后台刷满
的正是它们。领到任务后的每一次网关调用都在任务根 span 底下,不受影响。
"""
with suppress_instrumentation():
yield
def record_error(span: Span, exc: BaseException) -> None:
"""把异常记到 span 上并置 ERROR 状态。
单独抽出来是因为 AppError 带 `err_code`(对外错误码),排查时按码筛比按
异常类名筛更贴近上游看到的东西,值得单独落一个属性。
"""
span.record_exception(exc)
span.set_attribute("error.type", type(exc).__name__)
err_code = getattr(exc, "err_code", None)
if isinstance(err_code, int):
span.set_attribute("error.code", err_code)
span.set_status(Status(StatusCode.ERROR, f"{type(exc).__name__}: {exc}"))
def traced(
name: str,
*,
kind: SpanKind = SpanKind.INTERNAL,
) -> Callable[[Callable[P, Awaitable[R]]], Callable[P, Awaitable[R]]]:
"""给 async 方法套一层 span,异常自动记录后原样抛出。
交易侧的实际工作是 Playwright 页面操作,httpx 自动 instrumentation 完全看不到
(浏览器请求不走 httpx),所以这些步骤必须手工埋点,否则 trace 里只剩 worker
与网关之间的 HTTP 往返。用装饰器而不是在每个方法里写 with 块,是因为这些方法
的函数体都已经很长,再加一层缩进不利于阅读。
未启用 telemetry 时 tracer 是 noop,装饰器只多一次函数调用,可以无条件套。
"""
def decorate(fn: Callable[P, Awaitable[R]]) -> Callable[P, Awaitable[R]]:
@functools.wraps(fn)
async def wrapper(*args: P.args, **kwargs: P.kwargs) -> R:
tracer = trace.get_tracer(fn.__module__)
with tracer.start_as_current_span(name, kind=kind) as span:
try:
return await fn(*args, **kwargs)
except Exception as exc:
record_error(span, exc)
raise
return wrapper
return decorate
def set_attributes(span: Span, attributes: Mapping[str, AttributeValue | None]) -> None:
"""批量设置属性,跳过 None 值。
站点交互里大量字段是可选的(site_order_id 要到提交后才有、payable_yen 只在
确认页解析后才有),逐个 if 判断会把埋点代码写得比业务逻辑还长。
"""
for key, value in attributes.items():
if value is not None:
span.set_attribute(key, value)
def _parse_headers(raw: str | None) -> list[tuple[str, str]] | None:
"""解析 "k1=v1,k2=v2" 形式的 header 配置;空输入返回 None。"""
if not raw or not raw.strip():
+39 -3
View File
@@ -21,8 +21,12 @@ import logging
from datetime import datetime, timezone
from typing import TYPE_CHECKING, Any
from opentelemetry import trace
from opentelemetry.trace import SpanKind
from app.shared.errors import AppError, InvalidRequestError
from app.shared.task_state import AccountQueryKind
from app.shared.telemetry import record_error, set_attributes, suppressed
from app.trading.worker.client import GatewayClient
from app.trading.worker.models import QueryTask
from app.trading.worker.site_interact import SiteInteractor
@@ -32,6 +36,8 @@ if TYPE_CHECKING:
logger = logging.getLogger(__name__)
tracer = trace.get_tracer(__name__)
# 没给 since 时的窗口下界。用 epoch 而不是 None,是为了让翻页终止条件
# (`order_date < since`)与「不设下界」共用同一条代码路径,不多一个分支。
_EPOCH = datetime(1970, 1, 1, tzinfo=timezone.utc)
@@ -100,7 +106,9 @@ class QueryRunner:
logger.info("账号查询 worker 启动:worker_id=%s", self.worker_id)
while self._running:
try:
query = await self._gateway.lease_query(self.worker_id, wait=30)
# 与下单主循环同样:空转的长轮询不埋点,见 runner.py::run
with suppressed():
query = await self._gateway.lease_query(self.worker_id, wait=30)
except AppError as exc:
logger.warning("查询 lease 失败:%s (err=%s)", exc.message, exc.err_code)
await asyncio.sleep(5)
@@ -122,7 +130,27 @@ class QueryRunner:
# ---- 单张查询单 ----
async def handle(self, query: QueryTask) -> None:
"""执行一张查询单并回报结果。任何失败都转成一次「失败回报」,不抛出去"""
"""执行一张查询单并回报结果。任何失败都转成一次「失败回报」,不抛出去
这里开的 span 是一张查询单的根:底下挂着站点读取(`site.*`)与回报网关的
HTTP 调用,一个 query_id 对应一条 trace。查询失败不抛出去(都转成失败
回报),所以失败信息由各分支显式记到 span 上——否则 trace 里会显示成功。
"""
with tracer.start_as_current_span("account_query", kind=SpanKind.CONSUMER) as span:
set_attributes(
span,
{
"query.query_id": query.query_id,
"query.kind": query.kind,
"query.site": query.site,
"query.attempt": query.attempt,
"query.worker_id": self.worker_id,
},
)
await self._handle_traced(query, span)
async def _handle_traced(self, query: QueryTask, span: "trace.Span") -> None:
"""handle() 的实际执行体,拆出来只为让根 span 的 with 块保持一层缩进"""
logger.info(
"领到查询单:query_id=%s kind=%s attempt=%s",
query.query_id, query.kind, query.attempt,
@@ -132,11 +160,13 @@ class QueryRunner:
self.execute(query),
timeout=self._settings.account_query_timeout_seconds,
)
except asyncio.TimeoutError:
except asyncio.TimeoutError as exc:
logger.warning(
"查询超时(%s 秒,多半是下单任务正占着账号锁):query_id=%s",
self._settings.account_query_timeout_seconds, query.query_id,
)
span.set_attribute("query.outcome", "timeout")
record_error(span, exc)
await self._report_safe(
query,
success=False,
@@ -152,12 +182,16 @@ class QueryRunner:
"查询失败:query_id=%s code=%s msg=%s",
query.query_id, exc.err_code, exc.message,
)
span.set_attribute("query.outcome", "failed")
record_error(span, exc)
await self._report_safe(
query, success=False, error_code=exc.err_code, error_message=exc.message
)
return
except Exception as exc: # noqa: BLE001
logger.exception("查询未预期异常:query_id=%s", query.query_id)
span.set_attribute("query.outcome", "unexpected_error")
record_error(span, exc)
await self._report_safe(
query,
success=False,
@@ -168,10 +202,12 @@ class QueryRunner:
payload, oversized = self._enforce_result_size(result)
if oversized is not None:
span.set_attribute("query.outcome", "oversized")
await self._report_safe(
query, success=False, error_code=1003, error_message=oversized
)
return
span.set_attribute("query.outcome", "succeeded")
await self._report_safe(query, success=True, result=payload)
async def execute(self, query: QueryTask) -> dict[str, Any]:
+134 -35
View File
@@ -28,6 +28,9 @@ import contextlib
import logging
from typing import TYPE_CHECKING
from opentelemetry import trace
from opentelemetry.trace import SpanKind, Status, StatusCode
from app.shared.errors import (
AppError,
BrowserDeadError,
@@ -35,6 +38,7 @@ from app.shared.errors import (
OrderGuardError,
)
from app.shared.task_state import OrderState, TaskStatus
from app.shared.telemetry import record_error, set_attributes, suppressed
from app.trading.worker import verify
from app.trading.worker.client import GatewayClient
from app.trading.worker.evidence import EvidenceStore
@@ -47,6 +51,8 @@ if TYPE_CHECKING:
logger = logging.getLogger(__name__)
tracer = trace.get_tracer(__name__)
def _coerce_state(value: str | None) -> OrderState:
"""把网关返回的 state 字符串安全地包成 OrderState;None 或非法值回落到 CREATED"""
@@ -115,7 +121,11 @@ class WorkerRunner:
logger.info("worker 启动:worker_id=%s", self.worker_id)
while self._running:
try:
task = await self._gateway.lease(self.worker_id, wait=30)
# 空转的长轮询不埋点:30 秒一次、绝大多数返回空,每次都会变成一条
# 孤立 trace 把观测后台刷满。领到任务后的调用都在 handle() 的根
# span 底下,不受这里影响。
with suppressed():
task = await self._gateway.lease(self.worker_id, wait=30)
except AppError as exc:
logger.warning("lease 失败:%s (err=%s)", exc.message, exc.err_code)
await asyncio.sleep(5)
@@ -137,13 +147,42 @@ class WorkerRunner:
# ---- 单任务调度 ----
async def handle(self, task: LeaseTask) -> None:
"""单任务调度入口:本地幂等闸门 → 恢复核对 → 执行"""
"""单任务调度入口:本地幂等闸门 → 恢复核对 → 执行
这里开的 span 是**整条下单链路的根**:往下的每一步站点交互、每一次回报
网关都挂在它底下,一个 task_id 对应观测后台里的一条 trace。worker 是后台
asyncio 任务,没有 HTTP 入口,不开这个根 span 的话下游 httpx 调用会各自
散成孤立 trace(这正是「只有请求记录、没有链路」的原因)。
"""
with tracer.start_as_current_span("order.task", kind=SpanKind.CONSUMER) as span:
intent = task.intent or {}
set_attributes(
span,
{
"order.task_id": task.task_id,
"order.site": task.site,
"order.worker_id": self.worker_id,
"order.lease_count": task.lease_count,
"order.known_state": task.known_state,
"order.item_url": intent.get("item_url"),
"order.quantity": intent.get("quantity"),
},
)
try:
await self._dispatch(task, span)
except Exception as exc:
record_error(span, exc)
raise
async def _dispatch(self, task: LeaseTask, span: "trace.Span") -> None:
"""handle() 的实际分支逻辑,拆出来只为让根 span 的 with 块保持一层缩进"""
# 本地幂等闸门:之前已完成过的任务不再执行
if await self._db.has_finished(task.task_id):
final_state = await self._db.final_state(task.task_id)
logger.info(
"本地已完成,补报终态:task_id=%s state=%s", task.task_id, final_state
)
span.set_attribute("order.route", "already_finished")
await self._report_safe(
task,
state=_coerce_state(final_state),
@@ -154,10 +193,12 @@ class WorkerRunner:
# 恢复领取:lease_count > 1 表示 stale → reclaim,必须先核对站点订单
if task.lease_count > 1:
span.set_attribute("order.route", "recovery")
await self._handle_recovery(task)
return
# 常规执行
span.set_attribute("order.route", "execute")
await self._execute_with_renewal(task)
async def _handle_recovery(self, task: LeaseTask) -> None:
@@ -197,11 +238,24 @@ class WorkerRunner:
# ---- 常规执行 ----
async def _execute_recording_errors(self, task: LeaseTask) -> None:
"""execute() 外面加一层:异常先记到任务根 span 上,再原样抛出
下面那一串 except 分支会把异常**吞掉**转成 needs_human / failed 回报,
异常冒不到 handle() 的根 span。在这里统一记一次,比每个分支各写一遍省事,
也不会漏掉新增的分支。
"""
try:
await self.execute(task)
except Exception as exc:
record_error(trace.get_current_span(), exc)
raise
async def _execute_with_renewal(self, task: LeaseTask) -> None:
"""在租约自动续期的上下文里执行任务"""
async with self._renew_lease_every(task, interval=60):
try:
await self.execute(task)
await self._execute_recording_errors(task)
except NotImplementedError as exc:
# 站点交互未实现(规格 §10):上报 needs_human,不视为 worker 失败
logger.warning(
@@ -521,38 +575,65 @@ class WorkerRunner:
证据载体)时,其 html / screenshot 即本步骤要落盘的页面与整页截图;
显式传入的 `html` / `png` 优先级更高,供 step3/step4 等已经单独拿到
页面的调用点使用。
每步一个 span,挂在 handle() 的任务根 span 底下:一条 trace 就是一单的
完整流水(清车 → 加购 → 校验 → 确认 → 提交 → 付款),卡在哪一步、每步
耗时多少、证据落在哪个 evidence_ref 都能直接读出来。
"""
result = await action()
if isinstance(result, PageSnapshot):
if html is None:
html = result.html or None
if png is None:
png = result.screenshot or None
elif html is None and isinstance(result, str):
# 旧契约兼容:站点方法返回字符串时视为页面 HTML
html = result
meta = {
"step": step_name,
"state": state.value,
**(evidence_meta or {}),
}
evidence_ref = self._evidence.write_step(
task.task_id, step_no, step_name, html=html, png=png, meta=meta
)
await self._db.index_evidence(task.task_id, step_no, step_name, evidence_ref)
await self._db.record_event(
task.task_id, state.value, detail=detail, evidence_ref=evidence_ref
)
await self._gateway.report(
task.task_id,
self.worker_id,
state=state,
payable_yen=payable_yen,
pay_deadline=pay_deadline,
site_order_id=site_order_id,
evidence_ref=evidence_ref,
detail=detail,
)
with tracer.start_as_current_span(f"order.step.{step_name}") as span:
set_attributes(
span,
{
"order.task_id": task.task_id,
"order.step_no": step_no,
"order.step_name": step_name,
"order.state": state.value,
},
)
try:
result = await action()
except Exception as exc:
record_error(span, exc)
raise
if isinstance(result, PageSnapshot):
if html is None:
html = result.html or None
if png is None:
png = result.screenshot or None
elif html is None and isinstance(result, str):
# 旧契约兼容:站点方法返回字符串时视为页面 HTML
html = result
meta = {
"step": step_name,
"state": state.value,
**(evidence_meta or {}),
}
evidence_ref = self._evidence.write_step(
task.task_id, step_no, step_name, html=html, png=png, meta=meta
)
set_attributes(
span,
{
"order.evidence_ref": evidence_ref,
"order.site_order_id": site_order_id,
"order.payable_yen": payable_yen,
},
)
await self._db.index_evidence(task.task_id, step_no, step_name, evidence_ref)
await self._db.record_event(
task.task_id, state.value, detail=detail, evidence_ref=evidence_ref
)
await self._gateway.report(
task.task_id,
self.worker_id,
state=state,
payable_yen=payable_yen,
pay_deadline=pay_deadline,
site_order_id=site_order_id,
evidence_ref=evidence_ref,
detail=detail,
)
async def _report_safe(
self,
@@ -566,7 +647,25 @@ class WorkerRunner:
payable_yen: int | None = None,
pay_deadline: str | None = None,
) -> None:
"""回报 gateway,失败只记日志不抛——主循环不能因为回报失败退出"""
"""回报 gateway,失败只记日志不抛——主循环不能因为回报失败退出
顺带把终态标到当前 span 上。这一步是必要的:`_execute_with_renewal` 会把
闸门拦截、风控拦截、浏览器掉线等异常**吞掉**转成 needs_human 回报,异常
不会冒到 handle() 的根 span,于是一笔被拦下的单在 trace 里看起来跟成功
下单一模一样。所有这些分支都汇到这个方法,标在这里最省事也最不容易漏。
`is_recording()` 那道判断不是多余的:付款后监控(_monitor_order)是
`create_task` 起的后台任务,而 asyncio 在创建时就把当时的 context 复制了
进去——等它真正跑起来,任务根 span 早已结束,但 `get_current_span()` 拿到
的仍是那个**已结束**的 span(不是 INVALID_SPAN)。往上写属性会打
"Setting attribute on ended span" 警告。当前监控路径不传 terminal_status
走不到这里,但这道判断保证以后传了也不会污染已完成的任务 span。
"""
span = trace.get_current_span()
if terminal_status is not None and span.is_recording():
span.set_attribute("order.terminal_status", terminal_status.value)
if terminal_status in (TaskStatus.NEEDS_HUMAN, TaskStatus.FAILED):
span.set_status(Status(StatusCode.ERROR, detail))
try:
await self._gateway.report(
task.task_id,
+16
View File
@@ -96,6 +96,8 @@ from dataclasses import dataclass, field
from datetime import datetime
from typing import TYPE_CHECKING, TypeVar
from opentelemetry.trace import SpanKind
from app.shared.errors import (
BrowserDeadError,
CartOperationError,
@@ -113,6 +115,8 @@ from app.shared.purchase_contract import (
)
from app.shared.proxy import playwright_launch_proxy
from app.shared.task_state import OrderState
from app.shared.telemetry import traced
from app.trading.core import auth_site
from app.trading.worker.models import LeaseTask
@@ -973,6 +977,7 @@ class SiteInteractor:
# ---- 已实现:add_to_cart / verify_cart / cart_status ----
@traced("site.add_to_cart", kind=SpanKind.CLIENT)
async def add_to_cart(self, task: LeaseTask) -> PageSnapshot:
"""加购(worker 入口):从 task.intent 取字段,调 _add_to_cart_with_fields
@@ -1015,6 +1020,7 @@ class SiteInteractor:
screenshot=result.get("screenshot") or b"",
)
@traced("site.add_to_cart_payload", kind=SpanKind.CLIENT)
async def add_to_cart_payload(
self,
*,
@@ -1163,6 +1169,7 @@ class SiteInteractor:
"screenshot": screenshot,
}
@traced("site.verify_cart", kind=SpanKind.CLIENT)
async def verify_cart(self, task: LeaseTask) -> PageSnapshot:
"""校验购物车里有没有刚加的商品
@@ -1196,6 +1203,7 @@ class SiteInteractor:
# 2. 渲染 cart 页确认 item_id 在里面
return await self._verify_item_in_cart_html(item_id, label=f"task_id={task.task_id}")
@traced("site.cart_status", kind=SpanKind.CLIENT)
async def cart_status(self) -> dict:
"""轻量查询购物车状态:调 cart count JSONP API,不渲染整页
@@ -1216,6 +1224,7 @@ class SiteInteractor:
# ---- 已实现:clear_cart / remove_item(Playwright UI 点击)----
@traced("site.clear_cart", kind=SpanKind.CLIENT)
async def clear_cart(self) -> dict:
"""清空购物车:渲染 cart SPA → 反复点第一个「削除」按钮 → count API 校验
@@ -1304,6 +1313,7 @@ class SiteInteractor:
"screenshot": screenshot,
}
@traced("site.remove_item", kind=SpanKind.CLIENT)
async def remove_item(self, item_id: str) -> dict:
"""删除购物车里指定 item_id 的商品
@@ -1506,6 +1516,7 @@ class SiteInteractor:
# ---- 已实现:enter_checkout(到下单确认页,中间步骤未经真实 HTML 验证)----
@traced("site.enter_checkout", kind=SpanKind.CLIENT)
async def enter_checkout(self, task: LeaseTask) -> PageSnapshot:
"""进入下单确认页:购物车 → 点「購入手続き」→ 依次处理中间步骤 → 落地确认页
@@ -2017,6 +2028,7 @@ class SiteInteractor:
"""
return _parse_checkout_summary(html)
@traced("site.submit_order", kind=SpanKind.CLIENT)
async def submit_order(self, task: LeaseTask) -> SubmitOutcome:
"""点击下单确认页的最终确认按钮,提交订单
@@ -2118,6 +2130,7 @@ class SiteInteractor:
evidence=PageSnapshot(html=html, screenshot=screenshot),
)
@traced("site.pay", kind=SpanKind.CLIENT)
async def pay(self, task: LeaseTask, site_order_id: str) -> PageSnapshot:
"""检查提交下单后是否已完成付款 / 是否触发了需要人工介入的验证环节
@@ -2181,6 +2194,7 @@ class SiteInteractor:
except Exception:
logger.debug("pay 关闭确认页失败", exc_info=True)
@traced("site.check_order_status", kind=SpanKind.CLIENT)
async def check_order_status(self, site_order_id: str) -> OrderStatusSnapshot:
"""付款后监控的单次探测:查一次订单详情页的配送阶段,不循环
@@ -2198,6 +2212,7 @@ class SiteInteractor:
"""
return (await self.fetch_order_detail(site_order_id)).status
@traced("site.fetch_order_detail", kind=SpanKind.CLIENT)
async def fetch_order_detail(self, site_order_id: str) -> OrderDetailSnapshot:
"""读一次订单详情页:配送阶段 + 页面原始 __INITIAL_STATE__
@@ -2269,6 +2284,7 @@ class SiteInteractor:
f"check_order_status site_order_id={site_order_id}", read
)
@traced("site.list_recent_orders", kind=SpanKind.CLIENT)
async def list_recent_orders(
self, *, since: datetime, max_pages: int | None = None
) -> OrderListWindow:
+73 -19
View File
@@ -67,15 +67,17 @@ async def build_session(handler, *, browser=None, settings=None) -> SiteSession:
return session
async def test_warmup_happens_before_first_fetch_and_is_reused():
async def test_happy_path_never_touches_the_home_page():
"""目标页自己会带回 Akamai cookie,正常路径不该多打一次首页。"""
seen: list[str] = []
def handler(request: httpx.Request) -> httpx.Response:
seen.append(str(request.url))
headers = {}
if request.url.host == "www.rakuten.co.jp":
headers["set-cookie"] = "ak_bmsc=abc; Domain=.rakuten.co.jp; Path=/"
return httpx.Response(200, text=GOOD_PAGE, headers=headers)
return httpx.Response(
200,
text=GOOD_PAGE,
headers={"set-cookie": "ak_bmsc=abc; Domain=.rakuten.co.jp; Path=/"},
)
session = await build_session(handler)
try:
@@ -84,18 +86,20 @@ async def test_warmup_happens_before_first_fetch_and_is_reused():
finally:
await session.close()
# 首页预热只做一次,第二次抓取直接复用 cookie
assert seen.count("https://www.rakuten.co.jp/") == 1
assert seen.count(TARGET) == 2
assert seen == [TARGET, TARGET]
async def test_successful_warmup_is_reused_when_upstream_sets_no_cookie():
"""首页可能返回 200 但不下发 Akamai cookie,仍不得重复预热"""
seen: list[str] = []
async def test_cookies_from_target_page_are_reused_across_fetches():
"""第一次响应下发的 cookie 要带到后续请求上,不必再走首页"""
cookie_headers: list[str | None] = []
def handler(request: httpx.Request) -> httpx.Response:
seen.append(str(request.url))
return httpx.Response(200, text=GOOD_PAGE)
cookie_headers.append(request.headers.get("cookie"))
return httpx.Response(
200,
text=GOOD_PAGE,
headers={"set-cookie": "ak_bmsc=abc; Domain=.rakuten.co.jp; Path=/"},
)
session = await build_session(handler)
try:
@@ -104,8 +108,30 @@ async def test_successful_warmup_is_reused_when_upstream_sets_no_cookie():
finally:
await session.close()
assert seen.count("https://www.rakuten.co.jp/") == 1
assert seen.count(TARGET) == 2
assert cookie_headers[0] is None # 首个请求是冷的
assert "ak_bmsc=abc" in (cookie_headers[1] or "")
async def test_expired_cookies_are_dropped_before_next_fetch():
"""cookie 罐超过 session_ttl_seconds 后要清空,不能带着过期 cookie 去撞。"""
cookie_headers: list[str | None] = []
def handler(request: httpx.Request) -> httpx.Response:
cookie_headers.append(request.headers.get("cookie"))
return httpx.Response(
200,
text=GOOD_PAGE,
headers={"set-cookie": "ak_bmsc=abc; Domain=.rakuten.co.jp; Path=/"},
)
session = await build_session(handler, settings=make_settings(session_ttl_seconds=0.0))
try:
await session.fetch_html(TARGET, mobile=False)
await session.fetch_html(TARGET, mobile=False)
finally:
await session.close()
assert cookie_headers == [None, None] # 每次都从干净状态起步
async def test_missing_state_marker_is_treated_as_blocked():
@@ -139,6 +165,32 @@ async def test_retry_recovers_when_a_later_attempt_succeeds():
assert attempts["n"] == 2
async def test_first_failure_swaps_cookies_via_home_page():
"""首次失败时用首页换一套 cookie——这是首页 URL 唯一的用途。"""
seen: list[str] = []
def handler(request: httpx.Request) -> httpx.Response:
seen.append(str(request.url))
if request.url.host == "www.rakuten.co.jp":
return httpx.Response(
200,
text="home",
headers={"set-cookie": "ak_bmsc=fresh; Domain=.rakuten.co.jp; Path=/"},
)
if seen.count(TARGET) == 1:
return httpx.Response(200, text=BLOCK_PAGE)
return httpx.Response(200, text=GOOD_PAGE)
session = await build_session(handler)
try:
assert await session.fetch_html(TARGET, mobile=False) == GOOD_PAGE
finally:
await session.close()
# 首页只在失败之后出现一次,且排在两次目标页请求中间
assert seen == [TARGET, "https://www.rakuten.co.jp/", TARGET]
async def test_browser_fallback_supplies_cookies_and_page_on_persistent_block():
"""浏览器已经取到页面时应直接采用,不再多打一次 HTTP"""
browser = FakeBrowser(
@@ -264,9 +316,9 @@ async def test_search_and_detail_use_separate_cookie_jars():
def handler(request: httpx.Request) -> httpx.Response:
mobile = request.headers.get("sec-ch-ua-mobile") == "?1"
headers = {}
if request.url.host == "www.rakuten.co.jp":
headers["set-cookie"] = f"ak_bmsc={'sp' if mobile else 'pc'}; Domain=.rakuten.co.jp; Path=/"
headers = {
"set-cookie": f"ak_bmsc={'sp' if mobile else 'pc'}; Domain=.rakuten.co.jp; Path=/"
}
return httpx.Response(200, text=GOOD_PAGE, headers=headers)
session = await build_session(handler)
@@ -282,7 +334,9 @@ async def test_search_and_detail_use_separate_cookie_jars():
assert sp_cookie == "sp"
async def test_profile_status_reports_warmup_state():
async def test_profile_status_reports_cookie_state():
"""`warmed` 现在的语义是「当前有可复用的 Akamai cookie」,字段名为兼容保留。"""
def handler(request: httpx.Request) -> httpx.Response:
headers = {"set-cookie": "ak_bmsc=abc; Domain=.rakuten.co.jp; Path=/"}
return httpx.Response(200, text=GOOD_PAGE, headers=headers)
+111 -2
View File
@@ -1,22 +1,47 @@
"""telemetry 模块测试
验证两件事
验证
1. 默认配置otel_enabled=False setup noop不初始化任何 provider
2. enabled=true + endpoint setup 注册真实 TracerProvidershutdown 复位
3. instrument_app setup 之前调用也要真的装上中间件三个服务都是导入期打桩
4. traced / set_attributes / record_error 的行为
不打真实网络OTLPSpanExporter 创建时不发请求BatchSpanProcessor 异步批量
上报在没有 span 产生时也不会触发
上报在没有 span 产生时也不会触发span 断言用独立的 InMemory provider不碰
全局 providerOTel 的全局 provider 只允许设置一次测试间共享会互相污染
"""
from __future__ import annotations
import pytest
from fastapi import FastAPI
from opentelemetry import trace
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.trace import StatusCode
from app.shared import telemetry
from app.shared.config import Settings
from app.shared.errors import OrderGuardError
from app.shared.telemetry import is_initialized, setup_telemetry, shutdown_telemetry
@pytest.fixture
def spans(monkeypatch) -> InMemorySpanExporter:
"""把 telemetry 内部取到的 tracer 换成写内存的,用于断言 span
不用全局 provider`trace.set_tracer_provider` 只生效一次一旦别的用例先
设过这里再设就被忽略OTel 只打一条 warning断言会莫名其妙拿不到 span
"""
exporter = InMemorySpanExporter()
provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(exporter))
monkeypatch.setattr(telemetry.trace, "get_tracer", provider.get_tracer)
return exporter
def test_disabled_is_noop():
"""otel_enabled=False 时 setup/instrument/shutdown 都不初始化 provider"""
settings = Settings(_env_file=None) # 默认 otel_enabled=False
@@ -24,6 +49,90 @@ def test_disabled_is_noop():
assert not is_initialized()
def test_instrument_app_works_before_setup_telemetry(monkeypatch):
"""回归:instrument_app 必须在 setup_telemetry 之前也能真的装上中间件
三个服务都在模块导入时执行 `app = create_app()`内部调 instrument_app
setup_telemetry 要等 lifespan 才跑曾经这里用 `_provider is None` 做前置
判断于是导入期一律 returnFastAPI 从来没被打桩过一条 server span
没有这条用例把那个顺序钉住
"""
monkeypatch.setattr(
telemetry, "get_settings", lambda: Settings(_env_file=None, otel_enabled=True)
)
assert not is_initialized() # 尚未 setup,正是导入期的状态
app = FastAPI()
try:
telemetry.instrument_app(app)
assert app._is_instrumented_by_opentelemetry
finally:
FastAPIInstrumentor.uninstrument_app(app)
def test_instrument_app_skipped_when_otel_disabled(monkeypatch):
"""otel 关闭时不装中间件,省掉一层用不上的开销"""
monkeypatch.setattr(
telemetry, "get_settings", lambda: Settings(_env_file=None, otel_enabled=False)
)
app = FastAPI()
telemetry.instrument_app(app)
assert not getattr(app, "_is_instrumented_by_opentelemetry", False)
async def test_traced_records_span_and_reraises(spans):
"""traced 成功时留一个 span;异常时记 ERROR 状态并原样抛出"""
@telemetry.traced("unit.ok")
async def ok() -> str:
return "done"
@telemetry.traced("unit.boom")
async def boom() -> None:
raise OrderGuardError("金额超限")
assert await ok() == "done"
with pytest.raises(OrderGuardError):
await boom()
finished = {s.name: s for s in spans.get_finished_spans()}
assert finished["unit.ok"].status.status_code is not StatusCode.ERROR
failed = finished["unit.boom"]
assert failed.status.status_code is StatusCode.ERROR
# AppError 的对外错误码要落在 span 上:排查时按码筛比按异常类名筛更贴近上游
assert failed.attributes["error.type"] == "OrderGuardError"
assert failed.attributes["error.code"] == OrderGuardError("x").err_code
async def test_traced_nests_under_caller_span(spans):
"""traced 出来的 span 要挂在调用方的 span 底下,而不是各自成为孤立 trace"""
@telemetry.traced("unit.child")
async def child() -> None:
return None
tracer = telemetry.trace.get_tracer(__name__)
with tracer.start_as_current_span("unit.root"):
await child()
by_name = {s.name: s for s in spans.get_finished_spans()}
assert by_name["unit.child"].parent.span_id == by_name["unit.root"].context.span_id
# 同一条 trace 才能在观测后台里连成一条链路
assert by_name["unit.child"].context.trace_id == by_name["unit.root"].context.trace_id
def test_set_attributes_skips_none(spans):
"""可选字段为 None 时不落属性,避免一堆 None 噪声"""
tracer = telemetry.trace.get_tracer(__name__)
with tracer.start_as_current_span("unit.attrs") as span:
telemetry.set_attributes(span, {"a": 1, "b": None, "c": "x"})
attributes = spans.get_finished_spans()[0].attributes
assert attributes["a"] == 1
assert attributes["c"] == "x"
assert "b" not in attributes
def test_enabled_initializes_and_shutdown_releases():
"""enabled=true 时 setup 注册 TracerProvider,shutdown 后 _provider 复位"""
settings = Settings(
+130
View File
@@ -20,8 +20,14 @@ from typing import Any
import pytest
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.trace import StatusCode
from app.shared.errors import BrowserDeadError, CheckoutBlockedError, OrderGuardError
from app.shared.task_state import OrderState, TaskStatus
from app.trading.worker import runner as runner_module
from app.trading.worker import verify
from app.trading.worker.evidence import EvidenceStore
from app.trading.worker.local_db import LocalDB
@@ -900,6 +906,130 @@ async def test_monitor_order_stops_after_max_checks_without_finding_order(
assert gateway.reports == []
# ---- 链路追踪:一个任务一条 trace ----
@pytest.fixture
def spans(monkeypatch) -> InMemorySpanExporter:
"""把 runner 模块级 tracer 换成写内存的
不动全局 providerOTel 的全局 provider 只允许设置一次测试间共享会互相污染
"""
exporter = InMemorySpanExporter()
provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(exporter))
monkeypatch.setattr(runner_module, "tracer", provider.get_tracer("test"))
return exporter
def _stub_happy_path(runner: WorkerRunner) -> None:
"""把整条下单流程替换成桩,让 execute() 一路走到付款完成"""
async def _clear():
return {"removed_count": 0, "cart_count": 0, "html": "<html/>", "screenshot": b"p"}
async def _snapshot(*args, **kwargs):
return PageSnapshot(html="<html/>", screenshot=b"p")
async def _parse(html: str):
return CheckoutSummary(payable_yen=297)
async def _submit(task):
return SubmitOutcome(
site_order_id="ord-1", evidence=PageSnapshot(html="<html/>", screenshot=b"p")
)
async def _unchanged(order_id: str):
return OrderStatusSnapshot(found=True, order_state=None)
runner._site.clear_cart = _clear # type: ignore[assignment]
runner._site.add_to_cart = _snapshot # type: ignore[assignment]
runner._site.verify_cart = _snapshot # type: ignore[assignment]
runner._site.enter_checkout = _snapshot # type: ignore[assignment]
runner._site.parse_checkout = _parse # type: ignore[assignment]
runner._site.submit_order = _submit # type: ignore[assignment]
runner._site.pay = _snapshot # type: ignore[assignment]
runner._site.check_order_status = _unchanged # type: ignore[assignment]
async def test_order_task_emits_one_trace_with_a_span_per_step(
runner: WorkerRunner, spans: InMemorySpanExporter
):
"""一笔下单 = 一条 trace,每个步骤一个子 span,全部挂在任务根 span 底下
worker 是后台 asyncio 任务没有 HTTP 入口不开这个根 span 的话下游每次
出站调用都各自成为孤立 trace观测后台上就只剩一堆请求记录看不出
这是同一笔单也看不出卡在哪一步这条用例把链路结构钉住
"""
_stub_happy_path(runner)
await runner.handle(_make_task(task_id="t1", intent={"item_url": "https://x/y/", "quantity": 2}))
finished = spans.get_finished_spans()
root = next(s for s in finished if s.name == "order.task")
steps = [s for s in finished if s.name.startswith("order.step.")]
# 步骤 span 覆盖整条流水,且顺序与执行顺序一致
assert [s.name for s in steps] == [
"order.step.cart-add",
"order.step.cart-check",
"order.step.order-confirm",
"order.step.order-submit",
"order.step.payment",
]
# 每个步骤都挂在任务根底下,同一条 trace
for step in steps:
assert step.parent.span_id == root.context.span_id
assert step.context.trace_id == root.context.trace_id
assert root.attributes["order.task_id"] == "t1"
assert root.attributes["order.route"] == "execute"
assert root.attributes["order.item_url"] == "https://x/y/"
# 证据路径落在 span 上:看到失败步骤能直接找到现场
assert steps[0].attributes["order.evidence_ref"]
assert steps[3].attributes["order.site_order_id"] == "ord-1"
async def test_guard_failure_marks_task_span_as_error(
runner: WorkerRunner, spans: InMemorySpanExporter
):
"""金额守卫拦下时任务根 span 要是 ERROR,并带上对外错误码
闸门拦截会被 _execute_with_renewal 吞掉转 needs_human不抛出 handle
所以根 span 很容易显示成功这里确认它没有
"""
_stub_happy_path(runner)
async def _pricey(html: str):
return CheckoutSummary(payable_yen=999_999)
runner._site.parse_checkout = _pricey # type: ignore[assignment]
await runner.handle(_make_task(task_id="t1"))
root = next(s for s in spans.get_finished_spans() if s.name == "order.task")
assert root.status.status_code is StatusCode.ERROR
assert root.attributes["error.type"] == "OrderGuardError"
# 提交步骤没有发生:拦在确认页之后、提交之前
names = [s.name for s in spans.get_finished_spans()]
assert "order.step.order-submit" not in names
async def test_idle_lease_does_not_emit_spans(runner: WorkerRunner, spans: InMemorySpanExporter):
"""空转的长轮询不产生 span——否则每 30 秒一条孤立 trace 会把观测后台刷满"""
runner._running = True
async def _lease_then_stop(worker_id: str, *, wait: int = 30, site: str | None = None):
runner.stop() # 领到空就停,避免死循环
return None
runner._gateway.lease = _lease_then_stop # type: ignore[assignment]
await runner.run()
assert spans.get_finished_spans() == ()
async def test_monitor_order_swallows_check_errors_and_keeps_polling(
runner: WorkerRunner,
):