q792602257andClaude Opus 5
3c7618a1d6
两个问题一起处理，都与「出站请求与可观测性」有关。 ## 抓取：正常路径不再多打一次首页 site_session 原先每条通道每 30 分钟打一次 www.rakuten.co.jp/ 做预热，而且预热 返回非 2xx 时 warmed_at 不置位——那种情况下每个请求前都会再打一次首页。 Akamai 的 cookie 随任意页面响应下发，目标页自己就会带回来，专门先打一次首页除了 多一个出站请求（以及多一次被风控计数的机会）之外没有额外收益：首个请求无论打哪个 URL 都是冷的 ~11s，之后都复用 cookie。 改为 cookie 由目标页响应建立（_note_cookies）、超 TTL 主动清空 （_drop_expired_cookies）。首页只保留在失败修复路径上（_rewarm_on_home）：目标页 已经吃了挑战页时，拿首页换一套干净 cookie 比继续撞同一个 URL 更安全。happy path 的出站请求数 2 → 1。 _note_cookies 刻意不在每次响应时刷新时刻：TTL 要从「这套 cookie 第一次出现」算起， 每次都刷新会让一套 cookie 被无限续命，反而绕过了 session_ttl_seconds 的本意。 profile_status() 的 warmed 字段名保留（上游健康检查看板在用），语义改为「当前有 可复用的 Akamai cookie」，不再代表「已专门预热过首页」。 ## 交易：此前没有任何有意义的链路数据 根因是 trading 的实际工作两类自动埋点都覆盖不到：站点交互走 Playwright（不经 httpx），worker 主循环是后台 asyncio 任务（没有 HTTP 入口，因此没有根 span）。 于是发给网关的每次 httpx 调用各自成为孤立 trace——观测后台上只剩一堆请求记录。 新增手工埋点： - order.task：一笔下单的根 span，一个 task_id 一条 trace，带 order.route （execute / recovery / already_finished）与终态 order.terminal_status - order.step.*：清车 → 加购 → 校验 → 确认 → 提交 → 付款，每步一个子 span， 带 order.evidence_ref，可从 span 直接定位落盘证据 - site.*：12 个 Playwright 交互方法（用 traced 装饰器而非 with 块——这些方法的 函数体本就很长，再加一层缩进不利于阅读） - account_query：只读查询单的根 span，带 query.outcome 空转的长轮询（30 秒一次、绝大多数返回空）用 suppressed() 屏蔽：量大且没有信息量， 把观测后台刷满的正是它们。领到任务后的网关调用都在任务根 span 底下，不受影响。 闸门 / 风控拦截会被 _execute_with_renewal 吞掉转 needs_human，异常冒不到根 span， 被拦下的单在 trace 里跟成功下单一模一样。加 _execute_recording_errors 一层统一 记录，比每个 except 分支各写一遍省事，也不会漏掉后续新增的分支。 _report_safe 写 span 属性前判断 is_recording()：付款后监控是 create_task 起的， asyncio 在创建时就把 context 复制了进去，等它真正跑起来根 span 早已结束—— get_current_span() 拿到的仍是那个已结束的 span（不是 INVALID_SPAN），写属性会打 "Setting attribute on ended span"。当前监控路径不传 terminal_status 走不到那里， 这道判断是防以后。 ## 顺带修掉：instrument_app 从未生效 instrument_app 用 _provider is None 做前置判断，但三个服务都在模块导入时执行 app = create_app()，而 setup_telemetry 要等 lifespan 才跑——那时 _provider 还是 None，照着判断直接 return。**FastAPI 从来没被打桩过，三个服务一条 server span 都没有。** 实测确认两件事：导入期打桩能出 span，lifespan 内打桩出不来（instrument_app 是加 中间件，应用开始服务后加进去不生效）；provider 后设也不影响 ProxyTracer 委托到 真实 provider。所以只能在导入期装，判断条件改为 otel_enabled。 app/gateway/main.py 此前完全没接 telemetry，worker 出站请求带过来的 traceparent 没人接上，一条下单链路在网关这里断掉，只看得到 worker 侧那半截。补上 setup_telemetry(service_name="rakuten-gateway") 与 instrument_app / shutdown。 ## 验证 新增 8 个用例：首页零请求、cookie 复用与过期清空、失败后用首页换 cookie、一任务 一 trace 的父子结构、闸门失败标 ERROR、空转不埋点，以及 instrument_app 调用顺序 的回归测试。全量 526 passed。 Playwright 那些 site.* 埋点只做了静态验证（测试用桩替换站点方法），没有跑真实 浏览器下单确认 span 真的落地。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
354 lines
12 KiB
Python
354 lines
12 KiB
Python
"""会话层测试：cookie 预热、失败升级、浏览器兜底降级
|
|
|
|
全部用 httpx.MockTransport 拦截，不触达真实站点。
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import httpx
|
|
import pytest
|
|
|
|
from app.scraping.core import site
|
|
from app.shared.config import Settings
|
|
from app.shared.errors import (
|
|
ItemNotFoundError,
|
|
OffIchibaRedirectError,
|
|
UpstreamBlockedError,
|
|
UpstreamRequestError,
|
|
)
|
|
from app.scraping.parsers.subsites import build_item_page_validator
|
|
from app.scraping.services.browser_fallback import BrowserVisit
|
|
from app.scraping.services.site_session import SiteSession
|
|
|
|
GOOD_PAGE = '<html><script>window.__INITIAL_STATE__ = {"ok":1};</script></html>'
|
|
BLOCK_PAGE = "<html><body>Access Denied. Reference #18.abc</body></html>"
|
|
TARGET = "https://search.rakuten.co.jp/search/mall/x/"
|
|
|
|
|
|
class FakeBrowser:
|
|
"""可控的浏览器兜底替身"""
|
|
|
|
def __init__(self, visit: BrowserVisit | None = None):
|
|
self.visit_result = visit
|
|
self.calls: list[tuple[str, bool]] = []
|
|
self.enabled = True
|
|
self.unavailable_reason = None if visit else "playwright is not installed"
|
|
self.ready = visit is not None
|
|
|
|
async def visit(self, url: str, *, mobile: bool) -> BrowserVisit | None:
|
|
self.calls.append((url, mobile))
|
|
return self.visit_result
|
|
|
|
async def close(self) -> None:
|
|
pass
|
|
|
|
|
|
def make_settings(**overrides) -> Settings:
|
|
base = {
|
|
"http_max_attempts": 3,
|
|
"request_timeout_seconds": 5.0,
|
|
"max_site_concurrency": 4,
|
|
"browser_fallback_enabled": True,
|
|
}
|
|
base.update(overrides)
|
|
return Settings(**base)
|
|
|
|
|
|
async def build_session(handler, *, browser=None, settings=None) -> SiteSession:
|
|
"""构建 SiteSession 并把两条通道的 client 换成 MockTransport 版本"""
|
|
session = SiteSession(settings or make_settings(), browser or FakeBrowser())
|
|
await session.start()
|
|
for profile in session._profiles.values():
|
|
await profile.client.aclose()
|
|
profile.client = httpx.AsyncClient(
|
|
headers=site.default_headers(mobile=profile.mobile),
|
|
transport=httpx.MockTransport(handler),
|
|
follow_redirects=True,
|
|
)
|
|
return session
|
|
|
|
|
|
async def test_happy_path_never_touches_the_home_page():
|
|
"""目标页自己会带回 Akamai cookie，正常路径不该多打一次首页。"""
|
|
seen: list[str] = []
|
|
|
|
def handler(request: httpx.Request) -> httpx.Response:
|
|
seen.append(str(request.url))
|
|
return httpx.Response(
|
|
200,
|
|
text=GOOD_PAGE,
|
|
headers={"set-cookie": "ak_bmsc=abc; Domain=.rakuten.co.jp; Path=/"},
|
|
)
|
|
|
|
session = await build_session(handler)
|
|
try:
|
|
await session.fetch_html(TARGET, mobile=False)
|
|
await session.fetch_html(TARGET, mobile=False)
|
|
finally:
|
|
await session.close()
|
|
|
|
assert seen == [TARGET, TARGET]
|
|
|
|
|
|
async def test_cookies_from_target_page_are_reused_across_fetches():
|
|
"""第一次响应下发的 cookie 要带到后续请求上，不必再走首页。"""
|
|
cookie_headers: list[str | None] = []
|
|
|
|
def handler(request: httpx.Request) -> httpx.Response:
|
|
cookie_headers.append(request.headers.get("cookie"))
|
|
return httpx.Response(
|
|
200,
|
|
text=GOOD_PAGE,
|
|
headers={"set-cookie": "ak_bmsc=abc; Domain=.rakuten.co.jp; Path=/"},
|
|
)
|
|
|
|
session = await build_session(handler)
|
|
try:
|
|
await session.fetch_html(TARGET, mobile=False)
|
|
await session.fetch_html(TARGET, mobile=False)
|
|
finally:
|
|
await session.close()
|
|
|
|
assert cookie_headers[0] is None # 首个请求是冷的
|
|
assert "ak_bmsc=abc" in (cookie_headers[1] or "")
|
|
|
|
|
|
async def test_expired_cookies_are_dropped_before_next_fetch():
|
|
"""cookie 罐超过 session_ttl_seconds 后要清空，不能带着过期 cookie 去撞。"""
|
|
cookie_headers: list[str | None] = []
|
|
|
|
def handler(request: httpx.Request) -> httpx.Response:
|
|
cookie_headers.append(request.headers.get("cookie"))
|
|
return httpx.Response(
|
|
200,
|
|
text=GOOD_PAGE,
|
|
headers={"set-cookie": "ak_bmsc=abc; Domain=.rakuten.co.jp; Path=/"},
|
|
)
|
|
|
|
session = await build_session(handler, settings=make_settings(session_ttl_seconds=0.0))
|
|
try:
|
|
await session.fetch_html(TARGET, mobile=False)
|
|
await session.fetch_html(TARGET, mobile=False)
|
|
finally:
|
|
await session.close()
|
|
|
|
assert cookie_headers == [None, None] # 每次都从干净状态起步
|
|
|
|
|
|
async def test_missing_state_marker_is_treated_as_blocked():
|
|
def handler(request: httpx.Request) -> httpx.Response:
|
|
return httpx.Response(200, text="<html>no state here</html>")
|
|
|
|
session = await build_session(handler)
|
|
try:
|
|
with pytest.raises(UpstreamBlockedError):
|
|
await session.fetch_html(TARGET, mobile=False)
|
|
finally:
|
|
await session.close()
|
|
|
|
|
|
async def test_retry_recovers_when_a_later_attempt_succeeds():
|
|
attempts = {"n": 0}
|
|
|
|
def handler(request: httpx.Request) -> httpx.Response:
|
|
if request.url.host == "www.rakuten.co.jp":
|
|
return httpx.Response(200, text="home")
|
|
attempts["n"] += 1
|
|
if attempts["n"] == 1:
|
|
return httpx.Response(200, text=BLOCK_PAGE)
|
|
return httpx.Response(200, text=GOOD_PAGE)
|
|
|
|
session = await build_session(handler)
|
|
try:
|
|
assert await session.fetch_html(TARGET, mobile=False) == GOOD_PAGE
|
|
finally:
|
|
await session.close()
|
|
assert attempts["n"] == 2
|
|
|
|
|
|
async def test_first_failure_swaps_cookies_via_home_page():
|
|
"""首次失败时用首页换一套 cookie——这是首页 URL 唯一的用途。"""
|
|
seen: list[str] = []
|
|
|
|
def handler(request: httpx.Request) -> httpx.Response:
|
|
seen.append(str(request.url))
|
|
if request.url.host == "www.rakuten.co.jp":
|
|
return httpx.Response(
|
|
200,
|
|
text="home",
|
|
headers={"set-cookie": "ak_bmsc=fresh; Domain=.rakuten.co.jp; Path=/"},
|
|
)
|
|
if seen.count(TARGET) == 1:
|
|
return httpx.Response(200, text=BLOCK_PAGE)
|
|
return httpx.Response(200, text=GOOD_PAGE)
|
|
|
|
session = await build_session(handler)
|
|
try:
|
|
assert await session.fetch_html(TARGET, mobile=False) == GOOD_PAGE
|
|
finally:
|
|
await session.close()
|
|
|
|
# 首页只在失败之后出现一次，且排在两次目标页请求中间
|
|
assert seen == [TARGET, "https://www.rakuten.co.jp/", TARGET]
|
|
|
|
|
|
async def test_browser_fallback_supplies_cookies_and_page_on_persistent_block():
|
|
"""浏览器已经取到页面时应直接采用，不再多打一次 HTTP"""
|
|
browser = FakeBrowser(
|
|
BrowserVisit(
|
|
cookies=[{"name": "bm_sv", "value": "xyz", "domain": ".rakuten.co.jp", "path": "/"}],
|
|
html=GOOD_PAGE,
|
|
)
|
|
)
|
|
|
|
def handler(request: httpx.Request) -> httpx.Response:
|
|
if request.url.host == "www.rakuten.co.jp":
|
|
return httpx.Response(200, text="home")
|
|
return httpx.Response(200, text=BLOCK_PAGE)
|
|
|
|
session = await build_session(handler, browser=browser)
|
|
try:
|
|
assert await session.fetch_html(TARGET, mobile=False) == GOOD_PAGE
|
|
finally:
|
|
cookie_names = {cookie.name for cookie in session._profiles["pc"].client.cookies.jar}
|
|
await session.close()
|
|
|
|
assert browser.calls == [(TARGET, False)]
|
|
assert "bm_sv" in cookie_names # cookie 已回灌，后续请求可复用
|
|
|
|
|
|
async def test_missing_playwright_degrades_to_blocked_error_not_crash():
|
|
browser = FakeBrowser(None) # 模拟未安装 playwright
|
|
|
|
def handler(request: httpx.Request) -> httpx.Response:
|
|
return httpx.Response(200, text=BLOCK_PAGE)
|
|
|
|
session = await build_session(handler, browser=browser)
|
|
try:
|
|
with pytest.raises(UpstreamBlockedError):
|
|
await session.fetch_html(TARGET, mobile=False)
|
|
finally:
|
|
await session.close()
|
|
assert browser.calls # 尝试过兜底
|
|
|
|
|
|
async def test_404_fails_fast_without_retrying():
|
|
attempts = {"n": 0}
|
|
|
|
def handler(request: httpx.Request) -> httpx.Response:
|
|
if request.url.host == "www.rakuten.co.jp":
|
|
return httpx.Response(200, text="home")
|
|
attempts["n"] += 1
|
|
return httpx.Response(404, text="not found")
|
|
|
|
session = await build_session(handler)
|
|
try:
|
|
with pytest.raises(ItemNotFoundError):
|
|
await session.fetch_html("https://item.rakuten.co.jp/s/c/", mobile=True)
|
|
finally:
|
|
await session.close()
|
|
assert attempts["n"] == 1
|
|
|
|
|
|
async def test_unsupported_subsite_redirect_fails_fast_without_retrying():
|
|
"""跳到未登记的乐天子站时重试没有意义，应立刻以专门的错误返回"""
|
|
attempts = {"n": 0}
|
|
requested = "https://item.rakuten.co.jp/somewhere/1/"
|
|
|
|
def handler(request: httpx.Request) -> httpx.Response:
|
|
if request.url.host == "www.rakuten.co.jp":
|
|
return httpx.Response(200, text="home")
|
|
if request.url.host == "item.rakuten.co.jp":
|
|
attempts["n"] += 1
|
|
return httpx.Response(302, headers={"location": "https://unknown.rakuten.co.jp/x/1/"})
|
|
return httpx.Response(200, text="<html>unknown site</html>")
|
|
|
|
session = await build_session(handler)
|
|
try:
|
|
with pytest.raises(OffIchibaRedirectError) as exc:
|
|
await session.fetch(
|
|
requested, mobile=True, validator=build_item_page_validator(requested)
|
|
)
|
|
finally:
|
|
await session.close()
|
|
assert attempts["n"] == 1
|
|
assert exc.value.final_url == "https://unknown.rakuten.co.jp/x/1/"
|
|
|
|
|
|
async def test_known_subsite_redirect_is_accepted_and_reports_final_url():
|
|
"""已登记的子站应正常通过校验，并把落地地址回报给调用方用于分派"""
|
|
requested = "https://item.rakuten.co.jp/biccamera/1/"
|
|
landed = "https://biccamera.rakuten.co.jp/item/1/"
|
|
|
|
def handler(request: httpx.Request) -> httpx.Response:
|
|
if request.url.host == "www.rakuten.co.jp":
|
|
return httpx.Response(200, text="home")
|
|
if request.url.host == "item.rakuten.co.jp":
|
|
return httpx.Response(302, headers={"location": landed})
|
|
return httpx.Response(200, text='<html><script>window.__NUXT__={"state":{}}</script></html>')
|
|
|
|
session = await build_session(handler)
|
|
try:
|
|
page = await session.fetch(
|
|
requested, mobile=True, validator=build_item_page_validator(requested)
|
|
)
|
|
finally:
|
|
await session.close()
|
|
assert page.url == landed
|
|
assert "__NUXT__" in page.html
|
|
|
|
|
|
async def test_upstream_5xx_surfaces_as_request_error():
|
|
def handler(request: httpx.Request) -> httpx.Response:
|
|
if request.url.host == "www.rakuten.co.jp":
|
|
return httpx.Response(200, text="home")
|
|
return httpx.Response(503, text="service unavailable")
|
|
|
|
session = await build_session(handler)
|
|
try:
|
|
with pytest.raises(UpstreamRequestError):
|
|
await session.fetch_html(TARGET, mobile=False)
|
|
finally:
|
|
await session.close()
|
|
|
|
|
|
async def test_search_and_detail_use_separate_cookie_jars():
|
|
"""PC 与手机两条通道的 cookie 不能互相污染"""
|
|
|
|
def handler(request: httpx.Request) -> httpx.Response:
|
|
mobile = request.headers.get("sec-ch-ua-mobile") == "?1"
|
|
headers = {
|
|
"set-cookie": f"ak_bmsc={'sp' if mobile else 'pc'}; Domain=.rakuten.co.jp; Path=/"
|
|
}
|
|
return httpx.Response(200, text=GOOD_PAGE, headers=headers)
|
|
|
|
session = await build_session(handler)
|
|
try:
|
|
await session.fetch_html(TARGET, mobile=False)
|
|
await session.fetch_html("https://item.rakuten.co.jp/s/c/", mobile=True)
|
|
pc_cookie = session._profiles["pc"].client.cookies.get("ak_bmsc")
|
|
sp_cookie = session._profiles["sp"].client.cookies.get("ak_bmsc")
|
|
finally:
|
|
await session.close()
|
|
|
|
assert pc_cookie == "pc"
|
|
assert sp_cookie == "sp"
|
|
|
|
|
|
async def test_profile_status_reports_cookie_state():
|
|
"""`warmed` 现在的语义是「当前有可复用的 Akamai cookie」，字段名为兼容保留。"""
|
|
|
|
def handler(request: httpx.Request) -> httpx.Response:
|
|
headers = {"set-cookie": "ak_bmsc=abc; Domain=.rakuten.co.jp; Path=/"}
|
|
return httpx.Response(200, text=GOOD_PAGE, headers=headers)
|
|
|
|
session = await build_session(handler)
|
|
try:
|
|
assert session.profile_status()["pc"]["warmed"] is False
|
|
await session.fetch_html(TARGET, mobile=False)
|
|
status = session.profile_status()
|
|
assert status["pc"]["warmed"] is True
|
|
assert "ak_bmsc" in status["pc"]["cookies"]
|
|
assert status["sp"]["warmed"] is False
|
|
finally:
|
|
await session.close()
|