q792602257andClaude Opus 5
104d7fef6b
把需要账号登录态的链路从抓取服务里拆出成独立进程。分界线不是「要不要登录」， 而是抓取无状态、幂等、可多开实例，而交易的写操作不可逆、登录态全局唯一、 订单监控是常驻轮询——同进程时抓取一扩容就会复制出 N 份登录态与 N 个轮询， 同一账号会被并发操作。 - app/shared：配置、错误码、日志、ApiResponse 信封 + Bearer 鉴权 + 异常处理器、 导航请求头构造器 - app/scraping：站点常量、会话、解析器与 10 个抓取接口，:31107，可多开 - app/trading：登录态查询/重载与健康检查，:31108，只能单实例 - 依赖方向锁为 scraping→shared、trading→shared，两侧互不 import； tests/test_architecture.py 用 AST 检查 import 并校验两个 app 的路径不串 - 登录态 UA 在 trading 独立持有：与抓取 UA 值相同但变更理由不同，抓取 UA 为绕 反爬可随时调整，登录 UA 一改可能触发设备校验使已落盘 cookie 失效 - scripts/login.py 与 AuthSession 共用 auth_site.PROFILES 与 is_logged_in，判据只写一遍 - 同一镜像两个启动命令，交易容器覆盖 command 并设 RAKUTEN_HEALTH_PORT 同时带上此前未提交的 ラクマ 分类接口与登录态基础设施。 验证：239 个离线用例全绿；两个入口真实启动，/health 与鉴权正常。 未验证：真实探测登录态（当前开发机无外网，对站点的连接全部超时）。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
490 lines
17 KiB
Python
490 lines
17 KiB
Python
"""API 层测试：鉴权、参数校验、错误映射与响应包装
|
|
|
|
抓取客户端被替换为桩实现，不触达真实站点。
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
from fastapi.testclient import TestClient
|
|
|
|
from app.shared.config import get_settings
|
|
from app.shared.errors import ItemNotFoundError, OffIchibaRedirectError, UpstreamBlockedError
|
|
from app.scraping.main import create_app
|
|
from app.scraping.models.scrape import (
|
|
GenreData,
|
|
GenreNode,
|
|
ItemDetailData,
|
|
RakumaCategoryData,
|
|
RakumaCategoryNode,
|
|
RakumaItemDetailData,
|
|
RakumaSearchItem,
|
|
RakumaSearchResultData,
|
|
RakumaShopDetailData,
|
|
RakumaShopItemsData,
|
|
SearchItem,
|
|
SearchResultData,
|
|
ShopDetailData,
|
|
)
|
|
|
|
TOKEN = get_settings().bearer_token
|
|
AUTH = {"Authorization": f"Bearer {TOKEN}"}
|
|
|
|
|
|
class StubClient:
|
|
"""记录入参并返回固定结果的抓取客户端桩"""
|
|
|
|
def __init__(self) -> None:
|
|
self.search_payload = None
|
|
self.detail_payload = None
|
|
self.genre_payload = None
|
|
self.shop_detail_payload = None
|
|
self.shop_items_payload = None
|
|
self.raise_on_search: Exception | None = None
|
|
self.raise_on_detail: Exception | None = None
|
|
|
|
async def search(self, payload) -> SearchResultData:
|
|
self.search_payload = payload
|
|
if self.raise_on_search:
|
|
raise self.raise_on_search
|
|
return SearchResultData(
|
|
keyword=payload.keyword,
|
|
page=payload.page,
|
|
page_size=45,
|
|
total_count=1234,
|
|
reachable_count=1234,
|
|
has_more=True,
|
|
ad_count=2,
|
|
request_url="https://search.rakuten.co.jp/search/mall/x/",
|
|
items=[SearchItem(item_id="1", item_code="c1", item_name="商品", price=100)],
|
|
)
|
|
|
|
async def item_detail(self, payload) -> ItemDetailData:
|
|
self.detail_payload = payload
|
|
if self.raise_on_detail:
|
|
raise self.raise_on_detail
|
|
return ItemDetailData(item_id="1", item_code="c1", item_name="商品", price=100)
|
|
|
|
async def genres(self, payload) -> GenreData:
|
|
self.genre_payload = payload
|
|
return GenreData(
|
|
genre_id=payload.genre_id or "",
|
|
name="テレビゲーム" if payload.genre_id else "",
|
|
children=[GenreNode(genre_id="565950", name="Nintendo Switch")],
|
|
)
|
|
|
|
async def shop_detail(self, payload) -> ShopDetailData:
|
|
self.shop_detail_payload = payload
|
|
return ShopDetailData(
|
|
shop_id=272415,
|
|
shop_code=payload.shop_code or "edion",
|
|
shop_name="エディオン 楽天市場店",
|
|
review_score=4.51,
|
|
review_count=128136,
|
|
)
|
|
|
|
async def shop_items(self, payload) -> SearchResultData:
|
|
self.shop_items_payload = payload
|
|
return await self.search(payload.to_search_request(payload.shop_id or 272415))
|
|
|
|
|
|
class StubRakumaClient:
|
|
"""ラクマ 抓取客户端桩"""
|
|
|
|
def __init__(self) -> None:
|
|
self.search_payload = None
|
|
self.category_payload = None
|
|
self.detail_payload = None
|
|
self.shop_detail_payload = None
|
|
self.shop_items_payload = None
|
|
self.raise_on_detail: Exception | None = None
|
|
|
|
async def search(self, payload) -> RakumaSearchResultData:
|
|
self.search_payload = payload
|
|
return RakumaSearchResultData(
|
|
keyword=payload.keyword,
|
|
page=payload.page,
|
|
page_size=40,
|
|
total_count=1196774,
|
|
has_more=True,
|
|
request_url="https://fril.jp/s?query=x",
|
|
items=[RakumaSearchItem(item_id="abc", item_name="商品", price=6299)],
|
|
)
|
|
|
|
async def categories(self, payload) -> RakumaCategoryData:
|
|
self.category_payload = payload
|
|
return RakumaCategoryData(
|
|
category_id=payload.category_id or "",
|
|
name="エンタメ/ホビー" if payload.category_id else "",
|
|
total_count=1686,
|
|
children=[RakumaCategoryNode(category_id="786", name="ゲームソフト/ゲーム機本体")],
|
|
)
|
|
|
|
async def item_detail(self, payload) -> RakumaItemDetailData:
|
|
self.detail_payload = payload
|
|
if self.raise_on_detail:
|
|
raise self.raise_on_detail
|
|
return RakumaItemDetailData(item_id="abc", item_name="商品", price=6299)
|
|
|
|
async def shop_detail(self, payload) -> RakumaShopDetailData:
|
|
self.shop_detail_payload = payload
|
|
return RakumaShopDetailData(shop_id="s1", shop_name="出品者", review_count=118)
|
|
|
|
async def shop_items(self, payload) -> RakumaShopItemsData:
|
|
self.shop_items_payload = payload
|
|
return RakumaShopItemsData(
|
|
shop_id="s1",
|
|
shop_name="出品者",
|
|
page=payload.page,
|
|
total_count=21,
|
|
items=[RakumaSearchItem(item_id="abc", item_name="商品", price=6299)],
|
|
)
|
|
|
|
|
|
@pytest.fixture
|
|
def client_and_stub():
|
|
app = create_app()
|
|
with TestClient(app) as client:
|
|
stub = StubClient()
|
|
rakuma_stub = StubRakumaClient()
|
|
app.state.container.rakuten_client = stub
|
|
app.state.container.rakuma_client = rakuma_stub
|
|
yield client, stub, rakuma_stub
|
|
|
|
|
|
@pytest.fixture
|
|
def client(client_and_stub):
|
|
return client_and_stub[0]
|
|
|
|
|
|
@pytest.fixture
|
|
def stub(client_and_stub):
|
|
return client_and_stub[1]
|
|
|
|
|
|
@pytest.fixture
|
|
def rakuma_stub(client_and_stub):
|
|
return client_and_stub[2]
|
|
|
|
|
|
# ---- 鉴权 ----
|
|
|
|
@pytest.mark.parametrize("path", ["/api/search", "/api/item_detail", "/api/genres"])
|
|
def test_endpoints_reject_missing_token(client, path):
|
|
response = client.post(path, json={"keyword": "a", "item_url": "https://item.rakuten.co.jp/s/c/"})
|
|
assert response.status_code == 401
|
|
body = response.json()
|
|
assert body["success"] is False
|
|
assert body["code"] == 1001
|
|
|
|
|
|
def test_endpoints_reject_wrong_token(client):
|
|
response = client.post(
|
|
"/api/search", json={"keyword": "a"}, headers={"Authorization": "Bearer wrong-token"}
|
|
)
|
|
assert response.status_code == 401
|
|
|
|
|
|
def test_endpoints_reject_non_bearer_scheme(client):
|
|
response = client.post("/api/search", json={"keyword": "a"}, headers={"Authorization": TOKEN})
|
|
assert response.status_code == 401
|
|
|
|
|
|
def test_health_needs_no_token(client):
|
|
response = client.get("/health")
|
|
assert response.status_code == 200
|
|
body = response.json()
|
|
assert body["data"]["status"] == "ok"
|
|
assert set(body["data"]["sessions"]) == {"pc", "sp", "rakuma"}
|
|
|
|
|
|
# ---- 参数校验 ----
|
|
|
|
def test_search_requires_a_target(client):
|
|
response = client.post("/api/search", json={}, headers=AUTH)
|
|
assert response.status_code == 422
|
|
assert response.json()["code"] == 1002
|
|
|
|
|
|
def test_search_rejects_page_beyond_site_limit(client):
|
|
response = client.post("/api/search", json={"keyword": "a", "page": 151}, headers=AUTH)
|
|
assert response.status_code == 422
|
|
|
|
|
|
def test_search_rejects_unknown_sort(client):
|
|
response = client.post("/api/search", json={"keyword": "a", "sort": "cheapest"}, headers=AUTH)
|
|
assert response.status_code == 422
|
|
|
|
|
|
def test_item_detail_requires_url_or_code_pair(client):
|
|
response = client.post("/api/item_detail", json={"shop_code": "edion"}, headers=AUTH)
|
|
assert response.status_code == 422
|
|
|
|
|
|
# ---- 正常响应 ----
|
|
|
|
def test_search_passes_filters_through_to_client(client, stub):
|
|
response = client.post(
|
|
"/api/search",
|
|
json={
|
|
"keyword": "switch",
|
|
"page": 2,
|
|
"sort": "price_asc",
|
|
"min_price": 1000,
|
|
"condition": "used",
|
|
"free_shipping": True,
|
|
"exclude_ads": False,
|
|
},
|
|
headers=AUTH,
|
|
)
|
|
assert response.status_code == 200
|
|
payload = stub.search_payload
|
|
assert payload.keyword == "switch"
|
|
assert payload.page == 2
|
|
assert payload.sort.value == "price_asc"
|
|
assert payload.min_price == 1000
|
|
assert payload.condition.value == "used"
|
|
assert payload.free_shipping is True
|
|
assert payload.exclude_ads is False
|
|
|
|
body = response.json()
|
|
assert body["success"] is True
|
|
assert body["code"] == 0
|
|
assert body["data"]["total_count"] == 1234
|
|
assert body["data"]["items"][0]["item_code"] == "c1"
|
|
|
|
|
|
def test_genres_accepts_empty_body_for_top_level(client, stub):
|
|
"""顶层分类不需要任何入参"""
|
|
response = client.post("/api/genres", json={}, headers=AUTH)
|
|
assert response.status_code == 200
|
|
assert stub.genre_payload.genre_id is None
|
|
assert response.json()["data"]["children"][0]["genre_id"] == "565950"
|
|
|
|
|
|
def test_genres_passes_genre_id_through(client, stub):
|
|
response = client.post("/api/genres", json={"genre_id": "101205"}, headers=AUTH)
|
|
assert response.status_code == 200
|
|
assert stub.genre_payload.genre_id == "101205"
|
|
assert response.json()["data"]["name"] == "テレビゲーム"
|
|
|
|
|
|
def test_item_detail_defaults_to_including_sku_variants(client, stub):
|
|
response = client.post(
|
|
"/api/item_detail",
|
|
json={"shop_code": "edion", "item_code": "4902370549263"},
|
|
headers=AUTH,
|
|
)
|
|
assert response.status_code == 200
|
|
assert stub.detail_payload.include_sku_variants is True
|
|
assert response.json()["data"]["item_name"] == "商品"
|
|
|
|
|
|
# ---- 抓取异常映射 ----
|
|
|
|
def test_blocked_upstream_maps_to_structured_error(client, stub):
|
|
stub.raise_on_search = UpstreamBlockedError("blocked by akamai")
|
|
response = client.post("/api/search", json={"keyword": "a"}, headers=AUTH)
|
|
assert response.status_code == 400
|
|
body = response.json()
|
|
assert body["success"] is False
|
|
assert body["code"] == 3002
|
|
assert "blocked" in body["msg"]
|
|
|
|
|
|
def test_off_ichiba_redirect_gets_its_own_error_code(client, stub):
|
|
"""楽天ブックス等官方子站的商品不该被当成反爬拦截，上游要能据此改走别的通道"""
|
|
stub.raise_on_detail = OffIchibaRedirectError(
|
|
"https://item.rakuten.co.jp/book/16033028/",
|
|
"https://books.rakuten.co.jp/rb/16033028/",
|
|
)
|
|
response = client.post(
|
|
"/api/item_detail", json={"shop_code": "book", "item_code": "16033028"}, headers=AUTH
|
|
)
|
|
assert response.status_code == 400
|
|
body = response.json()
|
|
assert body["code"] == 4002
|
|
assert "books.rakuten.co.jp" in body["msg"]
|
|
|
|
|
|
def test_missing_item_maps_to_404(client, stub):
|
|
stub.raise_on_detail = ItemNotFoundError("no such item")
|
|
response = client.post(
|
|
"/api/item_detail", json={"shop_code": "s", "item_code": "c"}, headers=AUTH
|
|
)
|
|
assert response.status_code == 404
|
|
assert response.json()["code"] == 4004
|
|
|
|
|
|
# ---- 乐天商家 ----
|
|
|
|
def test_shop_detail_returns_store_profile(client, stub):
|
|
response = client.post("/api/shop_detail", json={"shop_code": "edion"}, headers=AUTH)
|
|
assert response.status_code == 200
|
|
assert stub.shop_detail_payload.shop_code == "edion"
|
|
data = response.json()["data"]
|
|
assert data["shop_id"] == 272415
|
|
assert data["review_count"] == 128136
|
|
|
|
|
|
def test_shop_detail_requires_an_identifier(client):
|
|
response = client.post("/api/shop_detail", json={}, headers=AUTH)
|
|
assert response.status_code == 422
|
|
|
|
|
|
def test_shop_items_scopes_the_search_to_the_shop(client, stub):
|
|
"""商家商品实际是一次 sid 限定的搜索，筛选字段要原样传下去"""
|
|
response = client.post(
|
|
"/api/shop_items",
|
|
json={"shop_id": 272415, "page": 2, "keyword": "テレビ", "sort": "price_asc"},
|
|
headers=AUTH,
|
|
)
|
|
assert response.status_code == 200
|
|
assert stub.shop_items_payload.shop_id == 272415
|
|
# 转换后的搜索请求必须带上店铺限定
|
|
assert stub.search_payload.shop_id == 272415
|
|
assert stub.search_payload.keyword == "テレビ"
|
|
assert stub.search_payload.page == 2
|
|
assert response.json()["data"]["total_count"] == 1234
|
|
|
|
|
|
def test_shop_items_requires_an_identifier(client):
|
|
response = client.post("/api/shop_items", json={"page": 2}, headers=AUTH)
|
|
assert response.status_code == 422
|
|
|
|
|
|
# ---- ラクマ ----
|
|
|
|
@pytest.mark.parametrize(
|
|
"path",
|
|
[
|
|
"/api/rakuma/search",
|
|
"/api/rakuma/categories",
|
|
"/api/rakuma/item_detail",
|
|
"/api/rakuma/shop_detail",
|
|
"/api/rakuma/shop_items",
|
|
],
|
|
)
|
|
def test_rakuma_endpoints_reject_missing_token(client, path):
|
|
response = client.post(path, json={"keyword": "a", "item_id": "x", "shop_id": "s"})
|
|
assert response.status_code == 401
|
|
assert response.json()["code"] == 1001
|
|
|
|
|
|
def test_rakuma_search_passes_filters_through(client, rakuma_stub):
|
|
response = client.post(
|
|
"/api/rakuma/search",
|
|
json={
|
|
"keyword": "switch",
|
|
"page": 2,
|
|
"sort": "price_asc",
|
|
"conditions": ["new", "almost_new"],
|
|
"transaction": "on_sale",
|
|
"free_shipping": True,
|
|
"brand_id": "5296",
|
|
},
|
|
headers=AUTH,
|
|
)
|
|
assert response.status_code == 200
|
|
payload = rakuma_stub.search_payload
|
|
assert payload.keyword == "switch"
|
|
assert payload.page == 2
|
|
assert payload.sort.value == "price_asc"
|
|
assert [c.value for c in payload.conditions] == ["new", "almost_new"]
|
|
assert payload.transaction.value == "on_sale"
|
|
assert payload.free_shipping is True
|
|
assert payload.brand_id == "5296"
|
|
assert response.json()["data"]["total_count"] == 1196774
|
|
|
|
|
|
def test_rakuma_search_requires_a_target(client):
|
|
response = client.post("/api/rakuma/search", json={}, headers=AUTH)
|
|
assert response.status_code == 422
|
|
assert response.json()["code"] == 1002
|
|
|
|
|
|
def test_rakuma_search_rejects_page_beyond_site_limit(client):
|
|
"""站点 page>100 直接 404，不必打出去才知道"""
|
|
response = client.post(
|
|
"/api/rakuma/search", json={"keyword": "a", "page": 101}, headers=AUTH
|
|
)
|
|
assert response.status_code == 422
|
|
|
|
|
|
def test_rakuma_search_rejects_rakuten_only_sort(client):
|
|
"""两站排序枚举不同，乐天的 review_count 在 ラクマ 上不存在"""
|
|
response = client.post(
|
|
"/api/rakuma/search", json={"keyword": "a", "sort": "review_count"}, headers=AUTH
|
|
)
|
|
assert response.status_code == 422
|
|
|
|
|
|
def test_rakuma_categories_accepts_empty_body_for_top_level(client, rakuma_stub):
|
|
"""不传 category_id 时取顶层分类，不应因缺参数被拦下"""
|
|
response = client.post("/api/rakuma/categories", json={}, headers=AUTH)
|
|
assert response.status_code == 200
|
|
assert rakuma_stub.category_payload.category_id is None
|
|
assert rakuma_stub.category_payload.include_descendants is False
|
|
assert response.json()["data"]["total_count"] == 1686
|
|
|
|
|
|
def test_rakuma_categories_passes_options_through(client, rakuma_stub):
|
|
response = client.post(
|
|
"/api/rakuma/categories",
|
|
json={"category_id": "10007", "include_descendants": True},
|
|
headers=AUTH,
|
|
)
|
|
assert response.status_code == 200
|
|
assert rakuma_stub.category_payload.category_id == "10007"
|
|
assert rakuma_stub.category_payload.include_descendants is True
|
|
assert response.json()["data"]["children"][0]["category_id"] == "786"
|
|
|
|
|
|
def test_rakuma_item_detail_accepts_item_id(client, rakuma_stub):
|
|
response = client.post("/api/rakuma/item_detail", json={"item_id": "abc"}, headers=AUTH)
|
|
assert response.status_code == 200
|
|
assert rakuma_stub.detail_payload.item_id == "abc"
|
|
assert response.json()["data"]["price"] == 6299
|
|
|
|
|
|
def test_rakuma_item_detail_requires_an_identifier(client):
|
|
response = client.post("/api/rakuma/item_detail", json={}, headers=AUTH)
|
|
assert response.status_code == 422
|
|
|
|
|
|
def test_rakuma_missing_item_maps_to_404(client, rakuma_stub):
|
|
rakuma_stub.raise_on_detail = ItemNotFoundError("no such item")
|
|
response = client.post("/api/rakuma/item_detail", json={"item_id": "x"}, headers=AUTH)
|
|
assert response.status_code == 404
|
|
assert response.json()["code"] == 4004
|
|
|
|
|
|
def test_rakuma_shop_detail_skips_reviews_by_default(client, rakuma_stub):
|
|
"""评价明细要多打一次请求，默认不取"""
|
|
response = client.post("/api/rakuma/shop_detail", json={"shop_id": "s1"}, headers=AUTH)
|
|
assert response.status_code == 200
|
|
assert rakuma_stub.shop_detail_payload.include_reviews is False
|
|
assert response.json()["data"]["review_count"] == 118
|
|
|
|
|
|
def test_rakuma_shop_detail_can_request_reviews(client, rakuma_stub):
|
|
response = client.post(
|
|
"/api/rakuma/shop_detail", json={"shop_id": "s1", "include_reviews": True}, headers=AUTH
|
|
)
|
|
assert response.status_code == 200
|
|
assert rakuma_stub.shop_detail_payload.include_reviews is True
|
|
|
|
|
|
def test_rakuma_shop_items_passes_page_through(client, rakuma_stub):
|
|
response = client.post(
|
|
"/api/rakuma/shop_items", json={"shop_id": "s1", "page": 3}, headers=AUTH
|
|
)
|
|
assert response.status_code == 200
|
|
assert rakuma_stub.shop_items_payload.page == 3
|
|
assert response.json()["data"]["total_count"] == 21
|
|
|
|
|
|
@pytest.mark.parametrize("path", ["/api/rakuma/shop_detail", "/api/rakuma/shop_items"])
|
|
def test_rakuma_shop_endpoints_require_an_identifier(client, path):
|
|
response = client.post(path, json={}, headers=AUTH)
|
|
assert response.status_code == 422
|