拆分抓取与交易服务
把需要账号登录态的链路从抓取服务里拆出成独立进程。分界线不是「要不要登录」, 而是抓取无状态、幂等、可多开实例,而交易的写操作不可逆、登录态全局唯一、 订单监控是常驻轮询——同进程时抓取一扩容就会复制出 N 份登录态与 N 个轮询, 同一账号会被并发操作。 - app/shared:配置、错误码、日志、ApiResponse 信封 + Bearer 鉴权 + 异常处理器、 导航请求头构造器 - app/scraping:站点常量、会话、解析器与 10 个抓取接口,:31107,可多开 - app/trading:登录态查询/重载与健康检查,:31108,只能单实例 - 依赖方向锁为 scraping→shared、trading→shared,两侧互不 import; tests/test_architecture.py 用 AST 检查 import 并校验两个 app 的路径不串 - 登录态 UA 在 trading 独立持有:与抓取 UA 值相同但变更理由不同,抓取 UA 为绕 反爬可随时调整,登录 UA 一改可能触发设备校验使已落盘 cookie 失效 - scripts/login.py 与 AuthSession 共用 auth_site.PROFILES 与 is_logged_in,判据只写一遍 - 同一镜像两个启动命令,交易容器覆盖 command 并设 RAKUTEN_HEALTH_PORT 同时带上此前未提交的 ラクマ 分类接口与登录态基础设施。 验证:239 个离线用例全绿;两个入口真实启动,/health 与鉴权正常。 未验证:真实探测登录态(当前开发机无外网,对站点的连接全部超时)。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,89 @@
|
||||
"""登录态路由:查询与重新加载账号登录态
|
||||
|
||||
抓取接口全部匿名,只有加购与下单需要账号。登录本身不在这里做——两站登录都要过
|
||||
reCAPTCHA 与设备验证,由 `scripts/login.py` 起有头浏览器人工完成一次,
|
||||
本服务只读取落盘的 cookie。这里提供的是运维视角的两个动作:
|
||||
|
||||
- `/api/auth/status`:现在还登录着吗(默认真实打一次请求探测,不看缓存)
|
||||
- `/api/auth/reload`:人工重新登录后,免重启服务重新读取登录态
|
||||
"""
|
||||
from fastapi import APIRouter, Depends
|
||||
|
||||
from app.shared.api import ApiResponse, get_container, require_bearer_token
|
||||
from app.trading.container import TradingContainer
|
||||
from app.trading.models import (
|
||||
AuthReloadData,
|
||||
AuthReloadRequest,
|
||||
AuthSiteStatus,
|
||||
AuthStatusData,
|
||||
AuthStatusRequest,
|
||||
)
|
||||
from app.trading.services.auth_session import AuthStatus
|
||||
|
||||
router = APIRouter(prefix="/api/auth", tags=["auth"])
|
||||
|
||||
|
||||
def _to_model(status: AuthStatus) -> AuthSiteStatus:
|
||||
return AuthSiteStatus(**status.to_dict())
|
||||
|
||||
|
||||
@router.post(
|
||||
"/status",
|
||||
response_model=ApiResponse[AuthStatusData],
|
||||
dependencies=[Depends(require_bearer_token)],
|
||||
)
|
||||
async def auth_status(
|
||||
payload: AuthStatusRequest,
|
||||
container: TradingContainer = Depends(get_container),
|
||||
) -> ApiResponse[AuthStatusData]:
|
||||
"""查询账号登录态
|
||||
|
||||
`refresh=true`(默认)时会真实访问站点探测——登录态过期没有可靠的本地判据,
|
||||
cookie 上的 expires 与服务端会话不是一回事,只能问站点。
|
||||
不需要这次网络往返时传 `refresh=false`,此时返回上一次探测的缓存结果
|
||||
(`logged_in` 为 null 表示从未探测过)。
|
||||
|
||||
`logged_in=false` 时,加购与下单接口会直接返回 5001,需要重新跑
|
||||
`scripts/login.py --site <site>` 后调用 `/api/auth/reload`。
|
||||
"""
|
||||
sites = [payload.site.value] if payload.site else list(container.auth_session.sites)
|
||||
statuses = []
|
||||
for site in sites:
|
||||
status = (
|
||||
await container.auth_session.check(site)
|
||||
if payload.refresh
|
||||
else container.auth_session.status(site)
|
||||
)
|
||||
statuses.append(_to_model(status))
|
||||
|
||||
return ApiResponse[AuthStatusData](
|
||||
success=True,
|
||||
msg="success",
|
||||
data=AuthStatusData(sites=statuses),
|
||||
code=0,
|
||||
)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/reload",
|
||||
response_model=ApiResponse[AuthReloadData],
|
||||
dependencies=[Depends(require_bearer_token)],
|
||||
)
|
||||
async def auth_reload(
|
||||
payload: AuthReloadRequest,
|
||||
container: TradingContainer = Depends(get_container),
|
||||
) -> ApiResponse[AuthReloadData]:
|
||||
"""重新从磁盘加载登录态并立即探测
|
||||
|
||||
人工跑完 `scripts/login.py` 后调用,避免为了换一套 cookie 重启整个服务。
|
||||
"""
|
||||
sites = [payload.site.value] if payload.site else list(container.auth_session.sites)
|
||||
reloaded = {site: container.auth_session.reload(site) for site in sites}
|
||||
statuses = [_to_model(await container.auth_session.check(site)) for site in sites]
|
||||
|
||||
return ApiResponse[AuthReloadData](
|
||||
success=True,
|
||||
msg="success",
|
||||
data=AuthReloadData(reloaded=reloaded, sites=statuses),
|
||||
code=0,
|
||||
)
|
||||
@@ -0,0 +1,27 @@
|
||||
"""交易服务健康检查路由"""
|
||||
from fastapi import APIRouter, Depends
|
||||
|
||||
from app.shared.api import ApiResponse, get_container
|
||||
from app.trading.container import TradingContainer
|
||||
from app.trading.models import TradingHealthData
|
||||
|
||||
router = APIRouter(tags=["health"])
|
||||
|
||||
|
||||
@router.get("/health", response_model=ApiResponse[TradingHealthData])
|
||||
async def health(
|
||||
container: TradingContainer = Depends(get_container),
|
||||
) -> ApiResponse[TradingHealthData]:
|
||||
"""交易服务健康状态
|
||||
|
||||
auth 给出两站账号登录态。这里只读缓存、不触发网络探测,避免健康检查被
|
||||
上游高频轮询时反复打站点;要实时结果请用 POST /api/auth/status。
|
||||
|
||||
注意 `logged_in=null` 表示服务启动后还没探测过,不等于未登录。
|
||||
"""
|
||||
return ApiResponse[TradingHealthData](
|
||||
success=True,
|
||||
msg="success",
|
||||
data=TradingHealthData(status="ok", auth=container.auth_session.status_all()),
|
||||
code=0,
|
||||
)
|
||||
@@ -0,0 +1,23 @@
|
||||
"""交易服务容器:集中管理交易侧服务实例,用于依赖注入
|
||||
|
||||
目前只有登录态会话。加购、下单、付款、订单监控与页面证据留痕的服务实例
|
||||
后续挂在这里,它们共享同一份 auth_session——同一个账号的写操作必须走同一条
|
||||
cookie 通道,且必须串行,不能各建各的客户端。
|
||||
"""
|
||||
from dataclasses import dataclass
|
||||
|
||||
from app.shared.config import Settings
|
||||
from app.trading.services.auth_session import AuthSession
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class TradingContainer:
|
||||
"""交易服务容器
|
||||
|
||||
与抓取容器最本质的差别不是字段多少,而是**这个进程有状态**:登录态、
|
||||
订单、页面证据都属于某个具体账号,因此交易服务只能单实例运行
|
||||
(或按账号分片),不能像抓取服务那样随意横向扩容。
|
||||
"""
|
||||
|
||||
settings: Settings
|
||||
auth_session: AuthSession
|
||||
@@ -0,0 +1,115 @@
|
||||
"""登录态相关的站点常量
|
||||
|
||||
与抓取用的 `scraping/core/site.py` / `rakuma_site.py` 分开:那两个模块描述的是
|
||||
「匿名抓取怎么拿到页面」,这里描述的是「怎么判断这套 cookie 还登录着」以及登录
|
||||
入口在哪。
|
||||
|
||||
登录态探针的选取原则:挑一个**未登录时行为明确可辨**的页面,而不是靠首页上有没有
|
||||
用户名这类会随改版漂移的文案。两站各自的信号(均为实测):
|
||||
|
||||
- 乐天:手机版购物车页始终返回 200,未登录时正文含「現在ログインしていません」,
|
||||
登录后该串消失。购物车页同时是加购结果的校验页,一页两用。
|
||||
- ラクマ:`/mypage` 未登录时 302 到 `/users/sign_in`,登录后停在 `/mypage`。
|
||||
用落地 URL 判断比翻文案稳。
|
||||
|
||||
每站的这组事实收在 `PROFILES` 里,`scripts/login.py`(起浏览器人工登录)与
|
||||
`AuthSession`(在 httpx 里探测)共用同一份,避免两处各写一遍判据后悄悄漂移。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Final
|
||||
|
||||
from app.shared import headers
|
||||
|
||||
# ---- 乐天市场 ----
|
||||
# 手机版购物车。加购走的是 sp.basket 集群,校验也用手机版,保持同一套指纹。
|
||||
RAKUTEN_CART_URL: Final = "https://sp.cart.step.rakuten.co.jp/cart"
|
||||
|
||||
# 登录入口。人工登录时打开这个地址,站点会在登录成功后跳回 my.rakuten.co.jp。
|
||||
RAKUTEN_LOGIN_URL: Final = "https://www.rakuten.co.jp/myrakuten/"
|
||||
|
||||
# 购物车页上表示「当前会话未登录」的文案。出现即判定登录态失效。
|
||||
RAKUTEN_LOGGED_OUT_MARKER: Final = "現在ログインしていません"
|
||||
|
||||
# ---- ラクマ ----
|
||||
RAKUMA_MYPAGE_URL: Final = "https://fril.jp/mypage"
|
||||
RAKUMA_LOGIN_URL: Final = "https://fril.jp/users/sign_in"
|
||||
|
||||
# 未登录时会被重定向到的登录页路径特征
|
||||
RAKUMA_SIGN_IN_PATH: Final = "/users/sign_in"
|
||||
|
||||
# ---- 登录态请求指纹 ----
|
||||
# 这两个 UA 必须与 scripts/login.py 起浏览器时用的一致:cookie 是在那个 UA 下拿到的,
|
||||
# 服务端再拿它发请求时换了 UA,可能触发站点的设备校验让登录态提前失效。
|
||||
#
|
||||
# 值与抓取侧当前相同,但**刻意不复用**抓取侧的常量:抓取 UA 是为绕反爬服务的,
|
||||
# 随时可能为了成功率被调整,那种调整不该波及已落盘的账号 cookie。
|
||||
RAKUTEN_USER_AGENT: Final = (
|
||||
"Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15 "
|
||||
"(KHTML, like Gecko) Version/17.5 Mobile/15E148 Safari/604.1"
|
||||
)
|
||||
RAKUMA_USER_AGENT: Final = (
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
|
||||
"(KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class SiteAuthProfile:
|
||||
"""一个站点的登录态配置:登录入口、探针、指纹与落盘文件名"""
|
||||
|
||||
name: str
|
||||
label: str
|
||||
login_url: str
|
||||
probe_url: str # 判断登录态是否仍有效的页面
|
||||
user_agent: str
|
||||
mobile: bool
|
||||
state_filename: str # storage_state 落盘文件名(放在 settings.auth_state_dir 下)
|
||||
|
||||
def headers(self) -> dict[str, str]:
|
||||
"""该站登录态请求用的完整导航请求头"""
|
||||
return headers.navigation_headers(self.user_agent, mobile=self.mobile)
|
||||
|
||||
|
||||
PROFILES: Final[dict[str, SiteAuthProfile]] = {
|
||||
"rakuten": SiteAuthProfile(
|
||||
name="rakuten",
|
||||
label="楽天市場",
|
||||
login_url=RAKUTEN_LOGIN_URL,
|
||||
probe_url=RAKUTEN_CART_URL,
|
||||
user_agent=RAKUTEN_USER_AGENT,
|
||||
mobile=True,
|
||||
state_filename="rakuten_state.json",
|
||||
),
|
||||
"rakuma": SiteAuthProfile(
|
||||
name="rakuma",
|
||||
label="ラクマ",
|
||||
login_url=RAKUMA_LOGIN_URL,
|
||||
probe_url=RAKUMA_MYPAGE_URL,
|
||||
user_agent=RAKUMA_USER_AGENT,
|
||||
mobile=False,
|
||||
state_filename="rakuma_state.json",
|
||||
),
|
||||
}
|
||||
|
||||
SITES: Final = tuple(PROFILES)
|
||||
|
||||
|
||||
def profile(site: str) -> SiteAuthProfile:
|
||||
"""取某站的登录态配置,未知站点直接报错"""
|
||||
try:
|
||||
return PROFILES[site]
|
||||
except KeyError:
|
||||
raise ValueError(f"未知站点:{site}") from None
|
||||
|
||||
|
||||
def is_logged_in(site: str, *, final_url: str, body: str) -> bool:
|
||||
"""按该站判据,从探针页的落地 URL 与正文判断是否仍登录着
|
||||
|
||||
两处共用:`AuthSession` 传 httpx 响应的 URL 与文本,`scripts/login.py` 传
|
||||
浏览器页面的 URL 与内容。判据只写一遍,两条链路不会各判各的。
|
||||
"""
|
||||
if site == "rakuten":
|
||||
return RAKUTEN_LOGGED_OUT_MARKER not in body
|
||||
return RAKUMA_SIGN_IN_PATH not in final_url
|
||||
@@ -0,0 +1,84 @@
|
||||
"""交易服务入口:FastAPI 应用创建与生命周期管理
|
||||
|
||||
与抓取服务(app/scraping/main.py)分成两个进程运行,理由不是「要不要登录」
|
||||
这一条,而是运行特性根本不同:
|
||||
|
||||
- 抓取无状态、可重试、可多开实例;这里的写操作**不可逆**,重复提交就是重复下单。
|
||||
- 登录态 cookie 全局唯一,订单监控是常驻轮询;多开实例会让同一个账号被多个
|
||||
进程并发操作,也会让轮询重复触发。
|
||||
- 抓取被限速最多是慢,账号被风控是封号;两者不该共用出口 IP 与请求节奏。
|
||||
|
||||
因此本服务只能单实例运行(或按账号分片),扩容靠抓取服务那一侧。
|
||||
|
||||
当前只提供登录态的查询与重载;加购、下单、付款与订单监控在此基础上叠加。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
from fastapi import FastAPI
|
||||
|
||||
from app.shared.api import register_exception_handlers
|
||||
from app.shared.config import get_settings
|
||||
from app.shared.logging_setup import configure_logging
|
||||
from app.trading.api.routes.auth import router as auth_router
|
||||
from app.trading.api.routes.health import router as health_router
|
||||
from app.trading.container import TradingContainer
|
||||
from app.trading.services.auth_session import AuthSession
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def build_container() -> TradingContainer:
|
||||
"""构建交易服务容器,组装所有依赖"""
|
||||
settings = get_settings()
|
||||
return TradingContainer(settings=settings, auth_session=AuthSession(settings))
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
"""应用生命周期管理:启动时加载登录态,关闭时释放 HTTP 客户端"""
|
||||
container = build_container()
|
||||
app.state.container = container
|
||||
|
||||
configure_logging(container.settings)
|
||||
logger.info(
|
||||
"交易服务启动:%s:%s",
|
||||
container.settings.trading_host,
|
||||
container.settings.trading_port,
|
||||
)
|
||||
logger.info("当前环境:%s", container.settings.app_env)
|
||||
await container.auth_session.start()
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
await container.auth_session.close()
|
||||
|
||||
|
||||
def create_app() -> FastAPI:
|
||||
"""创建 FastAPI 应用实例,注册路由和异常处理器"""
|
||||
app = FastAPI(title="Rakuten Trading Service", lifespan=lifespan)
|
||||
app.include_router(health_router)
|
||||
app.include_router(auth_router)
|
||||
register_exception_handlers(app)
|
||||
return app
|
||||
|
||||
|
||||
app = create_app()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
|
||||
settings = get_settings()
|
||||
configure_logging(settings)
|
||||
uvicorn.run(
|
||||
"app.trading.main:app",
|
||||
host=settings.trading_host,
|
||||
port=settings.trading_port,
|
||||
log_config=None,
|
||||
timeout_keep_alive=120,
|
||||
# 单进程:登录态与后续的订单监控都不能有第二份
|
||||
workers=1,
|
||||
)
|
||||
@@ -0,0 +1,68 @@
|
||||
"""交易侧 API 数据模型:登录态、(后续的)加购、下单与订单监控
|
||||
|
||||
与抓取侧模型(app/scraping/models/scrape.py)分开的理由和服务本身拆开的理由
|
||||
一致:抓取模型描述的是「站点上有什么」,这里描述的是「我们对某个账号做了什么、
|
||||
现在处于哪一步」——后者有生命周期、有状态迁移,会持久化,不是一次请求的产物。
|
||||
|
||||
响应信封 `ApiResponse` 与抓取侧共用,在 app/shared/api.py。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from enum import StrEnum
|
||||
from typing import Any
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class AuthSite(StrEnum):
|
||||
"""支持登录的站点"""
|
||||
|
||||
RAKUTEN = "rakuten"
|
||||
RAKUMA = "rakuma"
|
||||
|
||||
|
||||
class AuthStatusRequest(BaseModel):
|
||||
"""登录态查询请求"""
|
||||
|
||||
site: AuthSite | None = None # 不传则返回两站
|
||||
refresh: bool = True # 是否真实打一次请求探测;false 时只读缓存
|
||||
|
||||
|
||||
class AuthSiteStatus(BaseModel):
|
||||
"""单站登录态"""
|
||||
|
||||
site: str
|
||||
state_file_exists: bool # 是否已跑过 scripts/login.py
|
||||
logged_in: bool | None # None 表示尚未探测
|
||||
checked_age_seconds: float | None = None
|
||||
detail: str = ""
|
||||
|
||||
|
||||
class AuthStatusData(BaseModel):
|
||||
"""登录态查询响应"""
|
||||
|
||||
sites: list[AuthSiteStatus] = Field(default_factory=list)
|
||||
|
||||
|
||||
class AuthReloadRequest(BaseModel):
|
||||
"""重新加载登录态请求(人工登录完成后调用,免重启服务)"""
|
||||
|
||||
site: AuthSite | None = None # 不传则两站都重载
|
||||
|
||||
|
||||
class AuthReloadData(BaseModel):
|
||||
"""重新加载登录态响应"""
|
||||
|
||||
reloaded: dict[str, int] = Field(default_factory=dict) # site -> cookie 条数
|
||||
sites: list[AuthSiteStatus] = Field(default_factory=list)
|
||||
|
||||
|
||||
class TradingHealthData(BaseModel):
|
||||
"""交易服务健康检查响应数据
|
||||
|
||||
只读缓存的登录态,不触发网络探测——健康检查会被高频轮询,实时结果请用
|
||||
POST /api/auth/status。
|
||||
"""
|
||||
|
||||
status: str
|
||||
auth: dict[str, Any] = Field(default_factory=dict)
|
||||
@@ -0,0 +1,260 @@
|
||||
"""登录态会话:持有已登录账号的 cookie,供加购与下单链路使用
|
||||
|
||||
与抓取链路(app/scraping 的 site_session / rakuma_session)不只是分模块,
|
||||
而是分进程运行,原因:
|
||||
|
||||
- 抓取是**匿名**的,cookie 只用来过 Akamai 限速,丢了重新预热即可,无状态可言。
|
||||
- 加购下单必须**带账号**,cookie 一旦失效不能自动恢复——乐天与 ラクマ 登录都有
|
||||
reCAPTCHA 与设备验证,只能由人重新登录一次。因此这里的失效处理是「明确报错让
|
||||
上游停下」,而不是像抓取那样静默重试。
|
||||
- 这份登录态在整个系统里只能有一份。跟抓取同进程的话,抓取一扩容就会复制出 N 份
|
||||
登录态与 N 个订单轮询,同一个账号被并发操作。
|
||||
|
||||
登录态来源是 Playwright 的 storage_state:由 `scripts/login.py` 起一个有头浏览器
|
||||
让人工登录一次后落盘,本服务只读取、不生产。账号密码不经过本服务,也不写日志。
|
||||
|
||||
cookie 会被同时喂给两处:
|
||||
- httpx 客户端 —— 加购这类纯表单提交走 HTTP 更快
|
||||
- Playwright context —— 下单确认页有 JS 参与,必要时用浏览器走完
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
from app.shared.config import Settings
|
||||
from app.shared.errors import NotLoggedInError, UpstreamRequestError
|
||||
from app.trading.core import auth_site
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class AuthStatus:
|
||||
"""一个站点的登录态快照"""
|
||||
|
||||
site: str
|
||||
state_file_exists: bool
|
||||
logged_in: bool | None # None 表示尚未探测过
|
||||
checked_at: float | None = None
|
||||
detail: str = ""
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"site": self.site,
|
||||
"state_file_exists": self.state_file_exists,
|
||||
"logged_in": self.logged_in,
|
||||
"checked_age_seconds": (
|
||||
round(time.monotonic() - self.checked_at, 1) if self.checked_at else None
|
||||
),
|
||||
"detail": self.detail,
|
||||
}
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class _SiteAuth:
|
||||
"""一个站点的登录通道:独立的 httpx 客户端与登录态缓存"""
|
||||
|
||||
name: str
|
||||
client: httpx.AsyncClient
|
||||
lock: asyncio.Lock = field(default_factory=asyncio.Lock)
|
||||
logged_in: bool | None = None
|
||||
checked_at: float | None = None
|
||||
detail: str = ""
|
||||
|
||||
|
||||
class AuthSession:
|
||||
"""持有两站登录态的会话
|
||||
|
||||
职责边界:只负责「有没有登录态、cookie 是什么、还有效吗」,
|
||||
具体加购/下单的业务请求由各自的 client 组装后借这里的 HTTP 客户端发出。
|
||||
"""
|
||||
|
||||
def __init__(self, settings: Settings):
|
||||
self._settings = settings
|
||||
self._sites: dict[str, _SiteAuth] = {}
|
||||
|
||||
# ---- 生命周期 ----
|
||||
|
||||
async def start(self) -> None:
|
||||
"""为两站创建带登录 cookie 的 HTTP 客户端
|
||||
|
||||
登录态文件不存在时同样创建客户端(只是没有 cookie),这样 /health 与
|
||||
/api/auth/status 能如实回报「未登录」,而不是整个服务起不来。
|
||||
"""
|
||||
for name, profile in auth_site.PROFILES.items():
|
||||
if name in self._sites:
|
||||
continue
|
||||
client = httpx.AsyncClient(
|
||||
headers=profile.headers(),
|
||||
timeout=self._settings.request_timeout_seconds,
|
||||
follow_redirects=True,
|
||||
proxy=self._settings.httpx_proxy,
|
||||
http2=True,
|
||||
)
|
||||
self._sites[name] = _SiteAuth(name=name, client=client)
|
||||
loaded = self._load_cookies(name)
|
||||
logger.info("登录通道已就绪:site=%s cookies=%s", name, loaded)
|
||||
|
||||
async def close(self) -> None:
|
||||
for auth in self._sites.values():
|
||||
try:
|
||||
await auth.client.aclose()
|
||||
except Exception:
|
||||
logger.debug("关闭登录 HTTP 客户端失败:site=%s", auth.name, exc_info=True)
|
||||
self._sites.clear()
|
||||
|
||||
# ---- 登录态文件 ----
|
||||
|
||||
def state_path(self, site: str) -> Path:
|
||||
"""某站点 storage_state 文件的落盘路径"""
|
||||
return self._settings.auth_state_path / auth_site.profile(site).state_filename
|
||||
|
||||
def _load_cookies(self, site: str) -> int:
|
||||
"""把 storage_state 里的 cookie 灌进该站的 httpx 客户端,返回条数"""
|
||||
path = self.state_path(site)
|
||||
if not path.exists():
|
||||
return 0
|
||||
|
||||
try:
|
||||
state = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError) as exc:
|
||||
logger.warning("登录态文件读取失败:site=%s path=%s err=%s", site, path, exc)
|
||||
return 0
|
||||
|
||||
auth = self._sites[site]
|
||||
auth.client.cookies.clear()
|
||||
count = 0
|
||||
for cookie in state.get("cookies", []):
|
||||
name = cookie.get("name")
|
||||
value = cookie.get("value")
|
||||
if not name or value is None:
|
||||
continue
|
||||
auth.client.cookies.set(
|
||||
name,
|
||||
value,
|
||||
domain=cookie.get("domain") or "",
|
||||
path=cookie.get("path") or "/",
|
||||
)
|
||||
count += 1
|
||||
return count
|
||||
|
||||
def reload(self, site: str) -> int:
|
||||
"""重新从磁盘加载登录态(人工登录完成后调用),返回 cookie 条数"""
|
||||
auth = self._sites.get(site)
|
||||
if auth is None:
|
||||
raise ValueError(f"未知站点:{site}")
|
||||
count = self._load_cookies(site)
|
||||
auth.logged_in = None
|
||||
auth.checked_at = None
|
||||
auth.detail = "已重新加载登录态,尚未探测"
|
||||
logger.info("登录态已重新加载:site=%s cookies=%s", site, count)
|
||||
return count
|
||||
|
||||
# ---- 登录态探测 ----
|
||||
|
||||
async def check(self, site: str) -> AuthStatus:
|
||||
"""探测某站登录态是否仍然有效
|
||||
|
||||
每次都真实打一次请求——登录态过期没有可靠的本地判据(cookie 的 expires
|
||||
与服务端会话不是一回事),只能问站点。
|
||||
"""
|
||||
auth = self._require_site(site)
|
||||
async with auth.lock:
|
||||
try:
|
||||
if site == "rakuten":
|
||||
logged_in, detail = await self._check_rakuten(auth)
|
||||
else:
|
||||
logged_in, detail = await self._check_rakuma(auth)
|
||||
except httpx.HTTPError as exc:
|
||||
raise UpstreamRequestError(
|
||||
f"登录态探测请求失败:site={site} err={type(exc).__name__}: {exc}"
|
||||
) from exc
|
||||
|
||||
auth.logged_in = logged_in
|
||||
auth.checked_at = time.monotonic()
|
||||
auth.detail = detail
|
||||
logger.info("登录态探测:site=%s logged_in=%s detail=%s", site, logged_in, detail)
|
||||
return self.status(site)
|
||||
|
||||
async def _check_rakuten(self, auth: _SiteAuth) -> tuple[bool, str]:
|
||||
"""购物车页含「現在ログインしていません」即未登录"""
|
||||
response = await auth.client.get(auth_site.PROFILES["rakuten"].probe_url)
|
||||
if response.status_code >= 400:
|
||||
return False, f"购物车页返回 status {response.status_code}"
|
||||
if not auth_site.is_logged_in(
|
||||
"rakuten", final_url=str(response.url), body=response.text
|
||||
):
|
||||
return False, "购物车页显示未登录"
|
||||
return True, "购物车页未出现未登录标记"
|
||||
|
||||
async def _check_rakuma(self, auth: _SiteAuth) -> tuple[bool, str]:
|
||||
"""/mypage 被重定向到 /users/sign_in 即未登录"""
|
||||
response = await auth.client.get(auth_site.PROFILES["rakuma"].probe_url)
|
||||
if response.status_code >= 400:
|
||||
return False, f"mypage 返回 status {response.status_code}"
|
||||
if not auth_site.is_logged_in(
|
||||
"rakuma", final_url=str(response.url), body=response.text
|
||||
):
|
||||
return False, "mypage 被重定向至登录页"
|
||||
return True, "mypage 正常返回"
|
||||
|
||||
async def require_logged_in(self, site: str) -> None:
|
||||
"""确保某站处于登录态,否则抛错
|
||||
|
||||
加购与下单前的统一入口。登录态失效时不做任何自动恢复尝试——两站登录都需要
|
||||
人工过验证码,只能让上游停下来重新跑一次 scripts/login.py。
|
||||
"""
|
||||
status = await self.check(site)
|
||||
if not status.logged_in:
|
||||
raise NotLoggedInError(site=site, detail=status.detail)
|
||||
|
||||
# ---- 对外访问 ----
|
||||
|
||||
@property
|
||||
def sites(self) -> tuple[str, ...]:
|
||||
"""已初始化的站点名"""
|
||||
return tuple(self._sites)
|
||||
|
||||
def client(self, site: str) -> httpx.AsyncClient:
|
||||
"""取该站带登录 cookie 的 HTTP 客户端"""
|
||||
return self._require_site(site).client
|
||||
|
||||
def cookies_for_browser(self, site: str) -> list[dict[str, Any]]:
|
||||
"""导出 cookie 供 Playwright context 使用"""
|
||||
path = self.state_path(site)
|
||||
if not path.exists():
|
||||
return []
|
||||
try:
|
||||
state = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError):
|
||||
return []
|
||||
return list(state.get("cookies", []))
|
||||
|
||||
def status(self, site: str) -> AuthStatus:
|
||||
"""该站登录态快照(不触发探测,用缓存结果)"""
|
||||
auth = self._require_site(site)
|
||||
return AuthStatus(
|
||||
site=site,
|
||||
state_file_exists=self.state_path(site).exists(),
|
||||
logged_in=auth.logged_in,
|
||||
checked_at=auth.checked_at,
|
||||
detail=auth.detail,
|
||||
)
|
||||
|
||||
def status_all(self) -> dict[str, dict[str, Any]]:
|
||||
"""两站登录态快照,供健康检查展示"""
|
||||
return {name: self.status(name).to_dict() for name in self._sites}
|
||||
|
||||
def _require_site(self, site: str) -> _SiteAuth:
|
||||
auth = self._sites.get(site)
|
||||
if auth is None:
|
||||
raise ValueError(f"未知站点或登录会话未初始化:{site}")
|
||||
return auth
|
||||
Reference in New Issue
Block a user