拆分抓取与交易服务
把需要账号登录态的链路从抓取服务里拆出成独立进程。分界线不是「要不要登录」, 而是抓取无状态、幂等、可多开实例,而交易的写操作不可逆、登录态全局唯一、 订单监控是常驻轮询——同进程时抓取一扩容就会复制出 N 份登录态与 N 个轮询, 同一账号会被并发操作。 - app/shared:配置、错误码、日志、ApiResponse 信封 + Bearer 鉴权 + 异常处理器、 导航请求头构造器 - app/scraping:站点常量、会话、解析器与 10 个抓取接口,:31107,可多开 - app/trading:登录态查询/重载与健康检查,:31108,只能单实例 - 依赖方向锁为 scraping→shared、trading→shared,两侧互不 import; tests/test_architecture.py 用 AST 检查 import 并校验两个 app 的路径不串 - 登录态 UA 在 trading 独立持有:与抓取 UA 值相同但变更理由不同,抓取 UA 为绕 反爬可随时调整,登录 UA 一改可能触发设备校验使已落盘 cookie 失效 - scripts/login.py 与 AuthSession 共用 auth_site.PROFILES 与 is_logged_in,判据只写一遍 - 同一镜像两个启动命令,交易容器覆盖 command 并设 RAKUTEN_HEALTH_PORT 同时带上此前未提交的 ラクマ 分类接口与登录态基础设施。 验证:239 个离线用例全绿;两个入口真实启动,/health 与鉴权正常。 未验证:真实探测登录态(当前开发机无外网,对站点的连接全部超时)。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,175 @@
|
||||
"""两个入口共用的 HTTP 层:响应信封、鉴权依赖、异常处理器
|
||||
|
||||
抓取服务与交易服务是两个独立进程(见 README「两个部署单元」),但对外契约必须
|
||||
一致:同一套 `ApiResponse` 信封、同一份错误码表、同一个 Bearer token。共用的部分
|
||||
集中在这里,两侧的差异只体现在各自注册的路由与容器。
|
||||
|
||||
这里刻意不放任何站点或业务知识——`get_container` 不标注具体容器类型,shared 因此
|
||||
不需要认识 `ScrapingContainer` / `TradingContainer`,避免共用层反向依赖两侧。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import secrets
|
||||
from typing import Any, Generic, TypeVar
|
||||
|
||||
from fastapi import Depends, FastAPI, Request
|
||||
from fastapi.exceptions import RequestValidationError
|
||||
from fastapi.responses import JSONResponse
|
||||
from pydantic import BaseModel, ValidationError
|
||||
from starlette.exceptions import HTTPException as StarletteHTTPException
|
||||
|
||||
from app.shared.errors import AppError, AuthenticationError
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
T = TypeVar("T")
|
||||
|
||||
|
||||
class ApiResponse(BaseModel, Generic[T]):
|
||||
"""统一 API 响应格式"""
|
||||
|
||||
success: bool
|
||||
msg: str
|
||||
data: T | None = None
|
||||
code: int
|
||||
|
||||
|
||||
# ---- 依赖注入 ----
|
||||
|
||||
|
||||
def get_container(request: Request) -> Any:
|
||||
"""从请求中获取服务容器
|
||||
|
||||
返回类型故意留成 Any:抓取侧与交易侧的容器结构不同,由各自路由标注具体类型。
|
||||
"""
|
||||
return request.app.state.container
|
||||
|
||||
|
||||
def require_bearer_token(
|
||||
request: Request,
|
||||
container: Any = Depends(get_container),
|
||||
) -> None:
|
||||
"""Bearer Token 鉴权依赖
|
||||
|
||||
从请求头 Authorization 中提取 Bearer Token,
|
||||
与服务端配置的 token 做安全比较(使用 secrets.compare_digest 防止时序攻击)。
|
||||
"""
|
||||
authorization = request.headers.get("Authorization")
|
||||
if not authorization:
|
||||
logger.warning("鉴权失败:缺少 Authorization 请求头")
|
||||
raise AuthenticationError("Missing Authorization header")
|
||||
|
||||
token = authorization.replace("Bearer ", "", 1).strip()
|
||||
if token == authorization:
|
||||
logger.warning("鉴权失败:Authorization scheme 非 Bearer")
|
||||
raise AuthenticationError("Invalid Authorization scheme")
|
||||
if not token:
|
||||
logger.warning("鉴权失败:Bearer token 为空")
|
||||
raise AuthenticationError("Invalid token")
|
||||
|
||||
if not secrets.compare_digest(token, container.settings.bearer_token):
|
||||
logger.warning("鉴权失败:token 不匹配")
|
||||
raise AuthenticationError("Invalid token")
|
||||
|
||||
|
||||
# ---- 异常处理 ----
|
||||
|
||||
|
||||
def _format_validation_msg(errors: list[dict]) -> str:
|
||||
"""将校验错误整理为便于前端展示的消息。"""
|
||||
if not errors:
|
||||
return "Validation error"
|
||||
|
||||
messages: list[str] = []
|
||||
for error in errors:
|
||||
loc = ".".join(str(part) for part in error.get("loc", []) if part != "body")
|
||||
msg = str(error.get("msg", "Validation error"))
|
||||
messages.append(f"{loc}: {msg}" if loc else msg)
|
||||
|
||||
return "; ".join(messages)
|
||||
|
||||
|
||||
def jsonable_errors(errors: list[dict]) -> list[dict]:
|
||||
"""剔除校验错误里不可 JSON 序列化的 ctx(如原始异常对象)"""
|
||||
return [{key: value for key, value in error.items() if key != "ctx"} for error in errors]
|
||||
|
||||
|
||||
def register_exception_handlers(app: FastAPI) -> None:
|
||||
"""给应用挂上全套异常处理器
|
||||
|
||||
两个入口都调用它,保证抓取失败与下单失败返回的错误结构完全一致,
|
||||
上游只需要按 code 分支,不必区分是哪个服务回的。
|
||||
"""
|
||||
|
||||
@app.exception_handler(AppError)
|
||||
async def app_error_handler(_: Request, exc: AppError) -> JSONResponse:
|
||||
"""业务异常处理器:返回结构化的错误响应"""
|
||||
return JSONResponse(
|
||||
status_code=exc.status_code,
|
||||
content=ApiResponse[None](
|
||||
success=False,
|
||||
code=exc.err_code,
|
||||
msg=exc.message,
|
||||
data=None,
|
||||
).model_dump(),
|
||||
headers=exc.headers,
|
||||
)
|
||||
|
||||
@app.exception_handler(RequestValidationError)
|
||||
async def validation_error_handler(_: Request, exc: RequestValidationError) -> JSONResponse:
|
||||
"""请求参数校验异常处理器"""
|
||||
errors = exc.errors()
|
||||
return JSONResponse(
|
||||
status_code=422,
|
||||
content=ApiResponse[object](
|
||||
success=False,
|
||||
msg=_format_validation_msg(errors),
|
||||
data=jsonable_errors(errors),
|
||||
code=1002,
|
||||
).model_dump(),
|
||||
)
|
||||
|
||||
@app.exception_handler(ValidationError)
|
||||
async def pydantic_validation_error_handler(_: Request, exc: ValidationError) -> JSONResponse:
|
||||
"""Pydantic 模型校验异常处理器"""
|
||||
errors = exc.errors()
|
||||
return JSONResponse(
|
||||
status_code=422,
|
||||
content=ApiResponse[object](
|
||||
success=False,
|
||||
msg=_format_validation_msg(errors),
|
||||
data=jsonable_errors(errors),
|
||||
code=1002,
|
||||
).model_dump(),
|
||||
)
|
||||
|
||||
@app.exception_handler(StarletteHTTPException)
|
||||
async def http_exception_handler(_: Request, exc: StarletteHTTPException) -> JSONResponse:
|
||||
"""HTTP 异常处理器(404、500 等)"""
|
||||
status_code = int(getattr(exc, "status_code", 500) or 500)
|
||||
err_code = 1404 if status_code == 404 else 1500
|
||||
return JSONResponse(
|
||||
status_code=status_code,
|
||||
content=ApiResponse[None](
|
||||
success=False,
|
||||
msg=str(getattr(exc, "detail", "HTTP error")),
|
||||
data=None,
|
||||
code=err_code,
|
||||
).model_dump(),
|
||||
headers=getattr(exc, "headers", None),
|
||||
)
|
||||
|
||||
@app.exception_handler(Exception)
|
||||
async def unhandled_exception_handler(_: Request, exc: Exception) -> JSONResponse:
|
||||
"""兜底异常处理器:捕获所有未处理的异常"""
|
||||
logger.exception("未处理异常:%s", exc)
|
||||
return JSONResponse(
|
||||
status_code=500,
|
||||
content=ApiResponse[None](
|
||||
success=False,
|
||||
msg="Internal server error",
|
||||
data=None,
|
||||
code=1500,
|
||||
).model_dump(),
|
||||
)
|
||||
@@ -0,0 +1,150 @@
|
||||
"""应用配置:通过环境变量和 .env 文件加载所有配置项
|
||||
|
||||
配置项统一使用 RAKUTEN_ 前缀,例如 RAKUTEN_APP_PORT=31107。
|
||||
支持 .env 文件自动加载。
|
||||
|
||||
抓取服务与交易服务是两个进程,但共用这一个 Settings 类:两边都要日志、代理、
|
||||
超时与同一个 Bearer token,拆成两份配置只会让部署时多维护一套。下面按
|
||||
「通用 / 仅抓取 / 仅交易」分区标注,各进程只读自己那部分。
|
||||
"""
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
from typing import Literal
|
||||
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
BASE_DIR = Path(__file__).resolve().parent.parent.parent
|
||||
|
||||
# 乐天市场首页。这里不 import app.scraping —— shared 不能反向依赖两侧任何一方,
|
||||
# 否则交易服务也会被迫加载整套抓取模块。
|
||||
DEFAULT_HOME_URL = "https://www.rakuten.co.jp/"
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
"""应用全局配置
|
||||
|
||||
所有配置项均可通过环境变量覆盖,前缀为 RAKUTEN_。
|
||||
例如:RAKUTEN_APP_PORT=31107 对应 app_port 配置项。
|
||||
"""
|
||||
|
||||
model_config = SettingsConfigDict(
|
||||
env_file=str(BASE_DIR / ".env"),
|
||||
env_file_encoding="utf-8",
|
||||
env_prefix="RAKUTEN_",
|
||||
extra="ignore",
|
||||
)
|
||||
|
||||
# ---- 服务基本配置(通用)----
|
||||
app_name: str = "Rakuten Scraper Service"
|
||||
app_env: Literal["dev", "prod", "test"] = "dev"
|
||||
|
||||
# 抓取服务监听地址;交易服务用下面的 trading_host / trading_port
|
||||
app_host: str = "0.0.0.0"
|
||||
app_port: int = 31107
|
||||
|
||||
# 交易服务监听地址。两个服务同机部署时端口必须错开;交易服务只能单实例,
|
||||
# 不要在它前面挂多副本负载均衡。
|
||||
trading_host: str = "0.0.0.0"
|
||||
trading_port: int = 31108
|
||||
|
||||
# ---- 日志配置 ----
|
||||
log_level: str = "INFO"
|
||||
log_to_file: bool | None = None # None 表示根据环境自动决定
|
||||
log_dir: str = "logs"
|
||||
log_rotation: str = "100 MB"
|
||||
log_retention: str = "14 days"
|
||||
log_compression: str = "zip"
|
||||
log_format: str = "{time:YYYY-MM-DD HH:mm:ss} {level} {message}"
|
||||
log_enqueue: bool = True
|
||||
|
||||
# ---- 鉴权配置(通用:两个服务共用同一个对外 token)----
|
||||
bearer_token: str = "REPLACE_WITH_TOKEN_32CHARS"
|
||||
|
||||
# ---- HTTP 抓取配置(仅抓取服务)----
|
||||
request_timeout_seconds: float = 30.0
|
||||
max_site_concurrency: int = 8 # 对站点的最大并发请求数
|
||||
http_max_attempts: int = 3 # 单次抓取的最大尝试次数(含首次)
|
||||
session_ttl_seconds: float = 1800.0 # Akamai cookie 会话最长复用时长,超时后重新预热
|
||||
|
||||
# ---- 浏览器兜底配置(仅抓取服务)----
|
||||
# 纯 HTTP 被 Akamai 拦截时,用 Playwright 打开页面取回 cookie 再回灌给
|
||||
# HTTP 客户端重试。日常流量不会触发;未安装 playwright 时自动降级为不兜底。
|
||||
browser_fallback_enabled: bool = True
|
||||
browser_headless: bool | None = None # None 表示根据环境自动决定
|
||||
browser_channel: str | None = None # 例如 chrome;留空使用 bundled chromium
|
||||
browser_launch_timeout_seconds: float = 60.0
|
||||
browser_nav_timeout_seconds: float = 60.0
|
||||
|
||||
# ---- 代理配置(通用,可选,用于日本 IP)----
|
||||
# 两个服务同机部署时通常各配各的:抓取高频匿名,出口 IP 被限速换掉即可;
|
||||
# 交易带账号,出口 IP 频繁漂移反而会触发风控。
|
||||
proxy_server: str | None = None
|
||||
proxy_username: str | None = None
|
||||
proxy_password: str | None = None
|
||||
|
||||
# ---- 登录态与下单配置(仅交易服务)----
|
||||
# 人工登录一次后落盘的 Playwright storage_state 目录(相对项目根目录)。
|
||||
# 目录里是可直接冒充账号的 cookie,务必不要提交到版本库。
|
||||
auth_state_dir: str = ".auth"
|
||||
# 下单金额上限(日元)。实际应付金额超过该值时拒绝提交,防止解析出错或
|
||||
# 页面改版导致买到远超预期的订单。设为 0 表示不设上限(不建议)。
|
||||
order_max_total_yen: int = 30000
|
||||
|
||||
# ---- 目标站点 ----
|
||||
home_url: str = DEFAULT_HOME_URL
|
||||
|
||||
@property
|
||||
def browser_headless_effective(self) -> bool:
|
||||
"""浏览器无头模式:显式配置优先,否则开发环境使用有头模式方便调试"""
|
||||
if self.browser_headless is not None:
|
||||
return self.browser_headless
|
||||
return self.app_env != "dev"
|
||||
|
||||
@property
|
||||
def log_to_file_effective(self) -> bool:
|
||||
"""是否写日志文件:显式配置优先,否则生产环境默认写文件"""
|
||||
if self.log_to_file is not None:
|
||||
return self.log_to_file
|
||||
return self.app_env == "prod"
|
||||
|
||||
@property
|
||||
def playwright_proxy(self) -> dict[str, str] | None:
|
||||
"""构建 Playwright 代理配置字典"""
|
||||
if not self.proxy_server:
|
||||
return None
|
||||
|
||||
proxy: dict[str, str] = {"server": self.proxy_server}
|
||||
if self.proxy_username:
|
||||
proxy["username"] = self.proxy_username
|
||||
if self.proxy_password:
|
||||
proxy["password"] = self.proxy_password
|
||||
return proxy
|
||||
|
||||
@property
|
||||
def httpx_proxy(self) -> str | None:
|
||||
"""构建 httpx 代理 URL(含认证信息)"""
|
||||
if not self.proxy_server:
|
||||
return None
|
||||
if not self.proxy_username:
|
||||
return self.proxy_server
|
||||
|
||||
scheme, _, rest = self.proxy_server.partition("://")
|
||||
if not rest:
|
||||
return self.proxy_server
|
||||
credentials = f"{self.proxy_username}:{self.proxy_password or ''}"
|
||||
return f"{scheme}://{credentials}@{rest}"
|
||||
|
||||
@property
|
||||
def auth_state_path(self) -> Path:
|
||||
"""登录态目录的绝对路径,不存在时创建"""
|
||||
path = Path(self.auth_state_dir)
|
||||
if not path.is_absolute():
|
||||
path = BASE_DIR / path
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
return path
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def get_settings() -> Settings:
|
||||
"""获取全局配置单例"""
|
||||
return Settings()
|
||||
@@ -0,0 +1,171 @@
|
||||
"""应用异常定义:所有业务异常均继承自 AppError
|
||||
|
||||
错误码规范:
|
||||
- 1xxx: 请求/鉴权错误
|
||||
- 2xxx: 抓取资源错误
|
||||
- 3xxx: 反爬/上游阻断相关错误
|
||||
- 4xxx: 页面解析错误
|
||||
- 5xxx: 加购/下单错误(需要账号登录态的写操作)
|
||||
"""
|
||||
|
||||
|
||||
class AppError(Exception):
|
||||
"""应用基础异常,携带错误码、HTTP 状态码和是否可重试信息"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
message: str,
|
||||
code: str,
|
||||
err_code: int,
|
||||
retryable: bool = False,
|
||||
status_code: int = 400,
|
||||
headers: dict[str, str] | None = None,
|
||||
):
|
||||
super().__init__(message)
|
||||
self.message = message
|
||||
self.code = code
|
||||
self.err_code = err_code
|
||||
self.retryable = retryable
|
||||
self.status_code = status_code
|
||||
self.headers = headers
|
||||
|
||||
|
||||
class AuthenticationError(AppError):
|
||||
"""鉴权失败(Bearer Token 无效或缺失)"""
|
||||
|
||||
def __init__(self, message: str = "Invalid credentials"):
|
||||
super().__init__(
|
||||
message=message,
|
||||
code="AUTH_INVALID",
|
||||
err_code=1001,
|
||||
retryable=False,
|
||||
status_code=401,
|
||||
headers={"WWW-Authenticate": "Bearer"},
|
||||
)
|
||||
|
||||
|
||||
class InvalidRequestError(AppError):
|
||||
"""请求参数不合法(如 URL 非乐天站点、缺少必填标识)"""
|
||||
|
||||
def __init__(self, message: str = "Invalid request"):
|
||||
super().__init__(message=message, code="INVALID_REQUEST", err_code=1003, retryable=False)
|
||||
|
||||
|
||||
class ResourceBusyError(AppError):
|
||||
"""抓取资源繁忙(等待并发槽位超时)"""
|
||||
|
||||
def __init__(self, message: str = "Timed out while waiting for a scrape slot"):
|
||||
super().__init__(message=message, code="RESOURCE_BUSY", err_code=2002, retryable=True)
|
||||
|
||||
|
||||
class UpstreamRequestError(AppError):
|
||||
"""上游请求失败(网络异常、超时、5xx)"""
|
||||
|
||||
def __init__(self, message: str = "The upstream request failed"):
|
||||
super().__init__(message=message, code="UPSTREAM_ERROR", err_code=3001, retryable=True)
|
||||
|
||||
|
||||
class UpstreamBlockedError(AppError):
|
||||
"""上游请求被反爬阻断(Akamai 挑战页 / 403 / 页面缺少渲染数据)"""
|
||||
|
||||
def __init__(self, message: str = "The upstream request was blocked"):
|
||||
super().__init__(message=message, code="UPSTREAM_BLOCKED", err_code=3002, retryable=True)
|
||||
|
||||
|
||||
class ItemNotFoundError(AppError):
|
||||
"""商品不存在或已下架(详情页 404)"""
|
||||
|
||||
def __init__(self, message: str = "Item not found"):
|
||||
super().__init__(
|
||||
message=message,
|
||||
code="ITEM_NOT_FOUND",
|
||||
err_code=4004,
|
||||
retryable=False,
|
||||
status_code=404,
|
||||
)
|
||||
|
||||
|
||||
class ScrapeParseError(AppError):
|
||||
"""页面解析失败"""
|
||||
|
||||
def __init__(self, message: str = "Failed to parse the target page"):
|
||||
super().__init__(message=message, code="PARSE_ERROR", err_code=4001, retryable=False)
|
||||
|
||||
|
||||
class OffIchibaRedirectError(AppError):
|
||||
"""商品页跳转到了市场之外的乐天官方子站
|
||||
|
||||
楽天ブックス(book → books.rakuten.co.jp)、ビックカメラ
|
||||
(biccamera → biccamera.rakuten.co.jp)等官方旗舰店有各自独立的站点与
|
||||
页面结构,不返回市场统一模板,本服务的详情解析不适用。
|
||||
|
||||
单独成一类错误是为了让上游能把这些商品路由到别处,而不是当成被反爬拦截去重试。
|
||||
"""
|
||||
|
||||
def __init__(self, requested_url: str, final_url: str):
|
||||
super().__init__(
|
||||
message=(
|
||||
f"商品页跳转至乐天市场以外的站点,当前不支持解析:"
|
||||
f"{requested_url} -> {final_url}"
|
||||
),
|
||||
code="OFF_ICHIBA_REDIRECT",
|
||||
err_code=4002,
|
||||
retryable=False,
|
||||
)
|
||||
self.requested_url = requested_url
|
||||
self.final_url = final_url
|
||||
|
||||
|
||||
class NotLoggedInError(AppError):
|
||||
"""账号登录态缺失或已失效
|
||||
|
||||
加购与下单必须带已登录的账号会话。两站登录都要过 reCAPTCHA / 设备验证,
|
||||
无法自动恢复,因此这里明确标记 retryable=False,让上游停下来走一次
|
||||
`scripts/login.py` 重新人工登录,而不是原地重试。
|
||||
"""
|
||||
|
||||
def __init__(self, site: str, detail: str = ""):
|
||||
suffix = f"({detail})" if detail else ""
|
||||
super().__init__(
|
||||
message=(
|
||||
f"{site} 账号未登录或登录态已失效{suffix},"
|
||||
f"请运行 scripts/login.py --site {site} 重新登录"
|
||||
),
|
||||
code="NOT_LOGGED_IN",
|
||||
err_code=5001,
|
||||
retryable=False,
|
||||
status_code=401,
|
||||
)
|
||||
self.site = site
|
||||
self.detail = detail
|
||||
|
||||
|
||||
class CartOperationError(AppError):
|
||||
"""加购失败
|
||||
|
||||
站点对加购请求几乎不返回结构化错误:缺必填选项、SKU 已售罄、商品下架
|
||||
都可能返回 200 并把用户导回商品页。因此判定依据是「加购后购物车里有没有
|
||||
这件商品」,而不是 HTTP 状态码。
|
||||
"""
|
||||
|
||||
def __init__(self, message: str = "加入购物车失败"):
|
||||
super().__init__(message=message, code="CART_FAILED", err_code=5002, retryable=False)
|
||||
|
||||
|
||||
class OrderOperationError(AppError):
|
||||
"""下单流程失败(确认页解析不出、金额校验不通过、提交被拒等)"""
|
||||
|
||||
def __init__(self, message: str = "下单失败"):
|
||||
super().__init__(message=message, code="ORDER_FAILED", err_code=5003, retryable=False)
|
||||
|
||||
|
||||
class OrderGuardError(AppError):
|
||||
"""下单安全闸门未通过
|
||||
|
||||
真实付款不可逆,因此把「调用方没有显式确认」「实际金额超出上限」这类拦截
|
||||
单独成一类错误,与站点侧失败区分开——前者是本服务主动拒绝,重试无意义,
|
||||
需要调用方修改入参后再来。
|
||||
"""
|
||||
|
||||
def __init__(self, message: str):
|
||||
super().__init__(message=message, code="ORDER_GUARD", err_code=5004, retryable=False)
|
||||
@@ -0,0 +1,45 @@
|
||||
"""浏览器导航请求头构造
|
||||
|
||||
抓取链路与登录态链路都要把 httpx 请求伪装成一次正常的浏览器页面导航,头部结构
|
||||
完全一致,只有 User-Agent 以及随之联动的 `sec-ch-ua-mobile` / `sec-ch-ua-platform`
|
||||
不同。结构留在这里共用,UA 常量则各自持有:
|
||||
|
||||
- 抓取侧的 UA(`scraping/core/site.py`、`scraping/core/rakuma_site.py`)为反爬表现服务,
|
||||
换了只影响抓取成功率。
|
||||
- 登录态侧的 UA(`trading/core/auth_site.py`)必须与人工登录时浏览器用的那一个一致,
|
||||
换了可能触发站点的设备校验,使已落盘的 cookie 直接失效。
|
||||
|
||||
值今天相同,变更理由不同,因此不合并成一份常量。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Final
|
||||
|
||||
ACCEPT_LANGUAGE: Final = "ja,en-US;q=0.9,en;q=0.8"
|
||||
|
||||
|
||||
def navigation_headers(user_agent: str, *, mobile: bool) -> dict[str, str]:
|
||||
"""构造一套完整的浏览器导航请求头
|
||||
|
||||
乐天前置 Akamai Bot Manager,请求头不完整时不会直接封禁,而是把响应拖到
|
||||
~11s(实测与响应体大小无关,22 字节的响应同样耗时 11s);补齐下列头并复用
|
||||
Akamai 下发的 cookie 后,稳定在 ~0.6-0.9s。ラクマ 无此限速,但沿用同一套头
|
||||
没有代价,两侧保持一致更省心。
|
||||
"""
|
||||
return {
|
||||
"User-Agent": user_agent,
|
||||
"Accept": (
|
||||
"text/html,application/xhtml+xml,application/xml;q=0.9,"
|
||||
"image/avif,image/webp,image/apng,*/*;q=0.8,"
|
||||
"application/signed-exchange;v=b3;q=0.7"
|
||||
),
|
||||
"Accept-Language": ACCEPT_LANGUAGE,
|
||||
"sec-ch-ua": '"Chromium";v="131", "Not_A Brand";v="24", "Google Chrome";v="131"',
|
||||
"sec-ch-ua-mobile": "?1" if mobile else "?0",
|
||||
"sec-ch-ua-platform": '"iOS"' if mobile else '"Windows"',
|
||||
"Sec-Fetch-Dest": "document",
|
||||
"Sec-Fetch-Mode": "navigate",
|
||||
"Sec-Fetch-Site": "none",
|
||||
"Sec-Fetch-User": "?1",
|
||||
"Upgrade-Insecure-Requests": "1",
|
||||
}
|
||||
@@ -0,0 +1,86 @@
|
||||
"""日志配置:使用 loguru 替代标准 logging,统一管理日志输出
|
||||
|
||||
- 控制台输出:始终启用
|
||||
- 文件输出:生产环境默认启用,支持日志轮转、压缩和保留策略
|
||||
- 拦截标准 logging:将 uvicorn 等第三方库的日志转发到 loguru
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from types import FrameType
|
||||
|
||||
from loguru import logger as loguru_logger
|
||||
|
||||
from app.shared.config import Settings
|
||||
|
||||
|
||||
class InterceptHandler(logging.Handler):
|
||||
"""拦截标准 logging 日志,转发到 loguru"""
|
||||
|
||||
def emit(self, record: logging.LogRecord) -> None:
|
||||
try:
|
||||
level: str | int = loguru_logger.level(record.levelname).name
|
||||
except ValueError:
|
||||
level = record.levelno
|
||||
|
||||
frame: FrameType | None = logging.currentframe()
|
||||
depth = 2
|
||||
while frame and frame.f_code.co_filename == logging.__file__:
|
||||
frame = frame.f_back
|
||||
depth += 1
|
||||
|
||||
loguru_logger.opt(depth=depth, exception=record.exc_info).log(level, record.getMessage())
|
||||
|
||||
|
||||
def configure_logging(settings: Settings) -> None:
|
||||
"""根据配置初始化日志系统
|
||||
|
||||
- 移除 loguru 默认 handler
|
||||
- 添加控制台输出
|
||||
- 生产环境添加文件输出(轮转、压缩、保留策略)
|
||||
- 拦截标准 logging,统一走 loguru 输出
|
||||
"""
|
||||
loguru_logger.remove()
|
||||
loguru_logger.add(
|
||||
sys.stdout,
|
||||
level=settings.log_level.upper(),
|
||||
format=settings.log_format,
|
||||
enqueue=settings.log_enqueue,
|
||||
backtrace=False,
|
||||
diagnose=False,
|
||||
)
|
||||
|
||||
if settings.log_to_file_effective:
|
||||
log_dir = Path(settings.log_dir)
|
||||
log_dir.mkdir(parents=True, exist_ok=True)
|
||||
log_path = log_dir / f"app_{settings.app_env}.log"
|
||||
loguru_logger.add(
|
||||
str(log_path),
|
||||
level=settings.log_level.upper(),
|
||||
format=settings.log_format,
|
||||
rotation=settings.log_rotation,
|
||||
retention=settings.log_retention,
|
||||
compression=settings.log_compression,
|
||||
enqueue=settings.log_enqueue,
|
||||
encoding="utf-8",
|
||||
backtrace=False,
|
||||
diagnose=False,
|
||||
)
|
||||
|
||||
intercept = InterceptHandler()
|
||||
logging.basicConfig(
|
||||
handlers=[intercept],
|
||||
level=getattr(logging, settings.log_level.upper(), logging.INFO),
|
||||
force=True,
|
||||
)
|
||||
|
||||
for name in ("uvicorn", "uvicorn.error", "uvicorn.access"):
|
||||
logger = logging.getLogger(name)
|
||||
logger.handlers = [intercept]
|
||||
logger.propagate = False
|
||||
|
||||
logging.getLogger("asyncio").setLevel(logging.WARNING)
|
||||
# httpx 每次请求都会打一条 INFO,抓取服务下噪音过大
|
||||
logging.getLogger("httpx").setLevel(logging.WARNING)
|
||||
Reference in New Issue
Block a user