拆分抓取与交易服务
把需要账号登录态的链路从抓取服务里拆出成独立进程。分界线不是「要不要登录」, 而是抓取无状态、幂等、可多开实例,而交易的写操作不可逆、登录态全局唯一、 订单监控是常驻轮询——同进程时抓取一扩容就会复制出 N 份登录态与 N 个轮询, 同一账号会被并发操作。 - app/shared:配置、错误码、日志、ApiResponse 信封 + Bearer 鉴权 + 异常处理器、 导航请求头构造器 - app/scraping:站点常量、会话、解析器与 10 个抓取接口,:31107,可多开 - app/trading:登录态查询/重载与健康检查,:31108,只能单实例 - 依赖方向锁为 scraping→shared、trading→shared,两侧互不 import; tests/test_architecture.py 用 AST 检查 import 并校验两个 app 的路径不串 - 登录态 UA 在 trading 独立持有:与抓取 UA 值相同但变更理由不同,抓取 UA 为绕 反爬可随时调整,登录 UA 一改可能触发设备校验使已落盘 cookie 失效 - scripts/login.py 与 AuthSession 共用 auth_site.PROFILES 与 is_logged_in,判据只写一遍 - 同一镜像两个启动命令,交易容器覆盖 command 并设 RAKUTEN_HEALTH_PORT 同时带上此前未提交的 ラクマ 分类接口与登录态基础设施。 验证:239 个离线用例全绿;两个入口真实启动,/health 与鉴权正常。 未验证:真实探测登录态(当前开发机无外网,对站点的连接全部超时)。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,96 @@
|
||||
"""抓取服务入口:FastAPI 应用创建与生命周期管理
|
||||
|
||||
职责:
|
||||
- 构建抓取服务容器(依赖注入)
|
||||
- 管理应用生命周期(启动/关闭抓取会话与兜底浏览器)
|
||||
- 注册抓取路由和全局异常处理器
|
||||
|
||||
这个进程全程匿名、无状态,可按需要多开实例。需要账号登录态的加购、下单与
|
||||
订单监控在交易服务里(`python -m app.trading.main`),两者独立部署。
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
from fastapi import FastAPI
|
||||
|
||||
from app.scraping.api.routes.health import router as health_router
|
||||
from app.scraping.api.routes.rakuma import router as rakuma_router
|
||||
from app.scraping.api.routes.scrape import router as scrape_router
|
||||
from app.scraping.container import ScrapingContainer
|
||||
from app.scraping.services.browser_fallback import BrowserFallback
|
||||
from app.scraping.services.rakuma_client import RakumaClient
|
||||
from app.scraping.services.rakuma_session import RakumaSession
|
||||
from app.scraping.services.rakuten_client import RakutenClient
|
||||
from app.scraping.services.site_session import SiteSession
|
||||
from app.shared.api import register_exception_handlers
|
||||
from app.shared.config import get_settings
|
||||
from app.shared.logging_setup import configure_logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def build_container() -> ScrapingContainer:
|
||||
"""构建抓取服务容器,组装所有依赖"""
|
||||
settings = get_settings()
|
||||
browser_fallback = BrowserFallback(settings)
|
||||
site_session = SiteSession(settings, browser_fallback)
|
||||
rakuten_client = RakutenClient(settings, site_session)
|
||||
rakuma_session = RakumaSession(settings)
|
||||
rakuma_client = RakumaClient(settings, rakuma_session)
|
||||
return ScrapingContainer(
|
||||
settings=settings,
|
||||
browser_fallback=browser_fallback,
|
||||
site_session=site_session,
|
||||
rakuten_client=rakuten_client,
|
||||
rakuma_session=rakuma_session,
|
||||
rakuma_client=rakuma_client,
|
||||
)
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
"""应用生命周期管理:启动时初始化各服务,关闭时释放资源"""
|
||||
container = build_container()
|
||||
app.state.container = container
|
||||
|
||||
configure_logging(container.settings)
|
||||
logger.info("抓取服务启动:%s:%s", container.settings.app_host, container.settings.app_port)
|
||||
logger.info("日志级别:%s", container.settings.log_level)
|
||||
logger.info("当前环境:%s", container.settings.app_env)
|
||||
await container.site_session.start()
|
||||
await container.rakuma_session.start()
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
await container.rakuma_session.close()
|
||||
await container.site_session.close()
|
||||
await container.browser_fallback.close()
|
||||
|
||||
|
||||
def create_app() -> FastAPI:
|
||||
"""创建 FastAPI 应用实例,注册路由和异常处理器"""
|
||||
app = FastAPI(title="Rakuten Scraper Service", lifespan=lifespan)
|
||||
app.include_router(health_router)
|
||||
app.include_router(scrape_router)
|
||||
app.include_router(rakuma_router)
|
||||
register_exception_handlers(app)
|
||||
return app
|
||||
|
||||
|
||||
app = create_app()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
|
||||
settings = get_settings()
|
||||
configure_logging(settings)
|
||||
uvicorn.run(
|
||||
"app.scraping.main:app",
|
||||
host=settings.app_host,
|
||||
port=settings.app_port,
|
||||
log_config=None,
|
||||
timeout_keep_alive=120,
|
||||
)
|
||||
Reference in New Issue
Block a user