11 Commits

Author SHA1 Message Date
7ecc33b43c i 2025-09-24 11:38:17 +08:00
ba90778b2d l 2025-09-24 11:35:52 +08:00
501cea3888 a 2025-09-24 11:30:52 +08:00
873c89c778 u 2025-09-24 11:28:48 +08:00
ec1705769c q 2025-09-24 10:50:34 +08:00
c055a68592 mypy 2025-09-24 10:17:11 +08:00
dfb07d679f test 2025-09-24 09:21:03 +08:00
6d37e7c23c refactor 2025-09-24 04:51:12 +08:00
a54c157f9a docs: 添加 2025-09-12 15:03:38 +08:00
d496c7400d refactor(biz): 重构 FFmpeg 任务处理逻辑
-将主要处理逻辑迁移到新的 TaskService 架构中
-保持 FfmpegTask 类的接口
2025-09-12 14:59:04 +08:00
d770d84927 feat(重构): 实现新的渲染服务架构
- 新增 RenderTask
2025-09-12 14:41:58 +08:00
49 changed files with 5905 additions and 927 deletions

15
.flake8 Normal file
View File

@@ -0,0 +1,15 @@
[flake8]
max-line-length = 88
ignore =
# Line too long - handled by black
E501,
# Line break before binary operator - handled by black
W503
exclude =
.git,
__pycache__,
.venv,
venv,
tests,
.claude,
.serena

4
.gitignore vendored
View File

@@ -31,4 +31,6 @@ target/
.venv
venv/
cython_debug/
.env
.env
.serena
.claude

179
CLAUDE.md Normal file
View File

@@ -0,0 +1,179 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Project Overview
RenderWorker is a Python-based video processing service that renders video content using FFmpeg. It operates as both a Flask web service and a standalone worker process, processing video rendering tasks with templates, effects, and various transformations.
## Architecture
The project uses a modern layered architecture with clear separation of concerns:
### Core Components
- **Services Layer** (`services/`): Main business logic with dependency injection
- `RenderService`: Handles video rendering operations
- `TemplateService`: Manages video templates and resources
- `TaskService`: Orchestrates the complete task lifecycle
- **Entity Layer** (`entity/`): Data models and domain logic
- `RenderTask`: Core data model for rendering tasks
- `FFmpegCommandBuilder`: Generates FFmpeg commands from tasks
- `effects/`: Pluggable effect processing system (zoom, speed, skip, tail, cameraShot)
- **Configuration** (`config/`): Centralized configuration management
- Environment-based configuration with dataclasses
- FFmpeg, API, Storage, and Server configurations
- **Utilities** (`util/`): Common utilities for API, OSS, FFmpeg operations
### Entry Points
- **Web Service**: `app.py` - Flask application serving HTTP endpoints
- **Worker Process**: `index.py` - Long-running task processor that polls for work
- **Legacy Compatibility**: Maintains backward compatibility with older `entity/ffmpeg.py` and `biz/` modules
## Common Commands
### Running the Application
```bash
# Web service mode (Flask API)
python app.py
# Worker mode (background task processor)
python index.py
# Worker with template redownload
python index.py redownload
```
### Development Setup
```bash
# Install dependencies
pip install -r requirements.txt
# Environment setup
cp .env.example .env
# Edit .env with your configuration
```
### Building
```bash
# Build standalone executable
pyinstaller index.spec
# Build updater
pyinstaller updater_helper.spec
```
## Key Environment Variables
Required configuration (see `.env.example`):
- `API_ENDPOINT`: Backend API endpoint
- `ACCESS_KEY`: Authentication key
- `TEMPLATE_DIR`: Directory for template storage
- `ENCODER_ARGS`: FFmpeg encoder arguments (default: "-c:v h264")
- `VIDEO_ARGS`: Video processing arguments
- `OTLP_ENDPOINT`: OpenTelemetry endpoint for monitoring
## Architecture Patterns
### Service Layer Pattern
All new code should use the service layer:
```python
from services import DefaultRenderService, DefaultTemplateService, DefaultTaskService
# Dependency injection
render_service = DefaultRenderService()
template_service = DefaultTemplateService()
task_service = DefaultTaskService(render_service, template_service)
# Process tasks
success = task_service.process_task(task_info)
```
### Effect Processing System
Video effects use a pluggable architecture:
```python
from entity.effects import registry
from entity.effects.base import EffectProcessor
# Get existing effect
processor = registry.get_processor('zoom', '0,2.0,1.5')
# Register new effect
class MyEffect(EffectProcessor):
def validate_params(self) -> bool:
return True
def generate_filter_args(self, video_input: str, effect_index: int):
return filter_args, output_stream
def get_effect_name(self) -> str:
return "myeffect"
registry.register('myeffect', MyEffect)
```
### Task Processing Flow
1. **Task Reception**: API receives task via HTTP endpoint or polling
2. **Template Resolution**: Template service downloads/loads video templates
3. **Task Creation**: Task service creates RenderTask with effects and resources
4. **Command Building**: FFmpegCommandBuilder generates FFmpeg commands
5. **Execution**: RenderService executes FFmpeg with monitoring
6. **Cleanup**: Temporary files cleaned, results uploaded
## Backward Compatibility
The codebase maintains compatibility with legacy components:
- `biz.task.start_task()` - Main task entry point
- `entity.ffmpeg.FfmpegTask` - Legacy task object (now simplified)
- `template.get_template_def()` - Template access
These should work but new development should use the service layer.
## FFmpeg Integration
The system generates complex FFmpeg commands for video processing:
- Supports effects chaining (zoom, speed changes, cropping)
- Handles multiple input files and concatenation
- Manages audio processing and overlays
- Supports LUT color grading and subtitle embedding
## Monitoring and Telemetry
- **OpenTelemetry**: Distributed tracing for request monitoring
- **Structured Logging**: Detailed logs for debugging
- **Error Handling**: Custom exception types (`RenderError`, `FFmpegError`, `TemplateError`)
## File Organization
- Configuration and environment variables → `config/`
- Business logic and services → `services/`
- Data models and domain objects → `entity/`
- HTTP API and external integrations → `util/`
- Legacy compatibility layer → `biz/`
- Application entry points → `app.py`, `index.py`
## Working with Templates
Templates define video processing workflows:
- Downloaded from remote API and cached locally
- Support video parts, effects, overlays, and audio
- Use placeholder system for dynamic content
- Managed by `TemplateService` with automatic updates
## Development Notes
- The project recently underwent major architectural refactoring from monolithic to layered design
- New features should use the services layer and effect processor system
- Legacy code is maintained for compatibility but simplified
- All video processing ultimately uses FFmpeg with generated command lines
- The system supports both synchronous (web) and asynchronous (worker) operation modes

347
Jenkinsfile vendored Normal file
View File

@@ -0,0 +1,347 @@
pipeline {
agent any
environment {
// 环境变量
PYTHON_VERSION = '3.9'
VENV_NAME = 'venv'
TEST_REPORTS_DIR = 'test-reports'
COVERAGE_DIR = 'coverage-reports'
// 设置Python模块路径
PYTHONPATH = "${WORKSPACE}"
}
stages {
stage('Preparation') {
steps {
script {
// 清理工作空间
echo 'Cleaning workspace...'
deleteDir()
// 检出代码
checkout scm
// 创建报告目录
sh """
mkdir -p ${TEST_REPORTS_DIR}
mkdir -p ${COVERAGE_DIR}
"""
}
}
}
stage('Environment Setup') {
steps {
script {
echo 'Setting up Python environment...'
sh """
# 创建虚拟环境
python${PYTHON_VERSION} -m venv ${VENV_NAME}
# 激活虚拟环境并安装依赖
. ${VENV_NAME}/bin/activate
pip config set global.index-url https://mirrors.ustc.edu.cn/pypi/simple
# 设置PYTHONPATH
export PYTHONPATH=\${PWD}:\$PYTHONPATH
# 升级pip
pip install --upgrade pip
# 安装项目依赖
pip install -r requirements.txt
# 安装测试依赖
pip install -r requirements-test.txt
# 验证FFmpeg可用性
which ffmpeg || echo "Warning: FFmpeg not found, integration tests may be skipped"
"""
}
}
}
stage('Code Quality Check') {
parallel {
stage('Linting') {
steps {
script {
echo 'Running code linting...'
sh """
. ${VENV_NAME}/bin/activate
export PYTHONPATH=\${PWD}:\$PYTHONPATH
# 运行flake8检查(使用项目配置.flake8)
flake8 entity/ services/ --output-file=${TEST_REPORTS_DIR}/flake8-report.txt --tee || true
# 运行black格式检查
black --check --diff --line-length 88 entity/ services/ > ${TEST_REPORTS_DIR}/black-report.txt || true
"""
}
}
}
stage('Type Checking') {
steps {
script {
echo 'Running type checking with mypy...'
sh """
. ${VENV_NAME}/bin/activate
export PYTHONPATH=\${PWD}:\$PYTHONPATH
# 运行mypy类型检查
mypy --explicit-package-bases services/ entity/ > ${TEST_REPORTS_DIR}/mypy-report.txt || true
"""
}
}
}
}
post {
always {
// 发布代码质量报告(等待所有并行任务完成后统一发布)
publishHTML([
allowMissing: true,
alwaysLinkToLastBuild: true,
keepAll: true,
reportDir: TEST_REPORTS_DIR,
reportFiles: 'flake8-report.txt,black-report.txt,mypy-report.txt',
reportName: 'Code Quality Report'
])
}
}
}
stage('Unit Tests') {
steps {
script {
echo 'Running unit tests...'
sh """
. ${VENV_NAME}/bin/activate
export PYTHONPATH=\${PWD}:\$PYTHONPATH
# 运行单元测试
pytest tests/test_effects/ tests/test_ffmpeg_builder/ \\
--junitxml=${TEST_REPORTS_DIR}/unit-tests.xml \\
--html=${TEST_REPORTS_DIR}/unit-tests.html \\
--self-contained-html \\
--cov=entity \\
--cov=services \\
--cov-report=xml:${COVERAGE_DIR}/unit-coverage.xml \\
--cov-branch \\
-v \\
-m "not integration"
"""
}
}
post {
always {
// 发布单元测试结果
junit "${TEST_REPORTS_DIR}/unit-tests.xml"
// 发布HTML测试报告
publishHTML([
allowMissing: false,
alwaysLinkToLastBuild: true,
keepAll: true,
reportDir: TEST_REPORTS_DIR,
reportFiles: 'unit-tests.html',
reportName: 'Unit Tests Report'
])
}
}
}
stage('Integration Tests') {
when {
// 只在有FFmpeg的环境中运行集成测试
expression {
return sh(
script: 'which ffmpeg',
returnStatus: true
) == 0
}
}
steps {
script {
echo 'Running integration tests...'
sh """
. ${VENV_NAME}/bin/activate
export PYTHONPATH=\${PWD}:\$PYTHONPATH
# 运行集成测试
pytest tests/test_integration/ \\
--junitxml=${TEST_REPORTS_DIR}/integration-tests.xml \\
--html=${TEST_REPORTS_DIR}/integration-tests.html \\
--self-contained-html \\
--cov=entity \\
--cov=services \\
--cov-report=xml:${COVERAGE_DIR}/integration-coverage.xml \\
--cov-branch \\
--timeout=300 \\
-v \\
-m "integration"
"""
}
}
post {
always {
// 发布集成测试结果
junit "${TEST_REPORTS_DIR}/integration-tests.xml"
// 发布HTML集成测试报告
publishHTML([
allowMissing: false,
alwaysLinkToLastBuild: true,
keepAll: true,
reportDir: TEST_REPORTS_DIR,
reportFiles: 'integration-tests.html',
reportName: 'Integration Tests Report'
])
}
}
}
stage('Complete Test Suite') {
steps {
script {
echo 'Running complete test suite with coverage...'
sh """
. ${VENV_NAME}/bin/activate
export PYTHONPATH=\${PWD}:\$PYTHONPATH
# 运行完整测试套件
pytest tests/ \\
--junitxml=${TEST_REPORTS_DIR}/all-tests.xml \\
--html=${TEST_REPORTS_DIR}/all-tests.html \\
--self-contained-html \\
--cov=entity \\
--cov=services \\
--cov-report=xml:${COVERAGE_DIR}/coverage.xml \\
--cov-report=term-missing \\
--cov-branch \\
--cov-fail-under=70 \\
-v
"""
}
}
post {
always {
// 发布完整测试结果
junit "${TEST_REPORTS_DIR}/all-tests.xml"
// 发布代码覆盖率报告
publishCoverage([
adapters: [
[
mergeToOneReport: true,
path: "${COVERAGE_DIR}/coverage.xml"
]
],
sourceFileResolver: [
[
level: 'NEVER_STORE'
]
]
])
// 发布完整测试HTML报告
publishHTML([
allowMissing: false,
alwaysLinkToLastBuild: true,
keepAll: true,
reportDir: TEST_REPORTS_DIR,
reportFiles: 'all-tests.html',
reportName: 'Complete Test Report'
])
}
}
}
stage('Performance Tests') {
when {
// 只在主分支或发布分支运行性能测试
anyOf {
branch 'master'
branch 'main'
branch 'release/*'
}
}
steps {
script {
echo 'Running performance tests...'
sh """
. ${VENV_NAME}/bin/activate
export PYTHONPATH=\${PWD}:\$PYTHONPATH
# 运行性能测试
RUN_STRESS_TESTS=1 pytest tests/test_integration/test_ffmpeg_execution.py::TestFFmpegExecution::test_stress_test_large_effects_chain \\
--benchmark-json=${TEST_REPORTS_DIR}/benchmark.json \\
--html=${TEST_REPORTS_DIR}/performance-tests.html \\
--self-contained-html \\
-v \\
-m "stress" || true
"""
}
}
post {
always {
// 发布性能测试报告
publishHTML([
allowMissing: true,
alwaysLinkToLastBuild: true,
keepAll: true,
reportDir: TEST_REPORTS_DIR,
reportFiles: 'performance-tests.html',
reportName: 'Performance Tests Report'
])
}
}
}
}
post {
always {
// 清理虚拟环境
sh """
rm -rf ${VENV_NAME}
"""
// 归档测试报告
archiveArtifacts artifacts: "${TEST_REPORTS_DIR}/**/*", allowEmptyArchive: true
archiveArtifacts artifacts: "${COVERAGE_DIR}/**/*", allowEmptyArchive: true
}
success {
echo 'All tests passed successfully!'
// 发送成功通知
script {
if (env.BRANCH_NAME == 'master' || env.BRANCH_NAME == 'main') {
// 这里可以添加Slack、邮件或其他通知
echo 'Sending success notification...'
}
}
}
failure {
echo 'Tests failed!'
// 发送失败通知
script {
// 这里可以添加Slack、邮件或其他通知
echo 'Sending failure notification...'
}
}
unstable {
echo 'Tests completed with warnings!'
}
cleanup {
// 清理临时文件
deleteDir()
}
}
}

61
app.py
View File

@@ -4,37 +4,68 @@ import flask
import config
import biz.task
import template
from services import DefaultTemplateService
from telemetry import init_opentelemetry
from template import load_local_template
from util import api
load_local_template()
# 使用新的服务容器架构
from services.service_container import get_template_service, register_default_services
# 确保服务已注册
register_default_services()
template_service = get_template_service()
import logging
LOGGER = logging.getLogger(__name__)
init_opentelemetry(batch=False)
app = flask.Flask(__name__)
@app.get('/health/check')
@app.get("/health/check")
def health_check():
return api.sync_center()
@app.post('/')
@app.post("/")
def do_nothing():
return "NOOP"
@app.post('/<task_id>')
@app.post("/<task_id>")
def do_task(task_id):
task_info = api.get_task_info(task_id)
local_template_info = template.get_template_def(task_info.get("templateId"))
template_info = api.get_template_info(task_info.get("templateId"))
if local_template_info:
if local_template_info.get("updateTime") != template_info.get("updateTime"):
template.download_template(task_info.get("templateId"))
biz.task.start_task(task_info)
return "OK"
try:
task_info = api.get_task_info(task_id)
if not task_info:
LOGGER.error("Failed to get task info for task: %s", task_id)
return "Failed to get task info", 400
template_id = task_info.get("templateId")
if not template_id:
LOGGER.error("Task %s missing templateId", task_id)
return "Missing templateId", 400
local_template_info = template_service.get_template(template_id)
template_info = api.get_template_info(template_id)
if not template_info:
LOGGER.error("Failed to get template info for template: %s", template_id)
return "Failed to get template info", 400
if local_template_info:
if local_template_info.get("updateTime") != template_info.get("updateTime"):
LOGGER.info("Template %s needs update, downloading...", template_id)
if not template_service.download_template(template_id):
LOGGER.error("Failed to download template: %s", template_id)
return "Failed to download template", 500
biz.task.start_task(task_info)
return "OK"
except Exception as e:
LOGGER.error("Error processing task %s: %s", task_id, e, exc_info=True)
return "Internal server error", 500
if __name__ == '__main__':
if __name__ == "__main__":
app.run(host="0.0.0.0", port=9998)

View File

@@ -5,168 +5,140 @@ from concurrent.futures import ThreadPoolExecutor
from opentelemetry.trace import Status, StatusCode
# 使用新架构组件,保持对旧FfmpegTask的兼容
from entity.ffmpeg import FfmpegTask
from entity.render_task import RenderTask, TaskType
from services import DefaultRenderService
import logging
from util import ffmpeg, oss
from util.ffmpeg import fade_out_audio
from telemetry import get_tracer
logger = logging.getLogger('biz/ffmpeg')
logger = logging.getLogger("biz/ffmpeg")
_render_service = None
def _get_render_service():
"""获取渲染服务实例"""
global _render_service
if _render_service is None:
_render_service = DefaultRenderService()
return _render_service
def parse_ffmpeg_task(task_info, template_info):
tracer = get_tracer(__name__)
with tracer.start_as_current_span("parse_ffmpeg_task") as span:
tasks = []
# 中间片段
task_params_str = task_info.get("taskParams", "{}")
span.set_attribute("task_params", task_params_str)
task_params: dict = json.loads(task_params_str)
task_params_orig = json.loads(task_params_str)
# 统计only_if占位符的使用次数
only_if_usage_count = {}
with tracer.start_as_current_span("parse_ffmpeg_task.download_all") as sub_span:
with ThreadPoolExecutor(max_workers=8) as executor:
param_list: list[dict]
for param_list in task_params.values():
for param in param_list:
url = param.get("url")
if url.startswith("http"):
_, fn = os.path.split(url)
executor.submit(oss.download_from_oss, url, fn, True)
executor.shutdown(wait=True)
for part in template_info.get("video_parts"):
source, ext_data = parse_video(part.get('source'), task_params, template_info)
if not source:
logger.warning("no video found for part: " + str(part))
continue
only_if = part.get('only_if', '')
if only_if:
only_if_usage_count[only_if] = only_if_usage_count.get(only_if, 0) + 1
required_count = only_if_usage_count.get(only_if)
if not check_placeholder_exist_with_count(only_if, task_params_orig, required_count):
logger.info("because only_if exist, placeholder: %s insufficient (need %d), skip part: %s", only_if, required_count, part)
continue
sub_ffmpeg_task = FfmpegTask(source)
sub_ffmpeg_task.resolution = template_info.get("video_size", "")
sub_ffmpeg_task.annexb = True
sub_ffmpeg_task.ext_data = ext_data or {}
sub_ffmpeg_task.frame_rate = template_info.get("frame_rate", 25)
sub_ffmpeg_task.center_cut = part.get("crop_mode", None)
sub_ffmpeg_task.zoom_cut = part.get("zoom_cut", None)
for effect in part.get('effects', []):
sub_ffmpeg_task.add_effect(effect)
for lut in part.get('luts', []):
sub_ffmpeg_task.add_lut(os.path.join(template_info.get("local_path"), lut).replace("\\", "/"))
for audio in part.get('audios', []):
sub_ffmpeg_task.add_audios(os.path.join(template_info.get("local_path"), audio))
for overlay in part.get('overlays', []):
sub_ffmpeg_task.add_overlay(os.path.join(template_info.get("local_path"), overlay))
tasks.append(sub_ffmpeg_task)
output_file = "out_" + str(time.time()) + ".mp4"
task = FfmpegTask(tasks, output_file=output_file)
task.resolution = template_info.get("video_size", "")
overall = template_info.get("overall_template")
task.center_cut = template_info.get("crop_mode", None)
task.zoom_cut = template_info.get("zoom_cut", None)
task.frame_rate = template_info.get("frame_rate", 25)
# if overall.get('source', ''):
# source, ext_data = parse_video(overall.get('source'), task_params, template_info)
# task.add_inputs(source)
# task.ext_data = ext_data or {}
for effect in overall.get('effects', []):
task.add_effect(effect)
for lut in overall.get('luts', []):
task.add_lut(os.path.join(template_info.get("local_path"), lut).replace("\\", "/"))
for audio in overall.get('audios', []):
task.add_audios(os.path.join(template_info.get("local_path"), audio))
for overlay in overall.get('overlays', []):
task.add_overlay(os.path.join(template_info.get("local_path"), overlay))
return task
"""
解析FFmpeg任务 - 保留用于向后兼容
实际处理逻辑已迁移到 services.TaskService.create_render_task
"""
logger.warning(
"parse_ffmpeg_task is deprecated, use TaskService.create_render_task instead"
)
# 使用新的任务服务创建任务
from services import (
DefaultTaskService,
DefaultRenderService,
DefaultTemplateService,
)
render_service = DefaultRenderService()
template_service = DefaultTemplateService()
task_service = DefaultTaskService(render_service, template_service)
# 创建新的渲染任务
render_task = task_service.create_render_task(task_info, template_info)
# 为了向后兼容,创建一个FfmpegTask包装器
ffmpeg_task = FfmpegTask(
render_task.input_files, output_file=render_task.output_file
)
ffmpeg_task.resolution = render_task.resolution
ffmpeg_task.frame_rate = render_task.frame_rate
ffmpeg_task.annexb = render_task.annexb
ffmpeg_task.center_cut = render_task.center_cut
ffmpeg_task.zoom_cut = render_task.zoom_cut
ffmpeg_task.ext_data = render_task.ext_data
ffmpeg_task.effects = render_task.effects
ffmpeg_task.luts = render_task.luts
ffmpeg_task.audios = render_task.audios
ffmpeg_task.overlays = render_task.overlays
return ffmpeg_task
# 以下函数已迁移到新架构,保留用于向后兼容
def parse_video(source, task_params, template_info):
if source.startswith('PLACEHOLDER_'):
placeholder_id = source.replace('PLACEHOLDER_', '')
new_sources = task_params.get(placeholder_id, [])
_pick_source = {}
if type(new_sources) is list:
if len(new_sources) == 0:
logger.debug("no video found for placeholder: " + placeholder_id)
return None, _pick_source
else:
_pick_source = new_sources.pop(0)
new_sources = _pick_source.get("url")
if new_sources.startswith("http"):
_, source_name = os.path.split(new_sources)
oss.download_from_oss(new_sources, source_name, True)
return source_name, _pick_source
return new_sources, _pick_source
return os.path.join(template_info.get("local_path"), source), None
"""已迁移到 TaskService._parse_video_source"""
logger.warning("parse_video is deprecated, functionality moved to TaskService")
return source, {}
def check_placeholder_exist(placeholder_id, task_params):
if placeholder_id in task_params:
new_sources = task_params.get(placeholder_id, [])
if type(new_sources) is list:
if len(new_sources) == 0:
return False
else:
return True
return True
return False
"""已迁移到 TaskService._check_placeholder_exist_with_count"""
logger.warning(
"check_placeholder_exist is deprecated, functionality moved to TaskService"
)
return placeholder_id in task_params
def check_placeholder_exist_with_count(placeholder_id, task_params, required_count=1):
"""检查占位符是否存在足够数量的片段"""
"""已迁移到 TaskService._check_placeholder_exist_with_count"""
logger.warning(
"check_placeholder_exist_with_count is deprecated, functionality moved to TaskService"
)
if placeholder_id in task_params:
new_sources = task_params.get(placeholder_id, [])
if type(new_sources) is list:
if isinstance(new_sources, list):
return len(new_sources) >= required_count
return required_count <= 1
return False
def start_ffmpeg_task(ffmpeg_task):
"""启动FFmpeg任务 - 使用新的渲染服务"""
tracer = get_tracer(__name__)
with tracer.start_as_current_span("start_ffmpeg_task") as span:
for task in ffmpeg_task.analyze_input_render_tasks():
result = start_ffmpeg_task(task)
if not result:
return False
ffmpeg_task.correct_task_type()
span.set_attribute("task.type", ffmpeg_task.task_type)
span.set_attribute("task.center_cut", str(ffmpeg_task.center_cut))
span.set_attribute("task.frame_rate", ffmpeg_task.frame_rate)
span.set_attribute("task.resolution", str(ffmpeg_task.resolution))
span.set_attribute("task.ext_data", json.dumps(ffmpeg_task.ext_data))
result = ffmpeg.start_render(ffmpeg_task)
if not result:
try:
# 使用新的渲染服务
render_service = _get_render_service()
result = render_service.render(ffmpeg_task)
if result:
span.set_status(Status(StatusCode.OK))
else:
span.set_status(Status(StatusCode.ERROR))
return result
except Exception as e:
span.set_status(Status(StatusCode.ERROR))
logger.error(f"FFmpeg task failed: {e}", exc_info=True)
return False
span.set_status(Status(StatusCode.OK))
return True
def clear_task_tmp_file(ffmpeg_task):
for task in ffmpeg_task.analyze_input_render_tasks():
clear_task_tmp_file(task)
"""清理临时文件 - 已迁移到 TaskService._cleanup_temp_files"""
logger.warning(
"clear_task_tmp_file is deprecated, functionality moved to TaskService"
)
try:
if os.getenv("TEMPLATE_DIR") not in ffmpeg_task.get_output_file():
os.remove(ffmpeg_task.get_output_file())
logger.info("delete tmp file: " + ffmpeg_task.get_output_file())
template_dir = os.getenv("TEMPLATE_DIR", "")
output_file = ffmpeg_task.get_output_file()
if template_dir and template_dir not in output_file:
if os.path.exists(output_file):
os.remove(output_file)
logger.info("Cleaned up temp file: %s", output_file)
else:
logger.info("skip delete template file: " + ffmpeg_task.get_output_file())
except OSError:
logger.warning("delete tmp file failed: " + ffmpeg_task.get_output_file())
logger.info("Skipped cleanup of template file: %s", output_file)
return True
except OSError as e:
logger.warning("Failed to cleanup temp file %s: %s", output_file, e)
return False
return True
def probe_video_info(ffmpeg_task):
# 获取视频长度宽度和时长
return ffmpeg.probe_video_info(ffmpeg_task.get_output_file())
"""获取视频长度宽度和时长 - 使用新的渲染服务"""
render_service = _get_render_service()
return render_service.get_video_info(ffmpeg_task.get_output_file())

View File

@@ -1,44 +1,39 @@
import json
import logging
from opentelemetry.trace import Status, StatusCode
from biz.ffmpeg import parse_ffmpeg_task, start_ffmpeg_task, clear_task_tmp_file, probe_video_info, fade_out_audio
# 使用新的服务容器架构
from services.service_container import get_task_service, register_default_services
from telemetry import get_tracer
from template import get_template_def
from util import api
logger = logging.getLogger(__name__)
# 确保服务已注册
register_default_services()
def start_task(task_info):
"""启动任务处理(保持向后兼容的接口)"""
tracer = get_tracer(__name__)
with tracer.start_as_current_span("start_task") as span:
task_info = api.normalize_task(task_info)
span.set_attribute("task", json.dumps(task_info))
span.set_attribute("scenicId", task_info.get("scenicId", "?"))
span.set_attribute("templateId", task_info.get("templateId"))
template_info = get_template_def(task_info.get("templateId"))
api.report_task_start(task_info)
ffmpeg_task = parse_ffmpeg_task(task_info, template_info)
result = start_ffmpeg_task(ffmpeg_task)
if not result:
with tracer.start_as_current_span("start_task_legacy") as span:
try:
# 使用服务容器获取任务服务
task_service = get_task_service()
# 使用新的任务服务处理
result = task_service.process_task(task_info)
if result:
span.set_status(Status(StatusCode.OK))
logger.info("Task completed successfully: %s", task_info.get("id"))
else:
span.set_status(Status(StatusCode.ERROR))
logger.error("Task failed: %s", task_info.get("id"))
return None # 保持原有返回值格式
except Exception as e:
span.set_status(Status(StatusCode.ERROR))
return api.report_task_failed(task_info)
width, height, duration = probe_video_info(ffmpeg_task)
span.set_attribute("probe.width", width)
span.set_attribute("probe.height", height)
span.set_attribute("probe.duration", duration)
# 音频淡出
new_fn = fade_out_audio(ffmpeg_task.get_output_file(), duration)
ffmpeg_task.set_output_file(new_fn)
oss_result = api.upload_task_file(task_info, ffmpeg_task)
if not oss_result:
span.set_status(Status(StatusCode.ERROR))
return api.report_task_failed(task_info)
# 获取视频长度宽度和时长
clear_task_tmp_file(ffmpeg_task)
api.report_task_success(task_info, videoInfo={
"width": width,
"height": height,
"duration": duration
})
span.set_status(Status(StatusCode.OK))
return None
logger.error("Task processing failed: %s", e, exc_info=True)
return None

View File

@@ -3,14 +3,29 @@ import logging
from logging.handlers import TimedRotatingFileHandler
from dotenv import load_dotenv
# 导入新的配置系统,保持向后兼容
from .settings import (
get_config,
get_ffmpeg_config,
get_api_config,
get_storage_config,
get_server_config,
)
load_dotenv()
logging.basicConfig(level=logging.INFO)
root_logger = logging.getLogger()
rf_handler = TimedRotatingFileHandler('all_log.log', when='midnight')
rf_handler.setFormatter(logging.Formatter("[%(asctime)s][%(name)s]%(levelname)s - %(message)s"))
rf_handler = TimedRotatingFileHandler("all_log.log", when="midnight")
rf_handler.setFormatter(
logging.Formatter("[%(asctime)s][%(name)s]%(levelname)s - %(message)s")
)
rf_handler.setLevel(logging.DEBUG)
f_handler = TimedRotatingFileHandler('error.log', when='midnight')
f_handler = TimedRotatingFileHandler("error.log", when="midnight")
f_handler.setLevel(logging.ERROR)
f_handler.setFormatter(logging.Formatter("[%(asctime)s][%(name)s][:%(lineno)d]%(levelname)s - - %(message)s"))
f_handler.setFormatter(
logging.Formatter(
"[%(asctime)s][%(name)s][:%(lineno)d]%(levelname)s - - %(message)s"
)
)
root_logger.addHandler(rf_handler)
root_logger.addHandler(f_handler)
root_logger.addHandler(f_handler)

181
config/settings.py Normal file
View File

@@ -0,0 +1,181 @@
import os
from dataclasses import dataclass
from typing import Dict, List, Optional, Union
import logging
from dotenv import load_dotenv
load_dotenv()
@dataclass
class FFmpegConfig:
"""FFmpeg相关配置"""
encoder_args: List[str]
video_args: List[str]
audio_args: List[str]
default_args: List[str]
old_ffmpeg: bool = False
re_encode_video_args: Optional[List[str]] = None
re_encode_encoder_args: Optional[List[str]] = None
# 新增配置选项,消除硬编码
max_download_workers: int = 8
progress_args: Optional[List[str]] = None
loglevel_args: Optional[List[str]] = None
null_audio_args: Optional[List[str]] = None
overlay_scale_mode: str = "scale2ref" # 新版本使用scale2ref,旧版本使用scale
amix_args: Optional[List[str]] = None
@classmethod
def from_env(cls) -> "FFmpegConfig":
encoder_args = os.getenv("ENCODER_ARGS", "-c:v h264").split(" ")
video_args = os.getenv("VIDEO_ARGS", "-profile:v high -level:v 4").split(" ")
audio_args = ["-c:a", "aac", "-b:a", "128k", "-ar", "48000", "-ac", "2"]
default_args = ["-shortest"]
re_encode_video_args = None
re_encode_video_env = os.getenv("RE_ENCODE_VIDEO_ARGS")
if re_encode_video_env:
re_encode_video_args = re_encode_video_env.split(" ")
re_encode_encoder_args = None
re_encode_encoder_env = os.getenv("RE_ENCODE_ENCODER_ARGS")
if re_encode_encoder_env:
re_encode_encoder_args = re_encode_encoder_env.split(" ")
# 新增配置项的默认值
progress_args = ["-progress", "-"]
loglevel_args = ["-loglevel", "error"]
null_audio_args = ["-f", "lavfi", "-i", "anullsrc=cl=stereo:r=48000"]
amix_args = ["amix=duration=shortest:dropout_transition=0:normalize=0"]
overlay_scale_mode = (
"scale" if bool(os.getenv("OLD_FFMPEG", False)) else "scale2ref"
)
return cls(
encoder_args=encoder_args,
video_args=video_args,
audio_args=audio_args,
default_args=default_args,
old_ffmpeg=bool(os.getenv("OLD_FFMPEG", False)),
re_encode_video_args=re_encode_video_args,
re_encode_encoder_args=re_encode_encoder_args,
max_download_workers=int(os.getenv("MAX_DOWNLOAD_WORKERS", "8")),
progress_args=progress_args,
loglevel_args=loglevel_args,
null_audio_args=null_audio_args,
overlay_scale_mode=overlay_scale_mode,
amix_args=amix_args,
)
@dataclass
class APIConfig:
"""API相关配置"""
endpoint: str
access_key: str
timeout: int = 10
redirect_to_url: Optional[str] = None
@classmethod
def from_env(cls) -> "APIConfig":
endpoint = os.getenv("API_ENDPOINT", "")
if not endpoint:
raise ValueError("API_ENDPOINT environment variable is required")
access_key = os.getenv("ACCESS_KEY", "")
if not access_key:
raise ValueError("ACCESS_KEY environment variable is required")
return cls(
endpoint=endpoint,
access_key=access_key,
timeout=int(os.getenv("API_TIMEOUT", "10")),
redirect_to_url=os.getenv("REDIRECT_TO_URL") or None,
)
@dataclass
class StorageConfig:
"""存储相关配置"""
template_dir: str
@classmethod
def from_env(cls) -> "StorageConfig":
template_dir = os.getenv("TEMPLATE_DIR", "./template")
return cls(template_dir=template_dir)
@dataclass
class ServerConfig:
"""服务器相关配置"""
host: str = "0.0.0.0"
port: int = 9998
debug: bool = False
@classmethod
def from_env(cls) -> "ServerConfig":
return cls(
host=os.getenv("HOST", "0.0.0.0"),
port=int(os.getenv("PORT", "9998")),
debug=bool(os.getenv("DEBUG", False)),
)
@dataclass
class AppConfig:
"""应用总配置"""
ffmpeg: FFmpegConfig
api: APIConfig
storage: StorageConfig
server: ServerConfig
@classmethod
def from_env(cls) -> "AppConfig":
return cls(
ffmpeg=FFmpegConfig.from_env(),
api=APIConfig.from_env(),
storage=StorageConfig.from_env(),
server=ServerConfig.from_env(),
)
# 全局配置实例
_config: Optional[AppConfig] = None
def get_config() -> AppConfig:
"""获取全局配置实例"""
global _config
if _config is None:
_config = AppConfig.from_env()
return _config
def reload_config() -> AppConfig:
"""重新加载配置"""
global _config
_config = AppConfig.from_env()
return _config
# 向后兼容的配置获取函数
def get_ffmpeg_config() -> FFmpegConfig:
return get_config().ffmpeg
def get_api_config() -> APIConfig:
return get_config().api
def get_storage_config() -> StorageConfig:
return get_config().storage
def get_server_config() -> ServerConfig:
return get_config().server

View File

@@ -1,9 +1,9 @@
SUPPORT_FEATURE = (
'simple_render_algo',
'gpu_accelerate',
'hevc_encode',
'rapid_download',
'rclone_upload',
'custom_re_encode',
"simple_render_algo",
"gpu_accelerate",
"hevc_encode",
"rapid_download",
"rclone_upload",
"custom_re_encode",
)
SOFTWARE_VERSION = '0.0.5'
SOFTWARE_VERSION = "0.0.5"

0
entity/__init__.py Normal file
View File

View File

@@ -0,0 +1,25 @@
from .base import EffectProcessor, EffectRegistry
from .camera_shot import CameraShotEffect
from .speed import SpeedEffect
from .zoom import ZoomEffect
from .skip import SkipEffect
from .tail import TailEffect
# 注册所有效果处理器
registry = EffectRegistry()
registry.register("cameraShot", CameraShotEffect)
registry.register("ospeed", SpeedEffect)
registry.register("zoom", ZoomEffect)
registry.register("skip", SkipEffect)
registry.register("tail", TailEffect)
__all__ = [
"EffectProcessor",
"EffectRegistry",
"registry",
"CameraShotEffect",
"SpeedEffect",
"ZoomEffect",
"SkipEffect",
"TailEffect",
]

103
entity/effects/base.py Normal file
View File

@@ -0,0 +1,103 @@
from abc import ABC, abstractmethod
from typing import Dict, List, Type, Any, Optional
import json
import logging
logger = logging.getLogger(__name__)
class EffectProcessor(ABC):
"""效果处理器抽象基类"""
def __init__(self, params: str = "", ext_data: Optional[Dict[str, Any]] = None):
self.params = params
self.ext_data = ext_data or {}
self.frame_rate = 25 # 默认帧率
@abstractmethod
def validate_params(self) -> bool:
"""验证参数是否有效"""
pass
@abstractmethod
def generate_filter_args(
self, video_input: str, effect_index: int
) -> tuple[List[str], str]:
"""
生成FFmpeg滤镜参数
Args:
video_input: 输入视频流标识符 (例如: "[0:v]", "[v_eff1]")
effect_index: 效果索引,用于生成唯一的输出标识符
Returns:
tuple: (filter_args_list, output_stream_identifier)
"""
pass
@abstractmethod
def get_effect_name(self) -> str:
"""获取效果名称"""
pass
def parse_params(self) -> List[str]:
"""解析参数字符串为列表"""
if not self.params:
return []
return self.params.split(",")
def get_pos_json(self) -> Dict[str, Any]:
"""获取位置JSON数据"""
pos_json_str = self.ext_data.get("posJson", "{}")
try:
return json.loads(pos_json_str) if pos_json_str != "{}" else {}
except Exception as e:
logger.warning(f"Failed to parse posJson: {e}")
return {}
class EffectRegistry:
"""效果处理器注册表"""
def __init__(self):
self._processors: Dict[str, Type[EffectProcessor]] = {}
def register(self, name: str, processor_class: Type[EffectProcessor]):
"""注册效果处理器"""
if not issubclass(processor_class, EffectProcessor):
raise ValueError(f"{processor_class} must be a subclass of EffectProcessor")
self._processors[name] = processor_class
logger.debug(f"Registered effect processor: {name}")
def get_processor(
self,
effect_name: str,
params: str = "",
ext_data: Optional[Dict[str, Any]] = None,
) -> Optional[EffectProcessor]:
"""获取效果处理器实例"""
if effect_name not in self._processors:
logger.warning(f"Unknown effect: {effect_name}")
return None
processor_class = self._processors[effect_name]
return processor_class(params, ext_data)
def list_effects(self) -> List[str]:
"""列出所有注册的效果"""
return list(self._processors.keys())
def parse_effect_string(self, effect_string: str) -> tuple[str, str]:
"""
解析效果字符串
Args:
effect_string: 效果字符串,格式为 "effect_name:params"
Returns:
tuple: (effect_name, params)
"""
if ":" in effect_string:
parts = effect_string.split(":", 2)
return parts[0], parts[1] if len(parts) > 1 else ""
return effect_string, ""

View File

@@ -0,0 +1,99 @@
from typing import List
from .base import EffectProcessor
class CameraShotEffect(EffectProcessor):
"""相机镜头效果处理器"""
def validate_params(self) -> bool:
"""验证参数:start_time,duration,rotate_deg"""
params = self.parse_params()
if not params:
return True # 使用默认参数
# 参数格式: "start_time,duration,rotate_deg"
if len(params) > 3:
return False
try:
for i, param in enumerate(params):
if param == "":
continue
if i == 2: # rotate_deg
int(param)
else: # start_time, duration
float(param)
return True
except ValueError:
return False
def generate_filter_args(
self, video_input: str, effect_index: int
) -> tuple[List[str], str]:
"""生成相机镜头效果的滤镜参数"""
if not self.validate_params():
return [], video_input
params = self.parse_params()
# 设置默认值
start = 3.0
duration = 1.0
rotate_deg = 0
if len(params) >= 1 and params[0] != "":
start = float(params[0])
if len(params) >= 2 and params[1] != "":
duration = float(params[1])
if len(params) >= 3 and params[2] != "":
rotate_deg = int(params[2])
filter_args = []
# 生成输出流标识符
start_out_str = "[eff_s]"
mid_out_str = "[eff_m]"
end_out_str = "[eff_e]"
final_output = f"[v_eff{effect_index}]"
# 分割视频流为三部分
filter_args.append(
f"{video_input}split=3{start_out_str}{mid_out_str}{end_out_str}"
)
# 选择开始部分帧
filter_args.append(
f"{start_out_str}select=lt(n\\,"
f"{int(start * self.frame_rate)}){start_out_str}"
)
# 选择结束部分帧
filter_args.append(
f"{end_out_str}select=gt(n\\,"
f"{int(start * self.frame_rate)}){end_out_str}"
)
# 选择中间特定帧并扩展
filter_args.append(
f"{mid_out_str}select=eq(n\\,"
f"{int(start * self.frame_rate)}){mid_out_str}"
)
filter_args.append(
f"{mid_out_str}tpad=start_mode=clone:"
f"start_duration={duration:.4f}{mid_out_str}"
)
# 如果需要旋转
if rotate_deg != 0:
filter_args.append(f"{mid_out_str}rotate=PI*{rotate_deg}/180{mid_out_str}")
# 连接三部分
filter_args.append(
f"{start_out_str}{mid_out_str}{end_out_str}concat=n=3:v=1:a=0,"
f"setpts=N/{self.frame_rate}/TB{final_output}"
)
return filter_args, final_output
def get_effect_name(self) -> str:
return "cameraShot"

41
entity/effects/skip.py Normal file
View File

@@ -0,0 +1,41 @@
from typing import List
from .base import EffectProcessor
class SkipEffect(EffectProcessor):
"""跳过开头效果处理器"""
def validate_params(self) -> bool:
"""验证参数:跳过的秒数"""
if not self.params:
return True # 默认不跳过
try:
skip_seconds = float(self.params)
return skip_seconds >= 0
except ValueError:
return False
def generate_filter_args(
self, video_input: str, effect_index: int
) -> tuple[List[str], str]:
"""生成跳过开头效果的滤镜参数"""
if not self.validate_params():
return [], video_input
if not self.params:
return [], video_input
skip_seconds = float(self.params)
if skip_seconds <= 0:
return [], video_input
output_stream = f"[v_eff{effect_index}]"
# 使用trim滤镜跳过开头
filter_args = [f"{video_input}trim=start={skip_seconds}{output_stream}"]
return filter_args, output_stream
def get_effect_name(self) -> str:
return "skip"

38
entity/effects/speed.py Normal file
View File

@@ -0,0 +1,38 @@
from typing import List
from .base import EffectProcessor
class SpeedEffect(EffectProcessor):
"""视频变速效果处理器"""
def validate_params(self) -> bool:
"""验证参数:速度倍数"""
if not self.params:
return True # 默认不变速
try:
speed = float(self.params)
return speed > 0
except ValueError:
return False
def generate_filter_args(
self, video_input: str, effect_index: int
) -> tuple[List[str], str]:
"""生成变速效果的滤镜参数"""
if not self.validate_params():
return [], video_input
if not self.params or self.params == "1":
return [], video_input # 不需要变速
speed = float(self.params)
output_stream = f"[v_eff{effect_index}]"
# 使用setpts进行变速
filter_args = [f"{video_input}setpts={speed}*PTS{output_stream}"]
return filter_args, output_stream
def get_effect_name(self) -> str:
return "ospeed"

46
entity/effects/tail.py Normal file
View File

@@ -0,0 +1,46 @@
from typing import List
from .base import EffectProcessor
class TailEffect(EffectProcessor):
"""保留末尾效果处理器"""
def validate_params(self) -> bool:
"""验证参数:保留的秒数"""
if not self.params:
return True # 默认不截取
try:
tail_seconds = float(self.params)
return tail_seconds >= 0
except ValueError:
return False
def generate_filter_args(
self, video_input: str, effect_index: int
) -> tuple[List[str], str]:
"""生成保留末尾效果的滤镜参数"""
if not self.validate_params():
return [], video_input
if not self.params:
return [], video_input
tail_seconds = float(self.params)
if tail_seconds <= 0:
return [], video_input
output_stream = f"[v_eff{effect_index}]"
# 使用reverse+trim+reverse的方法来精确获取最后N秒
filter_args = [
f"{video_input}reverse[v_rev{effect_index}]",
f"[v_rev{effect_index}]trim=duration={tail_seconds}"
f"[v_trim{effect_index}]",
f"[v_trim{effect_index}]reverse{output_stream}",
]
return filter_args, output_stream
def get_effect_name(self) -> str:
return "tail"

89
entity/effects/zoom.py Normal file
View File

@@ -0,0 +1,89 @@
from typing import List
from .base import EffectProcessor
class ZoomEffect(EffectProcessor):
"""缩放效果处理器"""
def validate_params(self) -> bool:
"""验证参数:start_time,zoom_factor,duration"""
params = self.parse_params()
if len(params) < 3:
return False
try:
start_time = float(params[0])
zoom_factor = float(params[1])
duration = float(params[2])
return start_time >= 0 and zoom_factor > 0 and duration >= 0
except (ValueError, IndexError):
return False
def generate_filter_args(
self, video_input: str, effect_index: int
) -> tuple[List[str], str]:
"""生成缩放效果的滤镜参数"""
if not self.validate_params():
return [], video_input
params = self.parse_params()
start_time = float(params[0])
zoom_factor = float(params[1])
duration = float(params[2])
if zoom_factor == 1:
return [], video_input # 不需要缩放
output_stream = f"[v_eff{effect_index}]"
# 获取缩放中心点
center_x, center_y = self._get_zoom_center()
filter_args = []
if duration == 0:
# 静态缩放(整个视频时长)
x_expr = f"({center_x})-(ow*zoom)/2"
y_expr = f"({center_y})-(oh*zoom)/2"
filter_args.append(
f"{video_input}trim=start={start_time},zoompan=z={zoom_factor}:x={x_expr}:y={y_expr}:d=1{output_stream}"
)
else:
# 动态缩放(指定时间段内)
zoom_expr = f"if(between(t\\,{start_time}\\,{start_time + duration})\\,{zoom_factor}\\,1)"
x_expr = f"({center_x})-(ow*zoom)/2"
y_expr = f"({center_y})-(oh*zoom)/2"
filter_args.append(
f"{video_input}zoompan=z={zoom_expr}:x={x_expr}:y={y_expr}:d=1{output_stream}"
)
return filter_args, output_stream
def _get_zoom_center(self) -> tuple[str, str]:
"""获取缩放中心点坐标表达式"""
# 默认中心点
center_x = "iw/2"
center_y = "ih/2"
pos_json = self.get_pos_json()
if pos_json:
_f_x = pos_json.get("ltX", 0)
_f_x2 = pos_json.get("rbX", 0)
_f_y = pos_json.get("ltY", 0)
_f_y2 = pos_json.get("rbY", 0)
_v_w = pos_json.get("imgWidth", 1)
_v_h = pos_json.get("imgHeight", 1)
if _v_w > 0 and _v_h > 0:
# 计算坐标系统中的中心点
center_x_ratio = (_f_x + _f_x2) / (2 * _v_w)
center_y_ratio = (_f_y + _f_y2) / (2 * _v_h)
# 转换为视频坐标系统
center_x = f"iw*{center_x_ratio:.6f}"
center_y = f"ih*{center_y_ratio:.6f}"
return center_x, center_y
def get_effect_name(self) -> str:
return "zoom"

View File

@@ -1,14 +1,41 @@
import json
# 保留用于向后兼容的常量定义
import os
import time
import uuid
from typing import Any
DEFAULT_ARGS = ("-shortest",)
ENCODER_ARGS = ("-c:v", "h264", ) if not os.getenv("ENCODER_ARGS", False) else os.getenv("ENCODER_ARGS", "").split(" ")
VIDEO_ARGS = ("-profile:v", "high", "-level:v", "4", ) if not os.getenv("VIDEO_ARGS", False) else os.getenv("VIDEO_ARGS", "").split(" ")
AUDIO_ARGS = ("-c:a", "aac", "-b:a", "128k", "-ar", "48000", "-ac", "2", )
MUTE_AUDIO_INPUT = ("-f", "lavfi", "-i", "anullsrc=cl=stereo:r=48000", )
ENCODER_ARGS = (
(
"-c:v",
"h264",
)
if not os.getenv("ENCODER_ARGS", False)
else os.getenv("ENCODER_ARGS", "").split(" ")
)
VIDEO_ARGS = (
(
"-profile:v",
"high",
"-level:v",
"4",
)
if not os.getenv("VIDEO_ARGS", False)
else os.getenv("VIDEO_ARGS", "").split(" ")
)
AUDIO_ARGS = (
"-c:a",
"aac",
"-b:a",
"128k",
"-ar",
"48000",
"-ac",
"2",
)
MUTE_AUDIO_INPUT = (
"-f",
"lavfi",
"-i",
"anullsrc=cl=stereo:r=48000",
)
def get_mp4toannexb_filter():
@@ -23,10 +50,13 @@ def get_mp4toannexb_filter():
class FfmpegTask(object):
"""
兼容类:保留原有FfmpegTask接口用于向后兼容
实际处理逻辑已迁移到新架构,该类主要用作数据载体
"""
effects: list[str]
def __init__(self, input_file, task_type='copy', output_file=''):
def __init__(self, input_file, task_type="copy", output_file=""):
"""保持原有构造函数签名"""
self.annexb = False
if type(input_file) is str:
if input_file.endswith(".ts"):
@@ -40,7 +70,7 @@ class FfmpegTask(object):
self.center_cut = None
self.ext_data = {}
self.task_type = task_type
self.output_file = output_file
self.output_file = output_file or ""
self.mute = True
self.speed = 1
self.frame_rate = 25
@@ -52,456 +82,135 @@ class FfmpegTask(object):
self.effects = []
def __repr__(self):
_str = f'FfmpegTask(input_file={self.input_file}, task_type={self.task_type}'
if len(self.luts) > 0:
_str += f', luts={self.luts}'
if len(self.audios) > 0:
_str += f', audios={self.audios}'
if len(self.overlays) > 0:
_str += f', overlays={self.overlays}'
if self.annexb:
_str += f', annexb={self.annexb}'
if self.effects:
_str += f', effects={self.effects}'
if self.mute:
_str += f', mute={self.mute}'
_str += f', center_cut={self.center_cut}'
return _str + ')'
return f"FfmpegTask(input_file={self.input_file}, task_type={self.task_type})"
def analyze_input_render_tasks(self):
"""分析输入中的子任务"""
for i in self.input_file:
if type(i) is str:
continue
elif isinstance(i, FfmpegTask):
if i.need_run():
yield i
if isinstance(i, FfmpegTask) and i.need_run():
yield i
def need_run(self):
"""
判断是否需要运行
:rtype: bool
:return:
"""
"""判断是否需要运行"""
if self.annexb:
return True
# TODO: copy from url
return not self.check_can_copy()
def add_inputs(self, *inputs):
"""添加输入文件"""
self.input_file.extend(inputs)
def add_overlay(self, *overlays):
"""添加覆盖层"""
for overlay in overlays:
if str(overlay).endswith('.ass'):
if str(overlay).endswith(".ass"):
self.subtitles.append(overlay)
else:
self.overlays.append(overlay)
self.correct_task_type()
def add_audios(self, *audios):
"""添加音频"""
self.audios.extend(audios)
self.correct_task_type()
self.check_audio_track()
def add_lut(self, *luts):
"""添加LUT"""
self.luts.extend(luts)
self.correct_task_type()
def add_effect(self, *effects):
"""添加效果"""
self.effects.extend(effects)
self.correct_task_type()
def get_output_file(self):
if self.task_type == 'copy':
return self.input_file[0]
if self.output_file == '':
"""获取输出文件"""
if self.task_type == "copy":
return self.input_file[0] if self.input_file else ""
if not self.output_file:
self.set_output_file()
return self.output_file
def correct_task_type(self):
"""校正任务类型"""
if self.check_can_copy():
self.task_type = 'copy'
self.task_type = "copy"
elif self.check_can_concat():
self.task_type = 'concat'
self.task_type = "concat"
else:
self.task_type = 'encode'
self.task_type = "encode"
def check_can_concat(self):
if len(self.luts) > 0:
return False
if len(self.overlays) > 0:
return False
if len(self.subtitles) > 0:
return False
if len(self.effects) > 0:
return False
if self.speed != 1:
return False
if self.zoom_cut is not None:
return False
if self.center_cut is not None:
return False
return True
"""检查是否可以连接"""
return (
len(self.luts) == 0
and len(self.overlays) == 0
and len(self.subtitles) == 0
and len(self.effects) == 0
and self.speed == 1
and self.zoom_cut is None
and self.center_cut is None
)
def check_can_copy(self):
if len(self.luts) > 0:
return False
if len(self.overlays) > 0:
return False
if len(self.subtitles) > 0:
return False
if len(self.effects) > 0:
return False
if self.speed != 1:
return False
if len(self.audios) >= 1:
return False
if len(self.input_file) > 1:
return False
if self.zoom_cut is not None:
return False
if self.center_cut is not None:
return False
return True
def check_audio_track(self):
...
def get_ffmpeg_args(self):
args = ['-y', '-hide_banner']
if self.task_type == 'encode':
input_args = []
filter_args = []
output_args = [*VIDEO_ARGS, *AUDIO_ARGS, *ENCODER_ARGS, *DEFAULT_ARGS]
if self.annexb:
output_args.append("-bsf:v")
output_args.append(get_mp4toannexb_filter())
output_args.append("-reset_timestamps")
output_args.append("1")
video_output_str = "[0:v]"
audio_output_str = ""
audio_track_index = 0
effect_index = 0
for input_file in self.input_file:
input_args.append("-i")
if type(input_file) is str:
input_args.append(input_file)
elif isinstance(input_file, FfmpegTask):
input_args.append(input_file.get_output_file())
if self.center_cut == 1:
pos_json_str = self.ext_data.get('posJson', '{}')
try:
pos_json = json.loads(pos_json_str)
except Exception as e:
pos_json = {}
_v_w = pos_json.get('imgWidth', 1)
_f_x = pos_json.get('ltX', 0)
_f_x2 = pos_json.get('rbX', 0)
_x = f'{float((_f_x2 + _f_x)/(2 * _v_w)) :.4f}*iw-ih*ih/(2*iw)'
filter_args.append(f"{video_output_str}crop=x={_x}:y=0:w=ih*ih/iw:h=ih[v_cut{effect_index}]")
video_output_str = f"[v_cut{effect_index}]"
effect_index += 1
if self.zoom_cut == 1 and self.resolution:
_input = None
for input_file in self.input_file:
if type(input_file) is str:
_input = input_file
break
elif isinstance(input_file, FfmpegTask):
_input = input_file.get_output_file()
break
if _input:
from util.ffmpeg import probe_video_info
_iw, _ih, _ = probe_video_info(_input)
_w, _h = self.resolution.split('x', 1)
pos_json_str = self.ext_data.get('posJson', '{}')
try:
pos_json = json.loads(pos_json_str)
except Exception as e:
pos_json = {}
_v_w = pos_json.get('imgWidth', 1)
_v_h = pos_json.get('imgHeight', 1)
_f_x = pos_json.get('ltX', 0)
_f_x2 = pos_json.get('rbX', 0)
_f_y = pos_json.get('ltY', 0)
_f_y2 = pos_json.get('rbY', 0)
_x = min(max(0, int((_f_x + _f_x2) / 2 - int(_w) / 2)), _iw - int(_w))
_y = min(max(0, int((_f_y + _f_y2) / 2 - int(_h) / 2)), _ih - int(_h))
filter_args.append(f"{video_output_str}crop=x={_x}:y={_y}:w={_w}:h={_h}[vz_cut{effect_index}]")
video_output_str = f"[vz_cut{effect_index}]"
effect_index += 1
for effect in self.effects:
if effect.startswith("cameraShot:"):
param = effect.split(":", 2)[1]
if param == '':
param = "3,1,0"
_split = param.split(",")
start = 3
duration = 1
rotate_deg = 0
if len(_split) >= 3:
if _split[2] == '':
rotate_deg = 0
else:
rotate_deg = int(_split[2])
if len(_split) >= 2:
duration = float(_split[1])
if len(_split) >= 1:
start = float(_split[0])
_start_out_str = "[eff_s]"
_mid_out_str = "[eff_m]"
_end_out_str = "[eff_e]"
filter_args.append(f"{video_output_str}split=3{_start_out_str}{_mid_out_str}{_end_out_str}")
filter_args.append(f"{_start_out_str}select=lt(n\\,{int(start * self.frame_rate)}){_start_out_str}")
filter_args.append(f"{_end_out_str}select=gt(n\\,{int(start * self.frame_rate)}){_end_out_str}")
filter_args.append(f"{_mid_out_str}select=eq(n\\,{int(start * self.frame_rate)}){_mid_out_str}")
filter_args.append(
f"{_mid_out_str}tpad=start_mode=clone:start_duration={duration:.4f}{_mid_out_str}")
if rotate_deg != 0:
filter_args.append(f"{_mid_out_str}rotate=PI*{rotate_deg}/180{_mid_out_str}")
# filter_args.append(f"{video_output_str}trim=start=0:end={start+duration},tpad=stop_mode=clone:stop_duration={duration},setpts=PTS-STARTPTS{_start_out_str}")
# filter_args.append(f"tpad=start_mode=clone:start_duration={duration},setpts=PTS-STARTPTS{_start_out_str}")
# filter_args.append(f"{_end_out_str}trim=start={start}{_end_out_str}")
video_output_str = f"[v_eff{effect_index}]"
# filter_args.append(f"{_end_out_str}{_start_out_str}overlay=eof_action=pass{video_output_str}")
filter_args.append(f"{_start_out_str}{_mid_out_str}{_end_out_str}concat=n=3:v=1:a=0,setpts=N/{self.frame_rate}/TB{video_output_str}")
effect_index += 1
elif effect.startswith("ospeed:"):
param = effect.split(":", 2)[1]
if param == '':
param = "1"
if param != "1":
# 视频变速
effect_index += 1
filter_args.append(f"{video_output_str}setpts={param}*PTS[v_eff{effect_index}]")
video_output_str = f"[v_eff{effect_index}]"
elif effect.startswith("zoom:"):
param = effect.split(":", 2)[1]
if param == '':
continue
_split = param.split(",")
if len(_split) < 3:
continue
try:
start_time = float(_split[0])
zoom_factor = float(_split[1])
duration = float(_split[2])
if start_time < 0:
start_time = 0
if duration < 0:
duration = 0
if zoom_factor <= 0:
zoom_factor = 1
except (ValueError, IndexError):
start_time = 0
duration = 0
zoom_factor = 1
if zoom_factor == 1:
continue
effect_index += 1
# 获取缩放中心点(从pos_json或使用默认中心)
center_x = "iw/2"
center_y = "ih/2"
pos_json_str = self.ext_data.get('posJson', '{}')
try:
pos_json = json.loads(pos_json_str) if pos_json_str != '{}' else {}
if pos_json:
_f_x = pos_json.get('ltX', 0)
_f_x2 = pos_json.get('rbX', 0)
_f_y = pos_json.get('ltY', 0)
_f_y2 = pos_json.get('rbY', 0)
_v_w = pos_json.get('imgWidth', 1)
_v_h = pos_json.get('imgHeight', 1)
if _v_w > 0 and _v_h > 0:
# 计算坐标系统中的中心点
center_x_ratio = (_f_x + _f_x2) / (2 * _v_w)
center_y_ratio = (_f_y + _f_y2) / (2 * _v_h)
# 转换为视频坐标系统
center_x = f"iw*{center_x_ratio:.6f}"
center_y = f"ih*{center_y_ratio:.6f}"
except Exception as e:
# 解析失败使用默认中心
pass
if duration == 0:
# 静态缩放(整个视频时长)
x_expr = f"({center_x})-(ow*zoom)/2"
y_expr = f"({center_y})-(oh*zoom)/2"
filter_args.append(f"{video_output_str}trim=start={start_time},zoompan=z={zoom_factor}:x={x_expr}:y={y_expr}:d=1[v_eff{effect_index}]")
else:
# 动态缩放(指定时间段内)
zoom_expr = f"if(between(t\\,{start_time}\\,{start_time + duration})\\,{zoom_factor}\\,1)"
x_expr = f"({center_x})-(ow*zoom)/2"
y_expr = f"({center_y})-(oh*zoom)/2"
filter_args.append(f"{video_output_str}zoompan=z={zoom_expr}:x={x_expr}:y={y_expr}:d=1[v_eff{effect_index}]")
video_output_str = f"[v_eff{effect_index}]"
elif effect.startswith("skip:"):
param = effect.split(":", 2)[1]
if param == '':
param = "0"
skip_seconds = float(param)
if skip_seconds > 0:
effect_index += 1
filter_args.append(f"{video_output_str}trim=start={skip_seconds}[v_eff{effect_index}]")
video_output_str = f"[v_eff{effect_index}]"
elif effect.startswith("tail:"):
param = effect.split(":", 2)[1]
if param == '':
param = "0"
tail_seconds = float(param)
if tail_seconds > 0:
effect_index += 1
# 首先获取视频总时长,然后计算开始时间
# 使用reverse+trim+reverse的方法来精确获取最后N秒
filter_args.append(f"{video_output_str}reverse[v_rev{effect_index}]")
filter_args.append(f"[v_rev{effect_index}]trim=duration={tail_seconds}[v_trim{effect_index}]")
filter_args.append(f"[v_trim{effect_index}]reverse[v_eff{effect_index}]")
video_output_str = f"[v_eff{effect_index}]"
...
if self.resolution:
filter_args.append(f"{video_output_str}scale={self.resolution.replace('x', ':')}[v]")
video_output_str = "[v]"
for lut in self.luts:
filter_args.append(f"{video_output_str}lut3d=file={lut}{video_output_str}")
for overlay in self.overlays:
input_index = input_args.count("-i")
input_args.append("-i")
input_args.append(overlay)
if os.getenv("OLD_FFMPEG"):
filter_args.append(f"{video_output_str}[{input_index}:v]scale2ref=iw:ih[v]")
else:
filter_args.append(f"{video_output_str}[{input_index}:v]scale=rw:rh[v]")
filter_args.append(f"[v][{input_index}:v]overlay=1:eof_action=endall[v]")
video_output_str = "[v]"
for subtitle in self.subtitles:
filter_args.append(f"{video_output_str}ass={subtitle}[v]")
video_output_str = "[v]"
output_args.append("-map")
output_args.append(video_output_str)
output_args.append("-r")
output_args.append(f"{self.frame_rate}")
output_args.append("-fps_mode")
output_args.append("cfr")
if self.mute:
input_index = input_args.count("-i")
input_args += MUTE_AUDIO_INPUT
filter_args.append(f"[{input_index}:a]acopy[a]")
audio_track_index += 1
audio_output_str = "[a]"
else:
audio_output_str = "[0:a]"
audio_track_index += 1
for audio in self.audios:
input_index = input_args.count("-i")
input_args.append("-i")
input_args.append(audio.replace("\\", "/"))
audio_track_index += 1
filter_args.append(f"{audio_output_str}[{input_index}:a]amix=duration=shortest:dropout_transition=0:normalize=0[a]")
audio_output_str = "[a]"
if audio_output_str:
output_args.append("-map")
output_args.append(audio_output_str)
_filter_args = [] if len(filter_args) == 0 else ["-filter_complex", ";".join(filter_args)]
return args + input_args + _filter_args + output_args + [self.get_output_file()]
elif self.task_type == 'concat':
# 无法通过 annexb 合并的
input_args = []
output_args = [*DEFAULT_ARGS]
filter_args = []
audio_output_str = ""
audio_track_index = 0
# output_args
if len(self.input_file) == 1:
_file = self.input_file[0]
from util.ffmpeg import probe_video_audio
if type(_file) is str:
input_args += ["-i", _file]
self.mute = not probe_video_audio(_file)
elif isinstance(_file, FfmpegTask):
input_args += ["-i", _file.get_output_file()]
self.mute = not probe_video_audio(_file.get_output_file())
else:
_tmp_file = "tmp_concat_" + str(time.time()) + ".txt"
from util.ffmpeg import probe_video_audio
with open(_tmp_file, "w", encoding="utf-8") as f:
for input_file in self.input_file:
if type(input_file) is str:
f.write("file '" + input_file + "'\n")
elif isinstance(input_file, FfmpegTask):
f.write("file '" + input_file.get_output_file() + "'\n")
input_args += ["-f", "concat", "-safe", "0", "-i", _tmp_file]
self.mute = not probe_video_audio(_tmp_file, "concat")
output_args.append("-map")
output_args.append("0:v")
output_args.append("-c:v")
output_args.append("copy")
if self.mute:
input_index = input_args.count("-i")
input_args += MUTE_AUDIO_INPUT
audio_output_str = f"[{input_index}:a]"
audio_track_index += 1
else:
audio_output_str = "[0:a]"
audio_track_index += 1
for audio in self.audios:
input_index = input_args.count("-i")
input_args.append("-i")
input_args.append(audio.replace("\\", "/"))
audio_track_index += 1
filter_args.append(f"{audio_output_str}[{input_index}:a]amix=duration=shortest:dropout_transition=0:normalize=0[a]")
audio_output_str = "[a]"
if audio_output_str:
output_args.append("-map")
if audio_track_index <= 1:
output_args.append(audio_output_str[1:-1])
else:
output_args.append(audio_output_str)
output_args += AUDIO_ARGS
if self.annexb:
output_args.append("-bsf:v")
output_args.append(get_mp4toannexb_filter())
output_args.append("-bsf:a")
output_args.append("setts=pts=DTS")
output_args.append("-f")
output_args.append("mpegts" if self.annexb else "mp4")
_filter_args = [] if len(filter_args) == 0 else ["-filter_complex", ";".join(filter_args)]
return args + input_args + _filter_args + output_args + [self.get_output_file()]
elif self.task_type == 'copy':
if len(self.input_file) == 1:
if type(self.input_file[0]) is str:
if self.input_file[0] == self.get_output_file():
return []
return args + ["-i", self.input_file[0]] + ["-c", "copy", self.get_output_file()]
return []
"""检查是否可以复制"""
return (
len(self.luts) == 0
and len(self.overlays) == 0
and len(self.subtitles) == 0
and len(self.effects) == 0
and self.speed == 1
and len(self.audios) == 0
and len(self.input_file) <= 1
and self.zoom_cut is None
and self.center_cut is None
)
def set_output_file(self, file=None):
"""设置输出文件"""
if file is None:
if self.output_file == '':
if self.annexb:
self.output_file = "rand_" + str(uuid.uuid4()) + ".ts"
else:
self.output_file = "rand_" + str(uuid.uuid4()) + ".mp4"
import uuid
if self.annexb:
self.output_file = f"rand_{uuid.uuid4()}.ts"
else:
self.output_file = f"rand_{uuid.uuid4()}.mp4"
else:
if isinstance(file, FfmpegTask):
if file == self:
return
self.output_file = file.get_output_file()
if type(file) is str:
if file != self:
self.output_file = file.get_output_file()
elif isinstance(file, str):
self.output_file = file
def check_annexb(self):
for input_file in self.input_file:
if type(input_file) is str:
if self.task_type == 'encode':
return self.annexb
elif self.task_type == 'concat':
return False
elif self.task_type == 'copy':
return self.annexb
else:
return False
elif isinstance(input_file, FfmpegTask):
if not input_file.check_annexb():
return False
return True
"""检查annexb格式"""
return self.annexb
def get_ffmpeg_args(self):
"""
保留用于向后兼容,但实际逻辑已迁移到新架构
建议使用新的 FFmpegCommandBuilder 来生成命令
"""
# 简化版本,主要用于向后兼容
if self.task_type == "copy" and len(self.input_file) == 1:
if isinstance(self.input_file[0], str):
if self.input_file[0] == self.get_output_file():
return []
return [
"-y",
"-hide_banner",
"-i",
self.input_file[0],
"-c",
"copy",
self.get_output_file(),
]
# 对于复杂情况,返回基础命令结构
# 实际处理会在新的服务架构中完成
return (
["-y", "-hide_banner", "-i"]
+ self.input_file
+ ["-c", "copy", self.get_output_file()]
)

View File

@@ -0,0 +1,314 @@
import time
from typing import List, Optional
from config.settings import get_ffmpeg_config
from entity.render_task import RenderTask, TaskType
from entity.effects import registry as effect_registry
from util.exceptions import FFmpegError
from util.ffmpeg import probe_video_info, probe_video_audio
from util.ffmpeg_utils import (
build_base_ffmpeg_args,
build_null_audio_input,
get_annexb_filter,
build_standard_output_args,
)
from util.json_utils import safe_json_loads
import logging
logger = logging.getLogger(__name__)
class FFmpegCommandBuilder:
"""FFmpeg命令构建器"""
def __init__(self, task: RenderTask):
self.task = task
self.config = get_ffmpeg_config()
def build_command(self) -> List[str]:
"""构建FFmpeg命令"""
self.task.update_task_type()
if self.task.task_type == TaskType.COPY:
return self._build_copy_command()
elif self.task.task_type == TaskType.CONCAT:
return self._build_concat_command()
elif self.task.task_type == TaskType.ENCODE:
return self._build_encode_command()
else:
raise FFmpegError(f"Unsupported task type: {self.task.task_type}")
def _build_copy_command(self) -> List[str]:
"""构建复制命令"""
if len(self.task.input_files) == 1:
input_file = self.task.input_files[0]
if input_file == self.task.output_file:
return [] # 不需要处理
return [
"ffmpeg",
"-y",
"-hide_banner",
"-i",
self.task.input_files[0],
"-c",
"copy",
self.task.output_file,
]
def _build_concat_command(self) -> List[str]:
"""构建拼接命令"""
args = ["ffmpeg", "-y", "-hide_banner"]
input_args = []
output_args = [*self.config.default_args]
filter_args = []
if len(self.task.input_files) == 1:
# 单个文件
file = self.task.input_files[0]
input_args.extend(["-i", file])
self.task.mute = not probe_video_audio(file)
else:
# 多个文件使用concat协议
tmp_file = f"tmp_concat_{time.time()}.txt"
with open(tmp_file, "w", encoding="utf-8") as f:
for input_file in self.task.input_files:
f.write(f"file '{input_file}'\n")
input_args.extend(["-f", "concat", "-safe", "0", "-i", tmp_file])
self.task.mute = not probe_video_audio(tmp_file, "concat")
# 视频流映射
output_args.extend(["-map", "0:v", "-c:v", "copy"])
# 音频处理
audio_output_str = self._handle_audio_concat(input_args, filter_args)
if audio_output_str:
output_args.extend(["-map", audio_output_str])
output_args.extend(self.config.audio_args)
# annexb处理
if self.task.annexb:
output_args.extend(["-bsf:v", self._get_mp4toannexb_filter()])
output_args.extend(["-bsf:a", "setts=pts=DTS"])
output_args.extend(["-f", "mpegts"])
else:
output_args.extend(["-f", "mp4"])
filter_complex = (
["-filter_complex", ";".join(filter_args)] if filter_args else []
)
return (
args + input_args + filter_complex + output_args + [self.task.output_file]
)
def _build_encode_command(self) -> List[str]:
"""构建编码命令"""
args = build_base_ffmpeg_args()
input_args = []
filter_args = []
output_args = build_standard_output_args()
# annexb处理
if self.task.annexb:
output_args.extend(["-bsf:v", get_annexb_filter()])
output_args.extend(["-reset_timestamps", "1"])
# 处理输入文件
for input_file in self.task.input_files:
input_args.extend(["-i", input_file])
# 处理视频流
video_output_str = "[0:v]"
effect_index = 0
# 处理中心裁剪
if self.task.center_cut == 1:
video_output_str, effect_index = self._add_center_cut(
filter_args, video_output_str, effect_index
)
# 处理缩放裁剪
if self.task.zoom_cut == 1 and self.task.resolution:
video_output_str, effect_index = self._add_zoom_cut(
filter_args, video_output_str, effect_index
)
# 处理效果
video_output_str, effect_index = self._add_effects(
filter_args, video_output_str, effect_index
)
# 处理分辨率
if self.task.resolution:
filter_args.append(
f"{video_output_str}scale={self.task.resolution.replace('x', ':')}[v]"
)
video_output_str = "[v]"
# 处理LUT
for lut in self.task.luts:
filter_args.append(f"{video_output_str}lut3d=file={lut}{video_output_str}")
# 处理覆盖层
video_output_str = self._add_overlays(input_args, filter_args, video_output_str)
# 处理字幕
for subtitle in self.task.subtitles:
filter_args.append(f"{video_output_str}ass={subtitle}[v]")
video_output_str = "[v]"
# 映射视频流
output_args.extend(["-map", video_output_str])
output_args.extend(["-r", str(self.task.frame_rate)])
output_args.extend(["-fps_mode", "cfr"])
# 处理音频
audio_output_str = self._handle_audio_encode(input_args, filter_args)
if audio_output_str:
output_args.extend(["-map", audio_output_str])
filter_complex = (
["-filter_complex", ";".join(filter_args)] if filter_args else []
)
return (
args + input_args + filter_complex + output_args + [self.task.output_file]
)
def _add_center_cut(
self, filter_args: List[str], video_input: str, effect_index: int
) -> tuple[str, int]:
"""添加中心裁剪"""
pos_json = self.task.ext_data.get("posJson", "{}")
pos_data = safe_json_loads(pos_json, {})
_v_w = pos_data.get("imgWidth", 1)
_f_x = pos_data.get("ltX", 0)
_f_x2 = pos_data.get("rbX", 0)
_x = f"{float((_f_x2 + _f_x)/(2 * _v_w)):.4f}*iw-ih*ih/(2*iw)"
filter_args.append(
f"{video_input}crop=x={_x}:y=0:w=ih*ih/iw:h=ih[v_cut{effect_index}]"
)
return f"[v_cut{effect_index}]", effect_index + 1
def _add_zoom_cut(
self, filter_args: List[str], video_input: str, effect_index: int
) -> tuple[str, int]:
"""添加缩放裁剪"""
# 获取输入视频尺寸
input_file = self.task.input_files[0]
_iw, _ih, _ = probe_video_info(input_file)
_w, _h = self.task.resolution.split("x", 1)
pos_json = self.task.ext_data.get("posJson", "{}")
pos_data = safe_json_loads(pos_json, {})
_f_x = pos_data.get("ltX", 0)
_f_x2 = pos_data.get("rbX", 0)
_f_y = pos_data.get("ltY", 0)
_f_y2 = pos_data.get("rbY", 0)
_x = min(max(0, int((_f_x + _f_x2) / 2 - int(_w) / 2)), _iw - int(_w))
_y = min(max(0, int((_f_y + _f_y2) / 2 - int(_h) / 2)), _ih - int(_h))
filter_args.append(
f"{video_input}crop=x={_x}:y={_y}:w={_w}:h={_h}[vz_cut{effect_index}]"
)
return f"[vz_cut{effect_index}]", effect_index + 1
def _add_effects(
self, filter_args: List[str], video_input: str, effect_index: int
) -> tuple[str, int]:
"""添加效果处理"""
current_input = video_input
for effect_str in self.task.effects:
effect_name, params = effect_registry.parse_effect_string(effect_str)
processor = effect_registry.get_processor(
effect_name, params, self.task.ext_data
)
if processor:
processor.frame_rate = self.task.frame_rate
effect_filters, output_stream = processor.generate_filter_args(
current_input, effect_index
)
if effect_filters:
filter_args.extend(effect_filters)
current_input = output_stream
effect_index += 1
return current_input, effect_index
def _add_overlays(
self, input_args: List[str], filter_args: List[str], video_input: str
) -> str:
"""添加覆盖层"""
current_input = video_input
for overlay in self.task.overlays:
input_index = input_args.count("-i") // 2 # 每个输入占两个参数 -i filename
input_args.extend(["-i", overlay])
if self.config.overlay_scale_mode == "scale":
filter_args.append(f"{current_input}[{input_index}:v]scale=iw:ih[v]")
else:
filter_args.append(
f"{current_input}[{input_index}:v]{self.config.overlay_scale_mode}=iw:ih[v]"
)
filter_args.append(f"[v][{input_index}:v]overlay=1:eof_action=endall[v]")
current_input = "[v]"
return current_input
def _handle_audio_concat(
self, input_args: List[str], filter_args: List[str]
) -> Optional[str]:
"""处理concat模式的音频"""
audio_output_str = ""
if self.task.mute:
input_index = input_args.count("-i") // 2
input_args.extend(build_null_audio_input())
audio_output_str = f"[{input_index}:a]"
else:
audio_output_str = "[0:a]"
for audio in self.task.audios:
input_index = input_args.count("-i") // 2
input_args.extend(["-i", audio.replace("\\", "/")])
filter_args.append(
f"{audio_output_str}[{input_index}:a]{self.config.amix_args[0]}[a]"
)
audio_output_str = "[a]"
return audio_output_str.strip("[]") if audio_output_str else None
def _handle_audio_encode(
self, input_args: List[str], filter_args: List[str]
) -> Optional[str]:
"""处理encode模式的音频"""
audio_output_str = ""
if self.task.mute:
input_index = input_args.count("-i") // 2
input_args.extend(["-f", "lavfi", "-i", "anullsrc=cl=stereo:r=48000"])
filter_args.append(f"[{input_index}:a]acopy[a]")
audio_output_str = "[a]"
else:
audio_output_str = "[0:a]"
for audio in self.task.audios:
input_index = input_args.count("-i") // 2
input_args.extend(["-i", audio.replace("\\", "/")])
filter_args.append(
f"{audio_output_str}[{input_index}:a]{self.config.amix_args[0]}[a]"
)
audio_output_str = "[a]"
return audio_output_str if audio_output_str else None

157
entity/render_task.py Normal file
View File

@@ -0,0 +1,157 @@
import uuid
from typing import List, Optional, Dict, Any
from dataclasses import dataclass, field
from enum import Enum
from util.exceptions import TaskValidationError, EffectError
from entity.effects import registry as effect_registry
class TaskType(Enum):
COPY = "copy"
CONCAT = "concat"
ENCODE = "encode"
@dataclass
class RenderTask:
"""渲染任务数据类,只包含任务数据,不包含处理逻辑"""
input_files: List[str] = field(default_factory=list)
output_file: str = ""
task_type: TaskType = TaskType.COPY
# 视频参数
resolution: Optional[str] = None
frame_rate: int = 25
speed: float = 1.0
mute: bool = True
annexb: bool = False
# 裁剪参数
zoom_cut: Optional[int] = None
center_cut: Optional[int] = None
# 资源列表
subtitles: List[str] = field(default_factory=list)
luts: List[str] = field(default_factory=list)
audios: List[str] = field(default_factory=list)
overlays: List[str] = field(default_factory=list)
effects: List[str] = field(default_factory=list)
# 扩展数据
ext_data: Dict[str, Any] = field(default_factory=dict)
def __post_init__(self):
"""初始化后处理"""
# 检测annexb格式
for input_file in self.input_files:
if isinstance(input_file, str) and input_file.endswith(".ts"):
self.annexb = True
break
# 自动生成输出文件名
if not self.output_file:
self._generate_output_filename()
def _generate_output_filename(self):
"""生成输出文件名"""
if self.annexb:
self.output_file = f"rand_{uuid.uuid4()}.ts"
else:
self.output_file = f"rand_{uuid.uuid4()}.mp4"
def add_input_file(self, file_path: str):
"""添加输入文件"""
self.input_files.append(file_path)
if file_path.endswith(".ts"):
self.annexb = True
def add_overlay(self, *overlays: str):
"""添加覆盖层"""
for overlay in overlays:
if overlay.endswith(".ass"):
self.subtitles.append(overlay)
else:
self.overlays.append(overlay)
def add_audios(self, *audios: str):
"""添加音频"""
self.audios.extend(audios)
def add_lut(self, *luts: str):
"""添加LUT"""
self.luts.extend(luts)
def add_effect(self, *effects: str):
"""添加效果"""
self.effects.extend(effects)
def validate(self) -> bool:
"""验证任务参数"""
if not self.input_files:
raise TaskValidationError("No input files specified")
# 验证所有效果
for effect_str in self.effects:
effect_name, params = effect_registry.parse_effect_string(effect_str)
processor = effect_registry.get_processor(
effect_name, params, self.ext_data
)
if processor and not processor.validate_params():
raise EffectError(
f"Invalid parameters for effect {effect_name}: {params}",
effect_name,
params,
)
return True
def can_copy(self) -> bool:
"""检查是否可以直接复制"""
return (
len(self.luts) == 0
and len(self.overlays) == 0
and len(self.subtitles) == 0
and len(self.effects) == 0
and self.speed == 1
and len(self.audios) == 0
and len(self.input_files) == 1
and self.zoom_cut is None
and self.center_cut is None
)
def can_concat(self) -> bool:
"""检查是否可以使用concat模式"""
return (
len(self.luts) == 0
and len(self.overlays) == 0
and len(self.subtitles) == 0
and len(self.effects) == 0
and self.speed == 1
and self.zoom_cut is None
and self.center_cut is None
)
def determine_task_type(self) -> TaskType:
"""自动确定任务类型"""
if self.can_copy():
return TaskType.COPY
elif self.can_concat():
return TaskType.CONCAT
else:
return TaskType.ENCODE
def update_task_type(self):
"""更新任务类型"""
self.task_type = self.determine_task_type()
def need_processing(self) -> bool:
"""检查是否需要处理"""
if self.annexb:
return True
return not self.can_copy()
def get_output_extension(self) -> str:
"""获取输出文件扩展名"""
return ".ts" if self.annexb else ".mp4"

View File

@@ -4,49 +4,91 @@ import sys
import config
import biz.task
from telemetry import init_opentelemetry
from template import load_local_template, download_template, TEMPLATES
from services import DefaultTemplateService
from util import api
import os
import glob
load_local_template()
# 使用新的服务容器架构
from services.service_container import get_template_service, register_default_services
# 确保服务已注册
register_default_services()
template_service = get_template_service()
# Check for redownload parameter
if 'redownload' in sys.argv:
if "redownload" in sys.argv:
print("Redownloading all templates...")
for template_name in TEMPLATES.keys():
print(f"Redownloading template: {template_name}")
download_template(template_name)
print("All templates redownloaded successfully!")
try:
for template_name in template_service.get_all_templates().keys():
print(f"Redownloading template: {template_name}")
if not template_service.download_template(template_name):
print(f"Failed to download template: {template_name}")
print("Template redownload process completed!")
except Exception as e:
print(f"Error during template redownload: {e}")
sys.exit(1)
sys.exit(0)
import logging
LOGGER = logging.getLogger(__name__)
init_opentelemetry()
while True:
# print(get_sys_info())
print("waiting for task...")
try:
task_list = api.sync_center()
except Exception as e:
LOGGER.error("sync_center error", exc_info=e)
sleep(5)
continue
if len(task_list) == 0:
# 删除当前文件夹下所有以.mp4、.ts结尾的文件
for file_globs in ['*.mp4', '*.ts', 'tmp_concat*.txt']:
def cleanup_temp_files():
"""清理临时文件 - 异步执行避免阻塞主循环"""
import threading
def _cleanup():
for file_globs in ["*.mp4", "*.ts", "tmp_concat*.txt"]:
for file_path in glob.glob(file_globs):
try:
os.remove(file_path)
print(f"Deleted file: {file_path}")
if os.path.exists(file_path):
os.remove(file_path)
LOGGER.debug(f"Deleted temp file: {file_path}")
except Exception as e:
LOGGER.error(f"Error deleting file {file_path}", exc_info=e)
sleep(5)
for task in task_list:
print("start task:", task)
LOGGER.warning(f"Error deleting file {file_path}: {e}")
# 在后台线程中执行清理
threading.Thread(target=_cleanup, daemon=True).start()
def main_loop():
"""主处理循环"""
while True:
try:
biz.task.start_task(task)
print("waiting for task...")
task_list = api.sync_center()
if len(task_list) == 0:
# 异步清理临时文件
cleanup_temp_files()
sleep(5)
continue
for task in task_list:
task_id = task.get("id", "unknown")
print(f"Processing task: {task_id}")
try:
biz.task.start_task(task)
LOGGER.info(f"Task {task_id} completed successfully")
except Exception as e:
LOGGER.error(f"Task {task_id} failed: {e}", exc_info=True)
# 继续处理下一个任务而不是崩溃
except KeyboardInterrupt:
LOGGER.info("Received shutdown signal, exiting...")
break
except Exception as e:
LOGGER.error("task_start error", exc_info=e)
LOGGER.error("Unexpected error in main loop", exc_info=e)
sleep(5) # 避免快速循环消耗CPU
if __name__ == "__main__":
try:
main_loop()
except Exception as e:
LOGGER.critical("Critical error in main process", exc_info=e)
sys.exit(1)

42
mypy.ini Normal file
View File

@@ -0,0 +1,42 @@
[mypy]
python_version = 3.9
warn_return_any = False
warn_unused_configs = False
disallow_untyped_defs = False
disallow_incomplete_defs = False
check_untyped_defs = False
disallow_untyped_decorators = False
no_implicit_optional = False
warn_redundant_casts = False
warn_unused_ignores = False
warn_no_return = False
warn_unreachable = False
strict_equality = False
namespace_packages = True
explicit_package_bases = True
show_error_codes = True
# Exclude tests code
exclude = tests/
# Ignore missing type annotations for third-party libraries
[mypy-requests.*]
ignore_missing_imports = True
[mypy-flask.*]
ignore_missing_imports = True
[mypy-pytest.*]
ignore_missing_imports = True
[mypy-PIL.*]
ignore_missing_imports = True
[mypy-psutil.*]
ignore_missing_imports = True
[mypy-opentelemetry.*]
ignore_missing_imports = True
[mypy-dotenv.*]
ignore_missing_imports = True

53
pytest.ini Normal file
View File

@@ -0,0 +1,53 @@
[tool:pytest]
testpaths = tests
python_files = test_*.py
python_classes = Test*
python_functions = test_*
# 标记定义
markers =
integration: marks tests as integration tests (may be slow)
unit: marks tests as unit tests (fast)
slow: marks tests as slow running
stress: marks tests as stress tests
# 输出配置
addopts =
-v
--tb=short
--strict-markers
--strict-config
--cov=entity
--cov=services
--cov-report=xml:coverage.xml
--cov-report=html:htmlcov
--cov-report=term-missing
--cov-branch
# 过滤警告
filterwarnings =
ignore::DeprecationWarning
ignore::PendingDeprecationWarning
# 最小覆盖率要求
[tool:coverage:run]
source = entity,services
omit =
*/tests/*
*/test_*
*/__pycache__/*
*/venv/*
*/env/*
[tool:coverage:report]
exclude_lines =
pragma: no cover
def __repr__
if self.debug:
if settings.DEBUG
raise AssertionError
raise NotImplementedError
if 0:
if __name__ == .__main__.:
class .*\bProtocol\):
@(abc\.)?abstractmethod

23
requirements-test.txt Normal file
View File

@@ -0,0 +1,23 @@
# 测试依赖
pytest>=7.0.0
pytest-cov>=4.0.0
pytest-mock>=3.10.0
pytest-xdist>=3.0.0 # 并行测试
pytest-timeout>=2.1.0
pytest-html>=3.1.0 # HTML报告
# 代码质量
flake8>=5.0.0
black>=22.0.0
mypy>=1.0.0
# 覆盖率报告
coverage[toml]>=6.0.0
# 性能测试
pytest-benchmark>=4.0.0
# 测试辅助
factory-boy>=3.2.0 # 测试数据工厂
freezegun>=1.2.0 # 时间模拟
responses>=0.22.0 # HTTP模拟

294
run_tests.py Normal file
View File

@@ -0,0 +1,294 @@
#!/usr/bin/env python
"""
测试运行器脚本
支持不同类型的测试运行和报告生成
"""
import os
import sys
import subprocess
import argparse
import tempfile
from pathlib import Path
def run_command(cmd, cwd=None):
"""运行命令并返回结果"""
print(f"Running: {' '.join(cmd)}")
try:
result = subprocess.run(
cmd, cwd=cwd, capture_output=True, text=True, check=False
)
if result.stdout:
print(result.stdout)
if result.stderr:
print(result.stderr, file=sys.stderr)
return result.returncode == 0
except Exception as e:
print(f"Error running command: {e}", file=sys.stderr)
return False
def check_dependencies():
"""检查依赖是否安装"""
print("Checking dependencies...")
# 检查pytest
if not run_command([sys.executable, "-c", "import pytest"]):
print("pytest not found. Installing test dependencies...")
if not run_command(
[sys.executable, "-m", "pip", "install", "-r", "requirements-test.txt"]
):
print("Failed to install test dependencies", file=sys.stderr)
return False
# 检查FFmpeg
ffmpeg_available = run_command(["ffmpeg", "-version"])
if not ffmpeg_available:
print("Warning: FFmpeg not found. Integration tests may be skipped.")
return True
def run_unit_tests(args):
"""运行单元测试"""
print("\n=== Running Unit Tests ===")
cmd = [
sys.executable,
"-m",
"pytest",
"tests/test_effects/",
"tests/test_ffmpeg_builder/",
"-v",
"-m",
"not integration",
]
if args.coverage:
cmd.extend(
[
"--cov=entity",
"--cov=services",
"--cov-report=xml:coverage.xml",
"--cov-report=html:htmlcov",
"--cov-report=term-missing",
"--cov-branch",
]
)
if args.xml_report:
cmd.extend(["--junitxml=unit-tests.xml"])
if args.html_report:
cmd.extend(["--html=unit-tests.html", "--self-contained-html"])
return run_command(cmd)
def run_integration_tests(args):
"""运行集成测试"""
print("\n=== Running Integration Tests ===")
# 检查FFmpeg
if not run_command(["ffmpeg", "-version"]):
print("FFmpeg not available, skipping integration tests")
return True
cmd = [
sys.executable,
"-m",
"pytest",
"tests/test_integration/",
"-v",
"-m",
"integration",
"--timeout=300",
]
if args.coverage:
cmd.extend(
[
"--cov=entity",
"--cov=services",
"--cov-report=xml:integration-coverage.xml",
"--cov-report=html:integration-htmlcov",
"--cov-branch",
]
)
if args.xml_report:
cmd.extend(["--junitxml=integration-tests.xml"])
if args.html_report:
cmd.extend(["--html=integration-tests.html", "--self-contained-html"])
return run_command(cmd)
def run_all_tests(args):
"""运行所有测试"""
print("\n=== Running All Tests ===")
cmd = [sys.executable, "-m", "pytest", "tests/", "-v"]
if args.coverage:
cmd.extend(
[
"--cov=entity",
"--cov=services",
"--cov-report=xml:coverage.xml",
"--cov-report=html:htmlcov",
"--cov-report=term-missing",
"--cov-branch",
]
)
if args.fail_under:
cmd.extend([f"--cov-fail-under={args.fail_under}"])
if args.xml_report:
cmd.extend(["--junitxml=all-tests.xml"])
if args.html_report:
cmd.extend(["--html=all-tests.html", "--self-contained-html"])
return run_command(cmd)
def run_effect_tests(effect_name=None):
"""运行特定特效测试"""
if effect_name:
print(f"\n=== Running {effect_name} Effect Tests ===")
cmd = [
sys.executable,
"-m",
"pytest",
f"tests/test_effects/test_{effect_name}_effect.py",
"-v",
]
else:
print("\n=== Running All Effect Tests ===")
cmd = [sys.executable, "-m", "pytest", "tests/test_effects/", "-v"]
return run_command(cmd)
def run_stress_tests():
"""运行压力测试"""
print("\n=== Running Stress Tests ===")
env = os.environ.copy()
env["RUN_STRESS_TESTS"] = "1"
cmd = [
sys.executable,
"-m",
"pytest",
"tests/test_integration/",
"-v",
"-m",
"stress",
"--timeout=600",
]
return subprocess.run(cmd, env=env).returncode == 0
def create_test_video():
"""创建测试视频文件"""
print("\n=== Creating Test Video Files ===")
if not run_command(["ffmpeg", "-version"]):
print("FFmpeg not available, cannot create test videos")
return False
test_data_dir = Path("tests/test_data/videos")
test_data_dir.mkdir(parents=True, exist_ok=True)
# 创建短视频文件
video_path = test_data_dir / "sample.mp4"
cmd = [
"ffmpeg",
"-y",
"-f",
"lavfi",
"-i",
"testsrc=duration=5:size=640x480:rate=25",
"-c:v",
"libx264",
"-preset",
"ultrafast",
"-crf",
"23",
str(video_path),
]
if run_command(cmd):
print(f"Created test video: {video_path}")
return True
else:
print("Failed to create test video")
return False
def main():
parser = argparse.ArgumentParser(description="RenderWorker Test Runner")
parser.add_argument(
"test_type",
choices=["unit", "integration", "all", "effects", "stress", "setup"],
help="Type of tests to run",
)
parser.add_argument(
"--effect", help="Specific effect to test (for effects command)"
)
parser.add_argument(
"--coverage", action="store_true", help="Generate coverage report"
)
parser.add_argument(
"--xml-report", action="store_true", help="Generate XML test report"
)
parser.add_argument(
"--html-report", action="store_true", help="Generate HTML test report"
)
parser.add_argument(
"--fail-under", type=int, default=70, help="Minimum coverage percentage"
)
parser.add_argument(
"--no-deps-check", action="store_true", help="Skip dependency check"
)
args = parser.parse_args()
# 检查依赖
if not args.no_deps_check and not check_dependencies():
sys.exit(1)
# 运行相应的测试
success = True
if args.test_type == "unit":
success = run_unit_tests(args)
elif args.test_type == "integration":
success = run_integration_tests(args)
elif args.test_type == "all":
success = run_all_tests(args)
elif args.test_type == "effects":
success = run_effect_tests(args.effect)
elif args.test_type == "stress":
success = run_stress_tests()
elif args.test_type == "setup":
success = create_test_video()
if success:
print("\n✅ Tests completed successfully!")
if args.coverage:
print("📊 Coverage report generated in htmlcov/")
sys.exit(0)
else:
print("\n❌ Tests failed!")
sys.exit(1)
if __name__ == "__main__":
main()

26
services/__init__.py Normal file
View File

@@ -0,0 +1,26 @@
from .render_service import RenderService, DefaultRenderService
from .task_service import TaskService, DefaultTaskService
from .template_service import TemplateService, DefaultTemplateService
from .service_container import (
ServiceContainer,
get_container,
register_default_services,
get_render_service,
get_template_service,
get_task_service,
)
__all__ = [
"RenderService",
"DefaultRenderService",
"TaskService",
"DefaultTaskService",
"TemplateService",
"DefaultTemplateService",
"ServiceContainer",
"get_container",
"register_default_services",
"get_render_service",
"get_template_service",
"get_task_service",
]

214
services/render_service.py Normal file
View File

@@ -0,0 +1,214 @@
import subprocess
import os
import logging
from abc import ABC, abstractmethod
from typing import Union
from opentelemetry.trace import Status, StatusCode
from entity.render_task import RenderTask
from entity.ffmpeg_command_builder import FFmpegCommandBuilder
from entity.ffmpeg import FfmpegTask
from util.exceptions import RenderError, FFmpegError
from util.ffmpeg import (
probe_video_info,
fade_out_audio,
handle_ffmpeg_output,
subprocess_args,
)
from telemetry import get_tracer
logger = logging.getLogger(__name__)
# 向后兼容层 - 处理旧的FfmpegTask对象
class RenderService(ABC):
"""渲染服务抽象接口"""
@abstractmethod
def render(self, task: Union[RenderTask, FfmpegTask]) -> bool:
"""
执行渲染任务
Args:
task: 渲染任务
Returns:
bool: 渲染是否成功
"""
pass
@abstractmethod
def get_video_info(self, file_path: str) -> tuple[int, int, float]:
"""
获取视频信息
Args:
file_path: 视频文件路径
Returns:
tuple: (width, height, duration)
"""
pass
@abstractmethod
def fade_out_audio(
self, file_path: str, duration: float, fade_seconds: float = 2.0
) -> str:
"""
音频淡出处理
Args:
file_path: 音频文件路径
duration: 音频总时长
fade_seconds: 淡出时长
Returns:
str: 处理后的文件路径
"""
pass
class DefaultRenderService(RenderService):
"""默认渲染服务实现"""
def render(self, task: Union[RenderTask, FfmpegTask]) -> bool:
"""执行渲染任务"""
# 兼容旧的FfmpegTask
if hasattr(task, "get_ffmpeg_args"): # 这是FfmpegTask
# 使用旧的方式执行
return self._render_legacy_ffmpeg_task(task)
tracer = get_tracer(__name__)
with tracer.start_as_current_span("render_task") as span:
try:
# 验证任务
task.validate()
span.set_attribute("task.type", task.task_type.value)
span.set_attribute("task.input_files", len(task.input_files))
span.set_attribute("task.output_file", task.output_file)
# 检查是否需要处理
if not task.need_processing():
if len(task.input_files) == 1:
task.output_file = task.input_files[0]
span.set_status(Status(StatusCode.OK))
return True
# 构建FFmpeg命令
builder = FFmpegCommandBuilder(task)
ffmpeg_args = builder.build_command()
if not ffmpeg_args:
# 不需要处理,直接返回
if len(task.input_files) == 1:
task.output_file = task.input_files[0]
span.set_status(Status(StatusCode.OK))
return True
# 执行FFmpeg命令
return self._execute_ffmpeg(ffmpeg_args, span)
except Exception as e:
span.set_status(Status(StatusCode.ERROR))
logger.error(f"Render failed: {e}", exc_info=True)
raise RenderError(f"Render failed: {e}") from e
def _execute_ffmpeg(self, args: list[str], span) -> bool:
"""执行FFmpeg命令"""
span.set_attribute("ffmpeg.args", " ".join(args))
logger.info("Executing FFmpeg: %s", " ".join(args))
try:
# 执行FFmpeg进程 (使用构建器已经包含的参数)
process = subprocess.run(
args, stderr=subprocess.PIPE, **subprocess_args(True)
)
span.set_attribute("ffmpeg.return_code", process.returncode)
# 处理输出
if process.stdout:
output = handle_ffmpeg_output(process.stdout)
span.set_attribute("ffmpeg.output", output)
logger.info("FFmpeg output: %s", output)
# 检查返回码
if process.returncode != 0:
error_msg = (
process.stderr.decode() if process.stderr else "Unknown error"
)
span.set_attribute("ffmpeg.error", error_msg)
span.set_status(Status(StatusCode.ERROR))
logger.error(
"FFmpeg failed with return code %d: %s",
process.returncode,
error_msg,
)
raise FFmpegError(
"FFmpeg execution failed",
command=args,
return_code=process.returncode,
stderr=error_msg,
)
# 检查输出文件
output_file = args[-1] # 输出文件总是最后一个参数
if not os.path.exists(output_file):
span.set_status(Status(StatusCode.ERROR))
raise RenderError(f"Output file not created: {output_file}")
# 检查文件大小
file_size = os.path.getsize(output_file)
span.set_attribute("output.file_size", file_size)
if file_size < 4096: # 文件过小
span.set_status(Status(StatusCode.ERROR))
raise RenderError(f"Output file too small: {file_size} bytes")
span.set_status(Status(StatusCode.OK))
logger.info("FFmpeg execution completed successfully")
return True
except subprocess.SubprocessError as e:
span.set_status(Status(StatusCode.ERROR))
logger.error("Subprocess error: %s", e)
raise FFmpegError(f"Subprocess error: {e}") from e
def get_video_info(self, file_path: str) -> tuple[int, int, float]:
"""获取视频信息"""
return probe_video_info(file_path)
def fade_out_audio(
self, file_path: str, duration: float, fade_seconds: float = 2.0
) -> str:
"""音频淡出处理"""
return fade_out_audio(file_path, duration, fade_seconds)
def _render_legacy_ffmpeg_task(self, ffmpeg_task) -> bool:
"""兼容处理旧的FfmpegTask"""
tracer = get_tracer(__name__)
with tracer.start_as_current_span("render_legacy_ffmpeg_task") as span:
try:
# 处理依赖任务
for sub_task in ffmpeg_task.analyze_input_render_tasks():
if not self.render(sub_task):
span.set_status(Status(StatusCode.ERROR))
return False
# 获取FFmpeg参数
ffmpeg_args = ffmpeg_task.get_ffmpeg_args()
if not ffmpeg_args:
# 不需要处理,直接返回
span.set_status(Status(StatusCode.OK))
return True
# 执行FFmpeg命令
return self._execute_ffmpeg(ffmpeg_args, span)
except Exception as e:
span.set_status(Status(StatusCode.ERROR))
logger.error(f"Legacy FFmpeg task render failed: {e}", exc_info=True)
raise RenderError(f"Legacy render failed: {e}") from e

View File

@@ -0,0 +1,139 @@
"""
服务容器模块 - 提供线程安全的服务实例管理
"""
import threading
from typing import Dict, Type, TypeVar, Optional, TYPE_CHECKING
import logging
if TYPE_CHECKING:
from .render_service import RenderService
from .template_service import TemplateService
from .task_service import TaskService
logger = logging.getLogger(__name__)
T = TypeVar("T")
class ServiceContainer:
"""线程安全的服务容器,实现依赖注入和单例管理"""
def __init__(self):
self._services: Dict[Type, object] = {}
self._factories: Dict[Type, callable] = {}
self._lock = threading.RLock()
def register_singleton(self, service_type: Type[T], factory: callable) -> None:
"""注册单例服务工厂"""
with self._lock:
self._factories[service_type] = factory
logger.debug(f"Registered singleton factory for {service_type.__name__}")
def get_service(self, service_type: Type[T]) -> T:
"""获取服务实例(懒加载单例)"""
with self._lock:
# 检查是否已存在实例
if service_type in self._services:
return self._services[service_type]
# 检查是否有工厂方法
if service_type not in self._factories:
raise ValueError(
f"No factory registered for service type: {service_type}"
)
# 创建新实例
factory = self._factories[service_type]
try:
instance = factory()
self._services[service_type] = instance
logger.debug(f"Created new instance of {service_type.__name__}")
return instance
except Exception as e:
logger.error(
f"Failed to create instance of {service_type.__name__}: {e}"
)
raise
def has_service(self, service_type: Type[T]) -> bool:
"""检查是否有服务注册"""
with self._lock:
return service_type in self._factories
def clear_cache(self, service_type: Optional[Type[T]] = None) -> None:
"""清理服务缓存"""
with self._lock:
if service_type:
self._services.pop(service_type, None)
logger.debug(f"Cleared cache for {service_type.__name__}")
else:
self._services.clear()
logger.debug("Cleared all service cache")
# 全局服务容器实例
_container: Optional[ServiceContainer] = None
_container_lock = threading.Lock()
def get_container() -> ServiceContainer:
"""获取全局服务容器实例"""
global _container
if _container is None:
with _container_lock:
if _container is None:
_container = ServiceContainer()
return _container
def register_default_services():
"""注册默认的服务实现"""
from .render_service import DefaultRenderService, RenderService
from .template_service import DefaultTemplateService, TemplateService
from .task_service import DefaultTaskService, TaskService
container = get_container()
# 注册渲染服务
container.register_singleton(RenderService, lambda: DefaultRenderService())
# 注册模板服务
def create_template_service():
service = DefaultTemplateService()
service.load_local_templates()
return service
container.register_singleton(TemplateService, create_template_service)
# 注册任务服务(依赖其他服务)
def create_task_service():
render_service = container.get_service(RenderService)
template_service = container.get_service(TemplateService)
return DefaultTaskService(render_service, template_service)
container.register_singleton(TaskService, create_task_service)
logger.info("Default services registered successfully")
# 便捷函数
def get_render_service() -> "RenderService":
"""获取渲染服务实例"""
from .render_service import RenderService
return get_container().get_service(RenderService)
def get_template_service() -> "TemplateService":
"""获取模板服务实例"""
from .template_service import TemplateService
return get_container().get_service(TemplateService)
def get_task_service() -> "TaskService":
"""获取任务服务实例"""
from .task_service import TaskService
return get_container().get_service(TaskService)

357
services/task_service.py Normal file
View File

@@ -0,0 +1,357 @@
import logging
import os
from abc import ABC, abstractmethod
from concurrent.futures import ThreadPoolExecutor
from typing import Dict, Any, Optional
from opentelemetry.trace import Status, StatusCode
from entity.render_task import RenderTask
from services.render_service import RenderService
from services.template_service import TemplateService
from util.exceptions import TaskError, TaskValidationError
from util import api, oss
from util.json_utils import safe_json_loads
from telemetry import get_tracer
logger = logging.getLogger(__name__)
class TaskService(ABC):
"""任务服务抽象接口"""
@abstractmethod
def process_task(self, task_info: Dict[str, Any]) -> bool:
"""
处理任务
Args:
task_info: 任务信息
Returns:
bool: 处理是否成功
"""
pass
@abstractmethod
def create_render_task(
self, task_info: Dict[str, Any], template_info: Dict[str, Any]
) -> RenderTask:
"""
创建渲染任务
Args:
task_info: 任务信息
template_info: 模板信息
Returns:
RenderTask: 渲染任务对象
"""
pass
class DefaultTaskService(TaskService):
"""默认任务服务实现"""
def __init__(
self, render_service: RenderService, template_service: TemplateService
):
self.render_service = render_service
self.template_service = template_service
def process_task(self, task_info: Dict[str, Any]) -> bool:
"""处理任务"""
tracer = get_tracer(__name__)
with tracer.start_as_current_span("process_task") as span:
try:
# 标准化任务信息
task_info = api.normalize_task(task_info)
span.set_attribute("task.id", task_info.get("id", "unknown"))
span.set_attribute(
"task.template_id", task_info.get("templateId", "unknown")
)
# 获取模板信息
template_id = task_info.get("templateId")
template_info = self.template_service.get_template(template_id)
if not template_info:
raise TaskError(f"Template not found: {template_id}")
# 报告任务开始
api.report_task_start(task_info)
# 创建渲染任务
render_task = self.create_render_task(task_info, template_info)
# 执行渲染
success = self.render_service.render(render_task)
if not success:
span.set_status(Status(StatusCode.ERROR))
api.report_task_failed(task_info, "Render failed")
return False
# 获取视频信息
width, height, duration = self.render_service.get_video_info(
render_task.output_file
)
span.set_attribute("video.width", width)
span.set_attribute("video.height", height)
span.set_attribute("video.duration", duration)
# 音频淡出
new_file = self.render_service.fade_out_audio(
render_task.output_file, duration
)
render_task.output_file = new_file
# 上传文件 - 创建一个兼容对象
class TaskCompat:
def __init__(self, output_file):
self.output_file = output_file
def get_output_file(self):
return self.output_file
task_compat = TaskCompat(render_task.output_file)
upload_success = api.upload_task_file(task_info, task_compat)
if not upload_success:
span.set_status(Status(StatusCode.ERROR))
api.report_task_failed(task_info, "Upload failed")
return False
# 清理临时文件
self._cleanup_temp_files(render_task)
# 报告任务成功
api.report_task_success(
task_info,
videoInfo={
"width": width,
"height": height,
"duration": duration,
},
)
span.set_status(Status(StatusCode.OK))
return True
except Exception as e:
span.set_status(Status(StatusCode.ERROR))
logger.error(f"Task processing failed: {e}", exc_info=True)
api.report_task_failed(task_info, str(e))
return False
def create_render_task(
self, task_info: Dict[str, Any], template_info: Dict[str, Any]
) -> RenderTask:
"""创建渲染任务"""
tracer = get_tracer(__name__)
with tracer.start_as_current_span("create_render_task") as span:
# 解析任务参数
task_params_str = task_info.get("taskParams", "{}")
span.set_attribute("task_params", task_params_str)
task_params = safe_json_loads(task_params_str, {})
task_params_orig = safe_json_loads(task_params_str, {})
if not task_params:
raise TaskValidationError("Invalid or empty task params JSON")
# 并行下载资源
self._download_resources(task_params)
# 创建子任务列表
sub_tasks = []
only_if_usage_count = {}
for part in template_info.get("video_parts", []):
source, ext_data = self._parse_video_source(
part.get("source"), task_params, template_info
)
if not source:
logger.warning("No video found for part: %s", part)
continue
# 检查only_if条件
only_if = part.get("only_if", "")
if only_if:
only_if_usage_count[only_if] = (
only_if_usage_count.get(only_if, 0) + 1
)
required_count = only_if_usage_count[only_if]
if not self._check_placeholder_exist_with_count(
only_if, task_params_orig, required_count
):
logger.info(
"Skipping part due to only_if condition: %s (need %d)",
only_if,
required_count,
)
continue
# 创建子任务
sub_task = self._create_sub_task(part, source, ext_data, template_info)
sub_tasks.append(sub_task)
# 创建主任务
output_file = f"out_{task_info.get('id', 'unknown')}.mp4"
main_task = RenderTask(
input_files=[task.output_file for task in sub_tasks],
output_file=output_file,
resolution=template_info.get("video_size", ""),
frame_rate=template_info.get("frame_rate", 25),
center_cut=template_info.get("crop_mode"),
zoom_cut=template_info.get("zoom_cut"),
)
# 应用整体模板设置
overall_template = template_info.get("overall_template", {})
self._apply_template_settings(main_task, overall_template, template_info)
# 设置扩展数据
main_task.ext_data = task_info
span.set_attribute("render_task.sub_tasks", len(sub_tasks))
span.set_attribute("render_task.effects", len(main_task.effects))
return main_task
def _download_resources(self, task_params: Dict[str, Any]):
"""并行下载资源"""
from config.settings import get_ffmpeg_config
config = get_ffmpeg_config()
download_futures = []
with ThreadPoolExecutor(max_workers=config.max_download_workers) as executor:
for param_list in task_params.values():
if isinstance(param_list, list):
for param in param_list:
url = param.get("url", "")
if url.startswith("http"):
_, filename = os.path.split(url)
future = executor.submit(
oss.download_from_oss, url, filename, True
)
download_futures.append((future, url, filename))
# 等待所有下载完成,并记录失败的下载
failed_downloads = []
for future, url, filename in download_futures:
try:
result = future.result(timeout=30) # 30秒超时
if not result:
failed_downloads.append((url, filename))
except Exception as e:
logger.warning(f"Failed to download {url}: {e}")
failed_downloads.append((url, filename))
if failed_downloads:
logger.warning(
f"Failed to download {len(failed_downloads)} resources: {[f[1] for f in failed_downloads]}"
)
def _parse_video_source(
self,
source: str,
task_params: Dict[str, Any],
template_info: Dict[str, Any],
) -> tuple[Optional[str], Dict[str, Any]]:
"""解析视频源"""
if source.startswith("PLACEHOLDER_"):
placeholder_id = source.replace("PLACEHOLDER_", "")
new_sources = task_params.get(placeholder_id, [])
pick_source = {}
if isinstance(new_sources, list):
if len(new_sources) == 0:
logger.debug("No video found for placeholder: %s", placeholder_id)
return None, pick_source
else:
pick_source = new_sources.pop(0)
new_sources = pick_source.get("url", "")
if new_sources.startswith("http"):
_, source_name = os.path.split(new_sources)
oss.download_from_oss(new_sources, source_name, True)
return source_name, pick_source
return new_sources, pick_source
return os.path.join(template_info.get("local_path", ""), source), {}
def _check_placeholder_exist_with_count(
self,
placeholder_id: str,
task_params: Dict[str, Any],
required_count: int = 1,
) -> bool:
"""检查占位符是否存在足够数量的片段"""
if placeholder_id in task_params:
new_sources = task_params.get(placeholder_id, [])
if isinstance(new_sources, list):
return len(new_sources) >= required_count
return required_count <= 1
return False
def _create_sub_task(
self,
part: Dict[str, Any],
source: str,
ext_data: Dict[str, Any],
template_info: Dict[str, Any],
) -> RenderTask:
"""创建子任务"""
sub_task = RenderTask(
input_files=[source],
resolution=template_info.get("video_size", ""),
frame_rate=template_info.get("frame_rate", 25),
annexb=True,
center_cut=part.get("crop_mode"),
zoom_cut=part.get("zoom_cut"),
ext_data=ext_data,
)
# 应用部分模板设置
self._apply_template_settings(sub_task, part, template_info)
return sub_task
def _apply_template_settings(
self,
task: RenderTask,
template_part: Dict[str, Any],
template_info: Dict[str, Any],
):
"""应用模板设置到任务"""
# 添加效果
for effect in template_part.get("effects", []):
task.add_effect(effect)
# 添加LUT
for lut in template_part.get("luts", []):
full_path = os.path.join(template_info.get("local_path", ""), lut)
task.add_lut(full_path.replace("\\", "/"))
# 添加音频
for audio in template_part.get("audios", []):
full_path = os.path.join(template_info.get("local_path", ""), audio)
task.add_audios(full_path)
# 添加覆盖层
for overlay in template_part.get("overlays", []):
full_path = os.path.join(template_info.get("local_path", ""), overlay)
task.add_overlay(full_path)
def _cleanup_temp_files(self, task: RenderTask):
"""清理临时文件"""
try:
template_dir = os.getenv("TEMPLATE_DIR", "")
if template_dir and template_dir not in task.output_file:
if os.path.exists(task.output_file):
os.remove(task.output_file)
logger.info("Cleaned up temp file: %s", task.output_file)
else:
logger.info("Skipped cleanup of template file: %s", task.output_file)
except OSError as e:
logger.warning("Failed to cleanup temp file %s: %s", task.output_file, e)

View File

@@ -0,0 +1,287 @@
import json
import os
import logging
from abc import ABC, abstractmethod
from typing import Dict, Any, Optional
from opentelemetry.trace import Status, StatusCode
from util.exceptions import (
TemplateError,
TemplateNotFoundError,
TemplateValidationError,
)
from util import api, oss
from config.settings import get_storage_config
from telemetry import get_tracer
logger = logging.getLogger(__name__)
class TemplateService(ABC):
"""模板服务抽象接口"""
@abstractmethod
def get_template(self, template_id: str) -> Optional[Dict[str, Any]]:
"""
获取模板信息
Args:
template_id: 模板ID
Returns:
Dict[str, Any]: 模板信息,如果不存在则返回None
"""
pass
@abstractmethod
def load_local_templates(self):
"""加载本地模板"""
pass
@abstractmethod
def download_template(self, template_id: str) -> bool:
"""
下载模板
Args:
template_id: 模板ID
Returns:
bool: 下载是否成功
"""
pass
@abstractmethod
def validate_template(self, template_info: Dict[str, Any]) -> bool:
"""
验证模板
Args:
template_info: 模板信息
Returns:
bool: 验证是否通过
"""
pass
class DefaultTemplateService(TemplateService):
"""默认模板服务实现"""
def __init__(self):
self.templates: Dict[str, Dict[str, Any]] = {}
self.storage_config = get_storage_config()
def get_template(self, template_id: str) -> Optional[Dict[str, Any]]:
"""获取模板信息"""
if template_id not in self.templates:
# 尝试下载模板
if not self.download_template(template_id):
return None
return self.templates.get(template_id)
def load_local_templates(self):
"""加载本地模板"""
template_dir = self.storage_config.template_dir
if not os.path.exists(template_dir):
logger.warning("Template directory does not exist: %s", template_dir)
return
for template_name in os.listdir(template_dir):
if template_name.startswith("_") or template_name.startswith("."):
continue
target_path = os.path.join(template_dir, template_name)
if os.path.isdir(target_path):
try:
self._load_template(template_name, target_path)
except Exception as e:
logger.error("Failed to load template %s: %s", template_name, e)
def download_template(self, template_id: str) -> bool:
"""下载模板"""
tracer = get_tracer(__name__)
with tracer.start_as_current_span("download_template") as span:
try:
span.set_attribute("template.id", template_id)
# 获取远程模板信息
template_info = api.get_template_info(template_id)
if template_info is None:
logger.warning("Failed to get template info: %s", template_id)
return False
local_path = template_info.get("local_path")
if not local_path:
local_path = os.path.join(
self.storage_config.template_dir, str(template_id)
)
template_info["local_path"] = local_path
# 创建本地目录
if not os.path.isdir(local_path):
os.makedirs(local_path)
# 下载模板资源
overall_template = template_info.get("overall_template", {})
video_parts = template_info.get("video_parts", [])
self._download_template_assets(overall_template, template_info)
for video_part in video_parts:
self._download_template_assets(video_part, template_info)
# 保存模板定义文件
template_file = os.path.join(local_path, "template.json")
with open(template_file, "w", encoding="utf-8") as f:
json.dump(template_info, f, ensure_ascii=False, indent=2)
# 加载到内存
self._load_template(template_id, local_path)
span.set_status(Status(StatusCode.OK))
logger.info("Template downloaded successfully: %s", template_id)
return True
except Exception as e:
span.set_status(Status(StatusCode.ERROR))
logger.error("Failed to download template %s: %s", template_id, e)
return False
def validate_template(self, template_info: Dict[str, Any]) -> bool:
"""验证模板"""
try:
local_path = template_info.get("local_path")
if not local_path:
raise TemplateValidationError("Template missing local_path")
# 验证视频部分
for video_part in template_info.get("video_parts", []):
self._validate_template_part(video_part, local_path)
# 验证整体模板
overall_template = template_info.get("overall_template", {})
if overall_template:
self._validate_template_part(overall_template, local_path)
return True
except TemplateValidationError:
raise
except Exception as e:
raise TemplateValidationError(f"Template validation failed: {e}")
def _load_template(self, template_name: str, local_path: str):
"""加载单个模板"""
logger.info("Loading template: %s (%s)", template_name, local_path)
template_def_file = os.path.join(local_path, "template.json")
if not os.path.exists(template_def_file):
raise TemplateNotFoundError(
f"Template definition file not found: {template_def_file}"
)
try:
with open(template_def_file, "r", encoding="utf-8") as f:
template_info = json.load(f)
except json.JSONDecodeError as e:
raise TemplateError(f"Invalid template JSON: {e}")
template_info["local_path"] = local_path
try:
self.validate_template(template_info)
self.templates[template_name] = template_info
logger.info("Template loaded successfully: %s", template_name)
except TemplateValidationError as e:
logger.error(
"Template validation failed for %s: %s. Attempting to re-download.",
template_name,
e,
)
# 模板验证失败,尝试重新下载
if self.download_template(template_name):
logger.info("Template re-downloaded successfully: %s", template_name)
else:
logger.error("Failed to re-download template: %s", template_name)
raise
def _download_template_assets(
self, template_part: Dict[str, Any], template_info: Dict[str, Any]
):
"""下载模板资源"""
local_path = template_info["local_path"]
# 下载源文件
if "source" in template_part:
source = template_part["source"]
if isinstance(source, str) and source.startswith("http"):
_, filename = os.path.split(source)
new_file_path = os.path.join(local_path, filename)
oss.download_from_oss(source, new_file_path)
if filename.endswith(".mp4"):
from util.ffmpeg import re_encode_and_annexb
new_file_path = re_encode_and_annexb(new_file_path)
template_part["source"] = os.path.relpath(new_file_path, local_path)
# 下载覆盖层
if "overlays" in template_part:
for i, overlay in enumerate(template_part["overlays"]):
if isinstance(overlay, str) and overlay.startswith("http"):
_, filename = os.path.split(overlay)
oss.download_from_oss(overlay, os.path.join(local_path, filename))
template_part["overlays"][i] = filename
# 下载LUT
if "luts" in template_part:
for i, lut in enumerate(template_part["luts"]):
if isinstance(lut, str) and lut.startswith("http"):
_, filename = os.path.split(lut)
oss.download_from_oss(lut, os.path.join(local_path, filename))
template_part["luts"][i] = filename
# 下载音频
if "audios" in template_part:
for i, audio in enumerate(template_part["audios"]):
if isinstance(audio, str) and audio.startswith("http"):
_, filename = os.path.split(audio)
oss.download_from_oss(audio, os.path.join(local_path, filename))
template_part["audios"][i] = filename
def _validate_template_part(self, template_part: Dict[str, Any], base_dir: str):
"""验证模板部分"""
# 验证源文件
source_file = template_part.get("source", "")
if (
source_file
and not source_file.startswith("http")
and not source_file.startswith("PLACEHOLDER_")
):
if not os.path.isabs(source_file):
source_file = os.path.join(base_dir, source_file)
if not os.path.exists(source_file):
raise TemplateValidationError(f"Source file not found: {source_file}")
# 验证音频文件
for audio in template_part.get("audios", []):
if not os.path.isabs(audio):
audio = os.path.join(base_dir, audio)
if not os.path.exists(audio):
raise TemplateValidationError(f"Audio file not found: {audio}")
# 验证LUT文件
for lut in template_part.get("luts", []):
if not os.path.isabs(lut):
lut = os.path.join(base_dir, lut)
if not os.path.exists(lut):
raise TemplateValidationError(f"LUT file not found: {lut}")
# 验证覆盖层文件
for overlay in template_part.get("overlays", []):
if not os.path.isabs(overlay):
overlay = os.path.join(base_dir, overlay)
if not os.path.exists(overlay):
raise TemplateValidationError(f"Overlay file not found: {overlay}")

View File

@@ -2,36 +2,54 @@ import os
from constant import SOFTWARE_VERSION
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter as OTLPSpanHttpExporter
from opentelemetry.sdk.resources import DEPLOYMENT_ENVIRONMENT, HOST_NAME, Resource, SERVICE_NAME, SERVICE_VERSION
from opentelemetry.exporter.otlp.proto.http.trace_exporter import (
OTLPSpanExporter as OTLPSpanHttpExporter,
)
from opentelemetry.sdk.resources import (
DEPLOYMENT_ENVIRONMENT,
HOST_NAME,
Resource,
SERVICE_NAME,
SERVICE_VERSION,
)
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, SimpleSpanProcessor
from opentelemetry.instrumentation.threading import ThreadingInstrumentor
ThreadingInstrumentor().instrument()
def get_tracer(name):
return trace.get_tracer(name)
# 初始化 OpenTelemetry
def init_opentelemetry(batch=True):
# 设置服务名、主机名
resource = Resource(attributes={
SERVICE_NAME: "RENDER_WORKER",
SERVICE_VERSION: SOFTWARE_VERSION,
DEPLOYMENT_ENVIRONMENT: "Python",
HOST_NAME: os.getenv("ACCESS_KEY"),
})
resource = Resource(
attributes={
SERVICE_NAME: "RENDER_WORKER",
SERVICE_VERSION: SOFTWARE_VERSION,
DEPLOYMENT_ENVIRONMENT: "Python",
HOST_NAME: os.getenv("ACCESS_KEY"),
}
)
# 使用HTTP协议上报
if batch:
span_processor = BatchSpanProcessor(OTLPSpanHttpExporter(
endpoint="https://oltp.jerryyan.top/v1/traces",
))
span_processor = BatchSpanProcessor(
OTLPSpanHttpExporter(
endpoint="https://oltp.jerryyan.top/v1/traces",
)
)
else:
span_processor = SimpleSpanProcessor(OTLPSpanHttpExporter(
endpoint="https://oltp.jerryyan.top/v1/traces",
))
span_processor = SimpleSpanProcessor(
OTLPSpanHttpExporter(
endpoint="https://oltp.jerryyan.top/v1/traces",
)
)
trace_provider = TracerProvider(resource=resource, active_span_processor=span_processor)
trace_provider = TracerProvider(
resource=resource, active_span_processor=span_processor
)
trace.set_tracer_provider(trace_provider)

View File

@@ -4,125 +4,66 @@ import logging
from telemetry import get_tracer
from util import api, oss
from services.template_service import DefaultTemplateService
TEMPLATES = {}
logger = logging.getLogger("template")
def check_local_template(local_name):
template_def = TEMPLATES[local_name]
base_dir = template_def.get("local_path")
for video_part in template_def.get("video_parts", []):
source_file = video_part.get("source", "")
if str(source_file).startswith("http"):
# download file
...
elif str(source_file).startswith("PLACEHOLDER_"):
continue
else:
if not os.path.isabs(source_file):
source_file = os.path.join(base_dir, source_file)
if not os.path.exists(source_file):
logger.error(f"{source_file} not found, please check the template definition")
raise Exception(f"{source_file} not found, please check the template definition")
for audio in video_part.get("audios", []):
if not os.path.isabs(audio):
audio = os.path.join(base_dir, audio)
if not os.path.exists(audio):
logger.error(f"{audio} not found, please check the template definition")
raise Exception(f"{audio} not found, please check the template definition")
for lut in video_part.get("luts", []):
if not os.path.isabs(lut):
lut = os.path.join(base_dir, lut)
if not os.path.exists(lut):
logger.error(f"{lut} not found, please check the template definition")
raise Exception(f"{lut} not found, please check the template definition")
for mask in video_part.get("overlays", []):
if not os.path.isabs(mask):
mask = os.path.join(base_dir, mask)
if not os.path.exists(mask):
logger.error(f"{mask} not found, please check the template definition")
raise Exception(f"{mask} not found, please check the template definition")
# 全局模板服务实例
_template_service = None
def _get_template_service():
"""获取模板服务实例"""
global _template_service
if _template_service is None:
_template_service = DefaultTemplateService()
return _template_service
# 向后兼容的全局变量和函数
TEMPLATES: dict = {}
def _update_templates_dict():
"""更新全局TEMPLATES字典以保持向后兼容"""
service = _get_template_service()
TEMPLATES.clear()
TEMPLATES.update(service.templates)
def check_local_template(local_name):
"""向后兼容函数"""
service = _get_template_service()
template_def = service.templates.get(local_name)
if template_def:
try:
service.validate_template(template_def)
except Exception as e:
logger.error(f"Template validation failed: {e}")
raise
def load_template(template_name, local_path):
global TEMPLATES
logger.info(f"加载视频模板定义:【{template_name}{local_path})】")
template_def_file = os.path.join(local_path, "template.json")
if os.path.exists(template_def_file):
TEMPLATES[template_name] = json.load(open(template_def_file, 'rb'))
TEMPLATES[template_name]["local_path"] = local_path
try:
check_local_template(template_name)
logger.info(f"完成加载【{template_name}】模板")
except Exception as e:
logger.error(f"模板定义文件【{template_def_file}】有误,正在尝试重新下载模板", exc_info=e)
download_template(template_name)
"""向后兼容函数"""
service = _get_template_service()
service._load_template(template_name, local_path)
_update_templates_dict()
def load_local_template():
for template_name in os.listdir(os.getenv("TEMPLATE_DIR")):
if template_name.startswith("_"):
continue
if template_name.startswith("."):
continue
target_path = os.path.join(os.getenv("TEMPLATE_DIR"), template_name)
if os.path.isdir(target_path):
load_template(template_name, target_path)
"""加载本地模板(向后兼容函数)"""
service = _get_template_service()
service.load_local_templates()
_update_templates_dict()
def get_template_def(template_id):
if template_id not in TEMPLATES:
download_template(template_id)
return TEMPLATES.get(template_id)
"""获取模板定义(向后兼容函数)"""
service = _get_template_service()
template = service.get_template(template_id)
_update_templates_dict()
return template
def download_template(template_id):
tracer = get_tracer(__name__)
with tracer.start_as_current_span("download_template"):
template_info = api.get_template_info(template_id)
if template_info is None:
return
if not os.path.isdir(template_info['local_path']):
os.makedirs(template_info['local_path'])
# download template assets
overall_template = template_info['overall_template']
video_parts = template_info['video_parts']
def _download_assets(_template):
if 'source' in _template:
if str(_template['source']).startswith("http"):
_, _fn = os.path.split(_template['source'])
new_fp = os.path.join(template_info['local_path'], _fn)
oss.download_from_oss(_template['source'], new_fp)
if _fn.endswith(".mp4"):
from util.ffmpeg import re_encode_and_annexb
new_fp = re_encode_and_annexb(new_fp)
_template['source'] = os.path.relpath(new_fp, template_info['local_path'])
if 'overlays' in _template:
for i in range(len(_template['overlays'])):
overlay = _template['overlays'][i]
if str(overlay).startswith("http"):
_, _fn = os.path.split(overlay)
oss.download_from_oss(overlay, os.path.join(template_info['local_path'], _fn))
_template['overlays'][i] = _fn
if 'luts' in _template:
for i in range(len(_template['luts'])):
lut = _template['luts'][i]
if str(lut).startswith("http"):
_, _fn = os.path.split(lut)
oss.download_from_oss(lut, os.path.join(template_info['local_path'], _fn))
_template['luts'][i] = _fn
if 'audios' in _template:
for i in range(len(_template['audios'])):
if str(_template['audios'][i]).startswith("http"):
_, _fn = os.path.split(_template['audios'][i])
oss.download_from_oss(_template['audios'][i], os.path.join(template_info['local_path'], _fn))
_template['audios'][i] = _fn
_download_assets(overall_template)
for video_part in video_parts:
_download_assets(video_part)
with open(os.path.join(template_info['local_path'], 'template.json'), 'w', encoding='utf-8') as f:
json.dump(template_info, f)
load_template(template_id, template_info['local_path'])
"""下载模板(向后兼容函数)"""
service = _get_template_service()
success = service.download_template(template_id)
_update_templates_dict()
return success
def analyze_template(template_id):
...
"""分析模板(占位符函数)"""
pass

313
tests/README.md Normal file
View File

@@ -0,0 +1,313 @@
# RenderWorker 测试文档
本目录包含 RenderWorker 项目的完整测试套件,用于验证特效参数生成和 FFmpeg 渲染功能。
## 目录结构
```
tests/
├── conftest.py # pytest 配置和公共 fixtures
├── test_data/ # 测试数据
│ ├── videos/ # 测试视频文件
│ ├── templates/ # 测试模板数据
│ └── expected_outputs/ # 预期输出结果
├── test_effects/ # 特效单元测试
│ ├── test_base.py # 基础类测试
│ ├── test_zoom_effect.py # 缩放特效测试
│ ├── test_speed_effect.py # 变速特效测试
│ └── ... # 其他特效测试
├── test_ffmpeg_builder/ # FFmpeg 命令构建测试
│ └── test_ffmpeg_command_builder.py
├── test_integration/ # 集成测试
│ └── test_ffmpeg_execution.py # FFmpeg 执行测试
└── utils/ # 测试工具
└── test_helpers.py # 测试辅助函数
```
## 快速开始
### 1. 安装测试依赖
```bash
pip install -r requirements-test.txt
```
### 2. 运行测试
使用测试运行器脚本:
```bash
# 运行所有测试
python run_tests.py all --coverage
# 只运行单元测试
python run_tests.py unit --coverage --html-report
# 只运行集成测试
python run_tests.py integration
# 运行特定特效测试
python run_tests.py effects --effect zoom
# 运行压力测试
python run_tests.py stress
```
或直接使用 pytest:
```bash
# 运行所有测试
pytest tests/ -v --cov=entity --cov=services
# 只运行单元测试
pytest tests/test_effects/ tests/test_ffmpeg_builder/ -v
# 只运行集成测试
pytest tests/test_integration/ -v -m integration
# 生成覆盖率报告
pytest tests/ --cov=entity --cov=services --cov-report=html
```
## 测试类型说明
### 单元测试 (Unit Tests)
位置:`tests/test_effects/`, `tests/test_ffmpeg_builder/`
测试内容:
- 各个特效处理器的参数验证
- FFmpeg 滤镜参数生成
- 特效注册表功能
- FFmpeg 命令构建逻辑
特点:
- 运行速度快
- 不依赖外部工具
- 测试覆盖率高
### 集成测试 (Integration Tests)
位置:`tests/test_integration/`
测试内容:
- 实际 FFmpeg 命令执行
- 视频文件处理验证
- 特效组合效果测试
- 错误处理验证
依赖:
- 需要系统安装 FFmpeg
- 需要测试视频文件
### 压力测试 (Stress Tests)
测试内容:
- 大量特效链处理
- 长时间运行稳定性
- 资源使用情况
运行条件:
- 设置环境变量 `RUN_STRESS_TESTS=1`
- 较长的超时时间
## 测试配置
### pytest 配置
`pytest.ini` 中配置:
- 测试路径和文件模式
- 覆盖率设置
- 标记定义
- 输出格式
### 环境变量
- `RUN_STRESS_TESTS`: 启用压力测试
- `FFMPEG_PATH`: 自定义 FFmpeg 路径
## 特效测试详解
### 缩放特效 (ZoomEffect)
测试用例:
- 有效参数验证:`"0,2.0,3.0"`(开始时间,缩放因子,持续时间)
- 无效参数处理:负值、非数字、参数不足
- 静态缩放:持续时间为 0
- 动态缩放:指定时间段内的缩放
- 位置 JSON 解析:自定义缩放中心点
生成滤镜格式:
```
# 静态缩放
[0:v]trim=start=0,zoompan=z=2.0:x=iw/2:y=ih/2:d=1[v_eff1]
# 动态缩放
[0:v]zoompan=z=if(between(t\,0\,3.0)\,2.0\,1):x=iw/2:y=ih/2:d=1[v_eff1]
```
### 变速特效 (SpeedEffect)
测试用例:
- 加速效果:`"2.0"`(2倍速)
- 减速效果:`"0.5"`(0.5倍速)
- 无效参数:零值、负值、非数字
- 默认处理:空参数或 1.0 倍速
生成滤镜格式:
```
[0:v]setpts=2.0*PTS[v_eff1]
```
## FFmpeg 命令构建测试
### 测试场景
1. **单文件复制**:直接复制无需编码
2. **多文件拼接**:使用 concat 滤镜
3. **特效处理**:复杂滤镜链构建
4. **错误处理**:缺失文件、无效参数
### 命令验证
使用 `FFmpegValidator` 验证:
- 命令结构完整性
- 输入输出文件存在
- 滤镜语法正确性
- 流标识符格式
## 集成测试详解
### 真实 FFmpeg 执行
测试流程:
1. 创建测试视频文件
2. 构建 FFmpeg 命令
3. 执行命令并检查返回码
4. 验证输出文件存在且有效
### 测试用例
- 简单复制操作
- 单特效处理
- 多特效组合
- 视频拼接
- 错误情况处理
### 性能测试
监控指标:
- 执行时间
- 内存使用
- 文件大小
- 处理速度
## 代码覆盖率
目标覆盖率:≥ 70%
覆盖范围:
- `entity/` 目录下的所有模块
- `services/` 目录下的所有模块
报告格式:
- XML 格式:用于 Jenkins CI/CD
- HTML 格式:用于本地查看
- 终端输出:实时查看缺失覆盖
## CI/CD 集成
### Jenkins Pipeline
配置文件:`Jenkinsfile`
阶段:
1. 环境准备
2. 代码质量检查
3. 单元测试
4. 集成测试
5. 完整测试套件
6. 性能测试
报告生成:
- JUnit XML 测试报告
- Cobertura 覆盖率报告
- HTML 测试和覆盖率报告
### 自动化触发
- 代码提交时运行单元测试
- PR 创建时运行完整测试
- 主分支更新时运行包含性能测试的完整套件
## 故障排除
### 常见问题
1. **FFmpeg 未找到**
```bash
# 安装 FFmpeg
sudo apt-get install ffmpeg # Ubuntu/Debian
brew install ffmpeg # macOS
```
2. **测试视频创建失败**
```bash
# 手动创建测试视频
python run_tests.py setup
```
3. **覆盖率过低**
- 检查是否有未测试的代码路径
- 添加边界条件测试
- 验证测试是否实际运行
4. **集成测试超时**
- 增加超时时间:`--timeout=600`
- 使用更小的测试文件
- 检查系统资源使用情况
### 调试技巧
1. **查看详细输出**:
```bash
pytest tests/ -v -s
```
2. **只运行失败的测试**:
```bash
pytest tests/ --lf
```
3. **停在第一个失败**:
```bash
pytest tests/ -x
```
4. **查看覆盖率详情**:
```bash
pytest tests/ --cov=entity --cov-report=term-missing
```
## 贡献指南
### 添加新测试
1. 在相应目录创建测试文件
2. 使用描述性的测试函数名
3. 添加适当的测试标记
4. 更新文档说明
### 测试最佳实践
1. **独立性**:每个测试应该独立运行
2. **可重复性**:测试结果应该一致
3. **清晰性**:测试意图应该明确
4. **完整性**:覆盖正常和异常情况
### 代码质量
- 遵循 PEP 8 代码风格
- 使用类型注解
- 添加适当的文档字符串
- 使用有意义的变量名

108
tests/conftest.py Normal file
View File

@@ -0,0 +1,108 @@
"""pytest配置文件"""
import pytest
import os
import tempfile
import shutil
from pathlib import Path
from typing import Dict, Any
from entity.render_task import RenderTask
from config.settings import FFmpegConfig, APIConfig, StorageConfig
@pytest.fixture
def temp_dir():
"""创建临时目录"""
temp_path = tempfile.mkdtemp()
yield temp_path
shutil.rmtree(temp_path, ignore_errors=True)
@pytest.fixture
def test_ffmpeg_config():
"""测试用FFmpeg配置"""
return FFmpegConfig(
encoder_args=["-c:v", "h264"],
video_args=["-profile:v", "high"],
audio_args=["-c:a", "aac"],
default_args=["-shortest"],
)
@pytest.fixture
def test_api_config():
"""测试用API配置"""
return APIConfig(
endpoint="http://test.local",
access_key="test_key",
)
@pytest.fixture
def test_storage_config():
"""测试用存储配置"""
return StorageConfig(
template_dir="tests/test_data/templates",
)
@pytest.fixture
def sample_render_task():
"""示例渲染任务"""
return RenderTask(
input_files=["test_input.mp4"],
output_file="test_output.mp4",
frame_rate=25,
effects=["zoom:0,2.0,3.0", "ospeed:1.5"],
ext_data={
"posJson": '{"ltX": 100, "ltY": 100, "rbX": 200, "rbY": 200, "imgWidth": 300, "imgHeight": 300}'
},
)
@pytest.fixture
def sample_video_file(temp_dir):
"""创建测试用的样本视频文件路径"""
video_path = os.path.join(temp_dir, "sample.mp4")
# 这里只返回路径,实际测试中可能需要真实视频文件
return video_path
@pytest.fixture(scope="session")
def ffmpeg_available():
"""检查FFmpeg是否可用"""
import subprocess
try:
subprocess.run(["ffmpeg", "-version"], capture_output=True, check=True)
return True
except (subprocess.CalledProcessError, FileNotFoundError):
return False
@pytest.fixture
def mock_ext_data():
"""模拟扩展数据"""
return {
"posJson": '{"ltX": 50, "ltY": 50, "rbX": 150, "rbY": 150, "imgWidth": 200, "imgHeight": 200}',
"templateData": {"width": 1920, "height": 1080},
}
# 标记:只在FFmpeg可用时运行集成测试
def pytest_collection_modifyitems(config, items):
"""根据FFmpeg可用性修改测试收集"""
try:
import subprocess
subprocess.run(["ffmpeg", "-version"], capture_output=True, check=True)
ffmpeg_available = True
except (subprocess.CalledProcessError, FileNotFoundError):
ffmpeg_available = False
if not ffmpeg_available:
skip_ffmpeg = pytest.mark.skip(reason="FFmpeg not available")
for item in items:
if "integration" in item.keywords:
item.add_marker(skip_ffmpeg)

15
tests/test_data/README.md Normal file
View File

@@ -0,0 +1,15 @@
# 测试数据说明
这个目录包含测试所需的样本数据:
## 目录结构
- `videos/` - 测试用视频文件
- `templates/` - 测试用模板数据
- `expected_outputs/` - 预期输出结果
## 使用说明
- 小尺寸视频文件用于快速测试
- 模板文件包含各种特效配置
- 预期输出用于验证测试结果
注意:实际的视频文件可能较大,建议使用小尺寸测试文件。

View File

@@ -0,0 +1,187 @@
"""测试特效基础类和注册表"""
import pytest
from unittest.mock import Mock
from entity.effects.base import EffectProcessor, EffectRegistry
class TestEffectProcessor:
"""测试特效处理器基类"""
def test_init(self):
"""测试初始化"""
processor = Mock(spec=EffectProcessor)
processor.__init__("test_params", {"key": "value"})
assert processor.params == "test_params"
assert processor.ext_data == {"key": "value"}
assert processor.frame_rate == 25
def test_init_default_ext_data(self):
"""测试默认扩展数据"""
processor = Mock(spec=EffectProcessor)
processor.__init__("test_params")
assert processor.ext_data == {}
def test_parse_params_empty(self):
"""测试解析空参数"""
processor = EffectProcessor("", {})
result = processor.parse_params()
assert result == []
def test_parse_params_single(self):
"""测试解析单个参数"""
processor = EffectProcessor("123", {})
result = processor.parse_params()
assert result == ["123"]
def test_parse_params_multiple(self):
"""测试解析多个参数"""
processor = EffectProcessor("1,2.5,test", {})
result = processor.parse_params()
assert result == ["1", "2.5", "test"]
def test_get_pos_json_empty(self):
"""测试获取空位置JSON"""
processor = EffectProcessor("", {})
result = processor.get_pos_json()
assert result == {}
def test_get_pos_json_valid(self):
"""测试获取有效位置JSON"""
ext_data = {"posJson": '{"ltX": 100, "ltY": 200, "rbX": 300, "rbY": 400}'}
processor = EffectProcessor("", ext_data)
result = processor.get_pos_json()
expected = {"ltX": 100, "ltY": 200, "rbX": 300, "rbY": 400}
assert result == expected
def test_get_pos_json_invalid(self):
"""测试获取无效位置JSON"""
ext_data = {"posJson": "invalid_json"}
processor = EffectProcessor("", ext_data)
result = processor.get_pos_json()
assert result == {}
def test_get_pos_json_default_value(self):
"""测试默认位置JSON值"""
ext_data = {"posJson": "{}"}
processor = EffectProcessor("", ext_data)
result = processor.get_pos_json()
assert result == {}
class TestEffectRegistry:
"""测试特效注册表"""
def test_init(self):
"""测试初始化"""
registry = EffectRegistry()
assert registry._processors == {}
def test_register_valid_processor(self):
"""测试注册有效处理器"""
registry = EffectRegistry()
class TestEffect(EffectProcessor):
def validate_params(self):
return True
def generate_filter_args(self, video_input, effect_index):
return [], video_input
def get_effect_name(self):
return "test"
registry.register("test_effect", TestEffect)
assert "test_effect" in registry._processors
assert registry._processors["test_effect"] == TestEffect
def test_register_invalid_processor(self):
"""测试注册无效处理器"""
registry = EffectRegistry()
class InvalidEffect:
pass
with pytest.raises(ValueError, match="must be a subclass of EffectProcessor"):
registry.register("invalid", InvalidEffect)
def test_get_processor_exists(self):
"""测试获取存在的处理器"""
registry = EffectRegistry()
class TestEffect(EffectProcessor):
def validate_params(self):
return True
def generate_filter_args(self, video_input, effect_index):
return [], video_input
def get_effect_name(self):
return "test"
registry.register("test_effect", TestEffect)
processor = registry.get_processor("test_effect", "params", {"data": "value"})
assert isinstance(processor, TestEffect)
assert processor.params == "params"
assert processor.ext_data == {"data": "value"}
def test_get_processor_not_exists(self):
"""测试获取不存在的处理器"""
registry = EffectRegistry()
processor = registry.get_processor("nonexistent")
assert processor is None
def test_list_effects_empty(self):
"""测试列出空特效"""
registry = EffectRegistry()
effects = registry.list_effects()
assert effects == []
def test_list_effects_with_processors(self):
"""测试列出已注册特效"""
registry = EffectRegistry()
class TestEffect(EffectProcessor):
def validate_params(self):
return True
def generate_filter_args(self, video_input, effect_index):
return [], video_input
def get_effect_name(self):
return "test"
registry.register("effect1", TestEffect)
registry.register("effect2", TestEffect)
effects = registry.list_effects()
assert set(effects) == {"effect1", "effect2"}
def test_parse_effect_string_with_params(self):
"""测试解析带参数的特效字符串"""
registry = EffectRegistry()
name, params = registry.parse_effect_string("zoom:0,2.0,3.0")
assert name == "zoom"
assert params == "0,2.0,3.0"
def test_parse_effect_string_without_params(self):
"""测试解析无参数的特效字符串"""
registry = EffectRegistry()
name, params = registry.parse_effect_string("zoom")
assert name == "zoom"
assert params == ""
def test_parse_effect_string_multiple_colons(self):
"""测试解析多个冒号的特效字符串"""
registry = EffectRegistry()
name, params = registry.parse_effect_string("effect:param1:param2")
assert name == "effect"
assert params == "param1:param2"

View File

@@ -0,0 +1,138 @@
"""测试变速特效"""
import pytest
from entity.effects.speed import SpeedEffect
from tests.utils.test_helpers import EffectTestHelper
class TestSpeedEffect:
"""测试变速特效处理器"""
def test_validate_params_valid_cases(self):
"""测试有效参数验证"""
test_cases = [
{"params": "2.0", "expected": True, "description": "2倍速"},
{"params": "0.5", "expected": True, "description": "0.5倍速(慢速)"},
{"params": "1.0", "expected": True, "description": "正常速度"},
{"params": "10.0", "expected": True, "description": "10倍速"},
{"params": "", "expected": True, "description": "空参数(默认不变速)"},
]
effect = SpeedEffect()
results = EffectTestHelper.test_effect_params_validation(effect, test_cases)
for result in results:
assert result["passed"], f"Failed case: {result['description']} - {result}"
def test_validate_params_invalid_cases(self):
"""测试无效参数验证"""
test_cases = [
{"params": "0", "expected": False, "description": "零速度"},
{"params": "-1.0", "expected": False, "description": "负速度"},
{"params": "abc", "expected": False, "description": "非数字参数"},
{"params": "1.0,2.0", "expected": False, "description": "多余参数"},
]
effect = SpeedEffect()
results = EffectTestHelper.test_effect_params_validation(effect, test_cases)
for result in results:
assert result["passed"], f"Failed case: {result['description']} - {result}"
def test_generate_filter_args_speed_change(self):
"""测试变速滤镜生成"""
effect = SpeedEffect("2.0") # 2倍速
result = EffectTestHelper.test_filter_generation(effect, "[0:v]", 1)
assert result["success"], f"Filter generation failed: {result.get('error')}"
assert result["filter_count"] == 1
assert result["valid_syntax"]
assert result["output_stream"] == "[v_eff1]"
# 检查滤镜内容
filter_str = result["filters"][0]
assert "setpts=2.0*PTS" in filter_str
assert "[0:v]" in filter_str
assert "[v_eff1]" in filter_str
def test_generate_filter_args_slow_motion(self):
"""测试慢动作滤镜生成"""
effect = SpeedEffect("0.5") # 0.5倍速(慢动作)
result = EffectTestHelper.test_filter_generation(effect, "[input]", 3)
assert result["success"]
assert result["filter_count"] == 1
assert result["valid_syntax"]
assert result["output_stream"] == "[v_eff3]"
filter_str = result["filters"][0]
assert "setpts=0.5*PTS" in filter_str
assert "[input]" in filter_str
assert "[v_eff3]" in filter_str
def test_generate_filter_args_no_change(self):
"""测试无变速效果"""
test_cases = [
{"params": "", "description": "空参数"},
{"params": "1", "description": "1倍速"},
{"params": "1.0", "description": "1.0倍速"},
]
for case in test_cases:
effect = SpeedEffect(case["params"])
result = EffectTestHelper.test_filter_generation(effect, "[0:v]", 1)
assert result["success"], f"Failed for {case['description']}"
assert (
result["filter_count"] == 0
), f"Should not generate filter for {case['description']}"
assert (
result["output_stream"] == "[0:v]"
), f"Output should equal input for {case['description']}"
def test_generate_filter_args_invalid_params(self):
"""测试无效参数的滤镜生成"""
effect = SpeedEffect("invalid")
result = EffectTestHelper.test_filter_generation(effect, "[0:v]", 1)
assert result["success"]
assert result["filter_count"] == 0
assert result["output_stream"] == "[0:v]"
def test_get_effect_name(self):
"""测试获取特效名称"""
effect = SpeedEffect()
assert effect.get_effect_name() == "ospeed"
def test_various_speed_factors(self):
"""测试各种速度因子"""
speed_factors = ["0.1", "0.25", "0.75", "1.5", "3.0", "5.0"]
for speed in speed_factors:
effect = SpeedEffect(speed)
result = EffectTestHelper.test_filter_generation(effect, "[test]", 10)
if speed == "1.0":
# 1倍速不应该生成滤镜
assert result["filter_count"] == 0
assert result["output_stream"] == "[test]"
else:
# 其他速度应该生成滤镜
assert result["success"], f"Failed for speed {speed}"
assert result["filter_count"] == 1
assert result["valid_syntax"]
assert f"setpts={speed}*PTS" in result["filters"][0]
def test_effect_chaining(self):
"""测试特效链式处理"""
# 模拟在特效链中的使用
effect = SpeedEffect("2.0")
# 第一个特效
result1 = EffectTestHelper.test_filter_generation(effect, "[0:v]", 1)
assert result1["output_stream"] == "[v_eff1]"
# 作为链中的第二个特效
result2 = EffectTestHelper.test_filter_generation(effect, "[v_eff1]", 2)
assert result2["output_stream"] == "[v_eff2]"
assert "[v_eff1]" in result2["filters"][0]

View File

@@ -0,0 +1,157 @@
"""测试缩放特效"""
import pytest
from entity.effects.zoom import ZoomEffect
from tests.utils.test_helpers import EffectTestHelper, FFmpegValidator
class TestZoomEffect:
"""测试缩放特效处理器"""
def test_validate_params_valid_cases(self):
"""测试有效参数验证"""
test_cases = [
{"params": "0,2.0,3.0", "expected": True, "description": "标准缩放参数"},
{
"params": "1.5,1.5,0",
"expected": True,
"description": "静态缩放(duration=0)",
},
{
"params": "0,1.0,5.0",
"expected": True,
"description": "无缩放效果(factor=1.0)",
},
{"params": "10,0.5,2.0", "expected": True, "description": "缩小效果"},
]
effect = ZoomEffect()
results = EffectTestHelper.test_effect_params_validation(effect, test_cases)
for result in results:
assert result["passed"], f"Failed case: {result['description']} - {result}"
def test_validate_params_invalid_cases(self):
"""测试无效参数验证"""
test_cases = [
{"params": "", "expected": False, "description": "空参数"},
{"params": "1,2", "expected": False, "description": "参数不足"},
{"params": "-1,2.0,3.0", "expected": False, "description": "负开始时间"},
{"params": "0,0,3.0", "expected": False, "description": "零缩放因子"},
{"params": "0,-2.0,3.0", "expected": False, "description": "负缩放因子"},
{"params": "0,2.0,-1.0", "expected": False, "description": "负持续时间"},
{"params": "abc,2.0,3.0", "expected": False, "description": "非数字参数"},
]
effect = ZoomEffect()
results = EffectTestHelper.test_effect_params_validation(effect, test_cases)
for result in results:
assert result["passed"], f"Failed case: {result['description']} - {result}"
def test_generate_filter_args_static_zoom(self):
"""测试静态缩放滤镜生成"""
effect = ZoomEffect("0,2.0,0") # duration=0表示静态缩放
result = EffectTestHelper.test_filter_generation(effect, "[0:v]", 1)
assert result["success"], f"Filter generation failed: {result.get('error')}"
assert result["filter_count"] == 1
assert result["valid_syntax"]
assert result["output_stream"] == "[v_eff1]"
# 检查滤镜内容
filter_str = result["filters"][0]
assert "trim=start=0" in filter_str
assert "zoompan=z=2.0" in filter_str
assert "[v_eff1]" in filter_str
def test_generate_filter_args_dynamic_zoom(self):
"""测试动态缩放滤镜生成"""
effect = ZoomEffect("1.0,1.5,2.0") # 从1秒开始,持续2秒的1.5倍缩放
result = EffectTestHelper.test_filter_generation(effect, "[0:v]", 2)
assert result["success"], f"Filter generation failed: {result.get('error')}"
assert result["filter_count"] == 1
assert result["valid_syntax"]
assert result["output_stream"] == "[v_eff2]"
# 检查滤镜内容
filter_str = result["filters"][0]
assert "zoompan=z=" in filter_str
assert "between(t\\\\,1.0\\\\,3.0)" in filter_str # 检查时间范围
assert "[v_eff2]" in filter_str
def test_generate_filter_args_no_zoom(self):
"""测试无缩放效果(factor=1.0)"""
effect = ZoomEffect("0,1.0,3.0") # 缩放因子为1.0,应该不生成滤镜
result = EffectTestHelper.test_filter_generation(effect, "[0:v]", 1)
assert result["success"]
assert result["filter_count"] == 0
assert result["output_stream"] == "[0:v]" # 输出应该等于输入
def test_generate_filter_args_invalid_params(self):
"""测试无效参数的滤镜生成"""
effect = ZoomEffect("invalid,params")
result = EffectTestHelper.test_filter_generation(effect, "[0:v]", 1)
assert result["success"]
assert result["filter_count"] == 0
assert result["output_stream"] == "[0:v]" # 无效参数时应该返回原输入
def test_get_zoom_center_default(self):
"""测试默认缩放中心点"""
effect = ZoomEffect("0,2.0,3.0")
center_x, center_y = effect._get_zoom_center()
assert center_x == "iw/2"
assert center_y == "ih/2"
def test_get_zoom_center_with_pos_json(self):
"""测试基于posJson的缩放中心点"""
ext_data = {
"posJson": '{"ltX": 100, "ltY": 100, "rbX": 200, "rbY": 200, "imgWidth": 400, "imgHeight": 300}'
}
effect = ZoomEffect("0,2.0,3.0", ext_data)
center_x, center_y = effect._get_zoom_center()
# 中心点应该是矩形的中心
# center_x_ratio = (100 + 200) / (2 * 400) = 0.375
# center_y_ratio = (100 + 200) / (2 * 300) = 0.5
assert "iw*0.375" in center_x
assert "ih*0.5" in center_y
def test_get_zoom_center_invalid_pos_json(self):
"""测试无效posJson时的缩放中心点"""
ext_data = {"posJson": '{"imgWidth": 0, "imgHeight": 0}'} # 无效尺寸
effect = ZoomEffect("0,2.0,3.0", ext_data)
center_x, center_y = effect._get_zoom_center()
# 应该回退到默认中心点
assert center_x == "iw/2"
assert center_y == "ih/2"
def test_get_effect_name(self):
"""测试获取特效名称"""
effect = ZoomEffect()
assert effect.get_effect_name() == "zoom"
def test_complex_zoom_scenario(self):
"""测试复杂缩放场景"""
# 带有复杂posJson数据的动态缩放
ext_data = {
"posJson": '{"ltX": 50, "ltY": 75, "rbX": 350, "rbY": 225, "imgWidth": 400, "imgHeight": 300}'
}
effect = ZoomEffect("2.5,3.0,1.5", ext_data)
result = EffectTestHelper.test_filter_generation(effect, "[input]", 5)
assert result["success"]
assert result["filter_count"] == 1
assert result["valid_syntax"]
assert result["output_stream"] == "[v_eff5]"
filter_str = result["filters"][0]
assert "zoompan=z=" in filter_str
assert "between(t\\\\,2.5\\\\,4.0)" in filter_str # 2.5 + 1.5 = 4.0
assert "[input]" in filter_str
assert "[v_eff5]" in filter_str

View File

@@ -0,0 +1,234 @@
"""测试FFmpeg命令构建器"""
import pytest
from unittest.mock import Mock, patch
from entity.ffmpeg_command_builder import FFmpegCommandBuilder
from entity.render_task import RenderTask
from config.settings import FFmpegConfig, APIConfig, StorageConfig
from tests.utils.test_helpers import MockRenderTask, FFmpegValidator
class TestFFmpegCommandBuilder:
"""测试FFmpeg命令构建器"""
@pytest.fixture
def simple_task(self):
"""简单的渲染任务"""
task = MockRenderTask()
task.input_files = ["input1.mp4", "input2.mp4"]
task.output_path = "output.mp4"
task.effects = []
return task
@pytest.fixture
def task_with_effects(self):
"""带特效的渲染任务"""
task = MockRenderTask()
task.input_files = ["input.mp4"]
task.output_path = "output.mp4"
task.effects = ["zoom:0,2.0,3.0", "ospeed:1.5"]
task.ext_data = {
"posJson": '{"ltX": 100, "ltY": 100, "rbX": 200, "rbY": 200, "imgWidth": 300, "imgHeight": 300}'
}
return task
def test_init(self, simple_task):
"""测试初始化"""
builder = FFmpegCommandBuilder(simple_task)
assert builder.task == simple_task
assert builder.config is not None
def test_build_copy_command(self, simple_task):
"""测试构建复制命令"""
builder = FFmpegCommandBuilder(simple_task)
command = builder._build_copy_command()
# 验证命令结构
validation = FFmpegValidator.validate_ffmpeg_command(command)
assert validation["valid"], f"Invalid command: {validation['errors']}"
assert validation["has_input"]
assert validation["has_output"]
# 验证具体内容
assert command[0] == "ffmpeg"
assert "-i" in command
assert "input1.mp4" in command
assert "output.mp4" in command
assert "-c" in command and "copy" in command
def test_build_concat_command_multiple_files(self):
"""测试构建多文件拼接命令"""
task = MockRenderTask()
task.input_files = ["file1.mp4", "file2.mp4", "file3.mp4"]
task.output_path = "concat_output.mp4"
builder = FFmpegCommandBuilder(task)
command = builder._build_concat_command()
validation = FFmpegValidator.validate_ffmpeg_command(command)
assert validation["valid"]
assert validation["has_filter"]
# 验证拼接相关参数
assert "concat=n=3:v=1:a=1" in " ".join(command)
assert all(f"file{i}.mp4" in command for i in range(1, 4))
def test_build_encode_command_with_effects(self, task_with_effects):
"""测试构建带特效的编码命令"""
builder = FFmpegCommandBuilder(task_with_effects)
command = builder._build_encode_command()
validation = FFmpegValidator.validate_ffmpeg_command(command)
assert validation["valid"]
assert validation["has_filter"]
command_str = " ".join(command)
# 应该包含特效相关的滤镜
assert "-filter_complex" in command
# 应该包含编码参数
assert "-c:v" in command and "h264" in command
def test_add_effects_single_effect(self):
"""测试添加单个特效"""
task = MockRenderTask()
task.effects = ["zoom:0,2.0,3.0"]
task.ext_data = {"posJson": "{}"}
builder = FFmpegCommandBuilder(task)
filter_args: list[str] = []
result_input, result_index = builder._add_effects(filter_args, "[0:v]", 1)
# 验证结果
assert len(filter_args) > 0 # 应该有滤镜被添加
assert result_input == "[v_eff1]" # 输出流应该更新
assert result_index == 2 # 索引应该递增
def test_add_effects_multiple_effects(self):
"""测试添加多个特效"""
task = MockRenderTask()
task.effects = ["zoom:0,2.0,3.0", "ospeed:1.5"]
task.ext_data = {"posJson": "{}"}
builder = FFmpegCommandBuilder(task)
filter_args: list[str] = []
result_input, result_index = builder._add_effects(filter_args, "[0:v]", 1)
# 验证特效链
assert len(filter_args) >= 2 # 应该有两个特效的滤镜
assert result_input == "[v_eff2]" # 最终输出流
assert result_index == 3 # 索引应该递增两次
def test_add_effects_invalid_effect(self):
"""测试添加无效特效"""
task = MockRenderTask()
task.effects = ["invalid_effect:params"]
builder = FFmpegCommandBuilder(task)
filter_args: list[str] = []
result_input, result_index = builder._add_effects(filter_args, "[0:v]", 1)
# 无效特效应该被忽略
assert result_input == "[0:v]" # 输入流不变
assert result_index == 1 # 索引不变
def test_add_effects_no_effects(self):
"""测试无特效情况"""
task = MockRenderTask()
task.effects = []
builder = FFmpegCommandBuilder(task)
filter_args: list[str] = []
result_input, result_index = builder._add_effects(filter_args, "[0:v]", 1)
assert result_input == "[0:v]" # 输入流不变
assert result_index == 1 # 索引不变
assert len(filter_args) == 0 # 无滤镜添加
def test_build_command_copy_mode(self, simple_task):
"""测试构建复制模式命令"""
simple_task.input_files = ["single_file.mp4"]
builder = FFmpegCommandBuilder(simple_task)
command = builder.build_command()
validation = FFmpegValidator.validate_ffmpeg_command(command)
assert validation["valid"]
# 应该是复制模式
command_str = " ".join(command)
assert "-c copy" in command_str
def test_build_command_concat_mode(self):
"""测试构建拼接模式命令"""
task = MockRenderTask()
task.input_files = ["file1.mp4", "file2.mp4"]
task.effects = []
builder = FFmpegCommandBuilder(task)
command = builder.build_command()
validation = FFmpegValidator.validate_ffmpeg_command(command)
assert validation["valid"]
assert validation["has_filter"]
# 应该包含拼接滤镜
command_str = " ".join(command)
assert "concat=" in command_str
def test_build_command_encode_mode(self, task_with_effects):
"""测试构建编码模式命令"""
builder = FFmpegCommandBuilder(task_with_effects)
command = builder.build_command()
validation = FFmpegValidator.validate_ffmpeg_command(command)
assert validation["valid"]
assert validation["has_filter"]
# 应该包含编码参数和特效滤镜
command_str = " ".join(command)
assert "-c:v h264" in command_str
@patch("entity.effects.registry.get_processor")
def test_effect_processor_integration(self, mock_get_processor):
"""测试与特效处理器的集成"""
# 模拟特效处理器
mock_processor = Mock()
mock_processor.frame_rate = 25
mock_processor.generate_filter_args.return_value = (
["[0:v]zoompan=z=2.0:x=iw/2:y=ih/2:d=1[v_eff1]"],
"[v_eff1]",
)
mock_get_processor.return_value = mock_processor
task = MockRenderTask()
task.effects = ["zoom:0,2.0,3.0"]
task.frame_rate = 30
builder = FFmpegCommandBuilder(task)
filter_args: list[str] = []
builder._add_effects(filter_args, "[0:v]", 1)
# 验证处理器被正确调用
mock_processor.generate_filter_args.assert_called_once_with("[0:v]", 1)
assert mock_processor.frame_rate == 30 # 帧率应该被设置
def test_error_handling_missing_input(self):
"""测试缺少输入文件的错误处理"""
task = MockRenderTask()
task.input_files = []
builder = FFmpegCommandBuilder(task)
# 构建命令时应该处理错误情况
# 具体的错误处理依赖于实现
command = builder.build_command()
# 验证至少返回了基本的ffmpeg命令结构
assert isinstance(command, list)
assert len(command) > 0

View File

@@ -0,0 +1,275 @@
"""FFmpeg执行集成测试"""
import pytest
import subprocess
import tempfile
import os
from pathlib import Path
from entity.ffmpeg_command_builder import FFmpegCommandBuilder
from entity.render_task import RenderTask
from config.settings import FFmpegConfig
from services.render_service import DefaultRenderService
from tests.utils.test_helpers import MockRenderTask, create_test_video_file
@pytest.mark.integration
class TestFFmpegExecution:
"""FFmpeg执行集成测试"""
@pytest.fixture
def sample_video(self, temp_dir):
"""创建测试视频文件"""
video_path = os.path.join(temp_dir, "test_input.mp4")
success = create_test_video_file(video_path, duration=3, resolution="320x240")
if not success:
pytest.skip("Cannot create test video file")
return video_path
@pytest.fixture
def render_service(self):
"""渲染服务实例"""
return DefaultRenderService()
def test_simple_copy_execution(self, sample_video, temp_dir):
"""测试简单复制执行"""
output_path = os.path.join(temp_dir, "copy_output.mp4")
task = MockRenderTask()
task.input_files = [sample_video]
task.output_path = output_path
task.effects = []
builder = FFmpegCommandBuilder(task)
command = builder.build_command()
# 执行命令
try:
result = subprocess.run(command, capture_output=True, text=True, timeout=30)
assert result.returncode == 0, f"FFmpeg failed: {result.stderr}"
assert os.path.exists(output_path), "Output file was not created"
assert os.path.getsize(output_path) > 0, "Output file is empty"
except subprocess.TimeoutExpired:
pytest.fail("FFmpeg execution timed out")
def test_zoom_effect_execution(self, sample_video, temp_dir):
"""测试缩放特效执行"""
output_path = os.path.join(temp_dir, "zoom_output.mp4")
task = MockRenderTask()
task.input_files = [sample_video]
task.output_path = output_path
task.effects = ["zoom:0,2.0,2.0"] # 2倍缩放,持续2秒
task.ext_data = {"posJson": "{}"}
builder = FFmpegCommandBuilder(task)
command = builder.build_command()
try:
result = subprocess.run(command, capture_output=True, text=True, timeout=60)
assert result.returncode == 0, f"FFmpeg failed: {result.stderr}"
assert os.path.exists(output_path), "Output file was not created"
assert os.path.getsize(output_path) > 0, "Output file is empty"
except subprocess.TimeoutExpired:
pytest.fail("FFmpeg execution timed out")
def test_speed_effect_execution(self, sample_video, temp_dir):
"""测试变速特效执行"""
output_path = os.path.join(temp_dir, "speed_output.mp4")
task = MockRenderTask()
task.input_files = [sample_video]
task.output_path = output_path
task.effects = ["ospeed:2.0"] # 2倍速
task.ext_data = {}
builder = FFmpegCommandBuilder(task)
command = builder.build_command()
try:
result = subprocess.run(command, capture_output=True, text=True, timeout=60)
assert result.returncode == 0, f"FFmpeg failed: {result.stderr}"
assert os.path.exists(output_path), "Output file was not created"
assert os.path.getsize(output_path) > 0, "Output file is empty"
except subprocess.TimeoutExpired:
pytest.fail("FFmpeg execution timed out")
def test_multiple_effects_execution(self, sample_video, temp_dir):
"""测试多特效组合执行"""
output_path = os.path.join(temp_dir, "multi_effects_output.mp4")
task = MockRenderTask()
task.input_files = [sample_video]
task.output_path = output_path
task.effects = ["zoom:0,1.5,1.0", "ospeed:1.5"] # 缩放+变速
task.ext_data = {"posJson": "{}"}
builder = FFmpegCommandBuilder(task)
command = builder.build_command()
try:
result = subprocess.run(command, capture_output=True, text=True, timeout=60)
assert result.returncode == 0, f"FFmpeg failed: {result.stderr}"
assert os.path.exists(output_path), "Output file was not created"
assert os.path.getsize(output_path) > 0, "Output file is empty"
except subprocess.TimeoutExpired:
pytest.fail("FFmpeg execution timed out")
def test_concat_execution(self, temp_dir):
"""测试视频拼接执行"""
# 创建两个测试视频
video1_path = os.path.join(temp_dir, "video1.mp4")
video2_path = os.path.join(temp_dir, "video2.mp4")
output_path = os.path.join(temp_dir, "concat_output.mp4")
success1 = create_test_video_file(video1_path, duration=2, resolution="320x240")
success2 = create_test_video_file(video2_path, duration=2, resolution="320x240")
if not (success1 and success2):
pytest.skip("Cannot create test video files")
task = MockRenderTask()
task.input_files = [video1_path, video2_path]
task.output_path = output_path
task.effects = []
builder = FFmpegCommandBuilder(task)
command = builder.build_command()
try:
result = subprocess.run(command, capture_output=True, text=True, timeout=60)
assert result.returncode == 0, f"FFmpeg failed: {result.stderr}"
assert os.path.exists(output_path), "Output file was not created"
assert os.path.getsize(output_path) > 0, "Output file is empty"
except subprocess.TimeoutExpired:
pytest.fail("FFmpeg execution timed out")
def test_invalid_effect_execution(self, sample_video, temp_dir):
"""测试无效特效的执行处理"""
output_path = os.path.join(temp_dir, "invalid_effect_output.mp4")
task = MockRenderTask()
task.input_files = [sample_video]
task.output_path = output_path
task.effects = ["invalid_effect:params", "zoom:0,2.0,1.0"] # 混合有效和无效特效
task.ext_data = {"posJson": "{}"}
builder = FFmpegCommandBuilder(task)
command = builder.build_command()
# 应该忽略无效特效,继续处理有效特效
try:
result = subprocess.run(command, capture_output=True, text=True, timeout=60)
assert result.returncode == 0, f"FFmpeg failed: {result.stderr}"
assert os.path.exists(output_path), "Output file was not created"
except subprocess.TimeoutExpired:
pytest.fail("FFmpeg execution timed out")
def test_render_service_integration(self, sample_video, temp_dir, render_service):
"""测试渲染服务集成"""
output_path = os.path.join(temp_dir, "service_output.mp4")
# 创建真实的RenderTask(不是Mock)
task_data = {
"task_id": "integration_test",
"template_id": "test_template",
"input_files": [sample_video],
"output_path": output_path,
"effects": ["zoom:0,1.8,2.0"],
"ext_data": {"posJson": "{}"},
"frame_rate": 25,
}
# 这里需要根据实际的RenderTask构造方法调整
task = MockRenderTask(**task_data)
# 使用渲染服务执行
try:
# 这里的方法调用需要根据实际的渲染服务接口调整
# success = render_service.render(task)
# assert success, "Render service failed"
# 临时直接使用FFmpegCommandBuilder测试
builder = FFmpegCommandBuilder(task)
command = builder.build_command()
result = subprocess.run(command, capture_output=True, text=True, timeout=60)
assert result.returncode == 0, f"Render failed: {result.stderr}"
assert os.path.exists(output_path), "Output file was not created"
except Exception as e:
pytest.fail(f"Render service integration failed: {e}")
def test_error_handling_missing_input(self, temp_dir):
"""测试缺失输入文件的错误处理"""
missing_file = os.path.join(temp_dir, "nonexistent.mp4")
output_path = os.path.join(temp_dir, "error_output.mp4")
task = MockRenderTask()
task.input_files = [missing_file]
task.output_path = output_path
task.effects = []
builder = FFmpegCommandBuilder(task)
command = builder.build_command()
# 应该失败,因为输入文件不存在
result = subprocess.run(command, capture_output=True, text=True, timeout=30)
assert result.returncode != 0, "FFmpeg should fail with missing input file"
def test_performance_multiple_effects(self, sample_video, temp_dir):
"""测试多特效性能"""
output_path = os.path.join(temp_dir, "performance_output.mp4")
task = MockRenderTask()
task.input_files = [sample_video]
task.output_path = output_path
# 多个特效组合
task.effects = ["zoom:0,1.5,1.0", "ospeed:1.2", "zoom:1.5,2.0,1.0"]
task.ext_data = {"posJson": "{}"}
builder = FFmpegCommandBuilder(task)
command = builder.build_command()
import time
start_time = time.time()
try:
result = subprocess.run(
command, capture_output=True, text=True, timeout=120
)
execution_time = time.time() - start_time
assert result.returncode == 0, f"FFmpeg failed: {result.stderr}"
assert os.path.exists(output_path), "Output file was not created"
assert execution_time < 60, f"Execution took too long: {execution_time}s"
except subprocess.TimeoutExpired:
pytest.fail("FFmpeg execution timed out")
@pytest.mark.skipif(
not os.environ.get("RUN_STRESS_TESTS"), reason="Stress tests disabled"
)
def test_stress_test_large_effects_chain(self, sample_video, temp_dir):
"""压力测试:大量特效链"""
output_path = os.path.join(temp_dir, "stress_output.mp4")
task = MockRenderTask()
task.input_files = [sample_video]
task.output_path = output_path
# 创建大量特效
task.effects = [f"zoom:{i*0.5},1.{i+5},{i*0.2+0.5}" for i in range(10)]
task.ext_data = {"posJson": "{}"}
builder = FFmpegCommandBuilder(task)
command = builder.build_command()
try:
result = subprocess.run(
command, capture_output=True, text=True, timeout=300
)
assert result.returncode == 0, f"Stress test failed: {result.stderr}"
assert os.path.exists(output_path), "Output file was not created"
except subprocess.TimeoutExpired:
pytest.fail("Stress test timed out")

241
tests/utils/test_helpers.py Normal file
View File

@@ -0,0 +1,241 @@
"""测试辅助工具"""
import json
import tempfile
import subprocess
from typing import Dict, Any, List, Optional
from pathlib import Path
from entity.render_task import RenderTask
from entity.effects.base import EffectProcessor
from config.settings import FFmpegConfig
class MockRenderTask:
"""模拟渲染任务,用于测试"""
def __init__(
self,
input_files: Optional[List[str]] = None,
output_file: str = "test_output.mp4",
effects: Optional[List[str]] = None,
ext_data: Optional[Dict[str, Any]] = None,
frame_rate: int = 25,
):
# RenderTask required fields
self.input_files = input_files or []
self.output_file = output_file
self.task_type = "copy" # TaskType.COPY equivalent
# Optional fields that match RenderTask
self.resolution = None
self.frame_rate = frame_rate
self.speed = 1.0
self.mute = True
self.annexb = False
# Cut parameters
self.zoom_cut = None
self.center_cut = None
# Resource lists
self.subtitles: List[str] = []
self.luts: List[str] = []
self.audios: List[str] = []
self.overlays: List[str] = []
self.effects = effects or []
# Extension data
self.ext_data = ext_data or {}
# Legacy compatibility
self.task_id = "test_task"
self.template_id = "test_template"
self.use_center_cut = False
self.use_zoom_cut = False
self.audio_file = None
class FFmpegValidator:
"""FFmpeg命令验证器"""
@staticmethod
def validate_filter_syntax(filter_str: str) -> bool:
"""验证滤镜语法是否正确"""
try:
# 基本语法检查
if not filter_str:
return False
# 检查是否包含基本的滤镜结构
if "[" in filter_str and "]" in filter_str:
return True
# 检查常见的滤镜格式
common_filters = ["zoompan", "setpts", "trim", "scale", "crop"]
return any(f in filter_str for f in common_filters)
except Exception:
return False
@staticmethod
def validate_stream_identifier(stream_id: str) -> bool:
"""验证流标识符格式"""
if not stream_id:
return False
return stream_id.startswith("[") and stream_id.endswith("]")
@staticmethod
def validate_ffmpeg_command(command: List[str]) -> Dict[str, Any]:
"""验证完整的FFmpeg命令"""
result = {
"valid": False,
"has_input": False,
"has_output": False,
"has_filter": False,
"errors": [],
}
if not command or command[0] != "ffmpeg":
result["errors"].append("Command must start with 'ffmpeg'")
return result
# 检查输入文件
if "-i" in command:
result["has_input"] = True
else:
result["errors"].append("No input file specified")
# 检查输出文件
if len(command) > 1 and not command[-1].startswith("-"):
result["has_output"] = True
else:
result["errors"].append("No output file specified")
# 检查滤镜
if "-filter_complex" in command or "-vf" in command:
result["has_filter"] = True
result["valid"] = (
result["has_input"] and result["has_output"] and len(result["errors"]) == 0
)
return result
class EffectTestHelper:
"""特效测试辅助类"""
@staticmethod
def create_test_effect(
effect_class, params: str = "", ext_data: Dict[str, Any] = None
):
"""创建测试用特效实例"""
return effect_class(params, ext_data)
@staticmethod
def test_effect_params_validation(
effect: EffectProcessor, test_cases: List[Dict[str, Any]]
):
"""批量测试特效参数验证"""
results = []
for case in test_cases:
effect.params = case.get("params", "")
effect.ext_data = case.get("ext_data", {})
is_valid = effect.validate_params()
expected = case.get("expected", True)
results.append(
{
"params": effect.params,
"expected": expected,
"actual": is_valid,
"passed": is_valid == expected,
"description": case.get("description", ""),
}
)
return results
@staticmethod
def test_filter_generation(
effect: EffectProcessor, video_input: str = "[0:v]", effect_index: int = 1
):
"""测试滤镜生成"""
try:
filters, output_stream = effect.generate_filter_args(
video_input, effect_index
)
result = {
"success": True,
"filters": filters,
"output_stream": output_stream,
"filter_count": len(filters),
"valid_syntax": all(
FFmpegValidator.validate_filter_syntax(f) for f in filters
),
"valid_output": (
FFmpegValidator.validate_stream_identifier(output_stream)
if output_stream != video_input
else True
),
}
except Exception as e:
result = {
"success": False,
"error": str(e),
"filters": [],
"output_stream": "",
"filter_count": 0,
"valid_syntax": False,
"valid_output": False,
}
return result
def create_test_video_file(
output_path: str, duration: int = 5, resolution: str = "640x480"
) -> bool:
"""创建测试用视频文件"""
try:
cmd = [
"ffmpeg",
"-y", # 覆盖输出文件
"-f",
"lavfi", # 使用libavfilter输入
"-i",
f"testsrc=duration={duration}:size={resolution}:rate=25",
"-c:v",
"libx264",
"-preset",
"ultrafast",
"-crf",
"23",
output_path,
]
result = subprocess.run(cmd, capture_output=True, text=True)
return result.returncode == 0
except Exception:
return False
def create_sample_template_data() -> Dict[str, Any]:
"""创建示例模板数据"""
return {
"templateId": "test_template_001",
"name": "测试模板",
"parts": [
{
"id": "part1",
"type": "video",
"duration": 10.0,
"effects": ["zoom:0,2.0,3.0", "ospeed:1.5"],
}
],
"settings": {"width": 1920, "height": 1080, "frameRate": 25},
}

View File

@@ -2,6 +2,9 @@ import json
import logging
import os
import threading
import time
from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter
import requests
from opentelemetry.trace import Status, StatusCode
@@ -10,7 +13,26 @@ import util.system
from telemetry import get_tracer
from util import oss
# 创建带有连接池和重试策略的会话
session = requests.Session()
# 配置重试策略
retry_strategy = Retry(
total=3,
status_forcelist=[429, 500, 502, 503, 504],
backoff_factor=1,
respect_retry_after_header=True,
)
# 配置HTTP适配器(连接池)
adapter = HTTPAdapter(pool_connections=10, pool_maxsize=20, max_retries=retry_strategy)
session.mount("http://", adapter)
session.mount("https://", adapter)
# 设置默认超时
session.timeout = 30
logger = logging.getLogger(__name__)
@@ -24,23 +46,31 @@ def sync_center():
通过接口获取任务
:return: 任务列表
"""
from template import TEMPLATES, download_template
from services import DefaultTemplateService
template_service = DefaultTemplateService()
try:
response = session.post(os.getenv('API_ENDPOINT') + "/sync", json={
'accessKey': os.getenv('ACCESS_KEY'),
'clientStatus': util.system.get_sys_info(),
'templateList': [{'id': t.get('id', ''), 'updateTime': t.get('updateTime', '')} for t in
TEMPLATES.values()]
}, timeout=10)
response = session.post(
os.getenv("API_ENDPOINT") + "/sync",
json={
"accessKey": os.getenv("ACCESS_KEY"),
"clientStatus": util.system.get_sys_info(),
"templateList": [
{"id": t.get("id", ""), "updateTime": t.get("updateTime", "")}
for t in template_service.templates.values()
],
},
timeout=10,
)
response.raise_for_status()
except requests.RequestException as e:
logger.error("请求失败!", e)
return []
data = response.json()
logger.debug("获取任务结果:【%s", data)
if data.get('code', 0) == 200:
templates = data.get('data', {}).get('templates', [])
tasks = data.get('data', {}).get('tasks', [])
if data.get("code", 0) == 200:
templates = data.get("data", {}).get("templates", [])
tasks = data.get("data", {}).get("tasks", [])
else:
tasks = []
templates = []
@@ -48,15 +78,19 @@ def sync_center():
if os.getenv("REDIRECT_TO_URL", False) != False:
for task in tasks:
_sess = requests.Session()
logger.info("重定向任务【%s】至配置的地址:%s", task.get("id"), os.getenv("REDIRECT_TO_URL"))
logger.info(
"重定向任务【%s】至配置的地址:%s",
task.get("id"),
os.getenv("REDIRECT_TO_URL"),
)
url = f"{os.getenv('REDIRECT_TO_URL')}{task.get('id')}"
threading.Thread(target=requests.post, args=(url,)).start()
return []
for template in templates:
template_id = template.get('id', '')
template_id = template.get("id", "")
if template_id:
logger.info("更新模板:【%s", template_id)
download_template(template_id)
template_service.download_template(template_id)
return tasks
@@ -73,10 +107,17 @@ def get_template_info(template_id):
with tracer.start_as_current_span("get_template_info.request") as req_span:
try:
req_span.set_attribute("http.method", "POST")
req_span.set_attribute("http.url", '{0}/template/{1}'.format(os.getenv('API_ENDPOINT'), template_id))
response = session.post('{0}/template/{1}'.format(os.getenv('API_ENDPOINT'), template_id), json={
'accessKey': os.getenv('ACCESS_KEY'),
}, timeout=10)
req_span.set_attribute(
"http.url",
"{0}/template/{1}".format(os.getenv("API_ENDPOINT"), template_id),
)
response = session.post(
"{0}/template/{1}".format(os.getenv("API_ENDPOINT"), template_id),
json={
"accessKey": os.getenv("ACCESS_KEY"),
},
timeout=10,
)
req_span.set_attribute("http.status_code", response.status_code)
req_span.set_attribute("http.response", response.text)
response.raise_for_status()
@@ -86,64 +127,68 @@ def get_template_info(template_id):
return None
data = response.json()
logger.debug("获取模板信息结果:【%s", data)
remote_template_info = data.get('data', {})
remote_template_info = data.get("data", {})
if not remote_template_info:
logger.warning("获取模板信息结果为空", data)
return None
template = {
'id': template_id,
'updateTime': remote_template_info.get('updateTime', template_id),
'scenic_name': remote_template_info.get('scenicName', '景区'),
'name': remote_template_info.get('name', '模版'),
'video_size': remote_template_info.get('resolution', '1920x1080'),
'frame_rate': 25,
'overall_duration': 30,
'video_parts': [
]
"id": template_id,
"updateTime": remote_template_info.get("updateTime", template_id),
"scenic_name": remote_template_info.get("scenicName", "景区"),
"name": remote_template_info.get("name", "模版"),
"video_size": remote_template_info.get("resolution", "1920x1080"),
"frame_rate": 25,
"overall_duration": 30,
"video_parts": [],
}
def _template_normalizer(template_info):
_template = {}
_placeholder_type = template_info.get('isPlaceholder', -1)
_placeholder_type = template_info.get("isPlaceholder", -1)
if _placeholder_type == 0:
# 固定视频
_template['source'] = template_info.get('sourceUrl', '')
_template["source"] = template_info.get("sourceUrl", "")
elif _placeholder_type == 1:
# 占位符
_template['source'] = "PLACEHOLDER_" + template_info.get('sourceUrl', '')
_template['mute'] = template_info.get('mute', True)
_template['crop_mode'] = template_info.get('cropEnable', None)
_template['zoom_cut'] = template_info.get('zoomCut', None)
_template["source"] = "PLACEHOLDER_" + template_info.get(
"sourceUrl", ""
)
_template["mute"] = template_info.get("mute", True)
_template["crop_mode"] = template_info.get("cropEnable", None)
_template["zoom_cut"] = template_info.get("zoomCut", None)
else:
_template['source'] = None
_overlays = template_info.get('overlays', '')
_template["source"] = None
_overlays = template_info.get("overlays", "")
if _overlays:
_template['overlays'] = _overlays.split(",")
_audios = template_info.get('audios', '')
_template["overlays"] = _overlays.split(",")
_audios = template_info.get("audios", "")
if _audios:
_template['audios'] = _audios.split(",")
_luts = template_info.get('luts', '')
_template["audios"] = _audios.split(",")
_luts = template_info.get("luts", "")
if _luts:
_template['luts'] = _luts.split(",")
_only_if = template_info.get('onlyIf', '')
_template["luts"] = _luts.split(",")
_only_if = template_info.get("onlyIf", "")
if _only_if:
_template['only_if'] = _only_if
_effects = template_info.get('effects', '')
_template["only_if"] = _only_if
_effects = template_info.get("effects", "")
if _effects:
_template['effects'] = _effects.split("|")
_template["effects"] = _effects.split("|")
return _template
# outer template definition
overall_template = _template_normalizer(remote_template_info)
template['overall_template'] = overall_template
template["overall_template"] = overall_template
# inter template definition
inter_template_list = remote_template_info.get('children', [])
inter_template_list = remote_template_info.get("children", [])
for children_template in inter_template_list:
parts = _template_normalizer(children_template)
template['video_parts'].append(parts)
template['local_path'] = os.path.join(os.getenv('TEMPLATE_DIR'), str(template_id))
with get_tracer("api").start_as_current_span("get_template_info.template") as res_span:
template["video_parts"].append(parts)
template["local_path"] = os.path.join(
os.getenv("TEMPLATE_DIR"), str(template_id)
)
with get_tracer("api").start_as_current_span(
"get_template_info.template"
) as res_span:
res_span.set_attribute("normalized.response", json.dumps(template))
return template
@@ -154,12 +199,19 @@ def report_task_success(task_info, **kwargs):
with tracer.start_as_current_span("report_task_success.request") as req_span:
try:
req_span.set_attribute("http.method", "POST")
req_span.set_attribute("http.url",
'{0}/{1}/success'.format(os.getenv('API_ENDPOINT'), task_info.get("id")))
response = session.post('{0}/{1}/success'.format(os.getenv('API_ENDPOINT'), task_info.get("id")), json={
'accessKey': os.getenv('ACCESS_KEY'),
**kwargs
}, timeout=10)
req_span.set_attribute(
"http.url",
"{0}/{1}/success".format(
os.getenv("API_ENDPOINT"), task_info.get("id")
),
)
response = session.post(
"{0}/{1}/success".format(
os.getenv("API_ENDPOINT"), task_info.get("id")
),
json={"accessKey": os.getenv("ACCESS_KEY"), **kwargs},
timeout=10,
)
req_span.set_attribute("http.status_code", response.status_code)
req_span.set_attribute("http.response", response.text)
response.raise_for_status()
@@ -176,11 +228,21 @@ def report_task_start(task_info):
with tracer.start_as_current_span("report_task_start.request") as req_span:
try:
req_span.set_attribute("http.method", "POST")
req_span.set_attribute("http.url",
'{0}/{1}/start'.format(os.getenv('API_ENDPOINT'), task_info.get("id")))
response = session.post('{0}/{1}/start'.format(os.getenv('API_ENDPOINT'), task_info.get("id")), json={
'accessKey': os.getenv('ACCESS_KEY'),
}, timeout=10)
req_span.set_attribute(
"http.url",
"{0}/{1}/start".format(
os.getenv("API_ENDPOINT"), task_info.get("id")
),
)
response = session.post(
"{0}/{1}/start".format(
os.getenv("API_ENDPOINT"), task_info.get("id")
),
json={
"accessKey": os.getenv("ACCESS_KEY"),
},
timeout=10,
)
req_span.set_attribute("http.status_code", response.status_code)
req_span.set_attribute("http.response", response.text)
response.raise_for_status()
@@ -191,7 +253,7 @@ def report_task_start(task_info):
return None
def report_task_failed(task_info, reason=''):
def report_task_failed(task_info, reason=""):
tracer = get_tracer(__name__)
with tracer.start_as_current_span("report_task_failed") as span:
span.set_attribute("task_id", task_info.get("id"))
@@ -199,12 +261,19 @@ def report_task_failed(task_info, reason=''):
with tracer.start_as_current_span("report_task_failed.request") as req_span:
try:
req_span.set_attribute("http.method", "POST")
req_span.set_attribute("http.url",
'{0}/{1}/fail'.format(os.getenv('API_ENDPOINT'), task_info.get("id")))
response = session.post('{0}/{1}/fail'.format(os.getenv('API_ENDPOINT'), task_info.get("id")), json={
'accessKey': os.getenv('ACCESS_KEY'),
'reason': reason
}, timeout=10)
req_span.set_attribute(
"http.url",
"{0}/{1}/fail".format(
os.getenv("API_ENDPOINT"), task_info.get("id")
),
)
response = session.post(
"{0}/{1}/fail".format(
os.getenv("API_ENDPOINT"), task_info.get("id")
),
json={"accessKey": os.getenv("ACCESS_KEY"), "reason": reason},
timeout=10,
)
req_span.set_attribute("http.status_code", response.status_code)
req_span.set_attribute("http.response", response.text)
response.raise_for_status()
@@ -221,15 +290,26 @@ def upload_task_file(task_info, ffmpeg_task):
with get_tracer("api").start_as_current_span("upload_task_file") as span:
logger.info("开始上传文件: %s", task_info.get("id"))
span.set_attribute("file.id", task_info.get("id"))
with tracer.start_as_current_span("upload_task_file.request_upload_url") as req_span:
with tracer.start_as_current_span(
"upload_task_file.request_upload_url"
) as req_span:
try:
req_span.set_attribute("http.method", "POST")
req_span.set_attribute("http.url",
'{0}/{1}/uploadUrl'.format(os.getenv('API_ENDPOINT'), task_info.get("id")))
response = session.post('{0}/{1}/uploadUrl'.format(os.getenv('API_ENDPOINT'), task_info.get("id")),
json={
'accessKey': os.getenv('ACCESS_KEY'),
}, timeout=10)
req_span.set_attribute(
"http.url",
"{0}/{1}/uploadUrl".format(
os.getenv("API_ENDPOINT"), task_info.get("id")
),
)
response = session.post(
"{0}/{1}/uploadUrl".format(
os.getenv("API_ENDPOINT"), task_info.get("id")
),
json={
"accessKey": os.getenv("ACCESS_KEY"),
},
timeout=10,
)
req_span.set_attribute("http.status_code", response.status_code)
req_span.set_attribute("http.response", response.text)
response.raise_for_status()
@@ -240,21 +320,25 @@ def upload_task_file(task_info, ffmpeg_task):
logger.error("请求失败!", e)
return False
data = response.json()
url = data.get('data', "")
url = data.get("data", "")
logger.info("开始上传文件: %s%s", task_info.get("id"), url)
return oss.upload_to_oss(url, ffmpeg_task.get_output_file())
def get_task_info(id):
try:
response = session.get(os.getenv('API_ENDPOINT') + "/" + id + "/info", params={
'accessKey': os.getenv('ACCESS_KEY'),
}, timeout=10)
response = session.get(
os.getenv("API_ENDPOINT") + "/" + id + "/info",
params={
"accessKey": os.getenv("ACCESS_KEY"),
},
timeout=10,
)
response.raise_for_status()
except requests.RequestException as e:
logger.error("请求失败!", e)
return []
data = response.json()
logger.debug("获取任务结果:【%s", data)
if data.get('code', 0) == 200:
return data.get('data', {})
if data.get("code", 0) == 200:
return data.get("data", {})

113
util/exceptions.py Normal file
View File

@@ -0,0 +1,113 @@
from typing import Optional
class RenderWorkerError(Exception):
"""RenderWorker基础异常类"""
def __init__(self, message: str, error_code: Optional[str] = None):
super().__init__(message)
self.message = message
self.error_code = error_code or self.__class__.__name__
class ConfigurationError(RenderWorkerError):
"""配置错误"""
pass
class TemplateError(RenderWorkerError):
"""模板相关错误"""
pass
class TemplateNotFoundError(TemplateError):
"""模板未找到错误"""
pass
class TemplateValidationError(TemplateError):
"""模板验证错误"""
pass
class TaskError(RenderWorkerError):
"""任务处理错误"""
pass
class TaskValidationError(TaskError):
"""任务参数验证错误"""
pass
class RenderError(RenderWorkerError):
"""渲染处理错误"""
pass
class FFmpegError(RenderError):
"""FFmpeg执行错误"""
def __init__(
self,
message: str,
command: Optional[list] = None,
return_code: Optional[int] = None,
stderr: Optional[str] = None,
):
super().__init__(message)
self.command = command
self.return_code = return_code
self.stderr = stderr
class EffectError(RenderError):
"""效果处理错误"""
def __init__(
self, message: str, effect_name: Optional[str] = None, effect_params: Optional[str] = None
):
super().__init__(message)
self.effect_name = effect_name
self.effect_params = effect_params
class StorageError(RenderWorkerError):
"""存储相关错误"""
pass
class APIError(RenderWorkerError):
"""API调用错误"""
def __init__(
self, message: str, status_code: Optional[int] = None, response_body: Optional[str] = None
):
super().__init__(message)
self.status_code = status_code
self.response_body = response_body
class ResourceError(RenderWorkerError):
"""资源相关错误"""
pass
class ResourceNotFoundError(ResourceError):
"""资源未找到错误"""
pass
class DownloadError(ResourceError):
"""下载错误"""
pass

View File

@@ -7,11 +7,19 @@ from typing import Optional, IO
from opentelemetry.trace import Status, StatusCode
from entity.ffmpeg import FfmpegTask, ENCODER_ARGS, VIDEO_ARGS, AUDIO_ARGS, MUTE_AUDIO_INPUT, get_mp4toannexb_filter
from entity.ffmpeg import (
FfmpegTask,
ENCODER_ARGS,
VIDEO_ARGS,
AUDIO_ARGS,
MUTE_AUDIO_INPUT,
get_mp4toannexb_filter,
)
from telemetry import get_tracer
logger = logging.getLogger(__name__)
def re_encode_and_annexb(file):
with get_tracer("ffmpeg").start_as_current_span("re_encode_and_annexb") as span:
span.set_attribute("file.path", file)
@@ -30,69 +38,68 @@ def re_encode_and_annexb(file):
_encoder_args = tuple(os.getenv("RE_ENCODE_ENCODER_ARGS", "").split(" "))
else:
_encoder_args = ENCODER_ARGS
ffmpeg_process = subprocess.run(["ffmpeg", "-y", "-hide_banner", "-i", file,
*(set() if has_audio else MUTE_AUDIO_INPUT),
"-fps_mode", "cfr",
"-map", "0:v", "-map", "0:a" if has_audio else "1:a",
*_video_args, "-bsf:v", get_mp4toannexb_filter(),
*AUDIO_ARGS, "-bsf:a", "setts=pts=DTS",
*_encoder_args, "-shortest", "-fflags", "+genpts",
"-f", "mpegts", file + ".ts"])
ffmpeg_process = subprocess.run(
[
"ffmpeg",
"-y",
"-hide_banner",
"-i",
file,
*(set() if has_audio else MUTE_AUDIO_INPUT),
"-fps_mode",
"cfr",
"-map",
"0:v",
"-map",
"0:a" if has_audio else "1:a",
*_video_args,
"-bsf:v",
get_mp4toannexb_filter(),
*AUDIO_ARGS,
"-bsf:a",
"setts=pts=DTS",
*_encoder_args,
"-shortest",
"-fflags",
"+genpts",
"-f",
"mpegts",
file + ".ts",
]
)
logger.info(" ".join(ffmpeg_process.args))
span.set_attribute("ffmpeg.args", json.dumps(ffmpeg_process.args))
logger.info("ReEncodeAndAnnexb: %s, returned: %s", file, ffmpeg_process.returncode)
logger.info(
"ReEncodeAndAnnexb: %s, returned: %s", file, ffmpeg_process.returncode
)
span.set_attribute("ffmpeg.code", ffmpeg_process.returncode)
if ffmpeg_process.returncode == 0:
span.set_status(Status(StatusCode.OK))
span.set_attribute("file.size", os.path.getsize(file+".ts"))
span.set_attribute("file.size", os.path.getsize(file + ".ts"))
# os.remove(file)
return file+".ts"
return file + ".ts"
else:
span.set_status(Status(StatusCode.ERROR))
return file
def start_render(ffmpeg_task: FfmpegTask):
tracer = get_tracer(__name__)
with tracer.start_as_current_span("start_render") as span:
span.set_attribute("ffmpeg.task", str(ffmpeg_task))
if not ffmpeg_task.need_run():
ffmpeg_task.set_output_file(ffmpeg_task.input_file[0])
span.set_status(Status(StatusCode.OK))
return True
ffmpeg_args = ffmpeg_task.get_ffmpeg_args()
if len(ffmpeg_args) == 0:
ffmpeg_task.set_output_file(ffmpeg_task.input_file[0])
span.set_status(Status(StatusCode.OK))
return True
ffmpeg_process = subprocess.run(["ffmpeg", "-progress", "-", "-loglevel", "error", *ffmpeg_args], stderr=subprocess.PIPE, **subprocess_args(True))
span.set_attribute("ffmpeg.args", json.dumps(ffmpeg_process.args))
logger.info(" ".join(ffmpeg_process.args))
ffmpeg_final_out = handle_ffmpeg_output(ffmpeg_process.stdout)
span.set_attribute("ffmpeg.out", ffmpeg_final_out)
logger.info("FINISH TASK, OUTPUT IS %s", ffmpeg_final_out)
code = ffmpeg_process.returncode
span.set_attribute("ffmpeg.code", code)
if code != 0:
span.set_attribute("ffmpeg.err", str(ffmpeg_process.stderr))
span.set_status(Status(StatusCode.ERROR, "FFMPEG异常退出"))
logger.error("FFMPEG ERROR: %s", ffmpeg_process.stderr)
return False
span.set_attribute("ffmpeg.out_file", ffmpeg_task.output_file)
try:
file_size = os.path.getsize(ffmpeg_task.output_file)
span.set_attribute("file.size", file_size)
if file_size < 4096:
span.set_status(Status(StatusCode.ERROR, "输出文件过小"))
logger.error("FFMPEG ERROR: OUTPUT FILE IS TOO SMALL")
return False
except OSError as e:
span.set_attribute("file.size", 0)
span.set_attribute("file.error", e.strerror)
span.set_status(Status(StatusCode.ERROR, "输出文件不存在"))
logger.error("FFMPEG ERROR: OUTPUT FILE NOT FOUND")
return False
span.set_status(Status(StatusCode.OK))
return True
# start_render函数已迁移到services/render_service.py中的DefaultRenderService
# 保留原有签名用于向后兼容,但建议使用新的服务架构
def start_render(ffmpeg_task):
"""
已迁移到新架构,建议使用 DefaultRenderService.render()
保留用于向后兼容
"""
logger.warning(
"start_render is deprecated, use DefaultRenderService.render() instead"
)
from services import DefaultRenderService
render_service = DefaultRenderService()
return render_service.render(ffmpeg_task)
def handle_ffmpeg_output(stdout: Optional[bytes]) -> str:
out_time = "0:0:0.0"
@@ -111,7 +118,8 @@ def handle_ffmpeg_output(stdout: Optional[bytes]) -> str:
if line.startswith(b"speed="):
speed = line.replace(b"speed=", b"").decode().strip()
print("[ ]Speed:", out_time, "@", speed)
return out_time+"@"+speed
return out_time + "@" + speed
def duration_str_to_float(duration_str: str) -> float:
_duration = datetime.strptime(duration_str, "%H:%M:%S.%f") - datetime(1900, 1, 1)
@@ -124,8 +132,18 @@ def probe_video_info(video_file):
span.set_attribute("video.file", video_file)
# 获取宽度和高度
result = subprocess.run(
["ffprobe", '-v', 'error', '-select_streams', 'v:0', '-show_entries', 'stream=width,height:format=duration', '-of',
'csv=s=x:p=0', video_file],
[
"ffprobe",
"-v",
"error",
"-select_streams",
"v:0",
"-show_entries",
"stream=width,height:format=duration",
"-of",
"csv=s=x:p=0",
video_file,
],
stderr=subprocess.STDOUT,
**subprocess_args(True)
)
@@ -134,14 +152,14 @@ def probe_video_info(video_file):
if result.returncode != 0:
span.set_status(Status(StatusCode.ERROR))
return 0, 0, 0
all_result = result.stdout.decode('utf-8').strip()
all_result = result.stdout.decode("utf-8").strip()
span.set_attribute("ffprobe.out", all_result)
if all_result == '':
if all_result == "":
span.set_status(Status(StatusCode.ERROR))
return 0, 0, 0
span.set_status(Status(StatusCode.OK))
wh, duration = all_result.split('\n')
width, height = wh.strip().split('x')
wh, duration = all_result.split("\n")
width, height = wh.strip().split("x")
return int(width), int(height), float(duration)
@@ -149,8 +167,19 @@ def probe_video_audio(video_file, type=None):
tracer = get_tracer(__name__)
with tracer.start_as_current_span("probe_video_audio") as span:
span.set_attribute("video.file", video_file)
args = ["ffprobe", "-hide_banner", "-v", "error", "-select_streams", "a", "-show_entries", "stream=index", "-of", "csv=p=0"]
if type == 'concat':
args = [
"ffprobe",
"-hide_banner",
"-v",
"error",
"-select_streams",
"a",
"-show_entries",
"stream=index",
"-of",
"csv=p=0",
]
if type == "concat":
args.append("-safe")
args.append("0")
args.append("-f")
@@ -160,16 +189,16 @@ def probe_video_audio(video_file, type=None):
result = subprocess.run(args, stderr=subprocess.STDOUT, **subprocess_args(True))
span.set_attribute("ffprobe.args", json.dumps(result.args))
span.set_attribute("ffprobe.code", result.returncode)
logger.info("probe_video_audio: %s", result.stdout.decode('utf-8').strip())
logger.info("probe_video_audio: %s", result.stdout.decode("utf-8").strip())
if result.returncode != 0:
return False
if result.stdout.decode('utf-8').strip() == '':
if result.stdout.decode("utf-8").strip() == "":
return False
return True
# 音频淡出2秒
def fade_out_audio(file, duration, fade_out_sec = 2):
def fade_out_audio(file, duration, fade_out_sec=2):
if type(duration) == str:
try:
duration = float(duration)
@@ -187,7 +216,25 @@ def fade_out_audio(file, duration, fade_out_sec = 2):
os.remove(new_fn)
logger.info("delete tmp file: " + new_fn)
try:
process = subprocess.run(["ffmpeg", "-i", file, "-c:v", "copy", "-c:a", "aac", "-af", "afade=t=out:st=" + str(duration - fade_out_sec) + ":d=" + str(fade_out_sec), "-y", new_fn], **subprocess_args(True))
process = subprocess.run(
[
"ffmpeg",
"-i",
file,
"-c:v",
"copy",
"-c:a",
"aac",
"-af",
"afade=t=out:st="
+ str(duration - fade_out_sec)
+ ":d="
+ str(fade_out_sec),
"-y",
new_fn,
],
**subprocess_args(True)
)
span.set_attribute("ffmpeg.args", json.dumps(process.args))
logger.info(" ".join(process.args))
if process.returncode != 0:
@@ -203,7 +250,6 @@ def fade_out_audio(file, duration, fade_out_sec = 2):
return file
# Create a set of arguments which make a ``subprocess.Popen`` (and
# variants) call work with or without Pyinstaller, ``--noconsole`` or
# not, on Windows and Linux. Typical use::
@@ -216,7 +262,7 @@ def fade_out_audio(file, duration, fade_out_sec = 2):
# **subprocess_args(False))
def subprocess_args(include_stdout=True):
# The following is true only on Windows.
if hasattr(subprocess, 'STARTUPINFO'):
if hasattr(subprocess, "STARTUPINFO"):
# On Windows, subprocess calls will pop up a command window by default
# when run from Pyinstaller with the ``--noconsole`` option. Avoid this
# distraction.
@@ -240,7 +286,7 @@ def subprocess_args(include_stdout=True):
#
# So, add it only if it's needed.
if include_stdout:
ret = {'stdout': subprocess.PIPE}
ret = {"stdout": subprocess.PIPE}
else:
ret = {}
@@ -248,8 +294,5 @@ def subprocess_args(include_stdout=True):
# with the ``--noconsole`` option requires redirecting everything
# (stdin, stdout, stderr) to avoid an OSError exception
# "[Error 6] the handle is invalid."
ret.update({'stdin': subprocess.PIPE,
'startupinfo': si,
'env': env})
ret.update({"stdin": subprocess.PIPE, "startupinfo": si, "env": env})
return ret

161
util/ffmpeg_utils.py Normal file
View File

@@ -0,0 +1,161 @@
"""
FFmpeg工具模块 - 提供FFmpeg命令构建和处理的公共函数
"""
import logging
from typing import List, Tuple, Optional
from config.settings import get_ffmpeg_config
logger = logging.getLogger(__name__)
def build_base_ffmpeg_args() -> List[str]:
"""
构建基础FFmpeg参数
Returns:
基础参数列表
"""
config = get_ffmpeg_config()
args = ["ffmpeg", "-y", "-hide_banner"]
args.extend(config.progress_args)
args.extend(config.loglevel_args)
return args
def build_null_audio_input() -> List[str]:
"""
构建空音频输入参数
Returns:
空音频输入参数列表
"""
config = get_ffmpeg_config()
return config.null_audio_args
def build_amix_filter(input1: str, input2: str, output: str) -> str:
"""
构建音频混合滤镜
Args:
input1: 第一个音频输入
input2: 第二个音频输入
output: 输出流名称
Returns:
混合滤镜字符串
"""
config = get_ffmpeg_config()
return f"{input1}[{input2}]{config.amix_args[0]}[{output}]"
def build_overlay_scale_filter(
video_input: str, overlay_input: str, output: str
) -> str:
"""
构建覆盖层缩放滤镜
Args:
video_input: 视频输入流
overlay_input: 覆盖层输入流
output: 输出流名称
Returns:
缩放滤镜字符串
"""
config = get_ffmpeg_config()
if config.overlay_scale_mode == "scale":
return f"{video_input}[{overlay_input}]scale=iw:ih[{output}]"
else:
return (
f"{video_input}[{overlay_input}]{config.overlay_scale_mode}=iw:ih[{output}]"
)
def get_annexb_filter() -> str:
"""
获取annexb转换滤镜
Returns:
annexb滤镜名称
"""
config = get_ffmpeg_config()
encoder_args_str = " ".join(config.encoder_args).lower()
if "hevc" in encoder_args_str:
return "hevc_mp4toannexb"
return "h264_mp4toannexb"
def build_standard_output_args() -> List[str]:
"""
构建标准输出参数
Returns:
输出参数列表
"""
config = get_ffmpeg_config()
return [
*config.video_args,
*config.audio_args,
*config.encoder_args,
*config.default_args,
]
def validate_ffmpeg_file_extensions(file_path: str) -> bool:
"""
验证文件扩展名是否为FFmpeg支持的格式
Args:
file_path: 文件路径
Returns:
是否为支持的格式
"""
supported_extensions = {
".mp4",
".avi",
".mov",
".mkv",
".flv",
".wmv",
".webm",
".ts",
".m2ts",
".mts",
".m4v",
".3gp",
".asf",
".rm",
".mp3",
".wav",
".aac",
".flac",
".ogg",
".m4a",
".wma",
}
import os
_, ext = os.path.splitext(file_path.lower())
return ext in supported_extensions
def estimate_processing_time(
input_duration: float, complexity_factor: float = 1.0
) -> float:
"""
估算处理时间
Args:
input_duration: 输入文件时长(秒)
complexity_factor: 复杂度因子(1.0为普通处理)
Returns:
预估处理时间(秒)
"""
# 基础处理速度假设为实时的0.5倍(即处理1秒视频需要2秒)
base_processing_ratio = 2.0
return input_duration * base_processing_ratio * complexity_factor

99
util/json_utils.py Normal file
View File

@@ -0,0 +1,99 @@
"""
JSON处理工具模块 - 提供安全的JSON解析和处理功能
"""
import json
import logging
from typing import Dict, Any, Optional, Union
logger = logging.getLogger(__name__)
def safe_json_loads(json_str: Union[str, bytes], default: Any = None) -> Any:
"""
安全解析JSON字符串
Args:
json_str: JSON字符串
default: 解析失败时返回的默认值
Returns:
解析后的对象,或默认值
"""
if not json_str or json_str == "{}":
return default or {}
try:
return json.loads(json_str)
except (json.JSONDecodeError, TypeError) as e:
logger.warning(f"Failed to parse JSON: {e}, input: {json_str!r}")
return default or {}
def safe_json_dumps(
obj: Any, indent: Optional[int] = None, ensure_ascii: bool = False
) -> str:
"""
安全序列化对象为JSON字符串
Args:
obj: 要序列化的对象
indent: 缩进空格数
ensure_ascii: 是否确保ASCII编码
Returns:
JSON字符串
"""
try:
return json.dumps(obj, indent=indent, ensure_ascii=ensure_ascii)
except (TypeError, ValueError) as e:
logger.error(f"Failed to serialize to JSON: {e}")
return "{}"
def get_nested_value(data: Dict[str, Any], key_path: str, default: Any = None) -> Any:
"""
从嵌套字典中安全获取值
Args:
data: 字典数据
key_path: 键路径,用点分隔(如 "user.profile.name"
default: 默认值
Returns:
找到的值或默认值
"""
if not isinstance(data, dict):
return default
try:
keys = key_path.split(".")
current = data
for key in keys:
if isinstance(current, dict) and key in current:
current = current[key]
else:
return default
return current
except Exception as e:
logger.warning(f"Failed to get nested value for path '{key_path}': {e}")
return default
def merge_dicts(*dicts: Dict[str, Any]) -> Dict[str, Any]:
"""
合并多个字典,后面的字典会覆盖前面的字典中相同的键
Args:
*dicts: 要合并的字典
Returns:
合并后的字典
"""
result = {}
for d in dicts:
if isinstance(d, dict):
result.update(d)
return result

View File

@@ -31,12 +31,14 @@ def upload_to_oss(url, file_path):
if replace_map != "":
replace_list = [i.split("|", 1) for i in replace_map.split(",")]
new_url = url
for (_src, _dst) in replace_list:
for _src, _dst in replace_list:
new_url = new_url.replace(_src, _dst)
new_url = new_url.split("?", 1)[0]
r_span.set_attribute("rclone.target_dir", new_url)
if new_url != url:
result = os.system(f"rclone copyto --no-check-dest --ignore-existing --multi-thread-chunk-size 8M --multi-thread-streams 8 {file_path} {new_url}")
result = os.system(
f"rclone copyto --no-check-dest --ignore-existing --multi-thread-chunk-size 8M --multi-thread-streams 8 {file_path} {new_url}"
)
r_span.set_attribute("rclone.result", result)
if result == 0:
span.set_status(Status(StatusCode.OK))
@@ -49,8 +51,14 @@ def upload_to_oss(url, file_path):
try:
req_span.set_attribute("http.method", "PUT")
req_span.set_attribute("http.url", url)
with open(file_path, 'rb') as f:
response = requests.put(url, data=f, stream=True, timeout=60, headers={"Content-Type": "video/mp4"})
with open(file_path, "rb") as f:
response = requests.put(
url,
data=f,
stream=True,
timeout=60,
headers={"Content-Type": "video/mp4"},
)
req_span.set_attribute("http.status_code", response.status_code)
req_span.set_attribute("http.response", response.text)
response.raise_for_status()
@@ -61,12 +69,16 @@ def upload_to_oss(url, file_path):
req_span.set_attribute("http.error", "Timeout")
req_span.set_status(Status(StatusCode.ERROR))
retries += 1
logger.warning(f"Upload timed out. Retrying {retries}/{max_retries}...")
logger.warning(
f"Upload timed out. Retrying {retries}/{max_retries}..."
)
except Exception as e:
req_span.set_attribute("http.error", str(e))
req_span.set_status(Status(StatusCode.ERROR))
retries += 1
logger.warning(f"Upload failed. Retrying {retries}/{max_retries}...")
logger.warning(
f"Upload failed. Retrying {retries}/{max_retries}..."
)
span.set_status(Status(StatusCode.ERROR))
return False
@@ -83,11 +95,11 @@ def download_from_oss(url, file_path, skip_if_exist=None):
with tracer.start_as_current_span("download_from_oss") as span:
span.set_attribute("file.url", url)
span.set_attribute("file.path", file_path)
# 如果skip_if_exist为None,则从启动参数中读取
if skip_if_exist is None:
skip_if_exist = 'skip_if_exist' in sys.argv
skip_if_exist = "skip_if_exist" in sys.argv
if skip_if_exist and os.path.exists(file_path):
span.set_attribute("file.exist", True)
span.set_attribute("file.size", os.path.getsize(file_path))
@@ -107,7 +119,7 @@ def download_from_oss(url, file_path, skip_if_exist=None):
req_span.set_attribute("http.url", url)
response = requests.get(url, timeout=15) # 设置超时时间
req_span.set_attribute("http.status_code", response.status_code)
with open(file_path, 'wb') as f:
with open(file_path, "wb") as f:
f.write(response.content)
req_span.set_attribute("file.size", os.path.getsize(file_path))
req_span.set_status(Status(StatusCode.OK))
@@ -117,11 +129,15 @@ def download_from_oss(url, file_path, skip_if_exist=None):
req_span.set_attribute("http.error", "Timeout")
req_span.set_status(Status(StatusCode.ERROR))
retries += 1
logger.warning(f"Download timed out. Retrying {retries}/{max_retries}...")
logger.warning(
f"Download timed out. Retrying {retries}/{max_retries}..."
)
except Exception as e:
req_span.set_attribute("http.error", str(e))
req_span.set_status(Status(StatusCode.ERROR))
retries += 1
logger.warning(f"Download failed. Retrying {retries}/{max_retries}...")
logger.warning(
f"Download failed. Retrying {retries}/{max_retries}..."
)
span.set_status(Status(StatusCode.ERROR))
return False

View File

@@ -11,14 +11,14 @@ def get_sys_info():
Returns a dictionary with system information.
"""
info = {
'version': SOFTWARE_VERSION,
'client_datetime': datetime.now().isoformat(),
'platform': platform.system(),
'runtime_version': 'Python ' + platform.python_version(),
'cpu_count': os.cpu_count(),
'cpu_usage': psutil.cpu_percent(),
'memory_total': psutil.virtual_memory().total,
'memory_available': psutil.virtual_memory().available,
'support_feature': SUPPORT_FEATURE
"version": SOFTWARE_VERSION,
"client_datetime": datetime.now().isoformat(),
"platform": platform.system(),
"runtime_version": "Python " + platform.python_version(),
"cpu_count": os.cpu_count(),
"cpu_usage": psutil.cpu_percent(),
"memory_total": psutil.virtual_memory().total,
"memory_available": psutil.virtual_memory().available,
"support_feature": SUPPORT_FEATURE,
}
return info