Automatically recover running tasks with stale heartbeats on worker startup, preventing tasks from being permanently stuck after container restarts.
Key changes:
- Add HEARTBEAT_TIMEOUT_SECONDS constant (default 300s, configurable via env)
- Add _recover_stale_running_tasks() function:
* Scans for status='running' tasks with heartbeat timeout
* No progress (processed=0) → reset to pending (auto-retry)
* Has progress (processed>0) → mark as failed with Chinese error message
* Each task recovery is independent (single failure doesn't affect others)
* Skip recovery if timeout is 0 or negative (disable feature)
- Call recovery function in _worker_loop() before polling loop
- Update file header comments to reflect recovery mechanism
Recovery logic:
- Query: status='running' AND (heartbeat_at IS NULL OR heartbeat_at < NOW() - timeout)
- Decision based on processed_images count
- Clear run_token to allow other workers to claim
- Single transaction per task for atomicity
Edge cases handled:
- Database unavailable: recovery failure doesn't block worker startup
- Concurrent recovery: UPDATE WHERE status='running' prevents duplicates
- NULL heartbeat: extreme case (crash right after claim) also recovered
- stop_requested tasks: automatically excluded by _fetch_pending_task()
Testing:
- 8 unit tests all passed:
* No timeout tasks
* Timeout disabled
* No progress → pending
* Has progress → failed
* NULL heartbeat recovery
* Multiple tasks mixed processing
* DB error doesn't crash
* Negative timeout disables feature
q792602257
2fbfefdb91