Throttle progress updates to reduce database write operations during large dataset processing.
Key features:
- Add PROGRESS_UPDATE_INTERVAL config (default 2.0s, configurable via AUTO_ANNOTATION_PROGRESS_INTERVAL env)
- Conditional progress updates: Only write to DB when (now - last_update) >= interval
- Use time.monotonic() for timing (immune to system clock adjustments)
- Final status updates (completed/stopped/failed) always execute (not throttled)
Implementation:
- Initialize last_progress_update timestamp before as_completed() loop
- Replace unconditional _update_task_status() with conditional call based on time interval
- Update docstring to reflect throttling capability
Performance impact (T=2s):
- 1,000 files / 100s processing: DB writes reduced from 1,000 to ~50 (95% reduction)
- 10,000 files / 500s processing: DB writes reduced from 10,000 to ~250 (97.5% reduction)
- Small datasets (10 files): Minimal difference
Backward compatibility:
- PROGRESS_UPDATE_INTERVAL=0: Updates every file (identical to previous behavior)
- Heartbeat mechanism unaffected (2s interval << 300s timeout)
- Stop check mechanism independent of progress updates
- Final status updates always execute
Testing:
- 14 unit tests all passed (11 existing + 3 new):
* Fast processing with throttling
* PROGRESS_UPDATE_INTERVAL=0 updates every file
* Slow processing (per-file > T) updates every file
- py_compile syntax check passed
Edge cases handled:
- Single file task: Works normally
- Very slow processing: Degrades to per-file updates
- Concurrent FILE_WORKERS > 1: Counters accurate (lock-protected), DB reflects with max T seconds delay
Automatically recover running tasks with stale heartbeats on worker startup, preventing tasks from being permanently stuck after container restarts.
Key changes:
- Add HEARTBEAT_TIMEOUT_SECONDS constant (default 300s, configurable via env)
- Add _recover_stale_running_tasks() function:
* Scans for status='running' tasks with heartbeat timeout
* No progress (processed=0) → reset to pending (auto-retry)
* Has progress (processed>0) → mark as failed with Chinese error message
* Each task recovery is independent (single failure doesn't affect others)
* Skip recovery if timeout is 0 or negative (disable feature)
- Call recovery function in _worker_loop() before polling loop
- Update file header comments to reflect recovery mechanism
Recovery logic:
- Query: status='running' AND (heartbeat_at IS NULL OR heartbeat_at < NOW() - timeout)
- Decision based on processed_images count
- Clear run_token to allow other workers to claim
- Single transaction per task for atomicity
Edge cases handled:
- Database unavailable: recovery failure doesn't block worker startup
- Concurrent recovery: UPDATE WHERE status='running' prevents duplicates
- NULL heartbeat: extreme case (crash right after claim) also recovered
- stop_requested tasks: automatically excluded by _fetch_pending_task()
Testing:
- 8 unit tests all passed:
* No timeout tasks
* Timeout disabled
* No progress → pending
* Has progress → failed
* NULL heartbeat recovery
* Multiple tasks mixed processing
* DB error doesn't crash
* Negative timeout disables feature
Automatically convert auto-annotation outputs to Label Studio format and write to t_dm_annotation_results table, enabling seamless editing in the annotation editor.
New file:
- runtime/python-executor/datamate/annotation_result_converter.py
* 4 converters for different annotation types:
- convert_text_classification → choices type
- convert_ner → labels (span) type
- convert_relation_extraction → labels + relation type
- convert_object_detection → rectanglelabels type
* convert_annotation() dispatcher (auto-detects task_type)
* generate_label_config_xml() for dynamic XML generation
* Pipeline introspection utilities
* Label Studio ID generation logic
Modified file:
- runtime/python-executor/datamate/auto_annotation_worker.py
* Preserve file_id through processing loop (line 918)
* Collect file_results as (file_id, annotations) pairs
* New _create_labeling_project_with_annotations() function:
- Creates labeling project linked to source dataset
- Snapshots all files
- Converts results to Label Studio format
- Writes to t_dm_annotation_results in single transaction
* label_config XML stored in t_dm_labeling_projects.configuration
Key features:
- Supports 4 annotation types: text classification, NER, relation extraction, object detection
- Deterministic region IDs for entity references in relation extraction
- Pixel to percentage conversion for object detection
- XML escaping handled by xml.etree.ElementTree
- Partial results preserved on task stop
Users can now view and edit auto-annotation results seamlessly in the annotation editor.
Add three new LLM-powered auto-annotation operators:
- LLMTextClassification: Text classification using LLM
- LLMNamedEntityRecognition: Named entity recognition with type validation
- LLMRelationExtraction: Relation extraction with entity and relation type validation
Key features:
- Load LLM config from t_model_config table via modelId parameter
- Lazy loading of LLM configuration on first execute()
- Result validation with whitelist checking for entity/relation types
- Fault-tolerant: returns empty results on LLM failure instead of throwing
- Fully compatible with existing Worker pipeline
Files added:
- runtime/ops/annotation/_llm_utils.py: Shared LLM utilities
- runtime/ops/annotation/llm_text_classification/: Text classification operator
- runtime/ops/annotation/llm_named_entity_recognition/: NER operator
- runtime/ops/annotation/llm_relation_extraction/: Relation extraction operator
Files modified:
- runtime/ops/annotation/__init__.py: Register 3 new operators
- runtime/python-executor/datamate/auto_annotation_worker.py: Add to Worker whitelist
- frontend/src/pages/DataAnnotation/OperatorCreate/hooks/useOperatorOperations.ts: Add to frontend whitelist
新增 AnnotationMigrator 迁移算法，在 TEXT 类型数据集的文件版本更新时，
可选通过 difflib 位置偏移映射和文字二次匹配将旧版本标注迁移到新版本上。
前端版本切换对话框增加"保留标注"复选框（仅 TEXT 类型显示），后端 API
增加 preserveAnnotations 参数，完全向后兼容。
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Problem:
use_new_version returned 404 annotation not found for files without
annotation, preventing users from switching to new versions.
Solution:
1. Query latest file by logical_path
2. Update LabelingProjectFile to point to latest version
3. If annotation exists: clear it and update file_id
4. If no annotation: just update project file snapshot
5. Return new file_id in response
Change has_new_version logic to compare current file version with
latest version, regardless of whether annotation exists.
Before: Only show warning if annotation exists and version is outdated
After: Show warning if current file is not the latest version
This ensures users are informed when viewing an old file version,
even if they haven't started annotating yet.
Problem:
check_file_version was comparing annotation version with the passed
file_id's version, but when files are updated, new file records are
created with higher versions and old ones are marked ARCHIVED.
Solution:
1. Query the latest ACTIVE file by logical_path
2. Compare annotation version with latest file version
3. Return latestFileId so frontend can switch to new version
Changes:
- check_file_version now queries latest version by logical_path
- Added latest_file_id to FileVersionCheckResponse schema
- Updated descriptions to clarify currentFileVersion is latest version
Database scenario:
- old file: id=6dae9f2f, version=1, status=ARCHIVED
- new file: id=3365b4e7, version=3, status=ACTIVE
- Both have same logical_path='rufus.ini'
- Now correctly detects version 3 > annotation version
Replace datetime.datetime.now(datetime.UTC) with datetime.datetime.now()
to fix compatibility issues with Python 3.10 and earlier versions.
datetime.UTC is only available in Python 3.11+, causing 500 errors
in production environment.
Files fixed:
- app/module/dataset/service/pdf_extract.py
- app/module/generation/service/export_service.py
Update data-annotation-init.sql and Alembic migration to support both new and old deployments:
SQL Initialization Script (data-annotation-init.sql):
- Add file_version column to t_dm_annotation_results table
- Add Alembic version table creation and version insertion
- New deployments using this script will have latest schema and Alembic version marked
Alembic Migration (20250205_0001_add_file_version.py):
- Add column_exists() helper function to detect if column already exists
- Add compatibility check in upgrade(): skip if column exists (new SQL init)
- Add informative print messages for deployment clarity
- Enhanced docstrings explaining compatibility strategy
Deployment Scenarios:
1. New deployment with latest SQL script: Schema created with file_version, Alembic marked as applied
2. Old deployment upgrade: Alembic detects missing column and adds it
This ensures backward compatibility while supporting fresh installs with complete schema.
Add support for detecting new file versions and switching to them:
Backend Changes:
- Add file_version column to AnnotationResult model
- Create Alembic migration for database schema update
- Implement check_file_version() method to compare annotation and file versions
- Implement use_new_version() method to clear annotations and update version
- Update upsert_annotation() to record file version when saving
- Add new API endpoints: GET /version and POST /use-new-version
- Add FileVersionCheckResponse and UseNewVersionResponse schemas
Frontend Changes:
- Add checkFileVersionUsingGet and useNewVersionUsingPost API calls
- Add version warning banner showing current vs latest file version
- Add 'Use New Version' button with confirmation dialog
- Clear version info state when switching files to avoid stale warnings
Bug Fixes:
- Fix previousFileVersion returning updated value (save before update)
- Handle null file_version for historical data compatibility
- Fix segmented annotation clearing (preserve structure, clear results)
- Fix files without annotations incorrectly showing new version warnings
- Preserve total_segments when clearing segmented annotations
Files Modified:
- frontend/src/pages/DataAnnotation/Annotate/LabelStudioTextEditor.tsx
- frontend/src/pages/DataAnnotation/annotation.api.ts
- runtime/datamate-python/app/db/models/annotation_management.py
- runtime/datamate-python/app/module/annotation/interface/editor.py
- runtime/datamate-python/app/module/annotation/schema/editor.py
- runtime/datamate-python/app/module/annotation/service/editor.py
New Files:
- runtime/datamate-python/alembic.ini
- runtime/datamate-python/alembic/env.py
- runtime/datamate-python/alembic/script.py.mako
- runtime/datamate-python/alembic/versions/20250205_0001_add_file_version.py
