Commit Graph

28 Commits

Author SHA1 Message Date
9376382791 refactor(annotation): 简化知识库集查找逻辑
- 移除项目参数依赖,简化 _find_knowledge_set_by_name 方法
- 删除不再使用的 _parse_metadata 和 _metadata_matches_project 方法
- 更新知识库集创建流程中的查找调用方式
- 统一所有知识库集查找操作的参数结构
2026-01-22 20:18:42 +08:00
cb71712f51 feat(annotation): 优化知识集合同步服务逻辑
- 添加 KNOWLEDGE_SET_LIST_SIZE 常量配置知识集列表大小为50
- 实现现有知识集名称检查避免重复创建
- 添加 _list_knowledge_sets 方法用于查询知识集列表
- 实现 _parse_metadata 和 _metadata_matches_project 方法解析匹配项目元数据
- 添加 _find_knowledge_set_by_name 方法按名称查找知识集
- 优化知识集创建流程增加重试和回退机制
- 实现项目配置更新保存知识集ID和名称信息
2026-01-22 17:48:27 +08:00
1eee1e248e feat(annotation): 添加分段索引支持和优化标注编辑器
- 在前端 lsf.html 中添加 segmentIndex 字段解析逻辑
- 在 LabelStudioTextEditor 中添加分段索引相关类型定义和处理函数
- 使用 useCallback 优化组件中的异步函数性能
- 添加对驼峰命名和下划线命名的数据字段兼容处理
- 实现分段模式下的标注状态更新功能
- 添加任务 ID 验证防止过期保存请求
- 在后端 editor.py 中添加分段索引字段支持
- 统一前后端数据传输格式确保字段一致性
2026-01-22 17:14:37 +08:00
c638182c72 feat(annotation): 增强标注编辑器的文本数据处理功能
- 添加 JSON 和 XML 解析支持用于处理标注配置
- 实现文本占位符填充机制优化用户体验
- 集成标签工作室配置管理功能
- 添加文本对象类型检测和分类处理
- 实现标注配置装饰器增强编辑器
2026-01-22 16:22:32 +08:00
73f0ab65fa feat(annotation): 实现标注结果同步到知识管理功能
- 在知识条目实体中新增来源数据集ID和文件ID字段
- 实现标注编辑器中同步标注结果到知识管理的服务逻辑
- 添加知识同步服务类处理标注到知识条目的转换和同步
- 实现通过下载接口获取文本内容的独立服务模块
- 更新知识条目查询接口支持按来源数据集和文件ID过滤
- 自动创建和关联标注项目对应的知识集
- 支持文本和Markdown文件的内容合并标注结果
- 添加同步过程中的错误处理和日志记录机制
2026-01-21 16:09:34 +08:00
ec27e2fa3e Merge branch 'editor_next' into lsf 2026-01-21 13:28:01 +08:00
ea6765ea0f fix(annotation): 修改数据集文件状态查询逻辑
- 将文件状态查询从仅统计 ACTIVE 状态扩展为 ACTIVE 和 COMPLETED 状态
- 使用 in_ 操作符替代等于操作符以支持多状态查询
- 保持原有数据集标注计数功能不变
2026-01-20 00:30:21 +08:00
cc0a977349 feat(annotation): 添加标注任务的数据量统计功能
- 在前端表格中新增数据量和已标注列显示
- 添加标注完成百分比计算和提示功能
- 在后端schema中增加totalCount和annotatedCount字段
- 实现项目统计数据查询服务方法
- 集成前后端数据映射和接口响应更新
2026-01-19 22:43:41 +08:00
649ab2f6bb refactor(annotation): 移除调试日志和异常堆栈跟踪
- 移除了项目映射获取接口中的traceback打印
- 简化了内部服务器错误响应消息
- 删除了映射服务中的多个调试日志输出
- 清理了响应数据构建过程中的调试信息
2026-01-19 21:58:00 +08:00
496161b1f1 ```
chore(annotation): 添加调试日志到映射服务

- 在 _to_response_from_row 方法中添加配置和标签配置的调试日志
- 在 _to_response 方法中添加映射ID和配置信息的调试日志
- 添加响应数据键名的调试日志
- 优化配置解析逻辑以确保字典类型的正确检查
```
2026-01-19 21:52:01 +08:00
f4a86b4af1 feat(annotation): 添加 labelConfig 字段并优化配置解析逻辑
- 在 DatasetMappingResponse 模型中新增 label_config 字段
- 修改前端获取 labelConfig 的逻辑,优先使用任务自身配置
- 移除模板配置的 condition 分支,统一从 XML 解析配置
- 更新后端服务从 configuration JSON 字段中提取 label_config 和 description
- 优化前后端配置解析的一致性处理
2026-01-19 21:39:00 +08:00
71c4a8d8a6 feat(annotation): 添加文本分段标注功能
- 引入文本分割器实现长文本按200字符自动分段
- 增加分段状态管理和段落导航界面
- 支持按段落保存和加载标注数据
- 实现分段模式下的标注状态跟踪
- 扩展API接口支持段落索引参数
- 添加分段相关的数据模型定义
2026-01-19 18:18:19 +08:00
70ea998564 feat(annotation): 优化标注编辑器的标签配置获取逻辑
- 优先使用项目配置中的label_config(用户编辑版本)
- 其次使用模板默认配置作为备选方案
- 支持从项目配置字典中获取label_config字段
- 保持向后兼容性,当项目配置无效时回退到模板配置
2026-01-19 16:34:20 +08:00
b992b08b2c feat(annotation): 扩展标注模板功能支持多模态数据类型
- 扩展数据类型支持包括pdf/chat/html/table等多种格式
- 新增标注类型涵盖asr/ner/object-detection等专业领域
- 添加label_config字段用于Label Studio XML配置存储
- 更新模板分类体系为audio-speech/chat/computer-vision/nlp等
- 实现预定义label_config优先使用的配置加载逻辑
- 完善数据库初始化脚本包含多模态标注模板数据
2026-01-18 20:35:34 +08:00
0c97648a9e fix(annotation): 修复导出统计功能中的文件状态过滤问题
- 在获取总文件数时添加 ACTIVE 状态过滤条件
- 修改已标注文件数统计逻辑,使用 distinct(file_id) 进行计数
- 在导出功能中为所有文件查询添加 ACTIVE 状态过滤
- 增加日志记录以跟踪导出统计过程
- 修正
2026-01-18 17:35:40 +08:00
c48d2fdeb8 feat(annotation): 添加标注数据导出功能
- 新增导出对话框组件,支持多种格式选择
- 实现 JSON、JSONL、CSV、COCO、YOLO 五种导出格式
- 添加导出统计信息显示,包括总文件数和已标注数
- 集成前端导出按钮和后端 API 接口
- 支持仅导出已标注数据和包含原始数据选项
- 实现文件下载和命名功能
2026-01-18 16:54:02 +08:00
e1c41a93c3 refactor(annotation): 优化模板生成逻辑移除文本类型特殊处理
Some checks failed
CodeQL Advanced / Analyze (actions) (push) Has been cancelled
CodeQL Advanced / Analyze (java-kotlin) (push) Has been cancelled
CodeQL Advanced / Analyze (javascript-typescript) (push) Has been cancelled
CodeQL Advanced / Analyze (python) (push) Has been cancelled
- 移除了 text_object_types 变量定义
- 删除了 is_text_template 判断逻辑
- 移除了长文本优化的双栏布局实现
- 添加了关于 Label Studio 默认侧栏控件行为的说明
- 简化了 XML 结构生成逻辑
2026-01-09 18:52:52 +08:00
7de49feb66 feat(annotation): 优化标注模板生成和配置验证
- 添加文本对象类型集合用于模板类型判断
- 将XML生成部分拆分为对象和控件两个独立部分
- 为文本类模板添加响应式布局支持长文本标注
- 修复配置验证器中对象和控件查找逻辑
- 优化标签控件在长文本场景下的显示位置
2026-01-09 13:39:55 +08:00
08336e2a13 feat(annotation): 添加标注模板配置功能
- 在schema中新增choice和show_inline字段支持选择模式配置
- 为编辑器服务添加空标注创建逻辑避免前端异常
- 实现标签类型的标准化处理和大小写兼容
- 支持Choices标签的单选/多选和行内显示配置
- 优化前端界面滚动条显示控制样式
2026-01-09 13:05:09 +08:00
a82f4f1bc3 refactor(annotation): 移除对 Label Studio Server 的依赖并切换到内嵌编辑器模式
- 移除 LabelStudioClient 和 SyncService 的导入及使用
- 删除与 Label Studio 项目的创建、删除和同步相关代码
- 修改创建数据集映射功能,改为创建 DataMate 标注项目
- 更新删除映射接口,仅进行软删除不再删除 Label Studio 项目
- 修改同步接口为兼容性保留,实际操作为空操作
- 移除 Label Studio 连接诊断功能
- 更新文档说明以反映内嵌编辑器模式的变化
2026-01-09 12:31:03 +08:00
d5b75fee0d LSF 2026-01-07 00:00:16 +08:00
Kecheng Sha
3f1ad6a872 feat(auto-annotation): integrate YOLO auto-labeling and enhance data management (#223)
* feat(auto-annotation): initial setup

* chore: remove package-lock.json

* chore: 清理本地测试脚本与 Maven 设置

* chore: change package-lock.json
2026-01-05 14:22:44 +08:00
Jason Wang
df853a5177 feat: Enhance file tag update functionality with automatic format conversion (#84)
- Updated `update_file_tags` to support both simplified and full tag formats.
- Introduced `TagFormatConverter` to handle conversion from simplified external tags to internal storage format.
- Added logic to fetch and utilize the appropriate annotation template for conversion.
- Improved error handling for missing templates and unknown controls during tag updates.
- Created example script demonstrating the usage of the new tag format conversion feature.
- Added unit tests for `TagFormatConverter` to ensure correct functionality and edge case handling.
2025-11-14 12:42:39 +08:00
Jason Wang
45743f39f5 feat: add labeling template. refactor: switch to Poetry, build and deploy of backend Python (#79)
* feat: Enhance annotation module with template management and validation

- Added DatasetMappingCreateRequest and DatasetMappingUpdateRequest schemas to handle dataset mapping requests with camelCase and snake_case support.
- Introduced Annotation Template schemas including CreateAnnotationTemplateRequest, UpdateAnnotationTemplateRequest, and AnnotationTemplateResponse for managing annotation templates.
- Implemented AnnotationTemplateService for creating, updating, retrieving, and deleting annotation templates, including validation of configurations and XML generation.
- Added utility class LabelStudioConfigValidator for validating Label Studio configurations and XML formats.
- Updated database schema for annotation templates and labeling projects to include new fields and constraints.
- Seeded initial annotation templates for various use cases including image classification, object detection, and text classification.

* feat: Enhance TemplateForm with improved validation and dynamic field rendering; update LabelStudio config validation for camelCase support

* feat: Update docker-compose.yml to mark datamate dataset volume and network as external

* feat: Add tag configuration management and related components

- Introduced new components for tag selection and browsing in the frontend.
- Added API endpoint to fetch tag configuration from the backend.
- Implemented tag configuration management in the backend, including loading from YAML.
- Enhanced template service to support dynamic tag rendering based on configuration.
- Updated validation utilities to incorporate tag configuration checks.
- Refactored existing code to utilize the new tag configuration structure.

* feat: Refactor LabelStudioTagConfig for improved configuration loading and validation

* feat: Update Makefile to include backend-python-docker-build in the build process

* feat: Migrate to poetry for better deps management

* Add pyyaml dependency and update Dockerfile to use Poetry for dependency management

- Added pyyaml (>=6.0.3,<7.0.0) to pyproject.toml dependencies.
- Updated Dockerfile to install Poetry and manage dependencies using it.
- Improved layer caching by copying only dependency files before the application code.
- Removed unnecessary installation of build dependencies to keep the final image size small.

* feat: Remove duplicated backend-python-docker-build target from Makefile

* fix: airflow is not ready for adding yet

* feat: update Python version to 3.12 and remove project installation step in Dockerfile
2025-11-13 15:32:30 +08:00
Jason Wang
c5ccc56cca feat: Add labeling template (#72)
* feat: Enhance annotation module with template management and validation

- Added DatasetMappingCreateRequest and DatasetMappingUpdateRequest schemas to handle dataset mapping requests with camelCase and snake_case support.
- Introduced Annotation Template schemas including CreateAnnotationTemplateRequest, UpdateAnnotationTemplateRequest, and AnnotationTemplateResponse for managing annotation templates.
- Implemented AnnotationTemplateService for creating, updating, retrieving, and deleting annotation templates, including validation of configurations and XML generation.
- Added utility class LabelStudioConfigValidator for validating Label Studio configurations and XML formats.
- Updated database schema for annotation templates and labeling projects to include new fields and constraints.
- Seeded initial annotation templates for various use cases including image classification, object detection, and text classification.

* feat: Enhance TemplateForm with improved validation and dynamic field rendering; update LabelStudio config validation for camelCase support

* feat: Update docker-compose.yml to mark datamate dataset volume and network as external
2025-11-11 09:14:14 +08:00
Jason Wang
78f50ea520 feat: File and Annotation 2-way sync implementation (#63)
* feat: Refactor configuration and sync logic for improved dataset handling and logging

* feat: Enhance annotation synchronization and dataset file management

- Added new fields `tags_updated_at` to `DatasetFiles` model for tracking the last update time of tags.
- Implemented new asynchronous methods in the Label Studio client for fetching, creating, updating, and deleting task annotations.
- Introduced bidirectional synchronization for annotations between DataMate and Label Studio, allowing for flexible data management.
- Updated sync service to handle annotation conflicts based on timestamps, ensuring data integrity during synchronization.
- Enhanced dataset file response model to include tags and their update timestamps.
- Modified database initialization script to create a new column for `tags_updated_at` in the dataset files table.
- Updated requirements to ensure compatibility with the latest dependencies.
2025-11-07 15:03:07 +08:00
Jason Wang
ba0c69086a feat: data annotation page adaptation to backend API. Improve labeling project creation module.
* feat: data annotation page adaptation to the backend API.

* feat: Implement labeling configuration editor and enhance annotation task creation form
2025-10-31 15:56:29 +08:00
Jason Wang
2f7341dc1f refactor: Reorganize datamate-python (#34)
refactor: Reorganize datamate-python (previously label-studio-adapter) into a DDD style structure.
2025-10-30 01:32:59 +08:00