Commit Graph

29 Commits

Author SHA1 Message Date
afcb8783aa feat(kg): 实现 Phase 3.1 前端图谱浏览器
核心功能:
- G6 v5 力导向图,支持交互式缩放、平移、拖拽
- 5 种布局模式:force, circular, grid, radial, concentric
- 双击展开节点邻居到图中(增量探索)
- 全文搜索,类型过滤,结果高亮(变暗/高亮状态)
- 节点详情抽屉:实体属性、别名、置信度、关系列表(可导航)
- 关系详情抽屉:类型、源/目标、权重、置信度、属性
- 查询构建器:最短路径/全路径查询,可配置 maxDepth/maxPaths
- 基于 UUID 的图加载(输入或 URL 参数 ?graphId=...)
- 大图性能优化(200 节点阈值,超过时禁用动画)

新增文件(13 个):
- knowledge-graph.model.ts - TypeScript 接口,匹配 Java DTOs
- knowledge-graph.api.ts - API 服务,包含所有 KG REST 端点
- knowledge-graph.const.ts - 实体类型颜色、关系类型标签、中文显示名称
- graphTransform.ts - 后端数据 → G6 节点/边格式转换 + 合并工具
- graphConfig.ts - G6 v5 图配置(节点/边样式、行为、布局)
- hooks/useGraphData.ts - 数据钩子:加载子图、展开节点、搜索、合并
- hooks/useGraphLayout.ts - 布局钩子:5 种布局类型
- components/GraphCanvas.tsx - G6 v5 画布,力导向布局,缩放/平移/拖拽
- components/SearchPanel.tsx - 全文实体搜索,类型过滤
- components/NodeDetail.tsx - 实体详情抽屉
- components/RelationDetail.tsx - 关系详情抽屉
- components/QueryBuilder.tsx - 路径查询构建器
- Home/KnowledgeGraphPage.tsx - 主页面,整合所有组件

修改文件(5 个):
- package.json - 添加 @antv/g6 v5 依赖
- vite.config.ts - 添加 /knowledge-graph 代理规则
- auth/permissions.ts - 添加 knowledgeGraphRead/knowledgeGraphWrite
- pages/Layout/menu.tsx - 添加知识图谱菜单项(Network 图标)
- routes/routes.ts - 添加 /data/knowledge-graph 路由

新增文档(10 个):
- docs/knowledge-graph/ - 完整的知识图谱设计文档

Bug 修复(Codex 审查后修复):
- P1: 详情抽屉状态与选中状态不一致(显示旧数据)
- P1: 查询构建器未实现(最短路径/多路径查询)
- P2: 实体类型映射 Organization → Org(匹配后端)
- P2: getSubgraph depth 参数无效(改用正确端点)
- P2: AllPathsVO 字段名不一致(totalPaths → pathCount)
- P2: 搜索取消逻辑无效(传递 AbortController.signal)
- P2: 大图性能优化(动画降级)
- P3: 移除未使用的类型导入

构建验证:
- tsc --noEmit  clean
- eslint  0 errors/warnings
- vite build  successful
2026-02-20 19:13:46 +08:00
71f8f7d1c3 feat: 实现任务拆分和分配功能
## 功能概述
实现完整的任务拆分、分配和进度跟踪功能,支持将任务拆分为子任务并分配给不同用户。

## Phase 1: 数据库层
- 新增 t_task_meta 表(任务元数据协调表)
- 新增 t_task_assignment_log 表(分配日志表)
- 新增 3 个权限条目(read/write/assign)
- 新增 SQLAlchemy ORM 模型

## Phase 2: 后端 API (Java)
- 新增 task-coordination-service 模块(32 个文件)
- 实现 11 个 API 端点:
  - 任务查询(列表、子任务、我的任务)
  - 任务拆分(支持 4 种策略)
  - 任务分配(单个、批量、重新分配、撤回)
  - 进度管理(查询、更新、聚合)
  - 分配日志
- 集成权限控制和路由规则

## Phase 3: 前端 UI (React + TypeScript)
- 新增 10 个文件(模型、API、组件、页面)
- 实现 5 个核心组件:
  - SplitTaskDialog - 任务拆分对话框
  - AssignTaskDialog - 任务分配对话框
  - BatchAssignDialog - 批量分配对话框
  - TaskProgressPanel - 进度面板
  - AssignmentLogDrawer - 分配记录
- 实现 2 个页面:
  - TaskCoordination - 任务管理主页
  - MyTasks - 我的任务页面
- 集成侧边栏菜单和路由

## 问题修复
- 修复 getMyTasks 分页参数缺失
- 修复子任务 assignee 信息缺失(批量查询优化)
- 修复 proportion 精度计算(余量分配)

## 技术亮点
- 零侵入设计:通过独立协调表实现,不修改现有模块
- 批量查询优化:避免 N+1 查询问题
- 4 种拆分策略:按比例/数量/文件/手动
- 进度自动聚合:子任务更新自动聚合到父任务
- 权限细粒度控制:read/write/assign 三级权限

## 验证
- Maven 编译: 零错误
- TypeScript 编译: 零错误
- Vite 生产构建: 成功
2026-02-09 00:42:34 +08:00
78624915b7 feat(annotation): 添加标注任务算子编排前端页面和测试算子
## 功能概述
为标注任务通用算子编排功能添加完整的前端界面,包括任务创建、列表管理、详情查看等功能,并提供测试算子用于功能验证。

## 改动内容

### 前端功能

#### 1. 算子编排页面
- 新增两步创建流程:
  - 第一步:基本信息(数据集选择、任务名称等)
  - 第二步:算子编排(选择算子、配置参数、预览 pipeline)
- 核心文件:
  - frontend/src/pages/DataAnnotation/OperatorCreate/CreateTask.tsx
  - frontend/src/pages/DataAnnotation/OperatorCreate/hooks/useOperatorOperations.ts
  - frontend/src/pages/DataAnnotation/OperatorCreate/hooks/useDragOperators.ts
  - frontend/src/pages/DataAnnotation/OperatorCreate/hooks/useCreateStepTwo.tsx

#### 2. UI 组件
- 算子库(OperatorLibrary):显示可用算子,支持分类筛选
- 编排区(OperatorOrchestration):拖拽排序算子
- 参数面板(OperatorConfig):配置算子参数
- Pipeline 预览(PipelinePreview):预览算子链
- 核心文件:frontend/src/pages/DataAnnotation/OperatorCreate/components/

#### 3. 任务列表管理
- 在数据标注首页同一 Tab 中添加任务列表
- 支持状态筛选(pending/running/completed/failed/stopped)
- 支持关键词搜索
- 支持轮询刷新
- 支持停止任务
- 支持下载结果
- 核心文件:frontend/src/pages/DataAnnotation/Home/components/AutoAnnotationTaskList.tsx

#### 4. 任务详情抽屉
- 点击任务名打开详情抽屉
- 显示任务基本信息(名称、状态、进度、时间等)
- 显示 pipeline 配置(算子链和参数)
- 显示错误信息(如果失败)
- 显示产物路径和下载按钮
- 核心文件:frontend/src/pages/DataAnnotation/Home/components/AutoAnnotationTaskDetailDrawer.tsx

#### 5. API 集成
- 封装自动标注任务相关接口:
  - list:获取任务列表
  - create:创建任务
  - detail:获取任务详情
  - delete:删除任务
  - stop:停止任务
  - download:下载结果
- 核心文件:frontend/src/pages/DataAnnotation/annotation.api.ts

#### 6. 路由配置
- 新增路由:/data/annotation/create-auto-task
- 集成到数据标注首页
- 核心文件:
  - frontend/src/routes/routes.ts
  - frontend/src/pages/DataAnnotation/Home/DataAnnotation.tsx

#### 7. 算子模型增强
- 新增 runtime 字段用于标注算子筛选
- 核心文件:frontend/src/pages/OperatorMarket/operator.model.ts

### 后端功能

#### 1. 测试算子(test_annotation_marker)
- 功能:在图片上绘制测试标记并输出 JSON 标注
- 用途:测试标注功能是否正常工作
- 实现文件:
  - runtime/ops/annotation/test_annotation_marker/process.py
  - runtime/ops/annotation/test_annotation_marker/metadata.yml
  - runtime/ops/annotation/test_annotation_marker/__init__.py

#### 2. 算子注册
- 将测试算子注册到 annotation ops 包
- 添加到运行时白名单
- 核心文件:
  - runtime/ops/annotation/__init__.py
  - runtime/python-executor/datamate/auto_annotation_worker.py

#### 3. 数据库初始化
- 添加测试算子到数据库
- 添加算子分类关联
- 核心文件:scripts/db/data-operator-init.sql

### 问题修复

#### 1. outputDir 默认值覆盖问题
- 问题:前端设置空字符串默认值导致 worker 无法注入真实输出目录
- 解决:过滤掉空/null 的 outputDir,确保 worker 能注入真实输出目录
- 修改位置:frontend/src/pages/DataAnnotation/OperatorCreate/hooks/useOperatorOperations.ts

#### 2. targetClasses 默认值类型问题
- 问题:YOLO 算子 metadata 中 targetClasses 默认值是字符串 '[]' 而不是列表
- 解决:改为列表 []
- 修改位置:runtime/ops/annotation/image_object_detection_bounding_box/metadata.yml

## 关键特性

### 用户体验
- 统一的算子编排界面(与数据清洗保持一致)
- 直观的拖拽操作
- 实时的 pipeline 预览
- 完整的任务管理功能

### 功能完整性
- 任务创建:两步流程,清晰明了
- 任务管理:列表展示、状态筛选、搜索
- 任务操作:停止、下载
- 任务详情:完整的信息展示

### 可测试性
- 提供测试算子用于功能验证
- 支持快速测试标注流程

## 验证结果

- ESLint 检查: 通过
- 前端构建: 通过(10.91s)
- 功能测试: 所有功能正常

## 部署说明

1. 执行数据库初始化脚本(如果是新环境)
2. 重启前端服务
3. 重启后端服务(如果修改了 worker 白名单)

## 使用说明

1. 进入数据标注页面
2. 点击创建自动标注任务
3. 选择数据集和文件
4. 从算子库拖拽算子到编排区
5. 配置算子参数
6. 预览 pipeline
7. 提交任务
8. 在任务列表中查看进度
9. 点击任务名查看详情
10. 下载标注结果

## 相关文件

- 前端页面:frontend/src/pages/DataAnnotation/OperatorCreate/
- 任务管理:frontend/src/pages/DataAnnotation/Home/components/
- API 集成:frontend/src/pages/DataAnnotation/annotation.api.ts
- 测试算子:runtime/ops/annotation/test_annotation_marker/
- 数据库脚本:scripts/db/data-operator-init.sql
2026-02-08 08:17:35 +08:00
056cee11cc feat(auth): 完善API网关JWT认证和权限控制功能
- 实现网关侧JWT工具类和权限规则匹配器
- 集成JWT认证流程,支持Bearer Token验证
- 添加基于路径和HTTP方法的权限控制机制
- 配置白名单路由规则,优化认证性能
- 更新前端受保护路由组件,实现权限验证
- 添加403禁止访问页面和权限检查逻辑
- 重构登录页面,集成实际认证API调用
- 实现用户信息获取和权限加载功能
- 优化全局异常处理器中的认证错误状态码
- 集成FastJSON2和JJWT依赖库支持
2026-02-06 13:21:20 +08:00
7043a26ab3 feat(auth): 添加登录功能和路由保护
- 在侧边栏添加退出登录按钮并实现登出逻辑
- 添加 ProtectedRoute 组件用于路由权限控制
- 创建 LoginPage 组件实现登录界面和逻辑
- 集成本地登录验证到 authSlice 状态管理
- 配置路由表添加登录页面和保护路由
- 实现自动跳转到登录页面的重定向逻辑
2026-02-01 14:11:44 +08:00
790385bd80 feat(knowledge-management): 添加知识管理搜索功能和统计接口
- 新增知识条目搜索查询和响应DTO
- 实现知识管理统计功能,包括总数、文件数和总大小
- 添加数据库查询方法支持文件搜索和统计计算
- 创建知识条目搜索控制器提供REST API
- 在前端添加知识管理搜索页面和相关组件
- 更新前端路由配置添加搜索页面入口
- 移除RAG索引服务中的重复统计功能
- 优化前端页面统计数据显示和刷新逻辑
2026-01-31 09:30:37 +08:00
76f70a6847 feat(knowledge-base): 添加知识库文件全库检索功能
- 新增相对路径字段替代原有的metadata存储方式
- 实现跨知识库文件检索接口searchFiles
- 添加前端全库检索页面和相关API调用
- 优化文件路径处理和数据库索引配置
- 统一请求参数类型定义为RequestPayload和RequestParams
- 简化RagFile模型中的元数据结构设计
2026-01-30 22:24:12 +08:00
da6c7860cb refactor(routes): 重构路由配置和菜单结构
- 更新了路由定义文件以支持新的导航需求
- 调整了菜单组件的数据结构和渲染逻辑
- 统一了路由路径和菜单项的命名规范
- 优化了路由懒加载配置提升性能
- 修改了菜单权限验证机制
- 清理了废弃的路由配置项
2026-01-21 11:47:54 +08:00
8d6b4a4265 refactor(home): 移除外部链接跳转功能并优化路由导航
- 移除百度链接跳转功能,改为内部路由导航到内容生成页面
- 删除无用的 isChecking 状态和 handleChatClick 处理函数
- 移除菜单项中的 externalLink 属性配置
- 简化侧边栏点击事件处理逻辑,移除外部链接检查代码
- 添加内容生成页面的路由配置
2026-01-20 10:34:13 +08:00
d568de7d45 feat(layout): 隐藏数据归集和数据清洗菜单项并添加首页路由
- 注释掉数据归集菜单项
- 注释掉数据清洗菜单项
- 注释掉算子市场菜单项
- 注释掉侧边栏中的任务中心弹窗组件
- 添加首页路由配置
- 将根路径重定向改为首页组件渲染
2026-01-19 18:23:38 +08:00
87c2ef8a58 首页 2026-01-16 01:50:36 +08:00
d5b75fee0d LSF 2026-01-07 00:00:16 +08:00
Kecheng Sha
0df7a872e4 Revert "feat: fix the problem in the Operator Market frontend pages" 2025-12-29 12:00:37 +08:00
root
844add27ea feat: fix the problem in the Operator Market frontend pages 2025-12-29 11:38:47 +08:00
Dallas98
977d16339b feat(synthesis): add chunk-level synthesis data detail page & refine APIs/routing (#130)
* feat: implement synthesis data detail view with chunk selection and data display
2025-12-04 14:27:01 +08:00
Dallas98
7012a9ad98 feat: enhance backend deployment, frontend file selection and synthesis task management (#129)
* feat: Implement data synthesis task management with database models and API endpoints

* feat: Update Python version requirements and refine dependency constraints in configuration

* fix: Correctly extract file values from selectedFilesMap in AddDataDialog

* feat: Refactor synthesis task routes and enhance file task management in the API

* feat: Enhance SynthesisTaskTab with tooltip actions and add chunk data retrieval in API
2025-12-04 09:57:13 +08:00
hefanli
1d19cd3a62 feature: add data-evaluation
* feature: add evaluation task management function

* feature: add evaluation task detail page

* fix: delete duplicate definition for table t_model_config

* refactor: rename package synthesis to ratio

* refactor: add eval file table and  refactor related code

* fix: calling large models in parallel during evaluation
2025-12-04 09:23:54 +08:00
chenghh-9609
2660845b74 update data ratio detail page (#78)
* feat: Update site name to DataMate and refine text for AI data processing

* feat: Refactor settings page and implement model access functionality

- Created a new ModelAccess component for managing model configurations.
- Removed the old Settings component and replaced it with a new SettingsPage component that integrates ModelAccess, SystemConfig, and WebhookConfig.
- Added SystemConfig component for managing system settings.
- Implemented WebhookConfig component for managing webhook configurations.
- Updated API functions for model management in settings.apis.ts.
- Adjusted routing to point to the new SettingsPage component.

* feat: Implement Data Collection Page with Task Management and Execution Log

- Created DataCollectionPage component to manage data collection tasks.
- Added TaskManagement and ExecutionLog components for task handling and logging.
- Integrated task operations including start, stop, edit, and delete functionalities.
- Implemented filtering and searching capabilities in task management.
- Introduced SimpleCronScheduler for scheduling tasks with cron expressions.
- Updated CreateTask component to utilize new scheduling and template features.
- Enhanced BasicInformation component to conditionally render fields based on visibility settings.
- Refactored ImportConfiguration component to remove NAS import section.

* feat: Update task creation API endpoint and enhance task creation form with new fields and validation

* Refactor file upload and operator management components

- Removed unnecessary console logs from file download and export functions.
- Added size property to TaskItem interface for better task management.
- Simplified TaskUpload component by utilizing useFileSliceUpload hook for file upload logic.
- Enhanced OperatorPluginCreate component to handle file uploads and parsing more efficiently.
- Updated ConfigureStep component to use Ant Design Form for better data handling and validation.
- Improved PreviewStep component to navigate back to the operator market.
- Added support for additional file types in UploadStep component.
- Implemented delete operator functionality in OperatorMarketPage with confirmation prompts.
- Cleaned up unused API functions in operator.api.ts to streamline the codebase.
- Fixed number formatting utility to handle zero values correctly.

* Refactor Knowledge Generation to Knowledge Base

- Created new API service for Knowledge Base operations including querying, creating, updating, and deleting knowledge bases and files.
- Added constants for Knowledge Base status and type mappings.
- Defined models for Knowledge Base and related files.
- Removed obsolete Knowledge Base creation and home components, replacing them with new implementations under the Knowledge Base structure.
- Updated routing to reflect the new Knowledge Base paths.
- Adjusted menu items to align with the new Knowledge Base terminology.
- Modified ModelAccess interface to include modelName and type properties.

* feat: Implement Knowledge Base Page with CRUD operations and data management

- Added KnowledgeBasePage component for displaying and managing knowledge bases.
- Integrated search and filter functionalities with SearchControls component.
- Implemented CreateKnowledgeBase component for creating and editing knowledge bases.
- Enhanced AddDataDialog for file uploads and dataset selections.
- Introduced TableTransfer component for managing data transfers between tables.
- Updated API functions for knowledge base operations, including file management.
- Refactored knowledge base model to include file status and metadata.
- Adjusted routing to point to the new KnowledgeBasePage.

* feat: enhance OperatorPluginCreate and ConfigureStep for better upload handling and UI updates

* refactor: remove unused components and clean up API logging in KnowledgeBase

* feat: update icons in various components and improve styling for better UI consistency

* fix: adjust upload step handling and improve error display in configuration step

* feat: Add RatioTransfer component for dataset selection and configuration

- Implemented RatioTransfer component to manage dataset selection and ratio configuration.
- Integrated dataset fetching with search and filter capabilities.
- Added RatioConfig component for displaying and updating selected datasets' configurations.
- Enhanced SelectDataset component with improved UI and functionality for dataset selection.
- Updated RatioTasksPage to utilize new ratio task status mapping and improved error handling for task deletion.
- Refactored ratio model and constants for better type safety and clarity.
- Changed Vite configuration to use local backend service for development.

* feat: Add .editorconfig and enhance SystemConfig with table for settings display

* feat: Enhance parameter configuration for range inputs and update default values

* feat: Update site name to DataMate and refine text for AI data processing

* Refactor file upload and operator management components

- Removed unnecessary console logs from file download and export functions.
- Added size property to TaskItem interface for better task management.
- Simplified TaskUpload component by utilizing useFileSliceUpload hook for file upload logic.
- Enhanced OperatorPluginCreate component to handle file uploads and parsing more efficiently.
- Updated ConfigureStep component to use Ant Design Form for better data handling and validation.
- Improved PreviewStep component to navigate back to the operator market.
- Added support for additional file types in UploadStep component.
- Implemented delete operator functionality in OperatorMarketPage with confirmation prompts.
- Cleaned up unused API functions in operator.api.ts to streamline the codebase.
- Fixed number formatting utility to handle zero values correctly.

* Refactor Knowledge Generation to Knowledge Base

- Created new API service for Knowledge Base operations including querying, creating, updating, and deleting knowledge bases and files.
- Added constants for Knowledge Base status and type mappings.
- Defined models for Knowledge Base and related files.
- Removed obsolete Knowledge Base creation and home components, replacing them with new implementations under the Knowledge Base structure.
- Updated routing to reflect the new Knowledge Base paths.
- Adjusted menu items to align with the new Knowledge Base terminology.
- Modified ModelAccess interface to include modelName and type properties.

* feat: Implement Knowledge Base Page with CRUD operations and data management

- Added KnowledgeBasePage component for displaying and managing knowledge bases.
- Integrated search and filter functionalities with SearchControls component.
- Implemented CreateKnowledgeBase component for creating and editing knowledge bases.
- Enhanced AddDataDialog for file uploads and dataset selections.
- Introduced TableTransfer component for managing data transfers between tables.
- Updated API functions for knowledge base operations, including file management.
- Refactored knowledge base model to include file status and metadata.
- Adjusted routing to point to the new KnowledgeBasePage.

* feat: enhance OperatorPluginCreate and ConfigureStep for better upload handling and UI updates

* feat: update icons in various components and improve styling for better UI consistency

* fix: adjust upload step handling and improve error display in configuration step

* feat: Update site name to DataMate and refine text for AI data processing

* Refactor file upload and operator management components

- Removed unnecessary console logs from file download and export functions.
- Added size property to TaskItem interface for better task management.
- Simplified TaskUpload component by utilizing useFileSliceUpload hook for file upload logic.
- Enhanced OperatorPluginCreate component to handle file uploads and parsing more efficiently.
- Updated ConfigureStep component to use Ant Design Form for better data handling and validation.
- Improved PreviewStep component to navigate back to the operator market.
- Added support for additional file types in UploadStep component.
- Implemented delete operator functionality in OperatorMarketPage with confirmation prompts.
- Cleaned up unused API functions in operator.api.ts to streamline the codebase.
- Fixed number formatting utility to handle zero values correctly.

* Refactor Knowledge Generation to Knowledge Base

- Created new API service for Knowledge Base operations including querying, creating, updating, and deleting knowledge bases and files.
- Added constants for Knowledge Base status and type mappings.
- Defined models for Knowledge Base and related files.
- Removed obsolete Knowledge Base creation and home components, replacing them with new implementations under the Knowledge Base structure.
- Updated routing to reflect the new Knowledge Base paths.
- Adjusted menu items to align with the new Knowledge Base terminology.
- Modified ModelAccess interface to include modelName and type properties.

* feat: Implement Knowledge Base Page with CRUD operations and data management

- Added KnowledgeBasePage component for displaying and managing knowledge bases.
- Integrated search and filter functionalities with SearchControls component.
- Implemented CreateKnowledgeBase component for creating and editing knowledge bases.
- Enhanced AddDataDialog for file uploads and dataset selections.
- Introduced TableTransfer component for managing data transfers between tables.
- Updated API functions for knowledge base operations, including file management.
- Refactored knowledge base model to include file status and metadata.
- Adjusted routing to point to the new KnowledgeBasePage.

* feat: enhance OperatorPluginCreate and ConfigureStep for better upload handling and UI updates

* feat: update icons in various components and improve styling for better UI consistency

* fix: adjust upload step handling and improve error display in configuration step

* feat: add settings drawer and integrate SettingsPage component

* feat: add ratio task management features including detail view and API integration

* feat: enable destruction of hidden settings drawer to free up resources

* feat: add data metrics and ratio display components with charts for enhanced data visualization
2025-11-13 10:41:59 +08:00
hhhhsc701
6bbde0ec56 feature: 清洗任务详情页 (#73)
* feature: 清洗任务详情

* fix: 取消构建镜像,改为直接拉取

* fix: 增加清洗任务详情页

* fix: 增加清洗任务详情页

* fix: 算子列表可点击

* fix: 模板详情和更新
2025-11-12 18:00:19 +08:00
chenghh-9609
442e561817 change settings page and add ratio task detail page (#77)
* feat: Update site name to DataMate and refine text for AI data processing

* feat: Refactor settings page and implement model access functionality

- Created a new ModelAccess component for managing model configurations.
- Removed the old Settings component and replaced it with a new SettingsPage component that integrates ModelAccess, SystemConfig, and WebhookConfig.
- Added SystemConfig component for managing system settings.
- Implemented WebhookConfig component for managing webhook configurations.
- Updated API functions for model management in settings.apis.ts.
- Adjusted routing to point to the new SettingsPage component.

* feat: Implement Data Collection Page with Task Management and Execution Log

- Created DataCollectionPage component to manage data collection tasks.
- Added TaskManagement and ExecutionLog components for task handling and logging.
- Integrated task operations including start, stop, edit, and delete functionalities.
- Implemented filtering and searching capabilities in task management.
- Introduced SimpleCronScheduler for scheduling tasks with cron expressions.
- Updated CreateTask component to utilize new scheduling and template features.
- Enhanced BasicInformation component to conditionally render fields based on visibility settings.
- Refactored ImportConfiguration component to remove NAS import section.

* feat: Update task creation API endpoint and enhance task creation form with new fields and validation

* Refactor file upload and operator management components

- Removed unnecessary console logs from file download and export functions.
- Added size property to TaskItem interface for better task management.
- Simplified TaskUpload component by utilizing useFileSliceUpload hook for file upload logic.
- Enhanced OperatorPluginCreate component to handle file uploads and parsing more efficiently.
- Updated ConfigureStep component to use Ant Design Form for better data handling and validation.
- Improved PreviewStep component to navigate back to the operator market.
- Added support for additional file types in UploadStep component.
- Implemented delete operator functionality in OperatorMarketPage with confirmation prompts.
- Cleaned up unused API functions in operator.api.ts to streamline the codebase.
- Fixed number formatting utility to handle zero values correctly.

* Refactor Knowledge Generation to Knowledge Base

- Created new API service for Knowledge Base operations including querying, creating, updating, and deleting knowledge bases and files.
- Added constants for Knowledge Base status and type mappings.
- Defined models for Knowledge Base and related files.
- Removed obsolete Knowledge Base creation and home components, replacing them with new implementations under the Knowledge Base structure.
- Updated routing to reflect the new Knowledge Base paths.
- Adjusted menu items to align with the new Knowledge Base terminology.
- Modified ModelAccess interface to include modelName and type properties.

* feat: Implement Knowledge Base Page with CRUD operations and data management

- Added KnowledgeBasePage component for displaying and managing knowledge bases.
- Integrated search and filter functionalities with SearchControls component.
- Implemented CreateKnowledgeBase component for creating and editing knowledge bases.
- Enhanced AddDataDialog for file uploads and dataset selections.
- Introduced TableTransfer component for managing data transfers between tables.
- Updated API functions for knowledge base operations, including file management.
- Refactored knowledge base model to include file status and metadata.
- Adjusted routing to point to the new KnowledgeBasePage.

* feat: enhance OperatorPluginCreate and ConfigureStep for better upload handling and UI updates

* refactor: remove unused components and clean up API logging in KnowledgeBase

* feat: update icons in various components and improve styling for better UI consistency

* fix: adjust upload step handling and improve error display in configuration step

* feat: Add RatioTransfer component for dataset selection and configuration

- Implemented RatioTransfer component to manage dataset selection and ratio configuration.
- Integrated dataset fetching with search and filter capabilities.
- Added RatioConfig component for displaying and updating selected datasets' configurations.
- Enhanced SelectDataset component with improved UI and functionality for dataset selection.
- Updated RatioTasksPage to utilize new ratio task status mapping and improved error handling for task deletion.
- Refactored ratio model and constants for better type safety and clarity.
- Changed Vite configuration to use local backend service for development.

* feat: Add .editorconfig and enhance SystemConfig with table for settings display

* feat: Enhance parameter configuration for range inputs and update default values

* feat: Update site name to DataMate and refine text for AI data processing

* Refactor file upload and operator management components

- Removed unnecessary console logs from file download and export functions.
- Added size property to TaskItem interface for better task management.
- Simplified TaskUpload component by utilizing useFileSliceUpload hook for file upload logic.
- Enhanced OperatorPluginCreate component to handle file uploads and parsing more efficiently.
- Updated ConfigureStep component to use Ant Design Form for better data handling and validation.
- Improved PreviewStep component to navigate back to the operator market.
- Added support for additional file types in UploadStep component.
- Implemented delete operator functionality in OperatorMarketPage with confirmation prompts.
- Cleaned up unused API functions in operator.api.ts to streamline the codebase.
- Fixed number formatting utility to handle zero values correctly.

* Refactor Knowledge Generation to Knowledge Base

- Created new API service for Knowledge Base operations including querying, creating, updating, and deleting knowledge bases and files.
- Added constants for Knowledge Base status and type mappings.
- Defined models for Knowledge Base and related files.
- Removed obsolete Knowledge Base creation and home components, replacing them with new implementations under the Knowledge Base structure.
- Updated routing to reflect the new Knowledge Base paths.
- Adjusted menu items to align with the new Knowledge Base terminology.
- Modified ModelAccess interface to include modelName and type properties.

* feat: Implement Knowledge Base Page with CRUD operations and data management

- Added KnowledgeBasePage component for displaying and managing knowledge bases.
- Integrated search and filter functionalities with SearchControls component.
- Implemented CreateKnowledgeBase component for creating and editing knowledge bases.
- Enhanced AddDataDialog for file uploads and dataset selections.
- Introduced TableTransfer component for managing data transfers between tables.
- Updated API functions for knowledge base operations, including file management.
- Refactored knowledge base model to include file status and metadata.
- Adjusted routing to point to the new KnowledgeBasePage.

* feat: enhance OperatorPluginCreate and ConfigureStep for better upload handling and UI updates

* feat: update icons in various components and improve styling for better UI consistency

* fix: adjust upload step handling and improve error display in configuration step

* feat: Update site name to DataMate and refine text for AI data processing

* Refactor file upload and operator management components

- Removed unnecessary console logs from file download and export functions.
- Added size property to TaskItem interface for better task management.
- Simplified TaskUpload component by utilizing useFileSliceUpload hook for file upload logic.
- Enhanced OperatorPluginCreate component to handle file uploads and parsing more efficiently.
- Updated ConfigureStep component to use Ant Design Form for better data handling and validation.
- Improved PreviewStep component to navigate back to the operator market.
- Added support for additional file types in UploadStep component.
- Implemented delete operator functionality in OperatorMarketPage with confirmation prompts.
- Cleaned up unused API functions in operator.api.ts to streamline the codebase.
- Fixed number formatting utility to handle zero values correctly.

* Refactor Knowledge Generation to Knowledge Base

- Created new API service for Knowledge Base operations including querying, creating, updating, and deleting knowledge bases and files.
- Added constants for Knowledge Base status and type mappings.
- Defined models for Knowledge Base and related files.
- Removed obsolete Knowledge Base creation and home components, replacing them with new implementations under the Knowledge Base structure.
- Updated routing to reflect the new Knowledge Base paths.
- Adjusted menu items to align with the new Knowledge Base terminology.
- Modified ModelAccess interface to include modelName and type properties.

* feat: Implement Knowledge Base Page with CRUD operations and data management

- Added KnowledgeBasePage component for displaying and managing knowledge bases.
- Integrated search and filter functionalities with SearchControls component.
- Implemented CreateKnowledgeBase component for creating and editing knowledge bases.
- Enhanced AddDataDialog for file uploads and dataset selections.
- Introduced TableTransfer component for managing data transfers between tables.
- Updated API functions for knowledge base operations, including file management.
- Refactored knowledge base model to include file status and metadata.
- Adjusted routing to point to the new KnowledgeBasePage.

* feat: enhance OperatorPluginCreate and ConfigureStep for better upload handling and UI updates

* feat: update icons in various components and improve styling for better UI consistency

* fix: adjust upload step handling and improve error display in configuration step

* feat: add settings drawer and integrate SettingsPage component

* feat: add ratio task management features including detail view and API integration
2025-11-12 17:23:24 +08:00
Jason Wang
c5ccc56cca feat: Add labeling template (#72)
* feat: Enhance annotation module with template management and validation

- Added DatasetMappingCreateRequest and DatasetMappingUpdateRequest schemas to handle dataset mapping requests with camelCase and snake_case support.
- Introduced Annotation Template schemas including CreateAnnotationTemplateRequest, UpdateAnnotationTemplateRequest, and AnnotationTemplateResponse for managing annotation templates.
- Implemented AnnotationTemplateService for creating, updating, retrieving, and deleting annotation templates, including validation of configurations and XML generation.
- Added utility class LabelStudioConfigValidator for validating Label Studio configurations and XML formats.
- Updated database schema for annotation templates and labeling projects to include new fields and constraints.
- Seeded initial annotation templates for various use cases including image classification, object detection, and text classification.

* feat: Enhance TemplateForm with improved validation and dynamic field rendering; update LabelStudio config validation for camelCase support

* feat: Update docker-compose.yml to mark datamate dataset volume and network as external
2025-11-11 09:14:14 +08:00
hefanli
08bd4eca5c feature:增加数据配比功能 (#52)
* refactor: 修改调整数据归集实现,删除无用代码,优化代码结构

* feature: 每天凌晨00:00扫描所有数据集,检查数据集是否超过了预设的保留天数,超出保留天数的数据集调用删除接口进行删除

* fix: 修改删除数据集文件的逻辑,上传到数据集中的文件会同时删除数据库中的记录和文件系统中的文件,归集过来的文件仅删除数据库中的记录

* fix: 增加参数校验和接口定义,删除不使用的接口

* fix: 数据集统计数据默认为0

* feature: 数据集状态增加流转,创建时为草稿状态,上传文件或者归集文件后修改为活动状态

* refactor: 修改分页查询归集任务的代码

* fix: 更新后重新执行;归集任务执行增加事务控制

* feature: 创建归集任务时能够同步创建数据集,更新归集任务时能更新到指定数据集

* fix: 创建归集任务不需要创建数据集时不应该报错

* fix: 修复删除文件时数据集的统计数据不变动

* feature: 查询数据集详情时能够获取到文件标签分布

* fix: tags为空时不进行分析

* fix: 状态修改为ACTIVE

* fix: 修改解析tag的方法

* feature: 实现创建、分页查询、删除配比任务

* feature: 实现创建、分页查询、删除配比任务的前端交互

* fix: 修复进度计算异常导致的页面报错
2025-11-03 10:17:39 +08:00
chenghh-9609
c6958d1511 knowledge base pages (#43)
* feat: Update site name to DataMate and refine text for AI data processing

* feat: Refactor settings page and implement model access functionality

- Created a new ModelAccess component for managing model configurations.
- Removed the old Settings component and replaced it with a new SettingsPage component that integrates ModelAccess, SystemConfig, and WebhookConfig.
- Added SystemConfig component for managing system settings.
- Implemented WebhookConfig component for managing webhook configurations.
- Updated API functions for model management in settings.apis.ts.
- Adjusted routing to point to the new SettingsPage component.

* feat: Implement Data Collection Page with Task Management and Execution Log

- Created DataCollectionPage component to manage data collection tasks.
- Added TaskManagement and ExecutionLog components for task handling and logging.
- Integrated task operations including start, stop, edit, and delete functionalities.
- Implemented filtering and searching capabilities in task management.
- Introduced SimpleCronScheduler for scheduling tasks with cron expressions.
- Updated CreateTask component to utilize new scheduling and template features.
- Enhanced BasicInformation component to conditionally render fields based on visibility settings.
- Refactored ImportConfiguration component to remove NAS import section.

* feat: Update task creation API endpoint and enhance task creation form with new fields and validation

* Refactor file upload and operator management components

- Removed unnecessary console logs from file download and export functions.
- Added size property to TaskItem interface for better task management.
- Simplified TaskUpload component by utilizing useFileSliceUpload hook for file upload logic.
- Enhanced OperatorPluginCreate component to handle file uploads and parsing more efficiently.
- Updated ConfigureStep component to use Ant Design Form for better data handling and validation.
- Improved PreviewStep component to navigate back to the operator market.
- Added support for additional file types in UploadStep component.
- Implemented delete operator functionality in OperatorMarketPage with confirmation prompts.
- Cleaned up unused API functions in operator.api.ts to streamline the codebase.
- Fixed number formatting utility to handle zero values correctly.

* Refactor Knowledge Generation to Knowledge Base

- Created new API service for Knowledge Base operations including querying, creating, updating, and deleting knowledge bases and files.
- Added constants for Knowledge Base status and type mappings.
- Defined models for Knowledge Base and related files.
- Removed obsolete Knowledge Base creation and home components, replacing them with new implementations under the Knowledge Base structure.
- Updated routing to reflect the new Knowledge Base paths.
- Adjusted menu items to align with the new Knowledge Base terminology.
- Modified ModelAccess interface to include modelName and type properties.

* feat: Implement Knowledge Base Page with CRUD operations and data management

- Added KnowledgeBasePage component for displaying and managing knowledge bases.
- Integrated search and filter functionalities with SearchControls component.
- Implemented CreateKnowledgeBase component for creating and editing knowledge bases.
- Enhanced AddDataDialog for file uploads and dataset selections.
- Introduced TableTransfer component for managing data transfers between tables.
- Updated API functions for knowledge base operations, including file management.
- Refactored knowledge base model to include file status and metadata.
- Adjusted routing to point to the new KnowledgeBasePage.
2025-10-31 10:03:42 +08:00
chenghh-9609
d89811f238 add knowledgebase page (#39)
* feat: Update site name to DataMate and refine text for AI data processing

* feat: Refactor settings page and implement model access functionality

- Created a new ModelAccess component for managing model configurations.
- Removed the old Settings component and replaced it with a new SettingsPage component that integrates ModelAccess, SystemConfig, and WebhookConfig.
- Added SystemConfig component for managing system settings.
- Implemented WebhookConfig component for managing webhook configurations.
- Updated API functions for model management in settings.apis.ts.
- Adjusted routing to point to the new SettingsPage component.

* feat: Implement Data Collection Page with Task Management and Execution Log

- Created DataCollectionPage component to manage data collection tasks.
- Added TaskManagement and ExecutionLog components for task handling and logging.
- Integrated task operations including start, stop, edit, and delete functionalities.
- Implemented filtering and searching capabilities in task management.
- Introduced SimpleCronScheduler for scheduling tasks with cron expressions.
- Updated CreateTask component to utilize new scheduling and template features.
- Enhanced BasicInformation component to conditionally render fields based on visibility settings.
- Refactored ImportConfiguration component to remove NAS import section.

* feat: Update task creation API endpoint and enhance task creation form with new fields and validation

* Refactor file upload and operator management components

- Removed unnecessary console logs from file download and export functions.
- Added size property to TaskItem interface for better task management.
- Simplified TaskUpload component by utilizing useFileSliceUpload hook for file upload logic.
- Enhanced OperatorPluginCreate component to handle file uploads and parsing more efficiently.
- Updated ConfigureStep component to use Ant Design Form for better data handling and validation.
- Improved PreviewStep component to navigate back to the operator market.
- Added support for additional file types in UploadStep component.
- Implemented delete operator functionality in OperatorMarketPage with confirmation prompts.
- Cleaned up unused API functions in operator.api.ts to streamline the codebase.
- Fixed number formatting utility to handle zero values correctly.

* Refactor Knowledge Generation to Knowledge Base

- Created new API service for Knowledge Base operations including querying, creating, updating, and deleting knowledge bases and files.
- Added constants for Knowledge Base status and type mappings.
- Defined models for Knowledge Base and related files.
- Removed obsolete Knowledge Base creation and home components, replacing them with new implementations under the Knowledge Base structure.
- Updated routing to reflect the new Knowledge Base paths.
- Adjusted menu items to align with the new Knowledge Base terminology.
- Modified ModelAccess interface to include modelName and type properties.
2025-10-30 21:33:33 +08:00
hhhhsc701
ba6caabbc6 前端增加对deer-flow服务的检测 (#42)
前端增加对deer-flow服务的检测 (#42)
2025-10-30 20:21:25 +08:00
hhhhsc701
67eb571d8d feature: 对接deer-flow (#27)
feature: 对接deer-flow
2025-10-28 16:28:26 +08:00
chenghh-9609
acafe70d90 feat: add ModelAccess to settings page (#29)
* feat: Update site name to DataMate and refine text for AI data processing

* feat: Refactor settings page and implement model access functionality

- Created a new ModelAccess component for managing model configurations.
- Removed the old Settings component and replaced it with a new SettingsPage component that integrates ModelAccess, SystemConfig, and WebhookConfig.
- Added SystemConfig component for managing system settings.
- Implemented WebhookConfig component for managing webhook configurations.
- Updated API functions for model management in settings.apis.ts.
- Adjusted routing to point to the new SettingsPage component.
2025-10-28 16:02:18 +08:00
hhhhsc
a69b9f4921 feature: 对接deer-flow 2025-10-28 10:54:29 +08:00
Dallas98
1c97afed7d init datamate 2025-10-21 23:00:48 +08:00