Commit Graph

17 Commits

Author SHA1 Message Date
hefanli
ccfb84c034 feature: add mysql collection and starrocks collection (#222)
* fix: fix the path for backend-python imaage building

* feature: add mysql collection and starrocks collection

* feature: add mysql collection and starrocks collection

* fix: change the permission of those files which collected from nfs to 754

* fix: delete collected files, config files and log files while deleting collection task

* fix: add the collection task detail api

* fix: change the log of collecting for dataset

* fix: add collection task selecting while creating and updating dataset

* fix: set the umask value to 0022 for java process
2026-01-04 19:05:08 +08:00
o0Shark0o
cbed6fbcd7 Revert "Merge branch 'main' of https://github.com/ModelEngine-Group/DataMate"
This reverts commit a12f4c90a5, reversing
changes made to 34f08df86b.
2025-12-31 16:19:19 +08:00
hefanli
3a874fe699 fix: fix the collection for nfs (#218)
* fix: remove the datax-builder for the Backend Image

* fix: fix the collection for nfs
2025-12-31 15:56:01 +08:00
hefanli
63f4e3e447 refactor: modify data collection to python implementation (#214)
* feature: LabelStudio jumps without login

* refactor: modify data collection to python implementation

* refactor: modify data collection to python implementation

* refactor: modify data collection to python implementation

* refactor: modify data collection to python implementation

* refactor: modify data collection to python implementation

* refactor: modify data collection to python implementation

* fix: remove terrabase dependency

* feature: add the collection task executions page and the collection template page

* fix: fix the collection task creation

* fix: fix the collection task creation
2025-12-30 18:48:43 +08:00
Kecheng Sha
0df7a872e4 Revert "feat: fix the problem in the Operator Market frontend pages" 2025-12-29 12:00:37 +08:00
root
844add27ea feat: fix the problem in the Operator Market frontend pages 2025-12-29 11:38:47 +08:00
hhhhsc701
103c21945d 修复部分功能 (#138)
* feature: 版本统一

* feature: 定时同步时默认值展示异常,增加提示

* feature: 修复数据归集搜索

* feature: 优化标注模板查询

* feature: 屏蔽webhook功能
2025-12-10 14:31:05 +08:00
hhhhsc701
bb3345268e bugfix: 清洗/算子支持名称/描述搜索 (#116)
* bugfix: milvus适配etcd deploy部署

* bugfix: 可以在知识库界面跳转到创建模型
2025-11-29 18:15:43 +08:00
hefanli
a07fba23f2 feature:数据集导入数据集支持选择归集任务导入 (#92)
* feature: 实现obs归集

* feature: 增加数据集中出现同名文件时的处理方式

* feature: 前端数据集导入数据时增加可以选择归集任务导入
2025-11-19 11:05:33 +08:00
Vincent
5da992d312 feature:增加obs归集方式 (#90)
* feature:实现通过datax进行obs归集的插件

* feature:前端增加obs归集时的前缀参数
2025-11-18 09:24:07 +08:00
Vincent
2b09c7dfd1 feature:mysql数据库归集为csv文件 (#76)
* fix:配比任务需要能够跳转到目标数据集

* feature:增加配比任务详情接口

* fix:删除不存在的配比详情页面

* fix:使用正式的逻辑来展示标签

* fix:参数默认值去掉多余的-

* fix:修复配比任务相关操作

* fix:去除不需要的日志打印和import

* feature:数据归集创建时将obs、mysql归集也放出

* refactor:重构数据归集的代码

* refactor:重构数据归集的代码

* feature:增加实现mysql归集为csv文件
2025-11-12 17:05:31 +08:00
Vincent
b8d7aca8b7 refactor:重构数据归集部分代码 (#75)
* fix:配比任务需要能够跳转到目标数据集

* feature:增加配比任务详情接口

* fix:删除不存在的配比详情页面

* fix:使用正式的逻辑来展示标签

* fix:参数默认值去掉多余的-

* fix:修复配比任务相关操作

* fix:去除不需要的日志打印和import

* feature:数据归集创建时将obs、mysql归集也放出

* refactor:重构数据归集的代码

* refactor:重构数据归集的代码
2025-11-12 09:34:50 +08:00
chenghh-9609
5612c7cd91 add operator create page (#38)
* feat: Update site name to DataMate and refine text for AI data processing

* feat: Refactor settings page and implement model access functionality

- Created a new ModelAccess component for managing model configurations.
- Removed the old Settings component and replaced it with a new SettingsPage component that integrates ModelAccess, SystemConfig, and WebhookConfig.
- Added SystemConfig component for managing system settings.
- Implemented WebhookConfig component for managing webhook configurations.
- Updated API functions for model management in settings.apis.ts.
- Adjusted routing to point to the new SettingsPage component.

* feat: Implement Data Collection Page with Task Management and Execution Log

- Created DataCollectionPage component to manage data collection tasks.
- Added TaskManagement and ExecutionLog components for task handling and logging.
- Integrated task operations including start, stop, edit, and delete functionalities.
- Implemented filtering and searching capabilities in task management.
- Introduced SimpleCronScheduler for scheduling tasks with cron expressions.
- Updated CreateTask component to utilize new scheduling and template features.
- Enhanced BasicInformation component to conditionally render fields based on visibility settings.
- Refactored ImportConfiguration component to remove NAS import section.

* feat: Update task creation API endpoint and enhance task creation form with new fields and validation

* Refactor file upload and operator management components

- Removed unnecessary console logs from file download and export functions.
- Added size property to TaskItem interface for better task management.
- Simplified TaskUpload component by utilizing useFileSliceUpload hook for file upload logic.
- Enhanced OperatorPluginCreate component to handle file uploads and parsing more efficiently.
- Updated ConfigureStep component to use Ant Design Form for better data handling and validation.
- Improved PreviewStep component to navigate back to the operator market.
- Added support for additional file types in UploadStep component.
- Implemented delete operator functionality in OperatorMarketPage with confirmation prompts.
- Cleaned up unused API functions in operator.api.ts to streamline the codebase.
- Fixed number formatting utility to handle zero values correctly.
2025-10-30 16:30:01 +08:00
chenghh-9609
0614157c0b fix: data collection create task page (#33)
* feat: Update site name to DataMate and refine text for AI data processing

* feat: Refactor settings page and implement model access functionality

- Created a new ModelAccess component for managing model configurations.
- Removed the old Settings component and replaced it with a new SettingsPage component that integrates ModelAccess, SystemConfig, and WebhookConfig.
- Added SystemConfig component for managing system settings.
- Implemented WebhookConfig component for managing webhook configurations.
- Updated API functions for model management in settings.apis.ts.
- Adjusted routing to point to the new SettingsPage component.

* feat: Implement Data Collection Page with Task Management and Execution Log

- Created DataCollectionPage component to manage data collection tasks.
- Added TaskManagement and ExecutionLog components for task handling and logging.
- Integrated task operations including start, stop, edit, and delete functionalities.
- Implemented filtering and searching capabilities in task management.
- Introduced SimpleCronScheduler for scheduling tasks with cron expressions.
- Updated CreateTask component to utilize new scheduling and template features.
- Enhanced BasicInformation component to conditionally render fields based on visibility settings.
- Refactored ImportConfiguration component to remove NAS import section.

* feat: Update task creation API endpoint and enhance task creation form with new fields and validation
2025-10-28 17:41:59 +08:00
chenghh-9609
fad76e7477 data collection page (#31)
* feat: Update site name to DataMate and refine text for AI data processing

* feat: Refactor settings page and implement model access functionality

- Created a new ModelAccess component for managing model configurations.
- Removed the old Settings component and replaced it with a new SettingsPage component that integrates ModelAccess, SystemConfig, and WebhookConfig.
- Added SystemConfig component for managing system settings.
- Implemented WebhookConfig component for managing webhook configurations.
- Updated API functions for model management in settings.apis.ts.
- Adjusted routing to point to the new SettingsPage component.

* feat: Implement Data Collection Page with Task Management and Execution Log

- Created DataCollectionPage component to manage data collection tasks.
- Added TaskManagement and ExecutionLog components for task handling and logging.
- Integrated task operations including start, stop, edit, and delete functionalities.
- Implemented filtering and searching capabilities in task management.
- Introduced SimpleCronScheduler for scheduling tasks with cron expressions.
- Updated CreateTask component to utilize new scheduling and template features.
- Enhanced BasicInformation component to conditionally render fields based on visibility settings.
- Refactored ImportConfiguration component to remove NAS import section.
2025-10-28 16:15:06 +08:00
chenghh-9609
e8e2c1a96b refactor: 修复标签管理功能、优化数据选择项显示、屏蔽开发中功能 (#12)
* refactor: clean up tag management and dataset handling, update API endpoints

* feat: add showTime prop to DevelopmentInProgress component across multiple pages

* refactor: update component styles and improve layout with new utility classes
2025-10-22 16:09:03 +08:00
Dallas98
1c97afed7d init datamate 2025-10-21 23:00:48 +08:00