Commit Graph

500 Commits

Author SHA1 Message Date
Dallas98
8d2b41ed94 feature: Implement the basic knowledge generation function (#40) 2025-10-30 16:50:54 +08:00
chenghh-9609
5612c7cd91 add operator create page (#38)
* feat: Update site name to DataMate and refine text for AI data processing

* feat: Refactor settings page and implement model access functionality

- Created a new ModelAccess component for managing model configurations.
- Removed the old Settings component and replaced it with a new SettingsPage component that integrates ModelAccess, SystemConfig, and WebhookConfig.
- Added SystemConfig component for managing system settings.
- Implemented WebhookConfig component for managing webhook configurations.
- Updated API functions for model management in settings.apis.ts.
- Adjusted routing to point to the new SettingsPage component.

* feat: Implement Data Collection Page with Task Management and Execution Log

- Created DataCollectionPage component to manage data collection tasks.
- Added TaskManagement and ExecutionLog components for task handling and logging.
- Integrated task operations including start, stop, edit, and delete functionalities.
- Implemented filtering and searching capabilities in task management.
- Introduced SimpleCronScheduler for scheduling tasks with cron expressions.
- Updated CreateTask component to utilize new scheduling and template features.
- Enhanced BasicInformation component to conditionally render fields based on visibility settings.
- Refactored ImportConfiguration component to remove NAS import section.

* feat: Update task creation API endpoint and enhance task creation form with new fields and validation

* Refactor file upload and operator management components

- Removed unnecessary console logs from file download and export functions.
- Added size property to TaskItem interface for better task management.
- Simplified TaskUpload component by utilizing useFileSliceUpload hook for file upload logic.
- Enhanced OperatorPluginCreate component to handle file uploads and parsing more efficiently.
- Updated ConfigureStep component to use Ant Design Form for better data handling and validation.
- Improved PreviewStep component to navigate back to the operator market.
- Added support for additional file types in UploadStep component.
- Implemented delete operator functionality in OperatorMarketPage with confirmation prompts.
- Cleaned up unused API functions in operator.api.ts to streamline the codebase.
- Fixed number formatting utility to handle zero values correctly.
2025-10-30 16:30:01 +08:00
Jason Wang
e0884ab048 Develop py update schema (#37)
* feature: implement endpoints with multi-level response models
* refactor: move `/health` and `/config` endpoints to system module, remove example from base schemas
* refactor: remove unused get_standard_response_model()
2025-10-30 16:24:37 +08:00
Startalker
155603b1ca feature: add external pdf extract operator by using mineru (#36)
* feature: add UnstructuredFormatter

* feature: add UnstructuredFormatter in db

* feature: add unstructured[docx]==0.18.15

* feature: support doc

* feature: add mineru

* feature: add external pdf extract operator by using mineru

* feature: mineru docker install bugfix

---------

Co-authored-by: Startalker <438747480@qq.com>
2025-10-30 15:55:10 +08:00
Jason Wang
2f7341dc1f refactor: Reorganize datamate-python (#34)
refactor: Reorganize datamate-python (previously label-studio-adapter) into a DDD style structure.
2025-10-30 01:32:59 +08:00
chenghh-9609
0614157c0b fix: data collection create task page (#33)
* feat: Update site name to DataMate and refine text for AI data processing

* feat: Refactor settings page and implement model access functionality

- Created a new ModelAccess component for managing model configurations.
- Removed the old Settings component and replaced it with a new SettingsPage component that integrates ModelAccess, SystemConfig, and WebhookConfig.
- Added SystemConfig component for managing system settings.
- Implemented WebhookConfig component for managing webhook configurations.
- Updated API functions for model management in settings.apis.ts.
- Adjusted routing to point to the new SettingsPage component.

* feat: Implement Data Collection Page with Task Management and Execution Log

- Created DataCollectionPage component to manage data collection tasks.
- Added TaskManagement and ExecutionLog components for task handling and logging.
- Integrated task operations including start, stop, edit, and delete functionalities.
- Implemented filtering and searching capabilities in task management.
- Introduced SimpleCronScheduler for scheduling tasks with cron expressions.
- Updated CreateTask component to utilize new scheduling and template features.
- Enhanced BasicInformation component to conditionally render fields based on visibility settings.
- Refactored ImportConfiguration component to remove NAS import section.

* feat: Update task creation API endpoint and enhance task creation form with new fields and validation
2025-10-28 17:41:59 +08:00
Dallas98
3f484e988d feat: increase api_key length and enhance ModelConfig annotations (#32)
* refactor: rename artifactId and application name to 'datamate'; add model configuration and related services

* refactor: simplify package scanning by using wildcard for mapper packages

* feat: add model health check functionality and improve model configuration

* feat: increase api_key length and enhance ModelConfig annotations
2025-10-28 17:30:26 +08:00
hhhhsc701
67eb571d8d feature: 对接deer-flow (#27)
feature: 对接deer-flow
2025-10-28 16:28:26 +08:00
hhhhsc
4f5a9a9a83 refactor: simplify Dockerfile by removing redundant mirror configurations and cleaning up package installation commands 2025-10-28 16:24:40 +08:00
chenghh-9609
fad76e7477 data collection page (#31)
* feat: Update site name to DataMate and refine text for AI data processing

* feat: Refactor settings page and implement model access functionality

- Created a new ModelAccess component for managing model configurations.
- Removed the old Settings component and replaced it with a new SettingsPage component that integrates ModelAccess, SystemConfig, and WebhookConfig.
- Added SystemConfig component for managing system settings.
- Implemented WebhookConfig component for managing webhook configurations.
- Updated API functions for model management in settings.apis.ts.
- Adjusted routing to point to the new SettingsPage component.

* feat: Implement Data Collection Page with Task Management and Execution Log

- Created DataCollectionPage component to manage data collection tasks.
- Added TaskManagement and ExecutionLog components for task handling and logging.
- Integrated task operations including start, stop, edit, and delete functionalities.
- Implemented filtering and searching capabilities in task management.
- Introduced SimpleCronScheduler for scheduling tasks with cron expressions.
- Updated CreateTask component to utilize new scheduling and template features.
- Enhanced BasicInformation component to conditionally render fields based on visibility settings.
- Refactored ImportConfiguration component to remove NAS import section.
2025-10-28 16:15:06 +08:00
Dallas98
1a6e25758e feat: add model health check functionality and improve model configuration (#30)
* refactor: rename artifactId and application name to 'datamate'; add model configuration and related services

* refactor: simplify package scanning by using wildcard for mapper packages

* feat: add model health check functionality and improve model configuration
2025-10-28 16:06:53 +08:00
chenghh-9609
acafe70d90 feat: add ModelAccess to settings page (#29)
* feat: Update site name to DataMate and refine text for AI data processing

* feat: Refactor settings page and implement model access functionality

- Created a new ModelAccess component for managing model configurations.
- Removed the old Settings component and replaced it with a new SettingsPage component that integrates ModelAccess, SystemConfig, and WebhookConfig.
- Added SystemConfig component for managing system settings.
- Implemented WebhookConfig component for managing webhook configurations.
- Updated API functions for model management in settings.apis.ts.
- Adjusted routing to point to the new SettingsPage component.
2025-10-28 16:02:18 +08:00
Dallas98
a4b5238621 refactor: simplify package scanning by using wildcard for mapper packages (#28)
* refactor: rename artifactId and application name to 'datamate'; add model configuration and related services

* refactor: simplify package scanning by using wildcard for mapper packages
2025-10-28 14:12:44 +08:00
hhhhsc
41e7e684c3 Merge branch 'main' into develop_deer 2025-10-28 11:03:01 +08:00
hhhhsc
a69b9f4921 feature: 对接deer-flow 2025-10-28 10:54:29 +08:00
Dallas98
f54afddbeb refactor: rename artifactId and application name to 'datamate'; add model configuration and related services (#26) 2025-10-28 10:39:26 +08:00
Jinglong Wang
7f819563db Develop labeling module (#25)
* refactor: remove db table management from LS adapter (mv to scripts later); change adapter to use the same MySQL DB as other modules.

* refactor: Rename LS Adapter module to datamate-python
2025-10-27 16:16:14 +08:00
hefanli
46dfb389f1 feature:增加定时清除超出保留期限数据集的功能;增加数据归集任务绑定数据集的接口 (#24)
* refactor: 修改调整数据归集实现,删除无用代码,优化代码结构

* feature: 每天凌晨00:00扫描所有数据集,检查数据集是否超过了预设的保留天数,超出保留天数的数据集调用删除接口进行删除

* fix: 修改删除数据集文件的逻辑,上传到数据集中的文件会同时删除数据库中的记录和文件系统中的文件,归集过来的文件仅删除数据库中的记录

* fix: 增加参数校验和接口定义,删除不使用的接口

* fix: 数据集统计数据默认为0

* feature: 数据集状态增加流转,创建时为草稿状态,上传文件或者归集文件后修改为活动状态

* refactor: 修改分页查询归集任务的代码

* fix: 更新后重新执行;归集任务执行增加事务控制

* feature: 创建归集任务时能够同步创建数据集,更新归集任务时能更新到指定数据集
2025-10-25 15:59:36 +08:00
hhhhsc701
871ba5758d Merge pull request #23 from ModelEngine-Group/develop_op
refactor: update service and repository structure to use DTOs and imp…
2025-10-24 17:56:32 +08:00
hhhhsc
abc26c2c0e refactor: update service and repository structure to use DTOs and improve clarity 2025-10-24 17:55:41 +08:00
Jinglong Wang
ad9f41ffd7 feat: Dataset pagination; camelCase support in schemas (#22)
implement pagination for dataset mappings.
update response models to support camelCase parameters.
2025-10-24 17:14:42 +08:00
hhhhsc701
f9dbefd737 Merge pull request #21 from ModelEngine-Group/develop_db
refactor: rename and reorganize data models and repositories for clarity
2025-10-24 15:46:32 +08:00
hhhhsc
2d2419205a refactor: rename and reorganize data models and repositories for clarity 2025-10-24 15:33:46 +08:00
chenghh-9609
ebc3c2c09b Merge pull request #19 from chenghh-9609/main
Merge pull request #1 from ModelEngine-Group/main
2025-10-24 09:43:37 +08:00
chenghh-9609
05dc69026e feat: Add update event handling for dataset operations and improve dataset editing logic 2025-10-24 09:41:24 +08:00
chenghh-9609
5d937a7ef3 Merge branch 'ModelEngine-Group:main' into main 2025-10-24 09:41:06 +08:00
hefanli
cc072bbf90 refactor: 修改调整数据归集实现,删除无用代码,优化代码结构 (#20) 2025-10-23 21:10:57 +08:00
hhhhsc701
d58c2a0ac7 Merge pull request #18 from ModelEngine-Group/develop_doc
refactor: reorganize Helm chart structure and update service configur…
2025-10-23 17:17:05 +08:00
chenghh-9609
8e9a94971e Merge pull request #1 from ModelEngine-Group/main
merge
2025-10-23 17:13:24 +08:00
chenghh-9609
d66925410e Merge branch 'main' into main 2025-10-23 17:12:46 +08:00
hhhhsc
17e6cea1d9 refactor: reorganize Helm chart structure and update service configurations 2025-10-23 16:57:12 +08:00
chenghh-9609
4e53cc813b feat: Enhance CardView and DatasetManagement with loading state and pagination updates 2025-10-23 16:56:36 +08:00
Startalker
f86d4fae25 feature: add unstructured formatter operator for doc/docx (#17)
* feature: add UnstructuredFormatter

* feature: add UnstructuredFormatter in db

* feature: add unstructured[docx]==0.18.15

* feature: support doc

---------

Co-authored-by: Startalker <438747480@qq.com>
2025-10-23 16:49:03 +08:00
chenghh-9609
c52702b073 feat:修复下载数据集问题、删除数据确认框、修改标题、添加列表轮询刷新 (#16)
* refactor: clean up tag management and dataset handling, update API endpoints

* feat: add showTime prop to DevelopmentInProgress component across multiple pages

* refactor: update component styles and improve layout with new utility classes

* feat: enhance useFetchData hook with polling functionality and improve task progress tracking

* feat: enhance dataset management features with improved tag handling, download functionality, and UI updates

* feat: Enhance DatasetDetail component with delete functionality and improved download handling

feat: Add automatic data refresh and improved user feedback in DatasetManagementPage

fix: Update dataset API to streamline download functionality and improve error handling

* feat: Clear new tag input after successful addition in TagManager
2025-10-23 16:48:42 +08:00
chenghh-9609
aba7a3e1fb feat: Clear new tag input after successful addition in TagManager 2025-10-23 16:05:13 +08:00
chenghh-9609
bb116839ae feat: Enhance DatasetDetail component with delete functionality and improved download handling
feat: Add automatic data refresh and improved user feedback in DatasetManagementPage

fix: Update dataset API to streamline download functionality and improve error handling
2025-10-23 15:37:22 +08:00
chenghh-9609
a6d4b51601 feat: enhance dataset management features with improved tag handling, download functionality, and UI updates 2025-10-23 10:33:42 +08:00
chenghh-9609
add6998601 Merge branch 'main' of github.com:chenghh-9609/DataMate 2025-10-23 09:56:57 +08:00
chenghh-9609
17960f674f feat: enhance useFetchData hook with polling functionality and improve task progress tracking 2025-10-23 09:56:06 +08:00
chenghh-9609
665c11f2a1 Merge branch 'ModelEngine-Group:main' into main 2025-10-23 09:08:20 +08:00
Dallas98
c998de2e9d refactor: update Docker workflows to use reusable configuration and restrict branch triggers to 'main' (#15)
* refactor: update Docker workflows to use reusable configuration and restrict branch triggers to 'main'
2025-10-22 23:44:44 +08:00
hhhhsc701
31ef8bc265 [Feature] Refactor project to use 'datamate' naming convention for services and configurations (#14)
* Enhance CleaningTaskService to track cleaning process progress and update ExecutorType to DATAMATE

* Refactor project to use 'datamate' naming convention for services and configurations
2025-10-22 17:53:16 +08:00
Jinglong Wang
175d9ded93 [Feature] Add Label Studio adapter module and its build scipts.
[Feature] Add Label Studio adapter module and its build scipts.
2025-10-22 16:54:49 +08:00
Jinglong Wang
c6fbc5ac39 Merge branch 'ModelEngine-Group:main' into develop_label_tool 2025-10-22 16:51:40 +08:00
chenghh-9609
e8e2c1a96b refactor: 修复标签管理功能、优化数据选择项显示、屏蔽开发中功能 (#12)
* refactor: clean up tag management and dataset handling, update API endpoints

* feat: add showTime prop to DevelopmentInProgress component across multiple pages

* refactor: update component styles and improve layout with new utility classes
2025-10-22 16:09:03 +08:00
chenghh-9609
69b9517181 refactor: update component styles and improve layout with new utility classes 2025-10-22 15:51:50 +08:00
chenghh-9609
f17e82a3cc feat: add showTime prop to DevelopmentInProgress component across multiple pages 2025-10-22 15:19:24 +08:00
Jason Wang
c640105333 Add Label Studio adapter module and its build scipts. 2025-10-22 15:14:01 +08:00
chenghh-9609
b9a7e7a0a9 refactor: clean up tag management and dataset handling, update API endpoints 2025-10-22 14:41:14 +08:00
Dallas98
1c97afed7d init datamate 2025-10-21 23:00:48 +08:00