Commit Graph

155 Commits

Author SHA1 Message Date
o0Shark0o
fea7133dee Merge branch 'main' of github.com:ModelEngine-Group/DataMate 2025-12-10 15:07:28 +08:00
o0Shark0o
52accfb44c fix(synthesis-task): 修复文件选择列表中全选按钮无法勾选所有文件的问题 2025-12-10 15:05:14 +08:00
hhhhsc701
103c21945d 修复部分功能 (#138)
* feature: 版本统一

* feature: 定时同步时默认值展示异常,增加提示

* feature: 修复数据归集搜索

* feature: 优化标注模板查询

* feature: 屏蔽webhook功能
2025-12-10 14:31:05 +08:00
Dallas98
c18b7af2c4 docs: update README and Makefile for clarity and new development instructions (#147)
* feat(synthesis): add evaluation task creation functionality and UI enhancements

* feat(synthesis): implement synthesis data management features including loading, editing, and deleting

* feat(synthesis): add endpoints for deleting and updating synthesis data and chunks

* fix: Correctly extract file values from selectedFilesMap in AddDataDialog

* docs: update README and Makefile for clarity and new development instructions
2025-12-10 12:25:25 +08:00
Dallas98
6ccbc8a02f feat(dependencies): update greenlet to version 3.3.0 and adjust Python version compatibility (#145) 2025-12-10 10:17:48 +08:00
o0Shark0o
705009cab6 Merge branch 'main' of github.com:ModelEngine-Group/DataMate 2025-12-10 09:56:57 +08:00
uname
d9ce140c1f Merge branch 'main' of github.com:ModelEngine-Group/DataMate 2025-12-09 20:13:57 +08:00
uname
a728bc3100 修改数据标注:标注模板到主流语言。 2025-12-09 20:00:43 +08:00
hefanli
f8b32506cf fix: k8s部署场景下,backend-python服务挂载需要存储 (#144) 2025-12-09 19:09:51 +08:00
Dallas98
bef15f328d feat(config): add proxy configuration for evaluation API endpoint (#141) 2025-12-09 15:01:44 +08:00
hefanli
758cf93e36 feature: 增加压缩包上传功能 (#137)
* feature: 增加压缩包上传功能

* fix: 删除文件时数据集关于文件的相关统计信息也刷新

* fix: 增加k8s常见下评估服务的路由
2025-12-09 14:42:27 +08:00
Dallas98
015e738a7f feat(SynthDataDetail): add chunk/synthesis data management with edit/delete & UI enhancements (#139)
* feat(synthesis): add evaluation task creation functionality and UI enhancements

* feat(synthesis): implement synthesis data management features including loading, editing, and deleting

* feat(synthesis): add endpoints for deleting and updating synthesis data and chunks

* fix: Correctly extract file values from selectedFilesMap in AddDataDialog
2025-12-09 09:59:40 +08:00
Dallas98
cf20299af4 feat: add GitHub Actions workflow for releasing Docker images (#136) 2025-12-05 18:51:06 +08:00
hhhhsc701
d59c167da4 算子将抽取与落盘固定到流程中 (#134)
* feature: 将抽取动作移到每一个算子中

* feature: 落盘算子改为默认执行

* feature: 优化前端展示

* feature: 使用pyproject管理依赖
2025-12-05 17:26:29 +08:00
hefanli
744d15ba24 fix: 修复评估时模型输出json格式不对导致读取错误的问题 (#133)
* feature: add cot data evaluation function

* fix: added verification to evaluation results

* fix: fix the prompt for evaluating

* fix: 修复当评估结果为空导致读取失败的问题
2025-12-04 18:49:50 +08:00
Dallas98
31c4966608 feat(synthesis): add functionality to archive synthesis tasks to existing datasets (#132) 2025-12-04 17:11:43 +08:00
hhhhsc701
7a9530c1e3 feature: 增加对redis未部署时异常捕获 (#131)
* feature: 增加download-deer-flow

* feature: 增加对redis未部署时异常捕获

* feature: clean code
2025-12-04 16:09:29 +08:00
Dallas98
977d16339b feat(synthesis): add chunk-level synthesis data detail page & refine APIs/routing (#130)
* feat: implement synthesis data detail view with chunk selection and data display
2025-12-04 14:27:01 +08:00
Dallas98
7012a9ad98 feat: enhance backend deployment, frontend file selection and synthesis task management (#129)
* feat: Implement data synthesis task management with database models and API endpoints

* feat: Update Python version requirements and refine dependency constraints in configuration

* fix: Correctly extract file values from selectedFilesMap in AddDataDialog

* feat: Refactor synthesis task routes and enhance file task management in the API

* feat: Enhance SynthesisTaskTab with tooltip actions and add chunk data retrieval in API
2025-12-04 09:57:13 +08:00
hefanli
1d19cd3a62 feature: add data-evaluation
* feature: add evaluation task management function

* feature: add evaluation task detail page

* fix: delete duplicate definition for table t_model_config

* refactor: rename package synthesis to ratio

* refactor: add eval file table and  refactor related code

* fix: calling large models in parallel during evaluation
2025-12-04 09:23:54 +08:00
hhhhsc701
265e284fb8 feature: 修改算子开发指南 (#127) 2025-12-03 17:45:08 +08:00
hhhhsc701
c22683d635 优化部分问题 (#126)
* feature: 支持相对路径引用

* feature: 优化本地部署命令

* feature: 优化算子编排展示

* feature: 优化清洗任务失败后重试
2025-12-03 16:41:48 +08:00
Dallas98
04cff3f7e6 fix: Correctly extract file values from selectedFilesMap in AddDataDialog (#125)
* feat: Implement data synthesis task management with database models and API endpoints

* feat: Update Python version requirements and refine dependency constraints in configuration

* fix: Correctly extract file values from selectedFilesMap in AddDataDialog
2025-12-03 11:47:32 +08:00
hhhhsc701
b5fa8af900 bugfix: 修复deer-flow部署 (#124) 2025-12-02 19:23:30 +08:00
Dallas98
bcd1bc1534 feat: Update Python version requirements and refine dependency constraints in configuration (#123)
* feat: Implement data synthesis task management with database models and API endpoints

* feat: Update Python version requirements and refine dependency constraints in configuration
2025-12-02 16:34:22 +08:00
Dallas98
8b164cb012 feat: Implement data synthesis task management with database models and API endpoints (#122) 2025-12-02 15:23:58 +08:00
Dallas98
458afa2966 feat: Add original file ID to document metadata in RagEtlService #121 2025-12-02 15:10:07 +08:00
Jason Wang
d692f5fdae feat: new endpoint allowing only add file path to dataset record without any FS operations (#119)
* feat: Implement add files' path only to dataset

* refactor: Rename variable for clarity in metadata serialization
2025-12-01 20:31:06 +08:00
hhhhsc701
f730bd5b0c bugfix: 支持使用runtime单实例 (#118)
* bugfix: 支持使用runtime单实例
2025-12-01 19:05:50 +08:00
Dallas98
9fc35f066f feat: Add original file ID to document metadata in RagEtlService 2025-12-01 17:04:52 +08:00
hhhhsc701
bb3345268e bugfix: 清洗/算子支持名称/描述搜索 (#116)
* bugfix: milvus适配etcd deploy部署

* bugfix: 可以在知识库界面跳转到创建模型
2025-11-29 18:15:43 +08:00
chenghh-9609
5c178d5274 Integrated Redux for state management with auth and settings slices. (#117)
* feat: Implement DatasetFileTransfer component for file selection and management

* feat: Add pagination support to file list in Overview component

* feat: add DatasetFileTransfer and TagManagement components

- Added DatasetFileTransfer component for managing dataset files.
- Introduced TagManagement component for handling tags.
- Integrated Redux for state management with auth and settings slices.
- Updated package.json to include @reduxjs/toolkit and react-redux dependencies.
- Refactored existing components to utilize new DatasetFileTransfer and TagManagement components.
- Implemented hooks for typed dispatch and selector in Redux.
- Enhanced CreateKnowledgeBase and SynthesisTask components to support new features.
2025-11-29 17:37:36 +08:00
Jason Wang
2e13bb9b4c feat: Add download and load images functionality to Makefile (#115) 2025-11-29 10:53:19 +08:00
hhhhsc701
fe42b03548 bugfix: milvus部分组件支持镜像仓库 (#114) 2025-11-28 17:39:56 +08:00
hhhhsc701
07029d07ff 优化清洗重试机制,优化清洗进度展示,修复模板无法展示参数 (#113)
* bugfix: 模板无法展示参数

* bugfix: 优化清洗进度展示

* bugfix: 优化清洗重试机制
2025-11-28 15:28:10 +08:00
hhhhsc701
f1bffdcd61 bugfix: 创建清洗任务时修改数据集状态;无法删除已在模板/运行任务的算子
* bugfix: 创建清洗任务时修改数据集状态;无法删除已在模板/运行任务的算子
2025-11-27 17:34:53 +08:00
hhhhsc701
91390cace0 feature: 北向接口:支持通过模板创建清洗任务 (#111)
feature: 北向接口:支持通过模板创建清洗任务
2025-11-26 17:30:54 +08:00
Jason Wang
0ca5f29885 feat: Add build target for generic docker builds in Makefile (#110)
* feature: Add build target for generic docker builds in Makefile
2025-11-26 14:35:07 +08:00
hhhhsc701
bc2f57f2c0 feature: 修改缩进 (#109) 2025-11-26 11:24:50 +08:00
Dallas98
bc26cfba55 feat: Refactor knowledge base retrieval to return detailed search results and enhance API integration #108 2025-11-25 21:21:21 +08:00
hhhhsc701
b50c12d135 feature: 删除kuberay镜像推送 (#107)
feature: 删除kuberay镜像推送
2025-11-25 17:46:50 +08:00
Jason Wang
c097a06fbe feat: Improve makefile readability, Add user control on volume keep at uninstallation, Add Label Studio install and uninstall via Make. (#106)
* feature: Add label studio installation and uninstallation commands to Makefile

* feature: Enhance Makefile with detailed help commands and improve install/uninstall targets for services

* feature: Update Makefile help commands to clarify usage of local images

* feature: Improve error handling in Makefile for build, install, and uninstall targets

* feature: Enhance uninstall process in Makefile to prompt for volume deletion and update README with usage details

---------

Co-authored-by: Jason Wang <wjl_jason@qq.com>
2025-11-25 17:37:28 +08:00
hhhhsc701
af2a01e52d feature: milvus pvc支持本地目录 (#105)
feature: milvus pvc支持本地目录
2025-11-25 16:54:24 +08:00
hhhhsc701
fb399b74cf feature: pvc支持本地盘+配置sc (#104) 2025-11-24 17:29:32 +08:00
hefanli
c1352ab91f feature: multiple ratio configurations can be set for the data set. (#103)
feature: multiple ratio configurations can be set for the data set.
2025-11-24 15:28:17 +08:00
hhhhsc701
497a5688e9 feature: df构建双架构镜像 (#102) 2025-11-24 11:52:08 +08:00
hhhhsc701
a53f6776b8 feature: 构建双架构镜像 (#101)
feature: 构建双架构镜像
2025-11-24 11:34:53 +08:00
Dallas98
9858388084 feat: Refactor dataset file pagination and enhance retrieval functionality with new request structure #98
* feat: Enhance knowledge base management with collection renaming, imp…

* feat: Update Milvus integration with new API, enhance collection mana…

* Merge branch 'refs/heads/main' into dev

* feat: Refactor dataset file pagination and enhance retrieval function…

* Merge branch 'main' into dev
2025-11-21 17:28:25 +08:00
hhhhsc701
536ef9f556 feature: milvus service名称变更 兼容k8s (#97)
feature: milvus service名称变更  兼容k8s (#97)
2025-11-21 12:06:53 +08:00
chenghh-9609
fdfcfec1f1 update knowledge base file selection component (#96)
* feat: Implement DatasetFileTransfer component for file selection and management

* feat: Add pagination support to file list in Overview component
2025-11-21 11:39:26 +08:00