Commit Graph

316 Commits

Author SHA1 Message Date
hhhhsc701
786f13f9c3 feature: 增加data-juicer算子 (#157) 2025-12-11 10:32:19 +08:00
o0Shark0o
cfa6301e9e feat(annotation-templates): add new NLP templates for multilabel classification, keyword extraction, and text summarization 2025-12-11 09:37:32 +08:00
Dallas98
2f3ae21f8a feat: enhance dataset file fetching with improved pagination and document loading support (#156) 2025-12-10 22:39:24 +08:00
Dallas98
e9fd6a3ae1 fix: adjust pagination logic in dataset fetching to start from the current page 2025-12-10 19:52:06 +08:00
Dallas98
174359be9f feat(milvus): update Milvus configuration to use URI and remove deprecated host/port settings (#155) 2025-12-10 18:41:20 +08:00
hefanli
99fd46cb70 fix: fix the Data Evaluation Detail page (#154)
* fix: the Data Evaluation Detail Page should show the model used

* fix: fix the time format displayed

* fix: fix the Data Evaluation Detail page
2025-12-10 18:35:29 +08:00
Dallas98
4ed049ca11 feat(milvus): update Milvus configuration to use URI and remove deprecated host/port settings 2025-12-10 18:29:21 +08:00
Dallas98
44d72c446f feat(milvus): update Milvus configuration to use URI and remove deprecated host/port settings 2025-12-10 18:27:58 +08:00
hhhhsc701
19a04df276 feature: 增加水印去除/高级匿名化算子 (#151)
* feature: 增加水印去除算子

* feature: clean code

* feature: clean code

* feature: 增加高级匿名化算子
2025-12-10 18:12:47 +08:00
Dallas98
cbb146d3d7 feat(chart): add Helm chart for deploying Label Studio with PostgreSQL (#152)
* feat(chart): add Helm chart for deploying Label Studio with PostgreSQL

* feat(milvus): update Milvus configuration to use URI and remove deprecated host/port settings
2025-12-10 17:46:12 +08:00
hefanli
f87060490c feature: data management supports nested folders (#150)
* fix: k8s部署场景下,backend-python服务挂载需要存储

* fix: 增加数据集文件免拷贝的接口定义

* fix: 评估时评估结果赋予初始空值,防止未评估完成时接口报错

* feature: 数据管理支持嵌套文件夹(展示时按照文件系统展示;批量下载时带上相对路径)

* fix: 去除多余的文件重命名逻辑

* refactor: remove unused imports
2025-12-10 16:42:45 +08:00
o0Shark0o
fea7133dee Merge branch 'main' of github.com:ModelEngine-Group/DataMate 2025-12-10 15:07:28 +08:00
o0Shark0o
52accfb44c fix(synthesis-task): 修复文件选择列表中全选按钮无法勾选所有文件的问题 2025-12-10 15:05:14 +08:00
hhhhsc701
103c21945d 修复部分功能 (#138)
* feature: 版本统一

* feature: 定时同步时默认值展示异常,增加提示

* feature: 修复数据归集搜索

* feature: 优化标注模板查询

* feature: 屏蔽webhook功能
2025-12-10 14:31:05 +08:00
Dallas98
c18b7af2c4 docs: update README and Makefile for clarity and new development instructions (#147)
* feat(synthesis): add evaluation task creation functionality and UI enhancements

* feat(synthesis): implement synthesis data management features including loading, editing, and deleting

* feat(synthesis): add endpoints for deleting and updating synthesis data and chunks

* fix: Correctly extract file values from selectedFilesMap in AddDataDialog

* docs: update README and Makefile for clarity and new development instructions
2025-12-10 12:25:25 +08:00
Dallas98
6ccbc8a02f feat(dependencies): update greenlet to version 3.3.0 and adjust Python version compatibility (#145) 2025-12-10 10:17:48 +08:00
o0Shark0o
705009cab6 Merge branch 'main' of github.com:ModelEngine-Group/DataMate 2025-12-10 09:56:57 +08:00
uname
d9ce140c1f Merge branch 'main' of github.com:ModelEngine-Group/DataMate 2025-12-09 20:13:57 +08:00
uname
a728bc3100 修改数据标注:标注模板到主流语言。 2025-12-09 20:00:43 +08:00
hefanli
f8b32506cf fix: k8s部署场景下,backend-python服务挂载需要存储 (#144) 2025-12-09 19:09:51 +08:00
Dallas98
bef15f328d feat(config): add proxy configuration for evaluation API endpoint (#141) 2025-12-09 15:01:44 +08:00
hefanli
758cf93e36 feature: 增加压缩包上传功能 (#137)
* feature: 增加压缩包上传功能

* fix: 删除文件时数据集关于文件的相关统计信息也刷新

* fix: 增加k8s常见下评估服务的路由
2025-12-09 14:42:27 +08:00
Dallas98
015e738a7f feat(SynthDataDetail): add chunk/synthesis data management with edit/delete & UI enhancements (#139)
* feat(synthesis): add evaluation task creation functionality and UI enhancements

* feat(synthesis): implement synthesis data management features including loading, editing, and deleting

* feat(synthesis): add endpoints for deleting and updating synthesis data and chunks

* fix: Correctly extract file values from selectedFilesMap in AddDataDialog
2025-12-09 09:59:40 +08:00
Dallas98
cf20299af4 feat: add GitHub Actions workflow for releasing Docker images (#136) 2025-12-05 18:51:06 +08:00
hhhhsc701
d59c167da4 算子将抽取与落盘固定到流程中 (#134)
* feature: 将抽取动作移到每一个算子中

* feature: 落盘算子改为默认执行

* feature: 优化前端展示

* feature: 使用pyproject管理依赖
2025-12-05 17:26:29 +08:00
hefanli
744d15ba24 fix: 修复评估时模型输出json格式不对导致读取错误的问题 (#133)
* feature: add cot data evaluation function

* fix: added verification to evaluation results

* fix: fix the prompt for evaluating

* fix: 修复当评估结果为空导致读取失败的问题
2025-12-04 18:49:50 +08:00
Dallas98
31c4966608 feat(synthesis): add functionality to archive synthesis tasks to existing datasets (#132) 2025-12-04 17:11:43 +08:00
hhhhsc701
7a9530c1e3 feature: 增加对redis未部署时异常捕获 (#131)
* feature: 增加download-deer-flow

* feature: 增加对redis未部署时异常捕获

* feature: clean code
2025-12-04 16:09:29 +08:00
Dallas98
977d16339b feat(synthesis): add chunk-level synthesis data detail page & refine APIs/routing (#130)
* feat: implement synthesis data detail view with chunk selection and data display
2025-12-04 14:27:01 +08:00
Dallas98
7012a9ad98 feat: enhance backend deployment, frontend file selection and synthesis task management (#129)
* feat: Implement data synthesis task management with database models and API endpoints

* feat: Update Python version requirements and refine dependency constraints in configuration

* fix: Correctly extract file values from selectedFilesMap in AddDataDialog

* feat: Refactor synthesis task routes and enhance file task management in the API

* feat: Enhance SynthesisTaskTab with tooltip actions and add chunk data retrieval in API
2025-12-04 09:57:13 +08:00
hefanli
1d19cd3a62 feature: add data-evaluation
* feature: add evaluation task management function

* feature: add evaluation task detail page

* fix: delete duplicate definition for table t_model_config

* refactor: rename package synthesis to ratio

* refactor: add eval file table and  refactor related code

* fix: calling large models in parallel during evaluation
2025-12-04 09:23:54 +08:00
hhhhsc701
265e284fb8 feature: 修改算子开发指南 (#127) 2025-12-03 17:45:08 +08:00
hhhhsc701
c22683d635 优化部分问题 (#126)
* feature: 支持相对路径引用

* feature: 优化本地部署命令

* feature: 优化算子编排展示

* feature: 优化清洗任务失败后重试
2025-12-03 16:41:48 +08:00
Dallas98
04cff3f7e6 fix: Correctly extract file values from selectedFilesMap in AddDataDialog (#125)
* feat: Implement data synthesis task management with database models and API endpoints

* feat: Update Python version requirements and refine dependency constraints in configuration

* fix: Correctly extract file values from selectedFilesMap in AddDataDialog
2025-12-03 11:47:32 +08:00
hhhhsc701
b5fa8af900 bugfix: 修复deer-flow部署 (#124) 2025-12-02 19:23:30 +08:00
Dallas98
bcd1bc1534 feat: Update Python version requirements and refine dependency constraints in configuration (#123)
* feat: Implement data synthesis task management with database models and API endpoints

* feat: Update Python version requirements and refine dependency constraints in configuration
2025-12-02 16:34:22 +08:00
Dallas98
8b164cb012 feat: Implement data synthesis task management with database models and API endpoints (#122) 2025-12-02 15:23:58 +08:00
Dallas98
458afa2966 feat: Add original file ID to document metadata in RagEtlService #121 2025-12-02 15:10:07 +08:00
Jason Wang
d692f5fdae feat: new endpoint allowing only add file path to dataset record without any FS operations (#119)
* feat: Implement add files' path only to dataset

* refactor: Rename variable for clarity in metadata serialization
2025-12-01 20:31:06 +08:00
hhhhsc701
f730bd5b0c bugfix: 支持使用runtime单实例 (#118)
* bugfix: 支持使用runtime单实例
2025-12-01 19:05:50 +08:00
Dallas98
9fc35f066f feat: Add original file ID to document metadata in RagEtlService 2025-12-01 17:04:52 +08:00
hhhhsc701
bb3345268e bugfix: 清洗/算子支持名称/描述搜索 (#116)
* bugfix: milvus适配etcd deploy部署

* bugfix: 可以在知识库界面跳转到创建模型
2025-11-29 18:15:43 +08:00
chenghh-9609
5c178d5274 Integrated Redux for state management with auth and settings slices. (#117)
* feat: Implement DatasetFileTransfer component for file selection and management

* feat: Add pagination support to file list in Overview component

* feat: add DatasetFileTransfer and TagManagement components

- Added DatasetFileTransfer component for managing dataset files.
- Introduced TagManagement component for handling tags.
- Integrated Redux for state management with auth and settings slices.
- Updated package.json to include @reduxjs/toolkit and react-redux dependencies.
- Refactored existing components to utilize new DatasetFileTransfer and TagManagement components.
- Implemented hooks for typed dispatch and selector in Redux.
- Enhanced CreateKnowledgeBase and SynthesisTask components to support new features.
2025-11-29 17:37:36 +08:00
Jason Wang
2e13bb9b4c feat: Add download and load images functionality to Makefile (#115) 2025-11-29 10:53:19 +08:00
hhhhsc701
fe42b03548 bugfix: milvus部分组件支持镜像仓库 (#114) 2025-11-28 17:39:56 +08:00
hhhhsc701
07029d07ff 优化清洗重试机制,优化清洗进度展示,修复模板无法展示参数 (#113)
* bugfix: 模板无法展示参数

* bugfix: 优化清洗进度展示

* bugfix: 优化清洗重试机制
2025-11-28 15:28:10 +08:00
hhhhsc701
f1bffdcd61 bugfix: 创建清洗任务时修改数据集状态;无法删除已在模板/运行任务的算子
* bugfix: 创建清洗任务时修改数据集状态;无法删除已在模板/运行任务的算子
2025-11-27 17:34:53 +08:00
hhhhsc701
91390cace0 feature: 北向接口:支持通过模板创建清洗任务 (#111)
feature: 北向接口:支持通过模板创建清洗任务
2025-11-26 17:30:54 +08:00
Jason Wang
0ca5f29885 feat: Add build target for generic docker builds in Makefile (#110)
* feature: Add build target for generic docker builds in Makefile
2025-11-26 14:35:07 +08:00
hhhhsc701
bc2f57f2c0 feature: 修改缩进 (#109) 2025-11-26 11:24:50 +08:00