q792602257/DataMate
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
381 Commits 1 Branch 0 Tags
49b7f2a64cc8a0ea64475f1fee9511b10c2359c0
Commit Graph

381 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
hhhhsc701
3b4f8488e8 fix: update Dockerfile to improve pip installation process and remove unnecessary uninstalls (#173) 
* feature: unstructured支持简单pdf处理

* feature: update values.yaml to enhance ray-cluster configuration with security context, environment variables, and resource limits

* fix: update Dockerfile to improve pip installation process and remove unnecessary uninstalls
 2025-12-17 11:49:47 +08:00
hhhhsc701
62b91b6deb bugfix: update values.yaml to enhance ray-cluster configuration with security context, environment variables, and resource limits (#172) 
* feature: unstructured支持简单pdf处理

* feature: update values.yaml to enhance ray-cluster configuration with security context, environment variables, and resource limits
 2025-12-17 10:41:13 +08:00
hefanli
082aca1597 fix: the interface for querying data set files is compatible with ret… (#171) 
fix: the interface for querying data set files is compatible with returns in file system format and list returns.
 2025-12-16 11:31:52 +08:00
hefanli
b3558d3202 修改系统参数的预置数据 (#170) 
* feature: add the pipeline for pushing images to Huawei Cloud

* fix: updates the dataset-pvc name
 2025-12-15 16:43:22 +08:00
hefanli
4712f00196 feature: add the pipeline for pushing images to Huawei Cloud (#167) 2025-12-12 16:44:43 +08:00
hhhhsc701
c51058a867 feature: 商业版构建 (#166) 2025-12-12 16:19:46 +08:00
hhhhsc701
d8c0b0ed73 补充modal范围 (#165) 2025-12-12 13:34:03 +08:00
hhhhsc701
fc9fb07e77 bugfix (#164) 2025-12-11 23:17:01 +08:00
Dallas98
ec87e4f204 feat(frontend): 增强Synthesis Data Detail页面UX体验 (#163) 
* fix(chart): update Helm chart helpers and values for improved configuration

* feat(SynthesisTaskTab): enhance task table with tooltip support and improved column widths

* feat(CreateTask, SynthFileTask): improve task creation and detail view with enhanced payload handling and UI updates

* feat(SynthFileTask): enhance file display with progress tracking and delete action

* feat(SynthFileTask): enhance file display with progress tracking and delete action

* feat(SynthDataDetail): add delete action for chunks with confirmation prompt

* feat(SynthDataDetail): update edit and delete buttons to icon-only format

* feat(SynthDataDetail): add confirmation modals for chunk and synthesis data deletion
 2025-12-11 21:02:44 +08:00
hefanli
8f529952f6 Fix ratio (#162) 
* fix: fixed the issue where an error would be reported when only setting the proportioning quantity when creating a proportioning task

* fix: prevent adding the same file multiple times

* fix: implement a more flexible matching strategy, allowing only the tag name to be configured for matching
 2025-12-11 17:45:16 +08:00
hhhhsc701
bb8641bea2 docs: update README files to include instructions for accessing the f… (#161) 
docs: update README files to include instructions for accessing the front-end interface
 2025-12-11 16:15:23 +08:00
o0Shark0o
12529276ee fix(settings): improve ModelAccess table responsiveness during browser zoom 2025-12-11 14:53:01 +08:00
hhhhsc701
72669d1293 feat: add .env and conf.yaml for deer-flow configuration (#160) 
* fix: update MILVUS_URI in .env.example for correct service endpoint

* feat: add .env and conf.yaml for deer-flow configuration
 2025-12-11 14:33:06 +08:00
hhhhsc701
a6e82ce68b fix: update MILVUS_URI in .env.example for correct service endpoint (#159) 2025-12-11 14:17:02 +08:00
hhhhsc701
f69ed6b8aa Revert "feature: 增加data-juicer算子" (#158) 
Revert "feature: 增加data-juicer算子 (#157)"

This reverts commit 786f13f9c3.
 2025-12-11 10:32:53 +08:00
hhhhsc701
786f13f9c3 feature: 增加data-juicer算子 (#157) 2025-12-11 10:32:19 +08:00
o0Shark0o
cfa6301e9e feat(annotation-templates): add new NLP templates for multilabel classification, keyword extraction, and text summarization 2025-12-11 09:37:32 +08:00
Dallas98
2f3ae21f8a feat: enhance dataset file fetching with improved pagination and document loading support (#156) 2025-12-10 22:39:24 +08:00
Dallas98
e9fd6a3ae1 fix: adjust pagination logic in dataset fetching to start from the current page 2025-12-10 19:52:06 +08:00
Dallas98
174359be9f feat(milvus): update Milvus configuration to use URI and remove deprecated host/port settings (#155) 2025-12-10 18:41:20 +08:00
hefanli
99fd46cb70 fix: fix the Data Evaluation Detail page (#154) 
* fix: the Data Evaluation Detail Page should show the model used

* fix: fix the time format displayed

* fix: fix the Data Evaluation Detail page
 2025-12-10 18:35:29 +08:00
Dallas98
4ed049ca11 feat(milvus): update Milvus configuration to use URI and remove deprecated host/port settings 2025-12-10 18:29:21 +08:00
Dallas98
44d72c446f feat(milvus): update Milvus configuration to use URI and remove deprecated host/port settings 2025-12-10 18:27:58 +08:00
hhhhsc701
19a04df276 feature: 增加水印去除/高级匿名化算子 (#151) 
* feature: 增加水印去除算子

* feature: clean code

* feature: clean code

* feature: 增加高级匿名化算子
 2025-12-10 18:12:47 +08:00
Dallas98
cbb146d3d7 feat(chart): add Helm chart for deploying Label Studio with PostgreSQL (#152) 
* feat(chart): add Helm chart for deploying Label Studio with PostgreSQL

* feat(milvus): update Milvus configuration to use URI and remove deprecated host/port settings
 2025-12-10 17:46:12 +08:00
hefanli
f87060490c feature: data management supports nested folders (#150) 
* fix: k8s部署场景下，backend-python服务挂载需要存储

* fix: 增加数据集文件免拷贝的接口定义

* fix: 评估时评估结果赋予初始空值，防止未评估完成时接口报错

* feature: 数据管理支持嵌套文件夹（展示时按照文件系统展示；批量下载时带上相对路径）

* fix: 去除多余的文件重命名逻辑

* refactor: remove unused imports
 2025-12-10 16:42:45 +08:00
o0Shark0o
fea7133dee Merge branch 'main' of github.com:ModelEngine-Group/DataMate 2025-12-10 15:07:28 +08:00
o0Shark0o
52accfb44c fix(synthesis-task): 修复文件选择列表中全选按钮无法勾选所有文件的问题 2025-12-10 15:05:14 +08:00
hhhhsc701
103c21945d 修复部分功能 (#138) 
* feature: 版本统一

* feature: 定时同步时默认值展示异常，增加提示

* feature: 修复数据归集搜索

* feature: 优化标注模板查询

* feature: 屏蔽webhook功能
 2025-12-10 14:31:05 +08:00
Dallas98
c18b7af2c4 docs: update README and Makefile for clarity and new development instructions (#147) 
* feat(synthesis): add evaluation task creation functionality and UI enhancements

* feat(synthesis): implement synthesis data management features including loading, editing, and deleting

* feat(synthesis): add endpoints for deleting and updating synthesis data and chunks

* fix: Correctly extract file values from selectedFilesMap in AddDataDialog

* docs: update README and Makefile for clarity and new development instructions
 2025-12-10 12:25:25 +08:00
Dallas98
6ccbc8a02f feat(dependencies): update greenlet to version 3.3.0 and adjust Python version compatibility (#145) 2025-12-10 10:17:48 +08:00
o0Shark0o
705009cab6 Merge branch 'main' of github.com:ModelEngine-Group/DataMate 2025-12-10 09:56:57 +08:00
uname
d9ce140c1f Merge branch 'main' of github.com:ModelEngine-Group/DataMate 2025-12-09 20:13:57 +08:00
uname
a728bc3100 修改数据标注：标注模板到主流语言。 2025-12-09 20:00:43 +08:00
hefanli
f8b32506cf fix: k8s部署场景下，backend-python服务挂载需要存储 (#144) 2025-12-09 19:09:51 +08:00
Dallas98
bef15f328d feat(config): add proxy configuration for evaluation API endpoint (#141) 2025-12-09 15:01:44 +08:00
hefanli
758cf93e36 feature: 增加压缩包上传功能 (#137) 
* feature: 增加压缩包上传功能

* fix: 删除文件时数据集关于文件的相关统计信息也刷新

* fix: 增加k8s常见下评估服务的路由
 2025-12-09 14:42:27 +08:00
Dallas98
015e738a7f feat(SynthDataDetail): add chunk/synthesis data management with edit/delete & UI enhancements (#139) 
* feat(synthesis): add evaluation task creation functionality and UI enhancements

* feat(synthesis): implement synthesis data management features including loading, editing, and deleting

* feat(synthesis): add endpoints for deleting and updating synthesis data and chunks

* fix: Correctly extract file values from selectedFilesMap in AddDataDialog
 2025-12-09 09:59:40 +08:00
Dallas98
cf20299af4 feat: add GitHub Actions workflow for releasing Docker images (#136) 2025-12-05 18:51:06 +08:00
hhhhsc701
d59c167da4 算子将抽取与落盘固定到流程中 (#134) 
* feature: 将抽取动作移到每一个算子中

* feature: 落盘算子改为默认执行

* feature: 优化前端展示

* feature: 使用pyproject管理依赖
 2025-12-05 17:26:29 +08:00
hefanli
744d15ba24 fix: 修复评估时模型输出json格式不对导致读取错误的问题 (#133) 
* feature: add cot data evaluation function

* fix: added verification to evaluation results

* fix: fix the prompt for evaluating

* fix: 修复当评估结果为空导致读取失败的问题
 2025-12-04 18:49:50 +08:00
Dallas98
31c4966608 feat(synthesis): add functionality to archive synthesis tasks to existing datasets (#132) 2025-12-04 17:11:43 +08:00
hhhhsc701
7a9530c1e3 feature: 增加对redis未部署时异常捕获 (#131) 
* feature: 增加download-deer-flow

* feature: 增加对redis未部署时异常捕获

* feature: clean code
 2025-12-04 16:09:29 +08:00
Dallas98
977d16339b feat(synthesis): add chunk-level synthesis data detail page & refine APIs/routing (#130) 
* feat: implement synthesis data detail view with chunk selection and data display
 2025-12-04 14:27:01 +08:00
Dallas98
7012a9ad98 feat: enhance backend deployment, frontend file selection and synthesis task management (#129) 
* feat: Implement data synthesis task management with database models and API endpoints

* feat: Update Python version requirements and refine dependency constraints in configuration

* fix: Correctly extract file values from selectedFilesMap in AddDataDialog

* feat: Refactor synthesis task routes and enhance file task management in the API

* feat: Enhance SynthesisTaskTab with tooltip actions and add chunk data retrieval in API
 2025-12-04 09:57:13 +08:00
hefanli
1d19cd3a62 feature: add data-evaluation 
* feature: add evaluation task management function

* feature: add evaluation task detail page

* fix: delete duplicate definition for table t_model_config

* refactor: rename package synthesis to ratio

* refactor: add eval file table and  refactor related code

* fix: calling large models in parallel during evaluation
 2025-12-04 09:23:54 +08:00
hhhhsc701
265e284fb8 feature: 修改算子开发指南 (#127) 2025-12-03 17:45:08 +08:00
hhhhsc701
c22683d635 优化部分问题 (#126) 
* feature: 支持相对路径引用

* feature: 优化本地部署命令

* feature: 优化算子编排展示

* feature: 优化清洗任务失败后重试
 2025-12-03 16:41:48 +08:00
Dallas98
04cff3f7e6 fix: Correctly extract file values from selectedFilesMap in AddDataDialog (#125) 
* feat: Implement data synthesis task management with database models and API endpoints

* feat: Update Python version requirements and refine dependency constraints in configuration

* fix: Correctly extract file values from selectedFilesMap in AddDataDialog
 2025-12-03 11:47:32 +08:00
hhhhsc701
b5fa8af900 bugfix: 修复deer-flow部署 (#124) 2025-12-02 19:23:30 +08:00