hefanli
f87060490c
feature: data management supports nested folders ( #150 )
...
* fix: k8s部署场景下，backend-python服务挂载需要存储
* fix: 增加数据集文件免拷贝的接口定义
* fix: 评估时评估结果赋予初始空值，防止未评估完成时接口报错
* feature: 数据管理支持嵌套文件夹（展示时按照文件系统展示；批量下载时带上相对路径）
* fix: 去除多余的文件重命名逻辑
* refactor: remove unused imports
2025-12-10 16:42:45 +08:00
hhhhsc701
103c21945d
修复部分功能 ( #138 )
...
* feature: 版本统一
* feature: 定时同步时默认值展示异常，增加提示
* feature: 修复数据归集搜索
* feature: 优化标注模板查询
* feature: 屏蔽webhook功能
2025-12-10 14:31:05 +08:00
hefanli
758cf93e36
feature: 增加压缩包上传功能 ( #137 )
...
* feature: 增加压缩包上传功能
* fix: 删除文件时数据集关于文件的相关统计信息也刷新
* fix: 增加k8s常见下评估服务的路由
2025-12-09 14:42:27 +08:00
hhhhsc701
7a9530c1e3
feature: 增加对redis未部署时异常捕获 ( #131 )
...
* feature: 增加download-deer-flow
* feature: 增加对redis未部署时异常捕获
* feature: clean code
2025-12-04 16:09:29 +08:00
hefanli
1d19cd3a62
feature: add data-evaluation
...
* feature: add evaluation task management function
* feature: add evaluation task detail page
* fix: delete duplicate definition for table t_model_config
* refactor: rename package synthesis to ratio
* refactor: add eval file table and refactor related code
* fix: calling large models in parallel during evaluation
2025-12-04 09:23:54 +08:00
hhhhsc701
c22683d635
优化部分问题 ( #126 )
...
* feature: 支持相对路径引用
* feature: 优化本地部署命令
* feature: 优化算子编排展示
* feature: 优化清洗任务失败后重试
2025-12-03 16:41:48 +08:00
Dallas98
458afa2966
feat: Add original file ID to document metadata in RagEtlService #121
2025-12-02 15:10:07 +08:00
Jason Wang
d692f5fdae
feat: new endpoint allowing only add file path to dataset record without any FS operations ( #119 )
...
* feat: Implement add files' path only to dataset
* refactor: Rename variable for clarity in metadata serialization
2025-12-01 20:31:06 +08:00
Dallas98
9fc35f066f
feat: Add original file ID to document metadata in RagEtlService
2025-12-01 17:04:52 +08:00
hhhhsc701
bb3345268e
bugfix: 清洗/算子支持名称/描述搜索 ( #116 )
...
* bugfix: milvus适配etcd deploy部署
* bugfix: 可以在知识库界面跳转到创建模型
2025-11-29 18:15:43 +08:00
hhhhsc701
07029d07ff
优化清洗重试机制，优化清洗进度展示，修复模板无法展示参数 ( #113 )
...
* bugfix: 模板无法展示参数
* bugfix: 优化清洗进度展示
* bugfix: 优化清洗重试机制
2025-11-28 15:28:10 +08:00
hhhhsc701
f1bffdcd61
bugfix: 创建清洗任务时修改数据集状态；无法删除已在模板/运行任务的算子
...
* bugfix: 创建清洗任务时修改数据集状态；无法删除已在模板/运行任务的算子
2025-11-27 17:34:53 +08:00
hhhhsc701
91390cace0
feature: 北向接口：支持通过模板创建清洗任务 ( #111 )
...
feature: 北向接口：支持通过模板创建清洗任务
2025-11-26 17:30:54 +08:00
Dallas98
bc26cfba55
feat: Refactor knowledge base retrieval to return detailed search results and enhance API integration #108
2025-11-25 21:21:21 +08:00
hefanli
c1352ab91f
feature: multiple ratio configurations can be set for the data set. ( #103 )
...
feature: multiple ratio configurations can be set for the data set.
2025-11-24 15:28:17 +08:00
Dallas98
9858388084
feat: Refactor dataset file pagination and enhance retrieval functionality with new request structure #98
...
* feat: Enhance knowledge base management with collection renaming, imp…
* feat: Update Milvus integration with new API, enhance collection mana…
* Merge branch 'refs/heads/main' into dev
* feat: Refactor dataset file pagination and enhance retrieval function…
* Merge branch 'main' into dev
2025-11-21 17:28:25 +08:00
hhhhsc701
536ef9f556
feature: milvus service名称变更 兼容k8s ( #97 )
...
feature: milvus service名称变更 兼容k8s (#97 )
2025-11-21 12:06:53 +08:00
hefanli
a07fba23f2
feature：数据集导入数据集支持选择归集任务导入 ( #92 )
...
* feature: 实现obs归集
* feature: 增加数据集中出现同名文件时的处理方式
* feature: 前端数据集导入数据时增加可以选择归集任务导入
2025-11-19 11:05:33 +08:00
Dallas98
4506fa8a91
feat: Enhance AddDataDialog with dataset file selection and improved upload process ( #91 )
2025-11-18 20:48:28 +08:00
Dallas98
04a233b803
fix: 修复知识库问题 ( #89 )
...
* feat: Refactor system parameter management with new data structure and update logic
* feat: Enhance dataset file management with improved file copying
* feat: Enhance dataset file management with improved file copying
* fix: 修复知识库相关问题
* feat: Integrate Milvus service for enhanced knowledge base management and file deletion
2025-11-17 19:11:04 +08:00
Dallas98
145c154d1f
feat: Integrate Milvus service for enhanced knowledge base management and file deletion ( #88 )
...
* feat: Refactor system parameter management with new data structure and update logic
* fix: 修复知识库相关问题
2025-11-17 17:36:09 +08:00
Dallas98
e300d13c21
feat: Enhance dataset file management with improved file copying
2025-11-14 23:30:28 +08:00
Dallas98
5638bdcf1c
feat: add file copying functionality to dataset directory and update base path configuration
2025-11-14 18:05:40 +08:00
hhhhsc701
5cef9cb273
feature: deer-flow支持从datamate获取外部接入模型 ( #83 )
...
* feature: deer-flow支持从datamate获取外部接入模型
2025-11-13 20:13:16 +08:00
Dallas98
15498f27cf
feat: add file copying functionality to dataset directory and update base path configuration #80
2025-11-13 16:52:14 +08:00
hhhhsc701
6bbde0ec56
feature: 清洗任务详情页 ( #73 )
...
* feature: 清洗任务详情
* fix: 取消构建镜像，改为直接拉取
* fix: 增加清洗任务详情页
* fix: 增加清洗任务详情页
* fix: 算子列表可点击
* fix: 模板详情和更新
2025-11-12 18:00:19 +08:00
Vincent
2b09c7dfd1
feature：mysql数据库归集为csv文件 ( #76 )
...
* fix：配比任务需要能够跳转到目标数据集
* feature：增加配比任务详情接口
* fix：删除不存在的配比详情页面
* fix：使用正式的逻辑来展示标签
* fix：参数默认值去掉多余的-
* fix：修复配比任务相关操作
* fix：去除不需要的日志打印和import
* feature：数据归集创建时将obs、mysql归集也放出
* refactor：重构数据归集的代码
* refactor：重构数据归集的代码
* feature：增加实现mysql归集为csv文件
2025-11-12 17:05:31 +08:00
Vincent
b8d7aca8b7
refactor：重构数据归集部分代码 ( #75 )
...
* fix：配比任务需要能够跳转到目标数据集
* feature：增加配比任务详情接口
* fix：删除不存在的配比详情页面
* fix：使用正式的逻辑来展示标签
* fix：参数默认值去掉多余的-
* fix：修复配比任务相关操作
* fix：去除不需要的日志打印和import
* feature：数据归集创建时将obs、mysql归集也放出
* refactor：重构数据归集的代码
* refactor：重构数据归集的代码
2025-11-12 09:34:50 +08:00
Dallas98
aa01f52535
合并拉取请求 #74
...
* feat: Implement system parameter management with Redis integration
2025-11-11 22:13:14 +08:00
Vincent
60e2289019
fix：修复配比任务操作问题 ( #66 )
...
* fix：配比任务需要能够跳转到目标数据集
* feature：增加配比任务详情接口
* fix：删除不存在的配比详情页面
* fix：使用正式的逻辑来展示标签
* fix：参数默认值去掉多余的-
* fix：修复配比任务相关操作
2025-11-07 19:01:45 +08:00
hhhhsc701
2138ba23c7
feature: 增加算子详情页；优化算子上传更新逻辑 ( #64 )
...
* feature: 增加算子详情页；优化算子上传更新逻辑
2025-11-07 16:54:00 +08:00
hhhhsc701
05b26a2981
feature: 更新算子名称；增加创建任务、模板校验 ( #57 )
...
* feature: 更新算子名称；增加创建任务、模板校验
* feature: 镜像构建增加缓存
2025-11-05 17:38:03 +08:00
hhhhsc701
f3958f08d9
feature: 对接deer-flow ( #54 )
...
feature: 对接deer-flow
2025-11-04 20:30:40 +08:00
Dallas98
dc30b0d892
feat: update file deletion logic to accept multiple file IDs ( #53 )
...
* feat: update file deletion logic to accept multiple file IDs
2025-11-03 15:00:37 +08:00
hefanli
08bd4eca5c
feature：增加数据配比功能 ( #52 )
...
* refactor: 修改调整数据归集实现，删除无用代码，优化代码结构
* feature: 每天凌晨00：00扫描所有数据集，检查数据集是否超过了预设的保留天数，超出保留天数的数据集调用删除接口进行删除
* fix: 修改删除数据集文件的逻辑，上传到数据集中的文件会同时删除数据库中的记录和文件系统中的文件，归集过来的文件仅删除数据库中的记录
* fix: 增加参数校验和接口定义，删除不使用的接口
* fix: 数据集统计数据默认为0
* feature: 数据集状态增加流转，创建时为草稿状态，上传文件或者归集文件后修改为活动状态
* refactor: 修改分页查询归集任务的代码
* fix: 更新后重新执行；归集任务执行增加事务控制
* feature: 创建归集任务时能够同步创建数据集，更新归集任务时能更新到指定数据集
* fix: 创建归集任务不需要创建数据集时不应该报错
* fix: 修复删除文件时数据集的统计数据不变动
* feature: 查询数据集详情时能够获取到文件标签分布
* fix: tags为空时不进行分析
* fix: 状态修改为ACTIVE
* fix: 修改解析tag的方法
* feature: 实现创建、分页查询、删除配比任务
* feature: 实现创建、分页查询、删除配比任务的前端交互
* fix: 修复进度计算异常导致的页面报错
2025-11-03 10:17:39 +08:00
Dallas98
e854a0288a
feat: update knowledge base processing to use KnowledgeBase object and enhance configuration ( #46 )
...
* feat: update knowledge base processing to use KnowledgeBase object and enhance configuration
2025-10-31 13:16:05 +08:00
hhhhsc701
b9b97c1ac2
Develop op ( #35 )
...
* refactor: enhance CleaningTaskService and related components with validation and repository updates
* feature: 支持算子上传创建
2025-10-30 17:17:00 +08:00
Dallas98
8d2b41ed94
feature: Implement the basic knowledge generation function ( #40 )
2025-10-30 16:50:54 +08:00
Dallas98
3f484e988d
feat: increase api_key length and enhance ModelConfig annotations ( #32 )
...
* refactor: rename artifactId and application name to 'datamate'; add model configuration and related services
* refactor: simplify package scanning by using wildcard for mapper packages
* feat: add model health check functionality and improve model configuration
* feat: increase api_key length and enhance ModelConfig annotations
2025-10-28 17:30:26 +08:00
hhhhsc701
67eb571d8d
feature: 对接deer-flow ( #27 )
...
feature: 对接deer-flow
2025-10-28 16:28:26 +08:00
Dallas98
1a6e25758e
feat: add model health check functionality and improve model configuration ( #30 )
...
* refactor: rename artifactId and application name to 'datamate'; add model configuration and related services
* refactor: simplify package scanning by using wildcard for mapper packages
* feat: add model health check functionality and improve model configuration
2025-10-28 16:06:53 +08:00
Dallas98
a4b5238621
refactor: simplify package scanning by using wildcard for mapper packages ( #28 )
...
* refactor: rename artifactId and application name to 'datamate'; add model configuration and related services
* refactor: simplify package scanning by using wildcard for mapper packages
2025-10-28 14:12:44 +08:00
hhhhsc
41e7e684c3
Merge branch 'main' into develop_deer
2025-10-28 11:03:01 +08:00
hhhhsc
a69b9f4921
feature: 对接deer-flow
2025-10-28 10:54:29 +08:00
Dallas98
f54afddbeb
refactor: rename artifactId and application name to 'datamate'; add model configuration and related services ( #26 )
2025-10-28 10:39:26 +08:00
hefanli
46dfb389f1
feature：增加定时清除超出保留期限数据集的功能；增加数据归集任务绑定数据集的接口 ( #24 )
...
* refactor: 修改调整数据归集实现，删除无用代码，优化代码结构
* feature: 每天凌晨00：00扫描所有数据集，检查数据集是否超过了预设的保留天数，超出保留天数的数据集调用删除接口进行删除
* fix: 修改删除数据集文件的逻辑，上传到数据集中的文件会同时删除数据库中的记录和文件系统中的文件，归集过来的文件仅删除数据库中的记录
* fix: 增加参数校验和接口定义，删除不使用的接口
* fix: 数据集统计数据默认为0
* feature: 数据集状态增加流转，创建时为草稿状态，上传文件或者归集文件后修改为活动状态
* refactor: 修改分页查询归集任务的代码
* fix: 更新后重新执行；归集任务执行增加事务控制
* feature: 创建归集任务时能够同步创建数据集，更新归集任务时能更新到指定数据集
2025-10-25 15:59:36 +08:00
hhhhsc
abc26c2c0e
refactor: update service and repository structure to use DTOs and improve clarity
2025-10-24 17:55:41 +08:00
hhhhsc701
f9dbefd737
Merge pull request #21 from ModelEngine-Group/develop_db
...
refactor: rename and reorganize data models and repositories for clarity
2025-10-24 15:46:32 +08:00
hhhhsc
2d2419205a
refactor: rename and reorganize data models and repositories for clarity
2025-10-24 15:33:46 +08:00
hefanli
cc072bbf90
refactor: 修改调整数据归集实现，删除无用代码，优化代码结构 ( #20 )
2025-10-23 21:10:57 +08:00
