hhhhsc701
d82bff441a
fix: prevent deletion of predefined operators and improve error handling ( #192 )
...
* fix: prevent deletion of predefined operators and improve error handling
* fix: prevent deletion of predefined operators and improve error handling
2025-12-22 19:30:41 +08:00
hhhhsc701
46f4a8c219
feat: add download functionality for example operator and update Dock… ( #188 )
...
* feat: add download functionality for example operator and update Dockerfile
* feat: enhance download response by exposing content disposition header
* feat: update download function to accept filename parameter for example operator
2025-12-22 15:39:32 +08:00
hhhhsc701
ab4523b556
add export type settings and enhance metadata structure ( #181 )
...
* fix(session): enhance database connection settings with pool pre-ping and recycle options
* feat(metadata): add export type settings and enhance metadata structure
* fix(base_op): improve sample handling by introducing target_type key and consolidating text/data retrieval logic
* feat(metadata): add export type settings and enhance metadata structure
* feat(metadata): add export type settings and enhance metadata structure
2025-12-19 11:54:08 +08:00
hhhhsc701
be875086db
feat: add operator-packages-volume to docker-compose and update Docke… ( #179 )
...
* feat: add operator-packages-volume to docker-compose and update Dockerfile for site-packages path
* feat: add retry
2025-12-18 20:32:42 +08:00
hhhhsc701
761f7f6a51
fix: optimize PDF parsing by implementing concurrent processing with … ( #177 )
...
* fix: optimize PDF parsing by implementing concurrent processing with ThreadPoolExecutor
* Refactor to async processing for file extraction
Refactor the file processing to use asyncio for improved performance and concurrency.
2025-12-18 15:28:30 +08:00
hhhhsc701
924d977d6f
支持mineru npu处理 ( #174 )
...
* feature: unstructured支持简单pdf处理
* feature: update values.yaml to enhance ray-cluster configuration with security context, environment variables, and resource limits
* feature: update deploy.yaml and process.py for mineru server configuration and PDF processing enhancements
* feature: update deploy.yaml and process.py for mineru server configuration and PDF processing enhancements
* feature: improve PDF processing logic and update dependencies in process.py and pyproject.toml
* feature: improve PDF processing logic and update dependencies in process.py and pyproject.toml
* feature: update Dockerfile for improved package source mirrors and add mineru-npu to build targets
2025-12-17 16:31:06 +08:00
hhhhsc701
d8c0b0ed73
补充modal范围 ( #165 )
2025-12-12 13:34:03 +08:00
hhhhsc701
19a04df276
feature: 增加水印去除/高级匿名化算子 ( #151 )
...
* feature: 增加水印去除算子
* feature: clean code
* feature: clean code
* feature: 增加高级匿名化算子
2025-12-10 18:12:47 +08:00
hhhhsc701
d59c167da4
算子将抽取与落盘固定到流程中 ( #134 )
...
* feature: 将抽取动作移到每一个算子中
* feature: 落盘算子改为默认执行
* feature: 优化前端展示
* feature: 使用pyproject管理依赖
2025-12-05 17:26:29 +08:00
hhhhsc701
265e284fb8
feature: 修改算子开发指南 ( #127 )
2025-12-03 17:45:08 +08:00
hhhhsc701
07029d07ff
优化清洗重试机制，优化清洗进度展示，修复模板无法展示参数 ( #113 )
...
* bugfix: 模板无法展示参数
* bugfix: 优化清洗进度展示
* bugfix: 优化清洗重试机制
2025-11-28 15:28:10 +08:00
hhhhsc701
f1bffdcd61
bugfix: 创建清洗任务时修改数据集状态；无法删除已在模板/运行任务的算子
...
* bugfix: 创建清洗任务时修改数据集状态；无法删除已在模板/运行任务的算子
2025-11-27 17:34:53 +08:00
hhhhsc701
5cef9cb273
feature: deer-flow支持从datamate获取外部接入模型 ( #83 )
...
* feature: deer-flow支持从datamate获取外部接入模型
2025-11-13 20:13:16 +08:00
hhhhsc701
f78475e29f
Develop hsc ( #58 )
...
feature: 优化镜像构建/部署
2025-11-06 17:14:54 +08:00
hhhhsc701
05b26a2981
feature: 更新算子名称；增加创建任务、模板校验 ( #57 )
...
* feature: 更新算子名称；增加创建任务、模板校验
* feature: 镜像构建增加缓存
2025-11-05 17:38:03 +08:00
Startalker
a600c1d793
feature: modify UnstructuredFormatter and ExternalPDFFormatter description ( #44 )
...
* feature: add UnstructuredFormatter
* feature: add UnstructuredFormatter in db
* feature: add unstructured[docx]==0.18.15
* feature: support doc
* feature: add mineru
* feature: add external pdf extract operator by using mineru
* feature: mineru docker install bugfix
* feature: add unstructured xlsx/xls/csv/pptx/ppt
* feature: modify UnstructuredFormatter and ExternalPDFFormatter description
---------
Co-authored-by: Startalker <438747480@qq.com >
2025-10-31 10:32:14 +08:00
Startalker
06b05a65a9
feature: add unstructured xlsx/xls/csv/pptx/ppt ( #41 )
...
* feature: add UnstructuredFormatter
* feature: add UnstructuredFormatter in db
* feature: add unstructured[docx]==0.18.15
* feature: support doc
* feature: add mineru
* feature: add external pdf extract operator by using mineru
* feature: mineru docker install bugfix
* feature: add unstructured xlsx/xls/csv/pptx/ppt
---------
Co-authored-by: Startalker <438747480@qq.com >
2025-10-30 20:21:12 +08:00
Startalker
155603b1ca
feature: add external pdf extract operator by using mineru ( #36 )
...
* feature: add UnstructuredFormatter
* feature: add UnstructuredFormatter in db
* feature: add unstructured[docx]==0.18.15
* feature: support doc
* feature: add mineru
* feature: add external pdf extract operator by using mineru
* feature: mineru docker install bugfix
---------
Co-authored-by: Startalker <438747480@qq.com >
2025-10-30 15:55:10 +08:00
Startalker
f86d4fae25
feature: add unstructured formatter operator for doc/docx ( #17 )
...
* feature: add UnstructuredFormatter
* feature: add UnstructuredFormatter in db
* feature: add unstructured[docx]==0.18.15
* feature: support doc
---------
Co-authored-by: Startalker <438747480@qq.com >
2025-10-23 16:49:03 +08:00
hhhhsc701
31ef8bc265
[Feature] Refactor project to use 'datamate' naming convention for services and configurations ( #14 )
...
* Enhance CleaningTaskService to track cleaning process progress and update ExecutorType to DATAMATE
* Refactor project to use 'datamate' naming convention for services and configurations
2025-10-22 17:53:16 +08:00
Dallas98
1c97afed7d
init datamate
2025-10-21 23:00:48 +08:00
