Jerry Yan
37b478a052
修复内容： P1 级别（关键）： 1. 数据隔离漏洞：邻居查询添加 graph_id 路径约束，防止跨图谱数据泄漏 2. 空快照误删风险：添加 allowPurgeOnEmptySnapshot 保护开关（默认 false） 3. 弱默认凭据：启动自检，生产环境检测到默认密码直接拒绝启动 P2 级别（重要）： 4. 配置校验：importBatchSize 添加 @Min(1) 验证，启动时 fail-fast 5. N+1 性能：重写 upsertEntity 为单条 Cypher 查询（从 3 条优化到 1 条） 6. 服务认证：添加 mTLS/JWT 文档说明 7. 错误处理：改进 Schema 初始化和序列化错误处理 测试覆盖： - 新增 69 个单元测试，全部通过 - GraphEntityServiceTest: 13 个测试（CRUD、验证、分页） - GraphRelationServiceTest: 13 个测试（CRUD、方向验证） - GraphSyncServiceTest: 5 个测试（验证、全量同步） - GraphSyncStepServiceTest: 14 个测试（空快照保护、N+1 验证） - GraphQueryServiceTest: 13 个测试（邻居/路径/子图/搜索） - GraphInitializerTest: 11 个测试（凭据验证、Schema 初始化） 技术细节： - 数据隔离：使用 ALL() 函数约束路径中所有节点和关系的 graph_id - 空快照保护：新增配置项 allow-purge-on-empty-snapshot 和错误码 EMPTY_SNAPSHOT_PURGE_BLOCKED - 凭据检查：Java 和 Python 双端实现，根据环境（dev/test/prod）采取不同策略 - 性能优化：使用 SDN 复合属性格式（properties.key）在 MERGE 中直接设置属性 - 属性安全：使用白名单 [a-zA-Z0-9_] 防止 Cypher 注入 代码变更：+210 行，-29 行
DataMate All-in-One Data Work Platform
DataMate is an enterprise-level data processing platform for model fine-tuning and RAG retrieval, supporting core functions such as data collection, data management, operator marketplace, data cleaning, data synthesis, data annotation, data evaluation, and knowledge generation.
If you like this project, please give it a Star⭐️!
🌟 Core Features
- Core Modules: Data Collection, Data Management, Operator Marketplace, Data Cleaning, Data Synthesis, Data Annotation, Data Evaluation, Knowledge Generation.
- Visual Orchestration: Drag-and-drop data processing workflow design.
- Operator Ecosystem: Rich built-in operators and support for custom operators.
🚀 Quick Start
Prerequisites
- Git (for pulling source code)
- Make (for building and installing)
- Docker (for building images and deploying services)
- Docker-Compose (for service deployment - Docker method)
- Kubernetes (for service deployment - k8s method)
- Helm (for service deployment - k8s method)
This project supports deployment via two methods: docker-compose and helm. After executing the command, please enter the corresponding number for the deployment method. The command echo is as follows:
Choose a deployment method:
1. Docker/Docker-Compose
2. Kubernetes/Helm
Enter choice:
Clone the Code
git clone git@github.com:ModelEngine-Group/DataMate.git
cd DataMate
Deploy the basic services
make install
If the machine you are using does not have make installed, please run the following command to deploy it:
# Windows
set REGISTRY=ghcr.io/modelengine-group/
docker compose -f ./deployment/docker/datamate/docker-compose.yml up -d
docker compose -f ./deployment/docker/milvus/docker-compose.yml up -d
# Linux/Mac
export REGISTRY=ghcr.io/modelengine-group/
docker compose -f ./deployment/docker/datamate/docker-compose.yml up -d
docker compose -f ./deployment/docker/milvus/docker-compose.yml up -d
Once the container is running, access http://localhost:30000 in a browser to view the front-end interface.
To list all available Make targets, flags and help text, run:
make help
Build and deploy Mineru Enhanced PDF Processing
make build-mineru
make install-mineru
Deploy the DeerFlow service
make install-deer-flow
Local Development and Deployment
After modifying the local code, please execute the following commands to build the image and deploy using the local image.
make build
make install dev=true
Uninstall
make uninstall
When running make uninstall, the installer will prompt once whether to delete volumes; that single choice is applied to all components. The uninstall order is: milvus -> label-studio -> datamate, which ensures the datamate network is removed cleanly after services that use it have stopped.
🤝 Contribution Guidelines
Thank you for your interest in this project! We warmly welcome contributions from the community. Whether it's submitting bug reports, suggesting new features, or directly participating in code development, all forms of help make the project better.
• 📮 GitHub Issues: Submit bugs or feature suggestions.
• 🔧 GitHub Pull Requests: Contribute code improvements.
📄 License
DataMate is open source under the MIT license. You are free to use, modify, and distribute the code of this project in compliance with the license terms.