Jerry Yan
8b1ab8ff36
实现功能：
- 实现 GraphSyncService（同步编排器）
- 实现 GraphSyncStepService（同步步骤执行器）
- 实现 GraphSyncController（同步 API）
- 实现 GraphInitializer（图谱初始化）
- 实现 DataManagementClient（数据源客户端）
同步功能：
- syncDatasets：同步数据集实体
- syncFields：同步字段实体
- syncUsers：同步用户实体
- syncOrgs：同步组织实体
- buildHasFieldRelations：构建 HAS_FIELD 关系
- buildDerivedFromRelations：构建 DERIVED_FROM 关系
- buildBelongsToRelations：构建 BELONGS_TO 关系
- syncAll：全量同步（实体 + 关系 + 对账删除）
API 端点：
- POST /{graphId}/sync/full：全量同步
- POST /{graphId}/sync/datasets：同步数据集
- POST /{graphId}/sync/fields：同步字段
- POST /{graphId}/sync/users：同步用户
- POST /{graphId}/sync/orgs：同步组织
- POST /{graphId}/sync/relations/has-field：构建 HAS_FIELD
- POST /{graphId}/sync/relations/derived-from：构建 DERIVED_FROM
- POST /{graphId}/sync/relations/belongs-to：构建 BELONGS_TO
技术实现：
- Upsert 策略：
- 实体：两阶段（Cypher MERGE 原子创建 + SDN save 更新扩展属性）
- 关系：Cypher MERGE 幂等创建
- 全量对账删除：purgeStaleEntities() 删除 MySQL 中已删除的实体
- 并发安全：
- 图级互斥锁（ConcurrentHashMap<String, ReentrantLock>）
- 复合唯一约束（graph_id, source_id, type）
- 锁自动回收（releaseLock() 原子检查并移除空闲锁）
- 重试机制：HTTP 调用失败时按指数退避重试（默认 3 次）
- 错误处理：
- 逐条错误处理（单条失败不影响其他记录）
- 统一异常包装（BusinessException.of(SYNC_FAILED)）
- 错误信息脱敏（仅返回 errorCount + syncId）
- 事务管理：
- GraphSyncService（编排器，无事务）
- GraphSyncStepService（步骤执行器，@Transactional）
- 性能优化：
- 全量同步共享数据快照
- 批量日志跟踪
- 图谱初始化：
- 1 个唯一性约束（entity ID）
- 1 个复合唯一约束（graph_id, source_id, type）
- 9 个索引（5 个单字段 + 3 个复合 + 1 个全文）
- 幂等性保证（IF NOT EXISTS）
代码审查：
- 经过 3 轮 Codex 审查和 2 轮 Claude 修复
- 所有问题已解决（3个P0 + 5个P1 + 3个P2 + 1个P3）
- 编译验证通过（mvn compile SUCCESS）
设计决策：
- 最终一致性：允许短暂的数据不一致
- 对账机制：定期对比并修复差异
- 信任边界：网关负责鉴权，服务层只做格式校验
- 多实例部署：依赖复合唯一约束兜底
DataMate All-in-One Data Work Platform
DataMate is an enterprise-level data processing platform for model fine-tuning and RAG retrieval, supporting core functions such as data collection, data management, operator marketplace, data cleaning, data synthesis, data annotation, data evaluation, and knowledge generation.
If you like this project, please give it a Star⭐️!
🌟 Core Features
- Core Modules: Data Collection, Data Management, Operator Marketplace, Data Cleaning, Data Synthesis, Data Annotation, Data Evaluation, Knowledge Generation.
- Visual Orchestration: Drag-and-drop data processing workflow design.
- Operator Ecosystem: Rich built-in operators and support for custom operators.
🚀 Quick Start
Prerequisites
- Git (for pulling source code)
- Make (for building and installing)
- Docker (for building images and deploying services)
- Docker-Compose (for service deployment - Docker method)
- Kubernetes (for service deployment - k8s method)
- Helm (for service deployment - k8s method)
This project supports deployment via two methods: docker-compose and helm. After executing the command, please enter the corresponding number for the deployment method. The command echo is as follows:
Choose a deployment method:
1. Docker/Docker-Compose
2. Kubernetes/Helm
Enter choice:
Clone the Code
git clone git@github.com:ModelEngine-Group/DataMate.git
cd DataMate
Deploy the basic services
make install
If the machine you are using does not have make installed, please run the following command to deploy it:
# Windows
set REGISTRY=ghcr.io/modelengine-group/
docker compose -f ./deployment/docker/datamate/docker-compose.yml up -d
docker compose -f ./deployment/docker/milvus/docker-compose.yml up -d
# Linux/Mac
export REGISTRY=ghcr.io/modelengine-group/
docker compose -f ./deployment/docker/datamate/docker-compose.yml up -d
docker compose -f ./deployment/docker/milvus/docker-compose.yml up -d
Once the container is running, access http://localhost:30000 in a browser to view the front-end interface.
To list all available Make targets, flags and help text, run:
make help
Build and deploy Mineru Enhanced PDF Processing
make build-mineru
make install-mineru
Deploy the DeerFlow service
make install-deer-flow
Local Development and Deployment
After modifying the local code, please execute the following commands to build the image and deploy using the local image.
make build
make install dev=true
Uninstall
make uninstall
When running make uninstall, the installer will prompt once whether to delete volumes; that single choice is applied to all components. The uninstall order is: milvus -> label-studio -> datamate, which ensures the datamate network is removed cleanly after services that use it have stopped.
🤝 Contribution Guidelines
Thank you for your interest in this project! We warmly welcome contributions from the community. Whether it's submitting bug reports, suggesting new features, or directly participating in code development, all forms of help make the project better.
• 📮 GitHub Issues: Submit bugs or feature suggestions.
• 🔧 GitHub Pull Requests: Contribute code improvements.
📄 License
DataMate is open source under the MIT license. You are free to use, modify, and distribute the code of this project in compliance with the license terms.