Jerry Yan
75db6daeb5
新增功能：
- 查询阶段权限过滤：管理员看全部，普通用户只看自己创建的数据
- 结构实体（User、Org、Field）对所有用户可见
- 业务实体（Dataset、Workflow、Job、LabelTask、KnowledgeSet）按 created_by 过滤
- CONFIDENTIAL 敏感度过滤：需要特定权限才能查看
安全修复（四轮迭代）：
P1-1: CONFIDENTIAL 敏感度过滤
- 4 个查询入口统一计算 excludeConfidential
- assertEntityAccess / isEntityAccessible 新增保密数据检查
- buildPermissionPredicate 在 Cypher 中追加 sensitivity 条件
P1-2: 结构实体按类型白名单判定
- 新增常量 STRUCTURAL_ENTITY_TYPES = Set.of("User", "Org", "Field")
- 业务实体必须匹配 created_by（缺失则拒绝）
- Cypher 从 IS NULL OR 改为 type IN ['User', 'Org', 'Field'] OR
P2-1: getNeighborGraph 路径级权限旁路
- 改为 ALL(n IN nodes(p) WHERE ...) 路径全节点过滤
- 与 getShortestPath 保持一致
P2-2: CONFIDENTIAL 大小写归一化
- Cypher 用 toUpper(trim(...)) 比较
- Java 用 equalsIgnoreCase
- 与 data-management-service 保持一致
权限模型：
- 同步阶段：全量同步（保持图谱完整性）
- 查询阶段：根据用户权限过滤结果
- 使用 RequestUserContextHolder 和 ResourceAccessService
代码变更：+642 行，-32 行
测试结果：130 tests, 0 failures
新增 9 个测试用例
已知 P3 问题（非阻断，可后续优化）：
- 组件扫描范围偏大
- 测试质量可进一步增强
- 结构实体白名单重复维护
DataMate All-in-One Data Work Platform
DataMate is an enterprise-level data processing platform for model fine-tuning and RAG retrieval, supporting core functions such as data collection, data management, operator marketplace, data cleaning, data synthesis, data annotation, data evaluation, and knowledge generation.
If you like this project, please give it a Star⭐️!
🌟 Core Features
- Core Modules: Data Collection, Data Management, Operator Marketplace, Data Cleaning, Data Synthesis, Data Annotation, Data Evaluation, Knowledge Generation.
- Visual Orchestration: Drag-and-drop data processing workflow design.
- Operator Ecosystem: Rich built-in operators and support for custom operators.
🚀 Quick Start
Prerequisites
- Git (for pulling source code)
- Make (for building and installing)
- Docker (for building images and deploying services)
- Docker-Compose (for service deployment - Docker method)
- Kubernetes (for service deployment - k8s method)
- Helm (for service deployment - k8s method)
This project supports deployment via two methods: docker-compose and helm. After executing the command, please enter the corresponding number for the deployment method. The command echo is as follows:
Choose a deployment method:
1. Docker/Docker-Compose
2. Kubernetes/Helm
Enter choice:
Clone the Code
git clone git@github.com:ModelEngine-Group/DataMate.git
cd DataMate
Deploy the basic services
make install
If the machine you are using does not have make installed, please run the following command to deploy it:
# Windows
set REGISTRY=ghcr.io/modelengine-group/
docker compose -f ./deployment/docker/datamate/docker-compose.yml up -d
docker compose -f ./deployment/docker/milvus/docker-compose.yml up -d
# Linux/Mac
export REGISTRY=ghcr.io/modelengine-group/
docker compose -f ./deployment/docker/datamate/docker-compose.yml up -d
docker compose -f ./deployment/docker/milvus/docker-compose.yml up -d
Once the container is running, access http://localhost:30000 in a browser to view the front-end interface.
To list all available Make targets, flags and help text, run:
make help
Build and deploy Mineru Enhanced PDF Processing
make build-mineru
make install-mineru
Deploy the DeerFlow service
make install-deer-flow
Local Development and Deployment
After modifying the local code, please execute the following commands to build the image and deploy using the local image.
make build
make install dev=true
Uninstall
make uninstall
When running make uninstall, the installer will prompt once whether to delete volumes; that single choice is applied to all components. The uninstall order is: milvus -> label-studio -> datamate, which ensures the datamate network is removed cleanly after services that use it have stopped.
🤝 Contribution Guidelines
Thank you for your interest in this project! We warmly welcome contributions from the community. Whether it's submitting bug reports, suggesting new features, or directly participating in code development, all forms of help make the project better.
• 📮 GitHub Issues: Submit bugs or feature suggestions.
• 🔧 GitHub Pull Requests: Contribute code improvements.
📄 License
DataMate is open source under the MIT license. You are free to use, modify, and distribute the code of this project in compliance with the license terms.