q792602257/DataMate
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
565 Commits 1 Branch 0 Tags
a260134d7c3d02104007304e8e77075fdafe3fa7
Commit Graph

11 Commits

Author SHA1 Message Date
Jerry Yan
0e0782a452 feat(kg-extraction): 实现 Python 抽取器 FastAPI 接口 
实现功能：
- 创建 kg_extraction/interface.py（FastAPI 路由）
- 实现 POST /api/kg/extract（单条文本抽取）
- 实现 POST /api/kg/extract/batch（批量抽取，最多 50 条）
- 集成到 FastAPI 主路由（/api/kg/ 前缀）

技术实现：
- 配置管理：从环境变量读取 LLM 配置（API Key、Base URL、Model、Temperature）
- 安全性：
  - API Key 使用 SecretStr 保护
  - 错误信息脱敏（使用 trace_id，不暴露原始异常）
  - 请求文本不写入日志（使用 SHA-256 hash）
  - 强制要求 X-User-Id 头（鉴权边界）
- 超时控制：
  - kg_llm_timeout_seconds（60秒）
  - kg_llm_max_retries（2次）
- 输入校验：
  - graph_id 和 source_id 使用 UUID pattern
  - source_type 使用 Enum（4个值）
  - allowed_nodes/relationships 元素使用正则约束（ASCII，1-50字符）
- 审计日志：记录 caller、trace_id、text_hash

代码审查：
- 经过 3 轮 Codex 审查和 2 轮 Claude 修复
- 所有问题已解决（5个 P1/P2 + 3个 P3）
- 语法检查通过

API 端点：
- POST /api/kg/extract：单条文本抽取
- POST /api/kg/extract/batch：批量抽取（最多 50 条）

配置环境变量：
- KG_LLM_API_KEY：LLM API 密钥
- KG_LLM_BASE_URL：自定义端点（可选）
- KG_LLM_MODEL：模型名称（默认 gpt-4o-mini）
- KG_LLM_TEMPERATURE：生成温度（默认 0.0）
- KG_LLM_TIMEOUT_SECONDS：超时时间（默认 60）
- KG_LLM_MAX_RETRIES：重试次数（默认 2）
 2026-02-17 22:01:06 +08:00
Jerry Yan
7092c3f955 feat(annotation): 调整文本编辑器大小限制配置 
- 将editor_max_text_bytes默认值从2MB改为0，表示不限制
- 更新文本获取服务中的大小检查逻辑，只在max_bytes大于0时进行限制
- 修改错误提示信息中的字节限制显示
- 优化配置参数的条件判断流程
 2026-02-02 17:53:09 +08:00
Jerry Yan
d5b75fee0d LSF 2026-01-07 00:00:16 +08:00
hhhhsc701
d82bff441a fix: prevent deletion of predefined operators and improve error handling (#192) 
* fix: prevent deletion of predefined operators and improve error handling

* fix: prevent deletion of predefined operators and improve error handling
 2025-12-22 19:30:41 +08:00
hefanli
1d19cd3a62 feature: add data-evaluation 
* feature: add evaluation task management function

* feature: add evaluation task detail page

* fix: delete duplicate definition for table t_model_config

* refactor: rename package synthesis to ratio

* refactor: add eval file table and  refactor related code

* fix: calling large models in parallel during evaluation
 2025-12-04 09:23:54 +08:00
Dallas98
8b164cb012 feat: Implement data synthesis task management with database models and API endpoints (#122) 2025-12-02 15:23:58 +08:00
Jason Wang
78f50ea520 feat: File and Annotation 2-way sync implementation (#63) 
* feat: Refactor configuration and sync logic for improved dataset handling and logging

* feat: Enhance annotation synchronization and dataset file management

- Added new fields `tags_updated_at` to `DatasetFiles` model for tracking the last update time of tags.
- Implemented new asynchronous methods in the Label Studio client for fetching, creating, updating, and deleting task annotations.
- Introduced bidirectional synchronization for annotations between DataMate and Label Studio, allowing for flexible data management.
- Updated sync service to handle annotation conflicts based on timestamps, ensuring data integrity during synchronization.
- Enhanced dataset file response model to include tags and their update timestamps.
- Modified database initialization script to create a new column for `tags_updated_at` in the dataset files table.
- Updated requirements to ensure compatibility with the latest dependencies.
 2025-11-07 15:03:07 +08:00
Jason Wang
b5fe787c20 feat: Labeling Frontend adaptations + Backend build and deploy + Logging improvement (#55) 
* feat: Front-end data annotation page adaptation to the backend API.

* feat: Implement labeling configuration editor and enhance annotation task creation form

* feat: add python backend build and deployment; add backend configuration for Label Studio integration and improve logging setup

* refactor: remove duplicate log configuration
 2025-11-05 01:55:53 +08:00
Jason Wang
2f7341dc1f refactor: Reorganize datamate-python (#34) 
refactor: Reorganize datamate-python (previously label-studio-adapter) into a DDD style structure.
 2025-10-30 01:32:59 +08:00
hhhhsc
41e7e684c3 Merge branch 'main' into develop_deer 2025-10-28 11:03:01 +08:00
Jinglong Wang
7f819563db Develop labeling module (#25) 
* refactor: remove db table management from LS adapter (mv to scripts later); change adapter to use the same MySQL DB as other modules.

* refactor: Rename LS Adapter module to datamate-python
 2025-10-27 16:16:14 +08:00