* feat(synthesis): add evaluation task creation functionality and UI enhancements
* feat(synthesis): implement synthesis data management features including loading, editing, and deleting
* feat(synthesis): add endpoints for deleting and updating synthesis data and chunks
* fix: Correctly extract file values from selectedFilesMap in AddDataDialog
* feature: add cot data evaluation function
* fix: added verification to evaluation results
* fix: fix the prompt for evaluating
* fix: 修复当评估结果为空导致读取失败的问题
* feat: Implement data synthesis task management with database models and API endpoints
* feat: Update Python version requirements and refine dependency constraints in configuration
* fix: Correctly extract file values from selectedFilesMap in AddDataDialog
* feat: Refactor synthesis task routes and enhance file task management in the API
* feat: Enhance SynthesisTaskTab with tooltip actions and add chunk data retrieval in API
* feat: Implement data synthesis task management with database models and API endpoints
* feat: Update Python version requirements and refine dependency constraints in configuration
* fix: Correctly extract file values from selectedFilesMap in AddDataDialog
* feat: Implement DatasetFileTransfer component for file selection and management
* feat: Add pagination support to file list in Overview component
* feat: add DatasetFileTransfer and TagManagement components
- Added DatasetFileTransfer component for managing dataset files.
- Introduced TagManagement component for handling tags.
- Integrated Redux for state management with auth and settings slices.
- Updated package.json to include @reduxjs/toolkit and react-redux dependencies.
- Refactored existing components to utilize new DatasetFileTransfer and TagManagement components.
- Implemented hooks for typed dispatch and selector in Redux.
- Enhanced CreateKnowledgeBase and SynthesisTask components to support new features.
* feat: Refactor AddDataDialog and introduce DatasetFileTransfer component for improved file selection and management
* feat: Refactor SynthesisTask and InstructionTemplate components for improved UI and functionality; integrate DatasetFileTransfer for file management
* feat: Enhance CollectionTaskCreate form with additional fields for MYSQL configuration and prefix input
* feat: Refactor system parameter management with new data structure and update logic
* feat: Enhance dataset file management with improved file copying
* feat: Enhance dataset file management with improved file copying
* fix: 修复知识库相关问题
* feat: Integrate Milvus service for enhanced knowledge base management and file deletion
* feat: Enhance annotation module with template management and validation
- Added DatasetMappingCreateRequest and DatasetMappingUpdateRequest schemas to handle dataset mapping requests with camelCase and snake_case support.
- Introduced Annotation Template schemas including CreateAnnotationTemplateRequest, UpdateAnnotationTemplateRequest, and AnnotationTemplateResponse for managing annotation templates.
- Implemented AnnotationTemplateService for creating, updating, retrieving, and deleting annotation templates, including validation of configurations and XML generation.
- Added utility class LabelStudioConfigValidator for validating Label Studio configurations and XML formats.
- Updated database schema for annotation templates and labeling projects to include new fields and constraints.
- Seeded initial annotation templates for various use cases including image classification, object detection, and text classification.
* feat: Enhance TemplateForm with improved validation and dynamic field rendering; update LabelStudio config validation for camelCase support
* feat: Update docker-compose.yml to mark datamate dataset volume and network as external
* feat: Add tag configuration management and related components
- Introduced new components for tag selection and browsing in the frontend.
- Added API endpoint to fetch tag configuration from the backend.
- Implemented tag configuration management in the backend, including loading from YAML.
- Enhanced template service to support dynamic tag rendering based on configuration.
- Updated validation utilities to incorporate tag configuration checks.
- Refactored existing code to utilize the new tag configuration structure.
* feat: Refactor LabelStudioTagConfig for improved configuration loading and validation
* feat: Update Makefile to include backend-python-docker-build in the build process
* feat: Migrate to poetry for better deps management
* Add pyyaml dependency and update Dockerfile to use Poetry for dependency management
- Added pyyaml (>=6.0.3,<7.0.0) to pyproject.toml dependencies.
- Updated Dockerfile to install Poetry and manage dependencies using it.
- Improved layer caching by copying only dependency files before the application code.
- Removed unnecessary installation of build dependencies to keep the final image size small.
* feat: Remove duplicated backend-python-docker-build target from Makefile
* fix: airflow is not ready for adding yet
* feat: update Python version to 3.12 and remove project installation step in Dockerfile
* feat: Update site name to DataMate and refine text for AI data processing
* feat: Refactor settings page and implement model access functionality
- Created a new ModelAccess component for managing model configurations.
- Removed the old Settings component and replaced it with a new SettingsPage component that integrates ModelAccess, SystemConfig, and WebhookConfig.
- Added SystemConfig component for managing system settings.
- Implemented WebhookConfig component for managing webhook configurations.
- Updated API functions for model management in settings.apis.ts.
- Adjusted routing to point to the new SettingsPage component.
* feat: Implement Data Collection Page with Task Management and Execution Log
- Created DataCollectionPage component to manage data collection tasks.
- Added TaskManagement and ExecutionLog components for task handling and logging.
- Integrated task operations including start, stop, edit, and delete functionalities.
- Implemented filtering and searching capabilities in task management.
- Introduced SimpleCronScheduler for scheduling tasks with cron expressions.
- Updated CreateTask component to utilize new scheduling and template features.
- Enhanced BasicInformation component to conditionally render fields based on visibility settings.
- Refactored ImportConfiguration component to remove NAS import section.
* feat: Update task creation API endpoint and enhance task creation form with new fields and validation
* Refactor file upload and operator management components
- Removed unnecessary console logs from file download and export functions.
- Added size property to TaskItem interface for better task management.
- Simplified TaskUpload component by utilizing useFileSliceUpload hook for file upload logic.
- Enhanced OperatorPluginCreate component to handle file uploads and parsing more efficiently.
- Updated ConfigureStep component to use Ant Design Form for better data handling and validation.
- Improved PreviewStep component to navigate back to the operator market.
- Added support for additional file types in UploadStep component.
- Implemented delete operator functionality in OperatorMarketPage with confirmation prompts.
- Cleaned up unused API functions in operator.api.ts to streamline the codebase.
- Fixed number formatting utility to handle zero values correctly.
* Refactor Knowledge Generation to Knowledge Base
- Created new API service for Knowledge Base operations including querying, creating, updating, and deleting knowledge bases and files.
- Added constants for Knowledge Base status and type mappings.
- Defined models for Knowledge Base and related files.
- Removed obsolete Knowledge Base creation and home components, replacing them with new implementations under the Knowledge Base structure.
- Updated routing to reflect the new Knowledge Base paths.
- Adjusted menu items to align with the new Knowledge Base terminology.
- Modified ModelAccess interface to include modelName and type properties.
* feat: Implement Knowledge Base Page with CRUD operations and data management
- Added KnowledgeBasePage component for displaying and managing knowledge bases.
- Integrated search and filter functionalities with SearchControls component.
- Implemented CreateKnowledgeBase component for creating and editing knowledge bases.
- Enhanced AddDataDialog for file uploads and dataset selections.
- Introduced TableTransfer component for managing data transfers between tables.
- Updated API functions for knowledge base operations, including file management.
- Refactored knowledge base model to include file status and metadata.
- Adjusted routing to point to the new KnowledgeBasePage.
* feat: enhance OperatorPluginCreate and ConfigureStep for better upload handling and UI updates
* refactor: remove unused components and clean up API logging in KnowledgeBase
* feat: update icons in various components and improve styling for better UI consistency
* fix: adjust upload step handling and improve error display in configuration step
* feat: Add RatioTransfer component for dataset selection and configuration
- Implemented RatioTransfer component to manage dataset selection and ratio configuration.
- Integrated dataset fetching with search and filter capabilities.
- Added RatioConfig component for displaying and updating selected datasets' configurations.
- Enhanced SelectDataset component with improved UI and functionality for dataset selection.
- Updated RatioTasksPage to utilize new ratio task status mapping and improved error handling for task deletion.
- Refactored ratio model and constants for better type safety and clarity.
- Changed Vite configuration to use local backend service for development.
* feat: Add .editorconfig and enhance SystemConfig with table for settings display
* feat: Enhance parameter configuration for range inputs and update default values
* feat: Update site name to DataMate and refine text for AI data processing
* Refactor file upload and operator management components
- Removed unnecessary console logs from file download and export functions.
- Added size property to TaskItem interface for better task management.
- Simplified TaskUpload component by utilizing useFileSliceUpload hook for file upload logic.
- Enhanced OperatorPluginCreate component to handle file uploads and parsing more efficiently.
- Updated ConfigureStep component to use Ant Design Form for better data handling and validation.
- Improved PreviewStep component to navigate back to the operator market.
- Added support for additional file types in UploadStep component.
- Implemented delete operator functionality in OperatorMarketPage with confirmation prompts.
- Cleaned up unused API functions in operator.api.ts to streamline the codebase.
- Fixed number formatting utility to handle zero values correctly.
* Refactor Knowledge Generation to Knowledge Base
- Created new API service for Knowledge Base operations including querying, creating, updating, and deleting knowledge bases and files.
- Added constants for Knowledge Base status and type mappings.
- Defined models for Knowledge Base and related files.
- Removed obsolete Knowledge Base creation and home components, replacing them with new implementations under the Knowledge Base structure.
- Updated routing to reflect the new Knowledge Base paths.
- Adjusted menu items to align with the new Knowledge Base terminology.
- Modified ModelAccess interface to include modelName and type properties.
* feat: Implement Knowledge Base Page with CRUD operations and data management
- Added KnowledgeBasePage component for displaying and managing knowledge bases.
- Integrated search and filter functionalities with SearchControls component.
- Implemented CreateKnowledgeBase component for creating and editing knowledge bases.
- Enhanced AddDataDialog for file uploads and dataset selections.
- Introduced TableTransfer component for managing data transfers between tables.
- Updated API functions for knowledge base operations, including file management.
- Refactored knowledge base model to include file status and metadata.
- Adjusted routing to point to the new KnowledgeBasePage.
* feat: enhance OperatorPluginCreate and ConfigureStep for better upload handling and UI updates
* feat: update icons in various components and improve styling for better UI consistency
* fix: adjust upload step handling and improve error display in configuration step
* feat: Update site name to DataMate and refine text for AI data processing
* Refactor file upload and operator management components
- Removed unnecessary console logs from file download and export functions.
- Added size property to TaskItem interface for better task management.
- Simplified TaskUpload component by utilizing useFileSliceUpload hook for file upload logic.
- Enhanced OperatorPluginCreate component to handle file uploads and parsing more efficiently.
- Updated ConfigureStep component to use Ant Design Form for better data handling and validation.
- Improved PreviewStep component to navigate back to the operator market.
- Added support for additional file types in UploadStep component.
- Implemented delete operator functionality in OperatorMarketPage with confirmation prompts.
- Cleaned up unused API functions in operator.api.ts to streamline the codebase.
- Fixed number formatting utility to handle zero values correctly.
* Refactor Knowledge Generation to Knowledge Base
- Created new API service for Knowledge Base operations including querying, creating, updating, and deleting knowledge bases and files.
- Added constants for Knowledge Base status and type mappings.
- Defined models for Knowledge Base and related files.
- Removed obsolete Knowledge Base creation and home components, replacing them with new implementations under the Knowledge Base structure.
- Updated routing to reflect the new Knowledge Base paths.
- Adjusted menu items to align with the new Knowledge Base terminology.
- Modified ModelAccess interface to include modelName and type properties.
* feat: Implement Knowledge Base Page with CRUD operations and data management
- Added KnowledgeBasePage component for displaying and managing knowledge bases.
- Integrated search and filter functionalities with SearchControls component.
- Implemented CreateKnowledgeBase component for creating and editing knowledge bases.
- Enhanced AddDataDialog for file uploads and dataset selections.
- Introduced TableTransfer component for managing data transfers between tables.
- Updated API functions for knowledge base operations, including file management.
- Refactored knowledge base model to include file status and metadata.
- Adjusted routing to point to the new KnowledgeBasePage.
* feat: enhance OperatorPluginCreate and ConfigureStep for better upload handling and UI updates
* feat: update icons in various components and improve styling for better UI consistency
* fix: adjust upload step handling and improve error display in configuration step
* feat: add settings drawer and integrate SettingsPage component
* feat: add ratio task management features including detail view and API integration
* feat: enable destruction of hidden settings drawer to free up resources
* feat: add data metrics and ratio display components with charts for enhanced data visualization
* feat: Update site name to DataMate and refine text for AI data processing
* feat: Refactor settings page and implement model access functionality
- Created a new ModelAccess component for managing model configurations.
- Removed the old Settings component and replaced it with a new SettingsPage component that integrates ModelAccess, SystemConfig, and WebhookConfig.
- Added SystemConfig component for managing system settings.
- Implemented WebhookConfig component for managing webhook configurations.
- Updated API functions for model management in settings.apis.ts.
- Adjusted routing to point to the new SettingsPage component.
* feat: Implement Data Collection Page with Task Management and Execution Log
- Created DataCollectionPage component to manage data collection tasks.
- Added TaskManagement and ExecutionLog components for task handling and logging.
- Integrated task operations including start, stop, edit, and delete functionalities.
- Implemented filtering and searching capabilities in task management.
- Introduced SimpleCronScheduler for scheduling tasks with cron expressions.
- Updated CreateTask component to utilize new scheduling and template features.
- Enhanced BasicInformation component to conditionally render fields based on visibility settings.
- Refactored ImportConfiguration component to remove NAS import section.
* feat: Update task creation API endpoint and enhance task creation form with new fields and validation
* Refactor file upload and operator management components
- Removed unnecessary console logs from file download and export functions.
- Added size property to TaskItem interface for better task management.
- Simplified TaskUpload component by utilizing useFileSliceUpload hook for file upload logic.
- Enhanced OperatorPluginCreate component to handle file uploads and parsing more efficiently.
- Updated ConfigureStep component to use Ant Design Form for better data handling and validation.
- Improved PreviewStep component to navigate back to the operator market.
- Added support for additional file types in UploadStep component.
- Implemented delete operator functionality in OperatorMarketPage with confirmation prompts.
- Cleaned up unused API functions in operator.api.ts to streamline the codebase.
- Fixed number formatting utility to handle zero values correctly.
* Refactor Knowledge Generation to Knowledge Base
- Created new API service for Knowledge Base operations including querying, creating, updating, and deleting knowledge bases and files.
- Added constants for Knowledge Base status and type mappings.
- Defined models for Knowledge Base and related files.
- Removed obsolete Knowledge Base creation and home components, replacing them with new implementations under the Knowledge Base structure.
- Updated routing to reflect the new Knowledge Base paths.
- Adjusted menu items to align with the new Knowledge Base terminology.
- Modified ModelAccess interface to include modelName and type properties.
* feat: Implement Knowledge Base Page with CRUD operations and data management
- Added KnowledgeBasePage component for displaying and managing knowledge bases.
- Integrated search and filter functionalities with SearchControls component.
- Implemented CreateKnowledgeBase component for creating and editing knowledge bases.
- Enhanced AddDataDialog for file uploads and dataset selections.
- Introduced TableTransfer component for managing data transfers between tables.
- Updated API functions for knowledge base operations, including file management.
- Refactored knowledge base model to include file status and metadata.
- Adjusted routing to point to the new KnowledgeBasePage.
* feat: enhance OperatorPluginCreate and ConfigureStep for better upload handling and UI updates
* refactor: remove unused components and clean up API logging in KnowledgeBase
* feat: update icons in various components and improve styling for better UI consistency
* fix: adjust upload step handling and improve error display in configuration step
* feat: Add RatioTransfer component for dataset selection and configuration
- Implemented RatioTransfer component to manage dataset selection and ratio configuration.
- Integrated dataset fetching with search and filter capabilities.
- Added RatioConfig component for displaying and updating selected datasets' configurations.
- Enhanced SelectDataset component with improved UI and functionality for dataset selection.
- Updated RatioTasksPage to utilize new ratio task status mapping and improved error handling for task deletion.
- Refactored ratio model and constants for better type safety and clarity.
- Changed Vite configuration to use local backend service for development.
* feat: Add .editorconfig and enhance SystemConfig with table for settings display
* feat: Enhance parameter configuration for range inputs and update default values
* feat: Update site name to DataMate and refine text for AI data processing
* Refactor file upload and operator management components
- Removed unnecessary console logs from file download and export functions.
- Added size property to TaskItem interface for better task management.
- Simplified TaskUpload component by utilizing useFileSliceUpload hook for file upload logic.
- Enhanced OperatorPluginCreate component to handle file uploads and parsing more efficiently.
- Updated ConfigureStep component to use Ant Design Form for better data handling and validation.
- Improved PreviewStep component to navigate back to the operator market.
- Added support for additional file types in UploadStep component.
- Implemented delete operator functionality in OperatorMarketPage with confirmation prompts.
- Cleaned up unused API functions in operator.api.ts to streamline the codebase.
- Fixed number formatting utility to handle zero values correctly.
* Refactor Knowledge Generation to Knowledge Base
- Created new API service for Knowledge Base operations including querying, creating, updating, and deleting knowledge bases and files.
- Added constants for Knowledge Base status and type mappings.
- Defined models for Knowledge Base and related files.
- Removed obsolete Knowledge Base creation and home components, replacing them with new implementations under the Knowledge Base structure.
- Updated routing to reflect the new Knowledge Base paths.
- Adjusted menu items to align with the new Knowledge Base terminology.
- Modified ModelAccess interface to include modelName and type properties.
* feat: Implement Knowledge Base Page with CRUD operations and data management
- Added KnowledgeBasePage component for displaying and managing knowledge bases.
- Integrated search and filter functionalities with SearchControls component.
- Implemented CreateKnowledgeBase component for creating and editing knowledge bases.
- Enhanced AddDataDialog for file uploads and dataset selections.
- Introduced TableTransfer component for managing data transfers between tables.
- Updated API functions for knowledge base operations, including file management.
- Refactored knowledge base model to include file status and metadata.
- Adjusted routing to point to the new KnowledgeBasePage.
* feat: enhance OperatorPluginCreate and ConfigureStep for better upload handling and UI updates
* feat: update icons in various components and improve styling for better UI consistency
* fix: adjust upload step handling and improve error display in configuration step
* feat: Update site name to DataMate and refine text for AI data processing
* Refactor file upload and operator management components
- Removed unnecessary console logs from file download and export functions.
- Added size property to TaskItem interface for better task management.
- Simplified TaskUpload component by utilizing useFileSliceUpload hook for file upload logic.
- Enhanced OperatorPluginCreate component to handle file uploads and parsing more efficiently.
- Updated ConfigureStep component to use Ant Design Form for better data handling and validation.
- Improved PreviewStep component to navigate back to the operator market.
- Added support for additional file types in UploadStep component.
- Implemented delete operator functionality in OperatorMarketPage with confirmation prompts.
- Cleaned up unused API functions in operator.api.ts to streamline the codebase.
- Fixed number formatting utility to handle zero values correctly.
* Refactor Knowledge Generation to Knowledge Base
- Created new API service for Knowledge Base operations including querying, creating, updating, and deleting knowledge bases and files.
- Added constants for Knowledge Base status and type mappings.
- Defined models for Knowledge Base and related files.
- Removed obsolete Knowledge Base creation and home components, replacing them with new implementations under the Knowledge Base structure.
- Updated routing to reflect the new Knowledge Base paths.
- Adjusted menu items to align with the new Knowledge Base terminology.
- Modified ModelAccess interface to include modelName and type properties.
* feat: Implement Knowledge Base Page with CRUD operations and data management
- Added KnowledgeBasePage component for displaying and managing knowledge bases.
- Integrated search and filter functionalities with SearchControls component.
- Implemented CreateKnowledgeBase component for creating and editing knowledge bases.
- Enhanced AddDataDialog for file uploads and dataset selections.
- Introduced TableTransfer component for managing data transfers between tables.
- Updated API functions for knowledge base operations, including file management.
- Refactored knowledge base model to include file status and metadata.
- Adjusted routing to point to the new KnowledgeBasePage.
* feat: enhance OperatorPluginCreate and ConfigureStep for better upload handling and UI updates
* feat: update icons in various components and improve styling for better UI consistency
* fix: adjust upload step handling and improve error display in configuration step
* feat: add settings drawer and integrate SettingsPage component
* feat: add ratio task management features including detail view and API integration
* feat: Enhance annotation module with template management and validation
- Added DatasetMappingCreateRequest and DatasetMappingUpdateRequest schemas to handle dataset mapping requests with camelCase and snake_case support.
- Introduced Annotation Template schemas including CreateAnnotationTemplateRequest, UpdateAnnotationTemplateRequest, and AnnotationTemplateResponse for managing annotation templates.
- Implemented AnnotationTemplateService for creating, updating, retrieving, and deleting annotation templates, including validation of configurations and XML generation.
- Added utility class LabelStudioConfigValidator for validating Label Studio configurations and XML formats.
- Updated database schema for annotation templates and labeling projects to include new fields and constraints.
- Seeded initial annotation templates for various use cases including image classification, object detection, and text classification.
* feat: Enhance TemplateForm with improved validation and dynamic field rendering; update LabelStudio config validation for camelCase support
* feat: Update docker-compose.yml to mark datamate dataset volume and network as external
* feat: Refactor configuration and sync logic for improved dataset handling and logging
* feat: Enhance annotation synchronization and dataset file management
- Added new fields `tags_updated_at` to `DatasetFiles` model for tracking the last update time of tags.
- Implemented new asynchronous methods in the Label Studio client for fetching, creating, updating, and deleting task annotations.
- Introduced bidirectional synchronization for annotations between DataMate and Label Studio, allowing for flexible data management.
- Updated sync service to handle annotation conflicts based on timestamps, ensuring data integrity during synchronization.
- Enhanced dataset file response model to include tags and their update timestamps.
- Modified database initialization script to create a new column for `tags_updated_at` in the dataset files table.
- Updated requirements to ensure compatibility with the latest dependencies.
* fix: Update port mapping for label studio and adjust base URL in DataAnnotation component
