97 lines
7.1 KiB
Markdown
97 lines
7.1 KiB
Markdown
# task-scheduler Specification
|
|
|
|
## Purpose
|
|
TBD - created by archiving change cloud-runtime. Update Purpose after archive.
|
|
## Requirements
|
|
### Requirement: Task submission enqueues a scheduled task
|
|
The system SHALL allow a caller to submit a task (a goal string, or a reference to a `WorkflowDefinition`, plus optional device constraints: `driver_type`, required capability tags) and SHALL enqueue it as a `ScheduledTask` with status `queued`, returning a stable task id the caller can poll.
|
|
|
|
#### Scenario: Successful submission
|
|
- **WHEN** a caller submits a task with a goal and no constraints
|
|
- **THEN** the scheduler creates a `ScheduledTask` with status `queued`, assigns it a unique id, and returns that id to the caller without blocking for a device to become available
|
|
|
|
#### Scenario: Queue depth limit reached
|
|
- **WHEN** a caller submits a task while the queue already holds `config.max_queue_depth` queued tasks
|
|
- **THEN** the scheduler rejects the submission with a clear error rather than accepting an unbounded backlog
|
|
|
|
### Requirement: Assignment matches a queued task to an idle, constraint-matching device
|
|
The system SHALL assign a queued `ScheduledTask` to an idle `PooledDevice` (as reported by the `device-pool` capability) whose `driver_type` and capability tags satisfy the task's constraints, using a named, registrable `AssignmentStrategy`.
|
|
|
|
#### Scenario: Matching idle device available
|
|
- **WHEN** `assign()` runs and at least one idle `PooledDevice` matches the head-of-queue task's constraints
|
|
- **THEN** the scheduler selects one such device via the configured `AssignmentStrategy`, transitions the task to status `assigned`, and records the chosen `device_id`/`host_id`
|
|
|
|
#### Scenario: No matching device available
|
|
- **WHEN** `assign()` runs and no idle `PooledDevice` matches the head-of-queue task's constraints
|
|
- **THEN** the task remains `queued` (not failed), and `assign()` returns without error, ready to be retried on a later call
|
|
|
|
#### Scenario: Unknown assignment strategy configured
|
|
- **WHEN** `TaskScheduler` is configured with an `AssignmentStrategy` name that is not registered
|
|
- **THEN** the scheduler raises a clear configuration error at startup/first-assign rather than silently falling back to a default strategy
|
|
|
|
### Requirement: Assignment strategies are pluggable by name
|
|
The system SHALL provide an `AssignmentStrategy` registry mapping a strategy name to an implementation, with a default `fifo_match` strategy (oldest-queued matching task first, first matching idle device), and SHALL allow a new strategy to be added by registering a name without modifying `TaskScheduler`'s control flow.
|
|
|
|
#### Scenario: Default FIFO strategy orders by submission time
|
|
- **WHEN** two tasks with satisfiable, overlapping constraints are queued in order A then B, and one matching idle device exists
|
|
- **THEN** the default `fifo_match` strategy assigns the device to task A, leaving task B queued
|
|
|
|
#### Scenario: Adding a new strategy requires no scheduler edit
|
|
- **WHEN** a new `AssignmentStrategy` implementation is registered under a new name
|
|
- **THEN** `TaskScheduler` can be configured to use it by name alone, with no change to `scheduler.py`'s assignment control flow
|
|
|
|
### Requirement: Local dispatch executes an assignment via existing runners
|
|
The system SHALL provide a `TaskDispatcher` that, for an assignment whose device is owned by the local process's own host, executes the assigned task by composing the existing `agent-runtime` task-execution entry point (for a goal-based submission) or the `workflow-orchestration` workflow-execution entry point (for a workflow-based submission), without reimplementing planning/execution/retry logic.
|
|
|
|
#### Scenario: Dispatching a goal-based assignment
|
|
- **WHEN** `TaskDispatcher.dispatch()` is called with an assignment for a goal-based `ScheduledTask` whose device is local
|
|
- **THEN** the dispatcher constructs and runs a `Task` through the existing task-execution entry point, and updates the `ScheduledTask`'s status to `done` or `failed` based on the resulting task's outcome
|
|
|
|
#### Scenario: Dispatching a workflow-based assignment
|
|
- **WHEN** `TaskDispatcher.dispatch()` is called with an assignment referencing a `WorkflowDefinition` whose device is local
|
|
- **THEN** the dispatcher runs the definition through the existing workflow-execution entry point and updates the `ScheduledTask`'s status based on the resulting workflow run's outcome
|
|
|
|
### Requirement: Assignment and device reservation are atomic
|
|
The scheduler SHALL atomically bind a queued task to one eligible device, create a bounded lease attempt, and reserve that device so no other active task can be assigned to it.
|
|
|
|
#### Scenario: Scheduler assigns an idle device
|
|
- **WHEN** a queued task matches an idle pooled device with no active reservation
|
|
- **THEN** one transaction records the assigned task, owning host/device, incremented attempt, lease identifier, lease expiry, and device reservation
|
|
|
|
#### Scenario: Later scheduler iteration sees stale idle snapshot
|
|
- **WHEN** the host snapshot still reports a device idle but that device has an active assignment lease
|
|
- **THEN** the scheduler excludes the device from candidates for every other queued task
|
|
|
|
### Requirement: Host claim transitions assigned work to dispatched
|
|
The scheduler repository SHALL allow only the authenticated owning host to atomically claim an unexpired assigned attempt and transition it to `dispatched`.
|
|
|
|
#### Scenario: Owning host claims once
|
|
- **WHEN** the owning host requests available work and an unexpired assigned attempt exists
|
|
- **THEN** exactly one request receives the assignment and its status becomes dispatched
|
|
|
|
#### Scenario: Concurrent claims race
|
|
- **WHEN** multiple requests concurrently attempt to claim the same assignment
|
|
- **THEN** at most one request succeeds and every other request receives no assignment or a conflict
|
|
|
|
### Requirement: Expired attempts follow bounded retry policy
|
|
The system SHALL detect expired assigned or dispatched leases and SHALL either requeue the task with its reservation released or mark it failed when the configured attempt limit is reached.
|
|
|
|
#### Scenario: Lease expires with attempts remaining
|
|
- **WHEN** an active lease expires before a terminal result and the task has remaining attempts
|
|
- **THEN** the task returns to queued, the previous device reservation is released, and the expired attempt remains auditable
|
|
|
|
#### Scenario: Lease expires at attempt limit
|
|
- **WHEN** an active lease expires and the task has reached its maximum attempts
|
|
- **THEN** the task becomes failed with a lease-expiry reason and its device reservation is released
|
|
|
|
### Requirement: Terminal transitions validate the active lease
|
|
The system SHALL accept a `done` or `failed` result only from the current active task attempt and lease and SHALL make repeated identical terminal reports idempotent.
|
|
|
|
#### Scenario: Active lease reports completion
|
|
- **WHEN** the active lease owner reports a terminal result
|
|
- **THEN** the task transitions once to done or failed and releases its device reservation
|
|
|
|
#### Scenario: Superseded lease reports completion
|
|
- **WHEN** a result references a lease superseded by expiry and retry
|
|
- **THEN** the result is rejected and cannot overwrite the current task attempt
|