# Conflicts: # packages/cloud-platform/cloud/schema.py
This commit is contained in:
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-07-15
|
||||
@@ -0,0 +1,266 @@
|
||||
## Context
|
||||
|
||||
Task cancellation has no implementation anywhere in the stack today:
|
||||
|
||||
- `core/models.py`'s `TaskStatus` reserves `"cancelled"` but no code path assigns it.
|
||||
`runtime/task.py::TaskRunner._interrupt_task()` marks `should_stop`-triggered
|
||||
interruption as `status="failed", failure_reason="execution interrupted"` — this is the
|
||||
only existing consumer of `should_stop`, and it conflates "lease was lost" with "user
|
||||
asked to stop" (both produce `failed`).
|
||||
- `cloud/scheduler.py`'s `ScheduledTaskStatus` is `queued | assigned | dispatched | done |
|
||||
failed` — no `cancelled`, no cancel operation on `TaskScheduler` or `CloudRepository`.
|
||||
- The internal Host↔Cloud protocol (`cloud/internal_api/models.py`) has exactly one
|
||||
channel from Cloud back to Host during execution: `LeaseRenewalResponse`, currently
|
||||
`{status: Literal["renewed"], lease_expires_at: datetime}`. There is no push channel;
|
||||
the Host Agent is purely outbound (heartbeat, claim long-poll, renew, report result).
|
||||
- `workflow/` already has the target shape: `WorkflowRunStatus` includes `cancelled`,
|
||||
`WorkflowRunner._drive()` checks `should_stop()` at each step boundary and calls
|
||||
`self._checkpoint(run, ..., "cancelled")`. This design generalizes that pattern to
|
||||
`TaskRunner`/`ScheduledTask` rather than inventing a new one.
|
||||
- Three prior changes (`cloud-console`, `cloud-control-plane-integration`,
|
||||
`cloud-console-governance`) explicitly deferred cancellation, citing "no
|
||||
scheduler/repository operation for this exists." This change adds that operation.
|
||||
|
||||
Existing collaborative-stop infrastructure this design reuses instead of replacing:
|
||||
`ActiveAssignmentRunner.run()` (`apps/device-host-agent/host_agent/lease.py`) races
|
||||
execution against a renewal loop; the renewal loop is the only place the Host Agent
|
||||
talks to Cloud while a task is running. `should_stop` already flows
|
||||
`ActiveAssignmentRunner` → `AssignmentExecutor.execute()` →
|
||||
`TaskRunner.run()`/`WorkflowRunner.run()`, checked at step boundaries (never mid-step).
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Let an authorized caller cancel a task in `queued`, `assigned`, or `dispatched` state
|
||||
through the public SDK.
|
||||
- For `queued` tasks (never dispatched to a Host), cancellation is synchronous and
|
||||
immediate — no Host round-trip needed.
|
||||
- For `assigned`/`dispatched` tasks (a Host may be actively executing), cancellation is
|
||||
collaborative: the Host learns about it at its next lease renewal (≤ ~1/3 of
|
||||
`lease_duration_seconds`, matching the existing lease-loss detection latency) and stops
|
||||
at the next step boundary, exactly like a lost lease does today.
|
||||
- Make `cancelled` a real, reachable terminal state end-to-end: `core/models.py`'s
|
||||
`TaskStatus`, `cloud/scheduler.py`'s `ScheduledTaskStatus`, the Cloud Console, and the
|
||||
Host Agent local console.
|
||||
- Distinguish "cancelled by request" from "failed" in every layer that currently reports
|
||||
`should_stop`-triggered stops as generic failure, so operators can tell the two apart.
|
||||
- Cancellation is idempotent: cancelling an already-cancelled or already-terminal task is
|
||||
a no-op with a clear response, not an error that implies something changed.
|
||||
|
||||
**Non-Goals:**
|
||||
- No mid-step interruption. A single in-flight action (a tap, an LLM planning call) is
|
||||
never aborted mid-flight; the existing step-boundary granularity of `should_stop` is
|
||||
unchanged. A slow single step still finishes before a `dispatched` task's cancellation
|
||||
takes effect.
|
||||
- No forced/instant kill of a Host Agent process or its OS-level subprocess tree. This is
|
||||
cooperative cancellation only, consistent with the project's existing lease-loss
|
||||
handling — not a new capability class.
|
||||
- No per-task ownership/ACL model. The Cloud repository does not track which principal
|
||||
submitted a task; cancel authorization reuses the existing `tasks:submit` scope
|
||||
(whoever can submit a task can cancel any task). Adding submitter-scoped authorization
|
||||
is a distinct governance change, out of scope here (parallels
|
||||
`cloud-console-governance`'s existing target-based, not ownership-based, model).
|
||||
- No retry-from-UI or task editing. Only stopping a task early; resubmission remains a
|
||||
separate, already-existing "submit a new task" action.
|
||||
- No cancellation of individual workflow steps independent of the whole run; workflow
|
||||
cancellation continues to mean "stop the run," which `workflow/` already implements.
|
||||
- No change to `TaskDispatcher` (`cloud/dispatch.py`) — confirmed dead/dev-only code per
|
||||
prior investigation; not worth extending for cancellation.
|
||||
|
||||
## Decisions
|
||||
|
||||
### D1: `LeaseRenewalResponse.cancel_requested: bool` is the only new wire signal
|
||||
|
||||
Rejected alternatives: (a) a new dedicated Host-polled "check cancellation" endpoint —
|
||||
adds a second polling loop and a second latency bound to reason about, when renewal
|
||||
already runs on a well-understood cadence; (b) a push/webhook mechanism — breaks the
|
||||
explicit "Host Agent operation requires no inbound cloud connection" requirement in
|
||||
`host-agent-protocol`'s spec, which is a hard architectural constraint, not just current
|
||||
practice.
|
||||
|
||||
`cancel_requested` defaults to `False`. When the Cloud repository's `renew_lease` finds
|
||||
the active `ScheduledTask` has a pending cancellation record, it still renews the lease
|
||||
normally (a task must keep a live lease while the Host works through shutdown) but flags
|
||||
`cancel_requested=True` in the response. The Host Agent's `_renew_while_running` loop
|
||||
treats this the same way it treats `StaleLeaseError`: mark the `LeaseGuard` (extended
|
||||
with a distinct reason string, e.g. `"cancellation requested by control plane"`) and let
|
||||
the existing `should_stop` composite (`guard.is_lost() or self._stop_requested.is_set()`)
|
||||
do the rest — no new boolean plumbed through `ActiveAssignmentRunner`/`AssignmentExecutor`
|
||||
signatures.
|
||||
|
||||
### D2: Cancellation request is durable state, not a fire-and-forget signal
|
||||
|
||||
A `POST .../cancel` on an `assigned`/`dispatched` task writes a `cancel_requested_at`
|
||||
timestamp (see D5 schema) rather than only flipping an in-memory flag or directly setting
|
||||
`status="cancelled"`. Rationale: the Host may not renew again for up to
|
||||
`lease_duration_seconds / 3`; if the Cloud process restarts in that window, an in-memory
|
||||
signal would be lost silently, defeating the cancellation with no user-visible error. A
|
||||
durable row survives restarts and is what `renew_lease` reads.
|
||||
|
||||
The scheduled task's `status` only transitions to `cancelled` once the Host has actually
|
||||
stopped and reported it (D3) — or, for the immediate `queued` case, synchronously in the
|
||||
same request. This means `assigned`/`dispatched` tasks pass through an intermediate,
|
||||
observable "cancellation pending" state (surfaced to callers as the existing `status`
|
||||
value plus a non-null `cancel_requested_at`, not a new status literal — see D6) rather
|
||||
than jumping straight to `cancelled` before the Host has confirmed.
|
||||
|
||||
### D3: Host reports `cancelled` as a new terminal outcome via the existing result-report endpoint
|
||||
|
||||
`TerminalResultRequest.status` (`internal_api/models.py`) is `Literal["done", "failed"]`.
|
||||
This design adds `"cancelled"` to that literal rather than inventing a parallel
|
||||
cancellation-report endpoint, because `report_result`'s idempotency/lease-validation
|
||||
logic (`record_task_result` in `sql_repository.py`) already handles exactly the
|
||||
concurrency shape needed (attempt/lease-id conflict checks, `already_recorded` replay
|
||||
safety) — duplicating it for a cancel-specific endpoint would be pure risk with no
|
||||
benefit. `AssignmentExecutionResult.status` (`host_agent/assignment.py`) similarly gains
|
||||
a `"cancelled"` value alongside `"done"`/`"failed"`, set when `TaskRunner.run()` /
|
||||
`WorkflowRunner.run()` returns because of a cancellation-flavored stop rather than a lost
|
||||
lease or genuine step failure (see D4).
|
||||
|
||||
### D4: `should_stop` becomes a richer signal than a bare boolean at the `TaskRunner` boundary
|
||||
|
||||
Today `StopRequested = Callable[[], bool]`. Distinguishing "cancelled" from "lease
|
||||
lost"/"shutdown requested" only matters for the *reason recorded on the terminal
|
||||
status* — the control-flow behavior (stop at the next step boundary) is identical. Rather
|
||||
than changing the `should_stop` callable's signature (which would ripple through
|
||||
`workflow/runner.py`, tests, and every caller), `TaskRunner._interrupt_task()` and
|
||||
`WorkflowRunner._drive()`'s stop branch gain an optional second callable,
|
||||
`stop_reason: Callable[[], str] | None`, defaulting to the existing generic
|
||||
`"execution interrupted"` string when absent. `ActiveAssignmentRunner` supplies a
|
||||
`stop_reason` that reads `LeaseGuard.reason` (already a string field) so `"cancellation
|
||||
requested by control plane"` vs `"lease rejected by control plane"` flows through
|
||||
unchanged plumbing. `TaskRunner._interrupt_task()` sets `status="cancelled"` when the
|
||||
reason string indicates cancellation, else keeps `status="failed"` (lease loss remains a
|
||||
failure, not a cancellation, matching current behavior for that case). This is the
|
||||
narrowest change that gets a real `cancelled` status without touching every
|
||||
`should_stop`-typed parameter across the codebase.
|
||||
|
||||
Alternative considered and rejected: add a third `CancelRequested` callable parallel to
|
||||
`should_stop`. Rejected because it doubles the number of callables threaded through
|
||||
`ActiveAssignmentRunner → AssignmentExecutor → TaskRunner/WorkflowRunner` for
|
||||
information that's only needed at the one moment execution actually stops.
|
||||
|
||||
### D5: Schema — one new nullable column pair on `scheduled_tasks`, no new table
|
||||
|
||||
`packages/cloud-platform/cloud/migrations/versions/0012_task_cancellation.py` adds to
|
||||
`ScheduledTaskRow`/`scheduled_tasks`:
|
||||
- `cancel_requested_at: str | None` (ISO datetime, nullable) — set when a cancel request
|
||||
is recorded against an `assigned`/`dispatched` task; cleared (`NULL`) when the task
|
||||
reaches any terminal status, so a subsequent, different attempt of the same task id (if
|
||||
retry-on-lease-expiry logic in `reap_expired_leases` requeues it) doesn't inherit a
|
||||
stale cancellation.
|
||||
|
||||
No new table: the volume and access pattern (one pending value per task, read on every
|
||||
renewal, written once per cancel call) don't justify the join/joinless-read trade-off a
|
||||
separate `task_cancellation_requests` table would add, and there is exactly one prior
|
||||
migration precedent for this shape — `0008_task_progress_columns` added nullable
|
||||
scalar columns directly to `scheduled_tasks` for the same reason (frequently-read,
|
||||
single-value-per-task state). `TaskAttemptRow`/`task_attempts` needs no schema change:
|
||||
its `status` column already accepts free-form strings and gains `"cancelled"` as a value
|
||||
alongside `"assigned"/"dispatched"/"expired"/"done"/"failed"`.
|
||||
|
||||
`CloudRepository` Protocol gains:
|
||||
- `request_task_cancellation(task_id, *, requested_at) -> CancellationRequestStatus`
|
||||
where `CancellationRequestStatus = Literal["requested", "already_terminal",
|
||||
"already_requested", "not_found"]` — synchronously transitions a `queued` task straight
|
||||
to `cancelled` (there's no Host attempt in flight to notify) and otherwise sets
|
||||
`cancel_requested_at` on an `assigned`/`dispatched` task.
|
||||
- `renew_lease` gains a `cancel_requested` boolean in its return path (or the caller
|
||||
re-reads the row — implementation detail left to tasks.md) so `internal_api/api.py`'s
|
||||
`renew_assignment` handler can populate `LeaseRenewalResponse.cancel_requested`.
|
||||
- `record_task_result` accepts `status: Literal["done", "failed", "cancelled"]`
|
||||
(widened from today's `TerminalTaskStatus = Literal["done", "failed"]`) and clears
|
||||
`cancel_requested_at` on write.
|
||||
|
||||
### D6: No new `ScheduledTaskStatus` value for "cancellation pending"
|
||||
|
||||
Considered adding `"cancelling"` as a distinct status between `assigned`/`dispatched` and
|
||||
`cancelled`. Rejected: it would ripple into every place that already pattern-matches the
|
||||
existing five-value `ScheduledTaskStatus` (scheduler assignment logic, device
|
||||
reservation/`list_reserved_device_ids`, Cloud Console status filter, SDK response
|
||||
`Literal`), each needing to decide whether "cancelling" behaves like "assigned" (device
|
||||
still reserved, task still excluded from re-assignment) — which it always would, making
|
||||
the new status a strict synonym with an extra bit of information. Instead,
|
||||
"cancellation pending" is expressed as `status="assigned"` (or `"dispatched"`) plus
|
||||
non-null `cancel_requested_at` — every existing status-based code path (assignment
|
||||
matching, device reservation, retry-on-expiry) keeps working unmodified, and callers
|
||||
that want to show "cancelling…" in a UI check the extra field.
|
||||
|
||||
### D7: Public cancel endpoint shape
|
||||
|
||||
`POST /v1/tasks/{task_id}/cancel`, `202 Accepted` for the pending (assigned/dispatched)
|
||||
case and `200 OK` for the immediate (queued) case, both returning a small
|
||||
`TaskCancellationResponse {task_id, status}` reflecting the resulting `ScheduledTask`
|
||||
status. Repeated calls against an already-cancelled or already-`cancel_requested_at`-set
|
||||
task return the same response idempotently (HTTP 200, not a conflict) — matching the
|
||||
project's established idempotency style for `report_result`/`renew_assignment`, which
|
||||
return `already_recorded`/success on replay rather than erroring. Calling cancel on a
|
||||
task already `done`/`failed` returns `409 Conflict` with a clear "task is already
|
||||
terminal" detail, mirroring `_stale_lease_conflict`'s existing shape.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- [Operators expect cancellation to be instant] → It isn't, by design (D1's latency
|
||||
bound). The Cloud Console surfaces the `cancel_requested_at` pending state distinctly
|
||||
(D6) so operators see "cancellation requested" rather than a UI that looks stuck; docs
|
||||
(`docs/CLOUD_DEPLOYMENT.md`) get an explicit latency note per the proposal's Impact
|
||||
section.
|
||||
- [A Host Agent that never renews again (already offline, or wedged past its lease) never
|
||||
learns about the cancellation] → Existing `reap_expired_leases` already requeues or
|
||||
fails such tasks once the lease expires; this change doesn't need new machinery for
|
||||
that case — an offline Host's task eventually reaches a terminal state via the existing
|
||||
reaper, at which point `cancel_requested_at` being non-null is irrelevant (task is
|
||||
already terminal). Worth noting in tasks.md that the reaper's requeue path should NOT
|
||||
requeue (re-assign) a task with `cancel_requested_at` set — it should mark it
|
||||
`cancelled` instead of `queued`, or the cancellation would be silently dropped on
|
||||
retry.
|
||||
- [`tasks:submit`-scoped cancel with no per-task ownership means any authorized submitter
|
||||
can cancel any other submitter's task] → Accepted for this change (Non-Goals); this
|
||||
matches the existing coarse-grained scope model everywhere else in the SDK (task
|
||||
reading is likewise not ownership-scoped) and narrowing it is `cloud-console-governance`
|
||||
territory, not this change's.
|
||||
- [Widening `TaskStatus`/`ScheduledTaskStatus`/`TaskAttemptRow.status` literals is a
|
||||
backward-compatible additive change in Python, but any exhaustive `switch`/discriminated
|
||||
union in the TypeScript Console (`types.ts`'s `TaskStatus`) will fail to compile until
|
||||
updated] → Caught at Console build time (Vite/`vue-tsc`), not runtime; tasks.md includes
|
||||
updating `cloud-console/src/types.ts` and any exhaustiveness-checked switch in the same
|
||||
commit as the backend change.
|
||||
- [Race: cancel request arrives between the Host's `claim_assignment` (queued→dispatched
|
||||
transition happens inside `claim_assignment`, not `assign_task`) and its first renewal]
|
||||
→ Already handled by D2's durable-row design: the cancel call checks current `status`
|
||||
regardless of exactly when the Host claims, and `renew_lease` always re-reads the
|
||||
current row, so there's no window where the signal is lost — only a window (bounded by
|
||||
D1's latency) where it hasn't been observed yet.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
1. Ship the Alembic migration (0012) — additive nullable column, no backfill needed, safe
|
||||
to apply with the application running (existing tasks get `cancel_requested_at = NULL`,
|
||||
behaviorally identical to today).
|
||||
2. Deploy Cloud API with the new repository methods, internal `renew_assignment` response
|
||||
field (`cancel_requested`, defaults `False` — old Host Agents ignore unknown fields),
|
||||
and the new public cancel endpoint. This step alone is a no-op for existing Hosts:
|
||||
nothing calls the new endpoint yet, and `LeaseRenewalResponse` gaining an optional
|
||||
field is backward-compatible with any Host Agent version already deployed (Pydantic
|
||||
response models are additive-safe for JSON-decoding clients that only read known
|
||||
fields).
|
||||
3. Deploy Host Agents with the updated `lease.py`/`assignment.py`/`client.py` that read
|
||||
and act on `cancel_requested`. Hosts not yet updated simply never observe cancellation
|
||||
requests (task stays "pending cancellation" until its lease naturally expires and the
|
||||
reaper marks it `cancelled` per the Risks section) — a soft-fail, not a hard error.
|
||||
4. Ship the Cloud Console and Host Agent local console UI changes last, once the backend
|
||||
contract is stable.
|
||||
5. Rollback: the migration is purely additive and safe to leave in place even if the
|
||||
feature is disabled; no rollback migration is required beyond the standard Alembic
|
||||
downgrade (drop the column) if a full revert is ever needed.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Whether the public cancel endpoint should also accept an optional caller-supplied
|
||||
reason string (for audit/UI display) — left to tasks.md to decide during
|
||||
implementation; not a design-blocking question since it's purely additive to
|
||||
`TaskCancellationRequest` if added later.
|
||||
- Whether `reap_expired_leases`' "mark cancelled instead of requeue when
|
||||
`cancel_requested_at` is set" behavior (noted in Risks) needs its own explicit
|
||||
requirement in the `task-scheduler` delta spec or can be covered as an implementation
|
||||
detail of the existing restart-recovery requirement — resolve when writing specs.md.
|
||||
@@ -0,0 +1,81 @@
|
||||
## Why
|
||||
|
||||
Task cancellation has been an explicit, repeatedly-acknowledged gap: `cloud-console`,
|
||||
`cloud-control-plane-integration`, and `cloud-console-governance` each excluded it from
|
||||
scope and deferred it to "a later lifecycle capability." Operators currently have no way
|
||||
to stop a queued, assigned, or in-flight task short of letting it run to completion,
|
||||
failure, or lease expiry — including tasks stuck against an offline device or a runaway
|
||||
plan. `core/models.py`'s `TaskStatus` already reserves a `"cancelled"` value that no code
|
||||
path ever sets, and `workflow/` already proves the collaborative-stop pattern this change
|
||||
extends to goal-based cloud tasks.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Add a `cancelled` scheduler-side task status (`cloud/scheduler.py`'s
|
||||
`ScheduledTaskStatus`) reachable from `queued`, `assigned`, and `dispatched`.
|
||||
- Add a public SDK endpoint (`POST /v1/tasks/{task_id}/cancel`, gated by the existing
|
||||
`tasks:submit` scope — the repository has no per-task submitter/ownership tracking to
|
||||
authorize against more narrowly) that immediately cancels a `queued` task and otherwise
|
||||
records a cancellation request against an `assigned`/`dispatched` task.
|
||||
- Add an internal Host Agent protocol signal: `LeaseRenewalResponse` gains a
|
||||
`cancel_requested: bool` field; the Cloud repository's `renew_lease` reports it when the
|
||||
active attempt has a pending cancellation. This is the only new edge on the existing
|
||||
outbound-only Host Agent protocol — no inbound push, no new endpoint on the Host side.
|
||||
- Extend the Host Agent's existing `should_stop` collaborative-stop mechanism
|
||||
(`ActiveAssignmentRunner` → `AssignmentExecutor` → `TaskRunner`/`WorkflowRunner`) so a
|
||||
`cancel_requested` signal observed at lease-renewal time stops execution the same way a
|
||||
lost lease does today, and reports a `cancelled`-flavored terminal result.
|
||||
- Add `"cancelled"` as a real, reachable value of `core/models.py`'s `TaskStatus` (the
|
||||
`_interrupt_task` path already flowing through `should_stop` gains a
|
||||
cancellation-vs-lease-loss distinction) and confirm `TerminalResultRequest.status` can
|
||||
express it end to end.
|
||||
- Add an Alembic migration recording cancellation request/acknowledgement metadata on
|
||||
`scheduled_tasks`/`task_attempts` (requestor, requested-at, and the terminal
|
||||
`cancelled` outcome) — no schema change to unrelated tables.
|
||||
- Add a "Cancel" action to the Cloud Console `TasksView.vue` task-detail panel (visible
|
||||
for `queued`/`assigned`/`dispatched` tasks the operator is authorized to act on) and to
|
||||
the status filter dropdown; add a matching `POST /tasks/{id}/cancel` route + button to
|
||||
the Host Agent local console's task detail page for Host-local visibility/action on
|
||||
tasks running on that Host.
|
||||
- **BREAKING**: none of the existing status literals are renamed or removed; `cancelled`
|
||||
is purely additive. Callers that exhaustively `match`/switch over `TaskStatus` (Python)
|
||||
or `TaskStatus` (TypeScript) without a default arm will need to add a case — flagged in
|
||||
design.md's migration plan.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `task-cancellation`: cancellation request lifecycle across `task-scheduler` (queued/
|
||||
assigned/dispatched states), `host-agent-protocol` (collaborative cancel signal over
|
||||
lease renewal), and `agent-runtime`/workflow execution (stopping mid-task on a
|
||||
cancellation signal, distinct from lease loss).
|
||||
|
||||
### Modified Capabilities
|
||||
- `task-scheduler`: `ScheduledTaskStatus` gains `cancelled`; task submission/assignment
|
||||
requirements are unchanged, but the status-transition requirements need a new terminal
|
||||
transition path from `queued`/`assigned`/`dispatched`.
|
||||
- `host-agent-protocol`: the lease-renewal requirement ("Active execution renews its
|
||||
lease") gains a new SHALL for surfacing a cancellation request in the renewal response
|
||||
and treating it as a stop condition alongside lease loss.
|
||||
- `platform-sdk`: new cancel endpoint and scope-authorization requirement; task-status
|
||||
responses gain the `cancelled` status value.
|
||||
- `cloud-console-ui`: task list/detail view gains a Cancel action and the `cancelled`
|
||||
status value in filtering/display.
|
||||
|
||||
## Impact
|
||||
|
||||
- **Cloud API / persistence**: `packages/cloud-platform/cloud/scheduler.py`,
|
||||
`repository.py` (Protocol), `sql_repository.py`, `db_models.py`, new Alembic migration,
|
||||
`internal_api/models.py` + `internal_api/api.py` (renew/claim/cancel routes),
|
||||
`sdk/api.py` + `sdk/models.py` (new public cancel endpoint), `auth.py` (scope reuse).
|
||||
- **Host Agent**: `apps/device-host-agent/host_agent/lease.py` (`ActiveAssignmentRunner`
|
||||
cancellation-aware stop), `client.py` (surface `cancel_requested` from renew response),
|
||||
`processor.py`/`assignment.py` (terminal status reporting), local console
|
||||
(`host_agent/web/app.py` + `templates/task_detail.html`) new cancel route.
|
||||
- **Runtime**: `core/models.py` (`TaskStatus` reachability), `runtime/task.py`
|
||||
(`_interrupt_task` cancellation-vs-interruption distinction), `workflow/runner.py`
|
||||
(reuse of the already-existing `cancelled` terminal status — no change needed there).
|
||||
- **Cloud Console frontend**: `cloud-console/src/views/TasksView.vue`, `src/types.ts`,
|
||||
`src/api.ts` (new `cancelTask` client method).
|
||||
- **Docs**: `docs/CLOUD_DEPLOYMENT.md` gets a short note on cancellation being
|
||||
collaborative (not instantaneous) and its ~1/3-lease-period latency bound.
|
||||
@@ -0,0 +1,24 @@
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Task dashboard
|
||||
The console SHALL render a task view listing tasks by status with pagination, and SHALL show a task's detail including its attempt history, using the platform SDK's task-listing and attempt-history endpoints. The task detail view SHALL offer a Cancel action for tasks in status `queued`, `assigned`, or `dispatched`, using the platform SDK's cancel endpoint, and the status filter SHALL include `cancelled`.
|
||||
|
||||
#### Scenario: Browse the task queue
|
||||
- **WHEN** an operator with a `tasks:read`-scoped token opens the task view
|
||||
- **THEN** the console displays tasks with their status, goal or workflow reference, and assigned device/host, and lets the operator filter by status, including `cancelled`
|
||||
|
||||
#### Scenario: Inspect a task's attempt history
|
||||
- **WHEN** an operator selects a task from the list
|
||||
- **THEN** the console displays that task's recorded attempts in order, including each attempt's outcome
|
||||
|
||||
#### Scenario: Cancel a task from the detail view
|
||||
- **WHEN** an operator with a `tasks:submit`-scoped token views the detail of a task whose status is `queued`, `assigned`, or `dispatched`, and clicks Cancel
|
||||
- **THEN** the console calls the cancel endpoint and updates the displayed status to reflect the immediate or pending cancellation result
|
||||
|
||||
#### Scenario: Cancel action is absent for terminal tasks
|
||||
- **WHEN** an operator views the detail of a task whose status is `done`, `failed`, or `cancelled`
|
||||
- **THEN** the console does not offer a Cancel action for that task
|
||||
|
||||
#### Scenario: Cancel attempted without submit scope
|
||||
- **WHEN** an operator whose token lacks `tasks:submit` views a cancellable task's detail
|
||||
- **THEN** the console does not offer a Cancel action, or surfaces the API's authorization error without implying the task was cancelled
|
||||
@@ -0,0 +1,48 @@
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Active execution renews its lease
|
||||
The Host Agent SHALL renew the active assignment lease before expiry while execution continues, and SHALL treat loss or rejection of the lease, or an observed cancellation request, as a stop condition for further planned actions where interruption is possible.
|
||||
|
||||
#### Scenario: Lease renewal succeeds
|
||||
- **WHEN** the owning Host Agent renews an unexpired active lease
|
||||
- **THEN** the control plane extends its expiry without changing the task attempt or device assignment
|
||||
|
||||
#### Scenario: Lease is stale or foreign
|
||||
- **WHEN** a Host Agent attempts to renew an expired, replaced, or differently owned lease
|
||||
- **THEN** the control plane returns a conflict and does not revive or alter the current attempt
|
||||
|
||||
#### Scenario: Renewal response signals a pending cancellation
|
||||
- **WHEN** the control plane's renewal response for an active lease indicates a pending cancellation request
|
||||
- **THEN** the Host Agent stops further planned actions at the next available step boundary, the same way it stops on lease loss
|
||||
|
||||
### Requirement: Terminal result reporting is idempotent
|
||||
The Host Agent SHALL report a terminal result using the task, attempt, and lease identifiers, and repeating the same report SHALL return the already recorded outcome without duplicating state transitions. The terminal result SHALL be `done`, `failed`, or `cancelled`.
|
||||
|
||||
#### Scenario: Report a successful result
|
||||
- **WHEN** the active lease owner reports successful completion
|
||||
- **THEN** the control plane marks the scheduled task done, releases the device reservation, and records the result metadata
|
||||
|
||||
#### Scenario: Report a cancelled result
|
||||
- **WHEN** the active lease owner reports that its execution stopped because of an observed cancellation request
|
||||
- **THEN** the control plane marks the scheduled task cancelled, releases the device reservation, and records the result metadata
|
||||
|
||||
#### Scenario: Retry a result after response loss
|
||||
- **WHEN** the Host Agent repeats the identical terminal report for an already completed active lease
|
||||
- **THEN** the control plane returns the recorded terminal result without creating a new attempt or error
|
||||
|
||||
#### Scenario: Stale attempt reports after requeue
|
||||
- **WHEN** an expired earlier attempt reports after a newer attempt has been created
|
||||
- **THEN** the control plane rejects the stale report and preserves the newer attempt's state
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Host distinguishes cancellation stop from lease-loss stop when reporting outcome
|
||||
The Host Agent SHALL track whether its active execution stopped because of an observed cancellation request or for another stop reason (lost or rejected lease), and SHALL report `cancelled` only in the cancellation case, reporting `failed` for other stop reasons.
|
||||
|
||||
#### Scenario: Stop triggered by cancellation
|
||||
- **WHEN** the Host Agent's collaborative-stop mechanism is triggered by a renewal response signaling a pending cancellation
|
||||
- **THEN** the terminal result it reports for that attempt is `cancelled`
|
||||
|
||||
#### Scenario: Stop triggered by lease loss
|
||||
- **WHEN** the Host Agent's collaborative-stop mechanism is triggered by a rejected or lost lease unrelated to any cancellation signal
|
||||
- **THEN** the terminal result it reports for that attempt is `failed`, not `cancelled`
|
||||
@@ -0,0 +1,54 @@
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Task submission and status via the SDK
|
||||
The system SHALL allow an external integrator to submit a task (goal or workflow reference plus constraints) through the platform SDK's API, to query that task's current status by id, and to request cancellation of that task by id, backed by the `task-scheduler` capability.
|
||||
|
||||
#### Scenario: Submit a task via the API
|
||||
- **WHEN** an integrator calls the task-submission endpoint with a valid goal and optional constraints
|
||||
- **THEN** the API returns a task id that can be used to poll status, and the underlying `task-scheduler` records a new `queued` `ScheduledTask`
|
||||
|
||||
#### Scenario: Query status of a known task
|
||||
- **WHEN** an integrator requests status for a task id that exists
|
||||
- **THEN** the API returns that task's current status (`queued`, `assigned`, `dispatched`, `done`, `failed`, or `cancelled`)
|
||||
|
||||
#### Scenario: Query status of an unknown task
|
||||
- **WHEN** an integrator requests status for a task id that does not exist
|
||||
- **THEN** the API returns a not-found response rather than an unhandled server error
|
||||
|
||||
#### Scenario: Cancel a known task via the API
|
||||
- **WHEN** an integrator with the required scope calls the cancel endpoint for a task id that exists and is not already `done` or `failed`
|
||||
- **THEN** the API accepts the request and the underlying `task-scheduler` records the cancellation per its immediate or collaborative rules for that task's current status
|
||||
|
||||
#### Scenario: Cancel an unknown task
|
||||
- **WHEN** an integrator calls the cancel endpoint for a task id that does not exist
|
||||
- **THEN** the API returns a not-found response rather than an unhandled server error
|
||||
|
||||
### Requirement: Public API operations enforce scopes
|
||||
The public platform API SHALL require operation-specific scopes, including task submission, task cancellation, task reading, pool reading, plugin reading, and plugin administration.
|
||||
|
||||
#### Scenario: Submit token has task scope
|
||||
- **WHEN** a principal with `tasks:submit` calls the task-submission endpoint
|
||||
- **THEN** the request is authorized subject to normal task validation
|
||||
|
||||
#### Scenario: Submit-scoped token cancels a task
|
||||
- **WHEN** a principal with `tasks:submit` calls the task-cancellation endpoint for any task id
|
||||
- **THEN** the request is authorized; the platform SDK does not restrict cancellation to the task's original submitter, since no per-task submitter identity is tracked
|
||||
|
||||
#### Scenario: Read-only token attempts cancellation
|
||||
- **WHEN** a principal that holds only `tasks:read` calls the task-cancellation endpoint
|
||||
- **THEN** the API rejects the request before contacting the scheduler
|
||||
|
||||
#### Scenario: Non-admin token attempts plugin registration
|
||||
- **WHEN** an authenticated principal without `plugins:admin` calls plugin registration
|
||||
- **THEN** the API rejects the request before resolving or loading the plugin target
|
||||
|
||||
### Requirement: Python SDK client mirrors the REST API
|
||||
The system SHALL provide a Python client (`CloudClient`) exposing methods corresponding to each `/v1/...` route (submit task, get task status, cancel task, list devices, list hosts, list plugins, register plugin), so integrators do not need to hand-construct HTTP requests.
|
||||
|
||||
#### Scenario: Client submits a task and retrieves status
|
||||
- **WHEN** a caller uses `CloudClient` to submit a task and then fetch its status by the returned id
|
||||
- **THEN** the client's methods produce the same result as calling the corresponding `/v1/...` endpoints directly over HTTP
|
||||
|
||||
#### Scenario: Client cancels a task
|
||||
- **WHEN** a caller uses `CloudClient` to cancel a task by id
|
||||
- **THEN** the client's method produces the same result as calling the cancel endpoint directly over HTTP
|
||||
@@ -0,0 +1,59 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Queued task cancellation is immediate
|
||||
The system SHALL, when a cancellation is requested against a task in status `queued`, transition that task directly to status `cancelled` synchronously within the same request, without contacting any Host.
|
||||
|
||||
#### Scenario: Cancel a task that has not been assigned
|
||||
- **WHEN** an authorized caller requests cancellation of a task whose status is `queued`
|
||||
- **THEN** the task's status becomes `cancelled` in the same request and no assignment or lease is ever created for it
|
||||
|
||||
### Requirement: In-flight task cancellation is a durable, collaborative request
|
||||
The system SHALL, when a cancellation is requested against a task in status `assigned` or `dispatched`, durably record a cancellation request against that task rather than immediately marking it `cancelled`, and SHALL surface that pending request to the owning Host Agent no later than its next lease renewal.
|
||||
|
||||
#### Scenario: Cancel a task currently executing on a Host
|
||||
- **WHEN** an authorized caller requests cancellation of a task whose status is `dispatched`
|
||||
- **THEN** the system records the cancellation request against the task's current attempt, the task's status remains `dispatched` until the Host reports a terminal result, and the request survives a control-plane restart
|
||||
|
||||
#### Scenario: Owning Host observes the pending cancellation at lease renewal
|
||||
- **WHEN** the Host Agent executing the task renews its lease after a cancellation request was recorded
|
||||
- **THEN** the renewal response signals the pending cancellation and the Host Agent stops further planned actions at the next available step boundary
|
||||
|
||||
#### Scenario: Cancellation is not instantaneous
|
||||
- **WHEN** a cancellation is requested against a `dispatched` task
|
||||
- **THEN** the system does not guarantee the task reaches status `cancelled` before the owning Host's next lease-renewal cycle completes
|
||||
|
||||
### Requirement: Host reports a cancelled outcome distinct from a failed outcome
|
||||
The Host Agent SHALL report a terminal status of `cancelled`, distinct from `failed`, when its active execution stopped because of an observed cancellation request rather than a lease loss or an execution error, and the control plane SHALL record that task as status `cancelled`.
|
||||
|
||||
#### Scenario: Execution stops due to a cancellation request
|
||||
- **WHEN** the Host Agent's active `TaskRunner` or `WorkflowRunner` execution stops because a lease renewal signaled a pending cancellation
|
||||
- **THEN** the Host Agent reports terminal status `cancelled`, and the control plane transitions the task to status `cancelled` and releases its device reservation
|
||||
|
||||
#### Scenario: Execution stops due to lease loss unrelated to cancellation
|
||||
- **WHEN** the Host Agent's active execution stops because its lease was rejected or lost for a reason other than a pending cancellation
|
||||
- **THEN** the Host Agent reports terminal status `failed`, not `cancelled`
|
||||
|
||||
### Requirement: Cancellation requests are idempotent
|
||||
The system SHALL treat a repeated cancellation request against a task that already has a pending or completed cancellation as a no-op that returns the task's current status, rather than as an error.
|
||||
|
||||
#### Scenario: Cancel a task twice
|
||||
- **WHEN** an authorized caller requests cancellation of a task that already has a pending cancellation request recorded
|
||||
- **THEN** the system returns the same successful response as the first request without creating a duplicate cancellation record
|
||||
|
||||
#### Scenario: Cancel an already-cancelled task
|
||||
- **WHEN** an authorized caller requests cancellation of a task whose status is already `cancelled`
|
||||
- **THEN** the system returns success reflecting the `cancelled` status without error
|
||||
|
||||
### Requirement: Cancellation is rejected for tasks already in a terminal, non-cancelled state
|
||||
The system SHALL reject a cancellation request against a task whose status is already `done` or `failed` with a clear conflict error, without altering that task's recorded outcome.
|
||||
|
||||
#### Scenario: Cancel a completed task
|
||||
- **WHEN** an authorized caller requests cancellation of a task whose status is `done`
|
||||
- **THEN** the system rejects the request with a conflict error and the task's status and result remain unchanged
|
||||
|
||||
### Requirement: An expiring lease on a task with a pending cancellation resolves to cancelled, not requeued
|
||||
The system SHALL, when an active lease expires on a task that has a pending cancellation request, mark that task `cancelled` rather than returning it to `queued` for a further attempt.
|
||||
|
||||
#### Scenario: Lease expires while a cancellation is pending
|
||||
- **WHEN** the active lease on a `dispatched` task with a pending cancellation request expires before a terminal result is reported
|
||||
- **THEN** the task transitions to status `cancelled` and its device reservation is released, instead of being requeued for another attempt
|
||||
@@ -0,0 +1,54 @@
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Terminal transitions validate the active lease
|
||||
The system SHALL accept a `done`, `failed`, or `cancelled` result only from the current active task attempt and lease and SHALL make repeated identical terminal reports idempotent.
|
||||
|
||||
#### Scenario: Active lease reports completion
|
||||
- **WHEN** the active lease owner reports a terminal result
|
||||
- **THEN** the task transitions once to done, failed, or cancelled and releases its device reservation
|
||||
|
||||
#### Scenario: Superseded lease reports completion
|
||||
- **WHEN** a result references a lease superseded by expiry and retry
|
||||
- **THEN** the result is rejected and cannot overwrite the current task attempt
|
||||
|
||||
### Requirement: Expired attempts follow bounded retry policy
|
||||
The system SHALL detect expired assigned or dispatched leases and SHALL either requeue the task with its reservation released, mark it failed when the configured attempt limit is reached, or mark it cancelled when a cancellation request is pending against it.
|
||||
|
||||
#### Scenario: Lease expires with attempts remaining
|
||||
- **WHEN** an active lease expires before a terminal result and the task has remaining attempts and no pending cancellation request
|
||||
- **THEN** the task returns to queued, the previous device reservation is released, and the expired attempt remains auditable
|
||||
|
||||
#### Scenario: Lease expires at attempt limit
|
||||
- **WHEN** an active lease expires and the task has reached its maximum attempts
|
||||
- **THEN** the task becomes failed with a lease-expiry reason and its device reservation is released
|
||||
|
||||
#### Scenario: Lease expires with a cancellation pending
|
||||
- **WHEN** an active lease expires on a task that has a pending cancellation request, regardless of remaining attempts
|
||||
- **THEN** the task becomes cancelled rather than being requeued or marked failed, and its device reservation is released
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Task status includes a reachable cancelled value
|
||||
The `ScheduledTaskStatus` SHALL include `cancelled` as a terminal status reachable from `queued`, `assigned`, or `dispatched`, alongside the existing `done` and `failed` terminal statuses.
|
||||
|
||||
#### Scenario: Cancelled status is a valid terminal state
|
||||
- **WHEN** a task's cancellation completes, whether immediately from `queued` or after collaborative stop from `assigned`/`dispatched`
|
||||
- **THEN** the task's status is `cancelled`, and no further assignment, claim, or lease-renewal operation is accepted against it
|
||||
|
||||
### Requirement: Cancellation requests are recorded durably against in-flight tasks
|
||||
The scheduler repository SHALL persist a cancellation request against an `assigned` or `dispatched` task's current attempt such that the request is observable across a control-plane process restart, before the task reaches a terminal status.
|
||||
|
||||
#### Scenario: Cancellation request survives a restart
|
||||
- **WHEN** a cancellation request is recorded against a `dispatched` task and the control plane process restarts before the Host next renews its lease
|
||||
- **THEN** the pending cancellation request is still present and is surfaced to the Host on its next renewal after restart
|
||||
|
||||
### Requirement: Lease renewal surfaces a pending cancellation request
|
||||
The scheduler repository's lease-renewal operation SHALL report whether the renewing attempt has a pending cancellation request, without altering the normal lease-extension outcome.
|
||||
|
||||
#### Scenario: Renewal on a task with a pending cancellation
|
||||
- **WHEN** the owning host renews the lease for an attempt that has a pending cancellation request
|
||||
- **THEN** the lease is extended normally and the renewal result additionally indicates the pending cancellation
|
||||
|
||||
#### Scenario: Renewal on a task without a pending cancellation
|
||||
- **WHEN** the owning host renews the lease for an attempt with no pending cancellation request
|
||||
- **THEN** the lease is extended normally and the renewal result indicates no pending cancellation
|
||||
@@ -0,0 +1,67 @@
|
||||
## 1. Core and Runtime status plumbing
|
||||
|
||||
- [x] 1.1 Confirm `core/models.py`'s `TaskStatus` already includes `"cancelled"` (it does); add `StepStatus` no change needed — verify no other literal needs widening.
|
||||
- [x] 1.2 Add an optional `stop_reason: Callable[[], str] | None` parameter to `TaskRunner.run()` (`runtime/task.py`), defaulting to `None`.
|
||||
- [x] 1.3 Update `TaskRunner._interrupt_task()` to accept the resolved reason string, set `status="cancelled"` when the reason indicates cancellation (e.g. contains `"cancel"`), else keep `status="failed"` as today, and record the reason as `failure_reason` in both cases.
|
||||
- [x] 1.4 Update `WorkflowRunner`'s stop branch (`workflow/runner.py`) to accept and pass through the same optional `stop_reason`, reusing its existing `"cancelled"` checkpoint call — confirm no behavior change needed since it already lands on `"cancelled"` for any stop; only wire the reason through for consistency/logging. (Correction during implementation: WorkflowRunner previously collapsed *every* stop, including lease-loss, into `"cancelled"`, which contradicts the host-agent-protocol spec's requirement to distinguish cancellation from lease-loss stops. `_stop_status()` now branches on `stop_reason` the same way `TaskRunner` does, while preserving the exact pre-existing default (`"cancelled"`) when no `stop_reason` is supplied.)
|
||||
- [x] 1.5 Add/extend unit tests in `tests/` for `TaskRunner` covering: cancellation-flavored stop → `status="cancelled"`; lease-loss-flavored stop → `status="failed"` (existing behavior preserved). Also added matching `WorkflowRunner` coverage for the same `_stop_status` branching.
|
||||
|
||||
## 2. Cloud persistence: schema and repository
|
||||
|
||||
- [x] 2.1 Add Alembic migration `0012_task_cancellation.py` under `packages/cloud-platform/cloud/migrations/versions/`: add nullable `cancel_requested_at` column to `scheduled_tasks`, matching the style of `0008_task_progress_columns`.
|
||||
- [x] 2.2 Add `cancel_requested_at` to `ScheduledTaskRow` (`db_models.py`) and to the `ScheduledTask` dataclass (`cloud/scheduler.py`).
|
||||
- [x] 2.3 Widen `ScheduledTaskStatus` (`cloud/scheduler.py`) to include `"cancelled"`.
|
||||
- [x] 2.4 Widen `TerminalTaskStatus` and `record_task_result`'s accepted status literal (`repository.py` Protocol, `sql_repository.py`) to include `"cancelled"`; ensure the idempotency logic (`already_recorded`/`conflict`) treats a repeated `"cancelled"` report the same way it treats repeated `"done"`/`"failed"` reports today.
|
||||
- [x] 2.5 Add `CancellationRequestStatus = Literal["requested", "already_terminal", "already_requested", "not_found"]` and a `request_task_cancellation(task_id, *, requested_at) -> CancellationRequestStatus` method to the `CloudRepository` Protocol.
|
||||
- [x] 2.6 Implement `request_task_cancellation` in `SQLAlchemyCloudRepository`: for `queued` tasks, transition directly to `status="cancelled"`; for `assigned`/`dispatched` tasks, set `cancel_requested_at` if unset (return `"already_requested"` if already set); for `done`/`failed`/`cancelled` tasks, return `"already_terminal"`/success-idempotent as appropriate per design D7; for unknown task id, return `"not_found"`.
|
||||
- [x] 2.7 Extend `renew_lease` (`sql_repository.py`) to read `cancel_requested_at` on the current row and report whether it is set, without changing its existing lease-extension/progress-update behavior.
|
||||
- [x] 2.8 Update `reap_expired_leases` so a task whose `cancel_requested_at` is set resolves to `status="cancelled"` (clearing `cancel_requested_at`) instead of being requeued to `queued`, regardless of remaining attempts.
|
||||
- [x] 2.9 Ensure `record_task_result` and the immediate `queued`-cancellation path both clear `cancel_requested_at` on reaching any terminal status.
|
||||
- [x] 2.10 Add/extend repository-level tests (SQLite, matching existing test style) covering: immediate queued cancel; durable cancel request on assigned/dispatched surviving a simulated restart (re-fetch); renewal reporting the pending flag; expired lease with pending cancellation resolving to `cancelled`; idempotent repeat cancel calls; cancel rejected on `done`/`failed`.
|
||||
|
||||
## 3. Internal Host↔Cloud protocol
|
||||
|
||||
- [x] 3.1 Add `cancel_requested: bool = False` field to `LeaseRenewalResponse` (`internal_api/models.py`).
|
||||
- [x] 3.2 Widen `TerminalResultRequest.status` (`internal_api/models.py`) to `Literal["done", "failed", "cancelled"]`.
|
||||
- [x] 3.3 Update `renew_assignment` route (`internal_api/api.py`) to populate `LeaseRenewalResponse.cancel_requested` from the repository's `renew_lease` result.
|
||||
- [x] 3.4 Update `report_result` route (`internal_api/api.py`) to accept and forward the `"cancelled"` status to `record_task_result`.
|
||||
- [x] 3.5 Add/extend internal API tests covering a renewal response surfacing `cancel_requested=True` and a `"cancelled"` terminal report being accepted and idempotent on repeat.
|
||||
|
||||
## 4. Host Agent collaborative stop
|
||||
|
||||
- [x] 4.1 Extend `LeaseGuard` (`host_agent/lease.py`) with a `reason` attribute already implied by `mark_lost(reason)` — confirm it's readable, add a `is_cancellation` helper or convention (e.g. reason string prefix) to distinguish cancellation from other lost-lease reasons.
|
||||
- [x] 4.2 Update `ActiveAssignmentRunner._renew_while_running()` (`host_agent/lease.py`): when `response.cancel_requested` is true, call `guard.mark_lost("cancellation requested by control plane")` instead of continuing the renewal loop.
|
||||
- [x] 4.3 Update `client.py`'s `renew()` to ensure `LeaseRenewalResponse.cancel_requested` deserializes correctly (should be automatic via Pydantic model update, but add a test).
|
||||
- [x] 4.4 Update `AssignmentExecutor._execute_goal()` and `_execute_workflow()` (`host_agent/assignment.py`) to map a cancellation-flavored stop to `AssignmentExecutionResult.status = "cancelled"` (new value alongside `"done"`/`"failed"`), reading the underlying `Task`/`WorkflowRun` status (`"cancelled"`) instead of collapsing it to `"failed"`.
|
||||
- [x] 4.5 Update `AssignmentProcessor.process()` (`host_agent/processor.py`) so its `status = "done" if execution.status == "done" else "failed"` mapping becomes a three-way mapping that preserves `"cancelled"`, and `report_result` is called with `status="cancelled"` in that case.
|
||||
- [x] 4.6 Add/extend Host Agent tests covering: a renewal response with `cancel_requested=True` stops the active `should_stop`-driven loop; the resulting `AssignmentExecutionResult.status` and reported terminal status are `"cancelled"`; a lease-loss stop unrelated to cancellation still reports `"failed"`.
|
||||
|
||||
## 5. Public SDK endpoint
|
||||
|
||||
- [x] 5.1 Add `POST /v1/tasks/{task_id}/cancel` route to `cloud/sdk/api.py`, scope-gated by `TASKS_SUBMIT_SCOPE`, calling the new `TaskScheduler`/repository cancellation operation.
|
||||
- [x] 5.2 Add `TaskCancellationResponse {task_id, status}` model to `cloud/sdk/models.py` (or wherever SDK response models live); return `200 OK` for immediate/already-terminal-cancelled idempotent cases, `202 Accepted` for a newly recorded pending cancellation, `404` for unknown task id, `409 Conflict` for a `done`/`failed` task.
|
||||
- [x] 5.3 Widen the `status_filter` `Literal` on `list_tasks` (`cloud/sdk/api.py`) to include `"cancelled"`.
|
||||
- [x] 5.4 Add a `cancel_task(task_id)` method to `CloudClient` (`cloud/sdk/client.py`) mirroring the new route.
|
||||
- [x] 5.5 Add/extend SDK-level tests: submit-scoped caller cancels a queued task (200, immediate); cancels a dispatched task (202, pending); read-only-scoped caller rejected before reaching the scheduler; cancel on unknown id (404); cancel on terminal id (409); repeat cancel calls idempotent (200).
|
||||
|
||||
## 6. Frontend: Cloud Console
|
||||
|
||||
- [x] 6.1 Add `"cancelled"` to `TaskStatus` in `cloud-console/src/types.ts` and to the `STATUSES` array in `TasksView.vue`.
|
||||
- [x] 6.2 Add a `cancelTask(taskId)` method to `cloud-console/src/api.ts`.
|
||||
- [x] 6.3 Add a "Cancel" button to `TasksView.vue`'s task detail panel, visible only when the selected task's status is `queued`/`assigned`/`dispatched` and the operator's token has `tasks:submit`; on click, call `cancelTask` and refresh the displayed task.
|
||||
- [x] 6.4 Add/extend Cloud Console component tests (existing test style) covering: Cancel button visibility per status/scope; successful cancel updates displayed status; error response is surfaced without falsely showing cancelled. (Project has no Vue component-mounting test harness — `@vue/test-utils` isn't a dependency and no existing test exercises a `.vue` file directly. Followed the established pattern instead: extracted the visibility rule into a pure, unit-tested `taskCancellation.ts` module — mirroring `taskProgress.ts`/`plannerHistory.ts` — covering cancellable vs. terminal statuses and the `tasks:submit` scope gate. `cancelSelectedTask` in `TasksView.vue` only mutates `selectedTask.status` on a successful response and routes failures through the existing `handleError`/`errorMessage` path, so an error never flips the displayed status to cancelled.)
|
||||
|
||||
## 7. Frontend: Host Agent local console
|
||||
|
||||
- [x] 7.1 Add a `POST /tasks/{id}/cancel` route to the Host Agent local console (`host_agent/web/app.py`) that calls through to the same cancellation path used by the collaborative-stop mechanism for a locally-tracked task, consistent with existing local console read routes. (Correction during implementation: the Host Agent's own internal-API bearer credential — issued by `RepositoryHostAuthProvider` — carries an empty `scopes` frozenset and is authorized only via `Principal.require_host()` identity checks, not scopes. It therefore cannot call the public SDK's `tasks:submit`-scoped `POST /v1/tasks/{task_id}/cancel` endpoint from design.md/section 5. Added a new internal API route, `POST /internal/v1/hosts/{host_id}/tasks/{task_id}/cancel` (`cloud/internal_api/api.py`), authenticated the same way as the existing host self-submission route (`authorize_host`), with an added ownership check rejecting tasks whose `constraints.target_host_id != host_id` with 404. This mirrors the pre-existing pattern where hosts already self-serve create/execute their own tasks over this credential, and does not conflict with design.md's Non-Goal — that constraint (no per-task ACL; anyone with `tasks:submit` can cancel any task) is scoped to the public SDK layer, not the internal Host↔Cloud API. `HostAgentClient.cancel_task()` calls this new route directly rather than proxying to the public SDK.)
|
||||
- [x] 7.2 Add a Cancel button to `templates/task_detail.html` for tasks not yet in a terminal state.
|
||||
- [x] 7.3 Add/extend local console tests covering the new route and template rendering.
|
||||
|
||||
## 8. Docs
|
||||
|
||||
- [x] 8.1 Add a short section to `docs/CLOUD_DEPLOYMENT.md` documenting that cancellation is collaborative (not instantaneous) for `assigned`/`dispatched` tasks, bounded by roughly one third of the configured lease duration, with immediate effect for `queued` tasks.
|
||||
|
||||
## 9. End-to-end verification
|
||||
|
||||
- [x] 9.1 Run the full test suite (`uv run pytest` at repo root, plus `cloud-console` frontend tests) and confirm no regressions in existing task-scheduler, host-agent-protocol, platform-sdk, or workflow-orchestration tests. (869 passed, 50 skipped, 4 pre-existing failures unrelated to this change — `test_verifier_against_real_llm`, `test_reflector_against_real_llm`, `test_real_anthropic_ai_planner_selects_a_tool`, `test_real_anthropic_semantic_enrichment_returns_schema_valid_scene` all require live Anthropic API network access and fail the same way on `master`. `cloud-console`: 27 tests passed, `vue-tsc --noEmit` typecheck clean.)
|
||||
- [x] 9.2 Manually or via an integration test, exercise the full path: submit a task, cancel a `queued` task (immediate), submit and dispatch another task, cancel it mid-execution, and confirm it reaches `cancelled` within one lease-renewal cycle with `cancelled` visible in both the public API and the Cloud Console. (Added `test_cancellation_full_path_queued_immediate_and_dispatched_collaborative` in `tests/test_cloud_sdk_api.py`: submits and immediately cancels a queued task via `POST /v1/tasks/{id}/cancel` (200, `cancelled`); submits, dispatches, and cancels a second task mid-execution (202, pending); drives one lease renewal confirming `cancel_requested=True` is surfaced; reports a `cancelled` terminal result as the Host Agent would; and confirms the task shows `status="cancelled"` via both `GET /v1/tasks/{id}` and `GET /v1/tasks?status=cancelled`. The Cloud Console reads task status through this same public API and its `cancelled` rendering is covered by the Task 6.4 `taskCancellation.ts` unit tests, so this repository-to-API round trip is the full path exercised at the automated-test layer; no manual browser session was run.)
|
||||
Reference in New Issue
Block a user