- test_cancellation_full_path_queued_immediate_and_dispatched_collaborative exercises the full public-API cancellation path: immediate cancel of a queued task, collaborative cancel of a dispatched task surfaced through lease renewal and a cancelled terminal report, and visibility of the cancelled status via both the get and list endpoints. - Full backend suite (869 passed, 50 skipped) and cloud-console frontend suite (27 passed) + typecheck show no regressions; the only failures are 4 pre-existing live-LLM integration tests unrelated to this change.
12 KiB
12 KiB
1. Core and Runtime status plumbing
- 1.1 Confirm
core/models.py'sTaskStatusalready includes"cancelled"(it does); addStepStatusno change needed — verify no other literal needs widening. - 1.2 Add an optional
stop_reason: Callable[[], str] | Noneparameter toTaskRunner.run()(runtime/task.py), defaulting toNone. - 1.3 Update
TaskRunner._interrupt_task()to accept the resolved reason string, setstatus="cancelled"when the reason indicates cancellation (e.g. contains"cancel"), else keepstatus="failed"as today, and record the reason asfailure_reasonin both cases. - 1.4 Update
WorkflowRunner's stop branch (workflow/runner.py) to accept and pass through the same optionalstop_reason, reusing its existing"cancelled"checkpoint call — confirm no behavior change needed since it already lands on"cancelled"for any stop; only wire the reason through for consistency/logging. (Correction during implementation: WorkflowRunner previously collapsed every stop, including lease-loss, into"cancelled", which contradicts the host-agent-protocol spec's requirement to distinguish cancellation from lease-loss stops._stop_status()now branches onstop_reasonthe same wayTaskRunnerdoes, while preserving the exact pre-existing default ("cancelled") when nostop_reasonis supplied.) - 1.5 Add/extend unit tests in
tests/forTaskRunnercovering: cancellation-flavored stop →status="cancelled"; lease-loss-flavored stop →status="failed"(existing behavior preserved). Also added matchingWorkflowRunnercoverage for the same_stop_statusbranching.
2. Cloud persistence: schema and repository
- 2.1 Add Alembic migration
0012_task_cancellation.pyunderpackages/cloud-platform/cloud/migrations/versions/: add nullablecancel_requested_atcolumn toscheduled_tasks, matching the style of0008_task_progress_columns. - 2.2 Add
cancel_requested_attoScheduledTaskRow(db_models.py) and to theScheduledTaskdataclass (cloud/scheduler.py). - 2.3 Widen
ScheduledTaskStatus(cloud/scheduler.py) to include"cancelled". - 2.4 Widen
TerminalTaskStatusandrecord_task_result's accepted status literal (repository.pyProtocol,sql_repository.py) to include"cancelled"; ensure the idempotency logic (already_recorded/conflict) treats a repeated"cancelled"report the same way it treats repeated"done"/"failed"reports today. - 2.5 Add
CancellationRequestStatus = Literal["requested", "already_terminal", "already_requested", "not_found"]and arequest_task_cancellation(task_id, *, requested_at) -> CancellationRequestStatusmethod to theCloudRepositoryProtocol. - 2.6 Implement
request_task_cancellationinSQLAlchemyCloudRepository: forqueuedtasks, transition directly tostatus="cancelled"; forassigned/dispatchedtasks, setcancel_requested_atif unset (return"already_requested"if already set); fordone/failed/cancelledtasks, return"already_terminal"/success-idempotent as appropriate per design D7; for unknown task id, return"not_found". - 2.7 Extend
renew_lease(sql_repository.py) to readcancel_requested_aton the current row and report whether it is set, without changing its existing lease-extension/progress-update behavior. - 2.8 Update
reap_expired_leasesso a task whosecancel_requested_atis set resolves tostatus="cancelled"(clearingcancel_requested_at) instead of being requeued toqueued, regardless of remaining attempts. - 2.9 Ensure
record_task_resultand the immediatequeued-cancellation path both clearcancel_requested_aton reaching any terminal status. - 2.10 Add/extend repository-level tests (SQLite, matching existing test style) covering: immediate queued cancel; durable cancel request on assigned/dispatched surviving a simulated restart (re-fetch); renewal reporting the pending flag; expired lease with pending cancellation resolving to
cancelled; idempotent repeat cancel calls; cancel rejected ondone/failed.
3. Internal Host↔Cloud protocol
- 3.1 Add
cancel_requested: bool = Falsefield toLeaseRenewalResponse(internal_api/models.py). - 3.2 Widen
TerminalResultRequest.status(internal_api/models.py) toLiteral["done", "failed", "cancelled"]. - 3.3 Update
renew_assignmentroute (internal_api/api.py) to populateLeaseRenewalResponse.cancel_requestedfrom the repository'srenew_leaseresult. - 3.4 Update
report_resultroute (internal_api/api.py) to accept and forward the"cancelled"status torecord_task_result. - 3.5 Add/extend internal API tests covering a renewal response surfacing
cancel_requested=Trueand a"cancelled"terminal report being accepted and idempotent on repeat.
4. Host Agent collaborative stop
- 4.1 Extend
LeaseGuard(host_agent/lease.py) with areasonattribute already implied bymark_lost(reason)— confirm it's readable, add ais_cancellationhelper or convention (e.g. reason string prefix) to distinguish cancellation from other lost-lease reasons. - 4.2 Update
ActiveAssignmentRunner._renew_while_running()(host_agent/lease.py): whenresponse.cancel_requestedis true, callguard.mark_lost("cancellation requested by control plane")instead of continuing the renewal loop. - 4.3 Update
client.py'srenew()to ensureLeaseRenewalResponse.cancel_requesteddeserializes correctly (should be automatic via Pydantic model update, but add a test). - 4.4 Update
AssignmentExecutor._execute_goal()and_execute_workflow()(host_agent/assignment.py) to map a cancellation-flavored stop toAssignmentExecutionResult.status = "cancelled"(new value alongside"done"/"failed"), reading the underlyingTask/WorkflowRunstatus ("cancelled") instead of collapsing it to"failed". - 4.5 Update
AssignmentProcessor.process()(host_agent/processor.py) so itsstatus = "done" if execution.status == "done" else "failed"mapping becomes a three-way mapping that preserves"cancelled", andreport_resultis called withstatus="cancelled"in that case. - 4.6 Add/extend Host Agent tests covering: a renewal response with
cancel_requested=Truestops the activeshould_stop-driven loop; the resultingAssignmentExecutionResult.statusand reported terminal status are"cancelled"; a lease-loss stop unrelated to cancellation still reports"failed".
5. Public SDK endpoint
- 5.1 Add
POST /v1/tasks/{task_id}/cancelroute tocloud/sdk/api.py, scope-gated byTASKS_SUBMIT_SCOPE, calling the newTaskScheduler/repository cancellation operation. - 5.2 Add
TaskCancellationResponse {task_id, status}model tocloud/sdk/models.py(or wherever SDK response models live); return200 OKfor immediate/already-terminal-cancelled idempotent cases,202 Acceptedfor a newly recorded pending cancellation,404for unknown task id,409 Conflictfor adone/failedtask. - 5.3 Widen the
status_filterLiteralonlist_tasks(cloud/sdk/api.py) to include"cancelled". - 5.4 Add a
cancel_task(task_id)method toCloudClient(cloud/sdk/client.py) mirroring the new route. - 5.5 Add/extend SDK-level tests: submit-scoped caller cancels a queued task (200, immediate); cancels a dispatched task (202, pending); read-only-scoped caller rejected before reaching the scheduler; cancel on unknown id (404); cancel on terminal id (409); repeat cancel calls idempotent (200).
6. Frontend: Cloud Console
- 6.1 Add
"cancelled"toTaskStatusincloud-console/src/types.tsand to theSTATUSESarray inTasksView.vue. - 6.2 Add a
cancelTask(taskId)method tocloud-console/src/api.ts. - 6.3 Add a "Cancel" button to
TasksView.vue's task detail panel, visible only when the selected task's status isqueued/assigned/dispatchedand the operator's token hastasks:submit; on click, callcancelTaskand refresh the displayed task. - 6.4 Add/extend Cloud Console component tests (existing test style) covering: Cancel button visibility per status/scope; successful cancel updates displayed status; error response is surfaced without falsely showing cancelled. (Project has no Vue component-mounting test harness —
@vue/test-utilsisn't a dependency and no existing test exercises a.vuefile directly. Followed the established pattern instead: extracted the visibility rule into a pure, unit-testedtaskCancellation.tsmodule — mirroringtaskProgress.ts/plannerHistory.ts— covering cancellable vs. terminal statuses and thetasks:submitscope gate.cancelSelectedTaskinTasksView.vueonly mutatesselectedTask.statuson a successful response and routes failures through the existinghandleError/errorMessagepath, so an error never flips the displayed status to cancelled.)
7. Frontend: Host Agent local console
- 7.1 Add a
POST /tasks/{id}/cancelroute to the Host Agent local console (host_agent/web/app.py) that calls through to the same cancellation path used by the collaborative-stop mechanism for a locally-tracked task, consistent with existing local console read routes. (Correction during implementation: the Host Agent's own internal-API bearer credential — issued byRepositoryHostAuthProvider— carries an emptyscopesfrozenset and is authorized only viaPrincipal.require_host()identity checks, not scopes. It therefore cannot call the public SDK'stasks:submit-scopedPOST /v1/tasks/{task_id}/cancelendpoint from design.md/section 5. Added a new internal API route,POST /internal/v1/hosts/{host_id}/tasks/{task_id}/cancel(cloud/internal_api/api.py), authenticated the same way as the existing host self-submission route (authorize_host), with an added ownership check rejecting tasks whoseconstraints.target_host_id != host_idwith 404. This mirrors the pre-existing pattern where hosts already self-serve create/execute their own tasks over this credential, and does not conflict with design.md's Non-Goal — that constraint (no per-task ACL; anyone withtasks:submitcan cancel any task) is scoped to the public SDK layer, not the internal Host↔Cloud API.HostAgentClient.cancel_task()calls this new route directly rather than proxying to the public SDK.) - 7.2 Add a Cancel button to
templates/task_detail.htmlfor tasks not yet in a terminal state. - 7.3 Add/extend local console tests covering the new route and template rendering.
8. Docs
- 8.1 Add a short section to
docs/CLOUD_DEPLOYMENT.mddocumenting that cancellation is collaborative (not instantaneous) forassigned/dispatchedtasks, bounded by roughly one third of the configured lease duration, with immediate effect forqueuedtasks.
9. End-to-end verification
- 9.1 Run the full test suite (
uv run pytestat repo root, pluscloud-consolefrontend tests) and confirm no regressions in existing task-scheduler, host-agent-protocol, platform-sdk, or workflow-orchestration tests. (869 passed, 50 skipped, 4 pre-existing failures unrelated to this change —test_verifier_against_real_llm,test_reflector_against_real_llm,test_real_anthropic_ai_planner_selects_a_tool,test_real_anthropic_semantic_enrichment_returns_schema_valid_sceneall require live Anthropic API network access and fail the same way onmaster.cloud-console: 27 tests passed,vue-tsc --noEmittypecheck clean.) - 9.2 Manually or via an integration test, exercise the full path: submit a task, cancel a
queuedtask (immediate), submit and dispatch another task, cancel it mid-execution, and confirm it reachescancelledwithin one lease-renewal cycle withcancelledvisible in both the public API and the Cloud Console. (Addedtest_cancellation_full_path_queued_immediate_and_dispatched_collaborativeintests/test_cloud_sdk_api.py: submits and immediately cancels a queued task viaPOST /v1/tasks/{id}/cancel(200,cancelled); submits, dispatches, and cancels a second task mid-execution (202, pending); drives one lease renewal confirmingcancel_requested=Trueis surfaced; reports acancelledterminal result as the Host Agent would; and confirms the task showsstatus="cancelled"via bothGET /v1/tasks/{id}andGET /v1/tasks?status=cancelled. The Cloud Console reads task status through this same public API and itscancelledrendering is covered by the Task 6.4taskCancellation.tsunit tests, so this repository-to-API round trip is the full path exercised at the automated-test layer; no manual browser session was run.)