feat(api): server-rendered Jinja2 Runtime console at /ui/

Replaces the separate Vue/Vite `console/` SPA with a same-origin,
server-rendered console built on a module-level Jinja2 Environment
with select_autoescape(["html","xml"]).

- Add api/console_web.py with /ui/ routes (dashboard, tasks, task
  detail/timeline, config) and a _status_fragment polled every 10s.
- Refactor api/console.py into a typed ConsoleService shared by the
  JSON and HTML routers so validation/persistence cannot drift.
- Remove RUNTIME_CONSOLE_STATIC_DIR, SpaStaticFiles, and the wildcard
  CORS middleware from api/rest.py; GET / now redirects to /ui/.
- Delete the top-level console/ project; add jinja2 and python-multipart
  as direct dependencies and ship templates/CSS/JS via package-data.
- Add 31 tests (XSS probes, PRG flows, fragment refresh, no-static-dir
  and no-CORS regressions, wheel-packaging smoke test).

/console/* JSON endpoints remain unchanged. The console keeps the
trusted-network-only boundary; auth/CSRF is intentionally deferred.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-07-15 08:03:13 +08:00
co-authored by Claude Opus 4.6
parent 56f3f96363
commit e00c50e703
39 changed files with 1890 additions and 2228 deletions
@@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-07-14
@@ -0,0 +1,233 @@
## Context
The root Runtime API currently exposes console state and mutations through
`api/console.py` under `/console/*`. Its human interface is a separate
`console/` Vue/Vite SPA. `api/rest.py` only serves that SPA when
`RUNTIME_CONSOLE_STATIC_DIR` points to a built distribution, uses a
SPA-specific 404 fallback, and enables wildcard CORS for cross-origin Vite
development.
The existing `web-console` change is still unarchived and contains the
opposite design decision: an independent SPA with no backend templates. There
is no canonical Runtime-console spec in `openspec/specs/`, so this change
introduces a new capability rather than modifying a pending change's delta
spec. The Host Agent already uses Jinja2, but it is a distinct application
with local-account authentication and must not become a dependency of the
Runtime API.
The Runtime layering rule requires all HTTP, template, static-asset, and form
parsing concerns to remain in `api`. `core`, `driver`, `device`, `tools`,
`perception`, `storage`, and `runtime` retain their current framework-free
contracts.
## Goals / Non-Goals
**Goals:**
- Serve the Runtime operator console from the same FastAPI process at `/ui/`
without Node, Vite, a prebuilt SPA directory, or a second development
process.
- Preserve the observable behavior of the existing device, task, timeline,
and runtime-config console workflows, while retaining `/console/*` JSON
endpoints for programmatic clients.
- Ensure every HTML interpolation is protected by Jinja2 autoescaping and
ship templates and assets in the Runtime wheel.
- Keep mutations server-owned through ordinary POST/Redirect/GET form flows
and reuse one API-local implementation for page and JSON routes.
**Non-Goals:**
- Add authentication, authorization, sessions, CSRF protection, rate limits,
or a public-network deployment model for the Runtime console.
- Change `/agent/task`, `/devices`, MCP, `DeviceManager`, `TaskRunner`,
`TaskMetadataStore`, `Timeline`, or persisted data formats.
- Add new driver types, editable planner settings, Cloud Console features, or
Host Agent console features.
- Retain a generic client-side application framework or a Node build pipeline.
- Edit or archive the pending `web-console` change as part of this change.
## Decisions
### D1: Add a Runtime page router in `api/console_web.py`
`api/console_web.py` will expose `create_console_web_router()` with the `/ui`
prefix. `api/rest.py::create_app()` will build the existing Runtime stores and
manager once, construct the console service once, and include both the JSON
router and the page router in the same FastAPI application. `GET /` will
always redirect to `/ui/` because the console is no longer optional build
output.
The page surface will use stable server routes:
- `GET /ui/` for the device/status dashboard;
- `GET /ui/tasks` for task filtering and browsing;
- `GET /ui/tasks/{task_id}` for one task and its ordered timeline;
- `GET /ui/config` for device registration/removal and `max_steps`;
- POST routes below `/ui/` for device registration, device removal, and
configuration updates.
Successful POST handlers use `303 See Other` redirects. Validation failures
return the originating page with an HTML `400` response and preserved safe
form values. This is preferred over trying to preserve the SPA's client-side
state machine because normal browser navigation and forms are sufficient for
the console's low-frequency operational actions.
An alternative of replacing `/console/*` JSON endpoints with HTML endpoints
was rejected: the JSON API has explicit tests and remains useful to scripts
and future clients. An alternative of a separate FastAPI app was rejected:
it would duplicate the `DeviceManager`, stores, and `TaskRunner` composition.
### D2: Share API-local console operations rather than issuing loopback HTTP
requests
`api/console.py` will gain a small API-local `ConsoleService` (or equivalent
typed operation object) that owns the current console reads and mutations:
device listing/registration/removal, task listing/detail/timeline lookup, and
runtime configuration reads/updates. The existing JSON router and the new
page router receive the same service instance.
The service stays in `api` and delegates to the existing injected
`DeviceManager`, `TaskMetadataStore`, `Timeline`, `DeviceConfigStore`, and
`TaskRunner`. Page handlers MUST NOT make HTTP requests to the process's own
`/console/*` endpoints. This keeps validation, driver allow-listing,
persistence, error mapping, and state mutation single-sourced without moving
web concepts into a lower layer.
Duplicating the route bodies was rejected because future changes could make
JSON and form behavior diverge. Moving the service into `runtime` or
`storage` was rejected because it would introduce HTTP/UI-oriented
application concerns below the API adapter boundary.
### D3: Use a module-level Jinja2 environment and package-owned assets
`api/console_web.py` will construct one module-level Jinja2 `Environment`
using `FileSystemLoader` rooted at `api/templates/runtime_console/` and
`select_autoescape(["html", "xml"])`. Route handlers will render with
`get_template(...).render(...)` and return `HTMLResponse`; they will not
build HTML through f-strings, string concatenation, or post-process rendered
HTML.
Templates will include a shared `base.html`, dashboard, task list, task
detail, configuration page, and any small server-rendered dashboard fragment.
CSS and the narrow polling script will be regular package assets mounted under
`/ui/assets`, not Vite output. The root package will explicitly include the
template, CSS, and JavaScript globs in setuptools package data so an installed
wheel behaves like an editable checkout.
`jinja2>=3.1` and `python-multipart` will be direct root dependencies. The
latter is declared directly even though it is currently available through an
unrelated transitive dependency, because `Request.form()` is part of this
feature's runtime contract.
`starlette.templating.Jinja2Templates` was rejected because direct Jinja2
rendering is explicit, matches the existing Host Agent convention, and avoids
coupling the implementation to helper signature changes. Inline all-CSS HTML
was rejected because a dedicated static stylesheet keeps base templates
readable without restoring a frontend build system.
### D4: Retain server-owned live status with a narrow rendered fragment
The dashboard will keep the current approximately 10-second live-status
refresh through a small static browser script. It fetches an HTML fragment
rendered by the same Jinja2 environment and replaces only the dashboard's
live-status region. The browser does not fetch JSON and recreate console DOM
state; all stateful markup still originates on the server.
Task filters, task selection, timeline step selection, device mutations, and
configuration updates use normal GET/POST navigation. This retains useful
status freshness without reintroducing a SPA framework or making a full-page
reload the only refresh mechanism.
A zero-JavaScript periodic refresh was rejected because it would require full
page reloads and regress the existing dashboard behavior. Reusing the Vue
reactivity layer was rejected because it retains the independent build and
deployment boundary this change removes.
### D5: Preserve JSON compatibility while retiring SPA deployment wiring
The `/console/*` paths, methods, success payloads, error status codes, and
persistence semantics remain unchanged. Existing `tests/test_console_api.py`
continues to be the compatibility baseline. `api/rest.py` removes
`RUNTIME_CONSOLE_STATIC_DIR`, `SpaStaticFiles`, and its SPA-only fallback.
The broad CORS middleware added only for cross-origin Vite development is
removed; same-origin `/ui/` must not depend on it.
The top-level `console/` directory, Vite environment files, package lockfile,
and npm instructions are removed. Root documentation changes to a single
Runtime startup command followed by `/ui/`. The unrelated `cloud-console/`
SPA and its deployment remain unchanged.
Keeping `RUNTIME_CONSOLE_STATIC_DIR` as a deprecated fallback was rejected:
it would preserve an unsupported second rendering path and force every future
console change to be tested twice. A backward-compatible redirect from the
old external Vite development port is impossible because it is a separate
process, so the operator migration is documented as a breaking change.
### D6: Apply autoescaping uniformly and retain the trusted-network boundary
All interpolated device names, IDs, connection values, task goals, failure
reasons, timeline text, JSON-like tool/result data, and errors pass through
the shared autoescaping environment. Structured values use Jinja's `tojson`
filter only in safe text contexts; no current template uses `|safe` or a
global autoescape opt-out. Screenshot bytes remain data sourced from the
existing timeline and are rendered only as the established PNG data URI.
Removing wildcard CORS reduces the old SPA development surface but does not
provide authentication and does not by itself prevent cross-site HTML form
submission. The Runtime console therefore remains documented as
trusted-network-only. Authentication/session/CSRF design is intentionally
separate, so this migration does not create a misleading partial security
model.
## Risks / Trade-offs
- [A task, device name, or JSON value could carry HTML/script content] -> The
shared autoescaping environment, `tojson` for structured output, no safe
bypasses, and XSS regression tests make the protection mechanical.
- [Templates or assets work from a checkout but not an installed wheel] ->
Explicit setuptools package-data rules and a built-wheel smoke test verify
deployment behavior.
- [Removing the npm/Vite workflow disrupts an operator's existing runbook] ->
Mark the removal as breaking, update all Runtime console documentation, and
retain a Git-revert rollback path with no data migration.
- [Removing CORS breaks an undiscovered browser client] -> The existing CORS
configuration was introduced for the deleted Vite development flow; JSON
clients outside a browser remain unaffected. A future browser integration
must add an explicit origin policy rather than restore a wildcard.
- [Base64 screenshots make large task-detail responses expensive] -> Preserve
the existing bounded-task behavior and do not change timeline storage or
transfer format in this rendering migration.
- [The pending `web-console` change still contains a SPA decision and one
manual task] -> Record this change as the rendering-mechanism successor;
reconcile the older change only after the new server-rendered browser
workflow has been manually verified.
## Migration Plan
1. Add direct dependencies and package-data declarations, then create the
Jinja2 environment, templates, static assets, API-local console service,
and `/ui/` page routes while preserving existing JSON-route tests.
2. Add focused template, page-route, mutation/PRG, fragment-refresh, and XSS
tests. Build and install the Runtime wheel in an isolated environment to
verify package resources are present.
3. Remove `console/`, SPA static mounting, the static-directory environment
variable, Vite-specific CORS, and obsolete ignore/configuration files.
4. Update root Runtime and macOS setup documentation to direct operators to
`/ui/`, and document the removed npm/static-directory workflow.
5. Run formatting, linting, non-integration Runtime tests, wheel build/smoke
checks, strict OpenSpec validation, and a browser walkthrough covering
dashboard refresh, task replay, device mutation, and `max_steps` update.
Rollback is a source revert. It restores the Vue sources and static mount if
needed and does not alter device configuration, task metadata, or timeline
data, so no data rollback or schema migration is required.
## Open Questions
- The remaining manual verification task in the pending `web-console` change
must be reconciled before that older change is archived. It is not safe to
mark it complete solely because this proposal exists; the new `/ui/`
browser walkthrough supplies the replacement evidence after implementation.
- Authentication and CSRF protection remain deliberately deferred. Any plan to
expose the Runtime console beyond a trusted local network requires a
separate threat model and change proposal.
@@ -0,0 +1,60 @@
## Why
The local Runtime console is currently a separately built Vue/Vite SPA. It
requires a Node toolchain for development and an optional static-directory
configuration for same-process serving, even though its data and mutations
already live in the Runtime FastAPI process. Rendering the console with Jinja2
will make the operator surface deploy with the Runtime itself while preserving
the existing REST contract for programmatic clients.
## What Changes
- Add a same-origin, server-rendered Runtime console under `/ui/`, with Jinja2
pages for device status, task browsing/detail/timeline replay, device
registration/removal, and runtime configuration.
- Keep the existing `/console/*` JSON endpoints and make page handlers and
JSON handlers share API-layer console operations so their observable
registration, deletion, filtering, and configuration semantics cannot
drift.
- Package console templates and static assets with `device-agent-runtime`, add
direct Jinja2 and HTML form-parsing dependencies, and render every HTML page
through one autoescaping template environment.
- Replace the Vue/Vite `console/` project, `RUNTIME_CONSOLE_STATIC_DIR`, and
SPA fallback static mount with Runtime-owned templates and normal static
assets. Remove the permissive CORS configuration that existed only for
cross-origin Vite development.
- **BREAKING**: the independent `console/` npm workflow and
`RUNTIME_CONSOLE_STATIC_DIR` deployment mode are removed. Operators will
start the Runtime API normally and open `/ui/`; JSON API paths remain
unchanged.
## Capabilities
### New Capabilities
- `runtime-console-template-rendering`: Same-origin Jinja2-rendered Runtime
console pages, automatic HTML escaping, form-based mutations, and packaged
Runtime-owned web assets.
### Modified Capabilities
- None. The existing canonical specs do not define the pending `web-console`
SPA, and the `/console/*` JSON API contract remains unchanged.
## Impact
- Affected code: `api/rest.py`, `api/console.py`, a new API-layer page router,
Runtime template/static asset directories, root `pyproject.toml`, and
console-focused tests.
- Removed code/assets: top-level `console/` Vue/Vite sources, Node lockfile,
Vite environment configuration, and SPA deployment wiring.
- Documentation: Runtime startup and console guidance in `README.md`,
`docs/CONSTITUTION.md`, and `docs/MACOS_IPHONE_SETUP.md` change to describe
the built-in `/ui/` console.
- Security boundary: this change preserves the existing trusted-network,
unauthenticated Runtime console assumption. It does not add authentication,
authorization, or session/CSRF protection; non-trusted exposure needs a
separate security change.
- Architecture: all new HTTP, HTML, and template concerns remain in the outer
`api` layer. No `core`, `driver`, `device`, `tools`, `perception`, or
`runtime` package gains web-framework dependencies.
@@ -0,0 +1,113 @@
## ADDED Requirements
### Requirement: Runtime console pages SHALL be served by the Runtime FastAPI application
The Runtime FastAPI application SHALL serve its operator console from
same-origin `/ui/` routes without requiring a separately running frontend
process, a prebuilt SPA directory, or `RUNTIME_CONSOLE_STATIC_DIR`. `GET /`
SHALL redirect an operator to `/ui/`.
#### Scenario: Open the built-in console without static-directory configuration
- **WHEN** the Runtime application starts without `RUNTIME_CONSOLE_STATIC_DIR`
- **THEN** `GET /` redirects to `/ui/` and `GET /ui/` returns an HTML
dashboard rendered by the Runtime process
#### Scenario: Navigate the operator workflows through page routes
- **WHEN** an operator opens `/ui/tasks`, `/ui/tasks/{task_id}`, or
`/ui/config`
- **THEN** the Runtime returns HTML pages for task browsing, task timeline
detail, and device/runtime configuration respectively
### Requirement: Runtime console HTML SHALL use one autoescaping template environment
Every Runtime console HTML response SHALL be rendered through one
process-wide Jinja2 environment configured to autoescape `.html` and `.xml`
templates. Page handlers SHALL NOT construct HTML through f-strings, string
concatenation, or post-process rendered output to bypass that environment.
#### Scenario: Untrusted device and task values are rendered safely
- **WHEN** a device name, task goal, failure reason, or timeline value contains
`<script>alert(1)</script>`
- **THEN** the rendered console HTML contains an escaped text representation
and contains no script element originating from that value
#### Scenario: A future page inherits HTML autoescaping
- **WHEN** a future Runtime console route renders a `.html` template through
the shared environment
- **THEN** its interpolated values are HTML-escaped without route-specific
escaping configuration
### Requirement: Runtime console pages SHALL preserve console operational workflows
The server-rendered console SHALL let an operator inspect device status, list
and filter tasks, inspect a task's ordered timeline including available
screenshots, register or remove a supported device, and view or update
`max_steps`. Successful configuration mutations SHALL use POST/Redirect/GET;
invalid form input SHALL be re-rendered as a readable HTML error without
applying a partial mutation.
#### Scenario: Browse filtered tasks and inspect a timeline
- **WHEN** an operator selects a device or status filter and opens a known
task
- **THEN** the task list contains only matching tasks and the task page renders
its timeline in step order with its screenshot when one exists
#### Scenario: Register a device from the configuration page
- **WHEN** an operator submits valid supported-device form values
- **THEN** the Runtime registers and persists the device, responds with a
redirect to the configuration page, and the device is visible after the
redirect
#### Scenario: Reject invalid configuration without partial mutation
- **WHEN** an operator submits an unsupported driver type, malformed
connection value, or non-positive `max_steps`
- **THEN** the Runtime returns an HTML validation error, preserves the prior
Runtime/configuration state, and does not perform a redirect
### Requirement: Dashboard live status SHALL remain server-rendered
The dashboard SHALL retain periodic live-status refresh without restoring a
client-side application framework. Its browser enhancement SHALL request a
server-rendered HTML fragment and replace only the live-status region; it
SHALL NOT rebuild console state from a JSON API response.
#### Scenario: Refresh dashboard status after the polling interval
- **WHEN** the dashboard refresh enhancement runs while the Runtime is
available
- **THEN** it retrieves a Jinja2-rendered status fragment and updates the
dashboard's live-status region without a full-page reload
### Requirement: Existing console JSON API SHALL remain compatible
The Runtime SHALL continue to expose the existing `/console/devices`,
`/console/tasks`, `/console/tasks/{task_id}`, `/console/tasks/{task_id}/timeline`,
and `/console/config` JSON endpoints with their existing methods, status
codes, payloads, filtering behavior, and persistence semantics. Page and JSON
routes SHALL use the same API-local console operations rather than issuing
HTTP requests to each other.
#### Scenario: Programmatic client reads console data after the UI migration
- **WHEN** a client calls an existing `GET /console/*` endpoint after the
server-rendered console is deployed
- **THEN** it receives the same JSON response shape and status behavior as
before the migration
#### Scenario: A page mutation is visible through the JSON API
- **WHEN** an operator registers or removes a device or updates `max_steps`
through a `/ui/` form
- **THEN** the corresponding `/console/*` JSON endpoint reports the same
resulting Runtime state
### Requirement: Runtime console templates and assets SHALL ship with the Python package
The Runtime distribution SHALL package all console templates and static assets
needed by `/ui/`. The Runtime console SHALL not depend on the top-level Vue/Vite
`console/` project, Node package installation, Vite configuration, or the
SPA-only static-directory mount. Same-origin console operation SHALL not
require wildcard CORS configured for Vite development.
#### Scenario: Run the console from an installed Runtime wheel
- **WHEN** `device-agent-runtime` is built and installed outside the source
checkout
- **THEN** the Runtime can serve `/ui/` and its required CSS/browser assets
from packaged resources
#### Scenario: Start a Runtime after the SPA workflow is removed
- **WHEN** an operator starts the Runtime API using the documented Python
command
- **THEN** the console is available at `/ui/` without `npm install`,
`npm run build`, `VITE_API_BASE_URL`, or `RUNTIME_CONSOLE_STATIC_DIR`
@@ -0,0 +1,42 @@
## 1. Runtime package and web-resource setup
- [x] 1.1 Add direct `jinja2>=3.1` and `python-multipart` Runtime dependencies, configure setuptools package-data for Runtime console templates/CSS/JavaScript, regenerate `uv.lock`, and verify `uv lock --check`.
- [x] 1.2 Create the Runtime-owned template and static-asset layout under `api/` for the `/ui/` console, with names suitable for wheel packaging and standard static-file serving.
- [x] 1.3 Add a package-resource smoke test that builds and installs `device-agent-runtime` outside the source checkout and confirms `/ui/` can find its templates and assets.
## 2. Shared API-layer console operations
- [x] 2.1 Refactor `api/console.py` to expose an API-local typed console service for device, task, timeline, and runtime-config reads and mutations, preserving current validation and error semantics.
- [x] 2.2 Rewire `create_console_router()` to use the shared service and retain every existing `/console/*` method, JSON response shape, status code, filtering rule, and persistence behavior.
- [x] 2.3 Update `api/rest.py` to construct one shared service from its existing injected Runtime state and pass it to both JSON and HTML console routers without changing lower-layer dependencies.
## 3. Jinja2 Runtime console routes and pages
- [x] 3.1 Implement `api/console_web.py` with one module-level Jinja2 `Environment`, `FileSystemLoader`, `select_autoescape(["html", "xml"])`, and a small `HTMLResponse` render helper.
- [x] 3.2 Mount same-origin `/ui/` page routes and `/ui/assets` static resources; make `GET /` redirect to `/ui/` unconditionally.
- [x] 3.3 Implement the shared base layout, dashboard, and Jinja-rendered live-status fragment, including the small polling enhancement that replaces only the live region.
- [x] 3.4 Implement server-rendered task list/filter and task-detail/timeline pages, including ordered records, safe structured tool/result output, and available screenshot data URIs.
- [x] 3.5 Implement the configuration page and POST/Redirect/GET device registration/removal and `max_steps` update handlers, with readable `400` HTML validation errors that retain submitted safe values and perform no partial mutation.
- [x] 3.6 Port the existing console visual layout to package-owned CSS and the narrow polling script without adding a JavaScript framework or a frontend build step.
## 4. Retire SPA deployment wiring and update documentation
- [x] 4.1 Remove `RUNTIME_CONSOLE_STATIC_DIR`, `SpaStaticFiles`, the SPA fallback, and the wildcard CORS middleware used only for Vite development from `api/rest.py`.
- [x] 4.2 Delete the top-level `console/` Vue/Vite project and remove its obsolete environment, npm, and ignore-file references while leaving `cloud-console/` untouched.
- [x] 4.3 Update `README.md`, `docs/CONSTITUTION.md`, and `docs/MACOS_IPHONE_SETUP.md` to document normal Runtime startup followed by `/ui/`, the breaking removal of the npm/static-directory workflow, and the trusted-network-only security boundary.
- [x] 4.4 Remove stale Docker ignore/configuration entries that only described the deleted Runtime SPA, without changing Cloud Console build or deployment behavior.
## 5. Automated verification
- [x] 5.1 Keep and extend console JSON API tests to prove `/console/*` compatibility before and after page-route mutations.
- [x] 5.2 Add template tests covering every Runtime console template, module-level autoescape configuration, XSS probes in device/task/timeline/configuration values, safe structured JSON output, and absence of unintended `|safe` bypasses.
- [x] 5.3 Add FastAPI `TestClient` coverage for root redirect, page navigation, empty/populated dashboard state, task filters/detail/timeline, dashboard fragment refresh, static assets, form PRG success paths, and invalid-form no-mutation paths.
- [x] 5.4 Add regression coverage that same-origin `/ui/` works without `RUNTIME_CONSOLE_STATIC_DIR` and does not depend on the removed wildcard CORS middleware.
## 6. Validation and migration handoff
- [x] 6.1 Run targeted formatting, lint, compile, and Runtime console tests, then run the repository non-integration test suite; record any pre-existing failures separately from this change.
- [x] 6.2 Build the Runtime wheel and run the isolated package-resource smoke test after the final dependency lock update.
- [x] 6.3 Run `openspec validate runtime-console-jinja2-templates --strict` and resolve all validation failures.
- [ ] 6.4 Perform a browser walkthrough of dashboard live refresh, task filtering/timeline replay, valid and invalid device configuration, device removal, and `max_steps` update using a real Runtime process.
- [ ] 6.5 After human browser verification, reconcile the remaining manual verification and superseded SPA decision in the pending `web-console` change before any archive decision; do not mark it complete from automated evidence alone.