Compare commits
5
Commits
989fdbf878
...
56f3f96363
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
56f3f96363 | ||
|
|
8e37b965aa | ||
|
|
c01dd4c6b2 | ||
|
|
9669b52498 | ||
|
|
fa10cccf71 |
@@ -0,0 +1,127 @@
|
||||
# agent-runtime Specification
|
||||
|
||||
## Purpose
|
||||
Define the behavior of the AI Planner runtime, including LLM-driven action
|
||||
selection, completion signaling, provider abstraction, default-disabling, and
|
||||
the narrow screenshot exception to the Perception Boundary.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: LLM-driven Planner selects exactly one grounded action per turn
|
||||
The system SHALL provide a Planner implementation that, given a goal, the
|
||||
current Scene, and recent task history, uses native LLM tool/function
|
||||
calling to select exactly one action (or the completion signal defined
|
||||
below) per `plan()` invocation, grounding any coordinates in the current
|
||||
turn's Scene element bounds.
|
||||
|
||||
#### Scenario: Planner selects a single action for the current turn
|
||||
- **WHEN** the AI Planner is invoked with a goal and the current Scene
|
||||
- **THEN** it returns at most one `PlannedStep`, whose action and arguments
|
||||
come from exactly one tool call chosen by the underlying LLM for that turn
|
||||
|
||||
#### Scenario: Planner re-decides every turn from the current Scene
|
||||
- **WHEN** the AI Planner is invoked again after a prior step has executed
|
||||
- **THEN** its decision is grounded in the newly observed Scene for that
|
||||
turn, not in coordinates or assumptions carried over from a previous turn
|
||||
|
||||
### Requirement: Explicit finish_task completion and failure signal
|
||||
The system SHALL treat task completion and task failure as explicit,
|
||||
model-driven signals via a dedicated `finish_task(success, reason)` tool,
|
||||
rather than inferring either outcome from the model declining to call any
|
||||
tool.
|
||||
|
||||
#### Scenario: Model signals successful completion
|
||||
- **WHEN** the model calls `finish_task` with `success=True`
|
||||
- **THEN** the Planner returns an empty step list and the task is marked
|
||||
completed
|
||||
|
||||
#### Scenario: Model signals it cannot complete the goal
|
||||
- **WHEN** the model calls `finish_task` with `success=False` and a `reason`
|
||||
- **THEN** the task is marked failed with that reason, without attempting
|
||||
any further planning steps
|
||||
|
||||
### Requirement: Pluggable dual-provider tool-calling abstraction
|
||||
The system SHALL support at least two interchangeable LLM providers
|
||||
(Anthropic native tool use and OpenAI function calling) for the AI
|
||||
Planner's decision calls, selectable via configuration, with both providers
|
||||
constrained to return exactly one tool call per request. Independently of
|
||||
provider selection, the system SHALL support at least two transports for
|
||||
making that decision call -- direct-to-provider (the tool-calling client
|
||||
calls the provider's SDK itself, using locally configured credentials) and
|
||||
cloud-proxy (the tool-calling client calls the Cloud Control Plane's
|
||||
planner-decision endpoint, which calls the provider using cloud-held
|
||||
credentials) -- selectable via configuration without requiring any change
|
||||
to `AIPlanner`'s own decision logic.
|
||||
|
||||
#### Scenario: Provider selected via configuration
|
||||
- **WHEN** the AI Planner is configured with a given provider identifier
|
||||
- **THEN** it constructs and uses the tool-calling client for that provider
|
||||
without requiring any change to `AIPlanner`'s own decision logic
|
||||
|
||||
#### Scenario: Provider response resolves to a single decision
|
||||
- **WHEN** either supported provider returns a response to a tool-calling
|
||||
request
|
||||
- **THEN** the response is parsed into exactly one tool name and one
|
||||
arguments object, regardless of which provider produced it
|
||||
|
||||
#### Scenario: Transport selected via configuration
|
||||
- **WHEN** the Host Agent is configured with a given transport (direct or
|
||||
cloud-proxy)
|
||||
- **THEN** `AIPlanner` is constructed with the tool-calling client for that
|
||||
transport, and its own decision logic is unchanged regardless of which
|
||||
transport is in effect
|
||||
|
||||
#### Scenario: Cloud-proxy transport is the default
|
||||
- **WHEN** no transport is explicitly configured
|
||||
- **THEN** the AI Planner uses the cloud-proxy transport and the Cloud
|
||||
Control Plane's planner-decision endpoint
|
||||
|
||||
#### Scenario: Direct transport remains available by explicit configuration
|
||||
- **WHEN** the Host Agent is configured with the direct transport
|
||||
- **THEN** the AI Planner uses the direct-to-provider transport with locally
|
||||
configured credentials
|
||||
|
||||
#### Scenario: Cloud-proxy transport resolves a decision without a local provider client
|
||||
- **WHEN** the Host Agent is configured with the cloud-proxy transport
|
||||
- **THEN** its tool-calling client sends the decision request to the Cloud
|
||||
Control Plane's planner-decision endpoint instead of constructing a local
|
||||
Anthropic or OpenAI SDK client
|
||||
|
||||
### Requirement: AI Planner is disabled by default and additive to the existing Planner
|
||||
The system SHALL default to the existing non-LLM Planner unless the AI
|
||||
Planner is explicitly enabled via configuration, and SHALL NOT alter the
|
||||
existing Planner's behavior, dependencies, or any caller's construction of
|
||||
`TaskRunner` when left disabled.
|
||||
|
||||
#### Scenario: AI Planner disabled (default)
|
||||
- **WHEN** `TaskRunner` is constructed without an explicit `planner` and
|
||||
without the AI Planner enabled in configuration
|
||||
- **THEN** it uses the existing non-LLM Planner, unchanged from before this
|
||||
capability existed
|
||||
|
||||
#### Scenario: AI Planner enabled via configuration
|
||||
- **WHEN** `TaskRunner` is constructed without an explicit `planner` and with
|
||||
the AI Planner enabled in configuration
|
||||
- **THEN** it uses the AI Planner, configured with the selected provider and
|
||||
model
|
||||
|
||||
### Requirement: Screenshot access is a Planner-only, narrow exception to the Perception Boundary
|
||||
The system SHALL allow the AI Planner, and only the AI Planner, to receive
|
||||
the current step's raw screenshot bytes alongside the Scene for
|
||||
vision-grounded decision-making, while every other perception consumer
|
||||
SHALL continue to receive only the Scene.
|
||||
|
||||
#### Scenario: Planner receives both Scene and screenshot
|
||||
- **WHEN** a screenshot for the current step is available
|
||||
- **THEN** the AI Planner's decision call includes both the Scene JSON and
|
||||
the raw screenshot bytes for that step
|
||||
|
||||
#### Scenario: Screenshot unavailable does not block planning
|
||||
- **WHEN** a screenshot for the current step cannot be obtained
|
||||
- **THEN** the AI Planner still produces a decision using the Scene alone,
|
||||
and this is not treated as a task failure
|
||||
|
||||
#### Scenario: No other consumer receives raw screenshot bytes
|
||||
- **WHEN** any component other than the AI Planner (for example, `api`,
|
||||
`tools`, `perception`, or `storage`) consumes perception output
|
||||
- **THEN** it receives only the Scene, never raw screenshot bytes
|
||||
@@ -0,0 +1,72 @@
|
||||
# cloud-planner-proxy Specification
|
||||
|
||||
## Purpose
|
||||
Define the Cloud Control Plane's planner-decision proxy endpoint, which
|
||||
allows a Host Agent to delegate AI Planner LLM calls to the cloud using
|
||||
cloud-held provider credentials, removing the need for Host Agent-held
|
||||
provider API keys.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: Cloud Control Plane exposes an authenticated planner-decision endpoint
|
||||
The Cloud Control Plane SHALL expose an internal endpoint that accepts an AI Planner tool-calling decision request from an authenticated Host Agent, and SHALL require the same host-scoped bearer credential already used for heartbeat, claim, lease renewal, and result reporting -- no separate credential or enrollment step.
|
||||
|
||||
#### Scenario: Authenticated host requests a planner decision
|
||||
- **WHEN** a Host Agent presents its existing valid host-scoped credentials
|
||||
with a planner-decision request
|
||||
- **THEN** the Cloud Control Plane accepts and processes the request for
|
||||
that host
|
||||
|
||||
#### Scenario: Unauthenticated or foreign-host request is rejected
|
||||
- **WHEN** a request omits valid host-scoped credentials, or presents
|
||||
credentials bound to a different host than the one referenced in the
|
||||
request
|
||||
- **THEN** the Cloud Control Plane rejects the request without invoking any
|
||||
LLM provider
|
||||
|
||||
### Requirement: Endpoint resolves exactly one tool-call decision using cloud-held provider configuration
|
||||
The Cloud Control Plane SHALL use its own configured LLM provider, model, and credentials -- not any value supplied by the requesting Host Agent -- to resolve a planner-decision request to exactly one tool name and one arguments object, within the request's timeout. The endpoint SHALL resolve the active database-managed Provider profile for every request and use its provider, model, timeout, configured base URL, and encrypted cloud-held credential. It SHALL NOT read Cloud API planner Provider/model/timeout/API-key environment variables.
|
||||
|
||||
#### Scenario: Provider returns a usable decision
|
||||
- **WHEN** the configured provider responds to a planner-decision request
|
||||
with a tool call
|
||||
- **THEN** the Cloud Control Plane returns exactly one resolved tool name
|
||||
and arguments object to the requesting Host Agent
|
||||
|
||||
#### Scenario: Configured provider is unreachable or misconfigured
|
||||
- **WHEN** the Cloud Control Plane's configured provider call fails (for
|
||||
example, invalid credentials, provider error, or timeout)
|
||||
- **THEN** the endpoint returns a structured failure response rather than a
|
||||
fabricated decision, and does not crash the Cloud Control Plane process
|
||||
|
||||
#### Scenario: Database profile is the only planner configuration source
|
||||
- **WHEN** the Cloud API process has legacy planner environment variables
|
||||
- **THEN** subsequent planner-decision requests use only the active database
|
||||
profile and do not read a legacy Provider credential for that decision
|
||||
|
||||
### Requirement: Cloud-proxy transport removes the need for Host Agent-held provider credentials
|
||||
A Host Agent using the cloud-proxy transport for its AI Planner SHALL be able to execute AI-planned tasks without any locally configured LLM provider API key.
|
||||
|
||||
#### Scenario: Host configured for cloud-proxy transport has no local provider key
|
||||
- **WHEN** a Host Agent is configured to use the cloud-proxy transport and
|
||||
has no `ANTHROPIC_API_KEY`/`OPENAI_API_KEY` set in its own environment
|
||||
- **THEN** it can still obtain AI Planner decisions by calling the Cloud
|
||||
Control Plane's planner-decision endpoint
|
||||
|
||||
### Requirement: Planner-decision requests are not durably persisted
|
||||
The Cloud Control Plane SHALL process planner-decision requests, including any screenshot and prompt text they carry, without durably persisting that screenshot or prompt content; only request metadata (such as host identifier, resolved tool name, latency, and error classification) may be retained for observability.
|
||||
|
||||
#### Scenario: Request handling completes without storing prompt or screenshot content
|
||||
- **WHEN** the Cloud Control Plane finishes handling a planner-decision
|
||||
request
|
||||
- **THEN** the raw prompt text and screenshot bytes from that request are
|
||||
not present in any durable store or log the Cloud Control Plane retains
|
||||
|
||||
### Requirement: Planner-proxy failures do not silently substitute a default action
|
||||
The Cloud Control Plane SHALL report a failure to the requesting Host Agent when it cannot resolve a planner-decision request to a valid tool call, rather than returning a default, guessed, or previously cached decision.
|
||||
|
||||
#### Scenario: Endpoint cannot resolve a decision
|
||||
- **WHEN** the configured provider does not return a usable tool call for a
|
||||
planner-decision request
|
||||
- **THEN** the Cloud Control Plane's response indicates failure, and the
|
||||
requesting Host Agent treats the planner call for that turn as failed
|
||||
@@ -0,0 +1,121 @@
|
||||
# llm-provider-management Specification
|
||||
|
||||
## Purpose
|
||||
Define how the Cloud Control Plane stores, protects, and activates
|
||||
administrator-managed LLM Provider profiles in a durable database, so that
|
||||
cloud-transport planner decisions resolve provider, model, timeout, base URL,
|
||||
and API key from a single active database profile instead of from environment
|
||||
variables.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: Cloud-wide LLM Provider profiles are durable and validated
|
||||
The Cloud Control Plane SHALL persist administrator-managed LLM Provider
|
||||
profiles with a unique name, provider type, model, timeout, enabled state,
|
||||
revision, and timestamps. A profile's provider type SHALL be either
|
||||
`anthropic` or `openai-compatible`; either profile MAY specify an absolute
|
||||
HTTP(S) base URL and SHALL use its official provider endpoint when it does not.
|
||||
|
||||
#### Scenario: Administrator creates an OpenAI-compatible profile
|
||||
- **WHEN** an authorized administrator submits a unique profile name,
|
||||
`openai-compatible` provider type, model, valid timeout, API key, and an
|
||||
optional valid base URL
|
||||
- **THEN** the Cloud Control Plane persists an enabled profile with a new
|
||||
revision and returns its non-secret metadata
|
||||
|
||||
#### Scenario: Administrator creates an Anthropic profile with a custom base URL
|
||||
- **WHEN** an authorized administrator submits a unique profile name,
|
||||
`anthropic` provider type, model, valid timeout, API key, and a valid
|
||||
Anthropic-compatible base URL
|
||||
- **THEN** the Cloud Control Plane persists the normalized URL with the profile
|
||||
and returns its non-secret metadata
|
||||
|
||||
#### Scenario: Invalid profile configuration is rejected
|
||||
- **WHEN** an administrator submits an unsupported provider type, blank model,
|
||||
non-positive timeout, duplicate name, or invalid base URL
|
||||
- **THEN** the Cloud Control Plane rejects the write without creating or
|
||||
changing a profile
|
||||
|
||||
### Requirement: Provider API keys are encrypted and never disclosed
|
||||
The Cloud Control Plane SHALL encrypt Provider API keys before persistence
|
||||
using a deployment-held encryption key, and SHALL not expose plaintext keys in
|
||||
read responses, validation errors, audit records, or application logs.
|
||||
|
||||
#### Scenario: Provider profile is listed after creation
|
||||
- **WHEN** an authorized administrator lists Provider profiles after creating
|
||||
one with an API key
|
||||
- **THEN** every response reports only key-presence and rotation metadata and
|
||||
does not contain the submitted API key or its ciphertext
|
||||
|
||||
#### Scenario: Encryption configuration is unavailable
|
||||
- **WHEN** a database-managed profile is created, rotated, activated, or
|
||||
resolved without a valid deployment encryption key
|
||||
- **THEN** the operation fails with a controlled configuration error that does
|
||||
not reveal an API key
|
||||
|
||||
### Requirement: Administrators can manage and activate Provider profiles
|
||||
The Cloud Control Plane SHALL expose session-CSRF-protected and scope-guarded
|
||||
operations to list, create, update, rotate a key, enable, disable, activate,
|
||||
and delete inactive LLM Provider profiles. Mutating operations SHALL require
|
||||
the `llm-providers:admin` scope and record a non-secret audit event.
|
||||
|
||||
#### Scenario: Non-administrator attempts to modify a profile
|
||||
- **WHEN** a principal without `llm-providers:admin` invokes a Provider
|
||||
mutation endpoint
|
||||
- **THEN** the Cloud Control Plane rejects the request before decrypting or
|
||||
modifying a Provider credential
|
||||
|
||||
#### Scenario: Administrator switches the active profile
|
||||
- **WHEN** an authorized administrator activates an enabled profile
|
||||
- **THEN** the Cloud Control Plane atomically selects that profile as the one
|
||||
Cloud-wide active profile and records the activation without retaining an
|
||||
API key in the audit event
|
||||
|
||||
#### Scenario: Administrator attempts to retire the active profile
|
||||
- **WHEN** an administrator attempts to disable or delete the active profile
|
||||
before activating a replacement
|
||||
- **THEN** the Cloud Control Plane rejects the operation and preserves the
|
||||
active profile selection
|
||||
|
||||
### Requirement: Activation dynamically selects one Provider for cloud transport
|
||||
The Cloud Control Plane SHALL resolve the active database-managed profile for
|
||||
each Cloud planner decision and SHALL apply an activation to subsequent
|
||||
requests without requiring a Cloud API restart or any Host Agent
|
||||
reconfiguration.
|
||||
|
||||
#### Scenario: A Host requests a decision after a model switch
|
||||
- **WHEN** an administrator activates a different enabled profile and a
|
||||
cloud-transport Host Agent submits its next planner-decision request
|
||||
- **THEN** the Cloud Control Plane calls that profile's provider, model,
|
||||
timeout, base URL, and API key while the Host Agent continues using the same
|
||||
Cloud Control Plane endpoint
|
||||
|
||||
#### Scenario: Active OpenAI-compatible profile is used
|
||||
- **WHEN** the active profile is OpenAI-compatible and includes a base URL
|
||||
- **THEN** the Cloud Control Plane makes the existing OpenAI Chat Completions
|
||||
tool-calling request to that base URL using the profile's decrypted API key
|
||||
and returns the resulting single tool-call decision
|
||||
|
||||
#### Scenario: Active Anthropic profile is used with a custom base URL
|
||||
- **WHEN** the active profile is Anthropic and includes a base URL
|
||||
- **THEN** the Cloud Control Plane makes the existing Anthropic native
|
||||
tool-calling request to that base URL using the profile's decrypted API key
|
||||
and returns the resulting single tool-call decision
|
||||
|
||||
### Requirement: Cloud planner Provider configuration is database-only
|
||||
The Cloud Control Plane SHALL resolve Cloud planner Provider, model, timeout,
|
||||
base URL, and API key from the active database profile and SHALL NOT read
|
||||
planner Provider/model/timeout or Provider API-key environment variables.
|
||||
When no usable active profile exists, it SHALL fail a planner request with a
|
||||
structured unavailable response.
|
||||
|
||||
#### Scenario: Active profile is unavailable
|
||||
- **WHEN** the active database Provider profile is missing, disabled, or
|
||||
cannot be decrypted
|
||||
- **THEN** the planner decision fails without invoking a legacy
|
||||
environment-configured Provider
|
||||
|
||||
#### Scenario: Legacy environment variables are present
|
||||
- **WHEN** the Cloud API process has legacy planner Provider or API-key
|
||||
environment variables but an active database profile exists
|
||||
- **THEN** the planner decision uses only the active database profile
|
||||
@@ -0,0 +1,27 @@
|
||||
## Purpose
|
||||
|
||||
Define the supported Python interpreter version range for the project workspace, container images, and CI pipeline, ensuring all environments converge on a single Python baseline.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: Workspace-wide supported Python version
|
||||
The system SHALL declare a single, consistent supported Python interpreter version range across the `uv` workspace root and every workspace member (`apps/device-host-agent`, `apps/cloud-api`, `packages/cloud-platform`), such that the range admits Python 3.13 and excludes Python 3.14, so that `paddlepaddle` (which has no Python 3.14 wheel available on PyPI) can be installed in every environment that follows the project's documented setup.
|
||||
|
||||
#### Scenario: Workspace resolves and syncs on Python 3.13
|
||||
- **WHEN** an operator runs `uv python install 3.13` followed by `uv sync --locked --all-packages` in the repository root
|
||||
- **THEN** the sync succeeds and provisions a `.venv` using a Python 3.13 interpreter, with `uv.lock` resolved against that interpreter
|
||||
|
||||
#### Scenario: A Python 3.14 interpreter is rejected
|
||||
- **WHEN** `uv` attempts to resolve or sync the workspace against a Python 3.14 interpreter
|
||||
- **THEN** `uv` reports the interpreter as incompatible with the declared `requires-python` range, rather than silently proceeding
|
||||
|
||||
### Requirement: Container and CI images match the declared Python baseline
|
||||
The system SHALL build its Docker image and Jenkins CI pipeline using a base image whose Python version matches the workspace's declared `requires-python` range, so that containerized and CI environments never diverge from what local development targets.
|
||||
|
||||
#### Scenario: Docker image build uses the supported Python version
|
||||
- **WHEN** the project's `Dockerfile` is built
|
||||
- **THEN** the resulting image's base image is `astral-sh/uv:python3.13-bookworm-slim`
|
||||
|
||||
#### Scenario: CI pipeline uses the supported Python version
|
||||
- **WHEN** the Jenkins pipeline defined in `Jenkinsfile` executes
|
||||
- **THEN** it runs inside the `astral-sh/uv:python3.13-bookworm-slim` image
|
||||
@@ -0,0 +1,56 @@
|
||||
## Purpose
|
||||
|
||||
Define how the Python repository is structured as a uv workspace, including the shared lockfile, one-way dependency direction between the Runtime and cloud platform packages, import-path and behavior compatibility during the packaging migration, independent buildability of workspace members, and the boundary that keeps the frontend JavaScript project outside the Python workspace.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: Repository uses a uv workspace with a shared lockfile
|
||||
The repository SHALL define a uv workspace containing the root `device-agent-runtime` project and the extracted `device-cloud-platform` project, and SHALL commit one generated `uv.lock` that resolves all workspace members.
|
||||
|
||||
#### Scenario: Clean workspace synchronization
|
||||
- **WHEN** a developer synchronizes the repository from a clean checkout using uv
|
||||
- **THEN** uv installs the workspace members and their locked dependencies without requiring manual editable-install commands
|
||||
|
||||
#### Scenario: Lockfile consistency check
|
||||
- **WHEN** CI validates dependency metadata without updating dependencies
|
||||
- **THEN** the committed lockfile is accepted as current for every workspace member
|
||||
|
||||
### Requirement: Cloud packaging has an explicit one-way Runtime dependency
|
||||
The `device-cloud-platform` project SHALL declare `device-agent-runtime` as a uv workspace dependency, and the `device-agent-runtime` project MUST NOT depend on `device-cloud-platform` or import the `cloud` package.
|
||||
|
||||
#### Scenario: Cloud modules consume Runtime contracts
|
||||
- **WHEN** the cloud platform is installed through the workspace
|
||||
- **THEN** its imports of existing Runtime, workflow, storage, and driver contracts resolve through the declared Runtime workspace dependency
|
||||
|
||||
#### Scenario: Runtime remains cloud-independent
|
||||
- **WHEN** dependency-direction tests inspect Runtime-owned source packages
|
||||
- **THEN** no Runtime-owned package imports `cloud` or requires the cloud distribution to execute existing local capabilities
|
||||
|
||||
### Requirement: Existing cloud import paths and behavior remain compatible
|
||||
The packaging migration SHALL preserve the existing `cloud.*` Python import paths and SHALL NOT change REST, MCP, scheduling, dispatch, persistence, or SDK behavior.
|
||||
|
||||
#### Scenario: Existing cloud tests run after extraction
|
||||
- **WHEN** the existing cloud test suite runs from the workspace root after the package is moved
|
||||
- **THEN** the tests import the same `cloud.*` modules and retain their pre-migration behavior
|
||||
|
||||
#### Scenario: Local Runtime runs without cloud application members
|
||||
- **WHEN** a developer starts or tests the root Runtime project without starting a cloud application
|
||||
- **THEN** existing Runtime entry points and local device behavior remain available
|
||||
|
||||
### Requirement: Workspace members are independently buildable and selectable
|
||||
Each Python workspace member SHALL contain valid build metadata, SHALL produce an installable distribution, and SHALL support member-scoped uv commands while root-level tests remain supported.
|
||||
|
||||
#### Scenario: Build all distributions
|
||||
- **WHEN** CI builds the Runtime and cloud platform workspace members
|
||||
- **THEN** each build produces a wheel whose declared packages can be imported in an isolated environment
|
||||
|
||||
#### Scenario: Run a member-scoped command
|
||||
- **WHEN** a developer selects a workspace member with uv's package selection option
|
||||
- **THEN** the command executes with that member's declared dependencies and workspace sources
|
||||
|
||||
### Requirement: Frontend package management remains separate
|
||||
The Vue/Vite `console` project SHALL remain managed by its existing JavaScript package manager and SHALL NOT become a Python uv workspace member.
|
||||
|
||||
#### Scenario: Synchronize Python workspace
|
||||
- **WHEN** a developer runs uv synchronization at the repository root
|
||||
- **THEN** uv does not install or modify the console's JavaScript dependencies
|
||||
Reference in New Issue
Block a user