This commit is contained in:
2026-07-06 23:52:53 +08:00
parent d899b875ce
commit 8e29fcf3c2
54 changed files with 736 additions and 0 deletions
+49
View File
@@ -0,0 +1,49 @@
# device-pool Specification
## Purpose
TBD - created by archiving change cloud-runtime. Update Purpose after archive.
## Requirements
### Requirement: Host registration and heartbeat sync
The system SHALL provide a `DevicePool` that tracks a `HostRegistration` (host id, address, last-seen timestamp) for each host process that registers itself, and SHALL update a host's last-seen timestamp whenever that host pushes a device snapshot via `sync_host_devices(host_id, snapshot)`.
#### Scenario: New host registers and syncs devices
- **WHEN** a previously-unknown `host_id` calls `sync_host_devices` with a list of devices
- **THEN** the pool creates a new `HostRegistration` for that host, records the current time as its last-seen timestamp, and stores each synced device as a `PooledDevice` owned by that host
#### Scenario: Known host re-syncs
- **WHEN** an already-registered `host_id` calls `sync_host_devices` again with an updated device snapshot
- **THEN** the pool updates that host's last-seen timestamp and replaces its previously-stored `PooledDevice` records with the new snapshot, without duplicating or losing devices from other hosts
### Requirement: Aggregated device listing across hosts
The system SHALL provide a way to list all `PooledDevice` records across every registered host, including each device's owning `host_id`, `driver_type`, status, and capability tags.
#### Scenario: Listing devices across multiple hosts
- **WHEN** two hosts have each synced a non-empty device snapshot
- **THEN** a caller listing pool devices sees devices from both hosts in one combined result, each tagged with its correct `host_id`
#### Scenario: No hosts registered
- **WHEN** a caller lists pool devices before any host has ever synced
- **THEN** the pool returns an empty list rather than raising an error
### Requirement: Stale host devices degrade to unreachable
The system SHALL mark all `PooledDevice`s belonging to a host `unreachable` once that host's last-seen timestamp exceeds a configured staleness threshold, computed at read time, without requiring any background process and without raising an error for the stale host's absence.
#### Scenario: Host misses its sync interval
- **WHEN** a host's last-seen timestamp is older than `config.stale_after_seconds` at the time of a `list_devices()`/`get_device()` call
- **THEN** every `PooledDevice` owned by that host is reported with status `unreachable`, regardless of the status value in its last-synced snapshot
#### Scenario: Host resumes syncing after being stale
- **WHEN** a host previously marked stale calls `sync_host_devices` again
- **THEN** its devices immediately stop being reported `unreachable` and reflect the statuses in the new snapshot
### Requirement: Device lookup by id across the pool
The system SHALL allow looking up a single `PooledDevice` by `device_id` regardless of which host owns it, returning a clear not-found result when no host has ever reported that device id.
#### Scenario: Lookup finds device on any host
- **WHEN** a caller requests a device by id that exists in some host's synced snapshot
- **THEN** the pool returns that `PooledDevice` including its owning `host_id`
#### Scenario: Lookup for unknown device id
- **WHEN** a caller requests a device by id that no host has ever synced
- **THEN** the pool returns a not-found result (e.g. `None`) rather than raising an unhandled exception
+23
View File
@@ -0,0 +1,23 @@
# driver-registry Specification
## Purpose
TBD - created by archiving change device-agent-runtime-foundation. Update Purpose after archive.
## Requirements
### Requirement: Driver type registry lives in the driver layer
The system SHALL provide a registry, owned by the `driver` package, that maps a `driver_type` string (e.g. `"wda"`) to a builder function producing a `DriverFactory` for that type, so that any caller needing to construct a driver for a device does so without importing a concrete driver class directly.
#### Scenario: Building a factory for a known driver type
- **WHEN** a caller requests a driver factory for `driver_type="wda"` with connection info (e.g. `server_url`, `udid`)
- **THEN** the registry returns a `DriverFactory` that, when invoked, constructs a working `WDADriver` configured with that connection info
#### Scenario: Building a factory for an unknown driver type
- **WHEN** a caller requests a driver factory for a `driver_type` that is not registered
- **THEN** the registry raises a clear error naming the unsupported `driver_type`, instead of returning `None` or a factory that fails later at connect time
### Requirement: Adding a driver type requires no changes outside the driver layer
The system SHALL allow a new driver type to be added by registering it in the `driver` package's registry alone; no other package (`api/`, `device/`, `tools/`, `runtime/`) SHALL need code changes to support constructing devices of the new type.
#### Scenario: API layer is agnostic to registered driver types
- **WHEN** the console config API resolves a `driver_type` string into a driver factory to register a device
- **THEN** it does so by calling into the driver registry rather than maintaining its own mapping of `driver_type` to concrete driver classes
@@ -0,0 +1,63 @@
# multi-agent-collaboration Specification
## Purpose
TBD - created by archiving change multi-agent-runtime. Update Purpose after archive.
## Requirements
### Requirement: Handoff protocol data contracts
The system SHALL define `Observation`, `VerificationVerdict`, `ReflectionOutcome`, and `ReflectionAction` as plain dataclasses with `to_dict()`/`from_dict()` methods, forming the stable, spec'd contract passed between the Observer, Verifier, and Reflector roles, independent of any single role's internal implementation.
#### Scenario: Handoff dataclasses round-trip through serialization
- **WHEN** an `Observation`, `VerificationVerdict`, or `ReflectionOutcome` instance is serialized via `to_dict()` and then reconstructed via `from_dict()`
- **THEN** the reconstructed instance is equal to the original instance
### Requirement: Observer produces an Observation from available device state
The system SHALL provide an Observer role that produces an `Observation` summarizing current device state, using `SemanticScene` and `WorldState` when available and falling back to the raw `Scene`/`PlannedStep`/`StepResult` when either or both are absent.
#### Scenario: Observation succeeds with SemanticScene and WorldState present
- **WHEN** `Observer.observe(...)` is called and both `SemanticScene` and `WorldState` are available
- **THEN** it returns an `Observation` incorporating both as context
#### Scenario: Observation degrades gracefully when semantic state is absent
- **WHEN** `Observer.observe(...)` is called and `SemanticScene` and/or `WorldState` is `None`
- **THEN** it returns an `Observation` built from the raw `Scene`/`PlannedStep`/`StepResult` instead of raising an exception
### Requirement: Verifier checks whether an executed step achieved its intended effect
The system SHALL provide a Verifier role that, after a `PlannedStep` has already been executed by the existing `Executor`, compares a pre-step and post-step `Observation` against the step's stated intent and produces a `VerificationVerdict` indicating whether the intended effect was actually achieved, distinct from and running after the Executor's own low-level tool-call retry.
#### Scenario: Verifier confirms an achieved effect
- **WHEN** `Verifier.verify(...)` is called with a pre-step `Observation`, a post-step `Observation`, the executed `PlannedStep`, and its `StepResult`, and the post-step `Observation` reflects the step's stated intent
- **THEN** it returns a `VerificationVerdict` marking the step as achieved
#### Scenario: Verifier flags a mechanically-successful but semantically-failed step
- **WHEN** the `StepResult` reports mechanical success but the post-step `Observation` does not reflect the step's stated intent
- **THEN** `Verifier.verify(...)` returns a `VerificationVerdict` marking the step as not achieved
### Requirement: Reflector proposes bounded recovery, never a blind re-issue
The system SHALL invoke a Reflector role only when the Verifier produces a not-achieved `VerificationVerdict`, and the Reflector SHALL analyze the `Observation`/`PlannedStep`/`StepResult`/verdict to produce a `ReflectionOutcome` carrying either a bounded, distinct recovery `ReflectionAction` or a replan request back to the Planner, never an outcome that simply re-issues the identical failed step.
#### Scenario: Reflector proposes a distinct recovery action
- **WHEN** `Reflector.reflect(...)` is invoked following a not-achieved `VerificationVerdict` and a distinct corrective action is identifiable
- **THEN** it returns a `ReflectionOutcome` carrying a `ReflectionAction` that differs from the originally failed `PlannedStep`
#### Scenario: Reflector requests a replan when no bounded recovery action applies
- **WHEN** `Reflector.reflect(...)` is invoked and no bounded corrective action is identifiable from the available `Observation`/`PlannedStep`/`StepResult`/verdict
- **THEN** it returns a `ReflectionOutcome` carrying a replan request rather than re-issuing the failed step
### Requirement: Reflection-driven recovery is bounded by an explicit ceiling
The system SHALL enforce a configurable maximum number of Reflector-triggered recovery attempts per task, tracked independently of and in addition to the Executor's own `max_retries`, so that a persistently-failing step cannot loop indefinitely between the Verifier and Reflector.
#### Scenario: Reflection loop stops once the ceiling is reached
- **WHEN** a task's Verifier-Reflector loop reaches the configured maximum reflection-recovery attempts without a step being verified as achieved
- **THEN** the `CollaborativeTaskRunner` stops attempting further reflection-driven recovery for that task and surfaces the failure instead of continuing the loop
### Requirement: Multi-agent collaboration composes existing Planner/Executor without modifying them
The system SHALL provide a `CollaborativeTaskRunner` that composes the existing `Planner`, `Executor`, and `TaskRunner` (`agent-runtime`) by import, without modifying `runtime/planner.py`, `runtime/executor.py`, or `runtime/task.py`, and SHALL default to disabled so that applying this capability does not alter any existing task's behavior, latency, or cost unless explicitly enabled.
#### Scenario: Collaboration disabled by default leaves existing task behavior unchanged
- **WHEN** a task is run without explicitly enabling multi-agent collaboration
- **THEN** it executes exactly as `runtime/task.py`'s existing `TaskRunner.run()` today, with no Observer/Verifier/Reflector invoked
#### Scenario: Enabling collaboration does not require changes to Planner or Executor
- **WHEN** a task is run with multi-agent collaboration explicitly enabled via `CollaborativeTaskRunner`
- **THEN** the existing `Planner.plan()` and `Executor.execute()` are invoked unchanged, with the Observer/Verifier/Reflector roles composed around them
@@ -0,0 +1,26 @@
# perception-provider Specification
## Purpose
TBD - created by archiving change device-agent-runtime-foundation. Update Purpose after archive.
## Requirements
### Requirement: Perception is exposed through a swappable provider port
The system SHALL define a `PerceptionProvider` interface (`build_scene`) in the `perception` package that any concrete perception implementation (the existing OCR+tree fusion, a future cloud vision API, a future null/mock provider) must satisfy identically, so `tools/`, `runtime/`, and `api/` depend only on the port, never on a specific perception technique.
#### Scenario: Building a scene through the default provider
- **WHEN** a caller requests a Scene for a screenshot and UI tree via the default (OCR+tree fusion) provider
- **THEN** the provider returns a `Scene` identical in shape and content to what the existing fusion logic already produces — no change to `scene-perception`'s specified behavior
### Requirement: A null perception provider is available for testing and low-dependency development
The system SHALL provide a `NullPerceptionProvider` that returns an empty `Scene` (correct screen width/height, zero elements) without requiring OCR/vision dependencies to be installed or invoked.
#### Scenario: Running without OCR dependencies installed
- **WHEN** the runtime is configured to use `NullPerceptionProvider` (e.g. in a test or a minimal development environment)
- **THEN** `describe_screen`/Agent Runtime calls succeed and return an empty Scene instead of failing due to missing OCR dependencies
### Requirement: Adding a perception technique requires no changes outside the perception layer
The system SHALL allow a new perception provider (e.g. a cloud vision API) to be added by registering it within the `perception` package alone; `tools/`, `runtime/`, and `api/` SHALL NOT require code changes to use a newly registered provider.
#### Scenario: Runtime is agnostic to which provider is active
- **WHEN** the Agent Runtime requests a Scene for the current screenshot and UI tree
- **THEN** it does so through the `PerceptionProvider` port without importing or knowing about `scene_builder.py`, OCR, or any other concrete technique
+67
View File
@@ -0,0 +1,67 @@
# platform-sdk Specification
## Purpose
TBD - created by archiving change cloud-runtime. Update Purpose after archive.
## Requirements
### Requirement: Versioned public API surface
The system SHALL expose the platform SDK's REST endpoints under a versioned URL prefix (`/v1/...`), distinct from the `mcp-tool-server` and `console-status-api`/`console-config-api` surfaces, so external integrators have a stable base path that will not silently change shape.
#### Scenario: Routes are mounted under the version prefix
- **WHEN** the platform SDK's router is mounted into an application
- **THEN** every route it exposes (task submission, status queries, device/host listing, plugin listing/registration) is reachable only under the `/v1/` prefix
### Requirement: Task submission and status via the SDK
The system SHALL allow an external integrator to submit a task (goal or workflow reference plus constraints) through the platform SDK's API, and to query that task's current status by id, backed by the `task-scheduler` capability.
#### Scenario: Submit a task via the API
- **WHEN** an integrator calls the task-submission endpoint with a valid goal and optional constraints
- **THEN** the API returns a task id that can be used to poll status, and the underlying `task-scheduler` records a new `queued` `ScheduledTask`
#### Scenario: Query status of a known task
- **WHEN** an integrator requests status for a task id that exists
- **THEN** the API returns that task's current status (`queued`, `assigned`, `dispatched`, `done`, or `failed`)
#### Scenario: Query status of an unknown task
- **WHEN** an integrator requests status for a task id that does not exist
- **THEN** the API returns a not-found response rather than an unhandled server error
### Requirement: Device and host visibility via the SDK
The system SHALL allow an external integrator to list devices and hosts known to the `device-pool` capability through the platform SDK's API.
#### Scenario: List devices across the pool
- **WHEN** an integrator calls the device-listing endpoint
- **THEN** the API returns every `PooledDevice` known to the pool, including owning host id and current (possibly `unreachable`) status
#### Scenario: List registered hosts
- **WHEN** an integrator calls the host-listing endpoint
- **THEN** the API returns every `HostRegistration` known to the pool, including last-seen timestamp
### Requirement: Plugin listing and registration via the SDK
The system SHALL allow an external integrator to list registered plugins and submit a new plugin manifest for registration through the platform SDK's API, backed by the `plugin-system` capability.
#### Scenario: List registered plugins
- **WHEN** an integrator calls the plugin-listing endpoint
- **THEN** the API returns every registered `PluginManifest`, including its `entry_point_kind` and whether it is wired to an execution path
#### Scenario: Register a new plugin manifest
- **WHEN** an integrator submits a valid plugin manifest to the plugin-registration endpoint
- **THEN** the API registers it via `plugin-system`'s `PluginRegistry` and returns the stored manifest, or a clear validation/conflict error if registration fails
### Requirement: Pluggable authentication hook with a safe default
The system SHALL evaluate every platform SDK route through a configurable `AuthProvider` hook, defaulting to a no-op provider that treats every caller as an anonymous, authenticated principal, so real authentication can be added later without changing route signatures.
#### Scenario: Default configuration allows anonymous access
- **WHEN** no `AuthProvider` is explicitly configured
- **THEN** every route accepts requests without rejecting them for lack of credentials
#### Scenario: Custom AuthProvider is honored
- **WHEN** a caller configures a custom `AuthProvider` that rejects a request
- **THEN** the platform SDK's routes return an authorization error for that request instead of proceeding, without any route's own handler code needing to change
### Requirement: Python SDK client mirrors the REST API
The system SHALL provide a Python client (`CloudClient`) exposing methods corresponding to each `/v1/...` route (submit task, get task status, list devices, list hosts, list plugins, register plugin), so integrators do not need to hand-construct HTTP requests.
#### Scenario: Client submits a task and retrieves status
- **WHEN** a caller uses `CloudClient` to submit a task and then fetch its status by the returned id
- **THEN** the client's methods produce the same result as calling the corresponding `/v1/...` endpoints directly over HTTP
+57
View File
@@ -0,0 +1,57 @@
# plugin-system Specification
## Purpose
TBD - created by archiving change cloud-runtime. Update Purpose after archive.
## Requirements
### Requirement: Plugin manifest schema
The system SHALL define a `PluginManifest` schema with a unique `name`, a `version`, an `entry_point_kind` restricted to `driver`, `tool`, or `skill`, and a `target` (a dotted module:attribute reference to the plugin's implementation), and SHALL reject a manifest missing any required field or using an unrecognized `entry_point_kind`.
#### Scenario: Valid manifest accepted
- **WHEN** a manifest with all required fields and a recognized `entry_point_kind` is submitted for registration
- **THEN** the registry accepts it and stores it as a known plugin
#### Scenario: Manifest with unrecognized entry_point_kind rejected
- **WHEN** a manifest declares an `entry_point_kind` other than `driver`, `tool`, or `skill`
- **THEN** the registry rejects it with a clear validation error and does not register it
#### Scenario: Duplicate plugin name rejected
- **WHEN** a manifest is submitted whose `name` matches an already-registered plugin
- **THEN** the registry rejects the new registration with a clear conflict error rather than silently overwriting the existing entry
### Requirement: Plugin discovery via entry points and manifest files
The system SHALL discover plugin manifests both from installed Python packages declaring an entry point in the `device_agent_runtime.plugins` group and from local `plugin.json` files under a configured scan path, feeding both sources into the same validation-and-registration path.
#### Scenario: Discovery via installed entry point
- **WHEN** an installed package declares an entry point in the `device_agent_runtime.plugins` group resolving to a valid manifest
- **THEN** `PluginRegistry.discover()` finds and registers it
#### Scenario: Discovery via local manifest file
- **WHEN** a `plugin.json` file exists under the configured plugin scan path and parses into a valid manifest
- **THEN** `PluginRegistry.discover()` finds and registers it
#### Scenario: Malformed manifest file is skipped, not fatal
- **WHEN** a `plugin.json` file under the scan path fails to parse or fails schema validation
- **THEN** `PluginRegistry.discover()` skips that file, records it as a discovery error, and continues discovering remaining plugins rather than aborting the whole scan
### Requirement: Driver-kind plugins register into the driver registry extension point
The system SHALL, for a manifest with `entry_point_kind == "driver"`, resolve its `target` to a driver-factory builder and register it under the manifest's `name` as a new `driver_type` in the existing driver-registry extension point, without requiring any edit to the `driver` package's own files.
#### Scenario: Driver plugin registered successfully
- **WHEN** a valid `driver`-kind manifest is registered and its `target` resolves to a callable driver-factory builder
- **THEN** the manifest's `name` becomes usable as a `driver_type` value by any caller building a driver factory, with no change to existing driver-registry code
#### Scenario: Driver registry extension point unavailable
- **WHEN** a `driver`-kind manifest is registered but the driver-registry's registration function is not importable in the running environment
- **THEN** the registry raises a clear, explicit error naming the missing integration point, rather than silently accepting the manifest without wiring it
### Requirement: Tool and skill plugin manifests are accepted but explicitly marked unwired
The system SHALL accept and store `tool`- and `skill`-kind plugin manifests (listable like any other registered plugin) but SHALL report them as not wired to any execution path, rather than implying they are active.
#### Scenario: Tool-kind manifest registered
- **WHEN** a valid `tool`-kind manifest is registered
- **THEN** the registry stores it and it appears in a plugin listing with a `wired: false` indicator, and no tool dispatch path is modified as a result
#### Scenario: Skill-kind manifest registered
- **WHEN** a valid `skill`-kind manifest is registered
- **THEN** the registry stores it and it appears in a plugin listing with a `wired: false` indicator, and no skill store or execution path is modified as a result
+49
View File
@@ -0,0 +1,49 @@
# semantic-scene Specification
## Purpose
TBD - created by archiving change semantic-scene-runtime. Update Purpose after archive.
## Requirements
### Requirement: Semantic scene enrichment from an existing Scene
The system SHALL provide a function that, given an existing `Scene` (as produced by the `scene-perception` capability), produces a `SemanticScene` consisting of a page identity string, a list of plain-language supported intents, and a list of per-widget purpose labels referencing the `Scene`'s element IDs, using exactly one LLM call.
#### Scenario: Enrichment succeeds for a recognizable screen
- **WHEN** `enrich_scene()` is called with a `Scene` describing a recognizable app screen (e.g. a chat screen with a text input and a send button)
- **THEN** it returns a `SemanticScene` with a non-empty `page` string, a non-empty `intents` list of plain-language strings, and a `widgets` list where each entry's `element_id` matches an element ID present in the input `Scene`
#### Scenario: Widget purpose labels reference only known elements
- **WHEN** the LLM response includes a widget purpose label whose `element_id` does not match any element ID in the input `Scene`
- **THEN** `enrich_scene()` discards that widget entry from the returned `SemanticScene` rather than propagating a dangling element reference
### Requirement: Enrichment failure degrades to no semantic scene, never blocks the loop
The system SHALL treat any enrichment failure (LLM call timeout, connection error, rate limit, malformed or schema-invalid response, or enrichment disabled by configuration) as a non-fatal condition, returning an absence of a semantic scene rather than raising an exception, so that callers always have a defined fallback of using the raw `Scene`.
#### Scenario: LLM call times out
- **WHEN** the enrichment LLM call does not complete within the configured timeout
- **THEN** `enrich_scene()` returns `None` and the caller proceeds using the raw `Scene` without the task step being marked as failed
#### Scenario: LLM response fails schema validation
- **WHEN** the enrichment LLM call returns a response that does not conform to the expected `SemanticScene` JSON schema
- **THEN** `enrich_scene()` returns `None` instead of raising, and no partially-parsed `SemanticScene` is returned
#### Scenario: Enrichment disabled by configuration
- **WHEN** semantic enrichment is disabled in configuration
- **THEN** `enrich_scene()` returns `None` immediately without making an LLM call
### Requirement: Structured, schema-constrained LLM output
The system SHALL request the enrichment LLM call using a schema-constrained structured-output mechanism so that any successful response is guaranteed to be valid JSON matching the `SemanticScene` shape, rather than relying on free-text parsing of an unconstrained model reply.
#### Scenario: Successful call yields directly parseable output
- **WHEN** the enrichment LLM call completes successfully
- **THEN** the raw response body is valid JSON matching the declared `SemanticScene` schema without requiring text extraction, regex matching, or a JSON-repair step
### Requirement: Semantic enrichment is opt-in and does not alter existing tool behavior
The system SHALL expose semantic enrichment through a new, separate tool entry point rather than modifying the existing `describe_screen` tool's signature or behavior, so that every existing caller of `describe_screen` continues to receive only a `Scene`, unchanged, unless it explicitly opts into the new semantic-enriched entry point.
#### Scenario: Existing describe_screen callers are unaffected
- **WHEN** an existing caller invokes the `describe_screen` tool as it did before this change
- **THEN** it receives the same `Scene` result as before, with no semantic enrichment attempted and no new LLM-related dependency invoked
#### Scenario: A caller opts into semantic enrichment
- **WHEN** a caller invokes the new semantic-enrichment tool entry point for a given device
- **THEN** it receives both the underlying `Scene` and, when enrichment succeeds, the corresponding `SemanticScene`; when enrichment fails or is disabled, it receives the `Scene` with an explicit absence of a `SemanticScene` rather than a partial or error result
+64
View File
@@ -0,0 +1,64 @@
# skill-authoring Specification
## Purpose
TBD - created by archiving change skill-learning-runtime. Update Purpose after archive.
## Requirements
### Requirement: Synthesis triggers only on successful task completion
The system SHALL synthesize a flow-template skill only when a task's final status is `succeeded`, and SHALL NOT attempt synthesis for a task that failed, was cancelled, or is still running.
#### Scenario: Successful task triggers synthesis
- **WHEN** a task completes with status `succeeded` and Skill Authoring is enabled
- **THEN** the system reads that task's timeline and goal and produces a flow-template skill candidate
#### Scenario: Failed task does not trigger synthesis
- **WHEN** a task completes with status `failed` (or is cancelled/still running)
- **THEN** the system does not synthesize any skill from that task's timeline
### Requirement: Skill Authoring defaults to disabled
The system SHALL leave Skill Authoring disabled by default in configuration, so applying this capability does not change the cost or latency of any existing task run until a caller explicitly enables it.
#### Scenario: Default configuration performs no synthesis
- **WHEN** a task completes successfully and Skill Authoring has not been explicitly enabled in configuration
- **THEN** the system performs no synthesis work and the task's completion path behaves exactly as it would without this capability
#### Scenario: Explicit enable activates synthesis
- **WHEN** an operator enables Skill Authoring in configuration
- **THEN** subsequently completed successful tasks are eligible for synthesis
### Requirement: Flow-template skill synthesized from executed tool-call sequence
The system SHALL derive a flow-template skill's ordered steps from the sequence of mutating tool calls (e.g. `tap`, `swipe`, `input_text`, `launch_app`) recorded in the completed task's timeline, in the order they were executed, and SHALL exclude read-only/observational tool calls (e.g. `describe_screen`, `screenshot`, `ui_tree`) from the synthesized step list.
#### Scenario: Mutating steps are included in order
- **WHEN** a task's timeline contains a sequence of `launch_app`, `tap`, `input_text`, `tap` tool calls that all succeeded
- **THEN** the synthesized skill's steps list contains those four steps in that same order
#### Scenario: Read-only observation calls are excluded
- **WHEN** a task's timeline includes `describe_screen` or `screenshot` calls interleaved with mutating calls
- **THEN** the synthesized skill's steps list omits those read-only calls and retains only the mutating steps
### Requirement: Parameter abstraction from cross-execution argument diffing
The system SHALL abstract a synthesized skill's step arguments into named parameters by comparing the newly executed argument values against a previously stored skill with the same tool-name step sequence, promoting any argument value that differs between the two executions into a named placeholder, and SHALL leave a first-time synthesis (no prior matching skeleton) with zero parameters.
#### Scenario: First execution of a flow has no parameters
- **WHEN** no previously stored skill shares the newly executed tool-name sequence
- **THEN** the synthesized skill is stored with its literal argument values and an empty parameters list
#### Scenario: Second, divergent execution promotes a differing value to a parameter
- **WHEN** a later successful task executes the same tool-name sequence as a stored skill but with a different literal value at one argument position
- **THEN** the system promotes that argument position to a named parameter in the skill's `parameters` schema and replaces the literal in `steps` with a `{param}` placeholder referencing it
#### Scenario: Identical repeated execution does not spuriously add parameters
- **WHEN** a later successful task executes the same tool-name sequence as a stored skill with identical argument values at every position
- **THEN** the system does not introduce any new parameter for that skill
### Requirement: Locally-authored skills are stored separately from synced skills
The system SHALL persist locally-synthesized flow-template skills tagged with a `source` of `local-synthesis`, in a store owned by this capability, and SHALL NOT write into or modify the externally-synced skill catalog store or its sync-only write contract.
#### Scenario: Synthesized skill is tagged as locally-authored
- **WHEN** a flow-template skill is synthesized from a completed task
- **THEN** its stored record has `source = "local-synthesis"` and no subscription identifier
#### Scenario: Synced skill catalog is untouched by synthesis
- **WHEN** a skill is synthesized and stored by this capability
- **THEN** no record in the externally-synced skill catalog store is created, modified, or removed as a result
@@ -0,0 +1,45 @@
# skill-embedding-retrieval Specification
## Purpose
TBD - created by archiving change skill-learning-runtime. Update Purpose after archive.
## Requirements
### Requirement: Skill text embedded on synthesis and re-synthesis
The system SHALL compute and persist an embedding vector for each locally-authored skill's name, description, and originating goal text whenever that skill is first synthesized or a new version is stored, associated with that skill's id and version.
#### Scenario: New skill gains an embedding
- **WHEN** a flow-template skill is synthesized for the first time
- **THEN** the system computes an embedding vector from its name, description, and originating goal, and stores it alongside the skill record
#### Scenario: New version gains its own embedding
- **WHEN** a new version of an existing skill is created
- **THEN** the system computes and stores an embedding for that version, independent of any embedding stored for prior versions
### Requirement: Embedding failure degrades to non-retrievable-by-similarity, never blocks synthesis
The system SHALL NOT allow an embedding-provider failure (timeout, rate limit, disabled configuration, connection error) to prevent a skill from being synthesized or versioned; on such failure, the skill SHALL be stored without a similarity-searchable embedding.
#### Scenario: Embedding call fails but skill is still stored
- **WHEN** the embedding provider call fails or times out during synthesis of an otherwise-successful skill
- **THEN** the skill's flow-template record is still stored, and it is retrievable by exact name/id lookup but excluded from similarity-based retrieval results until a subsequent embedding attempt succeeds
### Requirement: Ranked retrieval of candidate skills by goal similarity
The system SHALL provide a function that, given a new goal string and a requested result count, returns locally-authored skills that have a stored embedding, ranked by descending semantic similarity between the goal and each skill's stored embedding.
#### Scenario: Similar goal returns matching skill highest-ranked
- **WHEN** a new goal is semantically similar to a previously-learned skill's originating goal
- **THEN** that skill appears in the ranked candidate results, ordered ahead of less-similar skills
#### Scenario: Requested count limits results
- **WHEN** a caller requests the top `k` candidate skills for a goal
- **THEN** the system returns at most `k` ranked results, even if more embedded skills exist
#### Scenario: No embedded skills yields an empty result
- **WHEN** no locally-authored skill currently has a stored embedding
- **THEN** the retrieval function returns an empty ranked list rather than raising an error
### Requirement: Retrieval scoped to locally-authored skills unless explicitly extended
The system SHALL restrict ranked candidate retrieval to skills stored by this capability's own local-synthesis store by default, and SHALL treat inclusion of externally-synced skills as a separate, explicit extension rather than an implicit default.
#### Scenario: Default retrieval excludes synced-only skills without embeddings
- **WHEN** the skill catalog contains externally-synced skills that have never been embedded by this capability
- **THEN** ranked candidate retrieval returns only locally-authored skills with stored embeddings, without erroring on the presence of unembedded synced skills
+42
View File
@@ -0,0 +1,42 @@
# skill-versioning Specification
## Purpose
TBD - created by archiving change skill-learning-runtime. Update Purpose after archive.
## Requirements
### Requirement: Structural divergence triggers a new version
The system SHALL compare a newly synthesized flow's tool-name step sequence against the currently-stored version of the matching skill, and SHALL create a new version (rather than overwriting the stored one) whenever the tool-name sequence differs by insertion, deletion, or reordering of a step.
#### Scenario: Extra step triggers a new version
- **WHEN** a newly executed flow for a matching skill contains an additional tap step not present in the currently-stored version's sequence
- **THEN** the system stores a new version of the skill rather than overwriting the existing stored version
#### Scenario: Reordered steps trigger a new version
- **WHEN** a newly executed flow for a matching skill executes the same tool names as the stored version but in a different order
- **THEN** the system stores a new version of the skill
#### Scenario: Argument-value-only differences do not trigger a version bump
- **WHEN** a newly executed flow has the identical tool-name sequence as the stored version and differs only in argument values already covered by parameter abstraction
- **THEN** the system does not create a new version, and instead updates the existing version's parameters per the skill-authoring capability
### Requirement: Version history is retained, never silently overwritten
The system SHALL retain every version of a skill it creates, each carrying an incrementing `version` number and a reference to the version it diverged from, and SHALL NOT delete or overwrite a prior version's stored record when a new version is created.
#### Scenario: New version references its parent
- **WHEN** a new version of a skill is created due to structural divergence
- **THEN** the new version's record stores a reference to the prior version's id and an incremented version number
#### Scenario: Prior version remains fetchable
- **WHEN** a new version of a skill has been created
- **THEN** the prior version's record remains retrievable by its own id, unmodified
### Requirement: Default retrieval surfaces the newest version
The system SHALL treat the highest-numbered version of a skill as the default result returned by a lookup-by-name/goal-family query, while still allowing an explicit lookup of any specific prior version by its id.
#### Scenario: Lookup by name returns newest version
- **WHEN** a caller looks up a skill by its name or goal-family without specifying a version
- **THEN** the system returns the highest-numbered stored version of that skill
#### Scenario: Explicit id lookup returns the requested version
- **WHEN** a caller requests a skill by a specific prior version's id
- **THEN** the system returns that exact version's record, not the newest version
+60
View File
@@ -0,0 +1,60 @@
# task-scheduler Specification
## Purpose
TBD - created by archiving change cloud-runtime. Update Purpose after archive.
## Requirements
### Requirement: Task submission enqueues a scheduled task
The system SHALL allow a caller to submit a task (a goal string, or a reference to a `WorkflowDefinition`, plus optional device constraints: `driver_type`, required capability tags) and SHALL enqueue it as a `ScheduledTask` with status `queued`, returning a stable task id the caller can poll.
#### Scenario: Successful submission
- **WHEN** a caller submits a task with a goal and no constraints
- **THEN** the scheduler creates a `ScheduledTask` with status `queued`, assigns it a unique id, and returns that id to the caller without blocking for a device to become available
#### Scenario: Queue depth limit reached
- **WHEN** a caller submits a task while the queue already holds `config.max_queue_depth` queued tasks
- **THEN** the scheduler rejects the submission with a clear error rather than accepting an unbounded backlog
### Requirement: Assignment matches a queued task to an idle, constraint-matching device
The system SHALL assign a queued `ScheduledTask` to an idle `PooledDevice` (as reported by the `device-pool` capability) whose `driver_type` and capability tags satisfy the task's constraints, using a named, registrable `AssignmentStrategy`.
#### Scenario: Matching idle device available
- **WHEN** `assign()` runs and at least one idle `PooledDevice` matches the head-of-queue task's constraints
- **THEN** the scheduler selects one such device via the configured `AssignmentStrategy`, transitions the task to status `assigned`, and records the chosen `device_id`/`host_id`
#### Scenario: No matching device available
- **WHEN** `assign()` runs and no idle `PooledDevice` matches the head-of-queue task's constraints
- **THEN** the task remains `queued` (not failed), and `assign()` returns without error, ready to be retried on a later call
#### Scenario: Unknown assignment strategy configured
- **WHEN** `TaskScheduler` is configured with an `AssignmentStrategy` name that is not registered
- **THEN** the scheduler raises a clear configuration error at startup/first-assign rather than silently falling back to a default strategy
### Requirement: Assignment strategies are pluggable by name
The system SHALL provide an `AssignmentStrategy` registry mapping a strategy name to an implementation, with a default `fifo_match` strategy (oldest-queued matching task first, first matching idle device), and SHALL allow a new strategy to be added by registering a name without modifying `TaskScheduler`'s control flow.
#### Scenario: Default FIFO strategy orders by submission time
- **WHEN** two tasks with satisfiable, overlapping constraints are queued in order A then B, and one matching idle device exists
- **THEN** the default `fifo_match` strategy assigns the device to task A, leaving task B queued
#### Scenario: Adding a new strategy requires no scheduler edit
- **WHEN** a new `AssignmentStrategy` implementation is registered under a new name
- **THEN** `TaskScheduler` can be configured to use it by name alone, with no change to `scheduler.py`'s assignment control flow
### Requirement: Local dispatch executes an assignment via existing runners
The system SHALL provide a `TaskDispatcher` that, for an assignment whose device is owned by the local process's own host, executes the assigned task by composing the existing `agent-runtime` task-execution entry point (for a goal-based submission) or the `workflow-orchestration` workflow-execution entry point (for a workflow-based submission), without reimplementing planning/execution/retry logic.
#### Scenario: Dispatching a goal-based assignment
- **WHEN** `TaskDispatcher.dispatch()` is called with an assignment for a goal-based `ScheduledTask` whose device is local
- **THEN** the dispatcher constructs and runs a `Task` through the existing task-execution entry point, and updates the `ScheduledTask`'s status to `done` or `failed` based on the resulting task's outcome
#### Scenario: Dispatching a workflow-based assignment
- **WHEN** `TaskDispatcher.dispatch()` is called with an assignment referencing a `WorkflowDefinition` whose device is local
- **THEN** the dispatcher runs the definition through the existing workflow-execution entry point and updates the `ScheduledTask`'s status based on the resulting workflow run's outcome
### Requirement: Remote assignments are rejected explicitly, not silently ignored
The system SHALL raise a distinct, typed error when `TaskDispatcher.dispatch()` is called for an assignment whose device is owned by a host other than the dispatching process's own host, rather than attempting execution or silently no-op'ing.
#### Scenario: Assignment targets a remote host's device
- **WHEN** `TaskDispatcher.dispatch()` is called with an assignment whose `host_id` does not match the local process's own host id
- **THEN** the dispatcher raises a `RemoteDispatchNotSupportedError` and leaves the `ScheduledTask`'s status unchanged from `assigned`
@@ -0,0 +1,101 @@
# workflow-orchestration Specification
## Purpose
TBD - created by archiving change workflow-orchestration-runtime. Update Purpose after archive.
## Requirements
### Requirement: Workflow definition as an ordered, branchable list of typed steps
The system SHALL provide a `WorkflowDefinition` model representing an ordered, possibly-branching list of `WorkflowStep`s, where each step is exactly one of four kinds: a planned-goal step (a natural-language sub-goal delegated to the existing single-goal Planner/Executor loop), a skill-invocation step (a reference to a locally-synthesized flow-template skill plus argument values), a wait-for-condition step (a named condition, timeout, and poll interval), or a branch step (a named condition plus two target step ids). Each step SHALL have a unique `step_id` within its `WorkflowDefinition`.
#### Scenario: Workflow with heterogeneous step kinds is constructed
- **WHEN** a `WorkflowDefinition` is built with a planned-goal step, a skill-invocation step, a wait-for-condition step, and a branch step in sequence
- **THEN** the system accepts the definition and each step retains its declared kind and fields without requiring fields belonging to another step kind
#### Scenario: Duplicate step id is rejected
- **WHEN** a `WorkflowDefinition` is constructed with two steps sharing the same `step_id`
- **THEN** the system rejects the definition before any run is created from it
### Requirement: Persisted, checkpointed workflow run
The system SHALL persist a `WorkflowRun` record for each execution of a `WorkflowDefinition`, containing the run's status, the currently active step id, workflow-scoped variables, and a per-step result log, and SHALL update this record after every completed step before advancing to the next one.
#### Scenario: Run status transitions as steps execute
- **WHEN** a `WorkflowRun` is started for a `WorkflowDefinition`
- **THEN** the system creates a persisted run record with status `running` and, as each step completes, updates the persisted `current_step_id` and per-step result log before the next step begins
#### Scenario: Run reaches terminal status
- **WHEN** all steps in a `WorkflowDefinition` complete successfully, or a step fails without a defined recovery path
- **THEN** the system updates the persisted `WorkflowRun` status to `completed` or `failed` respectively, and records a failure reason when failed
### Requirement: Resume from last checkpoint without re-executing completed steps
The system SHALL support resuming an interrupted `WorkflowRun` from its last persisted checkpoint, continuing execution from the currently active step without re-executing any step already recorded as completed in that run's step result log.
#### Scenario: Resume after simulated process restart
- **WHEN** a `WorkflowRun` has completed its first two steps and the process driving it stops before the third step completes, and a new `WorkflowRunner` instance is later pointed at the same persisted run id
- **THEN** the system resumes execution starting at the third step and does not re-invoke the tool calls or sub-goal already recorded as completed for the first two steps
#### Scenario: Resume on an already-completed run is a no-op
- **WHEN** `resume` is called with the id of a `WorkflowRun` whose status is already `completed`
- **THEN** the system returns the run's existing final state without executing any further steps
### Requirement: Planned-goal step delegates to the existing single-goal loop
The system SHALL execute a planned-goal step by delegating its sub-goal to the existing Planner/Executor Observe-Think-Act-Observe loop for a single task, and SHALL derive that step's success or failure from the resulting task's final status.
#### Scenario: Planned-goal step succeeds
- **WHEN** a planned-goal step's delegated task reaches a completed status
- **THEN** the workflow step is recorded as succeeded and the run advances to the next step
#### Scenario: Planned-goal step fails
- **WHEN** a planned-goal step's delegated task reaches a failed status
- **THEN** the workflow step is recorded as failed with the task's failure reason and the run's status becomes `failed` unless a branch step defines an alternate path
### Requirement: Skill-invocation step resolves parameters and executes a flow-template skill
The system SHALL execute a skill-invocation step by validating its supplied argument values against the referenced flow-template skill's declared parameters, substituting the validated values into the skill's stored tool-call template, and executing the resolved tool calls in order.
#### Scenario: Skill invocation with valid parameters executes resolved tool calls
- **WHEN** a skill-invocation step supplies argument values that satisfy the referenced skill's declared required parameters
- **THEN** the system substitutes those values into the skill's stored steps and executes the resulting tool calls in the skill's recorded order
#### Scenario: Skill invocation with a missing required parameter fails without executing any tool call
- **WHEN** a skill-invocation step omits a value for a parameter the referenced skill declares as required
- **THEN** the system fails the step before issuing any tool call and records the missing-parameter reason
#### Scenario: Skill invocation referencing a non-flow-template skill is rejected
- **WHEN** a skill-invocation step references a skill whose kind is not a flow-template
- **THEN** the system fails the step as a step-definition error rather than attempting to execute it
### Requirement: Wait-for-condition step polls until satisfied or timed out
The system SHALL execute a wait-for-condition step by repeatedly evaluating its named condition at the step's configured poll interval until the condition is satisfied or the step's configured timeout elapses.
#### Scenario: Condition becomes true before timeout
- **WHEN** a wait-for-condition step's condition evaluates true within its configured timeout
- **THEN** the system stops polling, records the step as succeeded, and advances the run to the next step
#### Scenario: Condition never becomes true before timeout
- **WHEN** a wait-for-condition step's condition has not evaluated true by its configured timeout
- **THEN** the system records the step as failed with a timeout reason and the run's status becomes `failed` unless a branch step defines an alternate path
### Requirement: Branch step selects the next step from a condition
The system SHALL execute a branch step by evaluating its named condition and setting the run's next active step to the branch's configured true-target step id or false-target step id accordingly, instead of advancing to the next step in definition order.
#### Scenario: Branch condition true selects the true-target step
- **WHEN** a branch step's condition evaluates true
- **THEN** the system sets the run's current step to the branch's `on_true` target step id
#### Scenario: Branch condition false selects the false-target step
- **WHEN** a branch step's condition evaluates false
- **THEN** the system sets the run's current step to the branch's `on_false` target step id
### Requirement: Condition kinds are pluggable via a registry
The system SHALL evaluate wait-for-condition and branch step conditions through a registry mapping a condition kind name to an evaluator, SHALL provide at least `scene_contains_text`, `world_variable_equals`, `elapsed_seconds`, and `step_result_success` as built-in kinds, and SHALL allow a new condition kind to be added without modifying the workflow runner's step-dispatch logic.
#### Scenario: Built-in condition kind evaluates against current state
- **WHEN** a wait-for-condition or branch step specifies the `scene_contains_text` kind with a target text value
- **THEN** the system evaluates the condition against the most recently observed scene and returns true only when the target text is present
#### Scenario: Unregistered condition kind fails the step
- **WHEN** a step specifies a condition kind that is not present in the registry
- **THEN** the system fails that step with an unrecognized-condition-kind reason instead of executing an undefined check
#### Scenario: World-state-dependent condition degrades safely when world state is absent
- **WHEN** a `world_variable_equals` condition is evaluated for a run whose task context has no `WorldState` available
- **THEN** the system treats the condition as not satisfied rather than raising an error, allowing the step to continue polling until its timeout
+90
View File
@@ -0,0 +1,90 @@
# world-model Specification
## Purpose
TBD - created by archiving change world-model-runtime. Update Purpose after archive.
## Requirements
### Requirement: Persistent per-task WorldState
The system SHALL maintain one `WorldState` per task, consisting of a current app identifier, a current page identifier, a `variables` mapping, and a bounded history of recent semantic-scene/action pairs, that persists across the task's steps rather than being re-derived from scratch each step.
#### Scenario: WorldState survives across steps within a task
- **WHEN** a task executes multiple steps in sequence
- **THEN** the `WorldState` object associated with the task is the same object (or reflects continuously accumulated updates) across those steps, not reset between steps
#### Scenario: WorldState is scoped to a single task
- **WHEN** two different tasks run (sequentially or concurrently) against the same or different devices
- **THEN** each task has its own independent `WorldState`, and neither task's `WorldState` reflects the other task's app/page/variables/history
### Requirement: Incremental update after each executed step
The system SHALL update a task's `WorldState` via a hook invoked once per executed step in the agent runtime's step loop, deriving the update from the step's observed `Scene`, optional `SemanticScene`, the `PlannedStep` that was executed, and its `StepResult`, without introducing a new LLM call or new network I/O.
#### Scenario: Update runs after a successful step
- **WHEN** the agent runtime's step loop executes a step and records a successful `StepResult`
- **THEN** the task's `WorldState` update hook is invoked with that step's `Scene`, `SemanticScene` (if any), the executed `PlannedStep`, and the `StepResult`, and updates the persisted `WorldState` accordingly
#### Scenario: Update runs after a failed step
- **WHEN** the agent runtime's step loop executes a step and records a failed `StepResult`
- **THEN** the task's `WorldState` update hook is still invoked with that step's data, and the update proceeds without raising an exception or blocking the loop's continuation/failure handling
#### Scenario: Update derivation makes no external calls
- **WHEN** the `WorldState` update hook runs for any step
- **THEN** the update completes using only the data already passed into the hook, without making any LLM call or other network request
### Requirement: Current app and page tracking
The system SHALL derive and refresh `current_app` from successful app-lifecycle actions (e.g. `launch_app`, `terminate_app`) and SHALL derive and refresh `current_page` from the current step's `SemanticScene` page identity when a `SemanticScene` is available, leaving each field unchanged when no corresponding signal is present in a given step.
#### Scenario: Launching an app updates current_app
- **WHEN** a step executes a successful `launch_app` action naming an app/bundle identifier
- **THEN** the task's `WorldState.current_app` is updated to that identifier
#### Scenario: A page identity from SemanticScene updates current_page
- **WHEN** a step's enrichment produces a `SemanticScene` with a non-empty `page` value
- **THEN** the task's `WorldState.current_page` is updated to that `page` value
#### Scenario: No page signal leaves current_page unchanged
- **WHEN** a step has no `SemanticScene` available (enrichment disabled, unavailable, or failed for that step)
- **THEN** the task's `WorldState.current_page` retains its previous value rather than being cleared or set to an empty/placeholder value
### Requirement: Explicit variable memorization
The system SHALL update `WorldState.variables` only when a step's `PlannedStep.args` contains an explicit memorization instruction, and SHALL NOT infer or write arbitrary variables from step content otherwise.
#### Scenario: A step explicitly remembers a value
- **WHEN** a step's `PlannedStep.args` includes an explicit key/value pair designated for memorization
- **THEN** the task's `WorldState.variables` is updated to include that key/value pair
#### Scenario: A step without a memorization instruction does not change variables
- **WHEN** a step's `PlannedStep.args` contains no explicit memorization instruction
- **THEN** the task's `WorldState.variables` is left unchanged by that step's update
### Requirement: Bounded history of recent scene/action pairs
The system SHALL maintain `WorldState.history` as a fixed-size, bounded collection of the most recent semantic-scene-or-scene/action pairs, automatically evicting the oldest entry when a new entry is added past the configured bound, so that history size never grows unboundedly with task length.
#### Scenario: History accumulates recent entries up to the bound
- **WHEN** a task executes a number of steps less than or equal to the configured history bound
- **THEN** `WorldState.history` contains one entry per executed step, in order from oldest to newest
#### Scenario: History evicts the oldest entry once the bound is exceeded
- **WHEN** a task executes more steps than the configured history bound
- **THEN** `WorldState.history` retains only the most recent entries up to the bound, with earlier entries evicted, and never exceeds the configured bound in length
### Requirement: Read-only WorldState available to the Planner
The system SHALL expose a task's current `WorldState` to the Planner as an additional, read-only input alongside the current `SemanticScene`/`Scene`, without requiring existing Planner implementations or call sites to change to keep working.
#### Scenario: Planner can read current WorldState
- **WHEN** the agent runtime invokes the Planner to produce the next steps for a task
- **THEN** the Planner is given access to the task's current `WorldState` (current app, current page, variables, bounded history) as of the most recently completed step
#### Scenario: Existing Planner call sites keep working unmodified
- **WHEN** an existing caller invokes the Planner's planning entry point without passing any world-state argument
- **THEN** the call succeeds exactly as it did before this capability existed, with the Planner treating the absence of world-state input as equivalent to "no world state available"
### Requirement: World Runtime tracking failure never blocks the task loop
The system SHALL treat any failure or unavailable input during a `WorldState` update (e.g. missing `SemanticScene`, missing expected `PlannedStep` arguments, disabled configuration) as non-fatal, leaving the affected `WorldState` fields unchanged rather than raising an exception that would interrupt the agent runtime's step loop.
#### Scenario: Missing expected data during update does not raise
- **WHEN** the `WorldState` update hook runs for a step whose data lacks a field an update rule expects (e.g. no app identifier on a `launch_app` step)
- **THEN** the update hook completes without raising, leaving the corresponding `WorldState` field at its prior value
#### Scenario: World Runtime tracking disabled by configuration
- **WHEN** World Runtime tracking is disabled in configuration for a task run
- **THEN** the agent runtime's step loop proceeds normally without invoking the `WorldState` update hook, and the Planner receives an absence of world-state input rather than a partially-updated or stale `WorldState`