feat: checkpoint device agent runtime milestones
This commit is contained in:
@@ -0,0 +1,60 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Synthesis triggers only on successful task completion
|
||||
The system SHALL synthesize a flow-template skill only when a task's final status is `succeeded`, and SHALL NOT attempt synthesis for a task that failed, was cancelled, or is still running.
|
||||
|
||||
#### Scenario: Successful task triggers synthesis
|
||||
- **WHEN** a task completes with status `succeeded` and Skill Authoring is enabled
|
||||
- **THEN** the system reads that task's timeline and goal and produces a flow-template skill candidate
|
||||
|
||||
#### Scenario: Failed task does not trigger synthesis
|
||||
- **WHEN** a task completes with status `failed` (or is cancelled/still running)
|
||||
- **THEN** the system does not synthesize any skill from that task's timeline
|
||||
|
||||
### Requirement: Skill Authoring defaults to disabled
|
||||
The system SHALL leave Skill Authoring disabled by default in configuration, so applying this capability does not change the cost or latency of any existing task run until a caller explicitly enables it.
|
||||
|
||||
#### Scenario: Default configuration performs no synthesis
|
||||
- **WHEN** a task completes successfully and Skill Authoring has not been explicitly enabled in configuration
|
||||
- **THEN** the system performs no synthesis work and the task's completion path behaves exactly as it would without this capability
|
||||
|
||||
#### Scenario: Explicit enable activates synthesis
|
||||
- **WHEN** an operator enables Skill Authoring in configuration
|
||||
- **THEN** subsequently completed successful tasks are eligible for synthesis
|
||||
|
||||
### Requirement: Flow-template skill synthesized from executed tool-call sequence
|
||||
The system SHALL derive a flow-template skill's ordered steps from the sequence of mutating tool calls (e.g. `tap`, `swipe`, `input_text`, `launch_app`) recorded in the completed task's timeline, in the order they were executed, and SHALL exclude read-only/observational tool calls (e.g. `describe_screen`, `screenshot`, `ui_tree`) from the synthesized step list.
|
||||
|
||||
#### Scenario: Mutating steps are included in order
|
||||
- **WHEN** a task's timeline contains a sequence of `launch_app`, `tap`, `input_text`, `tap` tool calls that all succeeded
|
||||
- **THEN** the synthesized skill's steps list contains those four steps in that same order
|
||||
|
||||
#### Scenario: Read-only observation calls are excluded
|
||||
- **WHEN** a task's timeline includes `describe_screen` or `screenshot` calls interleaved with mutating calls
|
||||
- **THEN** the synthesized skill's steps list omits those read-only calls and retains only the mutating steps
|
||||
|
||||
### Requirement: Parameter abstraction from cross-execution argument diffing
|
||||
The system SHALL abstract a synthesized skill's step arguments into named parameters by comparing the newly executed argument values against a previously stored skill with the same tool-name step sequence, promoting any argument value that differs between the two executions into a named placeholder, and SHALL leave a first-time synthesis (no prior matching skeleton) with zero parameters.
|
||||
|
||||
#### Scenario: First execution of a flow has no parameters
|
||||
- **WHEN** no previously stored skill shares the newly executed tool-name sequence
|
||||
- **THEN** the synthesized skill is stored with its literal argument values and an empty parameters list
|
||||
|
||||
#### Scenario: Second, divergent execution promotes a differing value to a parameter
|
||||
- **WHEN** a later successful task executes the same tool-name sequence as a stored skill but with a different literal value at one argument position
|
||||
- **THEN** the system promotes that argument position to a named parameter in the skill's `parameters` schema and replaces the literal in `steps` with a `{param}` placeholder referencing it
|
||||
|
||||
#### Scenario: Identical repeated execution does not spuriously add parameters
|
||||
- **WHEN** a later successful task executes the same tool-name sequence as a stored skill with identical argument values at every position
|
||||
- **THEN** the system does not introduce any new parameter for that skill
|
||||
|
||||
### Requirement: Locally-authored skills are stored separately from synced skills
|
||||
The system SHALL persist locally-synthesized flow-template skills tagged with a `source` of `local-synthesis`, in a store owned by this capability, and SHALL NOT write into or modify the externally-synced skill catalog store or its sync-only write contract.
|
||||
|
||||
#### Scenario: Synthesized skill is tagged as locally-authored
|
||||
- **WHEN** a flow-template skill is synthesized from a completed task
|
||||
- **THEN** its stored record has `source = "local-synthesis"` and no subscription identifier
|
||||
|
||||
#### Scenario: Synced skill catalog is untouched by synthesis
|
||||
- **WHEN** a skill is synthesized and stored by this capability
|
||||
- **THEN** no record in the externally-synced skill catalog store is created, modified, or removed as a result
|
||||
@@ -0,0 +1,41 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Skill text embedded on synthesis and re-synthesis
|
||||
The system SHALL compute and persist an embedding vector for each locally-authored skill's name, description, and originating goal text whenever that skill is first synthesized or a new version is stored, associated with that skill's id and version.
|
||||
|
||||
#### Scenario: New skill gains an embedding
|
||||
- **WHEN** a flow-template skill is synthesized for the first time
|
||||
- **THEN** the system computes an embedding vector from its name, description, and originating goal, and stores it alongside the skill record
|
||||
|
||||
#### Scenario: New version gains its own embedding
|
||||
- **WHEN** a new version of an existing skill is created
|
||||
- **THEN** the system computes and stores an embedding for that version, independent of any embedding stored for prior versions
|
||||
|
||||
### Requirement: Embedding failure degrades to non-retrievable-by-similarity, never blocks synthesis
|
||||
The system SHALL NOT allow an embedding-provider failure (timeout, rate limit, disabled configuration, connection error) to prevent a skill from being synthesized or versioned; on such failure, the skill SHALL be stored without a similarity-searchable embedding.
|
||||
|
||||
#### Scenario: Embedding call fails but skill is still stored
|
||||
- **WHEN** the embedding provider call fails or times out during synthesis of an otherwise-successful skill
|
||||
- **THEN** the skill's flow-template record is still stored, and it is retrievable by exact name/id lookup but excluded from similarity-based retrieval results until a subsequent embedding attempt succeeds
|
||||
|
||||
### Requirement: Ranked retrieval of candidate skills by goal similarity
|
||||
The system SHALL provide a function that, given a new goal string and a requested result count, returns locally-authored skills that have a stored embedding, ranked by descending semantic similarity between the goal and each skill's stored embedding.
|
||||
|
||||
#### Scenario: Similar goal returns matching skill highest-ranked
|
||||
- **WHEN** a new goal is semantically similar to a previously-learned skill's originating goal
|
||||
- **THEN** that skill appears in the ranked candidate results, ordered ahead of less-similar skills
|
||||
|
||||
#### Scenario: Requested count limits results
|
||||
- **WHEN** a caller requests the top `k` candidate skills for a goal
|
||||
- **THEN** the system returns at most `k` ranked results, even if more embedded skills exist
|
||||
|
||||
#### Scenario: No embedded skills yields an empty result
|
||||
- **WHEN** no locally-authored skill currently has a stored embedding
|
||||
- **THEN** the retrieval function returns an empty ranked list rather than raising an error
|
||||
|
||||
### Requirement: Retrieval scoped to locally-authored skills unless explicitly extended
|
||||
The system SHALL restrict ranked candidate retrieval to skills stored by this capability's own local-synthesis store by default, and SHALL treat inclusion of externally-synced skills as a separate, explicit extension rather than an implicit default.
|
||||
|
||||
#### Scenario: Default retrieval excludes synced-only skills without embeddings
|
||||
- **WHEN** the skill catalog contains externally-synced skills that have never been embedded by this capability
|
||||
- **THEN** ranked candidate retrieval returns only locally-authored skills with stored embeddings, without erroring on the presence of unembedded synced skills
|
||||
@@ -0,0 +1,38 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Structural divergence triggers a new version
|
||||
The system SHALL compare a newly synthesized flow's tool-name step sequence against the currently-stored version of the matching skill, and SHALL create a new version (rather than overwriting the stored one) whenever the tool-name sequence differs by insertion, deletion, or reordering of a step.
|
||||
|
||||
#### Scenario: Extra step triggers a new version
|
||||
- **WHEN** a newly executed flow for a matching skill contains an additional tap step not present in the currently-stored version's sequence
|
||||
- **THEN** the system stores a new version of the skill rather than overwriting the existing stored version
|
||||
|
||||
#### Scenario: Reordered steps trigger a new version
|
||||
- **WHEN** a newly executed flow for a matching skill executes the same tool names as the stored version but in a different order
|
||||
- **THEN** the system stores a new version of the skill
|
||||
|
||||
#### Scenario: Argument-value-only differences do not trigger a version bump
|
||||
- **WHEN** a newly executed flow has the identical tool-name sequence as the stored version and differs only in argument values already covered by parameter abstraction
|
||||
- **THEN** the system does not create a new version, and instead updates the existing version's parameters per the skill-authoring capability
|
||||
|
||||
### Requirement: Version history is retained, never silently overwritten
|
||||
The system SHALL retain every version of a skill it creates, each carrying an incrementing `version` number and a reference to the version it diverged from, and SHALL NOT delete or overwrite a prior version's stored record when a new version is created.
|
||||
|
||||
#### Scenario: New version references its parent
|
||||
- **WHEN** a new version of a skill is created due to structural divergence
|
||||
- **THEN** the new version's record stores a reference to the prior version's id and an incremented version number
|
||||
|
||||
#### Scenario: Prior version remains fetchable
|
||||
- **WHEN** a new version of a skill has been created
|
||||
- **THEN** the prior version's record remains retrievable by its own id, unmodified
|
||||
|
||||
### Requirement: Default retrieval surfaces the newest version
|
||||
The system SHALL treat the highest-numbered version of a skill as the default result returned by a lookup-by-name/goal-family query, while still allowing an explicit lookup of any specific prior version by its id.
|
||||
|
||||
#### Scenario: Lookup by name returns newest version
|
||||
- **WHEN** a caller looks up a skill by its name or goal-family without specifying a version
|
||||
- **THEN** the system returns the highest-numbered stored version of that skill
|
||||
|
||||
#### Scenario: Explicit id lookup returns the requested version
|
||||
- **WHEN** a caller requests a skill by a specific prior version's id
|
||||
- **THEN** the system returns that exact version's record, not the newest version
|
||||
Reference in New Issue
Block a user