feat: checkpoint device agent runtime milestones
This commit is contained in:
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-07-06
|
||||
@@ -0,0 +1,87 @@
|
||||
## Context
|
||||
|
||||
Two capabilities already exist as pending, unapplied changes that this milestone sits between: `task-memory` (`apex-agent-mvp`) persists a per-task `Timeline` — `storage/timeline.py`'s `Timeline.append()` writes one `TimelineRecord{index, scene, prompt, tool_call, result, timestamp, screenshot_path}` per executed step, retrievable in order via `Timeline.read(task_id)` — and `skill-catalog-subscription` defines a `Skill` data model (`kind = knowledge | flow_template`, shared metadata `id/name/description/version/tags`, plus for `flow_template` an ordered `steps` list of tool-name + args-template and a `parameters` schema) synced *from* an external Subscription Platform into a local read-mostly catalog (`skill-catalog-subscription`'s own spec explicitly rejects local writes outside of sync: "a caller attempts to create or edit a skill directly in the local catalog... reject or ignore"). Neither change gives the agent a way to go from "I just executed a successful sequence of tool calls" to "that sequence is now a reusable skill" — `task-memory`'s `Timeline` is write-once/read-only historical record, and `skill-catalog`'s local store is deliberately sync-only, not authorable.
|
||||
|
||||
This creates a real tension this design must resolve explicitly: `skill-catalog-subscription`'s spec says the local catalog rejects non-sync writes, but this change's whole point is to write locally-synthesized skills into a catalog. The resolution (D6 below) is that locally-authored skills live in a **separate local store** (`skills_learning/`'s own catalog table) that composes with, rather than writes into, the `skill-catalog-subscription` store — the two are presented to a Planner/MCP-tool layer as one logical namespace in prose, but this change does not touch `skill-catalog-subscription`'s schema, sync semantics, or its "sync-only" write rule.
|
||||
|
||||
`runtime/task.py`'s `TaskRunner.run()` (code-complete, unapplied) is the Observe→Think→Act→Observe loop; it already knows the task's `goal`, calls `Timeline.append()` once per step (via whatever wiring `apex-agent-mvp` gives it — the loop has access to everything `Timeline` needs), and terminates with a final status. This change adds exactly one new integration point at the very end of that loop: a hook fired once, only on success.
|
||||
|
||||
Two stakeholders: `runtime/task.py`'s `TaskRunner` (must gain a well-defined, optional, no-op-by-default hook), and a future LLM-driven Planner (not built in this change, matching `world-model-runtime`'s and `semantic-scene-runtime`'s precedent of "expose the input, defer the smart consumer") that will eventually call `retrieve_candidate_skills()` before planning from scratch.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Synthesize a parameterized `FlowTemplateSkill` from a completed task's `Timeline` + goal, automatically, with no manual authoring step.
|
||||
- Detect when a later execution that would synthesize into "the same" skill (same name/goal-family) has actually diverged in its step sequence, and version rather than silently overwrite.
|
||||
- Make locally-learned skills discoverable by semantic similarity to a new goal, not just exact name/id lookup, so a Planner can find "something like this" even when the new goal's wording differs from the one that produced the stored skill.
|
||||
- Compose cleanly with `skill-catalog-subscription`'s existing model and its "sync-only local writes" rule — do not require a delta against that change's spec.
|
||||
- Keep the synthesis/versioning/embedding path entirely off the hot Observe→Act loop — it runs once, after a task finishes, never mid-task.
|
||||
- Default to disabled, so applying this change does not silently add CPU/LLM cost to every successful task run.
|
||||
|
||||
**Non-Goals:**
|
||||
- No change to `skill-catalog-subscription`'s sync contract, MCP tool surface (`list_skills`/`search_skills`/`get_skill`), or subscription/visibility/entitlement model.
|
||||
- No Planner wiring that actually *consumes* `retrieve_candidate_skills()` results to skip planning — this change only makes retrieval available, matching `world-model-runtime`'s precedent of exposing a read-only input without teaching a Planner to use it.
|
||||
- No skill *execution* engine (running a `FlowTemplateSkill`'s steps against `tools/` with resolved parameters) — `skill-catalog-subscription`'s proposal already scoped a "run_skill_flow parameter-resolution helper" as its own concern; this change produces skills, it does not add a new runner for them.
|
||||
- No cross-device/cross-tenant skill sharing beyond whatever `skill-catalog`'s existing storage model already implies.
|
||||
- No UI (`web-console`'s domain, not touched here).
|
||||
- No multi-provider embedding abstraction — a single concrete embedding client is enough for this milestone, matching `semantic-scene-runtime`'s D-non-goal of not building a multi-provider LLM abstraction prematurely (YAGNI).
|
||||
- Failed or partially-completed tasks are never synthesized into skills — only `status = succeeded` timelines are eligible input.
|
||||
|
||||
## Decisions
|
||||
|
||||
### D1: New `skills_learning/` package, sibling to `semantic/` and `world/`, not folded into `runtime/` or `skills/`
|
||||
`skills_learning/` (`synthesis.py`, `versioning.py`, `embeddings.py`, `retrieval.py`, `config.py`) is its own package rather than extending `skill-catalog-subscription`'s planned `skills/` package (`skills/models.py`, `skills/catalog.py`, `skills/sync_client.py`, `skills/mcp_tools.py`) in place. Learning-from-execution has fundamentally different runtime characteristics from that package's sync-client role: it reads `task-memory`'s `Timeline`, it makes an embedding call, and it produces new skill records rather than pulling existing ones from an external platform. Keeping it separate means `skill-catalog-subscription`, when eventually implemented, does not need to anticipate a local-authoring code path it explicitly designed against (its own spec's "local catalog is not independently authored" scenario).
|
||||
**Alternative considered**: add authoring/versioning/embedding logic directly inside `skills/catalog.py` behind a feature flag. Rejected — it would make the sync-client package responsible for a write path its own spec explicitly forbids for the *synced* store, forcing every future reader of `skills/catalog.py` to reason about two different write-authority rules (sync-only vs. locally-authored) inside one module; a separate package with its own store keeps each write-authority rule enforceable at the package boundary, mirroring `semantic-scene-runtime`'s D1 and `world-model-runtime`'s D1 precedent of "new sibling package over folding into the layer below."
|
||||
|
||||
### D2: Synthesis triggered by an optional `TaskRunner.on_task_succeeded` hook, not a background scanner over `task-memory`
|
||||
`runtime/task.py`'s `TaskRunner.run()` gains an optional constructor argument `on_task_succeeded: Callable[[str, str, Timeline], None] | None = None` (task_id, goal, timeline), invoked exactly once, immediately after the loop determines final `status = succeeded`, before `run()` returns. When Skill Authoring is enabled in `skills_learning/config.py`, `TaskRunner` wires this to `synthesis.synthesize_flow_skill`; when disabled or left `None`, behavior is identical to today.
|
||||
**Alternative considered**: a separate offline batch job that periodically scans `storage/task_metadata.py`'s SQLite table for newly-succeeded tasks and synthesizes skills asynchronously. Rejected for this milestone — a batch job adds a second process/scheduling concern (when does it run, how does it avoid re-processing the same task twice) for a benefit (decoupling from the task loop's latency) that doesn't matter here, since synthesis is deliberately post-completion, not mid-loop; an in-process hook fired once at completion is simpler and has no scheduling state to get wrong. A batch/backfill mode remains a reasonable follow-up once there's a concrete need to synthesize skills from tasks that ran before this change existed.
|
||||
|
||||
### D3: Parameter abstraction via cross-goal diffing against the same tool-call skeleton, not NLP-based slot extraction from the goal string
|
||||
`synthesize_flow_skill(goal, timeline)` first extracts the ordered `(tool_name, args)` sequence from `timeline.read(task_id)`'s `tool_call` fields (dropping non-mutating/read-only calls like `describe_screen`/`screenshot`/`ui_tree` from the *template*, since those are re-derivable at replay time, while keeping `tap`/`swipe`/`input_text`/`launch_app`/similar mutating calls). It then looks up any already-stored skill whose stored steps have the same tool-name sequence (same skeleton, e.g. `launch_app → tap → input_text → tap`); if one exists, it diffs argument values position-by-position across the two executions and promotes any value that differs between the stored version and this run into a named `{param}` placeholder (e.g. a `input_text` call's literal text becomes `{search_query}`), inferring the parameter's `name` from the corresponding UI element's label/purpose when available (reusing `semantic-scene-runtime`'s `SemanticScene.widgets[].purpose` label as the naming source, when present, else falling back to a positional name like `param_2`). If no matching skeleton exists yet, the very first execution is stored as-is with zero parameters (nothing to diff against) and only gains parameters once a second, divergent-in-values execution is synthesized.
|
||||
**Alternative considered**: parse the natural-language `goal` string itself (e.g. via NER/pattern matching) to identify which words are "the variable part" (a search term, a contact name) and map those directly to template parameters. Rejected — goal phrasing is free-form and unconstrained (matching `semantic-scene-runtime`'s own open-vocabulary stance on intents), so goal-string parsing is fragile and would require its own NLP pipeline; diffing the *executed steps'* concrete argument values across two runs of "the same" flow is grounded in what the agent actually did, not what it said it wanted, and needs no parameter-worthy-token classifier of its own.
|
||||
|
||||
### D4: Versioning triggers on tool-name-sequence divergence beyond a configured tolerance, tracked as an explicit version chain
|
||||
`versioning.py`'s `diff_flow_versions(stored_steps, executed_steps) -> VersionDiff` computes: (a) whether the tool-name sequence itself differs (insertion/deletion/reorder of a step — always triggers a new version, no tolerance), and (b) for a matching skeleton, what fraction of argument positions newly qualify as "must become a parameter" per D3 (a configurable tolerance, default: any position that has ever varied is fine to keep parameterizing on the same version; only a *sequence*-level change forces a version bump). Every stored `FlowTemplateSkill` version keeps `version: int` (bumped) and `parent_version_id: str | None` (points at the version it diverged from), so history is a chain, not an overwrite; the newest version is what `skill-catalog`-style `get_skill`/list operations surface by default, but older versions remain fetchable by id.
|
||||
**Alternative considered**: version by content hash of the full step list (any byte-level difference is a new version, including argument-value-only changes that D3 would otherwise absorb into a parameter). Rejected — this would create a new version on essentially every synthesis run (since argument values differ per goal by design), defeating the entire point of parameterization; divergence has to be judged at the *tool-name-sequence* level (structural), with argument-value differences intentionally absorbed as parameters rather than treated as version-worthy changes, otherwise D3's parameter abstraction and D4's versioning contradict each other.
|
||||
|
||||
### D5: Embedding-based retrieval via a dedicated `EmbeddingIndex`, not full-text/keyword search reused from `skill-catalog`'s existing search
|
||||
`skill-embedding-retrieval` embeds `f"{name}: {description}\nOriginal goal: {goal}"` per locally-authored skill into a fixed-dimension vector using a single concrete embedding client (provider/model selected in implementation, config-overridable, mirroring `semantic-scene-runtime`'s D4 "config value, not hardcoded" stance), stores it in a new `skill_embeddings` local index (skill id → vector + model name + `updated_at`, re-embedded when a skill's description/goal changes at re-synthesis), and answers `retrieve_candidate_skills(goal, top_k)` via cosine similarity over that index, entirely in-process (no external vector-DB dependency introduced for this milestone's expected scale). `skill-catalog-subscription`'s existing `search_skills` (name/tag/description substring matching) is left untouched and is a different, complementary retrieval mode (exact/keyword vs. semantic).
|
||||
**Alternative considered**: extend `skill-catalog-subscription`'s planned `search_skills` MCP tool to also do embedding similarity internally. Rejected — that tool's contract (per its own spec) is a synchronous, no-network-round-trip local query over the synced catalog; folding an embedding-model call into it would either force every `search_skills` invocation to pay embedding-call latency, or require pre-computed embeddings for *subscription-sourced* skills too, which is `skill-catalog-subscription`'s decision to make (or not) in its own future revision, not something this change should impose on it. A standalone `retrieve_candidate_skills()` in `skills_learning/retrieval.py` keeps the new retrieval mode additive and independently callable.
|
||||
|
||||
### D6: Locally-authored skills live in `skills_learning/`'s own catalog table, tagged `source = "local-synthesis"`, composed with `skill-catalog` only in prose
|
||||
Synthesized `FlowTemplateSkill` records are persisted in a new local store owned by `skills_learning/` (same in-memory/on-disk shape as `skill-catalog-subscription`'s `Skill`/`FlowTemplateSkill` dataclasses, reused by import — not redefined — so both stores speak the same schema), with `source = "local-synthesis"` and no `subscription id`. This does not write into, nor require any schema change of, `skill-catalog-subscription`'s own store. Any future unification (one queryable namespace across "synced" and "locally-learned" skills, e.g. for a single `list_skills` MCP call to return both) is left as this change's Open Questions / a follow-on `skill-catalog-subscription` revision to make, since that store's spec today explicitly rejects non-sync writes and this change must not contradict that spec.
|
||||
**Alternative considered**: propose a `MODIFIED Requirements` delta against `skill-catalog-subscription`'s `skill-catalog` spec loosening its "reject non-sync writes" rule to allow a `source = local-synthesis` exception. Rejected per this session's explicit constraint — `openspec/specs/` has no applied baseline for `skill-catalog`, so there is nothing to diff a `MODIFIED` delta against yet, and `skill-catalog-subscription` remains its own pending, unmodified change; composing in prose (shared dataclass shapes, distinct stores) achieves the same practical reuse without touching another change's files.
|
||||
|
||||
### D7: Reuse `semantic-scene-runtime`'s LLM-integration pattern for the embedding call: narrow client, degrade-safe, never blocks
|
||||
The embedding client in `skills_learning/embeddings.py` follows the same shape `semantic-scene-runtime`'s `llm_client.py` established: a narrow, mockable client interface; `embed_skill_text(text) -> list[float] | None`, returning `None` (never raising) on timeout/rate-limit/disabled-config, exactly like `enrich_scene()`'s `SemanticScene | None` contract. Since embedding failure happens on the *post-task* synthesis path (not the live Observe→Act loop), a `None` result simply means "store the skill without an embedding, retrievable by name/id but not yet by similarity search" rather than blocking synthesis or versioning — `retrieve_candidate_skills()` skips any skill record with no stored vector.
|
||||
**Alternative considered**: make embedding a hard requirement of synthesis (skill authoring fails/rolls back entirely if embedding fails). Rejected — this would let a transient embedding-provider outage silently prevent skill learning even though the flow-template synthesis and versioning (the actually load-bearing artifact) succeeded independently; degrading to "skill stored, not yet semantically searchable" is strictly better than "skill not learned at all," and matches the project's established degrade-path precedent from `semantic-scene-runtime`.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- **[Risk]** Diffing argument values across two executions to infer parameters (D3) can misfire if the *first* two executions of a goal-family happen to share a value that later varies (e.g. always searching "coffee" twice, never generalizing to `{search_query}` until a third, different execution appears) → **Mitigation**: this is an inherent, accepted limitation of any 2-sample diff approach — document it in Open Questions as "parameterization confidence improves with more samples, not guaranteed correct after exactly one divergence"; the skill is still usable (with the shared literal baked in) until a genuinely divergent execution arrives to trigger parameterization, so nothing breaks, it just under-parameterizes early.
|
||||
- **[Risk]** Synthesizing a skill from a single successful task risks encoding an accidental/fragile path (e.g. steps that happened to work once but aren't a generally reliable flow) as if it were a validated reusable skill → **Mitigation**: out of scope to solve with a confidence/reliability score in this change (no repeated-execution signal exists yet for a single-run skill); `skills_learning/config.py` can gate authoring behind a "minimum N successful executions of this goal-family before first synthesis" threshold as a future config addition, but a real Planner retrieving and *trying* a skill (future work) is still expected to verify against the live `Scene`/`SemanticScene` before blindly trusting a one-shot-derived flow, matching `world-model-runtime`'s "advisory, not source-of-truth" stance on `WorldState`.
|
||||
- **[Risk]** The tool-name-sequence-skeleton matching in D3/D4 (used to decide "is this the same skill as before") has no formal goal-similarity threshold defined in this design — two semantically different goals that happen to execute the identical tool-name skeleton (e.g. two different searches) could be treated as "the same skill family" prematurely → **Mitigation**: skeleton-matching is scoped as a heuristic starting point, documented as an Open Question; if this proves too coarse in practice, a future revision can require skeleton match **and** a minimum goal-embedding similarity (reusing D5's embedding index) before treating two runs as the same skill family, rather than skeleton alone.
|
||||
- **[Trade-off]** Keeping locally-authored skills in a separate store from `skill-catalog-subscription`'s synced catalog (D6) means any consumer wanting "all skills, synced and learned" must query two stores and merge, not one → **Acceptable** because unifying them requires a `skill-catalog-subscription` spec change this session is explicitly scoped not to make; the shared dataclass shape (same `Skill`/`FlowTemplateSkill` types) keeps the merge trivial for whichever future change decides to do it (likely `skill-mcp-tools`' `list_skills` gaining a second source).
|
||||
- **[Trade-off]** Defaulting Skill Authoring to disabled (unlike `world-model-runtime`'s default-enabled precedent) means most existing/near-term task runs will not accumulate learned skills until explicitly turned on → **Acceptable**, matching `semantic-scene-runtime`'s reasoning: synthesis + embedding is a real CPU/LLM-adjacent cost on the success path, and applying this change must not silently change existing task cost/latency profiles for callers who haven't opted in.
|
||||
- **[Trade-off]** No skill-execution engine is built here (Non-Goal) — a synthesized `FlowTemplateSkill` is inert data until some future runner resolves its parameters and drives `tools/` — this is intentional scope discipline (per the proposal's Impact section) but means this milestone's value is not observable end-to-end (goal → learned skill → skill actually reused) until that follow-on work exists; acceptable since `skill-catalog-subscription`'s own proposal already scoped a "run_skill_flow parameter-resolution helper" as a distinct concern this change should not duplicate.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
This is purely additive, matching the "safe default, optional hook" shape of `semantic-scene-runtime` and `world-model-runtime`:
|
||||
1. Add the `skills_learning/` package: `synthesis.py`, `versioning.py`, `embeddings.py`, `retrieval.py`, `config.py` (enable flag defaulting to **disabled**, divergence tolerance, embedding model name, default `top_k`).
|
||||
2. Reuse (import, do not redefine) `Skill`/`FlowTemplateSkill`/`SkillMetadata` dataclass shapes from `skill-catalog-subscription`'s planned `skills/models.py` once that change is implemented; if implemented first, `skills_learning/` depends on it for shared types only, never for its sync/storage code paths. If `skills_learning/` is implemented before `skill-catalog-subscription`, define the minimal shared dataclass shape locally and note it must be reconciled (not duplicated) once that change lands.
|
||||
3. Add a new local store for synthesized skills (`skills_learning/`-owned catalog table, `source = "local-synthesis"`) plus the `skill_embeddings` index table.
|
||||
4. Add the optional `on_task_succeeded: Callable[[str, str, Timeline], None] | None = None` constructor argument to `TaskRunner` (`runtime/task.py`); when Skill Authoring is enabled in config and no explicit hook is passed, `TaskRunner` wires a default one calling `synthesis.synthesize_flow_skill`; call it once, only after `status = succeeded` is determined, at the very end of `run()`.
|
||||
5. Add the embedding-provider SDK dependency and `skills_learning*` to `pyproject.toml`'s `[tool.setuptools.packages.find].include` list; add embedding config with the enable flag defaulting to disabled so applying this change never silently changes existing task cost/latency (matching `semantic-scene-runtime`'s D-config precedent).
|
||||
6. No changes to `storage/timeline.py`'s persisted format, `skill-catalog-subscription`'s schema/spec, or any existing tool signature.
|
||||
7. Run `pytest` — full existing suite green with zero test-content edits; new tests (`tests/test_skill_synthesis.py`-style) exercise synthesis/versioning/retrieval logic against canned `Timeline` fixtures and a mocked embedding client, not live model calls.
|
||||
8. Rollback: net-new package plus one additive, default-`None` constructor argument with a config-gated default wiring — reverting is `git revert` of the commit(s); no data migration of existing stores, no external system beyond the embedding provider (itself optional/degrade-safe per D7).
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Whether the "same skill family" match (D3/D4's tool-name-skeleton matching) needs a goal-embedding-similarity floor in addition to skeleton equality, to avoid conflating two structurally-identical-but-semantically-different goals — left as a heuristic-first approach for this milestone, revisit once real synthesis data exists.
|
||||
- Whether a minimum-successful-executions threshold should gate first-time synthesis (to avoid learning from a single lucky run) — left as a future `skills_learning/config.py` addition once there's real usage data to tune it against, per the Risks section.
|
||||
- Whether/how a future `skill-catalog-subscription` revision should unify the synced and locally-authored stores behind one `list_skills`/`search_skills` surface (D6's deferred merge) — explicitly left to that change, not decided here.
|
||||
- Which embedding provider/model to default to, and whether it should be the same provider as `semantic-scene-runtime`'s enrichment client (operational simplicity: one API key/provider to manage) or an independent choice optimized purely for embedding quality/cost — left as an implementation-time decision, not fixed in this design.
|
||||
- Whether `WorldState.history` (from `world-model-runtime`, already flagged in that change's own Open Questions as a possible future Skill-synthesis consumer) should feed into synthesis here — e.g. using recent world-state transitions to help segment "which steps belong to this skill" versus incidental navigation — left open; this change synthesizes purely from `Timeline`'s tool-call sequence and does not yet consume `WorldState`.
|
||||
@@ -0,0 +1,30 @@
|
||||
## Why
|
||||
|
||||
`skill-catalog-subscription` (already proposed, unapplied) lets the agent *consume* skills that an external Subscription Platform authored, versioned, and pushed down — but it has no mechanism for the agent to *learn* a skill from its own experience. Every task the agent completes successfully today (once `apex-agent-mvp`'s `agent-runtime`/`task-memory` capabilities are applied) leaves behind a fully-recorded `Timeline` of scenes/prompts/tool-calls/results, and that recording is simply discarded once the task ends — there is no path from "the agent just did X successfully" to "the agent can do X again faster, or hand X to another task as a reusable flow." This change closes that gap: **Skill Runtime (learning half)** synthesizes a reusable, parameterized flow-template skill from a completed task's executed step sequence, detects when a later execution of "the same" skill diverges enough to warrant a new version, and makes locally-learned skills discoverable by semantic similarity to a new goal — so the Planner (present stub, future LLM-driven) has a growing, self-improving library of known-good flows to try before planning from scratch every time.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Add a **Skill Authoring** capability: after a task completes with `status = succeeded`, a synthesis pass reads that task's `Timeline` (from `task-memory`'s `storage/timeline.py`) plus the goal string that produced it, extracts the ordered sequence of executed tool calls (`tap`/`swipe`/`input_text`/`launch_app`/...), and produces a `FlowTemplateSkill` (matching `skill-catalog-subscription`'s `skill-catalog` shape: an ordered `steps` list of tool name + args-template, and a `parameters` schema) with literal argument values that vary across similar goals abstracted into named `{param}` placeholders.
|
||||
- Add a **Skill Versioning** capability: when synthesis produces a step sequence for a skill that already exists in local storage (matched by name/goal-family, not by id), diff the newly executed steps against the currently-stored version's steps; if they differ beyond a configured tolerance (extra/missing/reordered steps, or a materially different parameter set), store a new version and retain version history rather than overwriting silently.
|
||||
- Add a **Skill Embedding Retrieval** capability: embed each locally-authored skill's `name` + `description` + originating goal text into a vector, persist those vectors alongside the skill record, and expose a `retrieve_candidate_skills(goal, top_k)` function returning ranked candidates by cosine similarity to a new incoming goal, so a Planner can check "has something like this already been learned" before planning from scratch.
|
||||
- Extend `TaskRunner`'s completion path with an optional **post-task synthesis hook** (`on_task_succeeded(task_id, goal, timeline)`), called once, only on success, only when Skill Authoring is enabled in config (default **disabled**, since synthesis + embedding is extra CPU/LLM cost on the success path and should not silently change task latency for existing callers/tests) — mirroring `semantic-scene-runtime`'s default-off precedent for cost-bearing additions, not `world-model-runtime`'s default-on precedent (which is pure in-memory derivation, not an LLM/embedding call).
|
||||
- Compose with, but do not modify, `skill-catalog-subscription`'s `skill-catalog` storage model: locally-authored skills are written as `FlowTemplateSkill` records tagged with a `source = "local-synthesis"` discriminator (vs. `source = "<subscription-id>"` for externally-synced skills) so both kinds can be listed/searched through the same catalog surface without the sync client ever attempting to push a locally-authored skill upstream or the synthesis pass ever overwriting a subscription-sourced skill.
|
||||
- **BREAKING**: none. `TaskRunner`'s new completion hook defaults to a no-op when Skill Authoring is disabled; nothing existing changes shape or behavior.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `skill-authoring`: Synthesizes a parameterized `FlowTemplateSkill` from a completed task's `Timeline` (executed tool-call sequence) and the goal that produced it, abstracting goal-specific literals into named parameters.
|
||||
- `skill-versioning`: Detects divergence between a newly synthesized flow and the currently-stored version of "the same" skill, and manages version bump/history (never silent overwrite) when the executed sequence has materially changed.
|
||||
- `skill-embedding-retrieval`: Embeds locally-authored (and, read-only, subscription-sourced) skill descriptions/goals and retrieves a ranked list of candidate skills by semantic similarity to a new incoming goal, for a Planner to consider before planning from scratch.
|
||||
|
||||
### Modified Capabilities
|
||||
(none — `task-memory` (`apex-agent-mvp`) and `skill-catalog`/`skill-mcp-tools`/`skill-subscription-sync` (`skill-catalog-subscription`) are composed with in prose only; neither has an applied baseline in `openspec/specs/` to diff against, and this change does not alter either's specified behavior. `task-memory`'s `Timeline` is read as an input; `skill-catalog`'s storage shape is reused as the target format for synthesized skills, tagged with a new `source` value it already accommodates as a free-form field.)
|
||||
|
||||
## Impact
|
||||
|
||||
- **New package**: `skills_learning/` — `synthesis.py` (`synthesize_flow_skill(goal, timeline) -> FlowTemplateSkill`, tool-call-sequence extraction + parameter abstraction), `versioning.py` (`diff_flow_versions()`, `VersionStore` — bump/history logic), `embeddings.py` (`embed_skill_text()`, `EmbeddingIndex` — vector storage + cosine-similarity ranking), `retrieval.py` (`retrieve_candidate_skills(goal, top_k)`), `config.py` (enable flag, divergence tolerance, embedding model name, `top_k` default).
|
||||
- **Modified**: `runtime/task.py` (`TaskRunner` gains an optional `on_task_succeeded` hook invoked once at the end of a successful `run()`, default `None`/no-op); no change to `runtime/context.py`, `runtime/planner.py`, `runtime/executor.py` beyond the hook wiring — this change does not itself teach a Planner to call `retrieve_candidate_skills()` (that is a future LLM-driven Planner's job, matching `world-model-runtime`'s precedent of exposing a read-only input without building its consumer).
|
||||
- **Storage**: reuses `skill-catalog-subscription`'s catalog store for `FlowTemplateSkill` records (adding `source`, `version`, `parent_version_id` fields it already anticipates via free-form metadata); adds a new local `skill_embeddings` table/index (skill id → vector, model name, updated_at) — a new store, not a modification of `skill-catalog`'s schema, since `skill-catalog-subscription` does not define one today.
|
||||
- **External dependency**: introduces an embedding model call (provider TBD in design.md) as the second LLM-adjacent integration point in the codebase after `semantic-scene-runtime`'s enrichment call; reuses that change's pattern (narrow, mockable client; degrade-safe; never blocks the task loop it hooks into) rather than inventing a new one.
|
||||
- **Out of scope**: no changes to `skill-catalog-subscription`'s sync contract, MCP tool surface, or subscription/visibility model; no UI (that is `web-console`'s domain); no automatic skill *execution* triggering (a Planner choosing to run a retrieved skill is future Planner work); no cross-device or cross-tenant skill sharing beyond whatever the shared `skill-catalog` store already implies.
|
||||
@@ -0,0 +1,60 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Synthesis triggers only on successful task completion
|
||||
The system SHALL synthesize a flow-template skill only when a task's final status is `succeeded`, and SHALL NOT attempt synthesis for a task that failed, was cancelled, or is still running.
|
||||
|
||||
#### Scenario: Successful task triggers synthesis
|
||||
- **WHEN** a task completes with status `succeeded` and Skill Authoring is enabled
|
||||
- **THEN** the system reads that task's timeline and goal and produces a flow-template skill candidate
|
||||
|
||||
#### Scenario: Failed task does not trigger synthesis
|
||||
- **WHEN** a task completes with status `failed` (or is cancelled/still running)
|
||||
- **THEN** the system does not synthesize any skill from that task's timeline
|
||||
|
||||
### Requirement: Skill Authoring defaults to disabled
|
||||
The system SHALL leave Skill Authoring disabled by default in configuration, so applying this capability does not change the cost or latency of any existing task run until a caller explicitly enables it.
|
||||
|
||||
#### Scenario: Default configuration performs no synthesis
|
||||
- **WHEN** a task completes successfully and Skill Authoring has not been explicitly enabled in configuration
|
||||
- **THEN** the system performs no synthesis work and the task's completion path behaves exactly as it would without this capability
|
||||
|
||||
#### Scenario: Explicit enable activates synthesis
|
||||
- **WHEN** an operator enables Skill Authoring in configuration
|
||||
- **THEN** subsequently completed successful tasks are eligible for synthesis
|
||||
|
||||
### Requirement: Flow-template skill synthesized from executed tool-call sequence
|
||||
The system SHALL derive a flow-template skill's ordered steps from the sequence of mutating tool calls (e.g. `tap`, `swipe`, `input_text`, `launch_app`) recorded in the completed task's timeline, in the order they were executed, and SHALL exclude read-only/observational tool calls (e.g. `describe_screen`, `screenshot`, `ui_tree`) from the synthesized step list.
|
||||
|
||||
#### Scenario: Mutating steps are included in order
|
||||
- **WHEN** a task's timeline contains a sequence of `launch_app`, `tap`, `input_text`, `tap` tool calls that all succeeded
|
||||
- **THEN** the synthesized skill's steps list contains those four steps in that same order
|
||||
|
||||
#### Scenario: Read-only observation calls are excluded
|
||||
- **WHEN** a task's timeline includes `describe_screen` or `screenshot` calls interleaved with mutating calls
|
||||
- **THEN** the synthesized skill's steps list omits those read-only calls and retains only the mutating steps
|
||||
|
||||
### Requirement: Parameter abstraction from cross-execution argument diffing
|
||||
The system SHALL abstract a synthesized skill's step arguments into named parameters by comparing the newly executed argument values against a previously stored skill with the same tool-name step sequence, promoting any argument value that differs between the two executions into a named placeholder, and SHALL leave a first-time synthesis (no prior matching skeleton) with zero parameters.
|
||||
|
||||
#### Scenario: First execution of a flow has no parameters
|
||||
- **WHEN** no previously stored skill shares the newly executed tool-name sequence
|
||||
- **THEN** the synthesized skill is stored with its literal argument values and an empty parameters list
|
||||
|
||||
#### Scenario: Second, divergent execution promotes a differing value to a parameter
|
||||
- **WHEN** a later successful task executes the same tool-name sequence as a stored skill but with a different literal value at one argument position
|
||||
- **THEN** the system promotes that argument position to a named parameter in the skill's `parameters` schema and replaces the literal in `steps` with a `{param}` placeholder referencing it
|
||||
|
||||
#### Scenario: Identical repeated execution does not spuriously add parameters
|
||||
- **WHEN** a later successful task executes the same tool-name sequence as a stored skill with identical argument values at every position
|
||||
- **THEN** the system does not introduce any new parameter for that skill
|
||||
|
||||
### Requirement: Locally-authored skills are stored separately from synced skills
|
||||
The system SHALL persist locally-synthesized flow-template skills tagged with a `source` of `local-synthesis`, in a store owned by this capability, and SHALL NOT write into or modify the externally-synced skill catalog store or its sync-only write contract.
|
||||
|
||||
#### Scenario: Synthesized skill is tagged as locally-authored
|
||||
- **WHEN** a flow-template skill is synthesized from a completed task
|
||||
- **THEN** its stored record has `source = "local-synthesis"` and no subscription identifier
|
||||
|
||||
#### Scenario: Synced skill catalog is untouched by synthesis
|
||||
- **WHEN** a skill is synthesized and stored by this capability
|
||||
- **THEN** no record in the externally-synced skill catalog store is created, modified, or removed as a result
|
||||
@@ -0,0 +1,41 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Skill text embedded on synthesis and re-synthesis
|
||||
The system SHALL compute and persist an embedding vector for each locally-authored skill's name, description, and originating goal text whenever that skill is first synthesized or a new version is stored, associated with that skill's id and version.
|
||||
|
||||
#### Scenario: New skill gains an embedding
|
||||
- **WHEN** a flow-template skill is synthesized for the first time
|
||||
- **THEN** the system computes an embedding vector from its name, description, and originating goal, and stores it alongside the skill record
|
||||
|
||||
#### Scenario: New version gains its own embedding
|
||||
- **WHEN** a new version of an existing skill is created
|
||||
- **THEN** the system computes and stores an embedding for that version, independent of any embedding stored for prior versions
|
||||
|
||||
### Requirement: Embedding failure degrades to non-retrievable-by-similarity, never blocks synthesis
|
||||
The system SHALL NOT allow an embedding-provider failure (timeout, rate limit, disabled configuration, connection error) to prevent a skill from being synthesized or versioned; on such failure, the skill SHALL be stored without a similarity-searchable embedding.
|
||||
|
||||
#### Scenario: Embedding call fails but skill is still stored
|
||||
- **WHEN** the embedding provider call fails or times out during synthesis of an otherwise-successful skill
|
||||
- **THEN** the skill's flow-template record is still stored, and it is retrievable by exact name/id lookup but excluded from similarity-based retrieval results until a subsequent embedding attempt succeeds
|
||||
|
||||
### Requirement: Ranked retrieval of candidate skills by goal similarity
|
||||
The system SHALL provide a function that, given a new goal string and a requested result count, returns locally-authored skills that have a stored embedding, ranked by descending semantic similarity between the goal and each skill's stored embedding.
|
||||
|
||||
#### Scenario: Similar goal returns matching skill highest-ranked
|
||||
- **WHEN** a new goal is semantically similar to a previously-learned skill's originating goal
|
||||
- **THEN** that skill appears in the ranked candidate results, ordered ahead of less-similar skills
|
||||
|
||||
#### Scenario: Requested count limits results
|
||||
- **WHEN** a caller requests the top `k` candidate skills for a goal
|
||||
- **THEN** the system returns at most `k` ranked results, even if more embedded skills exist
|
||||
|
||||
#### Scenario: No embedded skills yields an empty result
|
||||
- **WHEN** no locally-authored skill currently has a stored embedding
|
||||
- **THEN** the retrieval function returns an empty ranked list rather than raising an error
|
||||
|
||||
### Requirement: Retrieval scoped to locally-authored skills unless explicitly extended
|
||||
The system SHALL restrict ranked candidate retrieval to skills stored by this capability's own local-synthesis store by default, and SHALL treat inclusion of externally-synced skills as a separate, explicit extension rather than an implicit default.
|
||||
|
||||
#### Scenario: Default retrieval excludes synced-only skills without embeddings
|
||||
- **WHEN** the skill catalog contains externally-synced skills that have never been embedded by this capability
|
||||
- **THEN** ranked candidate retrieval returns only locally-authored skills with stored embeddings, without erroring on the presence of unembedded synced skills
|
||||
@@ -0,0 +1,38 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Structural divergence triggers a new version
|
||||
The system SHALL compare a newly synthesized flow's tool-name step sequence against the currently-stored version of the matching skill, and SHALL create a new version (rather than overwriting the stored one) whenever the tool-name sequence differs by insertion, deletion, or reordering of a step.
|
||||
|
||||
#### Scenario: Extra step triggers a new version
|
||||
- **WHEN** a newly executed flow for a matching skill contains an additional tap step not present in the currently-stored version's sequence
|
||||
- **THEN** the system stores a new version of the skill rather than overwriting the existing stored version
|
||||
|
||||
#### Scenario: Reordered steps trigger a new version
|
||||
- **WHEN** a newly executed flow for a matching skill executes the same tool names as the stored version but in a different order
|
||||
- **THEN** the system stores a new version of the skill
|
||||
|
||||
#### Scenario: Argument-value-only differences do not trigger a version bump
|
||||
- **WHEN** a newly executed flow has the identical tool-name sequence as the stored version and differs only in argument values already covered by parameter abstraction
|
||||
- **THEN** the system does not create a new version, and instead updates the existing version's parameters per the skill-authoring capability
|
||||
|
||||
### Requirement: Version history is retained, never silently overwritten
|
||||
The system SHALL retain every version of a skill it creates, each carrying an incrementing `version` number and a reference to the version it diverged from, and SHALL NOT delete or overwrite a prior version's stored record when a new version is created.
|
||||
|
||||
#### Scenario: New version references its parent
|
||||
- **WHEN** a new version of a skill is created due to structural divergence
|
||||
- **THEN** the new version's record stores a reference to the prior version's id and an incremented version number
|
||||
|
||||
#### Scenario: Prior version remains fetchable
|
||||
- **WHEN** a new version of a skill has been created
|
||||
- **THEN** the prior version's record remains retrievable by its own id, unmodified
|
||||
|
||||
### Requirement: Default retrieval surfaces the newest version
|
||||
The system SHALL treat the highest-numbered version of a skill as the default result returned by a lookup-by-name/goal-family query, while still allowing an explicit lookup of any specific prior version by its id.
|
||||
|
||||
#### Scenario: Lookup by name returns newest version
|
||||
- **WHEN** a caller looks up a skill by its name or goal-family without specifying a version
|
||||
- **THEN** the system returns the highest-numbered stored version of that skill
|
||||
|
||||
#### Scenario: Explicit id lookup returns the requested version
|
||||
- **WHEN** a caller requests a skill by a specific prior version's id
|
||||
- **THEN** the system returns that exact version's record, not the newest version
|
||||
@@ -0,0 +1,49 @@
|
||||
## 1. Package scaffolding
|
||||
|
||||
- [ ] 1.1 Create `skills_learning/` package with `__init__.py`, `models.py`, `synthesis.py`, `versioning.py`, `embeddings.py`, `retrieval.py`, `config.py`, `store.py`
|
||||
- [ ] 1.2 Add `skills_learning*` to `pyproject.toml`'s `[tool.setuptools.packages.find].include` list and add the embedding-provider SDK dependency
|
||||
- [ ] 1.3 Add `skills_learning/config.py` with `SkillAuthoringConfig` (enable flag default `False`, divergence tolerance, embedding model name, default `top_k`) and a module-level accessor mirroring `semantic-scene-runtime`'s/`world-model-runtime`'s config-module pattern
|
||||
- [ ] 1.4 Add `skills_learning/models.py` defining (or importing, if `skill-catalog-subscription` is already implemented) the shared `Skill`/`FlowTemplateSkill`/`SkillMetadata` dataclass shapes, plus this capability's own `source`, `version`, `parent_version_id` fields
|
||||
|
||||
## 2. Local skill store (skill-authoring, skill-versioning)
|
||||
|
||||
- [ ] 2.1 Implement `skills_learning/store.py`: a local store for locally-synthesized `FlowTemplateSkill` records, separate from `skill-catalog-subscription`'s synced catalog, with `create_version()`, `get_by_id()`, `get_latest_by_name()`, `list_versions(name)`
|
||||
- [ ] 2.2 Enforce `source = "local-synthesis"` tagging on every record written by this store; add a guard/test that this store never writes into or imports a write-path of `skill-catalog-subscription`'s catalog module
|
||||
- [ ] 2.3 Write unit tests for the store's version-chain semantics: creating a new version does not delete/modify prior versions, and `get_latest_by_name()` returns the highest `version`
|
||||
|
||||
## 3. Timeline extraction and parameter abstraction (skill-authoring)
|
||||
|
||||
- [ ] 3.1 Implement `skills_learning/synthesis.py`'s tool-call extraction: given a `task_id`, read `storage.timeline.Timeline.read(task_id)` and produce an ordered list of `(tool_name, args)` pairs, filtering out read-only tool names (`describe_screen`, `screenshot`, `ui_tree`, `find_text`, `find_icon`)
|
||||
- [ ] 3.2 Implement skeleton matching: given an extracted tool-name sequence, look up any stored skill (via `store.py`) whose latest version has the identical tool-name sequence
|
||||
- [ ] 3.3 Implement cross-execution argument diffing: compare extracted argument values position-by-position against a matched stored version's steps, and promote any differing value into a named `{param}` placeholder plus a corresponding entry in the skill's `parameters` schema
|
||||
- [ ] 3.4 Implement parameter naming: prefer a name derived from the corresponding `SemanticScene.widgets[].purpose` label when available (optional dependency on `semantic/`'s output, degrading gracefully when absent), else fall back to a positional name (e.g. `param_2`)
|
||||
- [ ] 3.5 Implement `synthesize_flow_skill(goal, timeline) -> FlowTemplateSkill`: orchestrates extraction → skeleton match → diffing → parameter promotion → returns a candidate skill record (not yet persisted)
|
||||
- [ ] 3.6 Write unit tests for first-time synthesis (no prior match, zero parameters), second-execution parameter promotion, and identical-repeat synthesis (no spurious new parameters), using canned `TimelineRecord` fixtures
|
||||
|
||||
## 4. Version divergence detection (skill-versioning)
|
||||
|
||||
- [ ] 4.1 Implement `skills_learning/versioning.py`'s `diff_flow_versions(stored_steps, executed_steps) -> VersionDiff`: detect tool-name-sequence insertion/deletion/reorder (structural divergence) versus argument-value-only differences
|
||||
- [ ] 4.2 Implement version-bump logic: on structural divergence, construct a new `FlowTemplateSkill` version with incremented `version` and `parent_version_id` set to the prior version's id; on argument-only divergence, update the existing version's parameters in place (no bump)
|
||||
- [ ] 4.3 Write unit tests: extra/missing/reordered step triggers a version bump; identical-sequence-different-values does not bump but does update parameters; assert prior version records remain retrievable and unmodified after a bump
|
||||
|
||||
## 5. Post-task synthesis hook wiring
|
||||
|
||||
- [ ] 5.1 Add an optional `on_task_succeeded: Callable[[str, str, Timeline], None] | None = None` constructor argument to `TaskRunner` in `runtime/task.py`, invoked exactly once at the end of `run()` when the final status is `succeeded`
|
||||
- [ ] 5.2 Wire a default hook (when `on_task_succeeded` is not explicitly passed and Skill Authoring is enabled in `skills_learning/config.py`) that calls `synthesis.synthesize_flow_skill()`, runs versioning via `versioning.py`, and persists the result via `store.py`
|
||||
- [ ] 5.3 Verify that when Skill Authoring is disabled (default) or `on_task_succeeded` is left `None` and disabled, `TaskRunner.run()`'s behavior and return value are byte-for-byte identical to before this change
|
||||
- [ ] 5.4 Write a unit test that runs a fake successful `TaskRunner` loop with Skill Authoring enabled and asserts a skill record is stored after completion, and a test that asserts no store write occurs when disabled
|
||||
|
||||
## 6. Embedding and retrieval (skill-embedding-retrieval)
|
||||
|
||||
- [ ] 6.1 Implement `skills_learning/embeddings.py`'s embedding client interface: `embed_skill_text(text) -> list[float] | None`, catching timeout/rate-limit/disabled-config/connection-error internally and returning `None` rather than raising, mirroring `semantic/llm_client.py`'s degrade-safe contract
|
||||
- [ ] 6.2 Implement a local `skill_embeddings` index (skill id + version → vector, model name, `updated_at`) in `skills_learning/store.py` or a dedicated `skills_learning/embeddings_store.py`
|
||||
- [ ] 6.3 Wire embedding computation into the post-synthesis/versioning path: call `embed_skill_text()` on `name + description + goal` for every newly stored skill version, storing the resulting vector (or leaving the skill un-embedded if the call returns `None`)
|
||||
- [ ] 6.4 Implement `skills_learning/retrieval.py`'s `retrieve_candidate_skills(goal, top_k) -> list[ScoredSkill]`: embed the incoming goal, compute cosine similarity against every stored skill embedding, and return the top `top_k` ranked results, skipping skills with no stored embedding
|
||||
- [ ] 6.5 Write unit tests: ranked ordering for a goal similar to a stored skill's originating goal (using a fake/deterministic embedding function), `top_k` truncation, empty-result case when no skill has an embedding, and a case where an embedding call returns `None` and the skill is stored but excluded from retrieval results
|
||||
|
||||
## 7. Integration tests and validation
|
||||
|
||||
- [ ] 7.1 Write an end-to-end test: run a fake successful task twice with slightly different goal text/argument values through `TaskRunner` (Skill Authoring enabled, embedding client mocked), asserting the second run produces a new skill version with a promoted parameter and its own embedding
|
||||
- [ ] 7.2 Write an end-to-end test: run a fake successful task, then call `retrieve_candidate_skills()` with a new, semantically similar goal string, asserting the synthesized skill is returned
|
||||
- [ ] 7.3 Run the full existing `pytest` suite and confirm zero existing test files require content changes (only new `tests/test_skill_*.py`-style files are added)
|
||||
- [ ] 7.4 Add a smoke test importing `skills_learning` alongside existing `tests/` smoke coverage, confirming the package has no import-time dependency on `skill-catalog-subscription`'s sync client (only, optionally, its shared model shapes if already implemented)
|
||||
Reference in New Issue
Block a user