agentic-mobile-control/openspec/changes/archive/2026-07-06-skill-learning-runtime/proposal.md

Why

skill-catalog-subscription (already proposed, unapplied) lets the agent consume skills that an external Subscription Platform authored, versioned, and pushed down — but it has no mechanism for the agent to learn a skill from its own experience. Every task the agent completes successfully today (once apex-agent-mvp's agent-runtime/task-memory capabilities are applied) leaves behind a fully-recorded Timeline of scenes/prompts/tool-calls/results, and that recording is simply discarded once the task ends — there is no path from "the agent just did X successfully" to "the agent can do X again faster, or hand X to another task as a reusable flow." This change closes that gap: Skill Runtime (learning half) synthesizes a reusable, parameterized flow-template skill from a completed task's executed step sequence, detects when a later execution of "the same" skill diverges enough to warrant a new version, and makes locally-learned skills discoverable by semantic similarity to a new goal — so the Planner (present stub, future LLM-driven) has a growing, self-improving library of known-good flows to try before planning from scratch every time.

What Changes

• Add a Skill Authoring capability: after a task completes with status = succeeded, a synthesis pass reads that task's Timeline (from task-memory's storage/timeline.py) plus the goal string that produced it, extracts the ordered sequence of executed tool calls (tap/swipe/input_text/launch_app/...), and produces a FlowTemplateSkill (matching skill-catalog-subscription's skill-catalog shape: an ordered steps list of tool name + args-template, and a parameters schema) with literal argument values that vary across similar goals abstracted into named {param} placeholders.
• Add a Skill Versioning capability: when synthesis produces a step sequence for a skill that already exists in local storage (matched by name/goal-family, not by id), diff the newly executed steps against the currently-stored version's steps; if they differ beyond a configured tolerance (extra/missing/reordered steps, or a materially different parameter set), store a new version and retain version history rather than overwriting silently.
• Add a Skill Embedding Retrieval capability: embed each locally-authored skill's name + description + originating goal text into a vector, persist those vectors alongside the skill record, and expose a retrieve_candidate_skills(goal, top_k) function returning ranked candidates by cosine similarity to a new incoming goal, so a Planner can check "has something like this already been learned" before planning from scratch.
• Extend TaskRunner's completion path with an optional post-task synthesis hook (on_task_succeeded(task_id, goal, timeline)), called once, only on success, only when Skill Authoring is enabled in config (default disabled, since synthesis + embedding is extra CPU/LLM cost on the success path and should not silently change task latency for existing callers/tests) — mirroring semantic-scene-runtime's default-off precedent for cost-bearing additions, not world-model-runtime's default-on precedent (which is pure in-memory derivation, not an LLM/embedding call).
• Compose with, but do not modify, skill-catalog-subscription's skill-catalog storage model: locally-authored skills are written as FlowTemplateSkill records tagged with a source = "local-synthesis" discriminator (vs. source = "<subscription-id>" for externally-synced skills) so both kinds can be listed/searched through the same catalog surface without the sync client ever attempting to push a locally-authored skill upstream or the synthesis pass ever overwriting a subscription-sourced skill.
• BREAKING: none. TaskRunner's new completion hook defaults to a no-op when Skill Authoring is disabled; nothing existing changes shape or behavior.

Capabilities

New Capabilities

• skill-authoring: Synthesizes a parameterized FlowTemplateSkill from a completed task's Timeline (executed tool-call sequence) and the goal that produced it, abstracting goal-specific literals into named parameters.
• skill-versioning: Detects divergence between a newly synthesized flow and the currently-stored version of "the same" skill, and manages version bump/history (never silent overwrite) when the executed sequence has materially changed.
• skill-embedding-retrieval: Embeds locally-authored (and, read-only, subscription-sourced) skill descriptions/goals and retrieves a ranked list of candidate skills by semantic similarity to a new incoming goal, for a Planner to consider before planning from scratch.

Modified Capabilities

(none — task-memory (apex-agent-mvp) and skill-catalog/skill-mcp-tools/skill-subscription-sync (skill-catalog-subscription) are composed with in prose only; neither has an applied baseline in openspec/specs/ to diff against, and this change does not alter either's specified behavior. task-memory's Timeline is read as an input; skill-catalog's storage shape is reused as the target format for synthesized skills, tagged with a new source value it already accommodates as a free-form field.)

Impact

• New package: skills_learning/ — synthesis.py (synthesize_flow_skill(goal, timeline) -> FlowTemplateSkill, tool-call-sequence extraction + parameter abstraction), versioning.py (diff_flow_versions(), VersionStore — bump/history logic), embeddings.py (embed_skill_text(), EmbeddingIndex — vector storage + cosine-similarity ranking), retrieval.py (retrieve_candidate_skills(goal, top_k)), config.py (enable flag, divergence tolerance, embedding model name, top_k default).
• Modified: runtime/task.py (TaskRunner gains an optional on_task_succeeded hook invoked once at the end of a successful run(), default None/no-op); no change to runtime/context.py, runtime/planner.py, runtime/executor.py beyond the hook wiring — this change does not itself teach a Planner to call retrieve_candidate_skills() (that is a future LLM-driven Planner's job, matching world-model-runtime's precedent of exposing a read-only input without building its consumer).
• Storage: reuses skill-catalog-subscription's catalog store for FlowTemplateSkill records (adding source, version, parent_version_id fields it already anticipates via free-form metadata); adds a new local skill_embeddings table/index (skill id → vector, model name, updated_at) — a new store, not a modification of skill-catalog's schema, since skill-catalog-subscription does not define one today.
• External dependency: introduces an embedding model call (provider TBD in design.md) as the second LLM-adjacent integration point in the codebase after semantic-scene-runtime's enrichment call; reuses that change's pattern (narrow, mockable client; degrade-safe; never blocks the task loop it hooks into) rather than inventing a new one.
• Out of scope: no changes to skill-catalog-subscription's sync contract, MCP tool surface, or subscription/visibility model; no UI (that is web-console's domain); no automatic skill execution triggering (a Planner choosing to run a retrieved skill is future Planner work); no cross-device or cross-tenant skill sharing beyond whatever the shared skill-catalog store already implies.