perception-provider Specification

Purpose

TBD - created by archiving change device-agent-runtime-foundation. Update Purpose after archive.

Requirements

Requirement: Perception is exposed through a swappable provider port

The system SHALL define a PerceptionProvider interface (build_scene) in the perception package that any concrete perception implementation (the existing OCR+tree fusion, a future cloud vision API, a future null/mock provider) must satisfy identically, so tools/, runtime/, and api/ depend only on the port, never on a specific perception technique.

Scenario: Building a scene through the default provider

WHEN a caller requests a Scene for a screenshot and UI tree via the default (OCR+tree fusion) provider
THEN the provider returns a Scene identical in shape and content to what the existing fusion logic already produces — no change to scene-perception's specified behavior

Requirement: A null perception provider is available for testing and low-dependency development

The system SHALL provide a NullPerceptionProvider that returns an empty Scene (correct screen width/height, zero elements) without requiring OCR/vision dependencies to be installed or invoked.

Scenario: Running without OCR dependencies installed

WHEN the runtime is configured to use NullPerceptionProvider (e.g. in a test or a minimal development environment)
THEN describe_screen/Agent Runtime calls succeed and return an empty Scene instead of failing due to missing OCR dependencies

Requirement: Adding a perception technique requires no changes outside the perception layer

The system SHALL allow a new perception provider (e.g. a cloud vision API) to be added by registering it within the perception package alone; tools/, runtime/, and api/ SHALL NOT require code changes to use a newly registered provider.

Scenario: Runtime is agnostic to which provider is active

WHEN the Agent Runtime requests a Scene for the current screenshot and UI tree
THEN it does so through the PerceptionProvider port without importing or knowing about scene_builder.py, OCR, or any other concrete technique

2.2 KiB Raw Permalink Blame History