Many agent tasks end at the boundary of a screen or device that offers no useful API. The Agent Interface Experiments are a collection of attempts to cross that boundary without pretending that general computer control is automatically safe or reliable.
A collection of touch, window, camera, e-reader, and device-control experiments that give agents bounded ways to inspect and operate otherwise inaccessible interfaces.
The aim. Explore practical bridges between agent reasoning and the physical or graphical surfaces where real work happens.
01The problem behind the project
Many useful tasks end at a screen, phone, book reader, or device that lacks a clean API. Interface bridges test how agents can cross that boundary without pretending it is risk-free.
The experiments include touch, window navigation, Kindle and camera workflows, and device-specific control. What unifies them is not one product interface, but a recurring question: what is the smallest bridge that lets an agent complete a blocked real-world task?
Developers, accessibility researchers, automation users, and agent designers may benefit. Device owners and anyone visible on a screen or camera are affected.
02How it took shape
Small utilities and workflows spanning screen capture, controlled input, window navigation, Kindle experiments, camera streaming, and device-specific adapters.
Each utility exposes a bounded surface—capture a view, send a controlled input, stream a phone camera, or wrap a device interaction—so the agent can use it through an explicit capability rather than improvising unrestricted control.
Josiah identified blocked real-world workflows, designed the bridges and safety boundaries, and repeatedly directed agents to turn them into reusable capabilities.
Several individual utilities have worked for specific tasks, though the family has not been packaged as one product.
03What the project means now
The family suggests that the future of agent interfaces may be many narrow adapters rather than one omnipotent cursor. Scope, user visibility, consent, reversible actions, and inspection matter more as the agent moves closer to physical consequences.
Computer control is brittle and potentially destructive. Consent, scope, visual privacy, and reversible actions are essential.
The most useful agent interface is usually a narrow, inspectable bridge built for one blocked workflow.
Consolidate the experiments into a taxonomy of interface gaps, safe control patterns, and demonstrated utilities.