# Agent Interface Experiments

> Agent-facing counterpart to the [human project page](/projects/agent-interface-experiments/).

## Record metadata

- Record: 030
- Slug: agent-interface-experiments
- Domain: Software
- Domain code: SW
- Type: Interface lab
- Status: Active
- Period: 2026
- Portfolio role: Experiment family
- Publication state: Public methodology and safe demos
- Case-study readiness: Needs consolidated case study
- Compendium edition: 0.4

## Summary

A family of experiments that let AI agents see, touch, navigate, and manipulate interfaces beyond a text-only shell.

## Overview

Many agent tasks end at the boundary of a screen or device that offers no useful API. The Agent Interface Experiments are a collection of attempts to cross that boundary without pretending that general computer control is automatically safe or reliable.

A collection of touch, window, camera, e-reader, and device-control experiments that give agents bounded ways to inspect and operate otherwise inaccessible interfaces.

Purpose: Explore practical bridges between agent reasoning and the physical or graphical surfaces where real work happens.

## The problem behind the project

Many useful tasks end at a screen, phone, book reader, or device that lacks a clean API. Interface bridges test how agents can cross that boundary without pretending it is risk-free.

The experiments include touch, window navigation, Kindle and camera workflows, and device-specific control. What unifies them is not one product interface, but a recurring question: what is the smallest bridge that lets an agent complete a blocked real-world task?

Developers, accessibility researchers, automation users, and agent designers may benefit. Device owners and anyone visible on a screen or camera are affected.

## How it took shape

Small utilities and workflows spanning screen capture, controlled input, window navigation, Kindle experiments, camera streaming, and device-specific adapters.

Each utility exposes a bounded surface—capture a view, send a controlled input, stream a phone camera, or wrap a device interaction—so the agent can use it through an explicit capability rather than improvising unrestricted control.

Josiah identified blocked real-world workflows, designed the bridges and safety boundaries, and repeatedly directed agents to turn them into reusable capabilities.

Several individual utilities have worked for specific tasks, though the family has not been packaged as one product.

## What the project means now

The family suggests that the future of agent interfaces may be many narrow adapters rather than one omnipotent cursor. Scope, user visibility, consent, reversible actions, and inspection matter more as the agent moves closer to physical consequences.

Computer control is brittle and potentially destructive. Consent, scope, visual privacy, and reversible actions are essential.

The most useful agent interface is usually a narrow, inspectable bridge built for one blocked workflow.

Consolidate the experiments into a taxonomy of interface gaps, safe control patterns, and demonstrated utilities.

## Publication and interpretation notes

- Current classification: Active
- Portfolio readiness: Needs consolidated case study
- Publication boundary: Public methodology and safe demos

## Additional agent context

This is the umbrella record; individual utilities such as iPhone Camera Bridge may also have their own record. Avoid disclosing captured private screens or environments.

## Related project records

- [iPhone Camera Bridge](/projects/iphone-camera-bridge/llm/) — A local WebRTC bridge that lets a Mac-based agent use an iPhone as a user-controlled camera.
- [AgentWorkbench](/projects/agent-workbench/llm/) — A portable capability layer that gives different coding agents the same documented, repeatable tools.

## Navigation

- [Complete project index](/projects/llm/)
- [Human version of this record](/projects/agent-interface-experiments/)
- [About Josiah's working method](/about/llm/)
- [Agent discovery map](/llms.txt)