Android

Deputy

A phone agent that uses my apps for me

React NativeKotlinAccessibility APILLM tool callingMedia3

View source

Why I built it

Most phone chores are a few taps across one or two apps, done again and again: telling someone I'm running late, reordering lunch, clearing old screenshots, checking a balance. I wanted to say the task and have the phone do it.

No assistant did that end to end. Google's assistant reaches only apps that publish actions for it, and the open-source phone agents that can operate any app run their brain on a computer connected over a cable. Deputy runs the whole agent on the phone.

What it does

How it works

Android's accessibility service gives Deputy the screen as a tree of labelled elements and lets it act on them, with no root needed. The agent loop runs in native Kotlin, because React Native's timers stop when the app isn't in the foreground, which is exactly when Deputy is working. React Native only draws the screens.

It talks to any OpenAI-compatible model endpoint, and the default is a small, fast Gemini model. At each step the model sees the current screen and picks one action, and a policy layer in Kotlin decides which actions need my approval before they run.

Why build a phone agent when Google already has Gemini?

Gemini can act only inside apps that publish actions for it, and most of the apps I use every day don't. Deputy works through the accessibility layer, so it can use any app the way I would, and the safety rules are mine: it never sends, pays or deletes without asking.

The other agents that can drive any app expect a computer attached to the phone. That's fine on a desk and no use on the sofa or on the move.

Where it stands

Android only, on my own phone, installed from my private Store. It will never be on Play. There's no iOS version, because iOS offers apps nothing like Android's accessibility control.