Android
Deputy
A phone agent that uses my apps for me
Why I built it
Most phone chores are a few taps across one or two apps, done again and again: telling someone I'm running late, reordering lunch, clearing old screenshots, checking a balance. I wanted to say the task and have the phone do it.
No assistant did that end to end. Google's assistant reaches only apps that publish actions for it, and the open-source phone agents that can operate any app run their brain on a computer connected over a cable. Deputy runs the whole agent on the phone.
What it does
- Takes a task typed or spoken, then opens apps, reads the screen, and taps, types and scrolls until the task is done.
- Asks before any step that can't be undone: sending, paying, buying, deleting, posting or changing security settings.
- Treats financial and investment apps as read-only until I allow more.
- Asks before typing in a browser or tapping anything that looks like a link, so text on a web page or in a message can't steer it.
- Turns photos on the phone into a video itself, without driving an editing app.
- Hold the mic and it runs the task as soon as I stop talking. Tap it to dictate and check first.
How it works
Android's accessibility service gives Deputy the screen as a tree of labelled elements and lets it act on them, with no root needed. The agent loop runs in native Kotlin, because React Native's timers stop when the app isn't in the foreground, which is exactly when Deputy is working. React Native only draws the screens.
It talks to any OpenAI-compatible model endpoint, and the default is a small, fast Gemini model. At each step the model sees the current screen and picks one action, and a policy layer in Kotlin decides which actions need my approval before they run.
Why build a phone agent when Google already has Gemini?
Gemini can act only inside apps that publish actions for it, and most of the apps I use every day don't. Deputy works through the accessibility layer, so it can use any app the way I would, and the safety rules are mine: it never sends, pays or deletes without asking.
The other agents that can drive any app expect a computer attached to the phone. That's fine on a desk and no use on the sofa or on the move.
Where it stands
Android only, on my own phone, installed from my private Store. It will never be on Play. There's no iOS version, because iOS offers apps nothing like Android's accessibility control.