- Book: LLM Observability Pocket Guide
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
Your coding agent finishes a task. You open the pull request and read 40 changed lines. You leave a comment on line 12, ask for a fix, and approve the second round. If it breaks next week, you revert one commit. Now picture the same agent finishing a task by clicking through a settings window for six minutes. What do you open to review it?
On October 23, Runtime Wire reported that Cursor's coding agents can now click and type in desktop applications, including Apple's iOS Simulator, and that the agent can pick the tool on its own when a task needs it. Cursor's own docs describe computer use for self-hosted agents. OpenAI documents computer use for Codex and ChatGPT's desktop app on macOS and Windows, through a Computer Use plugin. Anthropic ships a computer use tool in its API with members such as screenshot, left_click and type.
A coding agent's work is reviewable and reversible for one reason: it lands as file changes that git can diff, bisect and revert. A click lands nowhere. Let GUI-driving agents look at the screen and report what they saw. Anything that changes state goes through files on a branch.
What computer use is actually for
The vendors pitch it sensibly. Cursor's headline example is an agent driving the iOS Simulator to test the feature it just built (Runtime Wire). OpenAI's docs list testing desktop or simulator flows, reproducing GUI-only bugs, and "changing app settings that require clicking through a UI" (OpenAI docs).
Two of those three only read the screen. The third writes state. That split is the whole argument.
Testing a flow is a check. The agent taps through signup, sees the button render off-screen, and goes back to the code to fix it. The fix is a file change. The click-through was evidence, and you can throw it away after reading it.
Changing a setting through a UI is a mutation. The result lives in some app's preferences, a cloud console, or a database row behind an admin panel. Unless that app writes it to a file in your repo, git never hears about it.
OpenAI's own docs are honest about this. Computer use actions can affect "app and system state outside your project workspace," so the docs recommend scoped tasks. They also note that desktop app changes "may not appear in the review pane until they're saved to disk and tracked by the project" (OpenAI docs). The review pane only sees what reaches disk.
A diff stays on the page after the work is done. A click doesn't.
Git is the review system you already trust
Look at what you get for free when an agent's work is a set of commits on a branch.
You get a diff. Every changed line is visible, in order, next to the line it replaced. On GitHub you can "hover over the line of code where you'd like to add a comment" and leave feedback on exactly that line (GitHub Docs). That comment is something the agent can act on.
You get undo per step. git revert takes an existing commit and records a new one that reverses it. If the agent made five commits and the third was wrong, you revert the third and keep the rest.
You get search through history. git bisect uses binary search to find the commit that introduced a bug, and git bisect run automates it with any script that exits 0 for good and 1 to 127 (except 125) for bad.
On an agent branch the whole loop is a handful of commands:
git switch -c agent/signup-fix # the agent works here, one commit per step git log --oneline main..agent/signup-fix git diff main...agent/signup-fix # step three was wrong: undo just that one git revert <sha-of-step-three> # something broke and you don't know where git bisect start agent/signup-fix main git bisect run npm test git bisect reset Try each of those on a GUI session. Clicks have no log you can comment on line by line, a toggled setting in a vendor console has no revert, and six minutes of mouse movement can't be bisected. You might get screenshots in the chat. Cursor's self-hosted docs describe screenshots as the way you confirm the setup works (Cursor docs). A screenshot shows the end state, not the steps that produced it.
The vendors already tell you to fence it
None of this is a secret the vendors are hiding. Their docs say it in plain words.
Anthropic recommends "a dedicated virtual machine or container with minimal privileges," keeping the model away from login details, and "asking a human to confirm decisions that might result in meaningful real-world consequences" (Anthropic docs). It also warns that instructions on webpages or inside images can override yours.
OpenAI says ChatGPT asks before it uses an app, "may also ask for permission before taking sensitive or disruptive actions," and that file edits and shell commands "still follow the sandbox and approval settings" (OpenAI docs). The docs also suggest a dedicated plugin or MCP server when one exists.
Cursor's self-hosted docs say that on macOS the agent "clicks, types, and captures the screen of the desktop session the worker runs in" (Cursor docs). If the worker runs on your own machine, that is your live session, with your apps and your logins.
Approval prompts are the GUI's answer to review. They help. But an approval happens once, before the action, with no reviewable artifact afterwards. A diff happens after the action, stays around, and can be read by someone who wasn't there.
A rule you can paste into your agent instructions
Split the work by what it touches.
Screen-only tasks are fine. Tap through the flow. Reproduce the bug. Check that the dark-mode toggle renders. Ask the agent to write what it saw to a file in the repo, say qa/signup-check.md with the steps and screenshots, so the evidence lands in the diff too.
State-changing tasks go through files on a branch. If the setting can live in a config file, an infrastructure template or a migration, make the agent change the file and let your normal deploy apply it. If the only path is a vendor's admin UI, a human does that click, after reading the PR that explains why.
When in doubt, ask "can I revert this tomorrow?" If the answer is no, it does not belong in an unattended GUI session.
This costs you a little speed on the rare task that really is GUI-only. It buys you the same review loop for every other change your agents make, and that is most of them.
Where Hermes fits
I build Hermes IDE, so weigh this section with that in mind. Hermes is built around the same bet as this post: what an agent did should show up as something git can see.
Hermes observes what agents report on purpose and what git shows. It never answers a permission prompt for you and never types into a terminal on its own (README). Each task can get its own branch and worktree. Every agent turn is saved, so you can diff or restore any single turn, with undo. The Review Desk (⌘G) shows changes by turn or by file, and sends a line comment back to the agent that wrote it. Land commits, opens a pull request or merges locally in one step, into the branch the task started from, and it can be undone.
The Review Desk in By turn view: two turns of a Claude Code task in orders-api, turn 1's diff of src/db.ts, and a commit message drafted from the turns.
It runs several CLI coding agents side by side. It is free to download, source-available under BSL 1.1, on macOS, Windows and Linux, and it is not affiliated with Anthropic, OpenAI or any agent vendor. It can't review a click either. That is the point: keep the agent's changes in files, and the tools you already have will do the rest.
Would you let a coding agent change state through a GUI with no diff behind it? Or does every change in your team go through a branch first? Tell me in the comments.
If this was useful
The question under this whole post is "how do I know what the agent did?" The LLM Observability Pocket Guide answers it for the rest of your LLM stack: traces, evals, cost and the tools that record them, so an agent's run leaves evidence you can read later. It's one of eight short books in Pocket Guides for Developers, covering agents, RAG, prompts, observability, events, databases and system design.
If you run several coding agents on one repo, Hermes IDE keeps each one's work on its own branch, turn by turn (GitHub).
