Understanding a screenshot is not the same as knowing where to click.
That gap is GUI grounding.
We've open-sourced Aico Grounding, a local grounding component built to translate a natural-language UI target into a location on the screen.
Screenshot + instruction → predicted UI target.
[attach annotated screenshot]
For a real-device agent, this is one of the critical steps between:
seeing the interface
→ understanding the task
→ taking the right action
The grounding layer runs locally. It identifies interactive elements across the screen, assigns each one a reference, and returns the predicted target location for the agent to act on.
The screenshot above shows this running on a real mobile interface — GitHub mobile with the UI in Chinese. The grounding model identifies elements correctly regardless of UI language.
Why local grounding matters
Cloud-routed grounding adds latency on every action. For a physical device agent that may take dozens of actions in a single task, that adds up fast.
Running grounding locally on the Aico hardware keeps the action loop fast and keeps screen content off any external server.
What's next
More benchmarks and demos coming — including instruction → detected target → actual click end-to-end.


