Introduction

Desktop apps are the hardest apps to test from the outside. A web page has a DOM and a dozen good test tools. A desktop app has a window.

So desktop UI tests usually go one of two ways. A script clicks at screen coordinates and compares pictures. Or you learn a different test framework for every operating system.

A person does neither. A person reads the window, finds the button that says "Unlock", and clicks it.

tamesu (試す, Japanese for to try, to test) is an open-source CLI written in Go that tests software that way. Version 1.22 adds a desktop target for macOS, Windows and Linux. You write a step in plain language. An agent reads the app through the accessibility tree, the same tree a screen reader uses, and acts on the elements it finds there. There are no screenshots to interpret and no coordinates.

This is the fourth article about tamesu. The first three cover web apps, AI agents and A2A agents. This one covers desktop apps.

tamesu drives agent-desktop, which reads your app through the accessibility API of the operating system

Why Desktop UI Tests Break

Coordinate and picture scripts tamesu
A step A click at x, y, or a match on a picture A sentence that says what the user wants
What the test reads Pixels The accessibility tree, as text
A renamed or moved button The test fails The agent finds the element by its role and name
The mouse pointer Moves across the screen Stays where it is: a click is an accessibility press
Cost of AI steps Every run pays for every step Plans are cached, so a stable window replays with no model calls for its goals
Passwords In the script Typed at the last moment, never sent to the model
macOS, Windows, Linux One tool per system One YAML format for the three

A tamesu test mixes three kinds of steps:

  • goal: a sentence. The agent acts until it is met.
  • expect: a sentence. The model says if the window satisfies it.
  • action: an exact, deterministic step, such as a click on a named button or a check on a text. No model is involved.

Use a goal where a person would say what they want. Use an action where the test must be exact.

How tamesu Reaches a Desktop App

tamesu does not talk to the app itself. It runs agent-desktop, a CLI from Vercel Labs that speaks the accessibility API of each system:

  • the AX API on macOS,
  • UI Automation on Windows,
  • AT-SPI on Linux.

tamesu pins one version of that CLI (1.2.0 in tamesu 1.22.0). On first use it downloads the binary into ~/.cache/tamesu/agent-desktop. It checks the file against a SHA-256 that ships inside tamesu before it runs it. To use your own binary, set TAMESU_AGENT_DESKTOP.

A goal step runs in a loop of three stages:

  1. Observe. The window becomes one line per element. A line with a number is something the agent can act on. A line without a number is text it can read.
  2. Plan. The agent gets the goal and those lines. It calls one tool: click, type, select, press, wait or done.
  3. Act. The tool runs on the app. The agent gets the new window and plans again.

The loop ends when the agent calls done. It also ends after max_goal_steps actions (15 by default), and then the step fails.

This is what the agent reads when the demo app of this guide starts:

 window "Mochi Stock" text "Mochi Stock" text "Enter the staff PIN to open the stock room" [1] textfield "PIN" (focused, secure) [2] button "Unlock" [3] button "Close" [4] button "Full Screen" [5] button "Minimize" text "Mochi Stock" 

That is the whole window: nine short lines of text. A small, fast model can drive a desktop app from that.

What You Will Build

You will test Mochi Stock, a small native Mac app for the stock room of Mochi Shoes, the demo shop of the earlier articles. It has a PIN screen, a product list with a search box and a form to receive stock. A command in the menu bar exports a CSV file.

The suite has four tests:

  1. Unlock the app and receive stock, written as goals.
  2. A wrong PIN keeps the app locked.
  3. The search box narrows the list.
  4. The export command in the menu bar shows a confirmation.

Then you will change the app twice. First you rename a button, and the test still passes. Then you add a real bug, and the test fails with the reason.

What You Need

  • A Mac with macOS 13 or newer, for the demo app of this guide. The desktop target also runs on Windows 10 (1809) or newer and on Linux (Ubuntu 24.04 or newer).
  • tamesu 1.22.0 or newer.
  • The Xcode Command Line Tools (xcode-select --install), only to build the demo app.
  • A Gemini API key in GEMINI_API_KEY. tamesu calls models through Genkit, so OpenAI, Anthropic and local models work too.

Step 1: Install tamesu and Give It Access

Install the CLI

brew install xavidop/tap/tamesu # macOS go install github.com/xavidop/tamesu/cmd/tamesu@latest # any platform 

Linux packages and a Scoop bucket for Windows are on the install page.

Give the terminal access

A desktop test needs a desktop: the session of a signed-in user, where the app can open a window. Each system asks for something more:

System What it takes
macOS The Accessibility permission for the app that runs tamesu (your terminal, your editor or your CI agent). It is in System Settings, Privacy & Security. Add Screen Recording to get screenshots.
Windows The interactive session of the signed-in user: not a service, not a locked screen.
Linux A graphical session with a window manager and the accessibility bus (at-spi2-core), on X11 or Wayland.

On macOS, grant the two permissions now. You will check them with tamesu doctor in Step 3.

Step 2: Build the App Under Test

Mochi Stock is one Swift file. Create a folder named mochi-stock and save this as MochiStock.swift:

import SwiftUI struct Product: Identifiable { let id: Int let name: String var units: Int } final class Store: ObservableObject { @Published var unlocked = false @Published var wrongPIN = false @Published var exported = false @Published var pin = "" @Published var query = "" @Published var selected = 0 @Published var units = "" @Published var products = [ Product(id: 0, name: "Trail runner", units: 12), Product(id: 1, name: "City sneaker", units: 30), Product(id: 2, name: "Canvas slip-on", units: 3), Product(id: 3, name: "Leather boot", units: 8), ] var total: Int { products.reduce(0) { $0 + $1.units } } func unlock() { unlocked = pin == "2468" wrongPIN = !unlocked } func receive() { products[selected].units += Int(units) ?? 0 units = "" } } @main struct MochiStockApp: App { @StateObject private var store = Store() var body: some Scene { WindowGroup("Mochi Stock") { Group { if store.unlocked { StockView() } else { LockView() } } .environmentObject(store) .frame(width: 560, height: 440) } .commands { CommandMenu("Stock") { Button("Export CSV") { store.exported = true } .disabled(!store.unlocked) } } } } struct LockView: View { @EnvironmentObject var store: Store var body: some View { VStack(spacing: 12) { Text("Mochi Stock").font(.largeTitle) Text("Enter the staff PIN to open the stock room") SecureField("PIN", text: $store.pin).frame(width: 160) Button("Unlock") { store.unlock() } if store.wrongPIN { Text("Wrong PIN. Try again.").foregroundColor(.red) } } .padding() } } struct StockView: View { @EnvironmentObject var store: Store var shown: [Product] { store.query.isEmpty ? store.products : store.products.filter { $0.name.localizedCaseInsensitiveContains(store.query) } } var body: some View { VStack(alignment: .leading, spacing: 12) { TextField("Search products", text: $store.query) List(shown) { product in HStack { Text(product.name) Spacer() if product.units < 5 { Text("Low stock").foregroundColor(.orange) } Text("\(product.units) in stock") } } HStack { Picker("Product", selection: $store.selected) { ForEach(store.products) { Text($0.name).tag($0.id) } } TextField("Units", text: $store.units).frame(width: 80) Button("Receive") { store.receive() } } Text("Total: \(store.total) units") } .padding() .alert("Exported 4 products to stock.csv", isPresented: $store.exported) { Button("OK") {} } } } 

Three details matter for the tests:

  • The PIN is 2468, and the PIN field is a secure field. It never shows its value.
  • "Canvas slip-on" starts with 3 units, so it shows a Low stock label. The label goes away at 5 units or more.
  • All the state lives in memory. Every launch starts locked, with 53 units in total.

Build and install it

Save this next to the Swift file as build.sh:

#!/bin/sh set -eu APP="Mochi Stock.app" rm -rf "$APP" mkdir -p "$APP/Contents/MacOS" swiftc -O -parse-as-library MochiStock.swift -o "$APP/Contents/MacOS/MochiStock" cat > "$APP/Contents/Info.plist" <<'EOF' <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>CFBundleName</key><string>Mochi Stock</string> <key>CFBundleIdentifier</key><string>dev.mochi.stock</string> <key>CFBundleExecutable</key><string>MochiStock</string> <key>CFBundlePackageType</key><string>APPL</string> <key>LSMinimumSystemVersion</key><string>13.0</string> </dict> </plist> EOF mkdir -p "$HOME/Applications" rm -rf "$HOME/Applications/$APP" cp -R "$APP" "$HOME/Applications/" echo "installed $HOME/Applications/$APP" 
chmod +x build.sh ./build.sh 
installed ~/Applications/Mochi Stock.app 

The script copies the app to ~/Applications for a reason. tamesu starts an app by its name or its bundle id, not by a path. So the app must live where macOS looks for apps.

Step 3: Configure the Project

Create a second folder for the tests. Add a file named tamesu.yaml:

version: 1 suites: ["suites/*.yaml"] genkit: plugins: [googlegenai] models: default: googleai/gemini-3.8-flash judge: googleai/gemini-3.8-flash browser_agent: googleai/gemini-3.8-flash defaults: { timeout: 3m, max_goal_steps: 10 } budgets: { max_cost_usd: 2 } reporters: [console, junit, html] secrets: GEMINI_API_KEY: env:GEMINI_API_KEY 
  • models.browser_agent is the model that drives the app in a goal step. The name comes from the browser target; the desktop target uses the same setting.
  • models.judge grades the whole session at the end.
  • budgets stops the run when the model spend passes the limit.
  • secrets are references, never values. env:GEMINI_API_KEY tells tamesu where the key lives.

Step 4: Write the First Test with Goals

Create suites/stock.yaml:

suite: stock tags: [desktop] secrets: STOCK_PIN: env:STOCK_PIN targets: stock: type: desktop app: Mochi Stock tests: - name: unlock and receive stock target: stock steps: - goal: "Unlock the app with the PIN {{secrets.STOCK_PIN}}" - expect: "The stock list is visible and the Canvas slip-on is marked as low stock" - goal: "Receive 5 units of the Canvas slip-on" - expect: "The Canvas slip-on has 8 units in stock and no low stock warning" - action: { assert_text: "Total: 58 units" } assert: - judge: "The steps show the app being unlocked and 5 units of the Canvas slip-on being received" - metric: { name: goal_steps, lte: 8 } 

The target is two lines: the type and the name of the app. The test reads like a note to a colleague. Two goals do the work, two expect steps check the meaning, and one action pins the exact total.

The assert block runs after the steps:

  • judge asks a model to grade the whole session against one sentence.
  • metric checks numbers that tamesu measured. Here the agent may use 8 actions at most.

Check the setup

export STOCK_PIN=2468 tamesu validate tamesu doctor 
1 suite, 1 test, 0 errors, 0 warnings ok config: tamesu.yaml (version 1) ok suites: 1 file, 1 test ok secret GEMINI_API_KEY: env (link 1 of 1) ok secret STOCK_PIN: env (link 1 of 1) ok models: default=googleai/gemini-3.8-flash (GEMINI_API_KEY resolved), judge=googleai/gemini-3.8-flash (GEMINI_API_KEY resolved), browser_agent=googleai/gemini-3.8-flash (GEMINI_API_KEY resolved) skip speech: no speech models configured (speech.tts / speech.stt); audio mode and sip need them ok chromium: Google Chrome 157.0.8103.79 (/Applications/Google Chrome.app/Contents/MacOS/Google Chrome); no browser target configured ok agent-desktop: 1.2.0 (~/.cache/tamesu/agent-desktop/1.2.0/agent-desktop) ok desktop permissions: the desktop can be read and captured skip livekit: no livekit target configured skip sip: no sip target configured 

The last two ok lines are the desktop checks. agent-desktop shows the binary a run will use. desktop permissions confirms that macOS lets tamesu read and capture the desktop.

Before the first run, the agent-desktop line says that version 1.2.0 "is not in the cache yet". The first tamesu run downloads it.

Run it

tamesu run 
tamesu run: 1 test, output in .tamesu/runs/2026-11-17T14-43-00 PASS stock / unlock and receive stock 1m13s 1 passed, 0 failed, 0 errors in 1m13s · 5 model calls, 6801 tokens, $0.0000 · llm cache: 0 hits, 1 misses 

Mochi Stock opens, unlocks, receives the stock and closes. Nobody touched the mouse.

Mochi Stock after the test received 5 units: the Canvas slip-on has 8 in stock and the total is 58

What the agent did

Every step is in steps.json, in the run directory. This command lists the actions of the two goals:

jq -c '.[] | select(.kind == "goal") | .actions[] | [.name, .args.element, .args.text]' \ .tamesu/runs/*/tests/unlock-and-receive-stock/run-1/steps.json 
["type","textfield \"PIN\"","{{secrets.STOCK_PIN}}"] ["click","button \"Unlock\"",null] ["click","combobox \"Product\"",null] ["click","menuitem \"Canvas slip-on\"",null] ["type","textfield \"Units\"","5"] ["click","button \"Receive\"",null] 

Two things to notice:

  • The agent found everything by role and name. Nobody told it that the product is picked in a dropdown named "Product".
  • The PIN is still a placeholder. The model planned with {{secrets.STOCK_PIN}} and never saw 2468. tamesu puts the value in at the moment of typing, and passes it to agent-desktop on standard input, never as a command-line argument.

After the unlock, the window is richer. This is what the agent reads before the second goal:

 window "Mochi Stock" (focused) [6] textfield "Search products" [7] scrollarea [8] outline [9] treeitem "Trail runner 12 in stock" [10] cell "Trail runner 12 in stock" (focused) text "Trail runner" text "12 in stock" [11] treeitem "City sneaker 30 in stock" [12] cell "City sneaker 30 in stock" (focused) text "City sneaker" text "30 in stock" [13] treeitem "Canvas slip-on Low stock 3 in stock" [14] cell "Canvas slip-on Low stock 3 in stock" (focused) text "Canvas slip-on" text "Low stock" text "3 in stock" [15] treeitem "Leather boot 8 in stock" [16] cell "Leather boot 8 in stock" (focused) text "Leather boot" text "8 in stock" text "Product" [17] combobox "Product" value="Trail runner" (readonly) [18] textfield "Units" [19] button "Receive" text "Total: 53 units" [3] button "Close" [4] button "Full Screen" [5] button "Minimize" text "Mochi Stock" 

A row of the list is a treeitem. The dropdown is a combobox with its current value. The total is plain text.

Good to know:

  • A click presses the element through the accessibility API. The pointer does not move.
  • Text is normally set the same way, with no key pressed. Some fields take text from the keyboard only. tamesu then focuses the field and types with the real keyboard, and the run records a keyboard event in result.json. The three text fields of Mochi Stock are of that kind.
  • The real keyboard types into whatever has the focus. Leave the keyboard and the mouse alone while a suite runs.
  • An expect step sees what the window shows. The secure field shows no value: textfield "PIN" (focused, secure).

Step 5: Run It Again: the Plan Cache

Run the same command a second time:

tamesu run 
tamesu run: 1 test, output in .tamesu/runs/2026-11-17T14-44-13 PASS stock / unlock and receive stock 31.3s 1 passed, 0 failed, 0 errors in 31.3s · 3 model calls, 1660 tokens, $0.0000 · llm cache: 0 hits, 1 misses 

The first run took 1m13s, 5 model calls and 6,801 tokens. This one took 31 seconds, 3 model calls and 1,660 tokens.

When a goal succeeds, tamesu saves its actions as a plan in .tamesu/cache/actions. This is the heart of the plan of the first goal:

{ "goal": "Unlock the app with the PIN {{secrets.STOCK_PIN}}", "url_path": "desktop://Mochi Stock", "key": "1487a403320f0602b9600cc0", "actions": [ { "name": "type", "args": { "element": "textfield \"PIN\"", "ref": 1, "text": "{{secrets.STOCK_PIN}}" } }, { "name": "click", "args": { "element": "button \"Unlock\"", "ref": 2 } } ] } 

The next run replays the plan and does not call the agent. The three calls that remain are the two expect steps and the judge, the steps that need judgment. The metrics of the second run say the same:

{ "cache_hits": 2, "cache_misses": 0, "desktop_actions": 6, "goal_steps": 6, "llm_cost_usd": 0, "llm_tokens": 1660, "observations": 10, "turns": 0 } 

A goal with the same window replays its plan with no model call; a changed window makes the agent plan again and save the new plan

The key of a plan is the goal, the app and the set of elements in the window (their roles and names). When the window changes, the key changes, and the agent plans again. You will see that in Step 9.

Two flags control the cache:

  • tamesu run --update-cache ignores the saved plans and plans every goal again.
  • tamesu run --replay uses only saved plans. A goal with no plan fails. This is the mode for CI.

Commit the .tamesu/cache folder with the suite.

Step 6: Pin Exact Behavior with Actions

A goal is flexible. Some checks must not be. Add two tests to suites/stock.yaml:

 - name: wrong PIN keeps the stock room closed target: stock steps: - action: { fill: { name: PIN, text: "0000" } } - action: { click: { role: button, name: Unlock } } - action: { assert_text: "Wrong PIN. Try again." } - expect: "The app is still locked and shows no products" - name: search narrows the list target: stock steps: - action: { fill: { name: PIN, text: "{{secrets.STOCK_PIN}}" } } - action: { click: Unlock } - action: { wait: { text: "Total: 53 units" } } - action: { fill: { name: "Search products", text: "boot" } } - action: { assert_text: "Leather boot" } - expect: "The list shows the Leather boot and no other product" 

No goal here, so no agent drives the app. Each action names its element the way a person would: by the text on it. The expect at the end of each test still asks the model one question.

The lock screen after a wrong PIN: the app shows "Wrong PIN. Try again."

The desktop target has six actions:

Act...Source:https://dev.to/xavidop/agentic-desktop-testing-with-tamesu-test-native-apps-the-way-a-person-uses-them-2jno

Related posts