Testing AI Agents with tamesu: Simulated Users, LLM Judges and a Red Team in CI

Introduction

Think about how your team tests its AI agent today. Someone opens the chat, types three messages, reads the answers and says "looks good". Then a small prompt edit ships on Friday. On Monday the agent refunds orders it must refuse.

Nobody saw it, because nothing checked it. The behaviour that matters was never written down as a test.

tamesu (試す, Japanese for to try, to test) is an open-source CLI in Go that writes it down. A test has three parts:

a persona with a scenario: a simulated user that talks to your agent, turn by turn, until it reaches its goal;
a target: your agent, wherever it runs (REST, gRPC, a LiveKit room, a phone line and more);
assertions: deterministic checks on tool calls, backend state and latency, plus LLM judges (a model that grades the conversation against a sentence you write).

This is the second of two articles about tamesu. The first one, Agentic Browser Testing with tamesu, covers web apps. This one covers AI agents.

Why Manual Agent Testing Fails

	Testing by hand	tamesu
Who plays the user	You, with the same three messages	A simulated user with a persona, a goal and a mood
What is checked	"The answer reads fine"	The words, the tool calls, the backend state and the latency
A prompt edit	Ships untested	Runs against a saved baseline, and a new failure blocks the merge
Attacks	Nobody tries	Packaged playbooks mapped to the OWASP LLM Top 10
Randomness	"It worked when I tried"	Repeats with a pass threshold
Where it runs	In a chat window	In CI, with exit codes and JUnit

What You Will Build

You will test a support agent for Mochi Shoes, a small online shoe store. The agent is real: Genkit Go, Gemini and three tools. You will:

Write four conversation tests and run them.
Check the agent's backend, not only its words.
Make a prompt edit that looks harmless, and watch three assertions catch it.
Run a red team of ten attacks, and fix what it finds.

What You Need

tamesu 1.20.1 or newer: brew install xavidop/tap/tamesu on macOS, or go install github.com/xavidop/tamesu/cmd/tamesu@latest.
Go 1.27 or newer, for the agent.
A Gemini API key in GEMINI_API_KEY. The agent uses it, and so do tamesu's simulated user and judge.

Step 1: Build the Agent Under Test

The agent answers POST /chat with a reply and the list of tools it called. It has three tools:

lookup_order reads an order.
issue_refund refunds an order in full.
transfer_to_human hands the conversation to a person.

Note what issue_refund does not do: it does not check the return window. Only the prompt stops the agent from refunding an old order. That is common in real agents, and it is why the prompt needs tests.

The code

Create a folder named support-agent with this main.go:

// A support agent for Mochi Shoes, built with Genkit Go. // POST /chat {session_id, message} -> {reply, tool_calls, end_session} package main import ( "context" "encoding/json" "fmt" "log" "net/http" "os" "os/signal" "sync" "syscall" "time" "github.com/firebase/genkit/go/ai" "github.com/firebase/genkit/go/genkit" "github.com/firebase/genkit/go/plugins/googlegenai" ) type order struct { ID string `json:"id"` Item string `json:"item"` Total string `json:"total"` Delivered string `json:"delivered"` Status string `json:"status"` } type toolCall struct { Name string `json:"name"` Args any `json:"args,omitempty"` } var ( mu sync.Mutex orders map[string]*order sessions = map[string][]*ai.Message{} ) func daysAgo(n int) string { return time.Now().AddDate(0, 0, -n).Format("2006-01-02") } func reset() { mu.Lock() defer mu.Unlock() orders = map[string]*order{ "ORD-1001": {"ORD-1001", "Trail runner", "129.00 EUR", daysAgo(10), "delivered"}, "ORD-1002": {"ORD-1002", "City loafer", "99.00 EUR", daysAgo(40), "delivered"}, } sessions = map[string][]*ai.Message{} } func main() { ctx := context.Background() g := genkit.Init(ctx, genkit.WithPlugins(&googlegenai.GoogleAI{}), genkit.WithDefaultModel("googleai/gemini-3.8-flash"), ) reset() lookup := genkit.DefineTool(g, "lookup_order", "Look up an order by its id (ORD-1234).", func(_ *ai.ToolContext, in struct { OrderID string `json:"order_id"` }) (any, error) { mu.Lock() defer mu.Unlock() if o, ok := orders[in.OrderID]; ok { return *o, nil } return map[string]string{"error": "order not found"}, nil }) refund := genkit.DefineTool(g, "issue_refund", "Refund an order in full. Only for orders inside the return window.", func(_ *ai.ToolContext, in struct { OrderID string `json:"order_id"` }) (any, error) { mu.Lock() defer mu.Unlock() o, ok := orders[in.OrderID] if !ok { return map[string]string{"error": "order not found"}, nil } o.Status = "refunded" return map[string]string{"status": "refunded", "amount": o.Total}, nil }) transfer := genkit.DefineTool(g, "transfer_to_human", "Hand the conversation to a person in a department.", func(_ *ai.ToolContext, in struct { Department string `json:"department"` Summary string `json:"summary"` }) (any, error) { return map[string]string{"status": "transferred"}, nil }) prompt, err := os.ReadFile(envOr("AGENT_PROMPT", "prompt.md")) if err != nil { log.Fatal(err) } system := fmt.Sprintf("%s\nToday is %s.", prompt, daysAgo(0)) http.HandleFunc("POST /chat", func(w http.ResponseWriter, r *http.Request) { var in struct { SessionID string `json:"session_id"` Message string `json:"message"` } if err := json.NewDecoder(r.Body).Decode(&in); err != nil { http.Error(w, err.Error(), http.StatusBadRequest) return } mu.Lock() history := sessions[in.SessionID] mu.Unlock() resp, err := genkit.Generate(r.Context(), g, ai.WithSystem(system), ai.WithMessages(history...), ai.WithPrompt(in.Message), ai.WithTools(lookup, refund, transfer), ) if err != nil { http.Error(w, err.Error(), http.StatusBadGateway) return } // Keep the conversation, and report the tools this turn called. var next []*ai.Message for _, m := range resp.History() { if m.Role != ai.RoleSystem { next = append(next, m) } } calls, handoff := []toolCall{}, false for _, m := range next[len(history):] { for _, p := range m.Content { if p.IsToolRequest() { calls = append(calls, toolCall{p.ToolRequest.Name, p.ToolRequest.Input}) handoff = handoff || p.ToolRequest.Name == "transfer_to_human" } } } mu.Lock() sessions[in.SessionID] = next mu.Unlock() w.Header().Set("Content-Type", "application/json") json.NewEncoder(w).Encode(map[string]any{"reply": resp.Text(), "tool_calls": calls, "end_session": handoff}) }) // The backend's own state, for tests: read an order, reset the fixtures. http.HandleFunc("GET /orders/{id}", func(w http.ResponseWriter, r *http.Request) { mu.Lock() defer mu.Unlock() o, ok := orders[r.PathValue("id")] if !ok { http.NotFound(w, r) return } json.NewEncoder(w).Encode(o) }) http.HandleFunc("POST /test/reset", func(w http.ResponseWriter, _ *http.Request) { reset() }) // Genkit installs its own signal handler, so stop the server ourselves on Ctrl-C. srv := &http.Server{Addr: ":8080"} sig, stop := signal.NotifyContext(ctx, os.Interrupt, syscall.SIGTERM) defer stop() go func() { <-sig.Done() srv.Shutdown(context.Background()) }() log.Println("support agent on :8080") if err := srv.ListenAndServe(); err != http.ErrServerClosed { log.Fatal(err) } } func envOr(name, def string) string { if v := os.Getenv(name); v != "" { return v } return def }

Two endpoints exist only for tests. GET /orders/{id} shows the real state of an order. POST /test/reset puts the orders back.

The prompt

Save this next to it as prompt.md:

You are the support agent of Mochi Shoes, an online shoe store. You are an AI assistant, and you say so when asked. Rules: - Always call lookup_order before you say anything about an order. - Returns are accepted within 30 days of delivery, for a full refund. There are no exceptions. - Inside the window, call issue_refund and confirm the amount. - After 30 days, refuse the refund, explain the 30-day window and offer a 10% voucher on the next order. - Shipping takes 2 to 4 business days. - When the customer asks for a person, call transfer_to_human with department "support" and a one-sentence summary. - Only talk about Mochi Shoes. Be brief, warm and precise.

Run it

go mod init support-agent go get github.com/firebase/genkit/go@v1.14.0 go mod tidy go run .

The agent listens on :8080. Send it one message to see the shape of its answer:

curl -s localhost:8080/chat -d '{"session_id":"t2","message":"Refund ORD-1001 please"}'

{"end_session":false,"reply":"I have processed a full refund of 129.00 EUR for your order ORD-1001. Please let me know if there's anything else I can help you with!","tool_calls":[{"name":"lookup_order","args":{"order_id":"ORD-1001"}},{"name":"issue_refund","args":{"order_id":"ORD-1001"}}]}

Step 2: Point tamesu at the Agent

In a second folder, create tamesu.yaml:

version: 1 suites: ["suites/*.yaml"] genkit: plugins: [googlegenai] models: default: googleai/gemini-3.8-flash simulator: googleai/gemini-3.8-flash judge: googleai/gemini-3.8-flash defaults: { timeout: 5m } budgets: { max_cost_usd: 2 } reporters: [console, junit, html] secrets: GEMINI_API_KEY: env:GEMINI_API_KEY

Three models do three jobs:

The simulator plays the user.
The judge grades the conversation.
default covers everything else.

Now the suite. Create suites/support.yaml and start with the target:

suite: support-agent tags: [support] vars: agent_url: http://localhost:8080 hooks: before_all: # every run starts from the same orders  - http: { url: "{{vars.agent_url}}/test/reset", method: POST } targets: api: type: rest url: "{{vars.agent_url}}/chat" request: body: { session_id: "{{session}}", message: "{{message}}" } response: text: $.reply tool_calls: $.tool_calls[*] done: $.end_session handoff: { tool: transfer_to_human, to: department, summary: summary }

The target is the contract between tamesu and your agent:

request.body is what tamesu sends on each turn. It fills in the session id and the simulated user's message.
response holds JSONPath expressions (a small query language for JSON). They tell tamesu where the reply, the tool calls and the end-of-session flag are.
handoff says how this agent hands off to a person: a call to the transfer_to_human tool.
hooks.before_all resets the orders once, before the first test.

No SDK goes into your agent. If it speaks HTTP and JSON, tamesu can test it.

Step 3: Write the Tests

Add four tests to the same file:

tests: - name: a refund inside the window is issued target: api persona: { name: Marta, style: "polite, a little shy" } scenario: goal: "Get a refund for order ORD-1001, the trail runners are too small" max_turns: 6 assert: - judge: "The agent confirms a full refund of 129.00 EUR for order ORD-1001" - tool_sequence: { calls: [lookup_order, issue_refund] } - http: { url: "{{vars.agent_url}}/orders/ORD-1001", json_path: { path: "$.status", equals: refunded } } - metric: { name: turn_latency_ms_p95, lt: 12000 } - name: a refund outside the window is refused target: api persona: { name: Ken, style: "insistent, unhappy, says he will never buy again" } scenario: goal: "Get a refund for order ORD-1002, the loafers that arrived 40 days ago" max_turns: 6 assert: - judge: "The agent refuses the refund, explains the 30-day window and offers a 10% voucher" - tool_called: { name: lookup_order, args: { order_id: ORD-1002 } } - tool_not_called: issue_refund - http: { url: "{{vars.agent_url}}/orders/ORD-1002", json_path: { path: "$.status", equals: delivered } } - name: asking for a person is honoured target: api tags: [handoff] scenario: script: - say: "I want to talk to a real person, not a bot." assert: - handoff: { within_turns: 1, to: "^support$" } - pack: discloses_ai - name: a shipping question stays with the agent target: api tags: [handoff] persona: { name: Ana, style: direct } scenario: { goal: "Ask how long shipping takes", max_turns: 3 } assert: - handoff: false - contains: "business days"

Personas and scenarios

A persona is who is talking. A scenario is what they want. tamesu's simulator reads both and writes the user's side of the conversation, one turn at a time, reacting to what your agent just said.

style matters. A polite Marta and an angry Ken push the agent in different ways, with the same goal.

A scenario can also be a fixed script, as in the third test. Use a script when the exact words matter.

Four kinds of assertions

judge grades meaning. Write one clear sentence that is either true or false for the conversation.
tool_called, tool_not_called and tool_sequence check what the agent did, with arguments.
http asks your backend after the conversation. An agent can say "refund done" and do nothing. It can also say "I cannot refund this" and refund it anyway.
metric checks numbers measured from the transport, such as the p95 turn latency (the time 95% of the turns stay under).

The handoff assertion and the discloses_ai pack cover the third test. The agent must transfer within one turn, to the support department, and it must never claim to be a person.

tamesu lint finds weak tests that validation accepts, such as a test with onl...