Playwright Agents Six Months Later: The Brutal Lessons That Made Them Better

Last time, my planner burned 156k tokens, 78% of my window, planning one feature. If I asked it to plan the next feature on the same page, it scanned the whole page again from scratch, as if the page might have changed overnight. Six months later my agents work like an actual D&D party, with a rulebook, a scout, a cleric on a budget, and me as the DM signing off on every big move. This is what changed in my Playwright agents, the skills and commands around them, and the flow that turns a Jira ticket into tests.

The Party Needed a Rulebook

The first fix wasn't an agent at all. CLAUDE.md started as one giant file with the whole framework in it, and every session loaded all of it, like reading the entire Player's Handbook aloud before anyone rolls initiative. The generator needed more than a summary of the framework, though. It needed to know how I actually write code, folder by folder.

So I split it. The root CLAUDE.md is a short pointer table now, and each folder gets its own guide that only loads when Claude is working there. The api/ guide explains that I treat endpoints like page objects, so the generator builds an API client the same way it would build a page. The tests/ui/ guide covers my three-file pattern for UI tests, and tests/api/ covers the singleton clients and helper rules.

It paid off in tokens too, which was the best tip in an article on cutting Claude Code's token burn (linked below). The root went from 12,306 characters to 5,177, which is 58% and roughly 1,790 tokens off every single session. That won't make anyone rich, but it comes off every session whether I send 2 messages or 200.

It's a tax cut, not a coupon.

Every session, back when CLAUDE.md read the whole framework aloud before doing anything.

Meet the Party

I started with three agents, all of them driving the browser and all of them doing a little of everything. Now each one has a class, and the party's bigger:

The party: each agent, its D&D class, and its job

The ranger is the big change. Before the page-mapper, the planner scouted every room itself on every plan, which is how one feature ate 78% of my window. Now the map gets written once to .claude/pages/, and every plan after that just reads it. That gave the planner one job: make the best plan it can, with all of its attention on what's worth testing instead of on clicking around.

The planner before the page-mapper, scouting the same page every single time.

I tried it first on a real ticket, an export drawer. Pages I hadn't mapped yet got test.fixme() instead of guesses, which is the ranger saying "haven't been there yet" instead of the wizard inventing a floor plan. I'll take an honest gap over a confident fake every time.

On the API side, my boss built the first generator, a paladin who had to call on a higher power for every move. It sent one Jira ticket to the Claude API along with a markdown file on how to plan it. When we moved to a Claude Code instance per QA person, I split it in two. An artificer plans, since API planning is the most technical job in the party, and a rogue writes the specs. The Anthropic SDK left the project for good, and Claude Code does all the thinking now.

I'm the DM Now

Hand it a Jira ticket key and the first thing it does is pull the ticket through the API and write a brief. The brief is the quest briefing: the ticket's requirements, its acceptance criteria, and the API clients the repo already has. That way the wizard knows what's in the party's inventory before planning anything. Point it at an epic and it pulls every child ticket too. It cuts the duplicates and writes one brief for the whole campaign, split into API and UI.

From there it runs one flow, and nothing gets past a checkpoint until I say so. The agents can roll all the dice they want. I approve the plan and the Manual tests before a single line of automation gets written. Sometimes the requirement itself is wrong. It can sail through development untouched, and if nobody reads the plan, the agents will write a test that proves the mistake works perfectly.

Manual tests come first on purpose. Writing the steps into the Xray case gives us real steps in our reporting. The Manual test and the automation both come from the same plan, so they line up when we tie them together. They also put a person in front of the feature. Human eyes still catch things the AI never thinks to look for.

The generate-tests skill runs all of this, and after the brief it stops and waits for my OK. I also wrote a rule into both the planner and the generator to check whether a test already exists before writing one. So when the fighter shows up to clear a room, the wizard can tell him it's already been cleared. I'd rather hear that than review a duplicate test I'm just going to delete.

The fighter, finding out the wizard already cleared this room.

The plans got stricter too. Every scenario reads like a manual test and names its Xray folder, which follows naming rules instead of whatever someone felt like typing that day. The generator writes the real Xray key into each automated test, so when one fails, it points straight back to the Manual test it came from.

House Rules

Last time I wrote about a filter test that checked the URL and never checked the table. That one was on me, bad prompting rather than a bad agent. Since then Nima Parvathy and I kept hitting failures in real use, and each one turned into a new rule in the agents:

• Don't wrap assertions in guards that let missing data pass quietly.
• Don't hide test.skip() behind a conditional where nobody will see it.
• Poll for eventually consistent data instead of waiting a fixed amount of time.
• Use the real pagination response shape, never guessed field names.
• When the plan and the Jira ticket disagree, assert what the ticket requires.

That last one is the big one, and Nima and I each added it at different times. It's rules as written versus the player who swears he rolled a 20. The point of a test is to prove the requirements are met, not to prove the code does what the code does. If the plan says something different from the ticket's requirements, the agent doesn't get to go with the plan. It asserts the acceptance criteria, lets the test fail, and reports the difference as a possible bug.

When I rebuilt this setup in public, plan review caught exactly that. One test expected the "Order Placed!" text the page showed instead of the message the site's own docs describe. Another accepted empty search results, so it would have passed with search completely broken. Both would have been green.

Both now fail when they should.

The search test, reporting green while search was completely broken.

Here's where it gets interesting. A rule in a prompt is a house rule, and house rules only hold while everyone at the table remembers them. In the public rebuild I moved every rule I could into lint, so serial mode, skipped tests, XPath, fixed waits, and .nth() never make it past npm run lint. The ticket-wins rule can't go there, because lint can't tell where an expected value came from, so that one still needs the DM reading the plan.

The Cleric Gets Three Spell Slots

In October I went after locators. The generator and healer share one set of rules now: anchor on a stable container, act on the element with a getBy* method, and never use XPath, class names, or .nth(). Class names don't change every build, but they can, and when one does, every locator built on it breaks at once. The generator also scopes to the right row or card on pages that repeat the same structure, instead of grabbing the first match it finds.

The healer gets three attempts per test, and then it stops. Without that limit it would run nonstop, chasing a locator that isn't flaky and never quite finding one. If it can't land a solid locator in three tries, the test gets test.fixme() with a comment saying what it tried, and I take it from there.

Three spell slots, then a long rest.

Sometimes nobody in the party is actually hurt. The public rebuild runs against a shared demo site, and at the default worker count 19 tests failed. Eight were real findings, the site disagreeing with its own docs, and most of the rest were the site answering "503 heavy load." Capping workers at 2 cut the overload failures from 11 to 2. Once I stopped smoke tests from running twice, the only failures left were the 7 findings.

The healer would have burned spell slots on every one of those 503s while the dungeon ceiling was the real problem. Nobody's filing a performance bug over a demo site with demo tests hitting it, so the config stays pinned at two workers and the agents check for 503s before blaming a locator.

The healer after fixing a locator that was never broken, while the site was throwing 503s

Packing the Party for Travel

Then our UI started splitting into plugins, and every plugin got its own Playwright repo. The party couldn't live in one repo anymore, so I packed it into a Claude Code plugin that any test repo can install. It carries the party, the skills that drive them, and the slash commands I reach for every day:

What the Claude Code plugin carries

Getting the whole party, their skills, and every command into one plugin.

The ticket commands got the best upgrade. They used to draft a ticket for me to paste in by hand. Now they create it once I confirm, and /bug-ticket with a ticket key files a Story-Bug under that ticket instead of a loose Bug. If Jira rejects the issue type, the error lists the valid ones and the command retries once on its own.

The bottleneck never went away. It moved from writing tests, to babysitting a planner that clicked through every page, to me sitting behind the DM screen reading plans. I'll take that trade, because the plan is the last place a wrong requirement gets caught before it turns green.

Links

• Adding Playwright Agents: Lessons from the Deep Trenches
• playwright-ai-qa-agents (the public version of the plugin)
• playwright-test-commons
• playwright-e2e-demo (a real suite written with the agents)
• Playwright Test Agents
• Claude Code Plugins
• 10 Tips to Stop Burning Your Tokens in Claude Code