I run a private agent at home. It has a profile file with my sizes, my fit, my colours, my brands, and it fetches things from shops I actually buy from.

Then I open-sourced the fetching engine as a separate tool. It had its own profile file. Its own shop list. Its own idea of what "my size" means.

For about three weeks they disagreed. Not loudly. The private agent said I was a size XL in trousers. The public tool said XL or XXL. One of them was right and I had no idea which, because the tool's output looked exactly as authoritative as the agent's.

The duplication I hadn't counted

I wrote the public tool first, then went back to look at my private agent. Same thing. Same twenty Shopify shops. Same /products.json endpoint. Same "per-variant stock means in stock in your size" logic.

Every shop was being hit twice, once by each system, on different schedules, with different parsing. Every week the two pools drifted a little further apart, because the private one had a fix for a size label the public one didn't, and vice versa.

The private agent was, in fairness, much older and much cleverer: it had six more adapters built on a headless renderer, photo matching, price-error detection, order history. But all of that sat on top of the same twenty shops I'd duplicated.

I had written the same fetcher twice and called it two features.

The obvious fix, and the bit that wasn't

The obvious fix is "have one call the other". Which one? That turned out to be the wrong question to ask first.

The right question was: which one owns the facts?

My agent's profile.yaml was the file I actually edited. When my size changed, I changed it there. So that file was the source of truth — and the public tool's profile was a copy of a copy, written by hand, already stale.

So the shape of the fix was:

 agent/profile.yaml <- the one file I edit (source of truth) | | bridge script: projects it into the tool's schema v tool/profile.yaml <- generated, never edited by hand tool/config.yaml <- generated: the shop list, from the agent's config

The bridge is about a hundred lines and does the unglamorous work: copy the sizes, map the fit rules, translate "never Zara" into a brand filter, and drop the fields the tool must not have.

It drops two fields on purpose. The tool's schema refuses them — and so does my agent's own rules. It stores my height and weight for fit reasoning, and it is not allowed to repeat those numbers to anyone, ever. So the bridge does not copy them, and the tool's validator rejects them if they appear. Two independent rules agreeing on one boundary is exactly the kind of thing you want, because it means the boundary can't be crossed by accident later.

The part I did not expect

Then I changed the fetch path. The twenty shared shops were now fetched by the tool — the tested, packaged one — and the agent read the tool's pool instead of fetching those shops itself. The agent kept its six extra adapters and its headless-rendered footwear retailers; it just stopped re-fetching the twenty shops it had in common with the tool.

The surprising bit is how little changed afterwards. The agent's pool format looked nothing like the tool's — different field names, different shape, one of them stores sizes as a JSON string. About forty lines of mapping sat between them, and then... it worked. The agent's daily digest started including the tool's items and there was no visible seam.

That forty lines is the whole trick. The two systems never had different facts. They had different shapes, and shapes are cheap to translate.

The rule I keep relearning

If two systems need the same facts, one of them must not gather them.

Not "should share them" — share them. Gather them once, in the place that owns them, and let everyone else read. The temptation with agent systems is that every soul feels entitled to its own scraper, because a soul that fetches its own data feels self-contained. But self-contained and duplicated are different things, and duplication doesn't announce itself: both copies look right, and the drift shows up months later as "why did the tool think my size was different".

Two smaller things I learned doing it:

  • A generated file needs a generator that runs, not a note that says it should. My first attempt was a script I ran by hand whenever I remembered. It was wrong within a fortnight. The version that works is three lines in the weekly job: bridge first, then fetch. Now it cannot drift, because it is regenerated on the same schedule that consumes it.
  • Keep the estate files out of the public repo. The moment the bridge and the merge plan were in the public repo, I had to take them back out again — the wiring belongs on the machine, not in the project. Small thing, but it's the difference between a reusable library and someone's private setup with a public URL attached.

What I'd tell past me

If you're about to open-source the engine that powers one of your own agents, decide up front which side owns the profile — and make the other side generated, with the generator wired into the schedule that uses it.

The tool is better for being public: it has tests it wouldn't have had as a private script, and it has a packaging that forced me to fix a fetcher that had never worked. But the private agent is better for being the source of truth, because it's the one where I actually edit the file.

Two systems, one profile, one fetch path. The code that glues them together is the boring part, and the boring part is where all of the duplication used to live.

🤖 Drafted with AI assistance from my own homelab notes, logs and repos, then reviewed and edited before publishing.

🌐 More field notes: https://iamtech.cn1-lab.uk