AI & ML
A finished scraper sat on a git branch for 19 days. Nothing noticed.
Devil Scrapes DEV Community
3 views
We keep a written do-not-build list. Vinted has been on it for months, with a one-line reason: DataDome, plus a saturated incumbent at 108k runs and a 5-star rating. That is a perfectly sensible thing to write down. It was also wrong, and it took us 19 days to find out — not because the wall was hard, but because nobody ever tried it.
Here is the whole story, with the numbers.
Quick answer
A Vinted scraper we had already finished sat unnoticed on a git branch for 19 days because our build lane counted "the agent said it was green" as done, instead of "the code is in main". When we finally ran it against the live site, it passed on the cheapest proxy tier available — no residential, no browser automation — for $0.00105 in platform cost. The "DataDome, infeasible" note in our backlog had never been tested against the actual site.
Two lessons, both cheap to copy: a NO-GO that doesn't name every tier you probed isn't a NO-GO, and "an agent reported success" and "the code shipped" are different facts that need reconciling.
The finished scraper nobody knew existed
We run our build pipeline as parallel agents, each in its own git worktree. An agent takes a target from spec to local-green, commits to its branch, and reports back. The orchestrator harvests the branch into main.
That last step is the one with no safety net. If an agent is killed by a watchdog, or a harvest is skipped, or the orchestrator simply moves on, the work stays on the branch — and nothing looks at branches again. The scoreboard counted a build the moment the agent said "local-green". No process ever asked whether that code reached main.
We found it by accident, while running an unrelated query that happened to list 97 accumulated worktree branches. The sweep that actually finds these is two lines:
for b in $(git branch --list 'worktree-agent-*' | tr -d ' *+'); do
git diff --name-only main..."$b" 2>/dev/null | grep '^actors/' |
cut -d/ -f2 | sort -u | while read -r slug; do
[ -d "actors/$slug" ] || echo "STRANDED: $slug (branch $b)"
done
done | sort -u
One hit. A complete Vinted scraper — models, client, parser, four test modules with real fixtures — committed on 2026-07-29 and invisible ever since.
We did not take the 19-day-old "local-green" claim on faith. Fresh virtualenv, full run: 73 tests passed, and the input-schema prefill validator confirmed the Actor's default input still deserializes into its input model. Then, and only then, it went to the cloud.
The tier that was never probed
Our note said DataDome. What we had actually done was: nothing. No cloud run had ever touched this Actor.
The first real run passed:
PASS vinted-sold-listings run f5QniR52Jf1UNC4oO SUCCEEDED, 3+ rows
The billing record is the interesting part, because logs will not tell you which network a run used:
ACTOR_COMPUTE_UNITS 0.00477417 $0.000955
KEY_VALUE_STORE_WRITES 1.00000000 $0.000050
DATA_TRANSFER_INTERNAL_GBYTES 0.00043767 $0.000022
DATASET_WRITES 3.00000000 $0.000015
DATA_TRANSFER_EXTERNAL_GBYTES 0.00001517 $0.000003
total $0.001050
no residential transfer on this run — it did NOT use the residential tier.
No residential line at all. It cleared without the expensive network, for about a tenth of a cent.
This is the second time this pattern has bitten us in the same direction. Earlier we shelved a business-directory target as "Camoufox-class, infeasible" after a residential-only probe — and later found it cleared cleanly on plain datacenter proxy while residential and a full stealth browser both failed. That is the inverse of the pecking order everyone assumes. Stronger tier does not mean higher success rate; different networks have different reputations at different targets, and the only way to know is to try each one.
So we made it a rule: a NO-GO is only valid if it names every proxy tier probed. "It's blocked" is not a finding. "It returns 403 on datacenter and on residential-with-country-pinning, and browser automation is untried" is a finding, and the next person can act on it.
Verify the tier from billing, not from your own config
The rule above has a trap, and we walked straight into it.
You can pass a proxy configuration and still not use it. In one batch of five Actors that we explicitly forced onto residential, two billed zero residential bytes. One preferred an environment variable over the input we handed it; the other's proxy helper failed and silently degraded to a direct connection. Both would have been filed as "residential probed, still blocked" — manufacturing exactly the false NO-GO the rule exists to prevent.
Requesting a tier is not the same as using one. The only honest check is the run's billing record: if PROXY_RESIDENTIAL_TRANSFER_GBYTES is zero, residential was not used, whatever your config said. Our QA tooling now refuses to record a verdict when the tier it asked for didn't actually apply.
One caveat worth stating plainly, because we can't resolve it from billing alone: datacenter proxy isn't billed as a separate line item, so a run that shows no proxy transfer is either datacenter or a direct connection. Residential is definitively ruled out; datacenter-versus-direct is not. We're comfortable with that here because the run succeeded either way, but it's the kind of thing worth knowing before you cite a number like this one.
What the scraper actually does
Vinted Listings & Condition Scraper pulls listing rows with the condition and price fields that matter for resale-price research, and it is priced pay-per-result — $0.0025 per row, so a run that returns nothing costs nothing beyond the start fee.
We are not going to tell you Vinted is easy to scrape. Marketplaces change their defenses, and the honest summary of this post is that we got a good result on one day, on one tier, and we wrote down exactly which one. That is the standard we hold our own notes to. When a target does start pushing back, absorbing the blocks, the rotation and the retries is our job, not yours.
If you want the same treatment on adjacent marketplaces, the ones built on this pattern are OfferUp, Poshmark, Craigslist and eBay.
The two changes we actually made
Findings that don't change a process are just anecdotes. Both of these are now committed:
The stranded-Actor sweep runs at every harvest, not when someone gets lucky with an unrelated query. "Agent reported local-green" and "code is in main" are reconciled explicitly, and anything recovered gets its tests re-run before it lands — a weeks-old green claim is a hypothesis, not a result.
Every tier verdict is read from billing. Our sweep annotates any run where the requested tier didn't apply, instead of letting the verdict stand.
The Vinted scraper is the smaller half of this. The bigger half is that we had a finished, working, revenue-capable product sitting in version control for 19 days and no part of our system noticed. If you run agents in parallel worktrees, go run that two-line sweep. We would genuinely like to hear whether you find one too.
Read original: https://dev.to/devil_scrapes/a-finished-scraper-sat-on-a-git-branch-for-19-days-nothing-noticed-4l4i
← Previous
Designing a 5-band parametric EQ from the biquad up, in MATLAB
Next →
Why We Open-Sourced Shaide
Related
Comments0
No comments yet — be the first