Cloud
A one-step probe told us YouTube was blocked. It was asking the wrong endpoint.
Devil Scrapes Dev.to (EN Zone)
3 views
Quick answer: If you POST to YouTube's internal /youtubei/v1/next with just a videoId and count zero comments in the response, you have not found a block. You have found the wrong endpoint. The watch payload never contains comments — they live behind a continuation token in the comments-section engagement panel, and you need a second request to get them. A one-step probe returns a confident false negative on every IP tier, including your home connection.
We know because that false negative cost us three months.
The shelf note that was true, and too broad
In June we probed YouTube's internal API from a datacenter proxy pool and wrote this down:
YouTube serves empty payloads to the WEB innertube client from all Apify datacenter IPs.
That is a real finding. We measured it, and for the endpoints we measured it on — the shorts and search payloads — it still holds. The problem was the last four words. "All Apify datacenter IPs" got generalised to "YouTube", and a note scoped to one surface became a blanket verdict on a whole target.
So when comment scraping came up again, the tooling answered before we did: the API host was already claimed by a shelved Actor, the shelf reason said IP-class block, and there was nothing to discuss. That is the dangerous kind of institutional memory — correct in its original scope, load-bearing outside it, and never re-tested.
What it was holding off
When we finally measured the demand instead of the block, this is what sat behind it:
Rival
30-day users
30-day runs
streamers/youtube-comments-scraper
23,259
83,460
apidojo/youtube-comments-scraper
2,852
99,081
138 competing Actors, 64 of them with real users. Each of those two runs roughly five times the monthly volume of our single best-earning scraper. It was the largest reachable demand cluster we had ever measured, and a three-month-old footnote was sitting on it.
Why the first probe lied
Here is the part worth stealing, whatever you are scraping.
Our first re-probe read 449 KB back from /youtubei/v1/next, counted zero comment nodes, and did it on all three tiers — direct, datacenter, residential. It looked exactly like the documented degradation. Same shape, same silence, three independent routes agreeing.
It was not a block. The watch payload does not carry comments at all. YouTube's own page does this in two steps:
Fetch the watch payload and pull a continuation token out of the comments-section engagement panel.
POST /youtubei/v1/next again, with that token, to get the actual comment nodes.
Step one returns a big, healthy, comment-free response. If you stop there and count comment nodes, you will measure zero on a perfectly working connection, and every tier you try will confirm it. Three agreeing tiers felt like strong evidence and were worth nothing, because the instrument was wrong in the same way on all three.
The actual result, once we asked the second question
Same video, same machine, step two:
Tier
status
bytes
comment nodes
next cursor
direct
200
281,238
20
yes
Apify datacenter
200
261,475
20
yes
Apify residential
200
261,421
20
yes
A full first page and a live cursor on every tier. The datacenter exit is not degraded for this endpoint.
The commercially interesting line in that table is the last one. Residential buys nothing here, and this endpoint returns ~260 KB per page. Residential proxy bandwidth runs about $8/GB. Reaching for the expensive tier "to be safe" on a target that does not need it is a bill you pay forever, on every page, for no additional data.
What we did not conclude
Three things this does not establish, stated plainly because the original mistake was overclaiming from a narrow probe:
It does not retract the June finding for the endpoints it was actually measured on. Scope it to those.
It does not prove a cloud run behaves like a local one. Local recon has overpredicted cloud results for us before. The remaining variable is the runtime, not the IP.
Three requests reaching a target says nothing about scraping a thousand pages through it. Reach is not volume — we have had a target clear recon and then hit a captcha at page 99.
What to take from this
Before you accept "this target is blocked", check that your probe asks for the thing you are counting. A probe that fetches the wrong endpoint and counts zero results is indistinguishable from a probe that found a block, and it will reproduce consistently across every proxy tier you own — which feels like confirmation and is actually just the same error, three times.
And when you write a NO-GO down, write down the endpoint it applies to. Ours was true. It was just wearing a label three sizes too big, and it quietly cost us the biggest thing on the board.
🔥 We packaged this one: YouTube Comments Scraper walks both steps for you — it finds the continuation token wherever YouTube nests it this month, pages the real comment feed, and hands you typed rows (author, text, likes, replies, pinned/hearted flags, full reply threads on request). It runs on datacenter proxy, because we measured that residential adds nothing here. $0.70 per 1,000 results, and you only pay for rows that land.
FAQ
Why does a single call to /youtubei/v1/next return no comments?
Because comments are not in the watch payload. They load from a separate continuation request, keyed by a token found in the comments-section engagement panel of the first response.
Do I need residential proxies to scrape YouTube comments?
We measured datacenter, residential and direct against the same video and got a full 20-record page with a live cursor on all three. For this endpoint, residential adds no data and costs roughly $8/GB.
Is YouTube's internal API blocked from datacenter IPs?
For the comment continuation endpoint, we found no degradation. Our earlier finding of empty payloads was measured on different surfaces (shorts and search) and should be scoped to those rather than to YouTube as a whole.
How do I avoid this class of false negative?
Make your probe assert on the thing you are counting. If you are counting comments, confirm the endpoint you called is the one that returns comments — otherwise a zero tells you nothing about whether you are blocked.
Read original: https://dev.to/devil_scrapes/a-one-step-probe-told-us-youtube-was-blocked-it-was-asking-the-wrong-endpoint-31l1
← Previous
When Falco Runs Out of Metadata
Next →
Texas judge rules TikTok misled users on child safety feature
Related
post-quantum tls is a platform migration, not a crypto project
Cloud
6
DEV Community
Bir Hafta Sonu Projesi: AWS EC2 Üzerinde V2Ray (3X-UI) ve Cloudflare WARP ile Datacenter IP Engellerini Aşmak
Cloud
5
Dev.to (EN Zone)
How to Set Up Auth0 Organizations (SSO, Branding & Invitations)
Cloud
7
DEV Community
Introducing Firebase spend caps
Cloud
0
Firebase Blog
Comments0
No comments yet — be the first