Most "how to bypass Cloudflare" guides give the same advice: set a real User-Agent, add some headers, rotate proxies. On a properly protected site that almost never decides the outcome. Each anti-bot vendor looks at different signals, and a fix that works on one of them can make another one worse.
These are field notes from running a scraping service against heavily protected sites every day. For each vendor you get how to recognise it, what it actually checks, what worked, and the traps. The numbers come from our own logs and test runs. I've left out site names on purpose, so you'll see "a big US retailer" instead.
Disclosure: I'm a co-founder of ScrapeUnblocker, a scraping API. You don't need it for anything below: every technique here works with open-source tools.
Step 0: find out what you're up against
Everything below depends on the vendor, so identify it first. The quickest way is the free Anti-Bot Detector we built. Paste a URL and it shows which bot protection, WAF, CDN and CAPTCHA the site uses, the cookies and headers behind each finding, and whether a plain HTTP request gets through. No signup.
If you'd rather check by hand, open DevTools and look at the cookies and response headers:
| What you see | Vendor |
|---|---|
cf-mitigated header, cf_clearance / __cf_bm cookies, _cf_chl_opt in the HTML | Cloudflare |
datadome cookie, x-datadome header, captcha-delivery.com | DataDome |
_px3, _pxhd, _pxvid cookies, a "Press & Hold" button | HUMAN (PerimeterX) |
_abck, bm_sz, ak_bmsc cookies, Server: AkamaiGHost | Akamai Bot Manager |
x-kpsdk-ct / x-kpsdk-cd headers, a script at /<uuid>/<uuid>/ips.js | Kasada |
aws-waf-token cookie, x-amzn-waf-action header, gokuProps in the page | AWS WAF |
reese84, visid_incap_*, incap_ses_* cookies, _Incapsula_Resource | Imperva (Incapsula) |
Six request headers that share one random prefix and end in -a, -b, -c, -d, -f, -z | F5 Shape |
Two warnings before you trust what you see:
- Cloudflare puts a script under
/cdn-cgi/challenge-platform/into normal pages, too. Seeing that path doesn't mean you're being challenged. - Go by raw headers and cookies, not by how a block page looks. When we went through about 105 failures that "looked like DataDome", only 8 actually were.
Five rules that apply to every vendor
1. Find the failing layer before you fix anything
A block can come from four places: the target itself, your exit IP, your client's fingerprint (TLS handshake and browser environment), or the way you solve the challenge. One cheap test separates them: send a plain HTTP client and a real browser through the same fresh exit IP.
- The HTTP client gets a challenge, the browser gets a hard block: your browser is the problem, not the IP.
- Blocked on some exits and fine on others: it's the exit.
- Blocked on every clean exit with one engine: that engine's fingerprint is flagged. Switch engines instead of buying more proxies.
And if several independent methods start failing at the same moment, the site changed its rules. Your code is probably fine.
2. HTTP 200 is not success
Challenge pages, sensor shells, consent walls and even proxy error pages can come back with status 200. We once lost 155,687 records because a proxy's authentication error came back as 200 and nothing checked the body. Always check for something only the real page has:
# Markers that only appear on challenge or block pages. # Don't match on "Just a moment...": Cloudflare translates it. CHALLENGE_MARKERS = ( "_cf_chl_opt", # Cloudflare challenge "Attention Required! | Cloudflare", # Cloudflare WAF block (1020) "captcha-delivery.com", # DataDome "px-captcha", # HUMAN "/_sec/verify", # Akamai interstitial "_Incapsula_Resource", # Imperva "gokuProps", # AWS WAF captcha ) def is_real_page(resp, must_contain: str) -> bool: if resp.status_code != 200 or "cf-mitigated" in resp.headers: return False html = resp.text if any(m in html for m in CHALLENGE_MARKERS): return False # The positive check is the one that matters: # a price, a product title, a JSON key you expect. return must_contain in html 3. Change the exit before you change the engine
When the block is about the IP, attempts on different exits are close to independent. On one site a single attempt through an ISP proxy worked 3 times in 8. Allowing up to 12 attempts on different ISP exits got 100%, still at plain HTTP speed (about 2.6 s per page). Exits also differ a lot by provider and network: on one DataDome site the same Chrome got 0/12 through one ISP pool and 6/6 through another.
The exception is rule 1: if it fails on every clean exit, stop rotating. Each extra attempt just burns IPs (and residential gigabytes).
4. Don't over-spoof
Stealth plugins that override navigator properties made things worse for us, not better. Removing puppeteer-stealth-style JavaScript overrides raised our pass rate on a mixed set of Akamai, Shape and Kasada sites from 22/78 to 35/78. On one hotel site it went from 0/6 to 5/6. Inconsistent spoofing looks worse than none.
What still gives you away is the environment, not the JavaScript properties:
- A software WebGL renderer on a VM without a GPU. On one DataDome site a GPU-less VM got 0 passes with every browser, and a server with a real integrated GPU got 4/4.
-
screen.availHeight === screen.height, which means there's no taskbar and so no real desktop. - WebRTC leaking your server's real IP behind the proxy. In Chrome only a profile preference fixed it; the command-line flag didn't.
5. Scale with more IPs, not more concurrency
Behavioural systems count what each IP does. On a hard Cloudflare site, raising concurrency from 1 to 2 tripped exit refusals within 14 minutes. Doubling the IP pool from 48 to 96 instead cut latency from 12-15 s to 2-4 s and raised output from about 16 to about 99 items a minute, with zero blocks.
The same goes for testing: don't benchmark behavioural vendors at high rates. A 1,000-call test at concurrency 3 made Akamai tighten its checks for all our exits after about 1.5 hours. Your load test trains their model.
The core trick: solve once in a browser, replay over HTTP
For most vendors the expensive part is a single challenge. Once a real browser passes it, the site gives that browser a clearance cookie, and you can reuse the cookie from a fast HTTP client that impersonates a browser's TLS handshake until it expires. On one Cloudflare site, 29 browser solves served 3,155 requests. On a DataDome site, one cookie served 40 out of 40 never-visited URLs in 108 seconds.
Three things have to match the browser that solved the challenge:
- The same exit IP. Cloudflare, DataDome and Imperva tie the cookie to it.
- The same User-Agent.
- A TLS fingerprint from the same browser family. A cookie issued to Firefox gets replayed with a Firefox profile.
With curl_cffi (MIT), the replay side looks like this:
from curl_cffi import requests # Taken from a real browser session that has just passed the challenge, # through the SAME proxy exit you're going to replay on. PROXY = "http://user:pass@exit-host:port" UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36" cookies = {"cf_clearance": "...", "__cf_bm": "..."} session = requests.Session( impersonate="chrome146", # same family and close to the browser's version headers={"User-Agent": UA}, cookies=cookies, ) resp = session.get( "https://example.com/some/deep/page", proxies={"https": PROXY, "http": PROXY}, timeout=30, ) print(resp.status_code, is_real_page(resp, must_contain="product-title")) Keep the session object between requests. Some vendors (Akamai, below) rewrite cookies on every response, and the session keeps the jar up to date for you.
Two notes on the browser side:
- Read cookies at browser level, not with
document.cookie. The clearance cookies are usuallyhttpOnly, so read them through CDP (Storage.getCookies) or your automation library's cookie API. - Don't authenticate the proxy through CDP request interception (the
Fetchdomain) or a proxy extension. On Cloudflare that gave an instant 403, and with DataDome and Imperva it broke the challenge entirely. Run a small local forwarding proxy that adds the credentials, and point the browser at that.
Cloudflare
What it checks. The hard blocks key on the TLS fingerprint (JA4), not the User-Agent. In our test, curl with a Chrome TLS fingerprint and a Safari User-Agent was blocked, while curl with a Safari fingerprint and a Chrome User-Agent passed.
Read the response first:
-
cf-mitigated: challengeplus a "Just a moment..." page means a challenge you can solve. - "Attention Required! | Cloudflare" or "Sorry, you have been blocked" (error 1020), around 700 bytes and with no
cf-mitigatedheader, is a WAF rule. Solving won't help: you have to change who you look like (fingerprint or exit). - 1015 or 429 is a rate limit.
What worked:
- Try several curl_cffi profiles before reaching for a browser. Safari profiles often pass where Chrome ones get challenged: 4/6 against 1/6 on one site. Pass rates are probabilistic, so rotate profiles.
- Browser solve plus replay (above). It cut latency from about 30 s per page (a full Turnstile on every call) to a few seconds.
- Respect the per-IP rate. On hard zones, about one request every 20-30 seconds per IP stayed clean. Two or three within a few seconds got a 403 or 429, and Cloudflare remembers the block per IP and site.
- Interactive Turnstile needs a clean browser. Use plain Chrome with no CDP client attached and no remote-debugging port, click with real OS mouse input, then read the cookie. With CDP attached, Cloudflare accepted the click but never issued a clearance.
- Click once and wait about 14 seconds: clicking again during "Verifying..." restarts the check.
- The widget sits in a closed shadow root, so find it by its position on a screenshot, not with CSS selectors.
- Measure your fingerprint on the machine that will run the browser (for example at tls.peet.ws). Cloudflare blocks specific browser builds: in one zone Chrome for Testing 131 got 0/4 while 128 got 2/2. A newer Chrome feature (the new ALPS codepoint) was on by default in the Windows build of the same version but not the Mac build, and that alone moved the JA4 from "challenge" to "block".
Traps:
-
cf_clearanceexisting doesn't mean you solved anything. Cloudflare sets a provisional one as soon as the challenge loads. Only real content proves a solve. - Measure the clearance lifetime instead of guessing. It's configurable per site (Cloudflare's default Challenge Passage is 30 minutes). On two hard sites we measured about 5 minutes, and replays did not extend it.
- Rapid testing burns exits. About 15 solves in a few minutes turned challenges into 1020 blocks on those IPs. Don't experiment from your home IP either.
- Rules can be path-scoped. The homepage may pass while deep pages are blocked, so test the page you actually need.
- A solved token is not trust. We haven't found an open-source tool that reliably produces a replayable clearance for interactive Turnstile, and a popular paid solving service managed 1 in 8 on one hard site.
DataDome
How to read it. The block page has an rt field: 'c' is a CAPTCHA (slider), and 'i' is a device check that returns 403 and then reloads itself into the real page after a few seconds. t:'bv' means the IP or session is banned.
What it checks: exit reputation (very much by provider and network), the device (a GPU matters, see rule 4) and the browser engine. Which engine it dislikes changes from site to site. On a property portal real Chrome got a CAPTCHA 25 times out of 26 while a patched Firefox passed 5/8. On an electronics retailer it was the other way round.
What worked:
- Harvest and replay, with a cap. The cookie is tied to the IP and short-lived. Cap it at about 40 replays and about 20 requests a minute per exit, then harvest a fresh one. On a travel review site that took success from 75% to 97% and the median response time from 8.5 s to 2.9 s.
- Wait out the device check. Real Chrome driven over CDP passed
rt:'i'15/18, 9/9 and 9/9 on three sites where a patched Firefox and Edge got CAPTCHAs. Give the interstitial up to 15 seconds before you treat the 403 as final. - Use a real mouse for the slider. Drag it with OS-level mouse input on a machine with a real display. One solve then feeds many replays, and ISP proxies were enough.
- Try Safari profiles. On a pet retailer, curl_cffi Safari profiles alone passed 24/24 while Chrome profiles got 0/12.
- Harvest on the right page. Some sites only accept a cookie issued on the same type of page: a cookie from the search page failed 4/4 on review pages.
Traps: a VM without a GPU gave false 0/N results; proxy authentication through an extension or CDP broke the challenge; and the cookie carries the harvest exit's country, so prices and currency follow the exit you harvested on.
HUMAN (PerimeterX)
What it checks: for browsers, the engine's fingerprint decides which challenge you get. On a big US retailer, through the same clean ISP exit with fresh profiles and no warm-up:
- real Chrome: 200
- real Chrome driven over CDP: 200
- Playwright Firefox: Press & Hold
- undetected-chromedriver: Press & Hold
For HTTP clients, it judges the TLS identity together with the shape of the request. On one site the curl_cffi chrome146 profile got the full page and chrome150 got a 307 to /blocked 5 times out of 5. On another site it was exactly the opposite.
What worked:
- Look for a no-browser path first. curl_cffi with the right identity plus a same-site
Refererhandled 1,019 calls at 100%, averaging 2.4 s (about 9 s in a browser). On another site: 1,240 calls at 99.8%. - Send the Referer on the very first request. On a home-goods retailer, deep URLs without a
Referergot 429 on every exit and every engine. With a same-originRefererandSec-Fetch-Site: same-originthey passed 38/38. A block poisons the exit for 45-90 seconds, so it has to be right from the start. - Rotate identities and rest refused exits. Rotate through a list of curl_cffi identities with the last one that worked first, and rest a refused exit for about 5 minutes.
- Use fresh sessions, not reused sticky ones. Reused sticky sessions got 6.2% against 100% for fresh ones.
- Press & Hold needs real OS mouse input. We didn't find a commercial solver for it that works.
Trap: a proxy set through a Manifest V2 extension silently stopped working in recent Chrome, so the traffic left from the server's datacenter IP. It looked exactly like a fingerprint block. Check your real egress IP before anything else.
Akamai Bot Manager
How to read it. "Access Denied" (often localised), an interstitial calling /_sec/verify, or a 200 with about 250 KB of JavaScript and an empty title. That last one is the sensor shell, and it's a failure even though it says 200.
What it checks: the engine fingerprint on fresh sessions (again it varies by site), plus a per-exit budget. On a large marketplace each exit got about 6-8 requests before being refused for 1-2 hours.
What worked:
- Keep the cookie jar rolling. Akamai rewrites
bm_s/bm_soon every response. A static jar replayed 1 time out of 12; the same jar updated from eachSet-Cookiewent 12/12. Use one jar for one request at a time, because concurrent use kills it. - Keep
_abckfresh. It goes stale within minutes. A background job re-harvesting every 90 seconds or so kept replays at about 0.6 s. - Harvest on a lightly protected page, then call the site's JSON API. One jar was good for 40-60 reads.
- Warm up like a visitor. Open the homepage, spend 5 seconds moving the mouse and scrolling, then go to the deep link in the same tab: 21/42 against 12/56 without the warm-up. With up to 4 attempts that took one site from 11% to 86%.
- Match the harvest country and don't assume the newest profile is best. Replays worked only through exits in the harvest country, and an older Chrome impersonate profile worked better than the newest.
Traps: counting the sensor shell as success. And residential proxies didn't help on the large marketplace: the limit there was per exit and per behaviour, not about IP type.
AWS WAF
The easiest one, if it's the CAPTCHA action.
How to read it: the CAPTCHA action returns HTTP 405 with x-amzn-waf-action: captcha, and the JS challenge returns 202.
What it checks: the CAPTCHA rule fires on every request that doesn't carry a solved token, on every exit and engine. Unlike the JS challenge, waiting doesn't help. Check geo first, though: one Japanese site served the CAPTCHA to every non-Japanese IP, while Japanese exits got the real page with plain curl.
What worked: solve the 3x3 image grid once (we sent the images to an image-classification API at a fraction of a cent each). The resulting aws-waf-token is a bearer token that isn't tied to the IP, User-Agent or TLS fingerprint. In our case it stayed valid for an hour, though sites can configure this. One token served 1,000 calls at 100%, with a median of 2.0 s.
Trap: a redirect between www and the bare domain drops the token.
Imperva (Incapsula)
How to read it: "Pardon Our Interruption" (Akamai uses the same phrase, so check the cookies). Also look for a JavaScript stub of about 200 bytes containing _Incapsula_Resource, which passes naive "is this the page?" checks, and for hCaptcha or GeeTest served from /_Incapsula_Resource.
What it checks:
- The exit IP. Cookies are tied to it; changing the User-Agent didn't matter.
- The exit's country, which decides the challenge type. On a genealogy site, EU and Japanese exits got a JS challenge and US, Canadian and British exits got hCaptcha.
- Datacenter IPs failed.
What worked:
- Harvest in Chrome, replay on the same exit. Real Chrome over ISP proxies passed in about 2 seconds. Then replay over curl_cffi from a pool of (cookie, exit) pairs, each used at most about 18 times with at least a second between uses.
- Copy the site's own headers for its JSON API. Capture them from the browser's network traffic. That gave 5,000 out of 5,000 at concurrency 20, with a median of 0.64 s.
- Set the site's language cookie before the first request, so you aren't geo-redirected into an hCaptcha.
Trap: sometimes a whole site switches to GeeTest for everyone. During one such wave, solving the GeeTest didn't help (still 403), and residential proxies were no better. Waiting it out was the only fix.
F5 Shape
The hardest one for us, because there's no cookie to reuse. The detector may not name Shape: it lives in obfuscated JavaScript rather than in cookies or headers. So if a plain request gets refused and nothing else shows up, look for the six prefixed headers on the site's own XHR calls, a JavaScript VM of about 400 KB, and 403 responses that come back in 20-30 ms.
What it checks:
- The total volume across all your sessions, not your IP or identity.
- A per-request signature. Every protected call carries fresh encrypted telemetry (the
-aheader alone is 1.6-3.6 KB).
On an airline the first ~50 calls went through almost 100%, then errors climbed to about 35%. How long each rate held:
| Searches per minute | Held for |
|---|---|
| ~6 | ~3.5 hours |
| 12-16 | 25-40 minutes |
| ~50 | ~10 minutes |
Recovery took 45 minutes to 3 hours of quiet.
What worked:
- Render in a real browser and capture the page's own XHR response. Replaying a captured call failed first time in about 7 sessions out of 8.
- Use an adaptive budget. Halve the rate whenever refusals reach 15%; that held 99% at 10 per minute for an hour.
- Rest the exits. 10 minutes after each search, 60 minutes after a refusal.
- Use deep links instead of filling in the search form.
- Make one protected call per page. A second protected call from the same page with no UI action in between was rejected 0/26.
On this one site, real Chrome over CDP was rejected while a patched Firefox passed, the opposite of most other vendors.
Trap: Playwright Firefox's all_headers() includes your proxy-authorization header. Strip it before replaying anything.
Kasada
We have less first-hand data on Kasada than on the others, so here's only what we measured. A real browser without stealth JavaScript injection did much better than patched setups: on a hotel chain, removing the overrides took Edge from 0/6 to 5/6. Start there before anything more exotic.
Cheat sheet
| Vendor | Mainly checks | Cheapest path that worked | Cookie replay? |
|---|---|---|---|
| Cloudflare | TLS fingerprint, per-IP rate | Rotate curl_cffi profiles, else browser solve + replay | Yes: same IP + UA, short-lived |
| DataDome | Exit reputation, device, engine | Browser solve + replay, capped per exit | Yes: same IP + TLS family |
| HUMAN | Engine, TLS identity + request shape | curl_cffi with the right identity + same-site Referer | Often not needed |
| Akamai | Engine on fresh sessions, per-exit budget | Rolling cookie jar, refreshed every ~90 s | Yes: keep the jar updated, one request at a time |
| AWS WAF | A missing token | Solve the CAPTCHA once, reuse the token | Yes: not tied to IP |
| Imperva | IP-bound cookies, exit country | Chrome harvest + replay on the same exit | Yes: same IP, ~18 uses |
| F5 Shape | Total volume, per-request signing | Real browser, capture its own XHR, adaptive rate | No |
| Kasada | Browser environment | Real browser, no stealth JS | Not tested enough |
What didn't work
- undetected-chromedriver. It got Press & Hold where real Chrome passed.
- Stealth plugins that override browser properties. They lowered pass rates.
- CAPTCHA token services on interactive Turnstile or Press & Hold. A token isn't trust.
- More residential proxies when the block is on your engine. You just burn gigabytes.
- Trusting
solved=true, status 200 or a cookie's existence. Only the content counts.
If you take one thing from this: identify the vendor first, then test the layers one at a time. Don't stack every trick at once. The Anti-Bot Detector covers the first part in a few seconds.
A note on tools: curl_cffi is MIT-licensed. If you drive real Chrome with nodriver or zendriver, both are AGPL-3.0, which matters if you ship them inside a product.
And the obvious part: stick to public pages, don't scrape behind logins, and keep your rates sane. The per-IP limits above are what keeps you unblocked, and they keep the load on the site reasonable too.
This article was written with AI assistance and checked against our own measurements.