Web scraping in 2026 is an arms race. Websites deploy increasingly sophisticated anti-bot systems, while scrapers develop new techniques to appear human. This guide covers everything you need to know to scrape reliably without getting blocked.

Understanding Anti-Bot Defenses

Modern websites use multiple layers of protection. Understanding each layer helps you build a strategy that addresses all of them.

Layer 1: Rate Limiting

The simplest defense. If you make 100 requests per second from one IP, you’re obviously not human. Rate limits are typically:

• Aggressive: 30-60 requests/minute (Google, Amazon)
• Moderate: 120-300 requests/minute (most e-commerce)
• Relaxed: 600+ requests/minute (static content sites)

Layer 2: IP Reputation

Anti-bot services maintain databases of “known bad” IPs. Datacenter IPs (AWS, GCP, DigitalOcean) are flagged immediately. Residential IPs are trusted by default.

IP Type	Trust Level	Cost	Speed
Datacenter	❌ Low	$2/month	Fast
Residential	✅ High	$5-15/GB	Variable
Mobile	✅ Very High	$15-30/GB	Slow
ISP (Static Residential)	✅ High	$3-8/IP/month	Fast

Layer 3: Browser Fingerprinting

This is where most scrapers fail. Even with rotating IPs and realistic headers, websites can detect automation through:

• Navigator properties — Headless Chrome has telltale differences (navigator.webdriver = true)
• Canvas/WebGL rendering — Identical fingerprints across requests = detected
• JavaScript execution — Bots don’t scroll, don’t move the mouse, don’t trigger hover events
• TLS fingerprint (JA3/JA4) — The SSL handshake itself reveals the client type
• HTTP/2 fingerprint — Frame ordering and settings differ between real browsers and HTTP libraries

Layer 4: Behavioral Analysis

The most sophisticated layer. AI models analyze:

• Click patterns (too regular = bot)
• Page navigation flow (going directly to product pages without browsing = suspicious)
• Session duration (too short or too long)
• Interaction timing (humans have natural variance; bots don’t)

The Anti-Bot Ecosystem

Service	Used By	Detection Level
Cloudflare	~20% of websites	Medium-High
Akamai Bot Manager	Enterprise sites	High
PerimeterX (HUMAN)	E-commerce	Very High
DataDome	Luxury/ticketing	Very High
reCAPTCHA v3	Widespread	Medium
hCaptcha	Privacy-focused sites	Medium

Proven Techniques for Unblocked Scraping

1. Use Real Browsers, Not HTTP Libraries

The single most impactful change you can make. Tools like requests (Python) or axios (Node.js) send HTTP requests that look nothing like a real browser.

Instead, use:

• Playwright — Microsoft’s browser automation library
• Puppeteer — Google’s Chrome automation
• Veilus + VeilusFlow — Record your scraping workflow visually, export to Playwright

// Bad: HTTP library (easily detected)
const response = await fetch('https://target.com/products');

// Good: Real browser (much harder to detect)
const browser = await playwright.chromium.launch();
const page = await browser.newPage();
await page.goto('https://target.com/products');
const data = await page.content();

2. Rotate Fingerprints, Not Just IPs

Most scrapers rotate proxies but use the same browser fingerprint for every request. This is like wearing the same unique outfit to every store while changing your car — the stores still recognize you.

Each scraping session needs:

• A unique canvas fingerprint
• Matching WebGL parameters
• Consistent navigator properties (don’t mix Windows UA with Mac fonts)
• Realistic screen resolution for the supposed device

This is exactly what anti-detect browsers like Veilus do — each profile gets a unique, internally consistent fingerprint.

3. Residential Proxies are Non-Negotiable

For any site with serious anti-bot protection, datacenter IPs are immediately flagged. You need residential proxies.

Recommended providers:

• Bright Data — Largest network, most reliable, expensive
• Smartproxy — Good balance of quality and price
• IPRoyal — Budget-friendly residential
• Oxylabs — Enterprise-grade

Pro tip: Use sticky sessions (same IP for the entire browsing session) rather than rotating on every request. Real users don’t change IP every 30 seconds.

4. Mimic Human Behavior

// Bad: Robot-like precision
await page.click('#add-to-cart');
await page.click('#checkout');

// Good: Human-like behavior
await page.mouse.move(randomX(), randomY()); // random movement
await sleep(random(500, 1500)); // natural pause
await page.click('#add-to-cart');
await sleep(random(2000, 4000)); // "thinking" time
await page.scroll(0, random(200, 400)); // scroll like a human
await page.click('#checkout');

Key behaviors to simulate:

• Random delays between actions (800ms-3s for clicks, 2-5s between page loads)
• Mouse movement before clicking (humans don’t teleport the cursor)
• Scrolling through content (don’t jump directly to the target element)
• Page dwell time (spend 10-30 seconds per page, not 0.5 seconds)

5. Manage Cookies and Sessions

Anti-bot systems track session behavior. A session that:

• Has no cookies → suspicious (everyone has cookies)
• Ignores Set-Cookie headers → definitely a bot
• Never accesses CSS/JS resources → headless browser detection

Solution: Use a real browser profile that maintains cookies, localStorage, and cache across sessions. Anti-detect browsers do this automatically.

6. Handle CAPTCHAs Gracefully

When you do encounter CAPTCHAs:

1. Slow down — CAPTCHAs often mean you’ve triggered a threshold
2. Switch fingerprint + IP — The current identity is flagged
3. Use solving services as a last resort (2Captcha, Anti-Captcha)
4. Wait and retry — Some CAPTCHAs are temporary rate-limit responses

Architecture for Scale

For serious scraping operations (10,000+ pages/day):

                    ┌─── Profile 1 (Fingerprint A + Proxy A)
                    │
Job Queue ──────────┼─── Profile 2 (Fingerprint B + Proxy B)
(URLs to scrape)    │
                    ├─── Profile 3 (Fingerprint C + Proxy C)
                    │
                    └─── Profile N (Fingerprint N + Proxy N)
                              │
                              ▼
                        Data Pipeline
                    (clean → store → export)

Key principles:

• Pool management — Rotate profiles after N requests or M minutes
• Error handling — If a profile gets CAPTCHAs, retire it and use a fresh one
• Rate limiting — Self-impose limits (1-3 requests/minute per profile is safe for most sites)
• Retry logic — Exponential backoff on failures

Tools We Recommend

Tool	Use Case	Price
Veilus	Multi-profile management + automation	Free (5 profiles)
Playwright	Browser automation scripting	Free
Bright Data	Residential proxies	From $5/GB
Scrapy	Large-scale structured scraping	Free
2Captcha	CAPTCHA solving (last resort)	From $2.99/1000

Common Mistakes

1. Using headless mode — Many anti-bot systems detect headless browsers. Use headed mode with a virtual display if needed.
2. Ignoring TLS fingerprints — Your JA3 hash reveals the client. Real Chrome has a specific TLS fingerprint that libraries don’t match.
3. Same user-agent for all requests — Rotate UAs, but keep them consistent within a session.
4. Scraping logged-in pages without session management — Cookies and auth tokens need careful handling.
5. Not respecting robots.txt — It won’t block you technically, but it can have legal implications.

Legal Considerations

Disclaimer: This guide is for educational purposes. Always check the website’s Terms of Service and applicable laws in your jurisdiction. Scraping publicly available data is generally legal (see hiQ v. LinkedIn), but scraping behind login walls or ignoring explicit restrictions may not be.

---

Need to manage multiple scraping profiles? Try Veilus free — 5 profiles with unique fingerprints, built-in automation, no credit card.