Modern anti-bot systems score three things: IP reputation (is this IP known for bot traffic?), browser fingerprint (do the headers, TLS signature, and JavaScript environment look like a real browser?), and behavioral patterns (is the request rate, navigation pattern, and timing consistent with a human visitor?).
Residential proxy rotation solves the IP signal automatically. Each request arrives from a unique household IP with no bot history. The other two signals require attention from your scraping code.
The most common mistake: sending requests with default library headers. Python's requests library announces itself as "python-requests/2.31.0" in the User-Agent. This is an instant bot signal.
Use realistic browser headers: a current Chrome User-Agent, Accept, Accept-Language, Accept-Encoding, and Connection headers that match what a real browser sends. Rotate User-Agent strings across a small set of recent versions -- do not randomize them, as random combinations are also detectable.
Residential rotation handles IP reputation. Realistic headers handle fingerprinting. Polite pacing handles behavioral detection.
Even with perfect IPs and headers, sending 100 requests per second to a single domain is detectable by rate analysis. The target sees a traffic spike from diverse IPs but with identical timing patterns.
Add realistic delays between requests to the same domain. A uniform random delay of 1-3 seconds between pages on the same site mimics casual browsing. For high-volume scraping, distribute requests across many domains simultaneously rather than hitting one domain fast.
Many targets embed structured data in JSON-LD, data attributes, or internal API endpoints that the page JavaScript calls. Hitting these endpoints directly through the proxy is faster, uses less bandwidth (cheaper), and is more stable than parsing rendered HTML.
Check the Network tab in browser DevTools to find the XHR/fetch calls the page makes. These endpoints often return clean JSON that is easier to parse than HTML and less likely to change layout.
Track your success rate per target domain over time. A gradual decline signals that the target has updated its anti-bot rules. When success drops below 95%, investigate: check if your headers are outdated, if the target has started using a new anti-bot vendor, or if your request pattern has become predictable.
Build a validation basket: a set of known URLs with known content. Check these regularly to confirm your scraper is returning accurate data, not soft blocks (200 responses with CAPTCHA pages instead of content).
import requests
import time
import random
proxy = "http://USER:PASS@gw.knoxproxy.com:7000"
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/126.0.0.0 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.9",
"Accept-Encoding": "gzip, deflate, br",
}
urls = ["https://example.com/page/1", "https://example.com/page/2"]
for url in urls:
r = requests.get(url,
proxies={"https": proxy},
headers=headers)
print(r.status_code, len(r.text))
time.sleep(random.uniform(1, 3)) # polite pacingResidential rotation handles IP reputation. Realistic headers handle fingerprinting. Polite pacing handles behavioral detection. Address all three signals and you will see 99%+ success on mainstream targets.
Start free, run a thousand requests, and log the exit IPs. The difference is measurable in the first batch.