Training corpora need petabyte-scale coverage without concentrating traffic on a handful of IPs that get rate-limited or banned. Datacenter proxies handle the bulk of open-web volume at $0.02/IP; residential covers the protected sources that block bulk collection outright.
Use datacenter proxies for the bulk of corpus collection -- open-web sources, government data, and permissive archives need throughput, not stealth. Route protected or paywalled sources through rotating residential, and reserve mobile for app-exclusive content. Blend the three to hit petabyte scale without one target's block list stalling the whole crawl.
| Expected success | 99%+ on open-web sources, 95%+ on protected |
| Throughput | Unlimited concurrency across all plans |
| Strategy | Hybrid routing -- match proxy type to source protection level |
| Cost fit | Blended $0.15-$2.10/GB depending on source mix |
import requests
dc_proxy = "http://USER:PASS@dc.knoxproxy.com:8000"res_proxy = "http://USER:PASS@gw.knoxproxy.com:7000"
def fetch(url, protected=False): proxy = res_proxy if protected else dc_proxy r = requests.get(url, proxies={"https": proxy}, timeout=15) return r.text if r.status_code == 200 else None
sources = load_source_manifest() # each entry flags protected=True/Falsefor src in sources: text = fetch(src["url"], protected=src["protected"]) if text: write_to_corpus(src["domain"], text)Corpus collection must stay within our public-data-only acceptable-use policy. Collect only publicly accessible pages, respect robots.txt and rate limits, and exclude sources that explicitly prohibit automated collection in their terms.
A crawl that stalls on the sources with the strongest anti-bot defenses produces a corpus biased toward whatever is easiest to reach. Datacenter proxies clear the majority of the open web that does not block automated traffic; residential rotation opens the protected long tail that would otherwise be entirely absent from the dataset.
The only reliable way to see what a real user sees is to become one.
Scheduler, proxy fetch, parser, store -- the proxy is one line in the fetch step. Everything else is pipeline you already run.
Tag each source as permissive or protected before the first fetch. Route permissive sources to datacenter and protected ones to residential -- guessing wrong wastes both cost and time.
When a datacenter-routed source drops below 90% success, move it to residential rather than retrying. Persistent failures usually mean the target upgraded its detection, not a transient block.
Near-duplicate pages -- pagination, tracking parameters, mirrors -- inflate storage and skew training weight. Hash normalized content before writing to the corpus, not after.
Failed fetches are never billed, so your effective cost tracks the success rate you actually observe.
Datacenter for the bulk of open, permissive sources -- it is the cheapest way to hit petabyte scale. Route protected and paywalled sources through rotating residential, since datacenter IPs get blocked on anti-bot-hardened domains.
Blended cost typically lands between $0.15 and $2.10 per GB depending on how much of the corpus comes from protected sources. Classifying sources correctly before crawling is the biggest lever on total cost.
Yes, within our public-data-only acceptable-use policy -- collect only publicly accessible pages, respect robots.txt, and exclude sources whose terms explicitly prohibit automated collection.
Route protected and paywalled sources through residential rotation instead of dropping them from the crawl. A datacenter-only crawl systematically excludes exactly the sources with the strongest defenses.
Free trial on rotating residential -- city targeting included, no credit card.