The essential points from this guide -- each one is explained in detail below.
Scrapy's built-in HttpProxyMiddleware reads HTTP_PROXY and HTTPS_PROXY environment variables automatically.
Set request.meta['proxy'] to assign a proxy to individual requests within a spider.
Custom downloader middleware gives you full control over proxy assignment, rotation, and error handling.
scrapy-rotating-proxies and other community middleware simplify pool-based proxy rotation.
Scrapy includes HttpProxyMiddleware enabled by default. It reads the HTTP_PROXY and HTTPS_PROXY environment variables and applies them to all requests. This is the zero-code approach for routing all spider traffic through a single proxy.
export HTTP_PROXY="http://USER:PASS@gw.knoxproxy.com:7000"
export HTTPS_PROXY="http://USER:PASS@gw.knoxproxy.com:7000"
scrapy crawl my_spiderWith these variables set, every request in every spider routes through KnoxProxy's gateway. The backconnect gateway assigns a different exit IP per request, so you get automatic IP rotation without any Scrapy-specific rotation logic. This approach works well for single-provider setups where all traffic should use the same proxy.
For more control, set the proxy directly on individual requests using the meta dictionary. This lets you assign different proxies to different requests within the same spider, which is useful for geo-targeting or mixing proxy providers.
import scrapy
class MySpider(scrapy.Spider):
name = 'my_spider'
def start_requests(self):
urls = ['https://example.com/1', 'https://example.com/2']
for url in urls:
yield scrapy.Request(
url,
callback=self.parse,
meta={
'proxy': 'http://USER:PASS@gw.knoxproxy.com:7000',
},
)
def parse(self, response):
self.logger.info(f'Status {response.status} from {response.url}')
# Parse responseThe meta proxy setting overrides any environment variable proxy for that specific request. When using a backconnect gateway, you do not need to vary the proxy URL -- each connection to the gateway gets a different exit IP automatically.
A custom downloader middleware gives you full control over proxy assignment, rotation logic, and error handling. This is the recommended approach for production scrapers that need to handle proxy failures, implement cooldowns, or select proxies based on request characteristics.
# middlewares.py
import random
class KnoxProxyMiddleware:
def __init__(self):
self.proxy_url = 'http://USER:PASS@gw.knoxproxy.com:7000'
self.failed_count = 0
self.max_retries = 3
@classmethod
def from_crawler(cls, crawler):
return cls()
def process_request(self, request, spider):
request.meta['proxy'] = self.proxy_url
request.meta['download_timeout'] = 30
def process_response(self, request, response, spider):
if response.status in (403, 429, 503):
retries = request.meta.get('proxy_retry_count', 0)
if retries < self.max_retries:
request.meta['proxy_retry_count'] = retries + 1
return request # Retry with new IP from gateway
return response
def process_exception(self, request, exception, spider):
retries = request.meta.get('proxy_retry_count', 0)
if retries < self.max_retries:
request.meta['proxy_retry_count'] = retries + 1
return requestEnable the middleware in settings.py by adding it to DOWNLOADER_MIDDLEWARES with a priority number lower than the default HttpProxyMiddleware (750). A priority of 350 ensures your middleware runs first and sets the proxy before the default middleware processes it.
Optimize Scrapy's settings for proxy-based scraping to balance throughput, respect target site limits, and handle the added latency of proxied connections. Key settings include concurrency, download delay, timeout, and retry configuration.
# settings.py
CONCURRENT_REQUESTS = 16
CONCURRENT_REQUESTS_PER_DOMAIN = 8
DOWNLOAD_DELAY = 1 # seconds between requests to same domain
DOWNLOAD_TIMEOUT = 30 # proxy connections may be slower
RETRY_TIMES = 3
RETRY_HTTP_CODES = [403, 429, 500, 502, 503]
DOWNLOADER_MIDDLEWARES = {
'myproject.middlewares.KnoxProxyMiddleware': 350,
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': None, # Disable default
}
# Disable cookies to prevent session tracking
COOKIES_ENABLED = False
# Rotate user agents
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'Disabling the default HttpProxyMiddleware prevents conflicts with your custom middleware. Set DOWNLOAD_DELAY to at least 1 second per domain to avoid triggering rate limits, even with rotating IPs. The backconnect gateway handles IP rotation, but request timing patterns can still reveal automated behavior.
Ready to put this into practice? Browse Residential Proxies
KnoxProxy Research Team · Technical Content
Network engineers and proxy infrastructure specialists with 10+ years in anti-bot systems, web scraping, and IP routing.
90.4M+ ethically sourced residential IPs across 195 countries. Start free -- no credit card required.