The essential points from this guide -- each one is explained in detail below.
Per-request rotation assigns a new IP for every request -- best for independent page fetches.
Time-based rotation holds an IP for a set duration -- ideal for multi-page flows requiring session continuity.
Failure-based rotation only switches IPs after a block or error -- maximizes each IP's useful life.
Hybrid strategies combine per-request rotation with sticky sessions for different parts of the same scraping job.
Gateway-side rotation eliminates client-side rotation code entirely.
Per-request rotation assigns a fresh IP to every HTTP request. This is the default strategy for stateless scraping where each request is independent -- fetching product pages, checking prices, downloading search results. The target site sees each request as coming from a different visitor.
With a backconnect gateway like KnoxProxy, per-request rotation requires zero code. Every connection to the gateway endpoint gets a different exit IP from the residential pool. You point your scraper at the gateway and it handles the rest. For client-side rotation, maintain a proxy list and select randomly:
import random
def get_next_proxy(proxy_list, used_recently=set()):
available = [p for p in proxy_list if p not in used_recently]
if not available:
used_recently.clear()
available = proxy_list
chosen = random.choice(available)
used_recently.add(chosen)
return chosenPer-request rotation uses IPs at the fastest rate, so it requires a large pool (or a provider with millions of IPs) to avoid reusing the same IP on the same target site within a short window.
Time-based rotation holds the same IP for a fixed duration -- typically 1 to 60 minutes -- before switching. This is essential for multi-step workflows: paginating through search results, navigating multi-page forms, maintaining a shopping cart, or scraping content behind a login.
KnoxProxy supports sticky sessions through the username parameter. Append a session ID to hold the same IP for the session's duration:
import time
def get_sticky_proxy(session_id, duration_minutes=10):
"""Same session_id returns same IP for the specified duration."""
return f'http://USER-session-{session_id}-sessTime-{duration_minutes}:PASS@gw.knoxproxy.com:7000'
# All requests in this session use the same exit IP
proxy = get_sticky_proxy('checkout_flow_42', duration_minutes=5)
for step_url in checkout_steps:
response = requests.get(step_url, proxies={'https': proxy})Choose the shortest session duration that covers your workflow. Longer sessions increase the risk of the IP getting blocked mid-flow, and they tie up IPs that could be serving other requests.
Failure-triggered rotation keeps using the same proxy until it fails -- returns a 403, 429, CAPTCHA, or times out. Only then does the scraper switch to a new IP. This strategy maximizes each IP's useful life and reduces proxy consumption compared to per-request rotation.
class FailureTriggeredRotator:
def __init__(self, proxy_list):
self.proxy_list = proxy_list
self.current_index = 0
self.failure_codes = {403, 429, 503}
def get_proxy(self):
return self.proxy_list[self.current_index % len(self.proxy_list)]
def report_result(self, status_code):
if status_code in self.failure_codes:
self.current_index += 1
return True # Rotated
return False
rotator = FailureTriggeredRotator(proxy_list)
for url in urls:
proxy = rotator.get_proxy()
response = requests.get(url, proxies={'https': proxy}, timeout=30)
rotator.report_result(response.status_code)This approach works best with datacenter or ISP proxies where each IP has a cost. For residential backconnect gateways that rotate per-request anyway, failure-triggered rotation adds complexity without saving resources.
Production scrapers rarely use a single rotation strategy. A hybrid approach combines per-request rotation for independent fetches with sticky sessions for multi-step flows, and adds failure-based fallback for error recovery.
For example, an e-commerce price monitor might use per-request rotation for fetching product listing pages (each is independent), switch to a 5-minute sticky session for navigating into product detail pages that require session cookies, and implement failure-triggered rotation that moves to a new sticky session if the current one gets blocked.
The key is matching the strategy to the request type. Categorize your requests: stateless reads use per-request rotation, authenticated flows use sticky sessions, and API endpoints with rate limits use failure-triggered rotation with backoff. Route each category through the appropriate strategy in your middleware layer.
Search engine scraping needs per-request rotation with geo-targeting -- search results vary by location and search engines aggressively block repeated IPs. Use residential proxies with country targeting and rotate every request.
Social media scraping needs sticky sessions of 5-30 minutes because platforms track session consistency. A new IP every request looks suspicious. Use residential proxies with sticky sessions and realistic browsing delays.
Price monitoring needs per-request rotation across many target sites. Speed matters more than stealth since you are making simple GET requests. Datacenter proxies with per-request rotation offer the best cost-per-request ratio.
Sneaker and limited-release purchasing needs the fastest possible proxies with per-request rotation across many checkout attempts. ISP proxies or dedicated datacenter proxies minimize latency while residential proxies provide the best success rates against sophisticated anti-bot systems.
Ready to put this into practice? Browse Rotating Proxies
KnoxProxy Research Team · Technical Content
Network engineers and proxy infrastructure specialists with 10+ years in anti-bot systems, web scraping, and IP routing.
90.4M+ ethically sourced residential IPs across 195 countries. Start free -- no credit card required.