The essential points from this guide -- each one is explained in detail below.
The requests library accepts a proxies dict with http and https keys pointing to your gateway URL.
httpx supports both sync and async proxy configuration through a single proxies parameter.
aiohttp uses a proxy parameter on each request or a connector-level proxy for session-wide config.
Environment variables HTTP_PROXY and HTTPS_PROXY work with requests, httpx, and most Python HTTP clients automatically.
Session-level proxy config avoids repeating the proxy argument on every request.
The requests library is the most common Python HTTP client. Pass a proxies dictionary to any request method, or set it on a Session object for reuse across multiple requests. The dictionary maps protocol schemes to proxy URLs, and the library routes traffic through the specified proxy automatically.
import requests
proxy_url = 'http://USER:PASS@gw.knoxproxy.com:7000'
proxies = {
'http': proxy_url,
'https': proxy_url,
}
# Per-request proxy
response = requests.get('https://httpbin.org/ip', proxies=proxies, timeout=30)
print(response.json())
# Session-level proxy (recommended for multiple requests)
session = requests.Session()
session.proxies.update(proxies)
for url in urls:
r = session.get(url, timeout=30)Session-level configuration is preferred when making many requests because it reuses the underlying TCP connection pool and avoids repeating the proxy argument. The session also handles cookies and headers consistently across requests.
httpx is a modern HTTP client that supports both synchronous and asynchronous usage with the same API surface. It handles proxy authentication, HTTP/2, and connection pooling out of the box. The proxy parameter accepts a URL string or a dictionary mapping schemes to proxy URLs.
import httpx
proxy_url = 'http://USER:PASS@gw.knoxproxy.com:7000'
# Synchronous client
with httpx.Client(proxy=proxy_url) as client:
r = client.get('https://httpbin.org/ip')
print(r.json())
# Async client for high-throughput scraping
async with httpx.AsyncClient(proxy=proxy_url) as client:
r = await client.get('https://httpbin.org/ip')
print(r.json())The async client is ideal for scraping at scale because it can maintain hundreds of concurrent connections without threading overhead. Each request through the backconnect gateway receives a different exit IP automatically.
aiohttp is Python's most established async HTTP client, widely used in scraping frameworks and web applications. It supports proxy configuration at the request level through the proxy parameter, with authentication credentials embedded in the URL.
import aiohttp
import asyncio
async def fetch(url):
proxy = 'http://USER:PASS@gw.knoxproxy.com:7000'
async with aiohttp.ClientSession() as session:
async with session.get(url, proxy=proxy, timeout=aiohttp.ClientTimeout(total=30)) as resp:
return await resp.text()
# Concurrent requests with different exit IPs
async def scrape(urls):
proxy = 'http://USER:PASS@gw.knoxproxy.com:7000'
async with aiohttp.ClientSession() as session:
tasks = [session.get(url, proxy=proxy) for url in urls]
responses = await asyncio.gather(*tasks, return_exceptions=True)
return responsesUnlike httpx, aiohttp requires you to pass the proxy parameter on each request rather than at the session level. For large-scale scraping, wrap the proxy logic in a helper function to keep your code clean.
Most Python HTTP clients -- including requests, httpx, and urllib3 -- automatically read the HTTP_PROXY and HTTPS_PROXY environment variables. This approach keeps credentials out of your source code and makes it easy to switch proxies without modifying the application.
export HTTP_PROXY="http://USER:PASS@gw.knoxproxy.com:7000"
export HTTPS_PROXY="http://USER:PASS@gw.knoxproxy.com:7000"
export NO_PROXY="localhost,127.0.0.1"Once these variables are set, any requests call without an explicit proxies argument will route through the configured proxy. The NO_PROXY variable excludes specific hosts from proxying, which is useful for local services and health check endpoints. In production deployments, store these values in your CI/CD secrets or a .env file loaded by python-dotenv rather than hardcoding them in shell profiles.
Ready to put this into practice? Browse Residential Proxies
KnoxProxy Research Team · Technical Content
Network engineers and proxy infrastructure specialists with 10+ years in anti-bot systems, web scraping, and IP routing.
90.4M+ ethically sourced residential IPs across 195 countries. Start free -- no credit card required.