Popular LLM application framework for building chains and agents. Route LangChain web retrieval requests through KnoxProxy to load web content for RAG pipelines, research agents, and knowledge-grounded LLM applications without IP blocks.
Install LangChain with community loaders and HTTP libraries.
pip install langchain langchain-community requests beautifulsoup4Configure HTTP proxy environment variables so LangChain web loaders route through KnoxProxy.
export HTTP_PROXY="http://USER:PASS@gw.knoxproxy.com:7000"
export HTTPS_PROXY="http://USER:PASS@gw.knoxproxy.com:7000"LangChain WebBaseLoader respects the requests session. Pass a custom session with proxy settings.
Modify the proxy username for country-specific content retrieval.
export HTTPS_PROXY="http://USER-country-us:PASS@gw.knoxproxy.com:7000"Combine the proxied loader with embeddings and a vector store for a full RAG pipeline.
Run the chain and verify that web content is loaded through the proxy by checking the source IP in logs.
import requests
from langchain_community.document_loaders import WebBaseLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
PROXY_URL = "http://USER:PASS@gw.knoxproxy.com:7000"
# Create a requests session with KnoxProxy
session = requests.Session()
session.proxies = {
"http": PROXY_URL,
"https": PROXY_URL,
}
session.headers.update({
"User-Agent": "Mozilla/5.0 (compatible; ResearchBot/1.0)",
})
# Load web pages through KnoxProxy
loader = WebBaseLoader(
web_paths=[
"https://en.wikipedia.org/wiki/Large_language_model",
"https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
],
requests_per_second=2,
session=session,
)
documents = loader.load()
# Split for RAG pipeline
splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200,
)
chunks = splitter.split_documents(documents)
print(f"Loaded {len(documents)} pages, split into {len(chunks)} chunks")
# Use with any LangChain chain:
# vectorstore = FAISS.from_documents(chunks, embeddings)
# qa_chain = RetrievalQA.from_chain_type(llm, retriever=vectorstore.as_retriever())Each WebBaseLoader.load() call creates new connections with fresh IPs. For sticky sessions across a multi-page load, use USER-session-{chain_run_id} as the proxy username.
| Problem | Fix |
|---|---|
| requests.exceptions.ProxyError: Cannot connect to proxy | Verify the proxy URL format: http://USER:PASS@gw.knoxproxy.com:7000. Test with curl first. |
| WebBaseLoader returns empty documents | Add a realistic User-Agent header to the session. Try a residential proxy by adding -type-residential to the username. |
| SSL: CERTIFICATE_VERIFY_FAILED | KnoxProxy does not perform SSL interception. This is likely a network issue. Set session.verify = True and check your system CA bundle. |
USER-country-de-city-berlin-session-profile07Order matters -- geo flags before the session flag. The session name is free text; use the profile ID so the mapping is self-documenting. Password stays as issued; no flags belong there. HTTP on :7000, SOCKS5 on :7001, same credentials.
Only if you use HTTP_PROXY environment variables. The session-based approach only proxies WebBaseLoader requests, leaving LLM API calls direct.
Yes. Set the proxy on aiohttp or httpx client config. AsyncHtmlLoader accepts a custom session via aiohttp.ClientSession(proxy=proxy_url).
Yes. Add cookies or auth headers to the requests session. KnoxProxy passes all headers through transparently.
Set the proxy username to USER-country-{cc} (e.g., USER-country-fr for France) to get an IP from that country.
Free trial on rotating residential -- 5 minutes setup, no credit card.