Autonomous AI research agent that generates comprehensive reports from web sources. Route GPT Researcher web requests through KnoxProxy to access global sources without IP blocks and retrieve geo-specific research data.
Install the gpt-researcher package and its dependencies.
pip install gpt-researcherConfigure your OpenAI or other LLM provider API key.
export OPENAI_API_KEY="sk-..."Set HTTP proxy environment variables so all web requests route through KnoxProxy.
export HTTP_PROXY="http://USER:PASS@gw.knoxproxy.com:7000"
export HTTPS_PROXY="http://USER:PASS@gw.knoxproxy.com:7000"Modify the proxy username in the env var for country-specific research.
export HTTPS_PROXY="http://USER-country-us:PASS@gw.knoxproxy.com:7000"Write a Python script that initializes GPTResearcher with a query and runs the research pipeline.
Execute the research script and review the generated report. Verify sources were accessed through the proxy.
python research_task.pyimport asyncio
import os
from gpt_researcher import GPTResearcher
# Configure KnoxProxy for all web requests
os.environ["HTTP_PROXY"] = "http://USER:PASS@gw.knoxproxy.com:7000"
os.environ["HTTPS_PROXY"] = "http://USER:PASS@gw.knoxproxy.com:7000"
# For geo-targeted research (e.g., research from UK perspective):
# os.environ["HTTPS_PROXY"] = "http://USER-country-gb:PASS@gw.knoxproxy.com:7000"
# Optional: bypass proxy for LLM API calls
os.environ["NO_PROXY"] = "api.openai.com,api.anthropic.com"
async def main():
# Initialize researcher with a query
researcher = GPTResearcher(
query="What are the latest trends in eSIM adoption worldwide?",
report_type="research_report",
)
# Conduct research (web searches + scraping through KnoxProxy)
await researcher.conduct_research()
# Generate the final report
report = await researcher.write_report()
print(report)
# Access individual sources
sources = researcher.get_source_urls()
print(f"\nResearch used {len(sources)} sources:")
for url in sources:
print(f" - {url}")
if __name__ == "__main__":
asyncio.run(main())Each web request during research gets a fresh IP. GPT Researcher makes many requests per run, and IP rotation helps avoid rate limits. For consistent identity during a research session, use USER-session-{research_id} as the proxy username.
| Problem | Fix |
|---|---|
| aiohttp.ClientProxyConnectionError: Cannot connect to proxy | Verify the proxy URL format. Test with: curl -x $HTTPS_PROXY https://httpbin.org/ip |
| Research returns no sources | Try residential proxies with USER-type-residential. Set NO_PROXY for search API endpoints if using direct API keys. |
| OpenAI API calls timing out | Add api.openai.com (or your LLM provider) to NO_PROXY environment variable. |
USER-country-de-city-berlin-session-profile07Order matters -- geo flags before the session flag. The session name is free text; use the profile ID so the mapping is self-documenting. Password stays as issued; no flags belong there. HTTP on :7000, SOCKS5 on :7001, same credentials.
Minimal impact. KnoxProxy adds ~50-100ms latency per request. Research quality improves because fewer requests are blocked.
Yes. GPT Researcher supports Ollama and other local LLMs. The proxy only affects web scraping requests, not local model inference.
Set HTTPS_PROXY with a country code (e.g., USER-country-de). Search results and scraped content will reflect that geography.
Yes. All sub-agents inherit the same proxy environment variables. Each agent request gets its own rotated IP.
Free trial on rotating residential -- 5 minutes setup, no credit card.