AI-powered web scraping API that converts websites to clean markdown for LLM consumption. Route self-hosted Firecrawl requests through KnoxProxy as an upstream proxy to bypass IP blocks and access geo-restricted content at scale.
Deploy Firecrawl self-hosted via Docker or sign up for the cloud API.
git clone https://github.com/mendableai/firecrawl
cd firecrawl && docker compose up -dSet KnoxProxy as the upstream proxy in Firecrawl environment variables.
PROXY_SERVER=http://USER:PASS@gw.knoxproxy.com:7000Install the Python or JavaScript SDK for your application.
pip install firecrawl-py
# or: npm install @mendable/firecrawl-jsCreate a Firecrawl client pointing to your self-hosted instance or cloud API.
Modify the proxy username for country-specific crawling.
PROXY_SERVER=http://USER-country-uk:PASS@gw.knoxproxy.com:7000Use the scrape or crawl endpoints to convert web pages to markdown through KnoxProxy.
import os
from firecrawl import FirecrawlApp
# Option 1: Self-hosted with KnoxProxy as upstream proxy
# Set in docker-compose.yml or .env:
# PROXY_SERVER=http://USER:PASS@gw.knoxproxy.com:7000
# Option 2: Use Firecrawl API with proxied requests via SDK
app = FirecrawlApp(
api_key="fc-YOUR_API_KEY",
# For self-hosted:
# api_url="http://localhost:3002",
)
# Scrape a single page to markdown
result = app.scrape_url(
"https://example.com/pricing",
params={
"formats": ["markdown", "html"],
"waitFor": 3000,
},
)
print(result["markdown"][:500])
# Crawl an entire site
crawl_result = app.crawl_url(
"https://example.com",
params={
"limit": 50,
"scrapeOptions": {
"formats": ["markdown"],
},
},
poll_interval=5,
)
for page in crawl_result.get("data", []):
print(f"Crawled: {page['metadata']['url']}")
print(f"Content: {page['markdown'][:200]}\n")
# ---
# Self-hosted docker-compose.yml proxy config:
# services:
# firecrawl:
# environment:
# - PROXY_SERVER=http://USER:PASS@gw.knoxproxy.com:7000
# # Country targeting:
# # - PROXY_SERVER=http://USER-country-us:PASS@gw.knoxproxy.com:7000Each page request through Firecrawl gets a fresh IP via KnoxProxy. For sticky sessions during a site crawl, use USER-session-{crawl_id} in the PROXY_SERVER username.
| Problem | Fix |
|---|---|
| Firecrawl returns empty markdown | Firecrawl uses Playwright internally. Ensure the proxy is set at the environment level so Playwright browser traffic is also proxied. |
| CrawlError: 407 Proxy Authentication Required | Verify the PROXY_SERVER environment variable format: http://USER:PASS@gw.knoxproxy.com:7000 |
| CrawlError: Connection timeout after 30s | Try a residential proxy by adding -type-residential to the username. Increase waitFor in scrape options. |
USER-country-de-city-berlin-session-profile07Order matters -- geo flags before the session flag. The session name is free text; use the profile ID so the mapping is self-documenting. Password stays as issued; no flags belong there. HTTP on :7000, SOCKS5 on :7001, same credentials.
Firecrawl cloud uses its own proxy infrastructure. KnoxProxy is most useful with self-hosted Firecrawl where you control the outbound proxy.
Yes. Change the PROXY_SERVER env var per crawl job, or run multiple Firecrawl instances with different proxy configs.
Firecrawl converts pages to clean markdown for LLMs automatically. Scrapy gives you raw HTML. Firecrawl is better for AI/RAG use cases.
Firecrawl uses Playwright which supports SOCKS5. Set PROXY_SERVER to socks5://USER:PASS@gw.knoxproxy.com:7001
Free trial on rotating residential -- 5 minutes setup, no credit card.