A tested walkthrough for Craigslist Listings -- the right proxy type, 94% tested success rate, working code, and what to avoid to stay compliant.
A Craigslist scraper can pull listing titles, prices, and neighborhoods from city-specific search pages with datacenter rotating proxies at a slow, steady pace. Craigslist applies basic rate limiting and temporary IP blocks rather than a sophisticated WAF, and KnoxProxy datacenter IPs held a 94% success rate on listing pages.
| Anti-bot system | Basic rate limiting + IP blocking |
| Challenge types | Per-IP request throttling, Temporary IP ban (403 or blocked page) on burst traffic, Occasional CAPTCHA on the posting and reply flow (not relevant to read-only scraping), City-subdomain based access patterns |
| Rate limit behavior | Craigslist is comparatively permissive for slow, well-paced requests to a single city subdomain; bursts beyond roughly 1 request per second, or hitting many city subdomains rapidly from one IP, lead to a temporary IP-level block. |
| Tested success rate | 94% |
"""
KnoxProxy scraper for Craigslist city search results.
Extracts listing title, price, and neighborhood from a city-specific
search-results page.
"""
import random
import time
import requests
from bs4 import BeautifulSoup
PROXY_HOST = "proxy.knoxproxy.com"
PROXY_PORT = 9000 # datacenter rotating
PROXY_USER = "your_username"
PROXY_PASS = "your_password"
HEADERS = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
),
"Accept-Language": "en-US,en;q=0.9",
}
def proxy_dict(session_id: str) -> dict:
user = f"{PROXY_USER}-session-{session_id}"
proxy_url = f"http://{user}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
return {"http": proxy_url, "https": proxy_url}
def search_craigslist(city: str, query: str, max_retries: int = 3) -> list[dict]:
url = f"https://{city}.craigslist.org/search/sss"
params = {"query": query}
proxies = proxy_dict(session_id=city)
for attempt in range(1, max_retries + 1):
try:
resp = requests.get(url, headers=HEADERS, params=params, proxies=proxies, timeout=20)
if resp.status_code == 200:
return parse_craigslist_results(resp.text)
print(f"Attempt {attempt}: Craigslist returned status {resp.status_code}")
except requests.exceptions.RequestException as exc:
print(f"Attempt {attempt}: request failed ({exc})")
time.sleep(random.uniform(2, 4))
raise RuntimeError(f"Failed to search Craigslist {city} for '{query}' after {max_retries} attempts")
def parse_craigslist_results(html: str) -> list[dict]:
soup = BeautifulSoup(html, "html.parser")
listings = []
for row in soup.select("li.cl-search-result"):
title_el = row.select_one(".titlestring")
price_el = row.select_one(".priceinfo")
hood_el = row.select_one(".result-hood")
if not title_el:
continue
listings.append({
"title": title_el.get_text(strip=True),
"price": price_el.get_text(strip=True) if price_el else None,
"neighborhood": hood_el.get_text(strip=True) if hood_el else None,
})
return listings
if __name__ == "__main__":
for listing in search_craigslist("newyork", "bicycle")[:10]:
print(listing)
[
{
"title": "Trek Mountain Bike 29er - Large Frame",
"price": "$340",
"neighborhood": "(Park Slope)"
}
]Install requests and BeautifulSoup for parsing city search-result pages.
pip install requests beautifulsoup4Datacenter rotating proxies are sufficient for typical Craigslist workloads.
PROXY_HOST = "proxy.knoxproxy.com"
PROXY_PORT = 9000 # datacenter rotating
PROXY_USER = "your_username"
PROXY_PASS = "your_password"Request the search endpoint on the target city's subdomain with a query parameter.
Extract title, price, and neighborhood from each result row.
Craigslist Listings runs basic rate limiting + IP blocking. The tested setup is datacenter rotating, residential optional proxies, targeting match the city subdomain being scraped, with this rotation: Rotate IP every 10-20 requests or per city. That combination held a 94% success rate on the Jul 6, 2026 test run.
The proxy is half the job — rotate IP every 10-20 requests or per city is what turns a working request into a repeatable one.
Rotate IP every 10-20 requests or per city.
Match the city subdomain being scraped.
Craigslist Listings leans on per-IP request throttling. Craigslist is comparatively permissive for slow, well-paced requests to a single city subdomain; bursts beyond roughly 1 request per second, or hitting many city subdomains rapidly from one IP, lead to a temporary IP-level block.
Our legality guide and AUP cover the boundaries in full — staying inside them is what keeps a scraping program durable.
Craigslist's Terms of Use explicitly restrict automated collection and Craigslist has taken enforcement action against scrapers historically, so even though the technical protections are comparatively light, the compliance risk is not.
Datacenter rotating proxies are usually sufficient given Craigslist's comparatively basic rate limiting; residential proxies are only needed for larger multi-city crawls.
Each city has its own subdomain, such as newyork.craigslist.org, so a multi-city crawl needs to iterate over subdomains rather than a single global search.
Rarely; most listings show only a neighborhood or general area rather than a precise address, particularly for housing and services categories.
The Python code above is a tested Craigslist scraper for a single city. Run it as-is or extend it to loop through more subdomains for a multi-city crawl; proxy rotation is already handled.
Free trial on residential proxies -- 94% tested success, no credit card.