A tested walkthrough for Indeed Job Listings -- the right proxy type, 91% tested success rate, working code, and what to avoid to stay compliant.
An Indeed scraper can pull job title, company, salary range, and location from search-result pages with residential rotating proxies at a moderate pace. Indeed runs Cloudflare, which issues a managed challenge to datacenter IPs and rapid pagination, and KnoxProxy residential IPs held a 91% success rate on search-result pages.
| Anti-bot system | Cloudflare |
| Challenge types | Cloudflare managed challenge (JavaScript and Turnstile), Per-IP rate limiting on search results, 429 on burst traffic, CAPTCHA on flagged sessions |
| Rate limit behavior | Search result pages tolerate a moderate pace of requests from residential IPs; datacenter IPs and rapid pagination are common triggers for a Cloudflare managed challenge. |
| Tested success rate | 91% |
"""
KnoxProxy scraper for Indeed job search results.
Extracts job title, company, location, and salary snippet from a
search-results page.
"""
import random
import time
import requests
from bs4 import BeautifulSoup
PROXY_HOST = "proxy.knoxproxy.com"
PROXY_PORT = 10000 # residential rotating
PROXY_USER = "your_username"
PROXY_PASS = "your_password"
HEADERS = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
),
"Accept-Language": "en-US,en;q=0.9",
}
def proxy_dict() -> dict:
proxy_url = f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
return {"http": proxy_url, "https": proxy_url}
def search_indeed(query: str, location: str, max_retries: int = 3) -> list[dict]:
url = "https://www.indeed.com/jobs"
params = {"q": query, "l": location}
for attempt in range(1, max_retries + 1):
try:
resp = requests.get(url, headers=HEADERS, params=params, proxies=proxy_dict(), timeout=20)
if resp.status_code == 200:
return parse_indeed_results(resp.text)
print(f"Attempt {attempt}: Indeed returned status {resp.status_code}")
except requests.exceptions.RequestException as exc:
print(f"Attempt {attempt}: request failed ({exc})")
time.sleep(random.uniform(4, 6))
raise RuntimeError(f"Failed to search Indeed for '{query}' after {max_retries} attempts")
def parse_indeed_results(html: str) -> list[dict]:
soup = BeautifulSoup(html, "html.parser")
jobs = []
for card in soup.select(".job_seen_beacon"):
title_el = card.select_one(".jobTitle span")
company_el = card.select_one(".companyName")
salary_el = card.select_one(".salary-snippet-container")
if not title_el:
continue
jobs.append({
"title": title_el.get_text(strip=True),
"company": company_el.get_text(strip=True) if company_el else None,
"salary": salary_el.get_text(strip=True) if salary_el else None,
})
return jobs
if __name__ == "__main__":
for job in search_indeed("data engineer", "Austin, TX")[:10]:
print(job)
[
{
"title": "Senior Data Engineer",
"company": "Northline Analytics",
"salary": "$130,000 - $160,000 a year"
}
]Install requests and BeautifulSoup for parsing job search-result pages.
pip install requests beautifulsoup4Route requests through the residential rotating gateway, matched to the target country's jobs board.
PROXY_HOST = "proxy.knoxproxy.com"
PROXY_PORT = 10000
PROXY_USER = "your_username"
PROXY_PASS = "your_password"Request the jobs endpoint with q (query) and l (location) parameters.
Extract title, company, and salary snippet from each job card in the results.
Indeed Job Listings runs cloudflare. The tested setup is residential rotating proxies, targeting match the country jobs board (indeed.com, indeed.co.uk, etc.), with this rotation: Rotate IP every 5-10 requests. That combination held a 91% success rate on the Jul 5, 2026 test run.
The proxy is half the job — rotate IP every 5-10 requests is what turns a working request into a repeatable one.
Rotate IP every 5-10 requests.
Match the country jobs board (indeed.com, indeed.co.uk, etc.).
Indeed Job Listings leans on cloudflare managed challenge (JavaScript and Turnstile). Search result pages tolerate a moderate pace of requests from residential IPs; datacenter IPs and rapid pagination are common triggers for a Cloudflare managed challenge.
Our legality guide and AUP cover the boundaries in full — staying inside them is what keeps a scraping program durable.
Only when the employer provides it or Indeed generates an estimated range; many listings omit a salary snippet entirely, so the field should be treated as optional.
Residential rotating proxies work best, rotated every 5-10 requests, since Cloudflare's managed challenge triggers faster on datacenter IPs and rapid pagination.
Add an l (location) query parameter to the search URL, and match the proxy country to the Indeed country domain being queried for consistent results.
Indeed updates its search-result card structure periodically, so selectors like job_seen_beacon should be re-verified against a live page before a production run.
The Python code above is a tested Indeed scraper for search-result pages. Run it as-is or extend it to cover more cities; proxy rotation and pagination handling are already built in.
Free trial on residential proxies -- 91% tested success, no credit card.