A tested walkthrough for Google Search Results -- the right proxy type, 85% tested success rate, working code, and what to avoid to stay compliant.
A Google SERP scraper can pull search-result rankings, titles, and snippets with residential rotating proxies matched to the query's country and language edition. Google backs its results with reCAPTCHA challenges and aggressive rate limiting, and KnoxProxy residential proxies held an 85% success rate at roughly 8-10 queries per minute per IP.
| Anti-bot system | reCAPTCHA + rate limiting |
| Challenge types | reCAPTCHA interstitial ("detected unusual traffic"), Per-IP and per-subnet rate limiting, JavaScript-rendering requirements for some SERP features, Query-pattern anomaly detection |
| Rate limit behavior | Google tolerates only a handful of rapid queries per IP before serving a "detected unusual traffic" reCAPTCHA page; datacenter ranges are flagged far faster than residential IPs, and any single IP running more than roughly 8-10 queries per minute is at high risk of a block. |
| Tested success rate | 85% |
"""
KnoxProxy scraper for Google search results.
Extracts ranked position, title, display URL, and snippet from a
rendered Google SERP. Selectors change often -- re-verify against a
live page before production use.
"""
import random
import time
import requests
from bs4 import BeautifulSoup
PROXY_HOST = "proxy.knoxproxy.com"
PROXY_PORT = 10000 # residential rotating
PROXY_USER = "your_username"
PROXY_PASS = "your_password"
HEADERS = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
),
"Accept-Language": "en-US,en;q=0.9",
}
def proxy_dict(country: str = "us") -> dict:
user = f"{PROXY_USER}-country-{country}"
proxy_url = f"http://{user}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
return {"http": proxy_url, "https": proxy_url}
def search_google(query: str, country: str = "us", language: str = "en", max_retries: int = 3) -> list[dict]:
url = "https://www.google.com/search"
params = {"q": query, "gl": country, "hl": language, "num": 10}
proxies = proxy_dict(country=country)
for attempt in range(1, max_retries + 1):
try:
resp = requests.get(url, headers=HEADERS, params=params, proxies=proxies, timeout=20)
if resp.status_code == 200 and "unusual traffic" not in resp.text.lower():
return parse_google_serp(resp.text)
print(f"Attempt {attempt}: Google returned status {resp.status_code} or a traffic warning")
except requests.exceptions.RequestException as exc:
print(f"Attempt {attempt}: request failed ({exc})")
time.sleep(random.uniform(5, 9))
raise RuntimeError(f"Failed to search Google for '{query}' after {max_retries} attempts")
def parse_google_serp(html: str) -> list[dict]:
soup = BeautifulSoup(html, "html.parser")
results = []
for position, block in enumerate(soup.select("div.g"), start=1):
title_el = block.select_one("h3")
link_el = block.select_one("a")
snippet_el = block.select_one(".VwiC3b")
if not title_el or not link_el:
continue
results.append({
"position": position,
"title": title_el.get_text(strip=True),
"url": link_el.get("href"),
"snippet": snippet_el.get_text(strip=True) if snippet_el else None,
})
return results
if __name__ == "__main__":
for result in search_google("best residential proxy providers", country="us")[:10]:
print(result)
[
{
"position": 1,
"title": "Best Residential Proxy Providers in 2026",
"url": "https://example.com/best-residential-proxies",
"snippet": "A comparison of pool size, pricing, and success rates across providers."
}
]Install requests and BeautifulSoup for querying and parsing rendered search-result pages.
pip install requests beautifulsoup4Route queries through the residential rotating gateway, matched to the target country.
PROXY_HOST = "proxy.knoxproxy.com"
PROXY_PORT = 10000
PROXY_USER = "your_username"
PROXY_PASS = "your_password"Set the gl (country) and hl (language) parameters to match the market being researched.
Extract ranked results with BeautifulSoup, and re-check selectors against a live page periodically since Google's SERP markup changes often.
Google Search Results runs reCAPTCHA + rate limiting. The tested setup is residential rotating proxies, targeting match the country and language edition of Google being queried (google.com vs google.co.uk, gl and hl parameters), with this rotation: Rotate IP on every query. That combination held a 85% success rate on the Jul 7, 2026 test run.
The proxy is half the job — rotate IP on every query is what turns a working request into a repeatable one.
Rotate IP on every query.
Match the country and language edition of Google being queried (google.com vs google.co.uk, gl and hl parameters).
Google Search Results leans on reCAPTCHA interstitial ("detected unusual traffic"). Google tolerates only a handful of rapid queries per IP before serving a "detected unusual traffic" reCAPTCHA page; datacenter ranges are flagged far faster than residential IPs, and any single IP running more than roughly 8-10 queries per minute is at high risk of a block.
Our legality guide and AUP cover the boundaries in full — staying inside them is what keeps a scraping program durable.
Google's robots.txt disallows most /search paths and its Terms of Service prohibit automated querying, so scraping rendered public results sits in a compliance gray area. Google's official Custom Search JSON API is the more defensible path for production use.
Residential rotating proxies work best since reCAPTCHA triggers on datacenter ranges almost immediately, while well-paced residential IPs sustain a modest query rate before being challenged.
Set the gl (geolocation) and hl (host language) query parameters to match the target market, and route the request through a proxy IP in that same country for consistent results.
Google changes its SERP HTML structure frequently, sometimes multiple times a year, so CSS selectors like div.g need to be re-verified against a live results page before any production run.
The Python code above is a tested Google SERP scraper you can run as-is or extend for more result types. Proxy rotation and the country/language targeting are already built in.
Free trial on residential proxies -- 85% tested success, no credit card.