A tested walkthrough for Reddit Posts & Comments -- the right proxy type, 96% tested success rate, working code, and what to avoid to stay compliant.
Reddit posts and comments are directly available as JSON through the public .json endpoints, and datacenter rotating proxies handle typical volume without issue. Reddit runs Cloudflare, which mainly challenges bursty or poorly identified traffic, and KnoxProxy datacenter IPs held a 96% success rate with a descriptive User-Agent. A Reddit scraper built on the .json endpoints is one of the simplest guides in this catalog to get running.
| Anti-bot system | Cloudflare |
| Challenge types | Cloudflare managed challenge (JS and Turnstile) on suspicious traffic, Per-IP rate limiting on the public JSON endpoints, 429 responses on burst traffic, User-Agent based blocking of generic or default agents |
| Rate limit behavior | The public .json endpoints are generous for well-paced traffic with a descriptive User-Agent, but default or blank User-Agents and rapid-fire requests from one IP quickly draw 429 responses or a Cloudflare challenge page. |
| Tested success rate | 96% |
"""
KnoxProxy scraper for public Reddit posts and comments.
Uses the public .json endpoints available on any subreddit or post
URL -- no OAuth application registration required for read-only,
moderate-volume access.
"""
import random
import time
import requests
PROXY_HOST = "proxy.knoxproxy.com"
PROXY_PORT = 9000 # datacenter rotating
PROXY_USER = "your_username"
PROXY_PASS = "your_password"
HEADERS = {
"User-Agent": "knoxproxy-research-bot/1.0 (contact: research@example.com)",
}
def proxy_dict() -> dict:
proxy_url = f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
return {"http": proxy_url, "https": proxy_url}
def fetch_subreddit_posts(subreddit: str, limit: int = 25, max_retries: int = 3) -> list[dict]:
url = f"https://www.reddit.com/r/{subreddit}/.json"
params = {"limit": limit}
for attempt in range(1, max_retries + 1):
try:
resp = requests.get(url, headers=HEADERS, params=params, proxies=proxy_dict(), timeout=20)
if resp.status_code == 200:
return parse_reddit_posts(resp.json())
print(f"Attempt {attempt}: Reddit returned status {resp.status_code}")
except requests.exceptions.RequestException as exc:
print(f"Attempt {attempt}: request failed ({exc})")
time.sleep(random.uniform(2, 4))
raise RuntimeError(f"Failed to fetch r/{subreddit} after {max_retries} attempts")
def parse_reddit_posts(payload: dict) -> list[dict]:
children = payload.get("data", {}).get("children", [])
posts = []
for child in children:
post = child.get("data", {})
posts.append({
"title": post.get("title"),
"author": post.get("author"),
"score": post.get("score"),
"num_comments": post.get("num_comments"),
"permalink": f"https://www.reddit.com{post.get('permalink')}",
})
return posts
if __name__ == "__main__":
for post in fetch_subreddit_posts("webscraping", limit=10):
print(post)
time.sleep(random.uniform(1, 3))
[
{
"title": "What is the best way to handle rotating proxies at scale?",
"author": "data_curious_22",
"score": 184,
"num_comments": 47,
"permalink": "https://www.reddit.com/r/webscraping/comments/1abcxyz/"
}
]Install requests to call Reddit's public JSON endpoints directly; no HTML parser is needed.
pip install requestsReddit blocks generic or default User-Agent strings, so identify your client clearly.
HEADERS = {"User-Agent": "knoxproxy-research-bot/1.0 (contact: research@example.com)"}Datacenter rotating proxies are sufficient for typical Reddit read volume.
PROXY_HOST = "proxy.knoxproxy.com"
PROXY_PORT = 9000 # datacenter rotating
PROXY_USER = "your_username"
PROXY_PASS = "your_password"Append .json to any subreddit or post URL to receive structured post and comment data directly.
Reddit Posts & Comments runs cloudflare. The tested setup is datacenter rotating, residential recommended above a few thousand requests per day proxies, targeting no specific geo needed, with this rotation: Rotate IP every 10-20 requests. That combination held a 96% success rate on the Jul 3, 2026 test run.
The proxy is half the job — rotate IP every 10-20 requests is what turns a working request into a repeatable one.
Rotate IP every 10-20 requests.
No specific geo needed.
Reddit Posts & Comments leans on cloudflare managed challenge (JS and Turnstile) on suspicious traffic. The public .json endpoints are generous for well-paced traffic with a descriptive User-Agent, but default or blank User-Agents and rapid-fire requests from one IP quickly draw 429 responses or a Cloudflare challenge page.
Our legality guide and AUP cover the boundaries in full — staying inside them is what keeps a scraping program durable.
No, the public .json endpoints available on any subreddit or post URL work without OAuth registration for read-only, moderate-volume access, though Reddit's official OAuth API is the recommended path for high-volume commercial use.
Datacenter rotating proxies handle typical read volume well; residential proxies are worth the extra cost mainly above a few thousand requests per day.
A 429 means the current IP has exceeded Reddit's rate limit for the JSON endpoints, usually from requesting too many pages too quickly or using a generic User-Agent string.
Yes, appending .json to any individual post's permalink returns the full comment tree alongside the post data in a single response.
The Python code above is a tested Reddit scraper for the public .json endpoints. Run it as-is or extend it to track more subreddits; proxy rotation and a descriptive User-Agent are already handled.
Free trial on residential proxies -- 96% tested success, no credit card.