A tested walkthrough for Twitter/X Posts -- the right proxy type, 77% tested success rate, working code, and what to avoid to stay compliant.
A Twitter/X scraper can reach individual post text, like count, and retweet count without login through the public syndication endpoint, using residential rotating proxies. X requires login for most browsing and applies tight per-IP rate limits plus Arkose Labs challenges on flagged sessions, and KnoxProxy residential IPs held a 77% success rate on the syndication endpoint.
| Anti-bot system | Login wall + rate limiting + Arkose Labs bot detection |
| Challenge types | Login wall for most timeline and search views, Arkose Labs FunCaptcha on flagged sessions, Aggressive per-IP and per-account rate limits, Guest-token validation on public endpoints |
| Rate limit behavior | Since 2023, X requires a login for most browsing, and even the syndication endpoint used for embedding single posts applies tight per-IP rate limits; exceeding a modest request rate returns an HTTP 429 or a rendered rate-limit page. |
| Tested success rate | 77% |
"""
KnoxProxy scraper for individual X (Twitter) posts.
Uses the public syndication endpoint designed for embedding tweets,
which does not require a logged-in session.
"""
import random
import time
import requests
PROXY_HOST = "proxy.knoxproxy.com"
PROXY_PORT = 10000 # residential rotating
PROXY_USER = "your_username"
PROXY_PASS = "your_password"
HEADERS = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
),
"Accept": "application/json",
}
def proxy_dict() -> dict:
proxy_url = f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
return {"http": proxy_url, "https": proxy_url}
def fetch_tweet(tweet_id: str, max_retries: int = 3) -> dict:
url = "https://cdn.syndication.twimg.com/tweet-result"
params = {"id": tweet_id, "lang": "en"}
for attempt in range(1, max_retries + 1):
try:
resp = requests.get(url, headers=HEADERS, params=params, proxies=proxy_dict(), timeout=20)
if resp.status_code == 200:
return parse_tweet(resp.json(), tweet_id)
if resp.status_code == 429:
print(f"Attempt {attempt}: rate limited, rotating IP before retry")
else:
print(f"Attempt {attempt}: X returned status {resp.status_code}")
except requests.exceptions.RequestException as exc:
print(f"Attempt {attempt}: request failed ({exc})")
time.sleep(random.uniform(4, 8))
raise RuntimeError(f"Failed to fetch tweet {tweet_id} after {max_retries} attempts")
def parse_tweet(payload: dict, tweet_id: str) -> dict:
return {
"tweet_id": tweet_id,
"text": payload.get("text"),
"author": payload.get("user", {}).get("screen_name"),
"like_count": payload.get("favorite_count"),
"retweet_count": payload.get("conversation_count"),
}
if __name__ == "__main__":
for tweet_id in ["1743456789012345678"]:
print(fetch_tweet(tweet_id))
time.sleep(random.uniform(4, 8))
{
"tweet_id": "1743456789012345678",
"text": "shipping a small update to the dashboard today",
"author": "examplehandle",
"like_count": 1420,
"retweet_count": 96
}Install requests for calling the public syndication endpoint used to embed individual posts.
pip install requestsRoute requests through the residential rotating gateway to reduce 429 rate-limit responses.
PROXY_HOST = "proxy.knoxproxy.com"
PROXY_PORT = 10000
PROXY_USER = "your_username"
PROXY_PASS = "your_password"Call the syndication endpoint with a post ID to retrieve text and engagement counts without logging in.
Back off and rotate IP whenever a rate-limit response is returned rather than retrying immediately.
Twitter/X Posts runs login wall + rate limiting + Arkose Labs bot detection. The tested setup is residential rotating proxies, targeting no specific geo requirement unless testing region-locked content, with this rotation: Rotate IP on every request. That combination held a 77% success rate on the Jun 30, 2026 test run.
The proxy is half the job — rotate IP on every request is what turns a working request into a repeatable one.
Rotate IP on every request.
No specific geo requirement unless testing region-locked content.
Twitter/X Posts leans on login wall for most timeline and search views. Since 2023, X requires a login for most browsing, and even the syndication endpoint used for embedding single posts applies tight per-IP rate limits; exceeding a modest request rate returns an HTTP 429 or a rendered rate-limit page.
Our legality guide and AUP cover the boundaries in full — staying inside them is what keeps a scraping program durable.
A limited amount of data is available anonymously through the public syndication endpoint used for embedding individual posts. Most timeline, search, and profile browsing now requires a login, which is outside anonymous public-data collection.
Residential rotating proxies work best since X's rate limiting is applied aggressively per IP, and datacenter ranges hit 429 responses noticeably faster.
X introduced a login requirement for most browsing in 2023 and tightened rate limits significantly, closing off much of what was previously reachable through anonymous HTML scraping.
It works for posts from public, non-protected accounts; posts from protected accounts or deleted posts return an error regardless of proxy quality.
The Python code above is a tested X (Twitter) scraper for the public syndication endpoint. Run it as-is for individual posts; proxy rotation is already handled, though timeline and search browsing need a login regardless of tooling.
Free trial on residential proxies -- 77% tested success, no credit card.