robots.txt is a plain text file placed at the root of a website (example.com/robots.txt) that tells web crawlers which pages or sections they are allowed or disallowed from accessing. It follows the Robots Exclusion Protocol.
Crawlers check for robots.txt before accessing a site. The file contains directives like User-agent (which crawler the rules apply to), Disallow (paths to skip), Allow (exceptions to disallow rules), and Crawl-delay (wait time between requests). Well-behaved crawlers follow these rules voluntarily. The file does not enforce access restrictions; it is a request, not a block.
Handle it deliberately in production scrapers -- most breakage traces back to skipping this step.
USER-country-de-session-task01Add this string to your scraper's proxy credentials and every request in the job shares one exit IP, which keeps robots.txt-related behavior consistent across the run. Change "task01" per worker to isolate parallel scrapes.
Isolate the logic for this step so every scraper in the project shares one tested implementation.
Sites change layouts and behavior over time -- recheck this part of the scraper on a schedule, not just at launch.
This works best over residential or ISP IPs, so the target sees ordinary browsing rather than clustered datacenter traffic.
Capture what actually failed so a broken selector or a new status code surfaces instead of getting masked by automatic retries.
A news website sets Disallow: /paywall/ in robots.txt to tell search engine crawlers not to index their premium subscriber content.
Respecting robots.txt is a baseline ethical requirement for web scraping. Ignoring it can lead to your IPs being blocked and may create legal complications.
robots.txt is a voluntary protocol, not a legal contract. However, courts have considered it in cases involving unauthorized access. Following robots.txt demonstrates good faith and responsible data collection practices.
No. robots.txt rules are based on the User-Agent header, not IP addresses. A website cannot use robots.txt to block specific IPs. IP-based blocking is done at the server or firewall level.
Ready to put this into practice? Ethical Scraping Guide
Start a free trial and test with real targets -- no credit card, no sales call.