Rare find

crawl-stop. A test page designed to validate web crawlers' compliance with restrictions and anti-scraping measures. Features rate limiting, access controls, and crawling deterrents.

crawlstop.com

Vaya's read on this project

Problem, audience, market, and the verdict — sign in to see it.

Updates

July 2025
  • Public headers: updated to allow any frame ancestors
June 2025
  • Headers: allowed cross origin and iframe
  • Styling: fixed page alignment
  • Robots.txt: do not reference managed policy inline
  • Robots.txt: clarified Cloudflare use
  • Robots.txt: clarified Cloudflare use
  • Robots.txt: clarified Cloudflare use
  • Styling: improved layout and responsiveness
  • Crawl stop: added base site UI and images