crawl-test. A test page designed for validating web crawlers, scrapers, and content extraction algorithms. Built with modern web standards including semantic HTML5, JSON-LD structured data, and proper meta tags.

crawltest.com

Vaya's read on this project

Problem, audience, market, and the verdict — sign in to see it.

Updates

June 2025
  • Update README.md
  • Readme: updated language on JS enablement
  • Assets: removed unused boilerplate
  • Layout: improve responsiveness
  • Merge pull request #1 from CameronWhiteside/update_worker_name_to_cra…
  • Update wrangler config name to crawltest
  • Build: fixed deps
  • Initial commit: CrawlTest.com - AI crawler testing site with semantic…
  • Initialize web application via create-cloudflare CLI