nrobots. The Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website which is otherwise publicly viewable. This project provides an easy-to-use class, implemented in C#, to work with robots.txt files.

github.com/sjdirect/nrobots

Vaya's read on this project

Problem, audience, market, and the verdict — sign in to see it.

Updates

No recent activity.