Spider. A tool that visits websites and collects all the links it finds.

thenewboston.com

Vaya's read on this project

Problem, audience, market, and the verdict — sign in to see it.

Updates

May 2016
  • Fixed file path bug
  • Merge pull request #23 from keatinge/master
April 2016
  • Changed delete_file_contents to be more readable and pythonic
  • Merge pull request #22 from srinath29/master
March 2016
  • used join function in the os.path to make it more pythonic
  • The set_to_file is changed and made more efficient. Previously, there…
  • Merge remote-tracking branch 'origin/master'
  • Improved URL checking logic (thanks to yjhao)
  • Merge pull request #17 from HTMLlama/master
  • Fixing getheader('Content-Type') value to simplify code.
  • Cleaning up code, will work on previously committed argument parsing …
  • Started adding changes manually. Have to pick up my sister from schoo…
  • Reverted back to original spider because argument parsing with f'ing …
  • Added documentation
  • Merge pull request #13 from BraxtonFair/patch-3
  • Remove redundant code
  • Merge pull request #12 from BraxtonFair/patch-2
  • Reverse docopt implementation
February 2016
  • Merge pull request #10 from BraxtonFair/patch-1
  • Merge pull request #9 from lwgray/twitter