Crawlson

Crawlson is a web crawler bot that indexes content for the Crawlson search engine by systematically browsing and analyzing web pages.