Kagi

kagibot is the web crawler for Kagi, a privacy-focused ad-free search engine. It crawls the web to build Kagi's independent search index to provide high-quality, personalized search results without tracking users.