Common Crawl

About

Common Crawl is a non-profit company that maintains a free, open source repository of web crawl data that can be used by anyone.

Detail

Website

https://commoncrawl.org/

Agents

CCBot

15% blocked

CCBot is a web crawler used by Common Crawl to maintainin an open source repository of web crawl data that is available for anyone to use. This repository has been used to train many LLMs (Large Language Models), including OpenAI's GPTs.

AI Data Scraper

See More →