magpie-crawler

Operated by Brandwatch. Last seen today.

What is magpie-crawler?

About

magpie-crawler is an intelligence gatherer operated by Brandwatch. It's not currently known to be artificially intelligent or AI-related. If you think that's incorrect or can provide more detail about its purpose, please contact us.

Detail

Website https://brandwatch.com/legal/magpie-crawler/

Type

Intelligence Gatherer
Searches for useful insights

Expected Behavior

The behavior of Intelligence Gatherers depends on the goals of their clients. For example, a client might be interested in brand sentiment, in which case the agent would crawl related social media or blog posts at a more frequent rate than unrelated websites.

Access Control

Using Robots.txt

User Agent Token Description
magpie-crawler Should match instances of magpie-crawler

You can block magpie-crawler or limit its access by setting user agent token rules in your website's robots.txt.

# robots.txt
# This blocks magpie-crawler

User-agent: magpie-crawler
Disallow: /

Instead of doing this manually, you can generate your robots.txt automatically using the free API or Wordpress plugin.

Get the Robots.txt

Insights

Other Websites

3%
of top websites are currently blocking magpie-crawler in some way
Updated today