FAST-WebCrawler

Last seen referenced today.

What is FAST-WebCrawler?

About

FAST-WebCrawler is an uncategorized agent. It's not currently known to be artificially intelligent or AI-related. If you think that's incorrect or can provide more detail about its purpose, please contact us.

Type

Uncategorized

Not currently assigned a type

Expected Behavior

Behavior will vary depending on whether this agent is a search engine crawler, data scraper, archiver, one-off fetcher, etc.

Insights

Other Websites

of top websites are currently blocking FAST-WebCrawler in some way

Learn How →

Access Control

Should I Block FAST-WebCrawler?

It's difficult to say without a type. Its purposes could either be good or bad for your website, depending on what it is.

Using Robots.txt

User Agent Token	Description
`FAST-WebCrawler`	Should match instances of FAST-WebCrawler

You can block FAST-WebCrawler or limit its access by setting user agent token rules in your website's robots.txt.

# robots.txt
# This blocks FAST-WebCrawler

User-agent: FAST-WebCrawler
Disallow: /

Instead of doing this manually, you can generate your robots.txt automatically using the free API or Wordpress plugin.

Get the Robots.txt