scraping@nytimes.com
What is scraping@nytimes.com?
About
NYTimes.com newsroom scraping bot collects publicly available, non-copyrighted data for journalistic projects including election result tracking, COVID-19 data aggregation, and other news analytics initiatives. You can see how often scraping@nytimes.com visits your website by setting up Dark Visitors agent analytics.
Expected Behavior
The behavior of intelligence gatherers depends on the goals of their clients. For example, a client might be interested in brand sentiment, in which case the agent would crawl related social media or blog posts at a more frequent rate than unrelated websites.
Type
Detail
| Operated By | The New York Times |
| Last Updated | 1 day ago |
Insights
Top Website Robots.txts
Country of Origin
Global Traffic
The percentage of all internet traffic coming from Intelligence Gatherers
Robots.txt
Should I Block scraping@nytimes.com?
Probably not, especially if you benefit from an intelligence gathering service yourself. However, you might choose to block them if you're concerned about things like server resource usage.
How Do I Block scraping@nytimes.com?
You can block scraping@nytimes.com or limit its access by setting user agent token rules in your website's robots.txt. Set up Dark Visitors agent analytics to check whether it's actually following them.
# robots.txt
# This should block scraping@nytimes.com
User-agent: scraping@nytimes.com
Disallow: /