What is Diffbot and how to unblock it?

Diffbot is an automated solution integrated with ADBRO that is used to extract content for analysis and classification. Diffbot works similar to search engine crawlers and fully respects directives from robots.txt and may be blocked by an application. ADBRO Team constantly monitors Diffbot availability for all connected Publishers and may contact you in order to perform required changes.

Unblocking with robots.txt

The following directive should be added to the robots.txt file on your Site:

User-agent: Diffbot
Disallow:

Learn more about robots.txt at The Web Robots Pages: About /robots.txt.