The crawler behind Visifluent
VisifluentAuditBot
VisifluentAuditBot is the crawler behind Visifluent, a tool that audits websites for search engine optimization. It reads a website only because someone asked Visifluent to audit it, or to look for links between its pages. It does not index the web, and it does not collect pages for any other purpose.
- Pages per audit
- 1000
- at most, from one website
- At a time
- 5
- requests to any one website, at most
- robots.txt
- Read first
- before it reads any page
- Sign-in
- Never
- public pages only
What it does
It says who it is
It identifies itself in every request with the user agent
VisifluentAuditBot/0.1 (+https://visifluent.com/bot).It reads a few pages, a few at a time
It reads up to 1000 pages of the one website per audit and checks the links and images on them, never more than 5 requests at a time to any one website. A link or image on another website gets a short status request, and only once that website's robots.txt allows it.
It follows what you publish
It follows the redirects and the sitemap the website itself offers. It reads only public pages and never signs in.
It stays out of private networks
It does not read private or internal network addresses, whatever a page links to.
Keeping it out
It obeys robots.txt, including a group written for VisifluentAuditBot. To keep it away from a whole website, or part of one, add:
User-agent: VisifluentAuditBot Disallow: /
Change / to a folder to block only that folder.
For example, Disallow: /private/ keeps it out of that folder and nothing else.
It reads robots.txt at the start of every audit, so a change counts from its next visit.
Contact
If it visits too often, use robots.txt as above: it takes effect at its next visit.