Skip to the content
Visifluent

Blog

How the Visifluent audit crawler treats your website

What VisifluentAuditBot does when it visits a site, how much it reads, which rules it obeys and how to keep it out with two lines in robots.txt.

  • By Visifluent
  • Published
  • 2 min read
  • crawler
  • robots.txt
  • transparency

When someone audits a website with Visifluent, a crawler visits it. If you run a website you may find that crawler in your server log, and you are entitled to know what it is, what it does and how to stop it. This post says so in plain terms. The crawler's own page, at visifluent.com/bot, says the same.

It says who it is

Every request carries a user agent that starts with VisifluentAuditBot and ends with the address of the crawler's page. It does not pretend to be a browser or another search engine. If you see it in your log, that page tells you what it is.

How much it reads

An audit reads up to 5,000 pages of one website, and checks the links and images on them. It never makes more than 5 requests at a time to any one website, so it should not slow your site down for your visitors.

A link or image that points to another website gets a short status request, only to see whether it works, and only if that other website's robots.txt allows it.

What it follows, and what it leaves alone

It follows the redirects your site sends and the sitemap your site offers, because those are the paths you have chosen to publish. It reads public pages only. It never signs in, so anything behind a login is out of reach for it.

It does not read private or internal network addresses, whatever a page links to.

It obeys robots.txt

The crawler reads your robots.txt at the start of every audit and follows it, including a group written for its name. That means you can change the rules at any time and the next audit will respect them.

To keep it away from a whole website, add these two lines:

User-agent: VisifluentAuditBot
Disallow: /

To keep it out of one folder only, change the path:

User-agent: VisifluentAuditBot
Disallow: /private/

Everything else on your site stays open to it.

If something looks wrong

If you think the crawler behaved in a way this post does not describe, write to info@visifluent.com. We would rather hear about a problem than have you block the crawler without knowing why. Blocking it is always your right, and the two lines above take effect at its next visit.

If you are curious what the audit is for, our guide to an on-page SEO checklist covers most of what it looks at.

More from the blog

  • 5 min read

    How to read Google Search Console: the numbers that matter

    A plain guide to the Performance report in Google Search Console: what clicks, impressions, click-through rate and position mean, and a ten-minute weekly routine.

    • guide
    • search console
    • google
    • reporting
  • 6 min read

    A practical on-page SEO checklist for a small website

    Eight checks, in the order to do them, that make a page easy for Google to find and for people to choose: indexing, titles, headings, links, speed and more.

    • guide
    • on-page seo
    • checklist
    • audit