Crawler disclosure
RigyxBot
RigyxBot is our crawler. It reads public pages a customer asked us to analyse — as part of a Prospect Run on their own site, or a competitor's public pages during that same run.
What it fetches
A homepage plus up to 5 linked pages — typically about, contact, legal and team pages. Nothing else.
How politely
- robots.txt is read and honoured.
- At least 1.2 seconds between requests to the same site.
- An 8-second timeout per request.
- Our own fetches are cached for 7 days, so a site isn't re-crawled on every run.
How to block it
Add this to your robots.txt:
User-agent: RigyxBot Disallow: /