LeadKestrelBot

If you found LeadKestrelBot in your server logs, this page explains what it is, what it does and how to stop it.

What it is

LeadKestrelBot is the crawler behind LeadKestrel, a service that checks whether the ways customers contact a business (contact forms, quote requests, WhatsApp and phone links) are actually working. It is run by Lynsco Group Limited, an Irish company (registration number 570133). It identifies itself with this user agent:

LeadKestrelBot/1.0 (+https://leadkestrel.com/bot)

What it does

• Only reads public pages. It makes plain GET requests over HTTP or HTTPS. It does not log in, click buttons, solve CAPTCHAs or fill in or submit any form.
• Keeps a small footprint. A check starts at one address, reads that site's robots.txt, and then looks at a handful of pages linked from it, such as the contact page. It does not crawl a whole site. Each request has a 10-second time limit and a 5 MB size limit, and it follows at most 5 redirects.
• Loads at most a couple of pages in a headless browser, only to see pages that are built with JavaScript. Images, video and fonts are not requested during that step.
• Respects robots.txt. After the first page, it checks your robots.txt before fetching any page it found by following a link, and skips anything you have disallowed for LeadKestrelBot (or for all crawlers).
• Never touches private networks. It refuses to connect to internal or private addresses.

How to stop it

Add this to your robots.txt to keep it off the rest of your site:

User-agent: LeadKestrelBot
Disallow: /

The first page of a check is the address that was entered, so a single request to it can still happen before robots.txt is read. To make sure your domain is not checked at all, email support@leadkestrel.com with the domain name and we will stop checking it.

Seen a "LeadKestrel Automated Test" enquiry instead?

That is a different thing from this crawler. A business that uses LeadKestrel can ask it to send a clearly labelled test enquiry through its own contact form to confirm the enquiry really arrives. We only do this for a website whose owner has proved they own it, never for a site that has not been verified, and the message is named "LeadKestrel Automated Test" so it is easy to recognise and ignore.

Contact

Questions or concerns about the crawler: support@leadkestrel.com.