ProveBot, our web page fetcher

Who we are

ProveBot is the program that Prove My Language uses to fetch a web page when a customer asks us to review it. We are the same company that runs this site. You can write to us at [email protected].

What it fetches

ProveBot fetches only an address that a signed-in customer has typed into the review form, and only that one page. It does not crawl. It does not follow the links on the page, it does not look for other pages on your site, and it does not fetch images, scripts or style sheets. It makes one request, plus at most four more if the page redirects.

Each request is an ordinary GET for the page. It sends no cookies, no login details and no referring address, and it identifies itself with this header:

User-Agent: ProveBot/1.0 (+https://provelang.com/bot)

It connects only to public internet addresses, on the usual web ports (80 and 443), and it checks the security certificate of every secure site. It gives up after 20 seconds and reads at most 5 MB of a page.

What happens to the page

We read the text of the page and show it to the customer who asked, who then decides whether to go ahead. We do not keep the address of the page. The text is held only while the review runs (encrypted, with a key kept only in memory) and is deleted when the job ends. See Security and privacy.

How to block ProveBot

Configure your web server or firewall to refuse requests whose User-Agent header contains the word ProveBot. The customer is then told that the page could not be fetched. ProveBot does not read robots.txt, because it never fetches a page that nobody asked it to fetch; it works for one person at a time, like a browser.

If something looks wrong, for example you see requests that you did not expect, write to [email protected] and tell us the time and the page that was fetched.