wheyproofbot
wheyproofbot is the crawler behind WheyProof. It reads public product prices from supplement stores so we can show each product's price history and send shoppers to the store that sells it. It also reads Shopify stores for our sister sites, which run on the same server: Protein Avcısı (supplement stores in Turkey) and CoffeeProof (coffee and tea prices in the US, the UK and the EU, coffeeproof.org).
What it fetches
Public pages only: a store's product catalog (on Shopify stores, /products.json), its home page to confirm the currency, and individual product pages for ratings and nutrition facts. It never logs in and never touches carts, checkouts or accounts.
How often
A supplement store's catalog about four times a day, a coffee or tea store's once a day: one request at a time per store, with a pause between pages.
How it identifies itself
- User-Agent
Mozilla/5.0 (compatible; wheyproofbot/1.0; +https://www.wheyproof.com/bot)- Web Bot Auth (HTTP Message Signatures) Signature-Agent
https://api.wheyproof.com- Public key directory
- https://api.wheyproof.com/.well-known/http-message-signatures-directory
Every request it sends to a store carries this User-Agent and is signed with Web Bot Auth, which a store can check against the key directory above.
Crawl-delay and 429 responses
If a store's robots.txt sets a Crawl-delay for wheyproofbot (or for every bot, with *), we leave at least that many seconds between two requests to that store. If it asks for more than five minutes, we don't read the store at all rather than read it faster.
If a store answers 429 (Too Many Requests), we stop asking it for the rest of that round. Only a catalog page is tried once more, after the time in the Retry-After header (a minute when there is none), and not at all if that is longer than five minutes.
robots.txt and how to block it
wheyproofbot reads each store's robots.txt at least once a day and follows the group named wheyproofbot, or the * group if there is none. To stop it reading your store:
User-agent: wheyproofbot
Disallow: /What we keep
Prices, product names, sizes, stock status, images and nutrition facts, stored on our own server and shown on our sites with a link back to the store. We don't sell or pass on the data.
Contact
To ask us to stop or slow down, or to report a problem, write to [email protected].