FeeUpdatesBot
FeeUpdatesBot reads public fee announcements so that changes can be recorded here. This page says what it reads, how often and how to stop it.
User agent
FeeUpdatesBot/1.0 (+https://feeupdates.com/bot; hello@feeupdates.com)
What it reads
- Only public fee pages, announcements, RSS feeds and sitemaps.
- It does not sign in, submit forms or read anything behind a login.
- It does not download PDF, XLSX and other documents. Those are linked, not fetched.
How often
- Each page once a day.
- If a request fails, at most 3 tries on the same day, at least 30 minutes apart.
- It follows robots.txt, including Crawl-delay.
- It sends conditional requests (ETag and If-Modified-Since), when the server supports it, so an unchanged page is not downloaded again.
What it keeps
A hash of each line of the page, and a short excerpt of the lines that changed. For feeds and sitemaps, it also keeps the item titles and links. The full page is not stored.
What is published
The numbers, a quote of two sentences or less, a link to the source and the date it was checked. Each entry is checked by a person before it is published.
UPS
UPS pages are not fetched by the bot. A person checks them once a week.
How to stop it
Add this to your robots.txt. It takes effect within 24 hours.
User-agent: FeeUpdatesBot
Disallow: /Or write to hello@feeupdates.com with the site you want excluded.
Follow the changes
The RSS feed has the 50 most recently published changes. Each platform page has its own feed.
Weekly email: one message on Fridays with the week's changes. Coming in November.