Ritalin Agent Tools

RitalinBot

RitalinBot is the web crawler for Ritalin Agent Tools. It reads a website’s sitemaps to keep track of which pages a site has and when they change. It doesn’t follow links, fill in forms, run scripts or log in.

User agent

RitalinBot sends this user agent with every request:

Mozilla/5.0 (compatible; RitalinBot/0.1; +https://ritalin.agnt.tools/robots) Ritalin Agent Tools site map reader

The robots.txt token is RitalinBot. The version number changes as the crawler is updated.

What it fetches

  1. Your robots.txt, to find your sitemaps and the rules you set for it.
  2. The sitemaps that robots.txt names, including sitemap indexes and gzipped sitemaps. If robots.txt doesn’t name any, it tries the usual addresses, such as /sitemap.xml.
  3. The pages listed in those sitemaps, so it can tell when a page’s content changes.

It only reads pages your sitemaps list, and only on the site those sitemaps belong to.

How it behaves

  • It obeys robots.txt: Disallow, Allow and Crawl-delay, following the robots.txt standard (RFC 9309).
  • One request at a time. It never sends two requests to your site at once, and waits at least 1 second between requests.
  • Crawl-delay is honoured up to 10 seconds. A longer Crawl-delay is treated as 10 seconds.
  • It backs off when asked. On a 429 Too Many Requests or 503 Service Unavailable, it waits for your Retry-After time (up to 60 seconds) and tries once more. It then waits twice as long between requests for the rest of that visit.
  • If robots.txt can’t be read because your server returns an error (5xx) or doesn’t answer, it fetches no pages from your site on that visit. A missing robots.txt (404) means no rules apply.
  • It asks before downloading again. Requests carry If-None-Match and If-Modified-Since, so your server can answer 304 Not Modified for unchanged pages.
  • It visits rarely. It normally visits once a day, overnight in Australia. A page whose sitemap date hasn’t changed is downloaded again only about once a week.
  • It stops early. Each response is cut off at 5 MB, each request times out after 20 seconds, and a visit ends after 25 failed requests in a row.

Limiting or blocking RitalinBot

To keep RitalinBot out of your whole site, add this to your robots.txt:

User-agent: RitalinBot
Disallow: /

To keep it out of one part of your site:

User-agent: RitalinBot
Disallow: /private/

To slow it down, set a delay in seconds (up to 10):

User-agent: RitalinBot
Crawl-delay: 5

RitalinBot reads robots.txt at the start of each visit, so a change takes effect on its next visit, within a day.

Contact

If RitalinBot is causing a problem on your site, email [email protected]. Tell us your domain and roughly when you saw the requests.