ClipleteBot

ClipleteBot is the crawler behind Cliplete, a recruiting platform that helps families of high-school athletes reach college coaching staffs. It identifies itself with this User-Agent:

ClipleteBot/1.0 (+https://cliplete.com/about/crawler)

What it reads

Public coaching-staff directory pages on college athletics websites: coaches' names, titles and the sport they coach, plus contact links the page itself publishes. It only fetches pages anyone can open without logging in, and it uses them to notice when a school's coaching staff changes.

It never adds, removes or edits coaches in our directory. When a page differs from our directory, it saves the names and titles that differ in an internal log. The only thing it updates is a note that a coach we already list is still on the page.

How it behaves

  • It checks robots.txt before every request, including each redirect it follows, and obeys the rules for ClipleteBot (or for * if there are none).
  • If your robots.txt can't be read (a timeout, a server error, or a refusal such as 403), it doesn't crawl your site. A missing robots.txt (404) counts as no restrictions, per the standard.
  • It never makes more than one request every 2 seconds to a site, and it honors a longer Crawl-delay (up to 30 seconds).
  • Each visit tries at most 6 pages for a school, follows at most 5 redirects, gives up after 10 seconds, and reads at most 2 MB of a page.
  • Automatic visits go only to schools that families on Cliplete have chosen as targets. The automatic run is scheduled once a day and visits a few schools each time. Our staff can also start that run by hand. Once a visit to a school ends, it won't automatically visit that school's staff pages again for 30 days (7 days when the visit ended in November through February, around the signing periods). A visit that runs out of time is finished on a later run, which may read some of the same pages again. Our staff can also start a visit to any school by hand.

Opting out

To stop ClipleteBot crawling your site, add this to your robots.txt:

User-agent: ClipleteBot
Disallow: /

You can also disallow just the paths you'd rather it skip. It re-reads robots.txt at least every six hours, so changes take effect within about six hours.

Questions

Email support@cliplete.com.