ClipleteBot
ClipleteBot is the crawler behind Cliplete, a recruiting platform that helps families of high-school athletes reach college coaching staffs. It identifies itself with this User-Agent:
ClipleteBot/1.0 (+https://cliplete.com/about/crawler)What it reads
Public coaching-staff directory pages on college athletics websites: coaches' names, titles and the sport they coach, plus contact links the page itself publishes. It only fetches pages anyone can open without logging in, and it uses them to notice when a school's coaching staff changes.
It never adds, removes or edits coaches in our directory. When a page differs from our directory, it saves the names and titles that differ in an internal log. The only thing it updates is a note that a coach we already list is still on the page.
How it behaves
- It checks
robots.txtbefore every request, including each redirect it follows, and obeys the rules forClipleteBot(or for*if there are none). - If your
robots.txtcan't be read (a timeout, a server error, or a refusal such as 403), it doesn't crawl your site. A missingrobots.txt(404) counts as no restrictions, per the standard. - It never makes more than one request every 2 seconds to a site, and it honors a longer
Crawl-delay(up to 30 seconds). - Each visit tries at most 6 pages for a school, follows at most 5 redirects, gives up after 10 seconds, and reads at most 2 MB of a page.
- Automatic visits go only to schools that families on Cliplete have chosen as targets. The automatic run is scheduled once a day and visits a few schools each time. Our staff can also start that run by hand. Once a visit to a school ends, it won't automatically visit that school's staff pages again for 30 days (7 days when the visit ended in November through February, around the signing periods). A visit that runs out of time is finished on a later run, which may read some of the same pages again. Our staff can also start a visit to any school by hand.
Opting out
To stop ClipleteBot crawling your site, add this to your robots.txt:
User-agent: ClipleteBot
Disallow: /You can also disallow just the paths you'd rather it skip. It re-reads robots.txt at least every six hours, so changes take effect within about six hours.
Questions
Email support@cliplete.com.