Songkick's robots.txt has a few special rules.
songkick.com
Songkick's robots.txt has a few special rules.
1–3 of 3 posts
Re: Songkick's robots.txt has a few special rules.
#2It's interesting they're disallowing 008. I work at a popular webhost, and we've had problems with this crawler too (It's distributed, and can crawl things a little too fast).
Re: Songkick's robots.txt has a few special rules.
#3It's interesting they're disallowing 008. I work at a popular webhost, and we've had problems with this crawler too (It's distributed, and can crawl things a little too fast).
Did you notice the bottom 3 rules? Maybe I should resubmit and change the title...