The irony of the situation is the crawlers specifically allowed are most likely the only ones that bother to abide by robots.txt in the first place. It'll be interesting to see if the change has any impact whatsoever.
Who exactly is crawling my site?
11–20 of 80 posts
Re: Who exactly is crawling my site?
#12"I've decided to block all crawlers to the site other than Google or Bing" And this is why I find people that respond to privacy complaints about Google with "if you want to switch search engines no-one is stopping you" frustrating. Additionally, I would like to point out that according to those numbers, there are ~41,000 (418,814 - 199,725 - 40,359 - 36,340 - 33,893 - 26,325 - 13,458 - 10,657 - 6,109 - 5,993 - 4,959…
I agree completely. Blocking everything but Google and Bing is horrible and extremely short-sighted.
Re: Who exactly is crawling my site?
#13Crawlers and spambots are the scourge of medium to small websites.
I run a small wiki that gets just a few thousand human hits a day. But according to the server logs 90% of server hits are crawlers and spambots, so I'm using 10 times the resources I really need to serve customers.
I finally resorted to blocking entire data centers and companies that crawl constantly but send no traffic.
I feel like search engines should crawl websites in proportion to the traffic they send. For example, Yandex, Baidu, and Bing were all crawling my website hundreds or thousands of times a day, but never sending a single visitor (or in the case of Bing sending single-digit visitors). It's an absurd waste of resources, so I blocked them completely.
Re: Who exactly is crawling my site?
#14I actually did this exact robots.txt for my site this summer. The net effect? A massive loss of secondary market traffic - enough that, after a month, we probably lost 10-15% in revenue. It surprised us. We rolled the old robots.txt back into place and bam - life went back to normal. You can question my pithy 300-character explanation all you want but I'll leave you with this: we were 100% certain that the robots.txt change was the difference.
In the end, we switched to network blocks for the common bots/spiders. Much better.
Re: Who exactly is crawling my site?
#15What about a system like this:
You crawl your own site every day (or generate the same content into files)
Put the content into files with file names representing the URLs (or HAR format?)
Zip em up
Put them on bittorrent
Tell the search engines to look there for your content
Wouldn't that save everyone a lot of work?
Re: Who exactly is crawling my site?
#16If everybody does this then a new search engine will never come to be. You better hope Google is the best search engine anyone could have ever created.
As a search engine developer that has tried to compete with Google in the past, I find this disheartening.
I agree with blocking anyone doing bad. But you are cutting out the good with the bad.
Re: Who exactly is crawling my site?
#17"I've decided to block all crawlers to the site other than Google or Bing" And this is why I find people that respond to privacy complaints about Google with "if you want to switch search engines no-one is stopping you" frustrating. Additionally, I would like to point out that according to those numbers, there are ~41,000 (418,814 - 199,725 - 40,359 - 36,340 - 33,893 - 26,325 - 13,458 - 10,657 - 6,109 - 5,993 - 4,959…
> But don't ban everything just because a few (proximic and ADmantX) are hammering the site.
I can understand blocking somebody that has a long-term and clear pattern of disrupting your site, not following the robots.txt rules, and not providing any links or anything back to you. But I find the idea of somebody preemptively blocking everything but Google and maybe Bing extremely distasteful.
If everybody out there blocked everything but Google/Bing, it would make it very difficult for anybody to ever try and create a new search engine or create new types of web services or analyze data in new ways.
Possibly a better solution is making the common crawl initiative a better project - make it more frequently updated, make it easier to get started with it, provide better documentation, etc. If there was a way to get every web service out there that wants to crawl the web to contribute to this, it would lighten the load on everybody. http://commoncrawl.org
Re: Who exactly is crawling my site?
#18You may want to update this to reflect the fact that "MJ12bot" is selling SEO Services, too.
They run a reward program for using their agent and getting pages crawled (data is then crunched on a central server cluster owned by the operators) - cash payouts that are seeded by the subscription models they have here http://www.majesticseo.com
Re: Who exactly is crawling my site?
#19"I've decided to block all crawlers to the site other than Google or Bing" And this is why I find people that respond to privacy complaints about Google with "if you want to switch search engines no-one is stopping you" frustrating. Additionally, I would like to point out that according to those numbers, there are ~41,000 (418,814 - 199,725 - 40,359 - 36,340 - 33,893 - 26,325 - 13,458 - 10,657 - 6,109 - 5,993 - 4,959…
This is a fair point, however blacklisting isn't necessarily a perfect solution either. It would require continuous manual effort in going through the logs and blocking bad bots, and if some new bot were to misbehave and crawl too aggressively, blacklisting would only help after the fact.
I do think I did make a mistake though. To your point, I shouldn't block crawlers that both behave and are attempting to help my site in some way (by driving traffic to it -- IE search engines). Whether or not they are currently driving traffic to the site is not important. I'll whitelist Yandex, Baidu, Scoutjet and any other related bots I see and edit the post.
Re: Who exactly is crawling my site?
#20Crawlers and spambots are the scourge of medium to small websites. I run a small wiki that gets just a few thousand human hits a day. But according to the server logs 90% of server hits are crawlers and spambots, so I'm using 10 times the resources I really need to serve customers. I finally resorted to blocking entire data centers and companies that crawl constantly but send no traffic. I feel like search engines sh…
Chicken and egg. Should they 1. crawl you first, or 2. send you traffic first? If 1, how long until they should either send traffic or stop, and if 2, how?