That robots.txt is so big that even with comments maybe google will drop it after trying to parse it for a while....
Robots.txt
61–70 of 142 posts
Re: Robots.txt
#62Keep up the good work Google. I hope this serves as encouragement for them to continue to screw over those who want to turn searching for relevant information into a war for who can cheat the system the best.
Totally with you. I was enjoying the amusing tone of the piece but he lost my sympathy at: "Before, SEO was the game of really skilled people, you know like professional poker players. You had to know the game, know when to raise, and when to bluff." Then with the "Maybe I should find another job?" I thought this has got to be satire. Well played, sir. The way the piece is framed, the writing style, surely a work of…
Re: Robots.txt
#63It seems that most people here would like it very much if SEO died a horrible death as quickly as possible. Let me try and paint a (probably somewhat sketchy) picture of why it should exist (I have nothing to do with SEO consulting btw): 1. You have a business. 2. Said business has a website. 3. Said website is bringing in no discernable revenue for your business. 4. Poking around for reasons, you discover that your…
Re: Robots.txt
#64Today, almost all large sites that people use to share links to original content use nofollow links. Because of this, it is common that when you Google for some original content, you will find links to aggregators (Stack Overflow, Reddit, HN, etc.) but the original page will be very far on the result list.
Re: Robots.txt
#65Nobody else read this as a joke? I read it as a joke (and a slightly racist one at that).
Re: Robots.txt
#66Earlier quoted context omitted.
I don't think this is a good analogy. Google's job is to make sure good content is ranked high and crap is ranked low. You should just create content that is useful to others and let Google figure out how to discover it and rank it high (they are good at it). To create useful content, you don't need to know Google internals.
And we would just do this if Google actually made good on their promises to do this, but I see an awful lot of crap still ranking in the SERPs especially because of hacked sites and link networks. Full disclosure - I work in SEO.
Note that I am not saying that you yourself perform any of this in your role. Our anger is not at site tuning and placement but the techniques used to do so, which you appear to be unnecessarily defending as essential for any site's survival.
Re: Robots.txt
#67That robots.txt is so big that even with comments maybe google will drop it after trying to parse it for a while....
Anybody know how big robots.txt can be and if anyone has ever given a spider grief by screwing with the size or download rate of robots.txt?
Re: Robots.txt
#68It seems that most people here would like it very much if SEO died a horrible death as quickly as possible. Let me try and paint a (probably somewhat sketchy) picture of why it should exist (I have nothing to do with SEO consulting btw): 1. You have a business. 2. Said business has a website. 3. Said website is bringing in no discernable revenue for your business. 4. Poking around for reasons, you discover that your…
I think the general contention is that the process you describe is just using someone who's going to write sensible HTML not SEO. The shady side of actually modifying code not for easy-to-read, browser-efficiency, etc. is SEO.
I don't see it as a game against Google... I try to make our sites as useful for humans as possible, and hope Google agrees with a decent enough ranking so humans can visit it and enjoy the content. But when it doesn't, then the game is between me and the "other SEOs" who may not see the point of making something useful for readers, not machines.
Re: Robots.txt
#69I think SEO is practically dead, and for good. The Social Web is supposedly lot more democratic and apparently a better choice.
Subverting "The Social Web" occurs to feed users to their advertising or other attempts at monetization, they don't get paid to Tweet.
Re: Robots.txt
#70Keep up the good work Google. I hope this serves as encouragement for them to continue to screw over those who want to turn searching for relevant information into a war for who can cheat the system the best.
Totally with you. I was enjoying the amusing tone of the piece but he lost my sympathy at: "Before, SEO was the game of really skilled people, you know like professional poker players. You had to know the game, know when to raise, and when to bluff." Then with the "Maybe I should find another job?" I thought this has got to be satire. Well played, sir. The way the piece is framed, the writing style, surely a work of…
-Rewarding relevant, informative, keyword-rich content focused on specific topics and punishing sites that construct thinly constructed (UGC or otherwise) pages, particularly those who do it in huge numbers.
-Rewarding sites who links from similarly relevant, authoritative sources gained organically and punishing those who acquire links through artificial means (either through direct purchases, purchasing domain portfolios with thousands of backlinks etc.)
There are nuances that are market-specific that require slightly different approaches and concentrations (e.g. health care) but those are the main things folks should concern themselves with. There's no black magic behind it unless you're doing something shady to begin with.