Live data from Hacker News

Robots.txt

explicitly.me

61–70 of 142 posts

Re: Robots.txt

#61
post #21

That robots.txt is so big that even with comments maybe google will drop it after trying to parse it for a while....

Anybody know how big robots.txt can be and if anyone has ever given a spider grief by screwing with the size or download rate of robots.txt?

Re: Robots.txt

#62
post #54
post #12

Keep up the good work Google. I hope this serves as encouragement for them to continue to screw over those who want to turn searching for relevant information into a war for who can cheat the system the best.

Totally with you. I was enjoying the amusing tone of the piece but he lost my sympathy at: "Before, SEO was the game of really skilled people, you know like professional poker players. You had to know the game, know when to raise, and when to bluff." Then with the "Maybe I should find another job?" I thought this has got to be satire. Well played, sir. The way the piece is framed, the writing style, surely a work of…

From his twitter postings, I don't believe this is fake.

Re: Robots.txt

#63

It seems that most people here would like it very much if SEO died a horrible death as quickly as possible. Let me try and paint a (probably somewhat sketchy) picture of why it should exist (I have nothing to do with SEO consulting btw): 1. You have a business. 2. Said business has a website. 3. Said website is bringing in no discernable revenue for your business. 4. Poking around for reasons, you discover that your…

I think the general contention is that the process you describe is just using someone who's going to write sensible HTML not SEO. The shady side of actually modifying code not for easy-to-read, browser-efficiency, etc. is SEO.

Re: Robots.txt

#64
SEO has made a lot of damage to the Internet. 10 years ago links really expressed people preferences, and it was easy to get linked and ranked high when you've published useful stuff.

Today, almost all large sites that people use to share links to original content use nofollow links. Because of this, it is common that when you Google for some original content, you will find links to aggregators (Stack Overflow, Reddit, HN, etc.) but the original page will be very far on the result list.

Re: Robots.txt

#65
post #31

Nobody else read this as a joke? I read it as a joke (and a slightly racist one at that).

I read it as a joke by the end but, worryingly, didn't automatically recontextualise it as racist in light of that. Which, I think I agree with you, it is, if it's a joke and all the East Africa stuff is made up for "comedy value".

Re: Robots.txt

#66

Earlier quoted context omitted.

I don't think this is a good analogy. Google's job is to make sure good content is ranked high and crap is ranked low. You should just create content that is useful to others and let Google figure out how to discover it and rank it high (they are good at it). To create useful content, you don't need to know Google internals.

And we would just do this if Google actually made good on their promises to do this, but I see an awful lot of crap still ranking in the SERPs especially because of hacked sites and link networks. Full disclosure - I work in SEO.

The solution to no-content black hattery is not to promote your no-content site with black hat techniques. Nobody here is criticizing site tuning and at the same time defending content farms. Why defend misbehavior as a justification for misbehavior?

Note that I am not saying that you yourself perform any of this in your role. Our anger is not at site tuning and placement but the techniques used to do so, which you appear to be unnecessarily defending as essential for any site's survival.

Re: Robots.txt

#67
post #21

That robots.txt is so big that even with comments maybe google will drop it after trying to parse it for a while....

Anybody know how big robots.txt can be and if anyone has ever given a spider grief by screwing with the size or download rate of robots.txt?

The robots.txt limit in Google's Webmasterpanelthing is 100,000 characters. Googlebot has been said to stop parsing robots.txt after 500,000 characters. I don't know if that's bytes or characters.

Re: Robots.txt

#68
post #63

It seems that most people here would like it very much if SEO died a horrible death as quickly as possible. Let me try and paint a (probably somewhat sketchy) picture of why it should exist (I have nothing to do with SEO consulting btw): 1. You have a business. 2. Said business has a website. 3. Said website is bringing in no discernable revenue for your business. 4. Poking around for reasons, you discover that your…

I think the general contention is that the process you describe is just using someone who's going to write sensible HTML not SEO. The shady side of actually modifying code not for easy-to-read, browser-efficiency, etc. is SEO.

I have to disagree. I consider part of my job to be some SEO (not my primary task and neither the most important, but I take care of it at my company) and its definitely the first part of what you sketch. And trying to clean all the second part stupid stuff someone else may have done before.

I don't see it as a game against Google... I try to make our sites as useful for humans as possible, and hope Google agrees with a decent enough ranking so humans can visit it and enjoy the content. But when it doesn't, then the game is between me and the "other SEOs" who may not see the point of making something useful for readers, not machines.

Re: Robots.txt

#69

I think SEO is practically dead, and for good. The Social Web is supposedly lot more democratic and apparently a better choice.

"I think SEO is practically dead, and for good. The Social Web is supposedly lot more democratic and apparently a better choice."

Subverting "The Social Web" occurs to feed users to their advertising or other attempts at monetization, they don't get paid to Tweet.

Re: Robots.txt

#70
post #54
post #12

Keep up the good work Google. I hope this serves as encouragement for them to continue to screw over those who want to turn searching for relevant information into a war for who can cheat the system the best.

Totally with you. I was enjoying the amusing tone of the piece but he lost my sympathy at: "Before, SEO was the game of really skilled people, you know like professional poker players. You had to know the game, know when to raise, and when to bluff." Then with the "Maybe I should find another job?" I thought this has got to be satire. Well played, sir. The way the piece is framed, the writing style, surely a work of…

I agree; that last comment gives it away. I've been doing enterprise-level SEO for several years now as a day job and it's not as if algo updates are so frequent and far reaching as to make it difficult to keep up with. Most major updates from Google and the other engines are predicated on two major themes:

-Rewarding relevant, informative, keyword-rich content focused on specific topics and punishing sites that construct thinly constructed (UGC or otherwise) pages, particularly those who do it in huge numbers.

-Rewarding sites who links from similarly relevant, authoritative sources gained organically and punishing those who acquire links through artificial means (either through direct purchases, purchasing domain portfolios with thousands of backlinks etc.)

There are nuances that are market-specific that require slightly different approaches and concentrations (e.g. health care) but those are the main things folks should concern themselves with. There's no black magic behind it unless you're doing something shady to begin with.

Post reply on HN