Live data from Hacker News

Robots.txt

explicitly.me

71–80 of 142 posts

Re: Robots.txt

#71

I wonder how this post got to the top even though the sentiment in the comments is that this posts sucks.

What others have said about commenters/lurkers and not being able to downvote, but also I think the "sentiment" is more that SEO sucks than that this piece is garbage per se. I upvoted it because it's entertaining and it did, along with the discussion here, broaden my perspective on SEO.

Re: Robots.txt

#73
post #34
post #23

Earlier quoted context omitted.

I think this is a common view, and I share it: I really don't like SEO. The thing is, SEO does provide real value, but it provides value because the system is broken: your website should reach the top of search results based on merit, not based on some arcane black magic. I think it's somewhat akin to lobbyists: I don't blame them for making money off and taking advantage off the political system, and they certainly…

In short, don't hate the player hate the game? I understand where you're coming from, but that's crap. Just because there is always someone willing to do ethically questionable things that fall within the letter of the law, doesn't mean that those types of people shouldn't be treated with disdain by the rest of us.

don't hate the player hate the game?

Why choose? I'm already somewhat of a misanthrope, but the fact that SEO exists just makes me hate everyone and everything.

Re: Robots.txt

#74

An SEO consultant complaining about Google's frequent algo changes is like a mechanic complaining that cars break down too often.

Not entirely - after all, to his clients an SEO consultant isn't just a black hole they throw money into. They do expect to see results in the form of higher search ranking, and Google's algorithm changes make it harder to deliver on that. I think I'd mirror others' analogies. It's more that the complaint is futile - and kind of sad. One comes away with the impression that the author is naive enough about the nature…

"to his clients an SEO consultant isn't just a black hole they throw money into. They do expect to see results in the form of higher search ranking, and Google's algorithm changes make it harder to deliver on that."

Wouldn't it then be harder to demonstrate objective returns if Google's algorithms are so obfuscated? If they're not afraid to game Google for their customer's benefit, they're certainly not going to be afraid to game customer expectations and fudge their results for a living.

Re: Robots.txt

#75
post #67

Earlier quoted context omitted.

Anybody know how big robots.txt can be and if anyone has ever given a spider grief by screwing with the size or download rate of robots.txt?

The robots.txt limit in Google's Webmasterpanelthing is 100,000 characters. Googlebot has been said to stop parsing robots.txt after 500,000 characters. I don't know if that's bytes or characters.

500kb is the hard upper limit in the crawler itself. Limits that are exposed elsewhere (Webmaster Tools, etc) are irrelevant.

Re: Robots.txt

#76

I think SEO is practically dead, and for good. The Social Web is supposedly lot more democratic and apparently a better choice.

"I think SEO is practically dead, and for good. The Social Web is supposedly lot more democratic and apparently a better choice." Subverting "The Social Web" occurs to feed users to their advertising or other attempts at monetization, they don't get paid to Tweet.

Well, there's a negative as well, hence I used keywords like supposedly, apparently etc. But wouldn't you rather your insurance-selling uncle visit you for yourself, instead of an intent to sell you the insurance? I think those days will be here eventually.

Re: Robots.txt

#77
post #13

Am i the only one who thinks that SEO is more a plague then it is a real profession? Google making it hard for SEO is exactly the right thing to do. Making it impossible for SEO should be the ultimate _goal_ of a good search engine.

Depends what you mean. If you mean people whose goal it is to help site owners learn how to structure their data intelligently for dissection by machines, people who make companies think very hard about their content, people who encourage high-quality crosslinks and meta tags, and people who generally help sites manage data, then no, SEOs are great. They help users and companies alike. If you mean the dirtball black…

Is the first group SEOs, or information architects? Good information navigation design will consider searchability and, more importantly, human accessibility (a large part of which will be searchability, but search serves the primary need of people to find information efficiently).

Re: Robots.txt

#78
post #50
post #39

Earlier quoted context omitted.

If you care about the underlying algorithms, I'd think it's pretty clear which kind of SEO we're talking about.

I think it would be reasonable to understand the underlying algorithms in order to optimise what white hat SEO to spend effort on. But I accept that this is probably a rare breed.

This is true. But there really are hundreds of thousands (probably millions and millions) of sites that are structured horribly where you can simply rely on knowing how the heck to structure a page correctly.

For instance...

I recently moved a corporate site for a major national law firm from Joomla to WordPress. They had 200+ pages with multiple H1 tags, nearly 300 pages missing descriptions, dozens and dozens of soft 404s, hundreds of pages with bad titles or URLs that are too long, dozens of pages with duplicate content because of bad sitemaps, and every single page was over 100 links because of the dummy that made the original nav toolbar put every subpage in there.

White hats can work for the next decade simply trying to save companies from themselves, their stupid CMS systems, and former well meaning SEO efforts that are now banned or deprecated.

Short term effect to fixing these issues? A nearly 20% increase in traffic since the switchover and slightly raised ranking for related firm content on OTHER sites.

Re: Robots.txt

#79

I wonder how this post got to the top even though the sentiment in the comments is that this posts sucks.

I find the perspective of the SEO guy interesting, even if I find what he does kind of skeezy and annoying.

The honesty is a bit refreshing, I agree. I'd love to read a good, anonymous confessional of the daily life of a SEO that's more intimate and less of their usual humblebrag PR.

Re: Robots.txt

#80
post #77

Earlier quoted context omitted.

Depends what you mean. If you mean people whose goal it is to help site owners learn how to structure their data intelligently for dissection by machines, people who make companies think very hard about their content, people who encourage high-quality crosslinks and meta tags, and people who generally help sites manage data, then no, SEOs are great. They help users and companies alike. If you mean the dirtball black…

Is the first group SEOs, or information architects? Good information navigation design will consider searchability and, more importantly, human accessibility (a large part of which will be searchability, but search serves the primary need of people to find information efficiently).

Good point, white hat SEO is simply applied IA.

Thanks for the resumé tip :)

Post reply on HN