Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How is search so bad? A case study

svilentodorov.xyz

Post reply on HN
Print view View original

181–190 of 416 posts

Re: How is search so bad? A case study

#181

Post by Swizec »

AznHisoka wrote: post #167
rahulchhabra07 wrote: post #32

I have been thinking about the same problem since a few weeks. The real problem with search engines is the fact that so many websites have hacked SEO that there is no meritocracy left. Results are not sorted based on relevance or quality but by SEO experts' efforts at making the search results favor themselves. I can possibly not find anything deep enough about any topic by searching on Google anymore. It's just surf…

“I can possibly not find anything deep enough about any topic by searching on Google anymore. It's just surface-level knowledge that I get from competing websites who just want to make money off pageviews.” Is it possible that there is no site providing non fluffy content on your query? For a lot of niche subjects, there really are very few if any substantial content on that topic.

> Is it possible that there is no site providing non fluffy content on your query? For a lot of niche subjects, there really are very few if any substantial content on that topic.

“Very few if any substantial”

The problem is that google won’t even show me the very few anymore. It’s just fluff upon fluff and depth (or real insight at least) is buried in twitter threads and reddit/hn comments, and github issue discussion.

I fear the seo problem has not only killed knowledge propagation, but also thoroughly disincentivized smart people from even trying. And that makes me sad.

Re: How is search so bad? A case study

#182

Post by anoncake »

zozbot234 wrote: post #94
derefr wrote: post #64

Earlier quoted context omitted.

You could even have all this under one roof: one common search spider that feeds this ensemble of different ranking algorithms to produce a set of indices, and then a search engine front end that round-robins queries out between the different indices. (Don’t like your query? Spin the algorithm wheel! “I’m Feeling Lucky” indeed .)

The Common Crawl is a thing already. Unfortunately, a "full" text crawl of the internets is a YUUUGE amount of data to manage, and I can't think of anything that could change that in the foreseeable future. That's why I think providing a federated Web directory standard, ala ODP/DMOZ except not limited to a single source, would be a far more impactful development.

Javascript, which Google coincidentally pushed and still pushes for, doesn't exactly make the web easier to crawl either.

Re: How is search so bad? A case study

#184

Post by nvarsj »

The first time I realized that Google search was bad was when del.icio.us got big. I was an avid user - and I stopped using Google except for basic things. You could search tags on del.icio.us - and the results were incredibly good, far better than Google, especially for niche areas.

I think, unfortunately, this kind of curated, social approach to search will never be compatible with monetization by ads. I'm not quite sure how to make a search engine profitable without significantly distorting its results. Maybe, depressingly, Google is the best thing possible given the constraint of making a profit?

Re: How is search so bad? A case study

#185

Post by reaperducer »

zozbot234 wrote: post #166
reaperducer wrote: post #154

Earlier quoted context omitted.

Unfortunately, a "full" text crawl of the internets is a YUUUGE amount of data to manage Maybe instead of a problem, there is an opportunity here. Back before Google ate the intarwebs, there used to be niche search engines. Perhaps that is an idea whose time has come again. For example, if I want information from a government source, I use a search engine that specializes in crawling only government web sites. If I w…

That's the problem that web directories solve. It's not that you're wrong, it's just largely orthogonal to the problem that you'd need a large crawl of the internets for, i.e. spotting sites about X niche that you wouldn't find even from other directly-related sites, and that are too obscure, new, etc. to be linked in any web directory.

That's the problem that web directories solve

Not really. A web directory is a directory of web sites. I can't search a web directory for content within the web sites, which is what a niche search engine would do.

Re: How is search so bad? A case study

#186

Post by z3t4 »

Public information should not be filtered by one single private entity. We need a distributed system with an open standard. Search should work more like DNS... In order to get your web site indexed, you only have to publish your search URL. There should be many index cache servers, so that your search URL only get a hit when a search string expires...

Re: How is search so bad? A case study

#187

Post by erikbye »

rahulchhabra07 wrote: post #32

I have been thinking about the same problem since a few weeks. The real problem with search engines is the fact that so many websites have hacked SEO that there is no meritocracy left. Results are not sorted based on relevance or quality but by SEO experts' efforts at making the search results favor themselves. I can possibly not find anything deep enough about any topic by searching on Google anymore. It's just surf…

> It's just surface-level knowledge that I get from competing websites who just want to make money off pageviews.

Can you give some examples of queries/topics? Not that I disagree, I often have the same problem, but have found ways to mitigate.

Re: How is search so bad? A case study

#188

Post by pbhjpbhj »

Razengan wrote: post #132
MayeulC wrote: post #51

Earlier quoted context omitted.

> A new breakthrough heuristic today will look something totally different, just as meritocratic and possibly resistant to gaming. I wonder how much of this could be obtained back by penalizing: 1. The number of javascript dependencies 2. The number of ads on the page, or the depth of the ad network This might start a virtuous circle, but in the end, this is just a game of cat-and-mouse, and website might optimize fo…

> What we might need to break this is ... Create a core protocol at the same level as DNS etc., that web servers can use to offer an index of everything they serve/relay. A multitude of user-side apps may then query that protocol, with each app using different algorithms, heuristics and offering different options.

How do you stop the server lying?

If a good site is on shared hosting will it always be dismissed because of the signal of the other [bad] sites on that same host? (you did say at DNS level, not domain level)

Re: How is search so bad? A case study

#190

Post by HomeDeLaPot »

reaperducer wrote: post #154
zozbot234 wrote: post #94

Earlier quoted context omitted.

The Common Crawl is a thing already. Unfortunately, a "full" text crawl of the internets is a YUUUGE amount of data to manage, and I can't think of anything that could change that in the foreseeable future. That's why I think providing a federated Web directory standard, ala ODP/DMOZ except not limited to a single source, would be a far more impactful development.

Unfortunately, a "full" text crawl of the internets is a YUUUGE amount of data to manage Maybe instead of a problem, there is an opportunity here. Back before Google ate the intarwebs, there used to be niche search engines. Perhaps that is an idea whose time has come again. For example, if I want information from a government source, I use a search engine that specializes in crawling only government web sites. If I w…

Don't forget the search engine search engine!
Post reply on HN