Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How is search so bad? A case study

svilentodorov.xyz

Post reply on HN
Print view View original

131–140 of 416 posts

Re: How is search so bad? A case study

#131

Post by fanf2 »

ethbro wrote: post #88
rahulchhabra07 wrote: post #32

I have been thinking about the same problem since a few weeks. The real problem with search engines is the fact that so many websites have hacked SEO that there is no meritocracy left. Results are not sorted based on relevance or quality but by SEO experts' efforts at making the search results favor themselves. I can possibly not find anything deep enough about any topic by searching on Google anymore. It's just surf…

> However, it will be interesting to figure the heuristics to deliver better quality search results today. If only there were some kind of analog for effective ways to locate information. Like if everything were written on paper, bound into collections, and then tossed into a large holding room. I guess it's past the Internet's event horizon now, but crawler-primary searching wasn't the only evolutionary path to sear…

Immediately before Google were search engines like AltaVista https://en.wikipedia.org/wiki/AltaVista (1995) and Lycos https://en.wikipedia.org/wiki/Lycos (1994) which were not directories like Yahoo. Google won by not being cluttered with non-search web portal clutter, and by the effectiveness of PageRank, and because by the late 1990s the web was too big to be indexed by a manually curated directory.

Re: How is search so bad? A case study

#132

Post by Razengan »

MayeulC wrote: post #51
rahulchhabra07 wrote: post #32

I have been thinking about the same problem since a few weeks. The real problem with search engines is the fact that so many websites have hacked SEO that there is no meritocracy left. Results are not sorted based on relevance or quality but by SEO experts' efforts at making the search results favor themselves. I can possibly not find anything deep enough about any topic by searching on Google anymore. It's just surf…

> A new breakthrough heuristic today will look something totally different, just as meritocratic and possibly resistant to gaming. I wonder how much of this could be obtained back by penalizing: 1. The number of javascript dependencies 2. The number of ads on the page, or the depth of the ad network This might start a virtuous circle, but in the end, this is just a game of cat-and-mouse, and website might optimize fo…

> What we might need to break this is ...

Create a core protocol at the same level as DNS etc., that web servers can use to offer an index of everything they serve/relay. A multitude of user-side apps may then query that protocol, with each app using different algorithms, heuristics and offering different options.

Re: How is search so bad? A case study

#133

Post by klingonopera »

dmfdmf wrote: post #127

The problem as I see it is that popularity ranking worked fine in the pre Eternal September era for the web (~10 years ago?). I think it is safe to say that most HN users skew toward searching for more technical, intellectual or scientific topics and get frustrated by their searches getting swamped by popular topics. What I'd like to see is a check box or slider bar to exclude or adjust the weighting for popularity i…

> pre Eternal September era for the web (~10 years ago?)

LOL, was that by chance the time that single from Green Day was released? ;)

["Eternal September or the September that never ended is Usenet slang for a period beginning in September 1993"]: https://en.wikipedia.org/wiki/Eternal_September

EDIT: TIL Green Day's single has nothing to do with that September. Huh. I gave them more nerdcred than they deserved...

Re: How is search so bad? A case study

#134

Post by Izkata »

basscomm wrote: post #44
tyingq wrote: post #7

Earlier quoted context omitted.

My guess is that suppressing spammy pages got too hard. So they applied some kind of big hammer that has a high false positive rate. You're getting the best of what's left. Maybe also some quality decline in their gradual shift to less hand weighted attributes and more ML.

My guess is that Google et al are all hell-bent on not telling you that your search returned zero results. They seem to go to great lengths to make sure that your results page has something on it by any means necessary, including: searching for synonyms for words I searched for instead of the specific words I chose, excluding words to increase the number of results (even though the words they exclude are usually the…

> I put in key words for something I'm looking for. I suspect that The Masses(tm) type in vague questions full of typos that search engines have to try to parse into a meaningful search query.

Yeeaap, sometime in gradeschool - I think somewhere around 5th grade, age 11 or so, which would be around 1999 - we had a section on computers, where we'd learn the basics about how to use them. One of the topics I remember was "how to do web searches", where a friend was surprised at how easily I found what I was looking for - the other kids had to be trained to use keywords instead of asking it questions.

Re: How is search so bad? A case study

#135

Post by Razengan »

Why can't every web server just remember (index) all the content that it serves?

Much like resolving DNS queries, I could then ask every server near me for specific terms. If they serve that content, they'll return a list of links containing those search terms.

We could have different apps with different algorithms to sort the bare results according to the criteria that's most relevant to each individual user.

Re: How is search so bad? A case study

#136

Post by yusef555 »

tsian2 wrote: post #92
yusef555 wrote: post #41

Earlier quoted context omitted.

But does anyone know why search on Reddit is broken? Perhaps intentionally? I don't want to get tin foil hatty but perhaps more not readily apparent false positives = more user clicks = more revenue via ad serving?

I often wonder why some fairly large companies that rely heavily on their own website don't seem to put more than a sole web developer worth of resources into them. Reddit fits into that category for me (Reddit has 400 employees). Initially I had the impression that search was hard to implement. However, spending a work week figuring it out with ElasticSearch, Solr and Sphinx changed my mind. Getting the solution to…

I wouldn't say it's a trivial ask, but yeah, if you have 400 employees at least assign some resources to get it right. Unless it's intentionally broken. Facebook's prioritization but also randomization of the feed is a feature not a bug.

Re: How is search so bad? A case study

#137

Post by zozbot234 »

Razengan wrote: post #132
MayeulC wrote: post #51

Earlier quoted context omitted.

> A new breakthrough heuristic today will look something totally different, just as meritocratic and possibly resistant to gaming. I wonder how much of this could be obtained back by penalizing: 1. The number of javascript dependencies 2. The number of ads on the page, or the depth of the ad network This might start a virtuous circle, but in the end, this is just a game of cat-and-mouse, and website might optimize fo…

> What we might need to break this is ... Create a core protocol at the same level as DNS etc., that web servers can use to offer an index of everything they serve/relay. A multitude of user-side apps may then query that protocol, with each app using different algorithms, heuristics and offering different options.

> Create a core protocol at the same level as DNS etc., that web servers can use to offer an index of everything they serve/relay.

So, back to gopher? That might actually work!

Re: How is search so bad? A case study

#138

Post by yusef555 »

graycat wrote: post #108
rahulchhabra07 wrote: post #32

I have been thinking about the same problem since a few weeks. The real problem with search engines is the fact that so many websites have hacked SEO that there is no meritocracy left. Results are not sorted based on relevance or quality but by SEO experts' efforts at making the search results favor themselves. I can possibly not find anything deep enough about any topic by searching on Google anymore. It's just surf…

Good to hear your concerns. > The real problem with search engines is the fact that so many websites have hacked SEO that there is no meritocracy left. I intend to announce the alpha test of my search engine here on HN. My search engine is immune to all SEO efforts. > I can possibly not find anything deep enough about any topic by searching on Google anymore. In simple terms my search engine gives users content with…

cool story, bro

Re: How is search so bad? A case study

#139

Post by dredmorbius »

Razengan wrote: post #132
MayeulC wrote: post #51

Earlier quoted context omitted.

> A new breakthrough heuristic today will look something totally different, just as meritocratic and possibly resistant to gaming. I wonder how much of this could be obtained back by penalizing: 1. The number of javascript dependencies 2. The number of ads on the page, or the depth of the ad network This might start a virtuous circle, but in the end, this is just a game of cat-and-mouse, and website might optimize fo…

> What we might need to break this is ... Create a core protocol at the same level as DNS etc., that web servers can use to offer an index of everything they serve/relay. A multitude of user-side apps may then query that protocol, with each app using different algorithms, heuristics and offering different options.

I've been thinking along similar lines for a year or so now.

There are several puzzling omissions from Web standards, particularly given that keyword-based search was part of the original CERN WWW discussion:

http://info.cern.ch/hypertext/WWW/Addressing/Search.html

IF we had a distributable search protocol, index, and infrastructure ... the entire online landscape might look rather different.

Note that you'd likely need some level of client support for this. And the world's leading client developer has a strongly-motivated incentive to NOT provide this functionality integrally.

A distributed self-provided search would also have numerous issues -- false or misleading results (keyword stuffing, etc.) would be harder to vet than the present situation. Which suggests that some form of vetting / verifying provided indices would be required.

Even a provided-index model would still require a reputational (ranking) mechanism. Arguably, Google's biggest innovation wasn't spidering, but ranking. The problem now is that Google's ranking ... both doesn't work, and incentivises behaviours strongly opposed to user interests. Penalising abusive practices has to be built into the system, with those penalties being rapid, effective, and for repeat offenders, highly durable.

The problem of potential for third-party malfeasance -- e.g., engaging in behaviours appearing to favour one site, but performed to harm that site's reputation through black-hat SEO penalties, also has to be considered.

As a user, the one thing I'd most like to be able to do is specify blacklists of sites / domains I never want to have appear in my search results. Without having to log in to a search provider and leave a "personalised" record of what those sites are.

(Some form of truly anonymised aggregation of such blocklists would, of course, be of some use, and facilitating this is an interesting challenge.)

Re: How is search so bad? A case study

#140

Post by dredmorbius »

basscomm wrote: post #44
tyingq wrote: post #7

Earlier quoted context omitted.

My guess is that suppressing spammy pages got too hard. So they applied some kind of big hammer that has a high false positive rate. You're getting the best of what's left. Maybe also some quality decline in their gradual shift to less hand weighted attributes and more ML.

My guess is that Google et al are all hell-bent on not telling you that your search returned zero results. They seem to go to great lengths to make sure that your results page has something on it by any means necessary, including: searching for synonyms for words I searched for instead of the specific words I chose, excluding words to increase the number of results (even though the words they exclude are usually the…

Tyranny of the minimum viable user.
Post reply on HN