Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How is search so bad? A case study

svilentodorov.xyz

Post reply on HN
Print view View original

111–120 of 416 posts

Re: How is search so bad? A case study

#111

Post by klingonopera »

bsanr2 wrote: post #54
yoz-y wrote: post #36

Earlier quoted context omitted.

I work for Google but not on search. I think in this case semantic web would not work, unless there was some way to weed out spam. There are currently multiple competing microdata formats out there than enable you to specify any kind of metadata but they still won't help if spammers fill those too. Maybe some sort of webring of trust where trusted people can endorse other sites and the chain breaks if somebody is fou…

In the interests of not causing a crisis when Top Level Trust Domain endorses the wrong site and the algorithm goes, "Uh uh," (or the endorsement is falsely labeled spam by malicious actors, or whatever), maybe the effect decreases the closer you are to that top level. But that's hierarchical in a very un-web-y way... Hm.

The internet is still kind of a hierarchy though, "changing" "ownership" from the government DARPA to the non-profit ICANN.

And that has worked... quite fine. I have no objections (maybe they're a bit too liberal with the new TLDs).

Most of the stuff that makes the hierarchies seem bad are actually faults of for-profit organizations (or other unsuited people/entities) being at the top, and not just that someone is at the top per se. In fact, in my experience, and contrary to popular expectation, when a hierarchy works well, an outsider shouldn't actually be able to immediately recognize it as such.

Re: How is search so bad? A case study

#112

Post by detaro »

endothrowho333 wrote: post #101
rahulchhabra07 wrote: post #32

I have been thinking about the same problem since a few weeks. The real problem with search engines is the fact that so many websites have hacked SEO that there is no meritocracy left. Results are not sorted based on relevance or quality but by SEO experts' efforts at making the search results favor themselves. I can possibly not find anything deep enough about any topic by searching on Google anymore. It's just surf…

This is very much misguided. Many websites do have "hacked" (blackhat/shady) SEO, but these websites do not last long, and are entirely wiped out (see: de-ranked) every major algorithm update. The major players you see on the top rankings today do utilize some blackhat SEO, but it's not at a level that significantly impacts their rankings. Blackhat SEO is inherently dangerous, because Google's algorithm will penalize…

> Google's algorithm works for its intended purposes, which is to serve pages that will benefit the highest amount of people searching for a specific term.

Does it? So for any product search, thrown-together comparison sites without actual substance but lots of affiliate links are really among the best results? Or maybe they are the most profitable result, and thus the one most able to invest in optimizing for ranking? Similarly, do we really expect results on (to a human) clearly hacked domains to be the best for anything, but Google will still put them in the top 20 for some queries? "Normal people want this crap" is a questionable starting point in many cases.

Re: How is search so bad? A case study

#113

Post by CM30 »

ethbro wrote: post #88
rahulchhabra07 wrote: post #32

I have been thinking about the same problem since a few weeks. The real problem with search engines is the fact that so many websites have hacked SEO that there is no meritocracy left. Results are not sorted based on relevance or quality but by SEO experts' efforts at making the search results favor themselves. I can possibly not find anything deep enough about any topic by searching on Google anymore. It's just surf…

> However, it will be interesting to figure the heuristics to deliver better quality search results today. If only there were some kind of analog for effective ways to locate information. Like if everything were written on paper, bound into collections, and then tossed into a large holding room. I guess it's past the Internet's event horizon now, but crawler-primary searching wasn't the only evolutionary path to sear…

Mixing human curation with crawlers is probably something that'd help with search results quality, but the issue comes in trying to get it to scale properly. Directories like the Open Directory Project/DMOZ and Yahoo's directory had a reputation for being slow to update, which left them miles behind Google and its ilk when it came to indexing new sites and information.

This is problematic when entire categories of sites were basically left out of the running, since the directory had no way to categorise them. I had that problem with a site about a video game system the directory hadn't added yet, and I suspect others would have it for say, a site about a newer TV show/film or a new JavaScript framework.

You've also got the increase in resources needed (you need tons of staff for effective curation), and the issues with potential corruption to deal with (another thing which significantly effected the ODP's usefulness in its later years).

Re: How is search so bad? A case study

#114

Post by klingonopera »

scottlocklin wrote: post #98

I agree google is shit, which is why I never use it. They're too busy being woke to run a search engine which compares to what they were doing 10-15 years ago. It's pretty obviously entirely that; smart people don't want to work in dying Brezhnevian bureaucratic hellscapes. Try Yandex for an example of a much smaller company doing a fine job at search: https://yandex.com/search/?text=reddit%20phone%20to%20buy&lr... P…

IIRC, Qwant piggybacks (-ed?) off Bing.

Re: How is search so bad? A case study

#115

Post by zozbot234 »

CM30 wrote: post #113
ethbro wrote: post #88

Earlier quoted context omitted.

> However, it will be interesting to figure the heuristics to deliver better quality search results today. If only there were some kind of analog for effective ways to locate information. Like if everything were written on paper, bound into collections, and then tossed into a large holding room. I guess it's past the Internet's event horizon now, but crawler-primary searching wasn't the only evolutionary path to sear…

Mixing human curation with crawlers is probably something that'd help with search results quality, but the issue comes in trying to get it to scale properly. Directories like the Open Directory Project/DMOZ and Yahoo's directory had a reputation for being slow to update, which left them miles behind Google and its ilk when it came to indexing new sites and information. This is problematic when entire categories of si…

Federation would help with both breadth and potential corruption, compared to what we had with ODP/DMOZ. A federated Web directory (with common naming/categorization standards, but very little beyond that) would probably have been infeasible back then simply because the Internet was so much smaller and fewer people were involved (and DMOZ itself partially made up for that lack by linking to "awesome"-like link lists where applicable) - but I'm quite sure that it could work today, particularly in the "commercial-ish" domain where corruption worries are most relevant.

Re: How is search so bad? A case study

#116

Post by Redoubts »

ma2rten wrote: post #9
arielweisberg wrote: post #3

Google has definitely stopped being able to find the things I need. Pasting stack traces and error messages. Needle in a haystack phrases from an article or book. None of it works anymore. Does this mean they are ripe for disruption or has search gotten harder?

Did you try putting them in quotes? EDIT: I don't know why this being downvoted. This is a genuine question to understand if the problem is the size of the index or the fuzzing matching that search engines do.

> I don't know why this being downvoted.

tbh, it's one of those "Are you sure you're not an idiot?" replies.

Re: How is search so bad? A case study

#117

Post by brownbat »

basscomm wrote: post #44
tyingq wrote: post #7

Earlier quoted context omitted.

My guess is that suppressing spammy pages got too hard. So they applied some kind of big hammer that has a high false positive rate. You're getting the best of what's left. Maybe also some quality decline in their gradual shift to less hand weighted attributes and more ML.

My guess is that Google et al are all hell-bent on not telling you that your search returned zero results. They seem to go to great lengths to make sure that your results page has something on it by any means necessary, including: searching for synonyms for words I searched for instead of the specific words I chose, excluding words to increase the number of results (even though the words they exclude are usually the…

Surely most engineers want the power of strict searching and less of the comforts of being always getting filler results, right?

So... is there an internal service at Google that works correctly but they're hiding from the world?

It might be useful for Google to make different search engines for different types of people. The behaviors of people are probably multi-modal, rather than normally distributed along some continuum where you should just assume the most common behavior and preferences. \

It would even be easier to target ads...

Or maybe this doesn't exist and spam is too hard.

Re: How is search so bad? A case study

#118

Post by woadwarrior01 »

ma2rten wrote: post #11

This article is essentially just complaining that DDG and Google don't have special parsing for reddit pages ("How come it doesn't know that thread didn't get many upvotes?", "How come it thinks some change to the site's layout was an update to the page?") Maybe if you want to search reddit, the best search engine is the search bar on reddit.com.

The search bar on reddit.com only searches through post titles and not comments. Using Google/DDG/ seaches through all content posted on Reddit and not just post titles. Until Reddit implements proper search, people will keep using search engines to search for content on it.

Re: How is search so bad? A case study

#119

Post by CM30 »

fortran77 wrote: post #109

I just did a google search for "piano". Just the word "piano" Only one link on the first page, the wikipedia entry for "piano" had anything to do with pianos, (i.e., the instrument invented in Italy 300+ years ago that has hammers, strings, and an iron frame).

What do you get when you search for that? Did a test right now, and apart from the Wikipedia page, I get videos about piano music/pianos, shop pages for buying pianos and local businesses that sell either pianos or piano lessons.

So I'm curious whether the issue is that there are too many shopping/business related pages (which is fair, but at least those seem to be piano related), or whether you're getting something completely different.

Re: How is search so bad? A case study

#120

Post by jszymborski »

Not sure if DDG patched this, but querying DDG w/o the month based tick results in a result that'll point you to the correct subreddit for finding which phone to buy in [0].

Although it's not using the "This Month" dropdown, doing the vanilla search still gets you the "most correct" answer imho.

[0] https://jszym.com/dl/imgs/20200119-ddg-example.png

Post reply on HN