Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reddit can't build a better search engine

ruky.me

Post reply on HN
Print view View original

271–280 of 289 posts

Re: Reddit can't build a better search engine

#271

Post by bduerst »

alkonaut wrote: post #202
bjarneh wrote: post #187

> I was too young to use Google when it first got started, but according to many, in the beginning, Google had better or more accurate search results. I'm old enough to remember when Google came, and the main difference was this: With Google you did not have to flip through 3 pages of completely unrelated search results; that were paid for. Google used to display advertisements on the right as well, i.e. not among th…

The early search engines (Webcrawler, Lycos, ...) ranked "relevance" based on some silly metric like "the more times the search term appears on the page, the more relevant the page must be". So the result was that the first pages were full spam sites that just used search terms as hidden text. Then Altavista came and it was a huge change for the better. They at least made some effort of trying to rank actual relevanc…

>but that Bing and DDG have improved so much that they are now on Par

Worth noting that these are essentially the same. DDG scrapes a handful of sites but they lean heavily on Bing's search API (or Yandex, if you search in russian).

It would be interesting if there could be some independent consumer group that would be able to grade search engine results and compare them regularly. To not only hold them accountable but give credit where it's due too.

Re: Reddit can't build a better search engine

#272

Post by quixoticaxolotl »

redwall_hp wrote: post #257
quixoticaxolotl wrote: post #206

I've searched for Reddit in the past because Reddit is the only site where I can get community answers to an important question. In those cases, the community aspect mattered to me - for example, I recently moved to Boston, so I solicited Bostonian's recommendations from r/boston; my partner and I were having troubles, and I wanted to hear from people who'd gone through similar experiences; I wanted to learn about co…

While there's a powerful inclination to take Reddit recommendations, it's also increasingly dangerous. Just from normal usage, it's increasingly apparent that a disturbing proportion of comments are astroturf. There's a flood of accounts that have generic names in the form of word-word-number or word_word_number, usually are only a few days or weeks old with little karma, and run around threads responding to everythi…

I think this is a really strong second-order point, in that Reddit is vulnerable to comments gamification in the same way Google was to SEO gamification, which sort of defeats the point of this whole enterprise.

Re: Reddit can't build a better search engine

#273

Post by bduerst »

lkxijlewlf wrote: post #266

A lot of people don't use search in Reddit. They'll ask the same questions over and over ("Why is "other" taking all my storage ???"). Reddit could make a better search engine than google and it wouldn't matter.

Yeah, much of reddit's content is repetitive, even so much that commenters will complain about "reposts".

If reddit ever built a search engine, it would need to build a graph that would cluster all these repetitive posts. Users would then be able to easily find a previous post before they went on to create their own repost.

This would lead to less posts and less avenues for other users to engage, which is counter to Reddit's own model and probably why Reddit doesn't want a search engine that works.

Re: Reddit can't build a better search engine

#274

Post by sndman »

noncoml wrote: post #268

I wonder if is Google the defacto search engine because they actually cannot be beaten or is it because it's virtually impossible for a new player to "scrape" the internet as easy as Google does. Websites, including Reddit, are happy to give all their data to Google bots, but if anyone else tries to crawl them they will get black-listed very fast even if they follow the robots.txt rules.

I lobbied heavily to crawl my company's intranet since our official IT folks would not include the ~10k "grey" (as in unofficial) web servers set up by technical people over the years.

So I tried this myself and quickly realized how hard it was. I stumbled upon thousands of devices that had TCP port 80 (and 443) open so I had to devise various ways of removing these devices.

By the end of my project, I had run out of disk space so many times it was laughable. And tuning the crawling and the resulting mountain of data was daunting and started to affect my day job so I eventually gave up.

A couple of months after my "project", and with enough warnings from IT and our network security folks, our company decided to purchase a couple google 1U servers.

Re: Reddit can't build a better search engine

#275

Post by bduerst »

notRobot wrote: post #19

Reddit doesn't really need to build a search engine, they could just have an algolia instance like HN does: https://hn.algolia.com/ . While not perfect, it does a pretty good job, in my experience. However, the article makes a valid point -- the moment people start using it heavily, we'll get a reddit comment equivalent of SEO, with keyword stuffing and whatnot.

Algolia was the first thing I though about when the post was made yesterday.

I commented this elsewhere but I don't think Reddit wants a good search option. Users tend to repost a non-trivial amount of content/questions/etc., and if those users could find the original content with a better search, it's likely reddit would see less user engagement.

Re: Reddit can't build a better search engine

#276

Post by Root_Denied »

lrem wrote: post #98
pojzon wrote: post #92

Earlier quoted context omitted.

Recently have noticed a business creating fictional threads on reddit (multiple on different communities) working as an AD to sell something. Example: „What is the cheapest best cooling for cpu now?” All discussions point to product that more experienced users know its crap. But still the wrong product is at the top with most (botted) upvotes. Companies realized that ppl look for opinions on reddit -> that created de…

Closed communities is where it's at... But they have the obvious inherent problem: they don't scale. If a closed community lets in too many people, it becomes a somewhat obscured open community, with all the downsides. And if it doesn't, well, you are unlikely to get in on a topic you are interested in.

That's something I've been grappling with recently, and the solution I landed on as "good enough" (though far from perfect) is to have a community where the ability to post and interact is closed, but the ability to view the content is open.

Some subreddits have tried a variation of this, such as the Economics subreddit granting a "Bureau Member" flair for individuals who have proven their knowledge of economics in long form posts, but without taking it farther than just extra flair. There's no weekly threads for Bureau Members only to post, or anything like that.

The issue then becomes how do you set up a judging body that determines the criteria for who is allowed to interact, and prevent that body from being co-opted in some way. Not an easy task, especially for a social media company where there's absolutely perverse incentives at play.

Re: Reddit can't build a better search engine

#278

Post by jiveturkey »

insightful.

Even Amazon can't build a better search engine. Besides reddit, I always now have to use Google to find specific products on Amazon. I can't understand why Amazon doesn't return them. These same products are not discoverable on Amazon by browsing either. But somehow Google is able to catalog them.

eg: 'ohto ceramic refill blue'

Post reply on HN