Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The next Google

dkb.io

Post reply on HN
Print view View original

231–240 of 561 posts

Re: The next Google

#231

Post by marginalia_nu »

sdoering wrote: post #204
marginalia_nu wrote: post #86

Earlier quoted context omitted.

All of this is a long series of solvable problems. I should know, I've dabbled in solving most of them. This is why I suggest actually taking a stab at it before you dismiss it as impossible. There are some problems that aren't as big as they seem. Parts of an SPA can't be reliably linked to anyway even if you find interesting text there, so you can just leave them out of the index. Likewise, there isn't as great of…

> Get rid of that noise and your hardware goes a lot longer. What qualifies? What defines signal, what noise? I agree, that a lot (probably nearly all) pages will receive very, very little traffic/search requests. But are these therefore not relevant? > I'm running a search engine on consumer hardware out of my living room that can index 100 million documents. That's extremely cool. I would love to know more. To me a…

I think I was editing the comment while you were replying. Sorry about that. I was just adding to it though, didn't really rug pull on your response so I think it's fine.

> What qualifies? What defines signal, what noise? I agree, that a lot (probably nearly all) pages will receive very, very little traffic/search requests. But are these therefore not relevant?

Now this is a proper difficult problem with (probably) fairly subjective answers. I do however think it's something that warrants serious investigation. It's probably a decent candidate for a machine learning model combined with some manual tweaking for sites similar to wikipedia or github that have absurd amounts of parallel historical content.

Developing heuristics for this is a bit of a hobby horse of mine. It feels tantalizingly almost doable with just a little bit more resources and time than I have.

> That's extremely cool. I would love to know more. To me an impressive feat already.

Yeah it's at https://search.marginalia.nu/>. I've built all the software myself from scratch in Java[1], and I'm doing my own crawling and indexing. The machine it's on is a Ryzen 3900X with 128 Gb RAM. Most of the index is on a single 1 Tb consumer grade SSD.

I do use a MariaDB database for some metadata, but I think it will have to go as its hardware demands is becoming a serious bottleneck.

[1] Despite using Java, I should say regarding the index. This is approaches sunk cost at this point. Building a search engine index is not something Java is at all suitable for, its limited low-level I/O capabilities is incredibly handicapping.

Re: The next Google

#232

Post by swethmandava »

I might be biased but I see myself depending on trusted sources - no SEO links, no spam! I want yelp for restaurants, stackoverflow for coding questions, reddit for opinions or allrecipes for recipes.

So I love you.com - it has an app for everything!

Re: The next Google

#233

Post by usui »

xtracto wrote: post #206
Nextgrid wrote: post #167

Earlier quoted context omitted.

Making spam unprofitable would be a good first step. Rather than trying to detect whether something is spam or not, just target how spam sites are funded: ads, analytics, affiliate links, etc and use those as a negative ranking signal. You'll still get spam if that's all that matches your query, but now all it takes is for someone to make a page matching the query without the aforementioned items to outrank the spam…

> Making spam unprofitable would be a good first step. Rather than trying to detect whether something is spam or not, just target how spam sites are funded: ads, analytics, affiliate links, etc and use those as a negative ranking signal. This would be a really interesting experiment: A search engine that ranks websites by the amount of Ads and other spam that they contain.

Kagi is experimenting with this already. It's one of the options to filter trackers in the page.

Re: The next Google

#234

Post by desiarnezjr »

focom wrote: post #21

> DuckDuckGo and Bing are not true alternatives – they’re just worse versions of Google. Glad to disagree, DuckDuckGo is good enough if not great.

Almost. Annoying are things like geographic context in results that are terrible (for me) but it's more adapting queries.

It's sad what today's web has become. Each year you see it degrade and there's no real fix I'm afraid.

Re: The next Google

#236

Post by hackernewds »

legohead wrote: post #98

I just went on a mini vacation to Vegas, and was thinking how nice it would be to just call up someone and ask for some simple advice. I did a bunch of online research before I booked things, but man was it painful. There are so many copy-cat blogs who just throw together a bunch of basic information with no real research done in order to get those clicks/adwords. For example: best hotel pool in vegas. Seems simple e…

Amex has a really good concierge that handles this. And it's free with most cards. Absolutely underrated, especially when you're traveling internationally

Re: The next Google

#237

Post by marginalia_nu »

Nextgrid wrote: post #228
marginalia_nu wrote: post #86

Earlier quoted context omitted.

All of this is a long series of solvable problems. I should know, I've dabbled in solving most of them. This is why I suggest actually taking a stab at it before you dismiss it as impossible. There are some problems that aren't as big as they seem. Parts of an SPA can't be reliably linked to anyway even if you find interesting text there, so you can just leave them out of the index. Likewise, there isn't as great of…

Out of curiosity, how much disk space does your index currently use, and what's the storage hardware (SSD or spinning rust)?

The reverse index is 180 Gb, on an SSD. I do think using SSDs are a major part of why this is possible on consumer hardware. I'd need a lot of spinning rust to get the sub-100ms response times I can get it to when the index is warmed up.

Should be said I do wear through this SSD at a pretty alarming rate. I'm at 193 TBW on this disk since I started using it as an index less than a year ago.

I do have a bunch of mechanical drives I use for archiving and as intermediate working areas as well, but the index itself is on an SSD.

Re: The next Google

#238

Post by dannywarner »

polote wrote: post #199
dannywarner wrote: post #112

Earlier quoted context omitted.

Did you try Andi search? It returns full search results alongside the answer and has full preview content for many of the websites, and the content is from the websites not Google or Bing's snippets. It's completely different to the way you described it. My biggest problem with it is that the alpha version still has errors and is weak for searches like local businesses. Also, big problem is I'm in Australia and every…

It feels like the engine understand well the query, but for test searches it didnt show better results than Google, it actually gave me almost the same answers. So if they use their own index thats pretty good. But not the Next Google

You might be missing the point. Try a difficult question instead of a regular web search. Something like this. "Why did Twitter add Elon Musk to its board of directors, and what impact did that have on twitter's share price?"

Then click on "View in Reader" on the results. This is not like Google dude.

People on the Discord are trying crazy questions like that and reporting what it does good or bad on. It screws some up but it is cool. It finds stuff Google doesn't.

Re: The next Google

#240

Post by richardsocher »

spiderice wrote: post #212
fallat wrote: post #88

Earlier quoted context omitted.

It's actually so good I plan to pay when they start charging.

It's really good in some ways, and very lacking (for me) in others. The search is fine. Good enough that I would switch. However, if I was out and about and quickly needed directions to Walmart, my normal flow with Google is Pull out phone -> Safari -> type "Walmart" -> click on the map -> Maps app opens and starts guiding me When I switched to Kagi, the flow went like Pull out phone -> Safari -> type "Walmart" -> to…

We found similar issues at https://you.com a while back. We just had to be good for more query families. Now we have both the walmart locations in a map app, eg https://you.com/search?q=walmart and coding related useful results, eg. https://you.com/search?q=how%20do%20I%20find%20all%20files%2...

or https://you.com/search?q=pyspark%20filter%20array%20element

Post reply on HN