Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dawn of a new era in Search: Balancing innovation, competition, and public good

blog.kagi.com

Post reply on HN
Print view View original

31–40 of 83 posts

Re: Dawn of a new era in Search: Balancing innovation, competition, and public good

#31

Post by scroot »

IgorPartola wrote: post #30
adamcharnock wrote: post #7

> The Google Search Index is a unique and irreplaceable resource within the digital ecosystem. Mandating fair access to it or treating it as an essential facility could address the core issues... The article estimates the Google Search Index at 12.5PB. If Kagi thinks that is a big enough moat to be the primary target then, well, I suppose they should know. But I'm also skeptical. You could fit that on about 50 Hetzne…

This puts it in enough perspective for me to ask: why doesn’t a university create a public/open source search index? Seems like a way to get a ton of attention. Moreover, archive.org has all the data and data storage capabilities many times over. What prevents them from creating an open source search engine?

Or the Library of Congress, if it had the right appropriations. Google itself started with an NSF grant to explore the future of libraries.

Re: Dawn of a new era in Search: Balancing innovation, competition, and public good

#33

Post by cube2222 »

freefaler wrote: post #27

I've used Kagi for a year and it was better than Google for most of the searches. However I think the model will be changed to something more like Perplexity.ai I've switched to Perplexity and for most of the searches it works better than Kagi. They'd need to add something like this to survive in the long run, because for exploratory searches tools like Perplexity are really good.

I admit I haven’t used Perplexity much, but isn’t that already mostly covered by the researcher[0] and newly added assistant that can do web searching [1]?

[0]: https://help.kagi.com/kagi/ai/assistant.html#research

[1]: https://kagi.com/changelog#4529

Re: Dawn of a new era in Search: Balancing innovation, competition, and public good

#34

Post by rychco »

I’ve been a happy Kagi customer for probably 2 years(?) now. Highlights for me, as a professional:

- I can blacklist low-value domains (such as geeksforgeeks) that dominate the top of many programming searches.

- I can increase/decrease the priority of domains or pin domains to the top of searches, such as official documentation for languages/libraries.

- I can use “Lenses” to filter results for programming/academic/forum results.

Re: Dawn of a new era in Search: Balancing innovation, competition, and public good

#35

Post by dmonitor »

adamcharnock wrote: post #7

> The Google Search Index is a unique and irreplaceable resource within the digital ecosystem. Mandating fair access to it or treating it as an essential facility could address the core issues... The article estimates the Google Search Index at 12.5PB. If Kagi thinks that is a big enough moat to be the primary target then, well, I suppose they should know. But I'm also skeptical. You could fit that on about 50 Hetzne…

I assume that the major hurdle is not storing an equivalently-sized search index, but building one from scratch. Crawling takes time, and Google has had a many years head start.

Re: Dawn of a new era in Search: Balancing innovation, competition, and public good

#36

Post by aDyslecticCrow »

pennybanks wrote: post #16
ldayley wrote: post #2

I've been using (and occasionally paying) Kagi on and off for a couple of years now. I truly think they're building something interesting and valuable! While I haven't agreed with every product decision they've made, the founder is very good at both understanding his business and also explaining their decisions. This is a well crafted explainer of the search business and the monopoly case-- much better for sharing wi…

i honestly cant imagine paying for a search engine with whats out there for free

The other three answers to your question were dumb, so I'll try to do better. Kagi has a free trial of 100 searches without requiring a credit card to sign up. Honest to god, try it. Just bookmark it and search once or twice when Google or Bing is giving you meh results.

As an engineer... googling for stuff is a good fraction of your job. It's quite reasonable to pay to improve that experience. The only real question is what is reasonable to pay.

Re: Dawn of a new era in Search: Balancing innovation, competition, and public good

#37

Post by aDyslecticCrow »

freefaler wrote: post #27

I've used Kagi for a year and it was better than Google for most of the searches. However I think the model will be changed to something more like Perplexity.ai I've switched to Perplexity and for most of the searches it works better than Kagi. They'd need to add something like this to survive in the long run, because for exploratory searches tools like Perplexity are really good.

AI is expensive to host, and I don't trust their puke. I'd rather get a classic result page. Kagi is great.

Re: Dawn of a new era in Search: Balancing innovation, competition, and public good

#38

Post by freediver »

adamcharnock wrote: post #7

> The Google Search Index is a unique and irreplaceable resource within the digital ecosystem. Mandating fair access to it or treating it as an essential facility could address the core issues... The article estimates the Google Search Index at 12.5PB. If Kagi thinks that is a big enough moat to be the primary target then, well, I suppose they should know. But I'm also skeptical. You could fit that on about 50 Hetzne…

> The article estimates the Google Search Index at 12.5PB.

I realize there was a mistake with the estimated number (thanks for pointing out, should be closer to 180 PB for raw crawl data). Since this is speculative and also does not account for other data needed to actually rank pages, hardware to do it in under 500ms at a scale of billions of queries per day and thus can be misleading in terms of true effort to do it, I edited that datapoint out of the article.

You are right, just crawling large number of pages (millions even billions) is indeed straightforward (eg [1]), it is about creating a searchable index of the web scale that has certain quality level that is simply impossible to do anymore for many reasons that would require another article to explain. Microsoft spent $100bn and last 20 years by their own account trying to match it and most people agree it is still not even close. At some point you reach diminishing returns. To use the analogy from the article, it is akin to someone trying to rebuild all of the US railroad network today. Sounds plausible, but not really in practice. That train has left the station in early 2000s.

[1] https://michaelnielsen.org/ddi/how-to-crawl-a-quarter-billio...

Re: Dawn of a new era in Search: Balancing innovation, competition, and public good

#39

Post by rqtwteye »

jpc0 wrote: post #9
poikroequ wrote: post #3

> Apple has stated that Bing does not match Google’s search result quality, and they are unwilling to compromise on user experience by offering subpar results. I wouldn't take this statement at face value. This is most likely a BS PR excuse for Apple to maintain their current deal with Google. I wouldn't expect anything less from any large corporation looking to protect $20 billion in annual revenue.

Universally every single time I've used bing, including this week. My response was to scroll through the entire first oage of results, swear, open google and with the exact same search params what I wanted was the first result. Bing is objectively worse.

Maybe Bing is managed by the same people that haven't managed to get Windows search right despite trying for many, many years?

Re: Dawn of a new era in Search: Balancing innovation, competition, and public good

#40

Post by freediver »

Destiner wrote: post #32

i feel bad for kagi/ddg they are still trying to fight the google by building pretty much the same product while perplexity is obviously in the lead by being ai-first

For many people (me included) being AI-first is a bug, not a feature :)
Post reply on HN