Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Two upstart search engines are teaming up to take on Google

wired.com

Post reply on HN
Print view View original

131–140 of 299 posts

Re: Two upstart search engines are teaming up to take on Google

#131

Post by eternal_braid »

IAmGraydon wrote: post #120
ThinkBeat wrote: post #91

For me, it seems the direction for search is going towards AI sites. (Gemini, ChatGPT) Trying to reinvent Google/ Search in 2024 seems a bit like jumping the shark

I agree, and I think it's half and half for me. Many times when I search, my query ends with a question mark. They're questions, looking for a simple answer. Those are the searches I have been going to LLMs for more and more lately. As the LLMs get better and hallucinate less, this is becoming more and more viable. Other searches are looking for a page. Where to buy a particular component. Where to download a particu…

I would argue that answer seeking queries have lower commercial value, compared to queries about products or services.

Re: Two upstart search engines are teaming up to take on Google

#132

Post by knadh »

cphoover wrote: post #37

Why isn’t there a distributed, decentralized or open index that all of these startups can utilize? I understand that these startups are all are focusing in on different problem areas, but doesn’t it make sense to have something like open street maps so that all of these companies can share their compute resources in order to maintain something competitive with the big guys? Or even if it’s not fully decentralized the…

Pretty much exactly what I have been thinking lately. Write about it recently here: https://nadh.in/blog/decentralised-open-indexes/

Re: Two upstart search engines are teaming up to take on Google

#133

Post by BLKNSLVR »

PittleyDunkin wrote: post #25
FredPret wrote: post #11

This is somehow not about Perplexity. Like many, I tried many other search engines, starting with DuckDuckGo way back when. I always ended up Googling (or !g… -ing). Perplexity is the first one that consistently works for both code questions (what’s this error message) and local questions (where’s my nearest store X and when do they close). Now they just need to speed it up a bit - Google queries are effectively inst…

I would strongly prefer a search engine that searches and doesn't attempt to proactively answer questions.

I've recently started using Perplexity and find that I've started using it for finding answers to specific questions, where I'll still use traditional search to find "a few sites on which I may be able to cobble together a useful set of information about ".

So, for me, "search" has fragmented into two separate use cases, one of which is served by the new not-really-a-search-engine.

Re: Two upstart search engines are teaming up to take on Google

#134

Post by ElijahLynn »

I've been using the chatGPT browser extension for a week now. It replaces Google search for me in Brave.

It was a little odd at first but now I find I won't go back. I love I can ask follow up questions, and it shows the listing results on the right just like a search engine. But it also gives my a such summary on the left, again, with the ability to ask follow up questions.

Anyone trying to build a search engine to provide a Google-like experience is going to fail, in my opinion.

chatGPT search is a game changer. I'll pay $20 month for this forever.

Re: Two upstart search engines are teaming up to take on Google

#135

Post by nickpsecurity »

heraclius1729 wrote: post #130
nickpsecurity wrote: post #114

Earlier quoted context omitted.

Search is still better for getting to specific, existing documents you need. Even the RAG people have been finding that out with hybrid models becoming more popular over time. I also think you can update search indexes more cheaply than further pretraining LLM’s.

Not to mention that the cost per search in terms of compute and energy is so much smaller for web search than for running an LLM. I forget the exact numbers now, but it was orders of magnitude as I recall. Search engines are just cheaper to run. I don't know that there's a good, long term model for a free LLM-based search replacement because of how much higher the operating costs are, ad supported or not.

On top of that, search usually uses CPU instead of GPU. A large infrastructure with CPU’s is easier to reuse for jobs other than search.

Re: Two upstart search engines are teaming up to take on Google

#136

Post by ricardo81 »

cphoover wrote: post #37

Why isn’t there a distributed, decentralized or open index that all of these startups can utilize? I understand that these startups are all are focusing in on different problem areas, but doesn’t it make sense to have something like open street maps so that all of these companies can share their compute resources in order to maintain something competitive with the big guys? Or even if it’s not fully decentralized the…

A shared index would surely be nice (Common crawl is perhaps an example of one that could be used) but say you had 10 search engines running from it. One decides a page is very important and updates constantly, so should be fetched every 30 minutes. Another search engine decides a page is spam and doesn't need to be recrawled. There's backend choices that affect the shape and crawl directions of the index.

Then things like whether the crawler should render the page (Using the end DOM content rather than the original source), does it do any tokenisation of the content, store other metrics etc, or does that need to be done by the end search engines.

Also there's issues with crawling Reddit, sites behind Cloudflare etc that others have went into more detail on this comment page.

Re: Two upstart search engines are teaming up to take on Google

#137

Post by analog31 »

arnaudsm wrote: post #4

Every new search engine I've seen was a Bing wrapper with sometimes light reranking. I understand that competing with Google was borderline impossible a decade ago. But in 2024, we have cheap compute, great OSS distributed DBs, powerful new vector search tech. Amateur search engines like Marginalia even run on consumer hardware. CommonCrawl text-only is ~100TB, and can fit on my home server. Why is no company buildin…

It occurs to me that competing with Google on their home turf and at their scale might be impossible. But doing what Google was good at, when they just a search engine, might not be all that much harder today than it was back then. And it may fly "under the radar" of Google's current business priorities.

Re: Two upstart search engines are teaming up to take on Google

#138

Post by dmafreezone »

arnaudsm wrote: post #4

Every new search engine I've seen was a Bing wrapper with sometimes light reranking. I understand that competing with Google was borderline impossible a decade ago. But in 2024, we have cheap compute, great OSS distributed DBs, powerful new vector search tech. Amateur search engines like Marginalia even run on consumer hardware. CommonCrawl text-only is ~100TB, and can fit on my home server. Why is no company buildin…

Isn’t it obvious? Look around. Hackers have been stamped out even on hacker news. Now it’s all FAANG lifers and MBA or VC types trying their hand at grifting. Nothing good comes of that. Whoever is making the next thing “for real” just gets acquired and shut down.

Moreover, get out of your echo chamber and you’ll see that for a majority of humanity Google is the internet. You have to supplant the utility, not just the brand. Most businesses cannot handle the latter let alone the former. If you want something to replace Google, you have to think about replacing the internet itself. But not many are that bold.

Re: Two upstart search engines are teaming up to take on Google

#139

Post by rajup »

ElijahLynn wrote: post #134

I've been using the chatGPT browser extension for a week now. It replaces Google search for me in Brave. It was a little odd at first but now I find I won't go back. I love I can ask follow up questions, and it shows the listing results on the right just like a search engine. But it also gives my a such summary on the left, again, with the ability to ask follow up questions. Anyone trying to build a search engine to…

It was the exact opposite for me interestingly. I couldn’t stand chatGPT search beyond a few days, I guess because my search patterns don’t need a bunch of useless words thrown in (locations, very specific keywords that lead to websites etc). The speed is also a big factor. Google search feels almost instantaneous and watching ChatGPT search emit a bunch of tokens before getting to what I need drove me to frustration.

Re: Two upstart search engines are teaming up to take on Google

#140

Post by Zambyte »

malfist wrote: post #14
pseudocomposer wrote: post #10

Earlier quoted context omitted.

I’m my second year into Kagi and loving it. I actually just upgraded to get Kagi Assistant (basically, cloud access to every LLM out there). But the search alone is worth every penny, and it’s built/operated fully in-house as far as I know. https://kagi.com

As much as I like kagi and wish it success, it's not a search engine from scratch. Kagi uses other search engines (Google and Bing) wraps them and does a light reranking

Light reranking is a huge understatement, given the amount of preferences they let you manually set for ranking.
Post reply on HN