Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Two upstart search engines are teaming up to take on Google

wired.com

Post reply on HN
Print view View original

41–50 of 299 posts

Re: Two upstart search engines are teaming up to take on Google

#41

Post by eitland »

stephantul wrote: post #6
arnaudsm wrote: post #4

Every new search engine I've seen was a Bing wrapper with sometimes light reranking. I understand that competing with Google was borderline impossible a decade ago. But in 2024, we have cheap compute, great OSS distributed DBs, powerful new vector search tech. Amateur search engines like Marginalia even run on consumer hardware. CommonCrawl text-only is ~100TB, and can fit on my home server. Why is no company buildin…

Because it is super expensive and difficult to keep an index up to date. People expect to be able to get current events, and expect search results to be updated in minutes/seconds.

The Swede behind search.marginalia.nu has had a working search engine running at a single desktop class computer in a living room, all programmed and maintained on his spare time, that was so good that in its niches (history, programming, open source comes to mind) it would often outshine Google.

Back before I found Kagi I used to use it everytime Google failed me.

So, yes, given he is the only one I know who manages this it isn't trivial.

But it clearly isn't impossible or that expensive either to run an index of the most useful and interesting parts of internet.

Re: Two upstart search engines are teaming up to take on Google

#42

Post by lexlambda »

> “We could de-rank results from unethical or unsustainable companies and rank good companies higher,” Kroll says of the eco-minded Ecosia.

Understandable knowing Ecosias goals, but I find it rather concerning their vision of a better search involves deciding what is good and bad.

Ranking by quality (against spam & SEO sites) is fine, but it should be applied equally to all Websites, and not target specific companies.

Re: Two upstart search engines are teaming up to take on Google

#43

Post by arnaudsm »

marginalia_nu wrote: post #38
stephantul wrote: post #6

Earlier quoted context omitted.

Because it is super expensive and difficult to keep an index up to date. People expect to be able to get current events, and expect search results to be updated in minutes/seconds.

No search engine is refreshing every website every minute. Most websites don't update frequently, and if you poll them more than once every month, your crawler will get blocked incredibly fast. The problem of being able to provide fresh results is best solved by having different tiers of indices, one for frequently updating content, and one for slowly updating content with a weekly or monthly cadence. You can get a l…

Big fan of your work Viktor, thanks for everything you build and how much you document it

Re: Two upstart search engines are teaming up to take on Google

#44

Post by beAbU »

bilekas wrote: post #22

It shouldn't be too hard to achieve what Google were good at before. Their recent search results for me (last 6-12 months) have been so far removed from what I'm searching it felt like a meme. Even after rephrasing things, more details, special quotations etc that everyone knows as the 'search tricks' the results are terrible.

I moved to DDG a couple of years ago, and initially, I found myself often using the `!g` switch, but I honestly can't recall the last time I needed to do that. Only when I'm shopping do I find the goog to be a slightly better tool for finding products sold by niche suppliers.

I honestly think google's monopoly on search at this time is 100% powered by momentum, there is almost no other reason to use it over something like DDG or hell, even Bing!

Re: Two upstart search engines are teaming up to take on Google

#45

Post by arnaudsm »

unionpivo wrote: post #35
arnaudsm wrote: post #4

Every new search engine I've seen was a Bing wrapper with sometimes light reranking. I understand that competing with Google was borderline impossible a decade ago. But in 2024, we have cheap compute, great OSS distributed DBs, powerful new vector search tech. Amateur search engines like Marginalia even run on consumer hardware. CommonCrawl text-only is ~100TB, and can fit on my home server. Why is no company buildin…

Because nowdays more than ever content you need is in silos. Your facebooks/twiters/instagram/stack overflow/reddit ... And they all have limited expensive api's, and have bulk scrapping detection. Sure you can clobber together something that will work for a while, but you can't runn a buissness on that. Aditionaly most paywalled sites (like news) explicitly whitlist google and bing, and if someone cretes new site, t…

This is the best (and saddest) answer. LLMs break the social contract of the internet, we're in a feudalisation process.

The decentralized nature of the internet was amazing for businesses, and monopolization could ruin the space and slow innovation down significantly.

Re: Two upstart search engines are teaming up to take on Google

#46

Post by marginalia_nu »

eitland wrote: post #41
stephantul wrote: post #6

Earlier quoted context omitted.

Because it is super expensive and difficult to keep an index up to date. People expect to be able to get current events, and expect search results to be updated in minutes/seconds.

The Swede behind search.marginalia.nu has had a working search engine running at a single desktop class computer in a living room, all programmed and maintained on his spare time, that was so good that in its niches (history, programming, open source comes to mind) it would often outshine Google. Back before I found Kagi I used to use it everytime Google failed me. So, yes, given he is the only one I know who manages…

I think the problem with search is that while it's relatively doable to build something that is competitive in one or a few niches, Google's real sticking power is how broad their offering is.

Google search has seamless integration with maps, with commercial directories, with translation, with their browser, with youtube, etc.

Even though there's more than a few queries they leave something to desire, the breadth of queries they can answer is very difficult to approach.

Re: Two upstart search engines are teaming up to take on Google

#47

Post by rkangel »

chabad360 wrote: post #20
malfist wrote: post #14

Earlier quoted context omitted.

As much as I like kagi and wish it success, it's not a search engine from scratch. Kagi uses other search engines (Google and Bing) wraps them and does a light reranking

No they do not, they have their own indexes. https://help.kagi.com/kagi/search-details/search-sources.htm...

That page includes the text "Our search results also include anonymized API calls to all major search result providers worldwide".

They source results from lots of places including Google. One way that you can confirm this is to search for something that only appears in a recent Reddit post. Google has done a deal with Reddit that they're the only company allowed to index Reddit since the summer.

DuckDuckGo gets no answers if you specify only results from the last week: https://duckduckgo.com/?q=caleb+williams+site%253Areddit.com...

Kagi is fine if you do the same: https://kagi.com/search?q=caleb+williams+site%3Areddit.com&d...

edit: I don't think this is a bad thing for Kagi. I'm a very happy subscriber, and it's nice for me that I still get results from Reddit. They're very useful!

Re: Two upstart search engines are teaming up to take on Google

#48

Post by eitland »

iamacyborg wrote: post #15
spamdotlol wrote: post #2

Google isn't even a competitor in the search space anymore. They've been completely unusable for a decade.

A cursory glance at their market share in the search space clearly says that’s not true. For a big site I help run, we’re getting about 8.2x the impressions on Google compared to Bing.

To interpret GP charitably I think they mean that Google is there not because they are a good search engine these days but because of inertia.

For you as a site owner, Google is the best: it delivers the impressions.

For a user who wants to search, Google has gone downhill since around 2009 and the only thing that confuse me is why DDG - who initially felt better - chose to run after Google down the path of insisting to give me results for things I didn't ask for.

(The usual answer is: "It is so much harder than in 2009 and SEO is so crazy these days, that nobody can do it, not even Google", to which I have to point out that even marginalia - run by one Swede - manages to do it in the niches it prioritizes.)

Re: Two upstart search engines are teaming up to take on Google

#49

Post by iterance »

lexlambda wrote: post #42

> “We could de-rank results from unethical or unsustainable companies and rank good companies higher,” Kroll says of the eco-minded Ecosia. Understandable knowing Ecosias goals, but I find it rather concerning their vision of a better search involves deciding what is good and bad. Ranking by quality (against spam & SEO sites) is fine, but it should be applied equally to all Websites, and not target specific companies…

The measure of success of a search engine is how quickly I leave it with the info I want in hand.

I too find this a bit strange. Downranking results that would otherwise be naturally highly ranked seems only to inhibit the operation of the search engine.

Re: Two upstart search engines are teaming up to take on Google

#50

Post by FredPret »

PittleyDunkin wrote: post #25
FredPret wrote: post #11

This is somehow not about Perplexity. Like many, I tried many other search engines, starting with DuckDuckGo way back when. I always ended up Googling (or !g… -ing). Perplexity is the first one that consistently works for both code questions (what’s this error message) and local questions (where’s my nearest store X and when do they close). Now they just need to speed it up a bit - Google queries are effectively inst…

I would strongly prefer a search engine that searches and doesn't attempt to proactively answer questions.

I like that too but the benefit of the answer model is near-zero spam. Perplexity gives an answer with references, effectively combining search with the answer.
Post reply on HN