This is quite welcomed! I like seeing more competition in such spaces. What would be really interesting is if DDG got involved...and those 3 entities (DDG, Ecosia, and Qwant) worked together. I know DDG is simply a wrapper around Bing...but i think one of these others has a similar Bing back-end...so if someoine who has the mindshare of DDG got into that, i think woul;d make it even more interesting. I know Google is quite the behemoth, but still, would be great to see some shake-ups (assuming of course that the results are good and provide value, etc.).
Two upstart search engines are teaming up to take on Google
71–80 of 299 posts
Re: Two upstart search engines are teaming up to take on Google
#72Earlier quoted context omitted.
As much as I like kagi and wish it success, it's not a search engine from scratch. Kagi uses other search engines (Google and Bing) wraps them and does a light reranking
Kagi is also building their own index at the background, and mixes these indexes as you search. When I search Kagi for "Hacker News", results start with this fine text: 65 relevant results in 1.09s. 47% unique Kagi results. So, other indexes are fillers for Kagi's own index. They can't target their bots to places, because they don't have the users' search history. They can only organically grow and process what they…
How is it possible that a search for "Hacker News" produces only 65 results? There are thousands of pages out there with that exact phrase on it (including many sub-pages of this site).
The first result is almost assuredly the right one, but either they're ruling out a lot of pages as not-what-you-meant, or their index is really small.
Re: Two upstart search engines are teaming up to take on Google
#73> “We could de-rank results from unethical or unsustainable companies and rank good companies higher,” Kroll says of the eco-minded Ecosia. Understandable knowing Ecosias goals, but I find it rather concerning their vision of a better search involves deciding what is good and bad. Ranking by quality (against spam & SEO sites) is fine, but it should be applied equally to all Websites, and not target specific companies…
> I find it rather concerning their vision of a better search involves deciding what is good and bad the entire purpose of a search engine is to do this, you've been grossly confused about the entire space if you think this isn't exactly what everyone is trying to do.
[deleted]
Re: Two upstart search engines are teaming up to take on Google
#74> “We could de-rank results from unethical or unsustainable companies and rank good companies higher,” Kroll says of the eco-minded Ecosia. Understandable knowing Ecosias goals, but I find it rather concerning their vision of a better search involves deciding what is good and bad. Ranking by quality (against spam & SEO sites) is fine, but it should be applied equally to all Websites, and not target specific companies…
Yes, people aren't going to use a search engine that politically skews the results. It will end up as a tiny website for a very narrow niche of person, similar to eg Mastodon.
Re: Two upstart search engines are teaming up to take on Google
#75Every new search engine I've seen was a Bing wrapper with sometimes light reranking. I understand that competing with Google was borderline impossible a decade ago. But in 2024, we have cheap compute, great OSS distributed DBs, powerful new vector search tech. Amateur search engines like Marginalia even run on consumer hardware. CommonCrawl text-only is ~100TB, and can fit on my home server. Why is no company buildin…
I’m my second year into Kagi and loving it. I actually just upgraded to get Kagi Assistant (basically, cloud access to every LLM out there). But the search alone is worth every penny, and it’s built/operated fully in-house as far as I know. https://kagi.com
I loved kagi, but their cost is prohibitive for me.
The only way there is a chance for me to afford Kagi might be to buy "search credit" without a subscription and without minimum consumption. And then it would only be good if they allowed more than 1000 domain rules and showed more results (when available)
Re: Two upstart search engines are teaming up to take on Google
#76Is it just me or does it feel like a lot of Big Tech empires are immovable? There are no winner takes all markets now. Massive incumbents are not replaced by another even more massive startup that gobbles up the market, but rather a collection of specialized alternatives. So Google is not being replaced, but is slowly losing bits and pieces to ChatGPT, Kagi, Perplexity, and others. Facebook/X lose a little to BlueSky, Mastodon, Threads and Telegram. Netflix by a dozen streamers. Etc. The age of massive upheavals is over? I can’t remember a single Big Tech company that went under completely in the last decade, just slowly became “not the only one”.
Re: Two upstart search engines are teaming up to take on Google
#77This is quite welcomed! I like seeing more competition in such spaces. What would be really interesting is if DDG got involved...and those 3 entities (DDG, Ecosia, and Qwant) worked together. I know DDG is simply a wrapper around Bing...but i think one of these others has a similar Bing back-end...so if someoine who has the mindshare of DDG got into that, i think woul;d make it even more interesting. I know Google is…
qwant had bing as a backend a few years back iirc.
Re: Two upstart search engines are teaming up to take on Google
#78Every new search engine I've seen was a Bing wrapper with sometimes light reranking. I understand that competing with Google was borderline impossible a decade ago. But in 2024, we have cheap compute, great OSS distributed DBs, powerful new vector search tech. Amateur search engines like Marginalia even run on consumer hardware. CommonCrawl text-only is ~100TB, and can fit on my home server. Why is no company buildin…
"CommonCrawl [being] text-only is only ~100TB, and can fit on my home server." Are any individual users downloading CC for potential use in the future? It may seem like a non-trivial task to process ~100TB at home today but in the future processing this amount of data will likely seem trivial. CC data is available for download to anyone but, to me, it appears only so-called "tech" companies and "researchers" are grab…
Is Common Crawl updated frequently?
Re: Two upstart search engines are teaming up to take on Google
#79Every new search engine I've seen was a Bing wrapper with sometimes light reranking. I understand that competing with Google was borderline impossible a decade ago. But in 2024, we have cheap compute, great OSS distributed DBs, powerful new vector search tech. Amateur search engines like Marginalia even run on consumer hardware. CommonCrawl text-only is ~100TB, and can fit on my home server. Why is no company buildin…
I’m my second year into Kagi and loving it. I actually just upgraded to get Kagi Assistant (basically, cloud access to every LLM out there). But the search alone is worth every penny, and it’s built/operated fully in-house as far as I know. https://kagi.com
They are highly dependent on outside search engines. Someone from Kagi gave an explanation on HN of their search costs and why they can't go lower, and calling the Google API on many (most?) search queries was a major driver of their costs.
It's great that they are developing their own index, but I'm skeptical that it makes up more than a tiny fraction of what they can get from Google/Bing. DDG has been making similar claims for years but are still heavily reliant on Bing.
This isn't to knock on upstart search engines. I think that Google Search has declined massively over the past 5-10 years and I rarely use it. More competition is sorely needed, but we should be be clear eyed about the landscape.
Re: Two upstart search engines are teaming up to take on Google
#80Earlier quoted context omitted.
> Bought tech Could you please elaborate?
Brave search is a continuation of cliqz. A German company that developed a proper search engine, with an independent index. They shut down, but the tech got sold off. Cliqz was the first time for me that a Google alternative actually worked really well - and it, or now brave search, is what parent was asking for :)
Yet we're still back to Larry Page and Sergey Brin's conclusion in their "The Anatomy of a Large-Scale Hypertextual Web Search Engine" research paper[0]:
> We expect that advertising funded search engines will be inherently biased towards the advertisers and away from the needs of the consumers.
Brave Search Premium hasn't been around nearly as long as their free tier serving ads, and I'm not confident this conflict of interest is gone.
Having independent indexes is a win regardless though.
[0] https://snap.stanford.edu/class/cs224w-readings/Brin98Anatom...