Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The world needs a non-profit search engine

daoudclarke.net

Post reply on HN
Print view View original

311–320 of 396 posts

Re: The world needs a non-profit search engine

#311

Post by Nextgrid »

DisjointedHunt wrote: post #161
larsrc wrote: post #155

Disclaimer: I work for Google, though far away from Search. Regardless of search engine design, there's HUGE money in SEO. Any successful search engine will be gamed. Do you have the developer power to go red-queen against all the large companies in the world?

And that money goes to support many industries beyond search itself. The author really needs to get off a computer for a minute and understand the economics of the web as it stands today and how the free ad supported model supports millions of peoples livelihoods before jumping to "This is all bullshit"

> the free ad supported model supports millions of peoples livelihoods

That doesn't necessarily make it right. Online scamming also supports a lot of people's livelihoods.

Re: The world needs a non-profit search engine

#312

Post by thatwasunusual »

8organicbits wrote: post #212
zaarn wrote: post #133

Earlier quoted context omitted.

Then I'll just add 3 million bots to the network (or just enough to have about 50%) and I can guarantee to win the A/B test against an honest client most of the time.

It's an arms race, but this is mostly a question of rate limiting account creation, assigning a trustworthiness score to different accounts, some network analysis to detect coordinated accounts, and having some trusted accounts (run by the project) that can help double check results. After an account loads poisoned data, you can detect this after the attack (user reported spam), and then block (or probably shadow ban…

> but this is mostly a question of rate limiting account creation, assigning a trustworthiness score to different accounts, some network analysis to detect coordinated accounts, and having some trusted accounts (run by the project) that can help double check results.

You make it sound easy. ;)

Re: The world needs a non-profit search engine

#313

Post by happymellon »

8organicbits wrote: post #212
zaarn wrote: post #133

Earlier quoted context omitted.

Then I'll just add 3 million bots to the network (or just enough to have about 50%) and I can guarantee to win the A/B test against an honest client most of the time.

It's an arms race, but this is mostly a question of rate limiting account creation, assigning a trustworthiness score to different accounts, some network analysis to detect coordinated accounts, and having some trusted accounts (run by the project) that can help double check results. After an account loads poisoned data, you can detect this after the attack (user reported spam), and then block (or probably shadow ban…

You can also then find all the other bots that are attempting to poison the well with the same junk and shadow ban them all.

Re: The world needs a non-profit search engine

#314

Post by thatwasunusual »

amelius wrote: post #291
javajosh wrote: post #249

>Google makes $40bn...If I can create something that just a tiny fraction of people find useful, then I can create a huge amount of value. You conflate two meanings of value: monetary value, and intrinsic value. Search engines are intrinsically but not monetarily valuable to users. Search engines are monetarily, but not intrinsically, valuable to advertisers. You can get into trouble when you conflate these two meani…

> It is surprisingly hard to extract monetary value from an intrinsically valuable solution. This is like saying that our entire economic system is backward ...

First an foremost it says that "the search engine economy" is f...ed up.

Re: The world needs a non-profit search engine

#315

Post by Nextgrid »

y42 wrote: post #9

I totally like the idea but I dare to doubt that this would solve the SEO problem. Website owners who are participating in those notorious affiliate programs or earn money with ads will still use the search engine to drag people onto their generic websites, using methodolgies to fit the search engines ranking mechanisms, no matter if they are public or not. SEO and all it's results seem to be immanent to the system.

> notorious affiliate programs

Great, detect the affiliate links and downrank them. Ban it completely if the problem is severe or the products are fake/etc.

> earn money with ads

Similar, detect them and downrank. If a particular ad network is knowingly malicious, ban it entirely.

Re: The world needs a non-profit search engine

#316

Post by ilaksh »

closedloop129 wrote: post #168
ilaksh wrote: post #64

We don't need another centralized search service. We need protocols for publishing and finding information that do not rely on servers.

Can you elaborate? https://yacy.net/ is a decentralized search engine. Are there shortcomings to the protocol?

Yacy is exactly the type of thing I am talking about. When I tried it, did not seem perfect but definitely a better overall paradigm.

It's weird that more people don't know about it.

But also there should be more than one decentralized search engine.

Re: The world needs a non-profit search engine

#317

Post by FrenchDevRemote »

naillo wrote: post #262
FrenchDevRemote wrote: post #261

Earlier quoted context omitted.

YALM is 200GB and require 200GB of GPU Memory to run....

Yeah you picked the hugest SOTA one of them all, but there's smaller ones like this https://bellard.org/libnc/gpt2tc.html that run well even on CPUs that might run well fine tuned specifically on search results (or at least just code queries).

The only significant difference between those models is the amount of data, the main(or the 2nd most important) reason why you're using a search engine is how much data is available on it.

If you want to search through an incredibly limited % of the web then yeah it can be a solution, but even the lamest search engine company out there would outperform a GPT-2 like model running from your laptop.

Re: The world needs a non-profit search engine

#318

Post by Cupertino95014 »

Tell me how this doesn't quickly devolve into a consensus-rules hellscape, where minority views are either ignored or certain minorities are artificially boosted.

There is no way that design choices (especially the ordering of results) can be made in a way that pleases everyone. So either you dumb it down to the point of meaninglessness OR you enforce a mainstream-only ruleset.

Re: The world needs a non-profit search engine

#319

Post by Nextgrid »

solarkraft wrote: post #122
renonce wrote: post #120

The biggest challenge with making a search engine is to combat adversarial SEO. It's an issue that's very easy to be overlooked when you are small, but at Google scale, your enemies have billions of dollars to make from your visitors. I bet Google spends at least as much to combat that, and it's extremely hard to deal with while being open-source. It's useless to call for a non-profit search engine without tackling t…

Are your sure they're combating it? It seems like they've given up.

They haven't given up, they are staying exactly where they need to be.

All the mainstream search engines' priority is to maximize ad clicks/impressions (or collect data to target future impressions), either directly on their own property, or indirectly when linking to websites that embed their ads.

There's no reason why they can't detect ads or analytics and use that as a negative ranking factor (so that all other factors being equal, a non-ad-infested result would rank higher than the ad-infested one), but this would go contrary to their business model.

Re: The world needs a non-profit search engine

#320

Post by Nextgrid »

Pakdef wrote: post #248
laserbeam wrote: post #208

> [google gets 40 billion a year from search.] I can’t even conceive how big it is. Just 1% of 1% of this would be more money than I’d know what to do with ($4m). Ouch. I wish you the best but that statement makes me lose hope. Employees are expensive. Servers aren't exactly cheap either. And unexpected mistakes along theyl way cost a lot.

> Employees are expensive. Google has too many of them and it's probably why Google Search is not improving.

I wouldn't say Google Search isn't improving because of the number of employees. Google Search is exactly what it needs to be to serve Google's interests - it's just that their interests don't align with yours.
Post reply on HN