Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The world needs a non-profit search engine

daoudclarke.net

Post reply on HN
Print view View original

211–220 of 396 posts

Re: The world needs a non-profit search engine

#211

Post by solarkraft »

dageshi wrote: post #177
solarkraft wrote: post #121

The internet, somewhat ironically, really needs a search engine that works in the current day. You can't find anything anymore. It's like Google has been un-invented. Hopefully some day soon the internet will be searchable again. Thanks to everyone involved in attempting to make this happen (preferably in a non-profit-maximized way). (Said before at https://news.ycombinator.com/item?id=32034390 )

I have a feeling that people stopped writing actual good website content for certain topics and moved to places like youtube instead. Youtube frankly offers better monetisation and most importantly easier user retention via subscriptions.

While monetization is certainly a great thing for a group of people wanting to go after things they and others enjoy, it's also exactly what incentivizes people to game the system and pretty much what brought us into our current place of spam, spam more spam and barely any way to discover things that are actually good.

I tend to think the quality of the content tends to be better when people don't think about stuff like user retention or subscriptions, but rather how it will actually reach people that care. Good search/curation is a key component for that.

Of course such a world free of implicit monetization will require it to be explicit (Patreon-style), but that should massively realign incentives.

Re: The world needs a non-profit search engine

#212

Post by 8organicbits »

zaarn wrote: post #133
happymellon wrote: post #59

Earlier quoted context omitted.

Get 3 people to scrape it and see if there are significant differences. Some might, because of A/B testing or news updating, but even updating news will get a positive similar page and those that don't should probably fall into an exceptions category until it can be determined what can be done about it. Maybe a flag in the URL to give you a static page or just accept that it changes often enough that even faked pages…

Then I'll just add 3 million bots to the network (or just enough to have about 50%) and I can guarantee to win the A/B test against an honest client most of the time.

It's an arms race, but this is mostly a question of rate limiting account creation, assigning a trustworthiness score to different accounts, some network analysis to detect coordinated accounts, and having some trusted accounts (run by the project) that can help double check results. After an account loads poisoned data, you can detect this after the attack (user reported spam), and then block (or probably shadow ban) the malicious account.

Re: The world needs a non-profit search engine

#213

Post by daoudc »

O__________O wrote: post #201

Worth noting attempts at non-profit search engines are not new. In 2015, Wikimedia Foundation attempted to start one called the “Knowledge Engine” using at least $250,000 from a grant. Wikipedia likely started the project as a response to Google’s use of “knowledge panels” based on Wikipedia Creative Commons license alongside search results in 2012, which reduced traffic to Wikipedia. https://en.m.wikipedia.org/wiki/…

Fascinating, thanks for this! Looks like it became controversial because of a lack of transparency.

Re: The world needs a non-profit search engine

#214

Post by 8organicbits »

zaarn wrote: post #210
closedloop129 wrote: post #207

Earlier quoted context omitted.

Then OP has to do things that don't scale: Review some pages and identify a subset that can be trusted. Then OP can compare their downloads to new accounts and mark the bots.

Then the botnet will just be honest for like a year before it abuses the network. Even better because now honest new clients can be kicked as they disagree with the bot majority. So now the network bleeds users.

Checking which account is honest isn't too hard, you detect that there is a "problematic mismatch" between two clients. So the project runs their own client to check. If one has an exact match, then you'd question the other.

There is a challenge for sites that serve different content based on GeoIP, A/B testing, dynamic content, etc. So some human review of the diff may help check for malice. If there's literally spam, human review would clearly detect this and that bot is distrusted.

Re: The world needs a non-profit search engine

#215

Post by zaarn »

8organicbits wrote: post #214
zaarn wrote: post #210

Earlier quoted context omitted.

Then the botnet will just be honest for like a year before it abuses the network. Even better because now honest new clients can be kicked as they disagree with the bot majority. So now the network bleeds users.

Checking which account is honest isn't too hard, you detect that there is a "problematic mismatch" between two clients. So the project runs their own client to check. If one has an exact match, then you'd question the other. There is a challenge for sites that serve different content based on GeoIP, A/B testing, dynamic content, etc. So some human review of the diff may help check for malice. If there's literally spa…

Then I'll simply use more bots to get 80% of the network, then I can almost always win any disagreements and your "problematic mismatch" never triggers.

Plus I can now cause you to have to run your own crawler anyway and either slow progress or cost you a lot of money.

Re: The world needs a non-profit search engine

#216

Post by jonathanstrange »

bsenftner wrote: post #184
larsrc wrote: post #155

Disclaimer: I work for Google, though far away from Search. Regardless of search engine design, there's HUGE money in SEO. Any successful search engine will be gamed. Do you have the developer power to go red-queen against all the large companies in the world?

Actually, it is entire possible to create search criteria that cannot be gamed. That is what Google Research's various arms ought to be working on. But we all know if is far more profitable to have a manipulatable system.

Could you expand on this a bit more? I don't see any obvious way to develop non-manipulable search criteria, I was suspecting that there might even be an impossibility theorem about this (which would depend a lot on the exact formulation, though), and I'd like to know what you have in mind.

Re: The world needs a non-profit search engine

#217

Post by klibertp »

DisjointedHunt wrote: post #159

This is the kind of idiocy that makes me despise the developer community every time i see something like this. It is one of my pet peeves, so if you're going to have an opinion on this comment, please, read the whole thing. The ad supported free internet is one of the most important business models the world has arguably ever seen. Very few can argue with the fact that poor kids in developing countries over the past…

> I'm astounded at how intelligent people who can write code can simultaneously be so fucking moronic in their grasp of economics.

Absolutely nothing surprising about that. Intelligence without knowledge is not very helpful. If all you know is how to write code, you will suck with other things, even if you're intelligent.

The bigger problem is that people tend to downplay the knowledge that's required to do something, simply because they do not know how much they don't know. It gets worse the more intelligent you are because you're more confident in yourself then.

Case in point: name of this project. I've read it like 10 times on this page already yet I still can't spell it from memory. I could paraphrase:

> I'm astounded at how intelligent people who can write code can simultaneously be so fucking moronic in their grasp of marketing.

(but won't, since, as I said, it's not astounding at all; just for illustration purposes)

Re: The world needs a non-profit search engine

#218

Post by jstummbillig »

nonameking2026 wrote: post #189
larsrc wrote: post #155

Disclaimer: I work for Google, though far away from Search. Regardless of search engine design, there's HUGE money in SEO. Any successful search engine will be gamed. Do you have the developer power to go red-queen against all the large companies in the world?

Allow users to blacklist sites. Share blacklisted sites. Have the option of instead of hiding blacklisted sites entirely - show them in a different column. Have 4 different types of list. Whitelist - highest scoring Not listed - these websites are not ranked. Yellowlist - show but keep in a separate column Blacklist - don't show https://search.brave.com/help/goggles is interesting.

How do you prevent the gaming of blacklisting?

Re: The world needs a non-profit search engine

#219

Post by O__________O »

daoudc wrote: post #213
O__________O wrote: post #201

Worth noting attempts at non-profit search engines are not new. In 2015, Wikimedia Foundation attempted to start one called the “Knowledge Engine” using at least $250,000 from a grant. Wikipedia likely started the project as a response to Google’s use of “knowledge panels” based on Wikipedia Creative Commons license alongside search results in 2012, which reduced traffic to Wikipedia. https://en.m.wikipedia.org/wiki/…

Fascinating, thanks for this! Looks like it became controversial because of a lack of transparency.

Which is the problem with non-profits, the transparency requirements are at best minimal and all non-profit vs for-profit means at a super high level is that there is: no equity distribution, government approves of its mission, and there is no distribution of excess cash flow.

There frequently non-profits that use excess funds to unnecessarily expand beyond the original mission, for example Wikipedia — or that pay staff, especially executives way beyond what most donors realize.

To me, being a non-profit is what it is, I don’t read too much into organization being a non-profit.

Re: The world needs a non-profit search engine

#220

Post by closedloop129 »

joshuamorton wrote: post #115
carschno wrote: post #89

Earlier quoted context omitted.

Not sure I missed the sarcasm here. Why would a non-profit not fight spam etc.? Why would a for-profit fight spam do so? I see the largest internet companies, including Facebook, Twitter, and also Google, fight spam and other harmful content only to the degree absolutely necessary to stay somewhat usable. Which makes sense because it's costly and does not generate profit. I would expect a non-profit, however, to focu…

The general answer here is that a nonprofit cannot maintain the (massive) amount of resources it takes to address spam/abuse/blackhat SEO etc, while a for-profit entity ostensibly has a profit motive to do a decent job, and the resources to do so. When a for profit entity is more successful at fighting abuse, their users are happier and they sell more ads and so can devote more resources to fighting spam. When a nonp…

Like @daoudc writes, Wikipedia shows that the nonprofit can maintain the resources because they attract volunteers.

If you create a search engine where users can report spam and get some form of karma for valid reports that is shown in their social network, then it's quite likely that the users have enough momentum to get ahead of the spam.

Post reply on HN