Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The world needs a non-profit search engine

daoudclarke.net

Post reply on HN
Print view View original

291–300 of 396 posts

Re: The world needs a non-profit search engine

#291

Post by amelius »

javajosh wrote: post #249

>Google makes $40bn...If I can create something that just a tiny fraction of people find useful, then I can create a huge amount of value. You conflate two meanings of value: monetary value, and intrinsic value. Search engines are intrinsically but not monetarily valuable to users. Search engines are monetarily, but not intrinsically, valuable to advertisers. You can get into trouble when you conflate these two meani…

> It is surprisingly hard to extract monetary value from an intrinsically valuable solution.

This is like saying that our entire economic system is backward ...

Re: The world needs a non-profit search engine

#292

Post by 8organicbits »

zaarn wrote: post #224
8organicbits wrote: post #212

Earlier quoted context omitted.

It's an arms race, but this is mostly a question of rate limiting account creation, assigning a trustworthiness score to different accounts, some network analysis to detect coordinated accounts, and having some trusted accounts (run by the project) that can help double check results. After an account loads poisoned data, you can detect this after the attack (user reported spam), and then block (or probably shadow ban…

You make it sound easy but companies have been trying to fight this stuff for ages. You can buy a trustworthy residential IP for low cost, you can buy them in bulk in the thousands. All of them are real residential IPs from any ISP of your choosing in any country. You can rent Chrome browsers running over those IPs, directed via remote desktop and accessibility protocols (good luck banning that without running awful…

In terms of spam injection (the concern from up thread) I don't think DDoS is relevant. If the core project manages asking clients to process URLs, they'd just IP ban any client that returns too many results. DDoS is a concern for other reasons though.

I think in this specific case, the spammer is on poor footing. The spammer wants to inject specific content, ideally many times. With double processing of URLs and the spammer controls 50% of the clients then there's a 50% chance that a simple diff would show the injected spam. The problem is that the spammer needs to do this many times, so their injection becomes statistically apparent. If the spammer can only inject a small number of messages before they are detected, then the cost per injected spam will be quite high. Long running spam campaigns could eventually be detected by content analysis, so the spammer also needs to rotate content.

Obviously you can play with the numbers, the attacker could try to control >>50% of the clients. The project could process URLs >2x. The project could re-process N% of URLs on trusted hardware, etc. It's not easy by any means, but you can tune the knobs to increase the cost for spammers.

Re: The world needs a non-profit search engine

#293

Post by rntksi »

Gareth321 wrote: post #271
IgorPartola wrote: post #266

Earlier quoted context omitted.

To your last point, I don’t quite agree. Google’s incentives are misaligned such that keeping you on Google.com just a little longer is better than not because you are more likely to click on an ad. But also yes the users and the UI both fail. When I used to search for something I would type in something like “gutter clog clean” but slowly started noticing that Google likes longer sentences like “how do I clean a clo…

>To your last point, I don’t quite agree. Google’s incentives are misaligned such that keeping you on Google.com just a little longer is better than not because you are more likely to click on an ad. I agree, which leads me to the conclusion that subscription is the best way to avoid this conflict of interest. Unfortunately, most of the world won't subscribe to a search engine, and doesn't seem to mind ads - to a deg…

Here's a search engine that I'm subscribed to: https://kagi.com

In the 20-30 searches that I do in a day, I still have to google about half of them. Either because it's stuff Google does well (currency conversion, for example), or Kagi just doesn't get what I'm trying to search.

I remember starting out with the Internet searching on Altavista and Yahoo and Lycos. The information that was present was nowhere near as now, and it was more "exploratory". Nowadays people just kind of know what they want and just wants to quickly get there.

Re: The world needs a non-profit search engine

#294

Post by cloverich »

sevazhidkov wrote: post #174
amelius wrote: post #166

Earlier quoted context omitted.

I think Google search is using the wrong approach. When I'm looking for e.g. a new camera, I want to use my network which I trust . E.g. I want to ask "what camera would HN recommend?" We should think more about how we can use trust as a basis for how we explore the internet.

Given that enough people use this heuristic, there will be companies focusing on earning karma on HN, writing comments and voting for products they are getting paid for.

While true, it is also not a given that you'd need to trust _all_ of HN. I visit sufficiently regularly that I see some people post and recognize their username. I often think -- I wish I could _follow_ them. Not so much of a stretch to think of rings of trust built around particular users. Bringing people in and kicking them out of these trust circles could play a role. PageRank -> TrustRank? Of course it would also be only one metric, among many possible trust rankings and many possible other signals and settings.

I bet the (niche) product as such wouldn't be as hard to build as it would be to scale. Imagine every user constantly tweaking (directly or indirectly) their search result settings, and having that impact millions (or more) indexed items, for every user.

Re: The world needs a non-profit search engine

#295

Post by permo-w »

lrvick wrote: post #235
slazaro wrote: post #200

Earlier quoted context omitted.

99%+ of people wouldn't use this, they would just try the search engine, see that the results suck, then stop using it. It could be useful for power users, though.

99%+ of people in the medical industry did not sanitize hands or equipment 100 years ago. 99%+ of people in the tech industry currently do not care to do the extra steps required for data neutrality, and privacy. 99%+ of people are lazy to the point of harming themselves and others. 1%- of people examine how the 99%+ do things and pioneer harm reduction tactics in spite of everyone constantly reminding them that no o…

>99%+ of people in the medical industry did not sanitize hands or equipment 100 years ago.

this isn't true. 170 years ago, maybe. handwashing became a thing in the late 1800s after Semmelweiss and Pasteur

Re: The world needs a non-profit search engine

#296

Post by throwawaymaths »

amelius wrote: post #291
javajosh wrote: post #249

>Google makes $40bn...If I can create something that just a tiny fraction of people find useful, then I can create a huge amount of value. You conflate two meanings of value: monetary value, and intrinsic value. Search engines are intrinsically but not monetarily valuable to users. Search engines are monetarily, but not intrinsically, valuable to advertisers. You can get into trouble when you conflate these two meani…

> It is surprisingly hard to extract monetary value from an intrinsically valuable solution. This is like saying that our entire economic system is backward ...

I have no idea what gp means by this. It can be surprisingly hard, but it's not always. Go buy a cooler of water bottles and sell them for 50c in the summer on the side of the street

Re: The world needs a non-profit search engine

#297

Post by irrational »

jstummbillig wrote: post #96
_def wrote: post #75

Earlier quoted context omitted.

Is it?

Yes? Offering ad space (and placing affiliate links) is how they make money. Difference to Google is how they position themselves in regards to privacy, and that Google actually built a search engine. Both make their money by providing ad space.

Where are the ads?

Re: The world needs a non-profit search engine

#298

Post by swayvil »

allendoerfer wrote: post #193
nonameking2026 wrote: post #189

Earlier quoted context omitted.

Allow users to blacklist sites. Share blacklisted sites. Have the option of instead of hiding blacklisted sites entirely - show them in a different column. Have 4 different types of list. Whitelist - highest scoring Not listed - these websites are not ranked. Yellowlist - show but keep in a separate column Blacklist - don't show https://search.brave.com/help/goggles is interesting.

I found the thought of adding a "yellowlist" funny, as some people are starting to find the use of "whitelist" and "blacklist" racist.

Some people couldn't find their butt with both hands and a map. Best not to take advice from them. Unless you find yourself surrounded. In which case smile and nod.

Re: The world needs a non-profit search engine

#299

Post by swayvil »

recuter wrote: post #282
fauigerzigerk wrote: post #225

Earlier quoted context omitted.

It would be more consistent to call it greenlist, yellowlist, redlist then.

How dare you, some of my best friends identify as traffic lights.

That term has been judged problematic. The appropriate term is "person of vehicular illuminativeness".

Re: The world needs a non-profit search engine

#300

Post by permo-w »

grogers wrote: post #274
_abox wrote: post #240

Earlier quoted context omitted.

But a company like Google would want supreme control. They're not inclined to rely on a community for anything Wikipedia shows it can work at scale.

Gmail's spam filter is basically community trained by people clicking spam/not spam (at least, that was what they said many years ago, it might have silently changed). What's different about search?

spam filter isn't their key flagship aquila product
Post reply on HN