Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The world needs a non-profit search engine

daoudclarke.net

Post reply on HN
Print view View original

231–240 of 396 posts

Re: The world needs a non-profit search engine

#231

Post by 6510 »

kolinko wrote: post #90

Perhaps a better approach would be building an open source www index or even a full current cache - as an enabler for people to build their own search engines? Right now it is extremely difficult to build your own web crawler that would compete with Google. And that is not because of the technology, but because multiple sites will prevent your bot from accessing them if you're not Google or Bing - either through robo…

> Right now it is extremely difficult to build your own web crawler that would compete with Google.

Nah, not a small task but you can break it down into well understood problems that have known solution.

The hard part is ranking everything.

Re: The world needs a non-profit search engine

#232

Post by mhoad »

nonameking2026 wrote: post #189
larsrc wrote: post #155

Disclaimer: I work for Google, though far away from Search. Regardless of search engine design, there's HUGE money in SEO. Any successful search engine will be gamed. Do you have the developer power to go red-queen against all the large companies in the world?

Allow users to blacklist sites. Share blacklisted sites. Have the option of instead of hiding blacklisted sites entirely - show them in a different column. Have 4 different types of list. Whitelist - highest scoring Not listed - these websites are not ranked. Yellowlist - show but keep in a separate column Blacklist - don't show https://search.brave.com/help/goggles is interesting.

I love how this is a problem that a huge number of people have worked on for around 20 years and have thrown stupid amounts of money at and people just jump in and boldly proclaim that they have the answers that just popped into their mind. Peak HackerNews.

Re: The world needs a non-profit search engine

#233

Post by [deleted] »

hnarn wrote: post #227
jstummbillig wrote: post #218

Earlier quoted context omitted.

How do you prevent the gaming of blacklisting?

That’s the same question as “how do you ensure complete trust” which is, of course, not possible. That doesn’t mean that “distributed trust-based blacklisting” still isn’t better than what Google is offering today, which is nothing.

[deleted]

Re: The world needs a non-profit search engine

#234

Post by samwillis »

The cost of building a “general” search engine for the “whole” web is astronomically high, in the 10s to 100s billions. It’s not achievable, Google were only able to do it by growing as a business at the same time as the internet itself. I don’t believe it’s possible to compete with Google (or Bing) by starting at zero.

The route forward, and what should be advocated for, is a distributed network of search engines, each for a specific vertical. If it operated as a cooperative they could share expertise and technology, they could then build a “meta” search engine for the co-op that combined all the results from the specialist niches. Each member basically “owning” the “franchise” for a specific type of search or category.

So, I don’t believe a single non-profit is the answer. More a co-op type arrangement where the co-op organisation (which may be a non-profit) has a mission to advance internet search through it’s network and strategic investment.

Re: The world needs a non-profit search engine

#235

Post by lrvick »

slazaro wrote: post #200
nonameking2026 wrote: post #189

Earlier quoted context omitted.

Allow users to blacklist sites. Share blacklisted sites. Have the option of instead of hiding blacklisted sites entirely - show them in a different column. Have 4 different types of list. Whitelist - highest scoring Not listed - these websites are not ranked. Yellowlist - show but keep in a separate column Blacklist - don't show https://search.brave.com/help/goggles is interesting.

99%+ of people wouldn't use this, they would just try the search engine, see that the results suck, then stop using it. It could be useful for power users, though.

99%+ of people in the medical industry did not sanitize hands or equipment 100 years ago.

99%+ of people in the tech industry currently do not care to do the extra steps required for data neutrality, and privacy.

99%+ of people are lazy to the point of harming themselves and others.

1%- of people examine how the 99%+ do things and pioneer harm reduction tactics in spite of everyone constantly reminding them that no one wants their help.

Re: The world needs a non-profit search engine

#236

Post by christophilus »

mhoad wrote: post #232
nonameking2026 wrote: post #189

Earlier quoted context omitted.

Allow users to blacklist sites. Share blacklisted sites. Have the option of instead of hiding blacklisted sites entirely - show them in a different column. Have 4 different types of list. Whitelist - highest scoring Not listed - these websites are not ranked. Yellowlist - show but keep in a separate column Blacklist - don't show https://search.brave.com/help/goggles is interesting.

I love how this is a problem that a huge number of people have worked on for around 20 years and have thrown stupid amounts of money at and people just jump in and boldly proclaim that they have the answers that just popped into their mind. Peak HackerNews.

I don’t think these ideas just popped into his mind. They’ve been discussed for a long time on HN, and some alternative search engines have started to incorporate them.

It’s worth the experiment.

Re: The world needs a non-profit search engine

#237

Post by [deleted] »

hnarn wrote: post #227
jstummbillig wrote: post #218

Earlier quoted context omitted.

How do you prevent the gaming of blacklisting?

That’s the same question as “how do you ensure complete trust” which is, of course, not possible. That doesn’t mean that “distributed trust-based blacklisting” still isn’t better than what Google is offering today, which is nothing.

[deleted]

Re: The world needs a non-profit search engine

#238

Post by Parzival99 »

allendoerfer wrote: post #193
nonameking2026 wrote: post #189

Earlier quoted context omitted.

Allow users to blacklist sites. Share blacklisted sites. Have the option of instead of hiding blacklisted sites entirely - show them in a different column. Have 4 different types of list. Whitelist - highest scoring Not listed - these websites are not ranked. Yellowlist - show but keep in a separate column Blacklist - don't show https://search.brave.com/help/goggles is interesting.

I found the thought of adding a "yellowlist" funny, as some people are starting to find the use of "whitelist" and "blacklist" racist.

lmao, feeling left out with no brown list then.

Re: The world needs a non-profit search engine

#239

Post by christophilus »

slazaro wrote: post #200
nonameking2026 wrote: post #189

Earlier quoted context omitted.

Allow users to blacklist sites. Share blacklisted sites. Have the option of instead of hiding blacklisted sites entirely - show them in a different column. Have 4 different types of list. Whitelist - highest scoring Not listed - these websites are not ranked. Yellowlist - show but keep in a separate column Blacklist - don't show https://search.brave.com/help/goggles is interesting.

99%+ of people wouldn't use this, they would just try the search engine, see that the results suck, then stop using it. It could be useful for power users, though.

Add a thumbs up / down icon, and your average Facebook user would likely use it without knowing what it was.

Re: The world needs a non-profit search engine

#240

Post by _abox »

mhoad wrote: post #232
nonameking2026 wrote: post #189

Earlier quoted context omitted.

Allow users to blacklist sites. Share blacklisted sites. Have the option of instead of hiding blacklisted sites entirely - show them in a different column. Have 4 different types of list. Whitelist - highest scoring Not listed - these websites are not ranked. Yellowlist - show but keep in a separate column Blacklist - don't show https://search.brave.com/help/goggles is interesting.

I love how this is a problem that a huge number of people have worked on for around 20 years and have thrown stupid amounts of money at and people just jump in and boldly proclaim that they have the answers that just popped into their mind. Peak HackerNews.

But a company like Google would want supreme control. They're not inclined to rely on a community for anything

Wikipedia shows it can work at scale.

Post reply on HN