Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The world needs a non-profit search engine

daoudclarke.net

Post reply on HN
Print view View original

271–280 of 396 posts

Re: The world needs a non-profit search engine

#271

Post by Gareth321 »

IgorPartola wrote: post #266
O__________O wrote: post #242

Earlier quoted context omitted.

For clarification, “red queen” means a conflict between two or more entities where the cost of engagement grows, but the relative advantage does not change. Simply put, search engines have been at war with SEO for over 30-years, which has significantly raised the bar not only being a search engine, but producing content; not to mention knowing how to search for information. With the introduction of machine generated…

To your last point, I don’t quite agree. Google’s incentives are misaligned such that keeping you on Google.com just a little longer is better than not because you are more likely to click on an ad. But also yes the users and the UI both fail. When I used to search for something I would type in something like “gutter clog clean” but slowly started noticing that Google likes longer sentences like “how do I clean a clo…

>To your last point, I don’t quite agree. Google’s incentives are misaligned such that keeping you on Google.com just a little longer is better than not because you are more likely to click on an ad.

I agree, which leads me to the conclusion that subscription is the best way to avoid this conflict of interest. Unfortunately, most of the world won't subscribe to a search engine, and doesn't seem to mind ads - to a degree. With Google looking more and more like AltaVista before its demise (to Google), my conclusion is that Google will strangle itself out of existence and give way for the next "new, streamlined, not-full-of-ads" competitor.

Re: The world needs a non-profit search engine

#272

Post by thfuran »

zo1 wrote: post #146
bosie wrote: post #119

Earlier quoted context omitted.

> and wrap it in ads, and derive content from it i am probably missing something but can you give an example where this happens?

I think they're referring to how Google "extracts" answers from your website and shows it on the search results page. Effectively meaning that the user doesn't even need to go to your site to get the answer, because Google extracted it and gave it to them directly.

It seems to me that what they usually extract is some junk only vaguely related to the query and often cut apart and reassembled in a way that's just wrong.

Re: The world needs a non-profit search engine

#273

Post by marginalia_nu »

larsrc wrote: post #155

Disclaimer: I work for Google, though far away from Search. Regardless of search engine design, there's HUGE money in SEO. Any successful search engine will be gamed. Do you have the developer power to go red-queen against all the large companies in the world?

I think this is mostly a problem if you are in Google's position, near total market dominance.

This not only means you exert massive selection pressure on the shape of websites. The SEO spammers don't need to be good or know what they are doing, they just need to be lucky once. If they get it right, they float to the top, and can iterate on that design. This effectively is saying that no matter how secret or smart your algorithms are, it doesn't matter if you're in Google's position. The numbers are stacked against you.

To make matters worse, any company with that sort of a market share has serious handcuffs in how heavy handed and "unfair" you can be without risking litigation for anti-competitive practices.

I think the best thing that could happen for Google is ironically serious competition in the search market. This would help both problems at once.

Re: The world needs a non-profit search engine

#274

Post by grogers »

_abox wrote: post #240
mhoad wrote: post #232

Earlier quoted context omitted.

I love how this is a problem that a huge number of people have worked on for around 20 years and have thrown stupid amounts of money at and people just jump in and boldly proclaim that they have the answers that just popped into their mind. Peak HackerNews.

But a company like Google would want supreme control. They're not inclined to rely on a community for anything Wikipedia shows it can work at scale.

Gmail's spam filter is basically community trained by people clicking spam/not spam (at least, that was what they said many years ago, it might have silently changed). What's different about search?

Re: The world needs a non-profit search engine

#275

Post by IgorPartola »

Gareth321 wrote: post #271
IgorPartola wrote: post #266

Earlier quoted context omitted.

To your last point, I don’t quite agree. Google’s incentives are misaligned such that keeping you on Google.com just a little longer is better than not because you are more likely to click on an ad. But also yes the users and the UI both fail. When I used to search for something I would type in something like “gutter clog clean” but slowly started noticing that Google likes longer sentences like “how do I clean a clo…

>To your last point, I don’t quite agree. Google’s incentives are misaligned such that keeping you on Google.com just a little longer is better than not because you are more likely to click on an ad. I agree, which leads me to the conclusion that subscription is the best way to avoid this conflict of interest. Unfortunately, most of the world won't subscribe to a search engine, and doesn't seem to mind ads - to a deg…

What about one funded by universities or libraries as a research project?

There have been lots of no ads (for now) attempts. DDG had like one small ad at one point. But people didn’t leave in droves. It’s almost like people are ok with ads.

Re: The world needs a non-profit search engine

#276

Post by tgv »

guerrilla wrote: post #102
tgv wrote: post #58

Earlier quoted context omitted.

It does, I just tried it with 102.1. You can add a "keyword" to use a search engine, and once you've done that, you can add it as a search engine to the url+search bar (the main one on top, I don't know how it's actually called) as well, and you can set it as your default search engine.

I mean I used to have that but I seem to now. Did it move? This is 102.0. Edit: You can make a bookmark and add a keyword, but that doesn't help me use it as a search engine. It used to be you could just create one from the contextual menu, what is this convoluted process.

The steps I took were:

1. Go to kagi.com

2. Right-click on the search field and choose "Add a keyword for this search..."

3. Fill in the keyword (e.g. kagi, as I did)

4. Right click in the top url+search field, and choose "Add Kagi search"

Now you can search via keyword, you can change to kagi while typing, and you can set the default search engine to kagi in the preferences.

Re: The world needs a non-profit search engine

#277

Post by kordlessagain »

kolinko wrote: post #90

Perhaps a better approach would be building an open source www index or even a full current cache - as an enabler for people to build their own search engines? Right now it is extremely difficult to build your own web crawler that would compete with Google. And that is not because of the technology, but because multiple sites will prevent your bot from accessing them if you're not Google or Bing - either through robo…

I built something with an API that uses Selenium to image a site. It works on a large percentage of the sites I feed it.

I don't recursively call links found in the pages. I expect the user to give me the URLs to crawl and save.

In order to "find" new content, I let the user specify where they want to search for things the engine hasn't "crawled" yet. So, a search for scooters to buy might end up searching Amazon directly, then lets the user "save" the site by passing the Amazon URL for the scooter they like to the system for imaging.

I use GPT-3 or other ML models to do some of the heavy lifting for adding labels to the pages or documents the user uploads.

This ends up being a "curated" list of documents important to the individual user, not an exhaustive crawl of all things which are important to all users.

Re: The world needs a non-profit search engine

#278

Post by rvz »

allendoerfer wrote: post #193
nonameking2026 wrote: post #189

Earlier quoted context omitted.

Allow users to blacklist sites. Share blacklisted sites. Have the option of instead of hiding blacklisted sites entirely - show them in a different column. Have 4 different types of list. Whitelist - highest scoring Not listed - these websites are not ranked. Yellowlist - show but keep in a separate column Blacklist - don't show https://search.brave.com/help/goggles is interesting.

I found the thought of adding a "yellowlist" funny, as some people are starting to find the use of "whitelist" and "blacklist" racist.

Don’t forget to avoid saying ‘blacksmith’ too. Another somewhat ‘racist’ one. /s

Still waiting for Mastercard to change their incredibly ‘racist’ company name. /s

Re: The world needs a non-profit search engine

#280

Post by marginalia_nu »

Honestly, the easy part is building a search engine, like just document retrieval stuff and domain ranking, SEO-mitigation etc. Anyone can build a Google '98 and get it to work well, not that hard, doesn't require all too much hardware. I have done that and got one running out of my living room.

The tricky part, if you want people to use your search engine for more than the novelty factor, and what most Google competitors struggle with is drawing the rest of the damn owl. For example, commercial searches, local businesses, that sort of thing. As much as Google flounders with some queries, the overall package is still really good.

Post reply on HN