Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Could we just re-invent original Google?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 40 posts

Re: Ask HN: Could we just re-invent original Google?

#21

Post by O1111OOO »

There was a search engine listed here yesterday (https://news.ycombinator.com/item?id=39203538) that displayed Google searches with the oldest first.

I played around with it, mostly typing in a bunch of DOS search terms. If this search engine is working the way it's suppose to.. I should have thousands of very old results. After a single page or 3, I was quickly looking at results from 2006/2007.

This might have been a problem with the search engine or a (more serious) problem with Google throwing away much of the past. We already know this is a problem. We just don't know how serious it is.

A recreation of Google would involve reindexing to include what Google (DDG/Bing and others) have abandoned (the recent past).

PS: I use https://wiby.me and https://yandex.com with much greater success in finding older material.

Re: Ask HN: Could we just re-invent original Google?

#22

Post by viraptor »

fsflover wrote: post #5

Here you go: distributed, peer-to-peer, FLOSS search engine: https://yacy.net

I love the idea itself, but the project is dying. Almost no maintenance, the P2P part is spammed / unusable, CloudFlare bot checks wreck your IP reputation, the deployment/setup is... idiosyncratic. I wish it was still practical to use YaCy.

Re: Ask HN: Could we just re-invent original Google?

#23

Post by BenoitP »

TL;DR: Goodhart's law rots everything it touches

SEO is in the structure of the internet now. Original Google was great because there was no incentive yet to buy a domain and blogspam it. Google getting shitty is just a natural instance of Goodhart's law, applied to domains and content.

Now, Google originally was based on PageRank; which based itself on every domain being a unit of authority. These have been compromised and drowned by SEO, but the concept remains valid and we could choose people as units of authority. For example PageRank on scientific papers accurately reproduce Nobel Prizes attributions. A person publishing papers is a solid enough foundation for this unit of authority.

It remains to be organized though. And if we take people as units of authority, it means they'd have to 'cite' or vote for each other. This has social consequences and might not be doable. Are you ready to refuse to cite your boss when he/she ask you to do so? Maybe if the vote is secret and delayed by 5 years?

Re: Ask HN: Could we just re-invent original Google?

#24

Post by Apreche »

Most people using machine learning to make search engines are replacing the search paradigm with a prompt + answer format.

I think there’s an easier way. Train an ML model to be able to tell apart legit web sites from garbage ones. It’ s just a binary classification. A site should either be blocked, or not.

Legit web sites being ones created by actual humans with actual content. Few to no ads. No malware, phishing, or other security threats. No content farms or SEO sites. No sites generated by other ML models. No paywalls, no pop-ups or other annoyances. Just real web sites.

You’re going to need a bunch of smart and trustworthy humans to spend hours and hours to help do this classification. But a model can help multiply the effectiveness of their efforts.

If the model works, then yes. You can make a very simple search engine. You just tell all the web crawlers to check the model, and only add sites to the index if the model says they are good web sites and not garbage sites.

Re: Ask HN: Could we just re-invent original Google?

#25

Post by krapp »

Apreche wrote: post #24

Most people using machine learning to make search engines are replacing the search paradigm with a prompt + answer format. I think there’s an easier way. Train an ML model to be able to tell apart legit web sites from garbage ones. It’ s just a binary classification. A site should either be blocked, or not. Legit web sites being ones created by actual humans with actual content. Few to no ads. No malware, phishing, o…

Websites that practice SEO, use paywalls, popups or "other annoyances" often also have actual content which should be relevant to a search engine. A search engine that refused to show me Wikipedia or IMDB or any mainstream news site wouldn't be useful to most people.

Also, tools that attempt to detect ML generated content tend not to work, and will only become less effective over time, as LLMs improve.

Re: Ask HN: Could we just re-invent original Google?

#26

Post by M95D »

tlb wrote: post #6

The real prize is to deshittify the content on the web. Ignoring copyrights for a moment, it should be possible to rewrite pages in 1000s of sites down to just the information. StackOverflow could just have the Q&As, news sites could just have the news, Pinterest would just have the pictures. Remove all the signup nags and popups. With good tools, one person could probably maintain the deshittifier for a few sites, a…

Nobody will create that. Too complicated to create. Too complicated to maintain.

I wish there was a browser add-on that could regex-replace html source of the page. Then I could write my own deshittfier list. Since I couldn't find any, i'm guessing it's because the plugin APIs won't allow it.

There was a MITM app that could do this, AdMuncher, but then the web switched to HTTPS and it didn't work anymore.

Re: Ask HN: Could we just re-invent original Google?

#28

Post by dinkleberg »

wildrhythms wrote: post #27
28304283409234 wrote: post #9

You mean, like https://www.kagi.com ? A search engine you pay for. Every link is there to serve _you_. There are not ads.

I haven't tried Kagi yet but I'm curious how does Kagi combat SEO spammers?

Pretty well to be honest.

Re: Ask HN: Could we just re-invent original Google?

#29

Post by drcongo »

wildrhythms wrote: post #27
28304283409234 wrote: post #9

You mean, like https://www.kagi.com ? A search engine you pay for. Every link is there to serve _you_. There are not ads.

I haven't tried Kagi yet but I'm curious how does Kagi combat SEO spammers?

Extremely well! No idea how it's doing it in the general results, but they're orders of magnitude better than Google / Bing etc. Plus you get to fine tune results yourself, ie: downrank w3schools etc so the garbage sites are less likely to appear in your results. You can even exclude them completely.

Re: Ask HN: Could we just re-invent original Google?

#30

Post by jqpabc123 »

Maybe if we just re-made the original and changed the business model it would be good again.

This has already been done --- see DuckDuckGo.

The problem is not ads --- it's privacy invasion (aka "personalized ads") and advertisers who respond to concept.

DuckDuckGo shows ads --- "context sensitive" ads --- you know, ads that are related to what you search for and might actually be helpful. Not something you did last week or last month that may no longer apply.

"Personalized ads" are one of the dumbest ideas ever --- virtually guaranteed to waste a lot of people's time and squander mental and electronic bandwidth. And yet Google makes billions --- because advertisers are stupid and lazy enough to literally turn over most of their ad budget to them.

We need to upend the idea the idea that, "No one ever got fired for using Google" --- and the only way to do that on a personal level is to stop using Google.

We need more ordinary people to grasp and respond to the idea that "Google" is just another word for "privacy invasion".

Post reply on HN