Earlier quoted context omitted.
To be fair, those searches are pretty weird.
We don't search for things because they're easy to find
Uh that has nothing to do with it? I don't immediatly know the URL of a website which shows me info about lamps?
A search engine that favors text-heavy sites and punishes modern web design
641–650 of 735 posts
Re: A search engine that favors text-heavy sites and punishes modern web design
#642Wow, that's awesome. Great work! For a simple test, I searched "fall of the roman empire". In your search engine, I got wikipedia, followed by academic talks, chapters of books, and long-form blogs. All extremely useful resources. When I search on google, I get wikipedia, followed by a listicle "8 Reasons Why Rome Fell", then the imdb page for a movie by the same name, and then two Amazon book links, which are totall…
I think it's a case where systems diversity can be an advantage. Much like how most malware was historically written for Windows and could be avoided by using Linux, the low-quality search engine bait is created for Google and can be avoided by using a different style of search engine.
Re: A search engine that favors text-heavy sites and punishes modern web design
#643Earlier quoted context omitted.
This got me thinking that maybe one of the other big reasons for this is that the algorithms prioritize newer pages over older pages. This produces the problem where instead of covering a topic and refining it over time, the incentive is to repackage it over and over again. It reminds me of an annoyance I have with the Kindle store. If I wanted to find a book on, let's say, Psychology, there is no option to find all-…
This is just a guess, but I believe that they use machine learning and rank it by the clicks. I took some coursera courses and Andrew Ng sort of suggested that as their strategy. The problem is that clickbait and low effort articles could be good enough to get the click, but low effort enough to drag society into the gutter. As time passes, the system is gamified more and more where the least effort for the most clic…
It sounds like the problem is, the search engine has no way to measure satisfaction after the click.
Re: A search engine that favors text-heavy sites and punishes modern web design
#644Earlier quoted context omitted.
The new evergreen is refreshed sludge for bottom dollar. College kids stealing Reddit comments or moving around paragraphs from old articles. Or linking to linked blogs that link elsewhere. It's all stamped with Google Ads, of course, and then Google ranks these pages high enough to rake in eyeballs and ad dollars. Also there's the fact that each year, the average webpage picks up two more video elements / ad players…
Really makes you wonder: you play whack a mole and tackle the symptoms with initiatives like this search engine. But the root of that problem and many many others is the same: advertising. Why don't we try to tackle that?
Perhaps a subscription-based search engine would avoid these incentives.
Re: A search engine that favors text-heavy sites and punishes modern web design
#645Yeah so this is my project. It's very much a work in progress, but occasionally I think it works remarkably well for something I cobbled together alone out of consumer hardware and home-made code :-)
Really like it, and love that you've done this yourself.
I'd prefer if it does just one thing and does that really well. Don't waste your time on calculator and conversion functions, or pseudo-natural language queries. There are plenty of good calculator & converter tools and websites, but we all need a really good search engine. I think you'd be better looking at handling synonyms and plurals.
Thanks.
Re: A search engine that favors text-heavy sites and punishes modern web design
#646Earlier quoted context omitted.
I added it now, but it turns out it's behind a CDN so I still can't crawl it.
Thanks for responding and especially thanks for the search engine. What a breath of fresh air, and access, it feels like, to real people.
Yeah, that's a large part of what I'm trying to accomplish. Great to see others understand as well.
Re: A search engine that favors text-heavy sites and punishes modern web design
#647This is a search engine indexing the internet on a mariadb database hosted on consumer hardware maintained by a single person as a hobby and it does not suffer from HN hug of death
Huh.
I've gone through the logs, and at most I got about 2 search queries per second as a sustained load over several hours. It's calmed down to about 0.5-1 QPS now.
This is ignoring other page loads. Not bad, if I do say so myself.
Re: A search engine that favors text-heavy sites and punishes modern web design
#648Earlier quoted context omitted.
This is just a guess, but I believe that they use machine learning and rank it by the clicks. I took some coursera courses and Andrew Ng sort of suggested that as their strategy. The problem is that clickbait and low effort articles could be good enough to get the click, but low effort enough to drag society into the gutter. As time passes, the system is gamified more and more where the least effort for the most clic…
It sounds like the problem is, the search engine has no way to measure satisfaction after the click.
But they have. or could have. At least Google (and to a smaller extend Microsoft), if you are using Chrome/Bing have exactly that signal. If you stay on the site and scroll (taking time, reading, not skimming) all this could be a signal to evaluate if the search result met your needs.
Re: A search engine that favors text-heavy sites and punishes modern web design
#649Earlier quoted context omitted.
This is just a guess, but I believe that they use machine learning and rank it by the clicks. I took some coursera courses and Andrew Ng sort of suggested that as their strategy. The problem is that clickbait and low effort articles could be good enough to get the click, but low effort enough to drag society into the gutter. As time passes, the system is gamified more and more where the least effort for the most clic…
It sounds like the problem is, the search engine has no way to measure satisfaction after the click.
They tried to insight this with the "bounce rate"
Re: A search engine that favors text-heavy sites and punishes modern web design
#650Earlier quoted context omitted.
Search engines whose revenue is based on advertising will ultimately be tuned to steer you to the ad foodchain. All the incentives are aligned towards and all the metrics ultimately in service of, profit for advertisers. Not in the 99% of people who can convinced to consume something by ads? Welp, screw you.
Search engines should be something you pay for. Surely search engine powerusers can afford to pay for such a service. If Google makes $1 per user per month or something, that's not too high a bar to get over.
The theoretical cap for this, if you include every human being on planet Earth, is 7 billion/month. This translates into $84 billion annual revenue.
Google's revenue last year was 146 billion, and it operates not anywhere near the theoretical maximum. Most of that revenue is advertisement.