Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Almost all searches on my independent search engine are now from SEO spam bots

blog.searchmysite.net

Post reply on HN
Print view View original

111–120 of 391 posts

Re: Almost all searches on my independent search engine are now from SEO spam bots

#111

Post by marginalia_nu »

ratww wrote: post #106
marginalia_nu wrote: post #84

Earlier quoted context omitted.

I do a traditional web crawl and exclude anything that looks too much like it wants a high google ranking. Nothing to it.

This might be controversial, but I wish Google would exclude those websites too. Google started punishing keyword spam, then it started punishing black-hat comment spam. Even Youtube backtracked on the "videos have to be 10 minutes to rank". I wish they would do the same for carefully manicured SEO content farms too, as those sites are causing a harm worse than keyword-spammer sites did.

They're probably doing all they can. The problem is their dominance, both means they have effectively an entire industry looking for loopholes in everything they do, as well as legal considerations (arbitrarily punishing individual smaller actors might skirt on the territory of anti-competitive behavior)

Re: Almost all searches on my independent search engine are now from SEO spam bots

#112

Post by honkdaddy »

Teandw wrote: post #62

This guy throws multiple reasons/conspiracies out there on why the website is really struggling to gain literally any sort of traction. Web is all bots, search engines not promoting competitors and being drowned out by SEO spam, yet he's failing to see the most obvious reason... the reason nearly all websites don't gain traction... Because it's a bad website. It provides no value to the user. I put in a few search te…

Searching for Astral Codex Ten, a popular, well-written, non-spammy blog which I would expect is indexed...

Returns only results in which _other_ bloggers are referencing ACX. Consider me as one of the datapoints that arrived from HN and likely won't be back, I'm afraid.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#113

Post by ori_b »

Elvie wrote: post #107
Gravityloss wrote: post #95

Earlier quoted context omitted.

And not only did this happen already over a decade ago, a lot of the current internet users have never known anything else. We had a discussion with coworkers and somebody mentioned irc. Explaining to younger colleagues what it was and that it was not a product of a company, but operators had servers that formed a network, and it was more like infrastructure. Felt weird.

isn't Discord a bit like IRC used to be?

How do I connect to a self hosted discord, and then connect it to my friends self hosted one?

And where do I get the RFC for the protocol so that I can write my own compatible implementation?

IRC isn't a product. It's a standardized protocol sufficiently simple to implement in a day or two.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#114

Post by baxtr »

FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

I am not so sure...

I think what happened is this: the WWW was everything back in the days. But in the "old days," only 10% of all people were online, the web elite. Then, AOL came, and the rest came online slowly but surely. The so-called "mainstream" people were no geeks, and these people were "just" ordinary people. Almost all were captured by what you call "big websites".

Now, we see the 100% being dominated by the 90%. That's why "Google results are bad". Bad for us! Not maybe (most probably) not for them.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#115

Post by pnutjam »

Cthulhu_ wrote: post #45
FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

I don't believe the WWW internet is dead; there's still millions of webpages being made and published every day. However, the traffic numbers are skewed in favor of the big socials and aggregators; I wouldn't be surprised if the 80/20 rule applies there.

There seems to be a tendancy towards video that undercuts the "old internet". I prefer instructions in a text or list format, but that's almost impossible to find for things like, changing the headlight bulb on my traverse.

1. turn the wheel so it is pointed hard in the direction of the bulb you are changing.

2. remove the hex screws from the shroud in the wheel well

3. pull the shroud down, it's pretty flexible plastic.

4. reach up and change the bulb. The wires are a bit short so you might need to get both hands in there. I have big hands and I'm able to do it.

---- There are innumerable videos explaining this process, but very few text directions.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#116

Post by ColinHayhurst »

Mojeek member here. We have always had a high level of spam bots; as any search engine/service will have. It's a constant battle to fend off new bots; folks can always use try out our API rather than freeloading, and some do. Many obviously do not. We are taking a look at whether things have also changed for us since mid-April 2022.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#117

Post by wccrawford »

honkdaddy wrote: post #112
Teandw wrote: post #62

This guy throws multiple reasons/conspiracies out there on why the website is really struggling to gain literally any sort of traction. Web is all bots, search engines not promoting competitors and being drowned out by SEO spam, yet he's failing to see the most obvious reason... the reason nearly all websites don't gain traction... Because it's a bad website. It provides no value to the user. I put in a few search te…

Searching for Astral Codex Ten, a popular, well-written, non-spammy blog which I would expect is indexed... Returns only results in which _other_ bloggers are referencing ACX. Consider me as one of the datapoints that arrived from HN and likely won't be back, I'm afraid.

I just search Kagi, Google, and DDG for "Astral Codex Ten" and it was the first result on each.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#118

Post by bachmeier »

buzzwords wrote: post #108

I have had very interesting conversations with people who are "casual" users of internet. They are still finding the results of the likes of Google, bing and duckduckgo perfectly suitable. Maybe it's most of us here who have different needs to what's available.

I suppose it depends what they're looking for. If you're a homeowner looking for a service of some kind...good luck. There are domains that aren't too bad, like programming, but you should go into a search with low expectations. Anyone that remembers the early days of Google will find today's search engines to be useless in comparison.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#119

Post by native_samples »

Teandw wrote: post #62

This guy throws multiple reasons/conspiracies out there on why the website is really struggling to gain literally any sort of traction. Web is all bots, search engines not promoting competitors and being drowned out by SEO spam, yet he's failing to see the most obvious reason... the reason nearly all websites don't gain traction... Because it's a bad website. It provides no value to the user. I put in a few search te…

Well, it's worse than that. The whole schtick is that it's only pure, real content by folksy people like us. The top reason to use it on the about page is:

Indexes only user-submitted sites with a moderation layer on top, for a community-based approach to content curation, rather than indexing the entire internet with all of its spam, "search engine optimisation" and "click-bait" content.

So I tried searching [kotlin] and got 123 results ...

https://searchmysite.net/search/?q=kotlin

... of which the 9th result is SEO spam! It reads:

PersonalSit.es | Yes we got hot and fresh sites https://personalsit.es/ ... Shandilyahttps://msfjarvis.devTagsandroid, kotlin, rust Go to feed Go to siteradoslawkoziel.plradoslawkoziel.pl ...

That looks like junk to me. How is that possible if what the developer says is true, that it's all verified and pre-moderated?

Re: Almost all searches on my independent search engine are now from SEO spam bots

#120

Post by sdoering »

marginalia_nu wrote: post #54
FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

I built my search engine in part to explore whether this was actually true, and I don't think it actually is. There's still a lot of organic human-made content still out there, possibly more than ever, it's just not able to compete with the SEO industry that completely displaces it from Google and social media.

I can only agree with my sister comment. I find this industrialized web more and more shallow and taxing to use.

While professionally I need to help (smaller, local) clients to reach their audiences I become more and more weary.

It is like walking through a supermarket with industrialized fast convenience food shouting in bright colors and advertising while ultimately not nourishing me like slow, real food could.

I am still looking for this digital slow food movement.

Post reply on HN