Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Almost all searches on my independent search engine are now from SEO spam bots

blog.searchmysite.net

Post reply on HN
Print view View original

121–130 of 391 posts

Re: Almost all searches on my independent search engine are now from SEO spam bots

#121

Post by LegitShady »

black_puppydog wrote: post #43
FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

These discussions always make me recal Jacob Applebaum. Think of him what you want, but this statement of his really stuck with me at the time. Paraphrasing: The real dark-net is facebook. Everything that goes in there never comes out again and is basically invisible to the world, except if you join facebook yourself. My own prime example of that used to be pinterest: it seems to be a 100% sink in the directed graph…

to me this is also discord - which seems to have become the chose alternative tk online forums for many communities and basically hides what used to be the public face of those communities.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#122

Post by pwiercinski »

Teandw wrote: post #62

This guy throws multiple reasons/conspiracies out there on why the website is really struggling to gain literally any sort of traction. Web is all bots, search engines not promoting competitors and being drowned out by SEO spam, yet he's failing to see the most obvious reason... the reason nearly all websites don't gain traction... Because it's a bad website. It provides no value to the user. I put in a few search te…

I guess the use-case just isn't that popular. It's a good website if you want to learn what some devs are up to, but barely anyone cares about that. Most people use search engines to find answers to their questions and Search My Site just doesn't work like that.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#123

Post by LegitShady »

maxwelldone wrote: post #104
mywaifuismeta wrote: post #29

Earlier quoted context omitted.

I no longer see Google as a neutral "search engine" the way it used to be. Now it's just another company that owns and promotes certain types of content, no different from reddit. For some things Google has the best content, for some things Twitter or Reddit have the best content.

Back in 2000s Google used to be the place for any type of search (IIRC). Now, I've been conditioned to use it only for specific use cases, mostly for convenience. Some examples include: 1. Anything programming related (searching for man pages, error codes etc) is straightforward. (I do have some UBO filters to exclude SO copycats) 2. Utility stuff like currency conversion, finding time in another city, weather etc. W…

another thing seems to be prioritizing current news over past news which makes searching for old.articles youve read quite difficult.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#124

Post by sdoering »

marginalia_nu wrote: post #111
ratww wrote: post #106

Earlier quoted context omitted.

This might be controversial, but I wish Google would exclude those websites too. Google started punishing keyword spam, then it started punishing black-hat comment spam. Even Youtube backtracked on the "videos have to be 10 minutes to rank". I wish they would do the same for carefully manicured SEO content farms too, as those sites are causing a harm worse than keyword-spammer sites did.

They're probably doing all they can. The problem is their dominance, both means they have effectively an entire industry looking for loopholes in everything they do, as well as legal considerations (arbitrarily punishing individual smaller actors might skirt on the territory of anti-competitive behavior)

I fear that Google also has a conflict of interest here. A lot of these non optimized sites are not interested in making money via ads. So Google wouldn't profit additionally from leading people there.

And a lot of people (myself often times included) are looking for a quick answer. A good enough answer. So good enough, SEO optimized is being surfaced. The result of an optimization war on both sides combined with the inevitable monetary interests.

I don't habe a solution. Sadly.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#125

Post by chairmanwow1 »

I created a temporary email service that was being used by about 10k users / week. Then several weeks ago, the number of users started growing like crazy up to about 60k users a day. Then we checked the recent email activity and 60k / 65k emails were from a social networking site.

Seems our service was being used to create fake bot accounts. The newly created accounts were obvious fakes. Rather than deal with the issue, we just shut the service off.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#126

Post by matsemann »

Cthulhu_ wrote: post #49
albatrosstrophy wrote: post #6

Ona tangential note, I remember a time when Google had the option to search only for 'discussions'. The results were amazing and accurate as it scoured online forums. Almost all issue I had (was following the rooting scene closely back then) were quickly resolved. Then suddenly it got removed for reasons unknown to me. Anyone knows if it's replicatable today?

I have a suspicion they removed it because of the amount of spam on those forums. There's tons of abandoned forums that are only occupied by spambots. There's even pretty convincing looking accounts and messages that turn out to be spam in the end, once they start trying to post links. I have Akismet on the comment section of the Wordpress front-end of the site I run, it basically said something like 99.99% of attemp…

Reminds of those "fake forums" I sometimes see when exhausting google's results. Found a screenshot of the concept here: https://www.reddit.com/r/Scams/comments/jxtr1k/but_it_requir...

Re: Almost all searches on my independent search engine are now from SEO spam bots

#127

Post by nl »

baxtr wrote: post #114
FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

I am not so sure... I think what happened is this: the WWW was everything back in the days. But in the "old days," only 10% of all people were online, the web elite. Then, AOL came, and the rest came online slowly but surely. The so-called "mainstream" people were no geeks, and these people were "just" ordinary people. Almost all were captured by what you call "big websites". Now, we see the 100% being dominated by t…

Eternal September was Sep 1993. AOL hit the internet in March 1994.

Netscape didn't launch until December 1994 (and the WWW was nothing before that. I subscribed to a mailing list with new sites that were released and I'd visit most new websites on the internet on most days with the Cello browser in my uni labs most days).

AOL users have been there since the beginning of the WWW.

https://en.m.wikipedia.org/wiki/Eternal_September

Re: Almost all searches on my independent search engine are now from SEO spam bots

#128

Post by psyc »

FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

Based on my observations over the past year, I’m certain that Google and Bing choose not to show us most of the web anymore.

I usually find what I’m looking for. It just takes literally three orders of magnitude longer than it used to for the same kind of stuff. I used to use Google a lot to jog my memory about various things I vaguely remembered. Type a few associative words and snippets, press Enter, done. Google’s useless for that now.

If you’re looking for hot pop shit in trendy publications, things to buy, commercial services to subscribe to - G has you covered. That’s what they do now.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#129

Post by NoGravitas »

api wrote: post #70
marginalia_nu wrote: post #54

Earlier quoted context omitted.

I built my search engine in part to explore whether this was actually true, and I don't think it actually is. There's still a lot of organic human-made content still out there, possibly more than ever, it's just not able to compete with the SEO industry that completely displaces it from Google and social media.

All open systems are destroyed by spam once they become popular enough to be profitable targets. This will eventually happen to the Fediverse too. If there is money to be made pissing all over the commons, the commons will be pissed all over. It even happens to proprietary silos if they are too open. Look at how many bots and spammers infest social media. Propaganda and disinformation can also be considered a form of…

You are probably right about the future; not necessarily because of spam, though that's a part of it, but just because of the toxicity of global, open to the world, mostly public social media. The Fediverse has mostly coasted by so far on obscurity, but it's not great, and it's bound to get worse. All of my online socializing these days is either through short-lived pseuds on topic-oriented fora, or invite-only Matrix rooms.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#130

Post by DethNinja »

Only solution is a webring based federated search engine.

1. You just put /webring.txt to your website. It shows links to other websites with a hard limit of 100 websites.

2. To combat spam and bots, search engine does accept blocklist as an input. So other people can curate the content.

3. People can personally rank the websites they like, so webring of the said website gets ranked higher for that specific user. This can be a community effort too.

4. Search engine itself should be under a commercial license so that other people can keep building it and add ads if they want to commercialise it.

I’m too busy to spend time with this but perhaps one day I can start coding it.

I’m convinced that search engine model of early internet is just dead, webrings are the way forward.

Post reply on HN