Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Almost all searches on my independent search engine are now from SEO spam bots

blog.searchmysite.net

Post reply on HN
Print view View original

81–90 of 391 posts

Re: Almost all searches on my independent search engine are now from SEO spam bots

#81

Post by pixl97 »

indigochill wrote: post #72
marginalia_nu wrote: post #54

Earlier quoted context omitted.

I built my search engine in part to explore whether this was actually true, and I don't think it actually is. There's still a lot of organic human-made content still out there, possibly more than ever, it's just not able to compete with the SEO industry that completely displaces it from Google and social media.

How do you surface organic human content? I happen to linger around the fediverse/tildeverse sphere where I see organic content from people I personally have a direct (digital) connection to (and I started self-hosting my music after Epic bought Bandcamp), but I'm not clear on how I'd go about digging that kind of stuff up in the more general case.

Welcome to the billion dollar question. Any place that is authentic will face the zombie horde attempting to fake authenticity in order to capture attention.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#82

Post by marginalia_nu »

api wrote: post #70
marginalia_nu wrote: post #54

Earlier quoted context omitted.

I built my search engine in part to explore whether this was actually true, and I don't think it actually is. There's still a lot of organic human-made content still out there, possibly more than ever, it's just not able to compete with the SEO industry that completely displaces it from Google and social media.

All open systems are destroyed by spam once they become popular enough to be profitable targets. This will eventually happen to the Fediverse too. If there is money to be made pissing all over the commons, the commons will be pissed all over. It even happens to proprietary silos if they are too open. Look at how many bots and spammers infest social media. Propaganda and disinformation can also be considered a form of…

As old open systems are destroyed, new ones are created to replace them. The Internet exists in a constant state of rebirth and transformation. You really can't step into the same river twice.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#83

Post by pixl97 »

api wrote: post #70
marginalia_nu wrote: post #54

Earlier quoted context omitted.

I built my search engine in part to explore whether this was actually true, and I don't think it actually is. There's still a lot of organic human-made content still out there, possibly more than ever, it's just not able to compete with the SEO industry that completely displaces it from Google and social media.

All open systems are destroyed by spam once they become popular enough to be profitable targets. This will eventually happen to the Fediverse too. If there is money to be made pissing all over the commons, the commons will be pissed all over. It even happens to proprietary silos if they are too open. Look at how many bots and spammers infest social media. Propaganda and disinformation can also be considered a form of…

It's not cynical, is how every system in nature works. Everything alive must develop an immune system or it is attacked and eaten.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#84

Post by marginalia_nu »

indigochill wrote: post #72
marginalia_nu wrote: post #54

Earlier quoted context omitted.

I built my search engine in part to explore whether this was actually true, and I don't think it actually is. There's still a lot of organic human-made content still out there, possibly more than ever, it's just not able to compete with the SEO industry that completely displaces it from Google and social media.

How do you surface organic human content? I happen to linger around the fediverse/tildeverse sphere where I see organic content from people I personally have a direct (digital) connection to (and I started self-hosting my music after Epic bought Bandcamp), but I'm not clear on how I'd go about digging that kind of stuff up in the more general case.

I do a traditional web crawl and exclude anything that looks too much like it wants a high google ranking. Nothing to it.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#85

Post by Auguste »

I'm disappointed that Search My Site isn't seeing many legitimate viewers.

Just wanted you to know that I'm a fan. I love reading peoples personal websites, and Search My Site has been great for discoverability. I visit the Newest Pages and Browse Sites pages once or twice a week to check out the new sites being indexed.

I don't know what the answer is to the spam bots, but you do have some real visitors out there. :)

Re: Almost all searches on my independent search engine are now from SEO spam bots

#86

Post by m-i-l »

jacquesm wrote: post #9

One day we'll have an internet for humans exclusively. On another note, with 160K requests / day from bots you could of course simply block the bots structurally assuming they are nice enough to identify themselves. Block all of AWS and Google, Russia, China, NK and a couple of other bot hot spots and the service may well become more successful for regular users because they get faster results. Bots can afford to wai…

This isn't indexing by search engine spiders, which are usually fairly benign and easy to identify with user agent etc. This is searches for "scraping footprints" executed en mass by "SEO proxy farms", which are designed to be very difficult to detect (e.g. originating from globally distributed residential IPs, quite possibly ordinary home user's machines which have been compromised). The main giveaway that something is a "scraping footprint" is the long search query which includes text that would appear on a template, e.g. ""This website is proudly using the open source classifieds software OSClass" rega turntables", for someone looking for OSClass-powered pages they could "search engine optimise" for the query "rega turntables".

Re: Almost all searches on my independent search engine are now from SEO spam bots

#87

Post by varun_ch »

pluc wrote: post #57
scambier wrote: post #30

Earlier quoted context omitted.

If you self-host Plausible, it's also possible to bundle the analytics package with the website, so that there's isn't an "ad-blockable" lone request for the .js file. https://github.com/plausible/plausible-tracker

Yeah there is. I surf with JS off because of people like you.

Most of the data you can collect with Plausible could just be collected server side instead, it's nothing like Google Analytics.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#88

Post by superasn »

To me it looks like some popular spamming software (like thebestspinner, etc) just integrated you and now everyone who is the software is now hitting your site.

The good news in this case is that's it'll be easy to spot the pattern and block it, the bad news is you're entering a never-ending cat and mouse game.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#89

Post by alxlaz »

marginalia_nu wrote: post #54
FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

I built my search engine in part to explore whether this was actually true, and I don't think it actually is. There's still a lot of organic human-made content still out there, possibly more than ever, it's just not able to compete with the SEO industry that completely displaces it from Google and social media.

This matches my findings 100%. The WWW is active and bubbling, but virtually all the cool websites I've found in the last 10 years or so came through friends, small IRC channels, or more recently through marginalia.nu :-). Google and friends are facilitators for the SEO and tracking industries, so of course they have zero interest to prioritize these things over content spam -- their whole business runs on content spam. But the WWW is as alive as it gets.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#90

Post by pjmlp »

jacquesm wrote: post #9

One day we'll have an internet for humans exclusively. On another note, with 160K requests / day from bots you could of course simply block the bots structurally assuming they are nice enough to identify themselves. Block all of AWS and Google, Russia, China, NK and a couple of other bot hot spots and the service may well become more successful for regular users because they get faster results. Bots can afford to wai…

And the cycle will reboot itself again.

The silos we have nowadays were there before the Internet took off, on BBS, Compuserve, Geocities, ....

Apparently the majority of regular humans likes to have centralized providers they can reach out to, instead of the freadom of decentralized content.

Post reply on HN