I had to put my search engine behind Cloudflare to deal with this. Like the volume grew to about 10x the traffic I saw sitting at the front page of Hacker News for a full week.
Almost all searches on my independent search engine are now from SEO spam bots
51–60 of 391 posts
Re: Almost all searches on my independent search engine are now from SEO spam bots
#52That's really interesting... and sad. For what it's worth, I've noticed comment bots dramatically increase over the last year too. They have always been there, but looking at Reddit, YouTube, etc, now there seem to be 10x more than there were a few years earlier. Even on HN it has gotten worse.
On two occasions I’ve read one of my comments here on HN copied and posted on Reddit. The user profiles that copied my comments in seemed like they were run by a real person but the rest of their posts might have all been scraped as well.
I only found out because I just so happened to be looking at the comments on a related news story and quickly realized the post sounded strangely familiar. I’m sure most of us here have had our comments copied without our knowledge.
Re: Almost all searches on my independent search engine are now from SEO spam bots
#53If this was the spam for a search engine (almost) nobody uses, it makes you wonder how much abuse the major search engines face
My understanding is that this wasn't about gaming this particular search engine itself, and more about the spammers using the search engine for its intended purpose of finding spam-free content so they can then use this content as copy for their spam posts.
Re: Almost all searches on my independent search engine are now from SEO spam bots
#54You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…
I built my search engine in part to explore whether this was actually true, and I don't think it actually is.
There's still a lot of organic human-made content still out there, possibly more than ever, it's just not able to compete with the SEO industry that completely displaces it from Google and social media.
Re: Almost all searches on my independent search engine are now from SEO spam bots
#55Earlier quoted context omitted.
I wonder how that could be accomplished. Maybe they'll build a brain interface to replace the "I'm not a robot" captchas/add a TPM chip to the brain. And then the spammers will start selling tools to fake the responses. Or pay Filipinos a few cents a month to have the chip implanted to their brains...
> Or pay Filipinos a few cents a month to have the chip implanted to their brains... That's the problem with blocking bots as opposed to malicious behavior. Bot blocking is actually trivial and very cheap to bypass as long as you can buy slave-like labor for peanuts. Ideally you'd want to block malicious behavior (when it comes to SEO spam, downrank anything for-profit such as ads, analytics, affiliate links, etc) in…
It's a principal-agent problem. Websites want to be paid for their content, rent ad space, advertisers want users to see ads, users want to find content.
The agent in the middle fucking over all three principals is hmm. Metaälphabetic, let's say.
Re: Almost all searches on my independent search engine are now from SEO spam bots
#56You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…
But Twitter, Reddit, HN, and most other such places are just websites and can be indexed fine. Same with Wikipedia, which is very much a silo (they don't have regular links in text in the hypertext spirit, but only footnotes).
Facebook and Instagram are more of a walled garden, like Quora, but there is a lot of junk there anyway.
It's sad for the WWW, but I don't really think it is a fundamental problem for search engines. In fact Twitter for example gives a direct pipe to Google. If you tweet something, it is immediately findable. Similar for StackExchange, but there I think the site is so "small" that Google can afford to just continuously index it.
Re: Almost all searches on my independent search engine are now from SEO spam bots
#57> I didn’t notice at first because the web analytics only shows real users, and the unusual activity could only be seen by looking at the server logs. Sounds like everyone blocking analytics (Plausible in this case), e.g. myself just now, is lumped in with spam bots. Of course, analytics blocking can’t meaningfully swing the ~99.99% statistic.
If you self-host Plausible, it's also possible to bundle the analytics package with the website, so that there's isn't an "ad-blockable" lone request for the .js file. https://github.com/plausible/plausible-tracker
Yeah there is. I surf with JS off because of people like you.
Re: Almost all searches on my independent search engine are now from SEO spam bots
#58Earlier quoted context omitted.
They face a lot. I always browse with incognito on safari, and I quite often have to do captchas on google and bing etc. to prove I’m not a computer… If there is money involved and value in being able to trick search engines, I’m not surprised it’s a thriving business of grift.
Why do you use Bing?
Why don't you like diversity, in this case diversity of search engines? Bing may have its problems but so does Google, the way to handle this is to either use many different engines or to use a meta-search engine like Searx. The latter is far easier so it is what I do. Just relying on a single source makes you an easy target for those who control that source.
Re: Almost all searches on my independent search engine are now from SEO spam bots
#59You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…
But Twitter, Reddit, HN, and most other such places are just websites and can be indexed fine. Same with Wikipedia, which is very much a silo (they don't have regular links in text in the hypertext spirit, but only footnotes). Facebook and Instagram are more of a walled garden, like Quora, but there is a lot of junk there anyway. It's sad for the WWW, but I don't really think it is a fundamental problem for search en…
Twitter and Reddit still can be indexed, but they've also become increasingly hard to use without an account. Reddit doesn't let you fully expand threads when you're unlogged. Twitter limits the amount of things you can read and shows a modal. Both of them heavily limit usage on mobile devices without installing an app.
Sure, an account is free but might require giving information you don't want to give. Twitter asks me for a phone number a few minutes after creating an account, even if I don't post anything). Reddit at least lets you skip giving an email.
Sure, there are workarounds such as using lite versions (old Reddit, mobile Twitter), but that's not known to all people coming from a search engine.
It feels as if HN are the only one that's not a partially walled garden yet (and Wikipedia of course).
Re: Almost all searches on my independent search engine are now from SEO spam bots
#60You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…
But Twitter, Reddit, HN, and most other such places are just websites and can be indexed fine. Same with Wikipedia, which is very much a silo (they don't have regular links in text in the hypertext spirit, but only footnotes). Facebook and Instagram are more of a walled garden, like Quora, but there is a lot of junk there anyway. It's sad for the WWW, but I don't really think it is a fundamental problem for search en…
And isn't great to get a link to Reddit or Twitter, and you click the link, and try to navigate to the comments for context or the answer, and you go to click the link to expand it, and then you get a demand to log in and install their app? Don't talk about walled gardens and not include Reddit or Twitter just because they let you look at one brick before demanding their tax.