Not surprised. I see many startups with Head of SEO (Search engine optimization) with huge salaries now a days.
Almost all searches on my independent search engine are now from SEO spam bots
11–20 of 391 posts
Re: Almost all searches on my independent search engine are now from SEO spam bots
#12It's been the same for unscatter.com for years but I've always attributed to that to me not having a real marketing strategy or even sticking with the ones I've tried to start.
Re: Almost all searches on my independent search engine are now from SEO spam bots
#13That's really interesting... and sad. For what it's worth, I've noticed comment bots dramatically increase over the last year too. They have always been there, but looking at Reddit, YouTube, etc, now there seem to be 10x more than there were a few years earlier. Even on HN it has gotten worse.
Re: Almost all searches on my independent search engine are now from SEO spam bots
#14Ona tangential note, I remember a time when Google had the option to search only for 'discussions'. The results were amazing and accurate as it scoured online forums. Almost all issue I had (was following the rooting scene closely back then) were quickly resolved. Then suddenly it got removed for reasons unknown to me. Anyone knows if it's replicatable today?
Sometimes adding "reddit" to a search query produces fantastic results.
I do this all the time
Re: Almost all searches on my independent search engine are now from SEO spam bots
#15One day we'll have an internet for humans exclusively. On another note, with 160K requests / day from bots you could of course simply block the bots structurally assuming they are nice enough to identify themselves. Block all of AWS and Google, Russia, China, NK and a couple of other bot hot spots and the service may well become more successful for regular users because they get faster results. Bots can afford to wai…
I wonder how that could be accomplished. Maybe they'll build a brain interface to replace the "I'm not a robot" captchas/add a TPM chip to the brain.
And then the spammers will start selling tools to fake the responses. Or pay Filipinos a few cents a month to have the chip implanted to their brains...
Re: Almost all searches on my independent search engine are now from SEO spam bots
#16> I didn’t notice at first because the web analytics only shows real users, and the unusual activity could only be seen by looking at the server logs. Sounds like everyone blocking analytics (Plausible in this case), e.g. myself just now, is lumped in with spam bots. Of course, analytics blocking can’t meaningfully swing the ~99.99% statistic.
I would argue that yes, it can. If the only people who are interested in using the website are those who block analytics - and, given the demographic of a niche search engine, it doesn't sound entirely implausible - then there's no telling how the 99.99% splits into bots and nerds.
Re: Almost all searches on my independent search engine are now from SEO spam bots
#17This is an awsome website that I was not aware of!
and there you have it: nobody uses it because nobody knows of it.
of course for a bot it is easy to remember your site, its just another url in a long list of others... but what does a human do? they go to their fav search site, be it duck duck go, google or bing... perhaps even yahoo.
i remember when google just started out, back then you would have used askjeeves, altavista or yahoo... google was really good compared to those... and the name was new, kinda orthogonal to existing search engines (except yahoo perhaps) and perhaps the most important bit: the site was "clean" except for the searchbar, there was nothing distracting there. you opened it and knew it is for looking up stuff
now, to join in, this late in the game? difficult. difficult.
maybe it would be easier if it specialized for some niche? idk.
dear OP: i'll try to remember your searchengine, but i cant promise to become a regular
Re: Almost all searches on my independent search engine are now from SEO spam bots
#18Google puts a _lot_ of effort into avoiding SEO spam, but it's a red queen problem.
Re: Almost all searches on my independent search engine are now from SEO spam bots
#19If this was the spam for a search engine (almost) nobody uses, it makes you wonder how much abuse the major search engines face
I'd assume they have more control though. I noticed whenever I use Google after connecting to NordVPN, it requires a captcha the first time.
Re: Almost all searches on my independent search engine are now from SEO spam bots
#20Ona tangential note, I remember a time when Google had the option to search only for 'discussions'. The results were amazing and accurate as it scoured online forums. Almost all issue I had (was following the rooting scene closely back then) were quickly resolved. Then suddenly it got removed for reasons unknown to me. Anyone knows if it's replicatable today?
Brave Search does have a discussion search section.