Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Almost all searches on my independent search engine are now from SEO spam bots

blog.searchmysite.net

Post reply on HN
Print view View original

351–360 of 391 posts

Re: Almost all searches on my independent search engine are now from SEO spam bots

#351

Post by go_prodev »

MicahKV wrote: post #298
Calavar wrote: post #291

Earlier quoted context omitted.

Agreed. I suspect that this is an arbitrage game on the part of the SEO spammers. Each search is cheaper for them than it is for a competitor who's using a major search engine with more extensive anti-spammer protections, and that difference equals $$$. A captcha doesn't have to be an unbeatable solution. It just has to provide enough of a barrier to equalize the cost.

I'm not so sure about this. The spammers goal is to build up as big a list of link spam targets as possible. If one spammer chooses to only scrape minor engines and another only major engines, the one scraping the major engines will probably come out on top despite the higher cost. Whoever is abusing OP's search engine is likely doing it to supplement the data they are already scraping from the major engines. For OP,…

Deliberately feeding the spam bots into an endless loop of captchas might slowly drain their accounts if they are paying 3rd party captcha farms.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#352

Post by MicahKV »

marginalia_nu wrote: post #350
MicahKV wrote: post #252

So spammers have latched onto your search engine because they are getting useful results. They are able to systematically discover websites built on certain platforms that allow users to post content containing links, which they can target for link spam. It is very difficult to fight this on a technical level because there is an entire industry built around blackhat SEO, with all kinds of softwares and services dedic…

> So spammers have latched onto your search engine because they are getting useful results. I'm not sure about this. At least with my search engine, it doesn't really seem to matter what response they get, I don't even think they look at the responses. They keep hammering away with tens of thousands of queries per day with the requests even though they've seen nothing but HTTP Status 403 since last October or so. My…

Huh, well I guess there goes my theory about the incentive. What a bummer. I would have thought that at least with search engine scraping, they would stop expending the effort once the results dried up.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#353

Post by efreak »

6510 wrote: post #61
Cthulhu_ wrote: post #49

Earlier quoted context omitted.

I have a suspicion they removed it because of the amount of spam on those forums. There's tons of abandoned forums that are only occupied by spambots. There's even pretty convincing looking accounts and messages that turn out to be spam in the end, once they start trying to post links. I have Akismet on the comment section of the Wordpress front-end of the site I run, it basically said something like 99.99% of attemp…

Everyone is a spammer according to Akismet. I wouldn't be surprised if 99% of that 99.9999% is false positives. You could start a website for people you don't like, flag all the comments as spam and they wont be allowed to post anything elsewhere - forever!

That percentage sounds about right to me. I've seen comments on blogs from ~10-15 years ago, that continue to have spam posted to them. The first 2-3 comments will be relevant, but comments 50-100 may have a single relevant comment along them, with a total of anywhere from 300-3000 comments. Older comments link mainly to blogs (*.WordPress.com) and such, while newer comments link to Facebook and Instagram.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#354

Post by pixl97 »

tomxor wrote: post #132
pixl97 wrote: post #81

Earlier quoted context omitted.

Welcome to the billion dollar question. Any place that is authentic will face the zombie horde attempting to fake authenticity in order to capture attention.

I think your almost right, but it's not necessarily authenticity... I think it's just money. Large "authentic" search engines can exist to serve the rest of the web, those personal blogs and other small communities. Those sites have a natural tendency to not be trying to turn everything into a revenue stream, so if that was the prerequisite for an engine, it would be a perfect match and naturally dissuade marketing t…

Authenticity is worth money.

When you have a 'real' community you're talking about real people with real salaries and desires, add in that you tend to develop a real trust between members. Think of this as fertilized soil. You can grow crops in it, but weed seeds will eventually land and try to take over it.

HackerNews is a good example of this, it takes a healthy amount of moderation to keep things on topic where things like politics get peared pretty ruthlessly. If for a minute Dang gave in found ways to additionally monetize the forums, something that would be profitable for a while at least, things would start down a bad path.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#355

Post by pascalxus »

MicahKV wrote: post #252

So spammers have latched onto your search engine because they are getting useful results. They are able to systematically discover websites built on certain platforms that allow users to post content containing links, which they can target for link spam. It is very difficult to fight this on a technical level because there is an entire industry built around blackhat SEO, with all kinds of softwares and services dedic…

just to throw out ideas: What if he decided to charge for each search?, say 1 cent or so. Users could purchase them in bulk, say 100 searches for a 1$.

The world is getting more and more desperate for a better search engine. the day may come, when people are willing to pay for better results.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#356

Post by miohtama »

marginalia_nu wrote: post #350
MicahKV wrote: post #252

So spammers have latched onto your search engine because they are getting useful results. They are able to systematically discover websites built on certain platforms that allow users to post content containing links, which they can target for link spam. It is very difficult to fight this on a technical level because there is an entire industry built around blackhat SEO, with all kinds of softwares and services dedic…

> So spammers have latched onto your search engine because they are getting useful results. I'm not sure about this. At least with my search engine, it doesn't really seem to matter what response they get, I don't even think they look at the responses. They keep hammering away with tens of thousands of queries per day with the requests even though they've seen nothing but HTTP Status 403 since last October or so. My…

Put a CloudFlare web application firewall at the front of the site and then use its rate limited / CAPTCHA features to throttle traffic. It is the easiest way to get rid of parasitic scraping and API abuse. Cost is $0.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#357

Post by wibyweb »

m-i-l wrote: post #254
wibyweb wrote: post #144

In late April up to now, Wiby (a small mostly unheard of search engine) began having the exact same issue. Tens of thousands of the exact same type of "powered by..." requests coming from thousands of IPs. They are using a tool called QHub.

Thanks for wiby.me. I have seen QHub coming up in the scraping footprints, but my assumption has been that the footprint query is looking for Question and Answers sites powered by QHub containing their targeted terms, e.g. because there's a known vulnerability with QHub that their scripts can exploit to auto-post backlinks or whatever it is they do. There are lots of other hosting tools, other than QHub, that come up…

Interesting, thanks for that extra info.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#358

Post by wolpoli »

MicahKV wrote: post #252

So spammers have latched onto your search engine because they are getting useful results. They are able to systematically discover websites built on certain platforms that allow users to post content containing links, which they can target for link spam. It is very difficult to fight this on a technical level because there is an entire industry built around blackhat SEO, with all kinds of softwares and services dedic…

Considering that as of Mar 12, this search engine only has 1001 sites indexed, I am not sure how useful this site is for getting SEO backlinks. Speaking of which, are backlinks still a thing these days?

Re: Almost all searches on my independent search engine are now from SEO spam bots

#359

Post by bladegash »

wolpoli wrote: post #358
MicahKV wrote: post #252

So spammers have latched onto your search engine because they are getting useful results. They are able to systematically discover websites built on certain platforms that allow users to post content containing links, which they can target for link spam. It is very difficult to fight this on a technical level because there is an entire industry built around blackhat SEO, with all kinds of softwares and services dedic…

Considering that as of Mar 12, this search engine only has 1001 sites indexed, I am not sure how useful this site is for getting SEO backlinks. Speaking of which, are backlinks still a thing these days?

They are, but the useful ones are those coming from sites with higher domain authority rankings.

That’s why you'll see fluff pieces (aka, paid content) from online publications like Forbes for the better funded entities.

Another approach is the reach out to site operators with offers of writing content or asking them to link to your site’s content in their existing content.

It’s expensive and/or incredibly time consuming to get back links that matter.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#360

Post by marginalia_nu »

miohtama wrote: post #356
marginalia_nu wrote: post #350

Earlier quoted context omitted.

> So spammers have latched onto your search engine because they are getting useful results. I'm not sure about this. At least with my search engine, it doesn't really seem to matter what response they get, I don't even think they look at the responses. They keep hammering away with tens of thousands of queries per day with the requests even though they've seen nothing but HTTP Status 403 since last October or so. My…

Put a CloudFlare web application firewall at the front of the site and then use its rate limited / CAPTCHA features to throttle traffic. It is the easiest way to get rid of parasitic scraping and API abuse. Cost is $0.

Yeah, that's essentially what I've done, except I'm paying for their cheapest non-free tier to have a bit more control over it. I really wish I didn't have to route all my traffic through an untrusted a 3rd party like that, but I guess we can't have nice things on the Internet anymore.
Post reply on HN