Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Almost all searches on my independent search engine are now from SEO spam bots

blog.searchmysite.net

Post reply on HN
Print view View original

291–300 of 391 posts

Re: Almost all searches on my independent search engine are now from SEO spam bots

#291

Post by Calavar »

minsc_and_boo wrote: post #279
MicahKV wrote: post #276

Earlier quoted context omitted.

That would probably help, but it's also a continuation of the cat and mouse game. There are plenty of captcha breaking services out there, it only cost about $1 to programmatically solve 1000 captchas.

Sure, but it increases friction that forces a re-eval of cost/benefit of the bot(s). Newest captcha services are a prediction score, not even a verification screen, and you can feed polluting data to bots you are certain to exist.

Agreed. I suspect that this is an arbitrage game on the part of the SEO spammers. Each search is cheaper for them than it is for a competitor who's using a major search engine with more extensive anti-spammer protections, and that difference equals $$$. A captcha doesn't have to be an unbeatable solution. It just has to provide enough of a barrier to equalize the cost.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#292

Post by dumbfounder »

m-i-l wrote: post #267
soheil wrote: post #231

Earlier quoted context omitted.

How exactly is a "general purpose search engine" different than a "search engine to try and find some of the fun and interesting content"?

The general purpose search engines search the whole internet, and as a result claim that you can search for anything on the whole internet, even going beyond that to answer questions which aren't on the internet as such, e.g. "What is my IP?" and "What time is it?". However, niche search engines only search specific parts of the internet, and only claim to be able to deliver results relating to their specific topic,…

I am a search guy and I would like you to succeed. But I don't get it. The name of the site is bland and makes me think you are a white label search service for websites. On the homepage it says "Open source search engine and search as a service for personal and independent websites." but it offers me to reason about why I (or anyone) would want to use it. The content it actually searches is random and of no real particular value as far as I can tell. Also, you are trying to avoid spam sites, but once you reach a certain size that's all you would see is people submitting spam sites. If you blocked people from submitting you would never get all the diamonds in the rough you are trying to expose.

You need to find an actual niche that solves a real problem people have and can understand and orient everything you do to tackling that. Then expand from there.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#293

Post by sylware »

MicahKV wrote: post #276
z3t4 wrote: post #271

Earlier quoted context omitted.

Or put those query results behind an anti-bot/"capcha" test.

That would probably help, but it's also a continuation of the cat and mouse game. There are plenty of captcha breaking services out there, it only cost about $1 to programmatically solve 1000 captchas.

... and there are the "click farms" with human beings.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#294

Post by iamwil »

To the OP of the article, this is great. I had just never known about it, to use it for searching.

Usually quality blog posts on specific technical topics are just things I run across through HN, lobsters, or twitter. Now it's one more channel to look for things that I'm specifically researching, like CRDTs. Kudos!

Re: Almost all searches on my independent search engine are now from SEO spam bots

#295

Post by blisterpeanuts »

john-radio wrote: post #260

Since everyone in this thread wants to jump down OP's throat about the quality of his web site, another interesting search engine is millionshort.com, which allows you to filter out the top N web sites from the results of your search. It's a great tool for looking past sites with good SEO; all you have to do is fiddle with the value of N. For example, searching for "electronic music box" as /u/ajnin suggested, with t…

That's an outstanding concept. One problem though: wouldn't it also filter out high quality curated results?

Re: Almost all searches on my independent search engine are now from SEO spam bots

#296

Post by lubesGordi »

ajnin wrote: post #38

This made me curious to try that search engine so I typed "electronic music box" (first thing that came to mind). As far as I can tell none or the 10+ pages of results include all those 3 words. I mean, you might not have any relevant sites in your database (likely if there are only 1000 sites or so as another of your blog posts imply), and I understand you want to show some result to the user, but if I want irreleva…

What the heck is an 'electronic music box'? I personally wouldn't expect those three words to show up on any sites served by a small search engine.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#297

Post by marginalia_nu »

oneeyedpigeon wrote: post #288
marginalia_nu wrote: post #281

Earlier quoted context omitted.

Is a comparison of document management systems something you expect actually find, as something written by humans? I wouldn't write such an article, I don't know who would. The only people who seem to be writing these types of comparison articles are spammers. I typed this reply without checking, but I checked now, and yeah -- if you google "document management system comparison", you get ads for document management…

2nd result I got from that exact search is an article from techradar: https://www.techradar.com/uk/best/best-document-management-s... Do you consider that search engine spam?

Yeah, that's affiliate marketing dressed up as a review. They're getting a kickback for several of the links in the review.

The deal on DocuWare is perhaps the most obvious, but the Abbyy-link also run through an affiliate marketing redirect service.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#298

Post by MicahKV »

Calavar wrote: post #291
minsc_and_boo wrote: post #279

Earlier quoted context omitted.

Sure, but it increases friction that forces a re-eval of cost/benefit of the bot(s). Newest captcha services are a prediction score, not even a verification screen, and you can feed polluting data to bots you are certain to exist.

Agreed. I suspect that this is an arbitrage game on the part of the SEO spammers. Each search is cheaper for them than it is for a competitor who's using a major search engine with more extensive anti-spammer protections, and that difference equals $$$. A captcha doesn't have to be an unbeatable solution. It just has to provide enough of a barrier to equalize the cost.

I'm not so sure about this. The spammers goal is to build up as big a list of link spam targets as possible. If one spammer chooses to only scrape minor engines and another only major engines, the one scraping the major engines will probably come out on top despite the higher cost. Whoever is abusing OP's search engine is likely doing it to supplement the data they are already scraping from the major engines.

For OP, I think simply not returning results at all is a more practical measure because it removes the reward completely. Captchas and bot detection keep the reward in play, while taking away the results entirely makes the entire pursuit futile.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#299

Post by randomstring »

Search traffic has always been mostly automated spam bots.

Even back in the Open Directory Days when we powered part of search.netscape.com I estimated 80+% of all search traffic was automated. At least most of it self-identified with the same Java useragent.

Later when working Topix, despite being a news search engine, most traffic was bot traffic. Most included the word “mortgage” in the query. Topix specialized in localized content, and that was very popular for SEO scrapers.

Lastly at Blekko, I estimate 90+% of traffic was automated. By then maybe half or more learned to change the user agent. Most used HTTP/1.0, a dead giveaway as no browser still uses 1.0. This was a major aspect in Blekko's load shedding strategy. If the servers started to get overloaded, we'd start bouncing suspected bot traffic to a redirect that would show in the logs. If there was a human with a modern browser running javascript on the other end, would get redirect to a link that wouldn't get bounced. I would check the logs weekly to see if any humans got caught. None ever did. This was a huge monetary savings, you only need 1/10th the servers if you can safely ignore the bots.

Often it's endless repetition of the same keywords in a random order with a place name appended, or prepended, or inserted. over and over. Often variations on known monetizatable SEO keywords. However, much of it doesn't make any sense.

I don't have any insight into Google's numbers but I would conservatively estimate 95% or more of all their queries are automated bots and not humans. And the level of spy-vs-spy going on for Google CPU resources vs SEO bots is probably pretty evolved by now. I stopped tracking many years ago when Google switched to densely packed obfuscated javascript for page renders. Maybe this is part of why automated queries are so high across the web, maybe google is too hard to crack for most.

Post reply on HN