Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Almost all searches on my independent search engine are now from SEO spam bots

blog.searchmysite.net

Post reply on HN
Print view View original

321–330 of 391 posts

Re: Almost all searches on my independent search engine are now from SEO spam bots

#321

Post by mfrye0 »

I run a data aggregation company that has a fairly advanced scraping infrastructure for collecting data across the web. Having built the scraping side, I'm pretty familiar with most of the strategies for avoiding bot detection.

Coming from that perspective, detecting and stopping at least the majority of bots out there is fairly doable, and I put together a rudimentary thing for a side project.

The core of it uses an IP API for looking up the requesting IP to identify the country and if it's coming from a data center, VPN, Tor, etc. If it passes that, I trigger Google Captcha to show up. Lastly, I track IPs that make it through and have some basic rules in place to try to detect patterns and block offenders that way.

There's a bunch more stuff you can check for, but the core of it is basically filtering out data center traffic to minimize the requests going to Google Captcha.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#322

Post by superjan »

randomstring wrote: post #299

Search traffic has always been mostly automated spam bots. Even back in the Open Directory Days when we powered part of search.netscape.com I estimated 80+% of all search traffic was automated. At least most of it self-identified with the same Java useragent. Later when working Topix, despite being a news search engine, most traffic was bot traffic. Most included the word “mortgage” in the query. Topix specialized in…

Almost sounds like it is justified to add a javascript crypto miner to your pages to make the bots pay for the use of your service.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#323

Post by nonrandomstring »

pde3 wrote: post #286
nonrandomstring wrote: post #226

Earlier quoted context omitted.

> The real value of the old internet was showing us what is possible. Of equal value is that it showed us what not to do. We have 30 years of documentation for research on exactly what a successful intra-planetary network needs to be immune to. A successful future network must build-in resistance all forms of human pyschopathology from the ground up.

This is a nice fantasy, but it's a fantasy. The tech stack and network we have is too dense a forest to be replaced by clean slate designs. But maybe some of the problems could be improved with some new platforms and APIs. Mind you, ML is making so much progress so quickly that what happened over the last thirty years is at best a partial model of the problem we have to solve now, and the tools we have to do it with.…

> ML is making so much progress so quickly that what happened over the last thirty years is at best a partial model of the problem we have to solve now, and the tools we have to do it with...

Sorry I don't see how ML can help here. It seems like another thing to pin hopes of repairing an already too broken system on.

"We cannot solve our problems with the same thinking we used when we created them." -- Albert Einstein

"A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it." -- Max Planck

We are the dying generation my friend. We built it. They came. It didn't work. Surely if ML can do anything it's telling us that we need to tear down the old system completely and start again, don't you think? Adding sticking tape won't help.

edit: turning a grunt into an honest question

Re: Almost all searches on my independent search engine are now from SEO spam bots

#324

Post by inetknght »

folkrav wrote: post #272
inetknght wrote: post #246

Earlier quoted context omitted.

> You surf with JS off because of sites abusing their users' data. This is not it. Wrong. I surf with JS off because of sites that use JS to collect information about me. If it's available on the server, then sure that might be considered fair game. But using javascript (or any other client-side tool) to do what you should instead do server-side is abusing users (or their data). Putting analytics inline so it's "not…

> Wrong. I surf with JS off because of sites that use JS to collect information about me. Plausible doesn't collect information about you, but the site's usage. Do you also object to physical stores putting up cameras? Here's their own instance, open to public. https://plausible.io/plausible.io > If it's available on the server, then sure that might be considered fair game. But using javascript (or any other client-s…

> Plausible doesn't collect information about you, but the site's usage. Do you also object to physical stores putting up cameras?

The difference is that the cameras don't get attached to my physical body, doesn't have any ability to monitor my actions after I have left the presence of the physical store, and can't force me to take any physical item or action.

Javascript, on the other hand, has the capability to become persistent, can monitor my computer's activity outside of your website, and can leave a lot (!) of additional data on my computer without my permission.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#325

Post by ajmurmann »

tmaly wrote: post #205
ColinHayhurst wrote: post #137

Earlier quoted context omitted.

Agreed. > If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just legacy content and spam. The "general" Internet is not dead. Though if you just want to participate in just Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord you might well think that. Users of marginalia (aut…

Everyone is trying to game the Google algorithm. The net result is all this long form content and cooking recipes that are 10 pages long. There seems to be a big disconnect with a typical users attention span and the length of a post.

I thought the recipe thing was to be able to copyright them

Re: Almost all searches on my independent search engine are now from SEO spam bots

#326

Post by [deleted] »

ColinHayhurst wrote: post #283
Domenic_S wrote: post #216

Earlier quoted context omitted.

> The "general" Internet is not dead. For some things it is. Good luck getting a non-sponsored/SEO-gamed review of a kitchen appliance or particular vacation mode such as a cruise. It's flabbergasting. Most times I just stick "inurl:reddit.com" in my search and try to get discussion threads about the thing I'm researching, but even that's getting filled up with shills.

Result #1 & #2 for kitchen appliance review (your personalised/local results might vary): Google: https://www.expertreviews.co.uk/home-garden/home-appliances https://www.goodhousekeeping.com/appliances/ Bing: https://www.which.co.uk/reviews/fitted-kitchens/article/plan... https://www.goodhousekeeping.com/appliances/ DDG: https://www.goodhousekeeping.com/appliances/ https://www.which.co.uk/reviews/fitted-kitchens/arti…

[deleted]

Re: Almost all searches on my independent search engine are now from SEO spam bots

#327

Post by FargaColora »

ColinHayhurst wrote: post #283
Domenic_S wrote: post #216

Earlier quoted context omitted.

> The "general" Internet is not dead. For some things it is. Good luck getting a non-sponsored/SEO-gamed review of a kitchen appliance or particular vacation mode such as a cruise. It's flabbergasting. Most times I just stick "inurl:reddit.com" in my search and try to get discussion threads about the thing I'm researching, but even that's getting filled up with shills.

Result #1 & #2 for kitchen appliance review (your personalised/local results might vary): Google: https://www.expertreviews.co.uk/home-garden/home-appliances https://www.goodhousekeeping.com/appliances/ Bing: https://www.which.co.uk/reviews/fitted-kitchens/article/plan... https://www.goodhousekeeping.com/appliances/ DDG: https://www.goodhousekeeping.com/appliances/ https://www.which.co.uk/reviews/fitted-kitchens/arti…

Most of these are spam. They contain affiliate links to Amazon to buy the product which is being reviewed, therefore the the review cannot be trusted.

"Which" looks to be the exception, but that is a paid-for service.

It's a sad state of affairs.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#328

Post by FargaColora »

marginalia_nu wrote: post #224
Domenic_S wrote: post #216

Earlier quoted context omitted.

> The "general" Internet is not dead. For some things it is. Good luck getting a non-sponsored/SEO-gamed review of a kitchen appliance or particular vacation mode such as a cruise. It's flabbergasting. Most times I just stick "inurl:reddit.com" in my search and try to get discussion threads about the thing I'm researching, but even that's getting filled up with shills.

¯\_(ツ)_/¯ http://www.jitterbuzz.com/indmix.html http://www.alaska.net/~akpassag/

These websites seem to be last updated decades ago, which is prehistoric to most casual browsers. There's no doubt there is great content on the general internet, but these examples I would classify as "legacy".

Re: Almost all searches on my independent search engine are now from SEO spam bots

#329

Post by tofuahdude »

z3t4 wrote: post #271
MicahKV wrote: post #252

So spammers have latched onto your search engine because they are getting useful results. They are able to systematically discover websites built on certain platforms that allow users to post content containing links, which they can target for link spam. It is very difficult to fight this on a technical level because there is an entire industry built around blackhat SEO, with all kinds of softwares and services dedic…

Or put those query results behind an anti-bot/"capcha" test.

Captcha breaking is SO easy these days; even the modern captchas are easy to defeat.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#330

Post by Jenk »

FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

> If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc.

High chances you will find a link to an external site over content actually on those big named sites though, right? That tells us the organic web isn't dead, it's just hard to discover/navigate - because of SEO wars, most probably... The problem isn't the lack of content, it's the number of shitty spammy sites standing in your way of the sites you actually want to see. Like a sleazy salesman trying to direct you to the crap laden three wheeled rust bucket when you were heading toward the family sedans.

Post reply on HN