Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Almost all searches on my independent search engine are now from SEO spam bots

blog.searchmysite.net

Post reply on HN
Print view View original

341–350 of 391 posts

Re: Almost all searches on my independent search engine are now from SEO spam bots

#341

Post by consp »

mdoms wrote: post #312
john-radio wrote: post #260

Since everyone in this thread wants to jump down OP's throat about the quality of his web site, another interesting search engine is millionshort.com, which allows you to filter out the top N web sites from the results of your search. It's a great tool for looking past sites with good SEO; all you have to do is fiddle with the value of N. For example, searching for "electronic music box" as /u/ajnin suggested, with t…

Million Short also has an option to remove only e-commerce results which is invaluable if you still want results from sites like Twitter, Wikipedia and YouTube but don't want online shopping spam.

Would this also work for the fake-sites-stealing-text-to-look-legit sites since they quickly end up in the top results?

Re: Almost all searches on my independent search engine are now from SEO spam bots

#342

Post by heavyset_go »

FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

I was once on this bandwagon, but I think it was just confirmation bias reflecting the way I used the internet at the time. The non-siloed internet is bigger than the pre-siloed internet ever was.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#343

Post by groby_b »

matheusmoreira wrote: post #215
eloisius wrote: post #189

Earlier quoted context omitted.

Good luck, spectrum is highly regulated in every country I can think of. If national governments don’t want you networking across borders, you’re definitely not going to be broadcasting long range radio transmissions that way. In fact, it’s currently illegal to transmit encrypted data or to relay packets via ham radio in the US.

Who knows? The whole point of decentralization is for there to be so many nodes in the network they can't possibly take them all down so that it's pointless to even try. What if all smartphones formed a mesh network? There aren't enough prisons in my country for all those criminals.

>What if all smartphones formed a mesh network? There aren't enough prisons in my country for all those criminals.

There don't need to be. You publicly gruesomely execute the first 100 or so you catch, and the practice of running a mesh node on your cell phone will fall so far out of fashion that the network breaks.

Societal shortcomings cannot be fixed via tech alone. If you can't build a society resilient to authoritarianism in the first place, tech will not help you. It can be used to increase resilience, but that's far from fixing the problem by itself.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#344

Post by [deleted] »

randomstring wrote: post #299

Search traffic has always been mostly automated spam bots. Even back in the Open Directory Days when we powered part of search.netscape.com I estimated 80+% of all search traffic was automated. At least most of it self-identified with the same Java useragent. Later when working Topix, despite being a news search engine, most traffic was bot traffic. Most included the word “mortgage” in the query. Topix specialized in…

[deleted]

Re: Almost all searches on my independent search engine are now from SEO spam bots

#345

Post by Nextgrid »

jrockway wrote: post #334
Nextgrid wrote: post #191

Earlier quoted context omitted.

One visit from the US police. US-unfriendly countries have no leverage over it, and similarly, the US has no leverage over satellite ISPs based in countries they aren't on good terms with.

> US-unfriendly countries have no leverage over it "Star Wars Episode 10: The one that's not fiction."

Internet censorship isn't worth going to war over and disclosing secret anti-satellite weapons that are better saved for a rainy day.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#346

Post by kelnage »

FargaColora wrote: post #327
ColinHayhurst wrote: post #283

Earlier quoted context omitted.

Result #1 & #2 for kitchen appliance review (your personalised/local results might vary): Google: https://www.expertreviews.co.uk/home-garden/home-appliances https://www.goodhousekeeping.com/appliances/ Bing: https://www.which.co.uk/reviews/fitted-kitchens/article/plan... https://www.goodhousekeeping.com/appliances/ DDG: https://www.goodhousekeeping.com/appliances/ https://www.which.co.uk/reviews/fitted-kitchens/arti…

Most of these are spam. They contain affiliate links to Amazon to buy the product which is being reviewed, therefore the the review cannot be trusted. "Which" looks to be the exception, but that is a paid-for service. It's a sad state of affairs.

I understand your opinion about affiliate links - but I use several review websites that use such links for all products they review, and have both positive and negative reviews for products. So I wouldn’t say it necessarily follows that affiliate links = biased reviews.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#347

Post by stevenicr »

randomstring wrote: post #299

Search traffic has always been mostly automated spam bots. Even back in the Open Directory Days when we powered part of search.netscape.com I estimated 80+% of all search traffic was automated. At least most of it self-identified with the same Java useragent. Later when working Topix, despite being a news search engine, most traffic was bot traffic. Most included the word “mortgage” in the query. Topix specialized in…

appreciate the sharing of info here.

I have recently been discovering and combating some similar, albeit much smaller issues.

I've been finding that a bunch of my recent 'resource sucks' have been constant spidering from petal-bot, semrush bot, alibiba-bot and a few others.

Using the wordpress plugin stop-bad-bots and it's logs has been eye-opening for me recently.

I understand many of these are not directly dark-seo related, but their aggressive nature is hurting the cpu and memory limits of some of my servers and sites so it's a big issue regardless of the intents behind them.

(kind of) glad someone else has dealt with these issues, and glad to see some of the 'how' for handling, identifying, and some actual real numbers for the impacts, as I've been guessing some of these things in my small projects, indeed it's a real thing. As well as a practical issue to pay attention to and work on.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#348

Post by jrockway »

Nextgrid wrote: post #345
jrockway wrote: post #334

Earlier quoted context omitted.

> US-unfriendly countries have no leverage over it "Star Wars Episode 10: The one that's not fiction."

Internet censorship isn't worth going to war over and disclosing secret anti-satellite weapons that are better saved for a rainy day.

It's probably easier to just cut off outgoing payments to Starlink anyway. They're not a charity, so if they don't get paid, they probably don't want to provide service just to send a message to some random government.

On the other hand, if you want to demonstrate that you have anti-satellite capability it's probably a better idea to shoot down a corporate satellite than a military one. The Soviet Union shot down Korean Air Lines Flight 007 and it didn't start a war, after all.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#349

Post by stevenicr »

m-i-l wrote: post #188
native_samples wrote: post #119

Earlier quoted context omitted.

Well, it's worse than that. The whole schtick is that it's only pure, real content by folksy people like us. The top reason to use it on the about page is: Indexes only user-submitted sites with a moderation layer on top, for a community-based approach to content curation, rather than indexing the entire internet with all of its spam, "search engine optimisation" and "click-bait" content. So I tried searching [kotlin…

Thanks for your feedback. It is just the home page which is moderated before indexing (and reviewed annually). When https://personalsit.es/ was listed it looked legitimate, but agreed the results for that site look infected with spam now. I've found at least one other site today where the home page and blog look genuinely legitimate, but which has a complete spam subdomain, quite possibly the victim of a subdomain ta…

As someone that has a few sites that can get user generated content - I must say that it saddens me that spam stuffing would get the main domain and site delisted - and likely never re-listed.

A couple times a year I get hit with a bunch of spam blogs / user profiles and when I discover and clean them up, I assume that at least google/bing see that the spam-to-real ratio has been fixed and rank it higher again.. but I'm not sure really, especially since google took keywords out of click traffic.

What would be nice is something like the 'site has been hacked page' that I've unfortunately seen a few times for sites - that lets you clean it up and submit a re-check it's clean now button thing.

I've also suggested that google make it so you have to vouch for links which would expose people using the spam stuffing techniques.. kind of the opposite of the disavow tool - but they never read any of my disavow submissions.

Sucks to get spammed, fight spam, and then be penalized for it more ways than one.

One of my older buddypress/wpmu sites I recently turned off blog creation for users because it's just so tiring fighting the spammers - which are only doing what they do because google - meh.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#350

Post by marginalia_nu »

MicahKV wrote: post #252

So spammers have latched onto your search engine because they are getting useful results. They are able to systematically discover websites built on certain platforms that allow users to post content containing links, which they can target for link spam. It is very difficult to fight this on a technical level because there is an entire industry built around blackhat SEO, with all kinds of softwares and services dedic…

> So spammers have latched onto your search engine because they are getting useful results.

I'm not sure about this. At least with my search engine, it doesn't really seem to matter what response they get, I don't even think they look at the responses. They keep hammering away with tens of thousands of queries per day with the requests even though they've seen nothing but HTTP Status 403 since last October or so.

My best guess is they're going after search engines in general in case they forward queries to google, in order to manipulate their typeahead suggestions.

Post reply on HN