Live data from Hacker News

Almost all searches on my independent search engine are now from SEO spam bots

blog.searchmysite.net

41–50 of 391 posts

Re: Almost all searches on my independent search engine are now from SEO spam bots

#41
post #3

This is for comment spam. It's trying to find a long tail of popular but not top listed blogs for the purpose of posting comments with the much desired links to the SEO target.

Does that work any more? I thought everyone put nofollow attributes on comment links.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#42
post #38

This made me curious to try that search engine so I typed "electronic music box" (first thing that came to mind). As far as I can tell none or the 10+ pages of results include all those 3 words. I mean, you might not have any relevant sites in your database (likely if there are only 1000 sites or so as another of your blog posts imply), and I understand you want to show some result to the user, but if I want irreleva…

Yeah same, I searched for Leeds grand theatre and the top result is something titled "June 2012 – Sam's Blog' which just mentions the word grand.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#43

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

These discussions always make me recal Jacob Applebaum. Think of him what you want, but this statement of his really stuck with me at the time. Paraphrasing:

The real dark-net is facebook. Everything that goes in there never comes out again and is basically invisible to the world, except if you join facebook yourself.

My own prime example of that used to be pinterest: it seems to be a 100% sink in the directed graph of internet links. But since Applebaum stated this, instagram (also facebook of course) is trying hard to push pinterest off that particular throne.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#44
post #9

One day we'll have an internet for humans exclusively. On another note, with 160K requests / day from bots you could of course simply block the bots structurally assuming they are nice enough to identify themselves. Block all of AWS and Google, Russia, China, NK and a couple of other bot hot spots and the service may well become more successful for regular users because they get faster results. Bots can afford to wai…

I wonder how that could be accomplished. Maybe they'll build a brain interface to replace the "I'm not a robot" captchas/add a TPM chip to the brain. And then the spammers will start selling tools to fake the responses. Or pay Filipinos a few cents a month to have the chip implanted to their brains...

> Or pay Filipinos a few cents a month to have the chip implanted to their brains...

That's the problem with blocking bots as opposed to malicious behavior. Bot blocking is actually trivial and very cheap to bypass as long as you can buy slave-like labor for peanuts.

Ideally you'd want to block malicious behavior (when it comes to SEO spam, downrank anything for-profit such as ads, analytics, affiliate links, etc) instead to remove the incentives for spamming, regardless of whether it's a bot or human.

In this case the only problem is that this search engine gives away resources (search queries) for free and then complains that people (in this case spammers) are taking it. It's not really a spam problem - they'd complain equally well if they had some legitimate user that happened to need tons of search queries to achieve their task.

The only solution here is to start charging for stuff that costs money, and then it doesn't really matter who is on the other side, as long as they pay the bill.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#45

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

I don't believe the WWW internet is dead; there's still millions of webpages being made and published every day. However, the traffic numbers are skewed in favor of the big socials and aggregators; I wouldn't be surprised if the 80/20 rule applies there.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#46
post #7

Ona tangential note, I remember a time when Google had the option to search only for 'discussions'. The results were amazing and accurate as it scoured online forums. Almost all issue I had (was following the rooting scene closely back then) were quickly resolved. Then suddenly it got removed for reasons unknown to me. Anyone knows if it's replicatable today?

Sometimes adding "reddit" to a search query produces fantastic results.

Not sure for how much longer this is going to work. Plenty of marketers make fake posts there in grassroots campaigns. Reddit itself is an advertising company.

God I hope they never find out about this site.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#47

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

These discussions always make me recal Jacob Applebaum. Think of him what you want, but this statement of his really stuck with me at the time. Paraphrasing: The real dark-net is facebook. Everything that goes in there never comes out again and is basically invisible to the world, except if you join facebook yourself. My own prime example of that used to be pinterest: it seems to be a 100% sink in the directed graph…

No post body was provided.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#48

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

Unfortunately, correct. The average Internet user accesses it via a phone, not a desktop, laptop, or even tablet these days. Most of that access is through apps, not a browser. To the extent that a user is looking for a factoid answer and does a search, a Google Knowledge Graph result with a Wikipedia link is probably enough in most cases. If they want a technical question answered, Stack Exchange; a product review, Reddit; nearby restaurants with reviews, Google Maps; etc.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#49

Ona tangential note, I remember a time when Google had the option to search only for 'discussions'. The results were amazing and accurate as it scoured online forums. Almost all issue I had (was following the rooting scene closely back then) were quickly resolved. Then suddenly it got removed for reasons unknown to me. Anyone knows if it's replicatable today?

I have a suspicion they removed it because of the amount of spam on those forums. There's tons of abandoned forums that are only occupied by spambots.

There's even pretty convincing looking accounts and messages that turn out to be spam in the end, once they start trying to post links.

I have Akismet on the comment section of the Wordpress front-end of the site I run, it basically said something like 99.99% of attempted comments were spam. I'm sure the same applies to e-mail and the like.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#50

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

People are doing this already. You just have to include the site name in the search on google e.g reddit. Search on these platforms is often broken.
Post reply on HN