Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Almost all searches on my independent search engine are now from SEO spam bots

blog.searchmysite.net

Post reply on HN
Print view View original

21–30 of 391 posts

Re: Almost all searches on my independent search engine are now from SEO spam bots

#21

Post by oefrha »

rhn_mk1 wrote: post #16
oefrha wrote: post #10

> I didn’t notice at first because the web analytics only shows real users, and the unusual activity could only be seen by looking at the server logs. Sounds like everyone blocking analytics (Plausible in this case), e.g. myself just now, is lumped in with spam bots. Of course, analytics blocking can’t meaningfully swing the ~99.99% statistic.

I would argue that yes, it can. If the only people who are interested in using the website are those who block analytics - and, given the demographic of a niche search engine, it doesn't sound entirely implausible - then there's no telling how the 99.99% splits into bots and nerds.

Not every "nerd" use a blocker. I know many who don't. Some want to support the sites they visit; some want to see the web as it is for most people; some say their mental filters are so well developed that ads don't bother them; etc.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#22

Post by Xylakant »

rhn_mk1 wrote: post #16
oefrha wrote: post #10

> I didn’t notice at first because the web analytics only shows real users, and the unusual activity could only be seen by looking at the server logs. Sounds like everyone blocking analytics (Plausible in this case), e.g. myself just now, is lumped in with spam bots. Of course, analytics blocking can’t meaningfully swing the ~99.99% statistic.

I would argue that yes, it can. If the only people who are interested in using the website are those who block analytics - and, given the demographic of a niche search engine, it doesn't sound entirely implausible - then there's no telling how the 99.99% splits into bots and nerds.

You could guesstimate by checking the IP address - blocks assigned to residential users are likely humans, blocks assigned to cloud providers etc. likely bots.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#23

Post by PaulHoule »

Spammers badly need spam-free content so they can mix some legitimate links with the junk they spew.

One great Black Hat SEO trick is to find where your competitors are getting clean links and insert your own links there so they do your spamming for you.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#24

Post by tunap »

blackhaz wrote: post #7
albatrosstrophy wrote: post #6

Ona tangential note, I remember a time when Google had the option to search only for 'discussions'. The results were amazing and accurate as it scoured online forums. Almost all issue I had (was following the rooting scene closely back then) were quickly resolved. Then suddenly it got removed for reasons unknown to me. Anyone knows if it's replicatable today?

Sometimes adding "reddit" to a search query produces fantastic results.

I have had some success adding "forum", when looking for trade discussions; eg: controls & automotive. With all the walled silos on the net, this is much less useful with every passing day. On the bright side, I don't have to use -twitter & -facebook, so there's that.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#25

Post by throwaway27727 »

blackhaz wrote: post #7
albatrosstrophy wrote: post #6

Ona tangential note, I remember a time when Google had the option to search only for 'discussions'. The results were amazing and accurate as it scoured online forums. Almost all issue I had (was following the rooting scene closely back then) were quickly resolved. Then suddenly it got removed for reasons unknown to me. Anyone knows if it's replicatable today?

Sometimes adding "reddit" to a search query produces fantastic results.

This is great but it seems reddit has done something to mess with their date reporting. When looking for recent posts, I might see a result on Google that says it was posted in the last few days, but on clicking the result will actually be from years ago.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#26

Post by jacquesm »

netsharc wrote: post #15
jacquesm wrote: post #9

One day we'll have an internet for humans exclusively. On another note, with 160K requests / day from bots you could of course simply block the bots structurally assuming they are nice enough to identify themselves. Block all of AWS and Google, Russia, China, NK and a couple of other bot hot spots and the service may well become more successful for regular users because they get faster results. Bots can afford to wai…

I wonder how that could be accomplished. Maybe they'll build a brain interface to replace the "I'm not a robot" captchas/add a TPM chip to the brain. And then the spammers will start selling tools to fake the responses. Or pay Filipinos a few cents a month to have the chip implanted to their brains...

Well, we can do it with the roads, I'm pretty sure if the incentives are right we can come up with a way to do it online. As long as we have not passed the Turing test ;)

The current web seems to favor machines talking to machines and that is definitely not how it was intended.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#27

Post by FargaColora »

You mention the "Dead Internet Theory" (not heard that phrase before!).

I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there.

If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc.

The general Internet is dead: it's just legacy content and spam.

If you think it's bad for you, imagine what it is like for Google Search! Their entire business is indexing a medium which no longer has any relevancy. People complain that Google no longer delivers good results. But what can Google do? The "good content" is no longer available for them to index.

Want to become rich? Make a search engine which indexes the fresh relevant data from the big siloed websites, and ignores the general dead Internet.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#28

Post by asddubs »

rhn_mk1 wrote: post #16
oefrha wrote: post #10

> I didn’t notice at first because the web analytics only shows real users, and the unusual activity could only be seen by looking at the server logs. Sounds like everyone blocking analytics (Plausible in this case), e.g. myself just now, is lumped in with spam bots. Of course, analytics blocking can’t meaningfully swing the ~99.99% statistic.

I would argue that yes, it can. If the only people who are interested in using the website are those who block analytics - and, given the demographic of a niche search engine, it doesn't sound entirely implausible - then there's no telling how the 99.99% splits into bots and nerds.

even if there's 99 people blocking analytics for every person who doesn't, the figure is still 99%

Re: Almost all searches on my independent search engine are now from SEO spam bots

#29

Post by mywaifuismeta »

FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

I no longer see Google as a neutral "search engine" the way it used to be. Now it's just another company that owns and promotes certain types of content, no different from reddit. For some things Google has the best content, for some things Twitter or Reddit have the best content.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#30

Post by scambier »

oefrha wrote: post #10

> I didn’t notice at first because the web analytics only shows real users, and the unusual activity could only be seen by looking at the server logs. Sounds like everyone blocking analytics (Plausible in this case), e.g. myself just now, is lumped in with spam bots. Of course, analytics blocking can’t meaningfully swing the ~99.99% statistic.

If you self-host Plausible, it's also possible to bundle the analytics package with the website, so that there's isn't an "ad-blockable" lone request for the .js file.

https://github.com/plausible/plausible-tracker

Post reply on HN