Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Almost all searches on my independent search engine are now from SEO spam bots

blog.searchmysite.net

Post reply on HN
Print view View original

171–180 of 391 posts

Re: Almost all searches on my independent search engine are now from SEO spam bots

#171

Post by jacquesm »

pjmlp wrote: post #90
jacquesm wrote: post #9

One day we'll have an internet for humans exclusively. On another note, with 160K requests / day from bots you could of course simply block the bots structurally assuming they are nice enough to identify themselves. Block all of AWS and Google, Russia, China, NK and a couple of other bot hot spots and the service may well become more successful for regular users because they get faster results. Bots can afford to wai…

And the cycle will reboot itself again. The silos we have nowadays were there before the Internet took off, on BBS, Compuserve, Geocities, .... Apparently the majority of regular humans likes to have centralized providers they can reach out to, instead of the freadom of decentralized content.

Yes, that's true. Bots tend to follow the money.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#172

Post by mc32 »

ColinHayhurst wrote: post #137
marginalia_nu wrote: post #54

Earlier quoted context omitted.

I built my search engine in part to explore whether this was actually true, and I don't think it actually is. There's still a lot of organic human-made content still out there, possibly more than ever, it's just not able to compete with the SEO industry that completely displaces it from Google and social media.

Agreed. > If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just legacy content and spam. The "general" Internet is not dead. Though if you just want to participate in just Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord you might well think that. Users of marginalia (aut…

Sounds like we’re back to AskJeeves and a number of failed answer engines from a couple of decades ago!

Re: Almost all searches on my independent search engine are now from SEO spam bots

#173

Post by marginalia_nu »

marginalia_nu wrote: post #51

I had to put my search engine behind Cloudflare to deal with this. Like the volume grew to about 10x the traffic I saw sitting at the front page of Hacker News for a full week.

This is the rate of rejected HTTP requests I'm seeing at this point: https://www.marginalia.nu/junk/spam.png

Real search traffic is about half that.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#174

Post by rchaud »

FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

> Want to become rich? Make a search engine which indexes the fresh relevant data from the big siloed websites, and ignores the general dead Internet.

That would be a great service, but it certainly wouldn't make you rich. Where's the money going to come from? Google got rich because they acquired an ads platform (DoubleClick) and an analytics platform (Urchin) and started monetizing the vast amounts of data they had. That was years after Google had established goodwill as the best search engine.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#175

Post by jspaetzel »

FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

This is so incredibly false, I've been working on a project for the last six months and MoM I've seen steady increase in usage. Tbh much much higher usage then I expected. Most users find my site via Google or Facebook however they are looking for content that's not in those silos and have no problems leaving them.

If you have high quality content and you get it indexed properly by Google, users will come.

There are reasons users are not using your website.

1. It's not solving a problem people have.

2. Users can't find it.

Who, in their right mind searches for search engines? Nobody I know.

If you want users you have to go out and get them (literally pound the pavement and talk to people) or create a LOT more content ironically, so they can find your site on the search engines they are using today.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#176

Post by netr0ute »

scambier wrote: post #170
pluc wrote: post #57

Earlier quoted context omitted.

Yeah there is. I surf with JS off because of people like you.

Also notice how I said "analytics package" and not "tracking" in my comment, because there is no tracking. I mean, unless you're the only visitor from a specific country, there is literally 0 identifying data in Plausible.

Analytics is still unnecessary JS and a bandwidth hog, so it has to go.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#177

Post by ColinHayhurst »

mc32 wrote: post #172
ColinHayhurst wrote: post #137

Earlier quoted context omitted.

Agreed. > If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just legacy content and spam. The "general" Internet is not dead. Though if you just want to participate in just Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord you might well think that. Users of marginalia (aut…

Sounds like we’re back to AskJeeves and a number of failed answer engines from a couple of decades ago!

AskBERT but now MUM knows best.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#178

Post by netr0ute »

varun_ch wrote: post #87
pluc wrote: post #57

Earlier quoted context omitted.

Yeah there is. I surf with JS off because of people like you.

Most of the data you can collect with Plausible could just be collected server side instead, it's nothing like Google Analytics.

> Most of the data you can collect with Plausible could just be collected server side instead

Then why not just use that instead?

Re: Almost all searches on my independent search engine are now from SEO spam bots

#179

Post by closedloop129 »

>I noted that there had been multiple weeks where not one single real person had visited a single blog entry for the whole week

The site is not on https://searchengine.party/ nor on seirdy.one's overview. Apart from the blog, how could users find that engine?

Is there some place where new search engines are announced and where new search engines band together to make themselves heard?

Re: Almost all searches on my independent search engine are now from SEO spam bots

#180

Post by rchaud »

boplicity wrote: post #164
FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

> No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. Really? We make our living running a small web based publication; around 40k readers a month. I know of many other sites like this. Google, and other search engines, depends on niche websites to provide quality search results. Without sites like ours, the internet would truly be dead, and search would be…

> Our "traffic sources" come from a mix of Facebook, Search, Reddit, etc, in addition to our many loyal readers.

40k/mo is a pretty good number for an independent website. As a word of warning though, relying on social media reach is a dangerous game, as there is anecdotal evidence that tweets with outbound links don't get as many impressions as those that link to in-site content, like another Twitter post.

As for Facebook, well, there's a good comic from The Oatmeal (enormously popular on FB back in 2010) that talks about what happened in the long run:

https://twitter.com/Oatmeal/status/923250055540219904

Post reply on HN