Live data from Hacker News

Almost all searches on my independent search engine are now from SEO spam bots

blog.searchmysite.net

151–160 of 391 posts

Re: Almost all searches on my independent search engine are now from SEO spam bots

#151
Makes me wonder whether Google tolerates bots on its search engine, to boost its ads revenue.

See also Twitter's extraordinary claim that 5% or less of its users are bots (or a claim from Twitter's detractors that up to 90% of its DAU are bots)

Re: Almost all searches on my independent search engine are now from SEO spam bots

#152
post #109

Will we ever see the return of hand curated directories of websites like the old days, categorized by topics and approved by human review?

Wikipedia, maybe?

The greater problem of curation is that it doesn't scale, and you need immense human effort to survey and curate both the breadth of questions -- what's a good table saw? what aspects of Egyptian culture were exported back to Greece? is HDPE plastic safe? give me some punk music. -- and also the breadth of answers, both every website and every type of table saw.

The lesser is that you cannot curate without introducing a voice, a set of preferences that may not be universal. Tastes are not universal, you can't recommend the same band for everyone. Resources are not universal, regardless of whether the $10000 table saw is more than 100x better than the $100 table saw, it's just out of reach of most people. And needs aren't universal -- a professional cabinet maker and a DIYer making a chicken coop don't need the same saw.

There's a set of priors behind every query, and you either need to get users to frame their queries in a way that captures all of the relevant priors, or you need to create a variety of voices that capture different sets of priors and curate answers appropriate to that voice. Are you asking Norm Abrams, Monica Mangin, or Shane Wighton for a recommendation on a table saw?

Re: Almost all searches on my independent search engine are now from SEO spam bots

#154

Earlier quoted context omitted.

Like Starlink?

Starlink connects to standard internet gateways on the ground. It cannot function without the 'regular internet', unless a replacement appears.

IIRC there was mention of it providing some p2p network style communication capabilities for Ukraine's military, and one of the reasons it's appealing to the US's military is the ability to route communications entirely within the network (well, with the gen 2 satellites which have laser interconnects).

So it can (at least eventually) function without 'regular internet', although I would still be hesitant to call it a viable infrastructure choice if the goal is to get around government control, simply from how much SpaceX have to appease the government to do anything space related.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#155

Mojeek member here. We have always had a high level of spam bots; as any search engine/service will have. It's a constant battle to fend off new bots; folks can always use try out our API rather than freeloading, and some do. Many obviously do not. We are taking a look at whether things have also changed for us since mid-April 2022.

Some evidence of an uptick here too. Historically it has been ~80%. 6 days ago we had to block 92%. Yesterday we blocked around three times that number of bot searches.

edit: the three times spike yesterday was one particular new attacker; general recent rise holds.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#156

Makes me wonder whether Google tolerates bots on its search engine, to boost its ads revenue. See also Twitter's extraordinary claim that 5% or less of its users are bots (or a claim from Twitter's detractors that up to 90% of its DAU are bots)

I don't think it does, I get a ~~middle finger~~ recaptcha every time I try google something

Re: Almost all searches on my independent search engine are now from SEO spam bots

#157

Earlier quoted context omitted.

Searching for Astral Codex Ten, a popular, well-written, non-spammy blog which I would expect is indexed... Returns only results in which _other_ bloggers are referencing ACX. Consider me as one of the datapoints that arrived from HN and likely won't be back, I'm afraid.

I just search Kagi, Google, and DDG for "Astral Codex Ten" and it was the first result on each.

Ironically the Kagi search engine is not in the first few results in Google when you search Kagi (at least in Thailand)

And when I did make it to the site, it looks like I have to sign up to use it? I'm not sure putting a locked gate in front of a search engine in 2022 makes sense but okay

Re: Almost all searches on my independent search engine are now from SEO spam bots

#158

Earlier quoted context omitted.

And not only did this happen already over a decade ago, a lot of the current internet users have never known anything else. We had a discussion with coworkers and somebody mentioned irc. Explaining to younger colleagues what it was and that it was not a product of a company, but operators had servers that formed a network, and it was more like infrastructure. Felt weird.

Most of the kids in my 3rd graders peer group understand federated infrastructures quite well because of Minecraft. Perhaps it wasn’t the federated nature of irc that was surprising but the fact that it was irc?

Isn't minecraft more decentralised than federated?

IRC networks usually have multiple servers connected together (historically, often run by a bunch of different people) and I didn't think people self-hosting minecraft servers usually did that?

Re: Almost all searches on my independent search engine are now from SEO spam bots

#159

Earlier quoted context omitted.

old.reddit will be gone soon, it is inevitable. Especially once they go public.

Yup. It's bound to happen. And when it does, Reddit will no longer exist in my eyes.

Agreed. IDK how I feel about Reddit. I've been on it since 2010 when Fark lost its spark. I remember some great times but a lot of it was "junk" content that in the end was very meaningless. I wish I could say I used it to develop my career in tech but that isn't true either; I use specific blogs, books, and tutorial sites to learn instead.

I suppose I mostly view it as a continuous party, yeah it's fun if you attend but after a few hours I wish I was doing something more productive.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#160
post #9

One day we'll have an internet for humans exclusively. On another note, with 160K requests / day from bots you could of course simply block the bots structurally assuming they are nice enough to identify themselves. Block all of AWS and Google, Russia, China, NK and a couple of other bot hot spots and the service may well become more successful for regular users because they get faster results. Bots can afford to wai…

> Block all of AWS and Google,

Google for "residential proxy". This is already a huge industry, and it's difficult not to see how we haven't lost this war a long time ago.

Post reply on HN