See also Twitter's extraordinary claim that 5% or less of its users are bots (or a claim from Twitter's detractors that up to 90% of its DAU are bots)
Almost all searches on my independent search engine are now from SEO spam bots
151–160 of 391 posts
Re: Almost all searches on my independent search engine are now from SEO spam bots
#152Will we ever see the return of hand curated directories of websites like the old days, categorized by topics and approved by human review?
The greater problem of curation is that it doesn't scale, and you need immense human effort to survey and curate both the breadth of questions -- what's a good table saw? what aspects of Egyptian culture were exported back to Greece? is HDPE plastic safe? give me some punk music. -- and also the breadth of answers, both every website and every type of table saw.
The lesser is that you cannot curate without introducing a voice, a set of preferences that may not be universal. Tastes are not universal, you can't recommend the same band for everyone. Resources are not universal, regardless of whether the $10000 table saw is more than 100x better than the $100 table saw, it's just out of reach of most people. And needs aren't universal -- a professional cabinet maker and a DIYer making a chicken coop don't need the same saw.
There's a set of priors behind every query, and you either need to get users to frame their queries in a way that captures all of the relevant priors, or you need to create a variety of voices that capture different sets of priors and curate answers appropriate to that voice. Are you asking Norm Abrams, Monica Mangin, or Shane Wighton for a recommendation on a table saw?
Re: Almost all searches on my independent search engine are now from SEO spam bots
#153Re: Almost all searches on my independent search engine are now from SEO spam bots
#154Earlier quoted context omitted.
Like Starlink?
Starlink connects to standard internet gateways on the ground. It cannot function without the 'regular internet', unless a replacement appears.
So it can (at least eventually) function without 'regular internet', although I would still be hesitant to call it a viable infrastructure choice if the goal is to get around government control, simply from how much SpaceX have to appease the government to do anything space related.
Re: Almost all searches on my independent search engine are now from SEO spam bots
#155Mojeek member here. We have always had a high level of spam bots; as any search engine/service will have. It's a constant battle to fend off new bots; folks can always use try out our API rather than freeloading, and some do. Many obviously do not. We are taking a look at whether things have also changed for us since mid-April 2022.
edit: the three times spike yesterday was one particular new attacker; general recent rise holds.
Re: Almost all searches on my independent search engine are now from SEO spam bots
#156Makes me wonder whether Google tolerates bots on its search engine, to boost its ads revenue. See also Twitter's extraordinary claim that 5% or less of its users are bots (or a claim from Twitter's detractors that up to 90% of its DAU are bots)
Re: Almost all searches on my independent search engine are now from SEO spam bots
#157Earlier quoted context omitted.
Searching for Astral Codex Ten, a popular, well-written, non-spammy blog which I would expect is indexed... Returns only results in which _other_ bloggers are referencing ACX. Consider me as one of the datapoints that arrived from HN and likely won't be back, I'm afraid.
I just search Kagi, Google, and DDG for "Astral Codex Ten" and it was the first result on each.
And when I did make it to the site, it looks like I have to sign up to use it? I'm not sure putting a locked gate in front of a search engine in 2022 makes sense but okay
Re: Almost all searches on my independent search engine are now from SEO spam bots
#158Earlier quoted context omitted.
And not only did this happen already over a decade ago, a lot of the current internet users have never known anything else. We had a discussion with coworkers and somebody mentioned irc. Explaining to younger colleagues what it was and that it was not a product of a company, but operators had servers that formed a network, and it was more like infrastructure. Felt weird.
Most of the kids in my 3rd graders peer group understand federated infrastructures quite well because of Minecraft. Perhaps it wasn’t the federated nature of irc that was surprising but the fact that it was irc?
IRC networks usually have multiple servers connected together (historically, often run by a bunch of different people) and I didn't think people self-hosting minecraft servers usually did that?
Re: Almost all searches on my independent search engine are now from SEO spam bots
#159Earlier quoted context omitted.
old.reddit will be gone soon, it is inevitable. Especially once they go public.
Yup. It's bound to happen. And when it does, Reddit will no longer exist in my eyes.
I suppose I mostly view it as a continuous party, yeah it's fun if you attend but after a few hours I wish I was doing something more productive.
Re: Almost all searches on my independent search engine are now from SEO spam bots
#160One day we'll have an internet for humans exclusively. On another note, with 160K requests / day from bots you could of course simply block the bots structurally assuming they are nice enough to identify themselves. Block all of AWS and Google, Russia, China, NK and a couple of other bot hot spots and the service may well become more successful for regular users because they get faster results. Bots can afford to wai…
Google for "residential proxy". This is already a huge industry, and it's difficult not to see how we haven't lost this war a long time ago.