Live data from Hacker News

Almost all searches on my independent search engine are now from SEO spam bots

blog.searchmysite.net

161–170 of 391 posts

Re: Almost all searches on my independent search engine are now from SEO spam bots

#161

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

"I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there."

That is not the Dead Internet Theory. That's just something anyone can see by looking at the world.

The Dead Internet Theory is that the Internet is already an echo chamber custom fed to you by a collection of bots and other such things, and that a lot of the "people" you think you're interacting with are already, today, faked. You're basically in a constructed echo chamber designed only with the interests of the creators of that chamber in mind, using the powerful social cues of homo sapiens effectively against you.

In particular, those silos aren't where people are communicating. Those silos are where you think you're communicating.

It is obviously not entirely true. When we physically meet friends, sometimes topics wander to "Did you see what I posted on Facebook?" So far, we've not caught Facebook actively forging posts from our real-life friends that we physically know. (Though we have caught them failing to disseminate posts in what seems to be a distinctly slanted manner.)

I am also not terribly convinced that the bots have mastered long-form content like you see on HN. I think we've had some try, and while they can sort of pass, they seem to expend so much effort on merely "passing" that they don't have much left over to actually drive the conversation. HN probably still requires real humans to manipulate things.

Where I do seriously wonder about this theory is Twitter. AI has progressed to the point that short-form content like that can be effectively generated and driven in a certain direction. There's been some chatter on the far-out rumor mills about just how bot-infested Twitter may be, how many people think they have thousands of followers, even having interacted with some of them as "people", and in fact may only have dozens of flesh-and-blood humans following them, if that. Stay tuned, this one is developing.

(Note that while this could be "a big plan", it is also a possible outcome of many groups independently coming to the conclusion that a Twitter bot horde could be useful. A few hundred from X trying to nudge you one way, a few hundred from Y trying to nudge you another, another few thousand from Z trying to nudge you yet another, before you know it, the vast vast majority of everyone's "followers" is bots bots bots, and there was no grand plan to produce that result. It just so happens that Twitter's ancient decision to be dedicated to short-form content, with no particular real-world connection to the conversation participants, where everyone is isolated on their own feed (even if that is shared in some ways) made it the first place where this could happen. Things with real-world connections, things where everyone is in the same "area" like an HN conversation, and long-form content will all be three things that will be harder for AIs to manipulate. Twitter is like the agar dish for this sort of thing, by its structure.)

Re: Almost all searches on my independent search engine are now from SEO spam bots

#162

Earlier quoted context omitted.

There seems to be a tendancy towards video that undercuts the "old internet". I prefer instructions in a text or list format, but that's almost impossible to find for things like, changing the headlight bulb on my traverse. 1. turn the wheel so it is pointed hard in the direction of the bulb you are changing. 2. remove the hex screws from the shroud in the wheel well 3. pull the shroud down, it's pretty flexible plas…

I think this is actually because real, fluent literacy is still rare even in highly developed places. It may be easier for a very literate someone to dash off those instructions but most people are 1000x more comfortable making a little video. Same goes for reading vs watching the video. This is my same theory about meetings being universally preferred to asynchronous email, even when literally all the questions some…

You're probably right, it's just so frustrating.

I think I'm going to start compiling stuff like this in my git repo.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#163

Spammers badly need spam-free content so they can mix some legitimate links with the junk they spew. One great Black Hat SEO trick is to find where your competitors are getting clean links and insert your own links there so they do your spamming for you.

Why does the mixing work? Shouldn't Google and Bing know what the original content is and automatically identify the sites that are copies?

Re: Almost all searches on my independent search engine are now from SEO spam bots

#164

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

> No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there.

Really? We make our living running a small web based publication; around 40k readers a month. I know of many other sites like this. Google, and other search engines, depends on niche websites to provide quality search results. Without sites like ours, the internet would truly be dead, and search would be mostly useless. Our "traffic sources" come from a mix of Facebook, Search, Reddit, etc, in addition to our many loyal readers.

Others in our niche are producing blog spam, which looks nearly identical to people who aren't experts in the field, but we have real experts, fact checkers, etc, as part of our production process. This is a big problem: These low quality websites get similar rankings to our own, which does make it much harder for people to get quality information via search. (Hence the general shift towards trusting social recommendations, such as from Reddit.)

In short, the WWW is alive and well, it's just buried under a bunch of #$#$%.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#165
post #109

Will we ever see the return of hand curated directories of websites like the old days, categorized by topics and approved by human review?

Coincidentally, such a site was submitted yesterday: https://news.ycombinator.com/item?id=31387592

Re: Almost all searches on my independent search engine are now from SEO spam bots

#166
post #136

Earlier quoted context omitted.

How do you surface organic human content? I happen to linger around the fediverse/tildeverse sphere where I see organic content from people I personally have a direct (digital) connection to (and I started self-hosting my music after Epic bought Bandcamp), but I'm not clear on how I'd go about digging that kind of stuff up in the more general case.

It's not about surfacing organic human content, it's about only indexing organic human content. The problem is automated indexing. So long as indexing works according to defined rules, the advantage will be to those able to shape their content to those rules, and the spammers and scammers will win. An idea I've had for a few years is making a social-network based index engine. The only pages that get indexed are page…

...so, blogrolls?

Re: Almost all searches on my independent search engine are now from SEO spam bots

#167

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

I no longer see Google as a neutral "search engine" the way it used to be. Now it's just another company that owns and promotes certain types of content, no different from reddit. For some things Google has the best content, for some things Twitter or Reddit have the best content.

I find one of the best ways to find interesting content on specific subjects using Google is now to start blocking all their top returns (a lot of SEO spam). This is somewhat tedious (lots of -site:seospam.com) and Google doesn't like automated queries. However, a few rounds of this often turns up interesting content down low in the search results. Just don't take what's on offer on page one of search results, basically.

Where it's gotten really bad is on news searches as Google either now has some kind of shitlist of independent news sites that it won't allow to show op on, for example, site:youtube.com searches - or, it's filtered through a guest list. It's hard to tell which strategy they're using, but news is definitely being heavily filtered based on very dubious propaganda-smelling agendas.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#168
post #136

Earlier quoted context omitted.

It's not about surfacing organic human content, it's about only indexing organic human content. The problem is automated indexing. So long as indexing works according to defined rules, the advantage will be to those able to shape their content to those rules, and the spammers and scammers will win. An idea I've had for a few years is making a social-network based index engine. The only pages that get indexed are page…

...so, blogrolls?

Not familiar with blogrolls, but not quite. The idea is more to have standard search engine user experience, but with the requirement that each result is vetted by someone the user trusts, or trusts by proxy.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#169

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

The internet itself is probably gonna die soon anyway. Every country wants to impose its own laws on it. I think it'll eventually fragment into multiple segregated continental networks, if not national ones, all with heavy filtering at the borders. I'm happy to have experienced the free internet. Truly a jewel of humanity.

> I think it'll eventually fragment into multiple segregated continental networks, if not national ones

That's exactly the world in which the Internet grew. There were multiple segregated national and sub-national networks, and the Internet was built as a means to interconnect them. After some time, the Internet protocols ended up being used even within these networks, but that was not originally the case. And even today, there are still things like the AS (Autonomous System) concept which permeates the core of the top-level Internet routing protocols, which still reflect the Internet being a "network of networks" instead of a single unified network.

That's why I'm not too worried about the Internet fragmenting; we've seen this before. What happens next is gateways between the networks, and there are already shades of these in the VPN providers which allow one to connect as if one were located in a different network, often from a different country.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#170
post #57

Earlier quoted context omitted.

If you self-host Plausible, it's also possible to bundle the analytics package with the website, so that there's isn't an "ad-blockable" lone request for the .js file. https://github.com/plausible/plausible-tracker

Yeah there is. I surf with JS off because of people like you.

Also notice how I said "analytics package" and not "tracking" in my comment, because there is no tracking. I mean, unless you're the only visitor from a specific country, there is literally 0 identifying data in Plausible.
Post reply on HN