Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Almost all searches on my independent search engine are now from SEO spam bots

blog.searchmysite.net

Post reply on HN
Print view View original

131–140 of 391 posts

Re: Almost all searches on my independent search engine are now from SEO spam bots

#131

Post by shortformblog »

Gravityloss wrote: post #95
shortformblog wrote: post #92

Earlier quoted context omitted.

I think this is a tad reductive, but I will say that we sure let a lot of big companies convince a huge portion of the population to create all of their content on platforms that they have no real control over. The problem is, many of them didn’t realize this was a problem until recently. That said, plenty of exciting stuff is happening outside of the walled garden, as long as you know how to find it.

And not only did this happen already over a decade ago, a lot of the current internet users have never known anything else. We had a discussion with coworkers and somebody mentioned irc. Explaining to younger colleagues what it was and that it was not a product of a company, but operators had servers that formed a network, and it was more like infrastructure. Felt weird.

I think honestly it highlights the power of marketing as much as anything else. In some ways, building an open network is always going to put you at a disadvantage to a company that can throw money at user acquisition and PR teams. That federated networks like Mastodon have seen growth reflects the fact that word of mouth still means something in 2022.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#132

Post by tomxor »

pixl97 wrote: post #81
indigochill wrote: post #72

Earlier quoted context omitted.

How do you surface organic human content? I happen to linger around the fediverse/tildeverse sphere where I see organic content from people I personally have a direct (digital) connection to (and I started self-hosting my music after Epic bought Bandcamp), but I'm not clear on how I'd go about digging that kind of stuff up in the more general case.

Welcome to the billion dollar question. Any place that is authentic will face the zombie horde attempting to fake authenticity in order to capture attention.

I think your almost right, but it's not necessarily authenticity... I think it's just money.

Large "authentic" search engines can exist to serve the rest of the web, those personal blogs and other small communities. Those sites have a natural tendency to not be trying to turn everything into a revenue stream, so if that was the prerequisite for an engine, it would be a perfect match and naturally dissuade marketing types.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#133

Post by ratww »

sdoering wrote: post #124
marginalia_nu wrote: post #111

Earlier quoted context omitted.

They're probably doing all they can. The problem is their dominance, both means they have effectively an entire industry looking for loopholes in everything they do, as well as legal considerations (arbitrarily punishing individual smaller actors might skirt on the territory of anti-competitive behavior)

I fear that Google also has a conflict of interest here. A lot of these non optimized sites are not interested in making money via ads. So Google wouldn't profit additionally from leading people there. And a lot of people (myself often times included) are looking for a quick answer. A good enough answer. So good enough, SEO optimized is being surfaced. The result of an optimization war on both sides combined with the…

I think there's two kinds of SEO spam going on.

The black-hat kind is definitely made to extract money from ads. But those are easy to avoid for web veterans IMO. And I also feel that Google is doing its part, even though it's costing them money from those sweet ads!

But the white-hat kind, also known as content marketing, is made to let legit companies save money. Instead of paying for Google Advertisement, they get traffic by means of organic content. Think "Michelin Guide" or "Red Bull". Which is a jolly fine idea and responsible for a lot of good stuff, but the problem is that this has been taken to extremes, and now the web is littered with low-effort content made by freelancer writers getting peanuts.

I would personally prefer if those freelancer writers were doing 10 interesting Red Bull articles per month rather than 500 rehashes of contents from other websites. But who am I to judge.

In the news industry things are also very similar.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#135

Post by mcv »

If you're trying to boost your user numbers, I'm in. Results on topics I search for are very sparse, but it's all content I hadn't seen before, which is great.

Sounds like your search engine is not suitable as a replacement for more traditional search engines, but it might complement them very well. I'll give it a try.

As for the SEO bots: can't you simply block those?

Re: Almost all searches on my independent search engine are now from SEO spam bots

#136

Post by ysavir »

indigochill wrote: post #72
marginalia_nu wrote: post #54

Earlier quoted context omitted.

I built my search engine in part to explore whether this was actually true, and I don't think it actually is. There's still a lot of organic human-made content still out there, possibly more than ever, it's just not able to compete with the SEO industry that completely displaces it from Google and social media.

How do you surface organic human content? I happen to linger around the fediverse/tildeverse sphere where I see organic content from people I personally have a direct (digital) connection to (and I started self-hosting my music after Epic bought Bandcamp), but I'm not clear on how I'd go about digging that kind of stuff up in the more general case.

It's not about surfacing organic human content, it's about only indexing organic human content. The problem is automated indexing. So long as indexing works according to defined rules, the advantage will be to those able to shape their content to those rules, and the spammers and scammers will win.

An idea I've had for a few years is making a social-network based index engine. The only pages that get indexed are pages that users themselves mark as worth indexing, and the only pages returned in your results are pages that were marked for indexing by people you added to your circles, or the people in their circles, or the people in those circles, etc (probably up to 5 or 6 degrees of separation).

Re: Almost all searches on my independent search engine are now from SEO spam bots

#137

Post by ColinHayhurst »

marginalia_nu wrote: post #54
FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

I built my search engine in part to explore whether this was actually true, and I don't think it actually is. There's still a lot of organic human-made content still out there, possibly more than ever, it's just not able to compete with the SEO industry that completely displaces it from Google and social media.

Agreed.

> If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just legacy content and spam.

The "general" Internet is not dead. Though if you just want to participate in just Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord you might well think that.

Users of marginalia (author above), Mojeek (disclosure: CEO) and others [0] are well aware that there are riches of organic human-made content; from years back and new. Yes, a lot of noise too, which Google has a bigger (SEO) struggle to compete against. But still there is good and different content available.

To find good content, using search, you need to use "search" engines which enable discovery, as Google used to do so. I stress the "search" as the emphasis of Google, Bing and thus their syndicates is increasingly on being "answer" engines.

[0] https://seirdy.one/2021/03/10/search-engines-with-own-indexe...

Re: Almost all searches on my independent search engine are now from SEO spam bots

#138

Post by fortran77 »

Teandw wrote: post #62

This guy throws multiple reasons/conspiracies out there on why the website is really struggling to gain literally any sort of traction. Web is all bots, search engines not promoting competitors and being drowned out by SEO spam, yet he's failing to see the most obvious reason... the reason nearly all websites don't gain traction... Because it's a bad website. It provides no value to the user. I put in a few search te…

I found a few pro-terrorism sites here. I don't think it's the OPs purpose, but he's being duped by the few users that do look for sites like this where they can add a "curated link" to their ISIS or Hezbollah or Hamas site with a slick facade.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#139

Post by folkrav »

pluc wrote: post #57
scambier wrote: post #30

Earlier quoted context omitted.

If you self-host Plausible, it's also possible to bundle the analytics package with the website, so that there's isn't an "ad-blockable" lone request for the .js file. https://github.com/plausible/plausible-tracker

Yeah there is. I surf with JS off because of people like you.

https://plausible.io/privacy-focused-web-analytics

You surf with JS off because of sites abusing their users' data. This is not it.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#140

Post by mcv »

DethNinja wrote: post #130

Only solution is a webring based federated search engine. 1. You just put /webring.txt to your website. It shows links to other websites with a hard limit of 100 websites. 2. To combat spam and bots, search engine does accept blocklist as an input. So other people can curate the content. 3. People can personally rank the websites they like, so webring of the said website gets ranked higher for that specific user. Thi…

Nice idea, but of course if it gets even slightly popular, every SEO content farm will immediately generate 10,000 sites that all list each other in their webring.txt.
Post reply on HN