Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Almost all searches on my independent search engine are now from SEO spam bots

blog.searchmysite.net

Post reply on HN
Print view View original

191–200 of 391 posts

Re: Almost all searches on my independent search engine are now from SEO spam bots

#191

Post by Nextgrid »

matheusmoreira wrote: post #184
politician wrote: post #91

Earlier quoted context omitted.

Like Starlink?

Starlink is maintained by a company, it's an internet service provider. One visit from the police and they'll censor anything. The mesh network should be made out of common hardware in order to be viable. I'd suggest phones but those devices are owned before they've even left the factory.

One visit from the US police. US-unfriendly countries have no leverage over it, and similarly, the US has no leverage over satellite ISPs based in countries they aren't on good terms with.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#192

Post by kmeisthax »

jacquesm wrote: post #9

One day we'll have an internet for humans exclusively. On another note, with 160K requests / day from bots you could of course simply block the bots structurally assuming they are nice enough to identify themselves. Block all of AWS and Google, Russia, China, NK and a couple of other bot hot spots and the service may well become more successful for regular users because they get faster results. Bots can afford to wai…

...so you're going to write your own HTTP requests? Encrypt your traffic and validate certificates by hand? Toggle in each TCP header from a memory debugger?

Most of the Internet is bots because humans don't actually generate HTTP traffic - they fire up a bot called a "browser" to do it for them. The challenge for anti-spam is to distinguish which bots are currently being directly controlled by humans and which ones are not-so-directly controlled by such. This isn't even a hard line; I've frequently hit Hacker News' bot detection just by upvoting a comment and then clicking reply too quickly.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#193

Post by hn_throwaway_99 »

FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

Doesn't this site, and all of the content it links to, pretty much disprove your theory?

Yes, sure, I often do go to the "top sites" when searching for content, but I still usually start at Google. And, despite all the SEO spam, Google still does a fairly decent of landing me on, for example, the appropriate Wikipedia page, Stackoverflow post, travel site, etc.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#194

Post by ntauthority »

FargaColora wrote: post #74
airstrike wrote: post #71

Earlier quoted context omitted.

> Reddit doesn't let you fully expand threads when you're unlogged. that's what old.reddit.com is for!

old.reddit will be gone soon, it is inevitable. Especially once they go public.

Isn't it a bit ironic that a site - or its operator - 'going public' means all the content on said site actually 'goes private'?

Re: Almost all searches on my independent search engine are now from SEO spam bots

#195

Post by lukev »

Teandw wrote: post #62

This guy throws multiple reasons/conspiracies out there on why the website is really struggling to gain literally any sort of traction. Web is all bots, search engines not promoting competitors and being drowned out by SEO spam, yet he's failing to see the most obvious reason... the reason nearly all websites don't gain traction... Because it's a bad website. It provides no value to the user. I put in a few search te…

I second this. Don't get me wrong, I applaud the concept and the effort, but this implementation isn't quite there.

I searched for "document management system comparison" since I am currently in the process of selecting one for our legal team at work. Some on-the-ground reports from real users would be hugely valuable. But this is the classic example of where Google utterly fails; document management is a 100 billion industry and there are absolutely no search results which are not SEO, marketing copy, or astroturfed listicles with nearly zero value.

Unfortunately, this website returned even less relevant results. Not a single result pertained to document management at all; instead it returned random matches on words like "system" and "management."

Whoever solves this problem could definitely unseat Google as the go-to search engine for most people. So it's a big prize. But it's also a super hard socio-technical problem, requiring incredibly sophisticated and powerful tech in a highly adversarial environment. However, regrettably, it looks like this attempt hasn't even got the basic search tech down.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#196

Post by jacquesm »

kmeisthax wrote: post #192
jacquesm wrote: post #9

One day we'll have an internet for humans exclusively. On another note, with 160K requests / day from bots you could of course simply block the bots structurally assuming they are nice enough to identify themselves. Block all of AWS and Google, Russia, China, NK and a couple of other bot hot spots and the service may well become more successful for regular users because they get faster results. Bots can afford to wai…

...so you're going to write your own HTTP requests? Encrypt your traffic and validate certificates by hand? Toggle in each TCP header from a memory debugger? Most of the Internet is bots because humans don't actually generate HTTP traffic - they fire up a bot called a "browser" to do it for them. The challenge for anti-spam is to distinguish which bots are currently being directly controlled by humans and which ones…

I really don't understand your comment.

Just so we don't have to argue about what constitutes a bot and what does not I propose we use this definition:

https://en.wikipedia.org/wiki/Internet_bot

Re: Almost all searches on my independent search engine are now from SEO spam bots

#197

Post by dylan604 »

marginalia_nu wrote: post #54
FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

I built my search engine in part to explore whether this was actually true, and I don't think it actually is. There's still a lot of organic human-made content still out there, possibly more than ever, it's just not able to compete with the SEO industry that completely displaces it from Google and social media.

And who uses your search? I had never heard of "you" until just now. And there is the problem with "new" search engines. Unless you can come up with what would have to be one of the greatest ad campaigns the world has ever seen, no significant number of users will know you exist. Where does the money to pay for that ad campaign come from? How will a search engine generate money to stay relevant? Once people see you becoming relevant, they will figure out how to game your system. It's just the nature of the beast. I don't think I'm being overly cynical about this either.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#198

Post by thesuitonym »

jerf wrote: post #161
FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

"I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there." That is not the Dead Internet Theory. That's just something anyone can see by looking at the world. The Dead Internet Theory is that the Internet is already an echo chamber custom fed to you by a collection of bots and other such things, and that a…

> (Though we have caught them failing to disseminate posts in what seems to be a distinctly slanted manner.)

I haven't seen this, but I'd be interested in reading about it, if you have a link!

Re: Almost all searches on my independent search engine are now from SEO spam bots

#199

Post by m-i-l »

Teandw wrote: post #62

This guy throws multiple reasons/conspiracies out there on why the website is really struggling to gain literally any sort of traction. Web is all bots, search engines not promoting competitors and being drowned out by SEO spam, yet he's failing to see the most obvious reason... the reason nearly all websites don't gain traction... Because it's a bad website. It provides no value to the user. I put in a few search te…

Hi, "this guy" here:-) If people come to a site but don't come back then it is reasonable to conclude that "it's a bad website", but as the blog entry put it "without any real users in the first place it is hard to gauge whether people like it or not".

Note also that it isn't intended to be a general purpose search engine, but a niche search engine to try and find some of the fun and interesting content, e.g. relating to hobbies and interests, which used to be at the core of the web but which can be difficult to find anywhere nowadays.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#200

Post by dageshi »

FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

I agree with you to an extent. The web is less useful than it used to be. BUT I would say a lot of that usefulness has diverted into youtube. There are people who would previously have made sites who are making youtube videos instead which of course is owned by google.
Post reply on HN