Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Almost all searches on my independent search engine are now from SEO spam bots

blog.searchmysite.net

Post reply on HN
Print view View original

331–340 of 391 posts

Re: Almost all searches on my independent search engine are now from SEO spam bots

#331

Post by ajmurmann »

marginalia_nu wrote: post #111
ratww wrote: post #106

Earlier quoted context omitted.

This might be controversial, but I wish Google would exclude those websites too. Google started punishing keyword spam, then it started punishing black-hat comment spam. Even Youtube backtracked on the "videos have to be 10 minutes to rank". I wish they would do the same for carefully manicured SEO content farms too, as those sites are causing a harm worse than keyword-spammer sites did.

They're probably doing all they can. The problem is their dominance, both means they have effectively an entire industry looking for loopholes in everything they do, as well as legal considerations (arbitrarily punishing individual smaller actors might skirt on the territory of anti-competitive behavior)

I love your search engine. Should I stop recommending it to friends to keep it safe?

I jest a little bit, but your comment genuinely makes me wonder if Marginalia++ is search results - Google - Marginalia

Re: Almost all searches on my independent search engine are now from SEO spam bots

#332

Post by bornfreddy »

MicahKV wrote: post #307
jfim wrote: post #303

Earlier quoted context omitted.

It might be a better idea to return low quality results than nothing at all. The idea is that it's pretty obvious when the bot is banned when it receives no results at all. Having to look at the results manually to determine whether one is banned is a much more time consuming endeavor.

Well what I'm suggesting isn't about blocking the bots, it's about removing the incentive. So in this case, I think the more obvious it is the better. I would want them to realize as soon as possible that they are 100% wasting their time. If anything, it might be best to return a page that explicitly states "Sorry, this search engine no longer supports SEO footprint search queries." *edit for typo & wording

On the other hand, making content difficult to parse is easy to do and a very strong weapon. Make them waste dev time... It is much easier to make variants of HTML than it is to parse it. You can even automate it to some degree.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#333

Post by ajmurmann »

galangalalgol wrote: post #148
sdoering wrote: post #124

Earlier quoted context omitted.

I fear that Google also has a conflict of interest here. A lot of these non optimized sites are not interested in making money via ads. So Google wouldn't profit additionally from leading people there. And a lot of people (myself often times included) are looking for a quick answer. A good enough answer. So good enough, SEO optimized is being surfaced. The result of an optimization war on both sides combined with the…

Does anyone have an ad free search engine? You'd start with blacklists from ublock origin, pi-hole, and similar, don't bother even crawling those, then have easy reporting for new or self hosted ads. Not much money in it if any, but it would be refreshing. Might even have a mode to nix anything with a payment method on the site, or that links to a site with a payment method.

> Does anyone have an ad free search engine

kagi.com search.marginalia.nu

Re: Almost all searches on my independent search engine are now from SEO spam bots

#334

Post by jrockway »

Nextgrid wrote: post #191
matheusmoreira wrote: post #184

Earlier quoted context omitted.

Starlink is maintained by a company, it's an internet service provider. One visit from the police and they'll censor anything. The mesh network should be made out of common hardware in order to be viable. I'd suggest phones but those devices are owned before they've even left the factory.

One visit from the US police. US-unfriendly countries have no leverage over it, and similarly, the US has no leverage over satellite ISPs based in countries they aren't on good terms with.

> US-unfriendly countries have no leverage over it

"Star Wars Episode 10: The one that's not fiction."

Re: Almost all searches on my independent search engine are now from SEO spam bots

#335

Post by marginalia_nu »

FargaColora wrote: post #328
marginalia_nu wrote: post #224

Earlier quoted context omitted.

¯\_(ツ)_/¯ http://www.jitterbuzz.com/indmix.html http://www.alaska.net/~akpassag/

These websites seem to be last updated decades ago, which is prehistoric to most casual browsers. There's no doubt there is great content on the general internet, but these examples I would classify as "legacy".

I can see why the website owners would be interested in getting traffic to recent websites, but why would you be interested in recently updated websites?

Re: Almost all searches on my independent search engine are now from SEO spam bots

#336

Post by tbm57 »

TremendousJudge wrote: post #319
alphabet9000 wrote: post #315

i recently built a habitat for spam bots, they eventually found it and now post peacefully https://upstairs.treehouse.telnet.asia/pharm/cylohexapine

It's beautiful to see nature healing

Maybe someone should start an internet rewilding project

Re: Almost all searches on my independent search engine are now from SEO spam bots

#337

Post by noAnswer »

MicahKV wrote: post #276
z3t4 wrote: post #271

Earlier quoted context omitted.

Or put those query results behind an anti-bot/"capcha" test.

That would probably help, but it's also a continuation of the cat and mouse game. There are plenty of captcha breaking services out there, it only cost about $1 to programmatically solve 1000 captchas.

> There are plenty of captcha breaking services out there

Give it a try and see what happens.

People said greylisting against email spam wouldn't work, since spammers would just resend. It works since 20 years. To get your IP off the DNSBL NiX Spam you just have to follow a link. People said spammers would automate that process. Never happened in 19 years. Sometimes spammers are just lazy.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#338

Post by gopher_space »

MicahKV wrote: post #252

So spammers have latched onto your search engine because they are getting useful results. They are able to systematically discover websites built on certain platforms that allow users to post content containing links, which they can target for link spam. It is very difficult to fight this on a technical level because there is an entire industry built around blackhat SEO, with all kinds of softwares and services dedic…

> It is very difficult to fight this on a technical level

It is when your base assumption is that you won't hire outside of engineering. There are more bored teenagers with phones than people creating quality content, so I'm not sure why you wouldn't just brute force checks against bad actors.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#339

Post by randomstring »

superjan wrote: post #322
randomstring wrote: post #299

Search traffic has always been mostly automated spam bots. Even back in the Open Directory Days when we powered part of search.netscape.com I estimated 80+% of all search traffic was automated. At least most of it self-identified with the same Java useragent. Later when working Topix, despite being a news search engine, most traffic was bot traffic. Most included the word “mortgage” in the query. Topix specialized in…

Almost sounds like it is justified to add a javascript crypto miner to your pages to make the bots pay for the use of your service.

The point is that the vast majority of scrapers do not bother to run javascript.
Post reply on HN