Live data from Hacker News

Almost all searches on my independent search engine are now from SEO spam bots

blog.searchmysite.net

361–370 of 391 posts

Re: Almost all searches on my independent search engine are now from SEO spam bots

#361

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

> If you think it's bad for you, imagine what it is like for Google Search! Their entire business is indexing a medium which no longer has any relevancy.

Google was the one (among many) that killed it - so I am not gonna shed any tears.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#362

Earlier quoted context omitted.

Internet censorship isn't worth going to war over and disclosing secret anti-satellite weapons that are better saved for a rainy day.

It's probably easier to just cut off outgoing payments to Starlink anyway. They're not a charity, so if they don't get paid, they probably don't want to provide service just to send a message to some random government. On the other hand, if you want to demonstrate that you have anti-satellite capability it's probably a better idea to shoot down a corporate satellite than a military one. The Soviet Union shot down Kor…

> It's probably easier to just cut off outgoing payments to Starlink anyway.

Cryptocurrencies might be a problem in this plan, and satellite internet access itself might become a currency (since unlike cryptocurrencies, this one both has almost an intrinsic value and provides its own infrastructure that's very hard to block, where as cryptos rely on external sources of Internet access).

It also depends - drugs have consistently won the war on drugs despite being a physical product that needs a local supply chain and various anti-money-laundering and banking/finance regulations that should make it hard to fund the operation. Satellite internet access is likely to be even easier as it doesn't rely on a physical product (if we reach this stage there's going to be clandestine satellite terminals built locally, so blocking shipments of the real thing isn't going to cut it).

The only solution, apart from North Korea-levels of isolation (and even then, NK has the advantage of their population being isolated & indoctrinated since birth, something most other countries won't achieve even if they turned authoritarian overnight) would be detection followed by harsh punishment, but this has the downside of not only wasting the disclosure of detection capabilities (that are useful to the military) but also outsourcing the R&D of evading such capabilities into the open which enemies will no doubt pick up on too and use against you in a conflict.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#363
I just learned of your site through this post, it looks useful (and not just for SEO spam), thank you.

Some feedback: some initial searches did not find blogs that were of interest. When browsing I noticed the tags. I wish there was a page of tags that I could browse. If there is, I could not find it.

Also many of the site titles do not provide useful information on their topics and are untagged. If you could auto generate some tags, perhaps based on word frequency in post titles (with some stopwords no doubt), that would improve the browsing experience, at least for me.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#364

Earlier quoted context omitted.

Most of these are spam. They contain affiliate links to Amazon to buy the product which is being reviewed, therefore the the review cannot be trusted. "Which" looks to be the exception, but that is a paid-for service. It's a sad state of affairs.

I understand your opinion about affiliate links - but I use several review websites that use such links for all products they review, and have both positive and negative reviews for products. So I wouldn’t say it necessarily follows that affiliate links = biased reviews.

How often do they give their best review score or opinion to a product without an affiliate link? Not every product will have an accessible affiliate link.

Isn’t Amazon commonly used for most affiliate links or has that changed in recent years? Amazon isn’t the cheapest all the time any more. Nor is its customer support the top any more

Re: Almost all searches on my independent search engine are now from SEO spam bots

#365

Earlier quoted context omitted.

Put a CloudFlare web application firewall at the front of the site and then use its rate limited / CAPTCHA features to throttle traffic. It is the easiest way to get rid of parasitic scraping and API abuse. Cost is $0.

Yeah, that's essentially what I've done, except I'm paying for their cheapest non-free tier to have a bit more control over it. I really wish I didn't have to route all my traffic through an untrusted a 3rd party like that, but I guess we can't have nice things on the Internet anymore.

> I guess we can't have nice things on the Internet anymore

Not since it left the larval stage and became "pay for play", no.

Oh, well, those taxpayer-funded years were nice for those of us who were around.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#366

Earlier quoted context omitted.

Yeah, that's essentially what I've done, except I'm paying for their cheapest non-free tier to have a bit more control over it. I really wish I didn't have to route all my traffic through an untrusted a 3rd party like that, but I guess we can't have nice things on the Internet anymore.

> I guess we can't have nice things on the Internet anymore Not since it left the larval stage and became "pay for play", no. Oh, well, those taxpayer-funded years were nice for those of us who were around.

I think I remember wondering, after the dotcom bust, if the whole web thing would actually take off.

The reasoning I vaguely remember reading was that the internet required government subsidy to exist - at first directly, then in the form of universities, and the bust was a sign that it couldn't exist without one.

I don't remember how prevalent the view was at the time though. Obviously it turned out to be wrong.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#367
post #205

Earlier quoted context omitted.

Everyone is trying to game the Google algorithm. The net result is all this long form content and cooking recipes that are 10 pages long. There seems to be a big disconnect with a typical users attention span and the length of a post.

I thought the recipe thing was to be able to copyright them

That’s just gaming a different algorithm.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#368

Since everyone in this thread wants to jump down OP's throat about the quality of his web site, another interesting search engine is millionshort.com, which allows you to filter out the top N web sites from the results of your search. It's a great tool for looking past sites with good SEO; all you have to do is fiddle with the value of N. For example, searching for "electronic music box" as /u/ajnin suggested, with t…

That's an outstanding concept. One problem though: wouldn't it also filter out high quality curated results?

Yes, such as Wikipedia, for example.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#369

i recently built a habitat for spam bots, they eventually found it and now post peacefully https://upstairs.treehouse.telnet.asia/pharm/cylohexapine

this is the best thing I have ever seen. Its art, engineering, biology and sociology. Do you write blog posts about it?

[deleted]

Re: Almost all searches on my independent search engine are now from SEO spam bots

#370
post #265

Earlier quoted context omitted.

I searched "best dress shoes reddit" as a test, and just got a random list of websites that had the word "shoes" on the page somewhere, including a Dinosaur Comic from 2008. So... yeah. Won't exactly be my first choice of search engine in the future.

It doesn't index sites like Reddit, so, not too surprising Reddit wasn't in the result.

Is there a reason for that? I am not super knowledgeable on search engines.
Post reply on HN