IMHO what you should try is excluding all sites with excessive third-party cookies, sluggish performance, and too many ads. That will slice the index down by 80% probably but it would be a really nice thing to see. It might push out low quality SEO results for a couple of years.
Almost all searches on my independent search engine are now from SEO spam bots
241–250 of 391 posts
Re: Almost all searches on my independent search engine are now from SEO spam bots
#242You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…
I no longer see Google as a neutral "search engine" the way it used to be. Now it's just another company that owns and promotes certain types of content, no different from reddit. For some things Google has the best content, for some things Twitter or Reddit have the best content.
Google is an advertising company. It has been for a good while.
Re: Almost all searches on my independent search engine are now from SEO spam bots
#243This guy throws multiple reasons/conspiracies out there on why the website is really struggling to gain literally any sort of traction. Web is all bots, search engines not promoting competitors and being drowned out by SEO spam, yet he's failing to see the most obvious reason... the reason nearly all websites don't gain traction... Because it's a bad website. It provides no value to the user. I put in a few search te…
Whether or not his site is meeting his goals is his business.
I find this a really interesting post, because I'm also dealing with excessive bot traffic (it's generally about half of my overall), and specifically how to salvage analytics data when there's so much noise. Seeing what other people are doing to combat it helps me, regardless of whether you might think of them as successful or not.
Re: Almost all searches on my independent search engine are now from SEO spam bots
#244Earlier quoted context omitted.
I second this. Don't get me wrong, I applaud the concept and the effort, but this implementation isn't quite there. I searched for "document management system comparison" since I am currently in the process of selecting one for our legal team at work. Some on-the-ground reports from real users would be hugely valuable. But this is the classic example of where Google utterly fails; document management is a 100 billion…
Typed this search into Kagi and got: - This results from an old site https://www.scanstore.com/Scanning_Software/Document_Managem... not sure if still relevant - A bunch of discussions from reddit and other forums (probably best lead) - One research paper https://arxiv.org/pdf/1403.3131.pdf - Listicles grouped togeter so you can skip them - The noncommercial filter gave a few more good results, but it seems like ther…
Thanks for building Kagi! Have been enjoying the experience of it this past month
Re: Almost all searches on my independent search engine are now from SEO spam bots
#245I'm not sure I understand the theory of what motivates the automated "powered by" searches; can anyone explain it (or an alternate theory) further?
Re: Almost all searches on my independent search engine are now from SEO spam bots
#246Earlier quoted context omitted.
Yeah there is. I surf with JS off because of people like you.
https://plausible.io/privacy-focused-web-analytics You surf with JS off because of sites abusing their users' data. This is not it.
> You surf with JS off because of sites abusing their users' data. This is not it.
Wrong. I surf with JS off because of sites that use JS to collect information about me.
If it's available on the server, then sure that might be considered fair game. But using javascript (or any other client-side tool) to do what you should instead do server-side is abusing users (or their data).
Putting analytics inline so it's "not ad-blocked by a url request" is absolutely disrespecting users and a perfect reason to turn off javascript.
Re: Almost all searches on my independent search engine are now from SEO spam bots
#247I am slowly convincing my coworkers that deploying the exact same binary as two different 'services' is a significant tool to have in your toolbox. Some disaster recovery work we're doing is making it a much easier sell.
I'm really just combining two very old tricks here. Traffic shaping based on class of service for two different requests, and for two different classes of users.
Segregating bot traffic improves consumer experience. Segregating admin traffic from both allows you to set an upper and lower bound on availability.
Re: Almost all searches on my independent search engine are now from SEO spam bots
#248You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…
Interesting thought. I just went though my browser history and realised that almost every time I use google search, I already know what website I want, I just don’t know the exact link/page. I’ll use google because the search on stack overflow or reddit sucks but I know I’m looking for a page on one particular site.
I had the same realization and ended up setting up a simple Cloudflare script to automatically do an “I’m Feeling Lucky” style search to return the first result: https://notes.npilk.com/custom-search
Re: Almost all searches on my independent search engine are now from SEO spam bots
#249You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…
This MUST be the reason that they threw their purchase of Postini in the garbage and my GMAIL INBOX is filled with spam, and my "social" and "promotions" tabs dont filter....
GMAIL is garbage now, I literally use it as my spam email any more. Which sucks because I have had it for a really long time.
Annecdote on Yahoo! Mail ; years ago I wrote to yahoo support asking when I created my Yahoo Mail account (i'd had it from the 90s when it was very early available...)
And support told me that they couldnt tell me when my account was created as that was *proprietary company information*
So I deleted my Yahoo account. Im about to DL all my gmail and do the same.
Re: Almost all searches on my independent search engine are now from SEO spam bots
#250Earlier quoted context omitted.
Good luck, spectrum is highly regulated in every country I can think of. If national governments don’t want you networking across borders, you’re definitely not going to be broadcasting long range radio transmissions that way. In fact, it’s currently illegal to transmit encrypted data or to relay packets via ham radio in the US.
Who knows? The whole point of decentralization is for there to be so many nodes in the network they can't possibly take them all down so that it's pointless to even try. What if all smartphones formed a mesh network? There aren't enough prisons in my country for all those criminals.
I agree with your ethos, but I don't share your optimism. If the state wants to enforce networking firewalls along national boundaries, no technological solution will save us in general. As a resourceful techie with the right know-how you may be able to sneak your packets through, just like people in Cuba receive a literal packet of data via sneakernet, but if the state doesn't want widespread meshnets circumventing their firewall, they will imprison you for emitting pirate radio signals, they will penalize any electronics manufacturer that makes non-compliant hardware, and rest assured that companies will go right along. Liberty requires more than technical solutions.
I'm saying this as someone who once wrote a decentralized P2P mesh for instant messaging[1]. I was inspired by the HK protests going on ~2014 after hearing that they were using Bluetooth chat apps. Luckily Matrix, Telegram, Signal, etc. mostly solved the problem. Still, I don't think any amount of mesh networking would turn back the tide of Hong Kong now.