381–390 of 391 posts
Earlier quoted context omitted.
...so, blogrolls?
Not familiar with blogrolls, but not quite. The idea is more to have standard search engine user experience, but with the requirement that each result is vetted by someone the user trusts, or trusts by proxy.
Not directed at you specifically but this is the actual problem.
We already had a good system for these things. Delicious, blogrolls, RSS, the folksonomy ..
Earlier quoted context omitted.
I built my search engine in part to explore whether this was actually true, and I don't think it actually is. There's still a lot of organic human-made content still out there, possibly more than ever, it's just not able to compete with the SEO industry that completely displaces it from Google and social media.
Agreed, the general internet is not dead, but the majority of internet users are on Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. From my perspective, we onboarded a lot (if not most) people to the internet after 2007 (the explosion of social media). People sticking to big sites really speaks to an inability to explore the larger internet and a lack of knowing why you would even want to.
Most (99%) people use the Internet most (99%) of the time to see or hear what other people are up to. The big sites are where all the other people are. QED.
(This comment falls into that space)
Chances are they're not going to take the time to figure out your specific site's mitigation measures and just let the bot fail crawling or move on.
So spammers have latched onto your search engine because they are getting useful results. They are able to systematically discover websites built on certain platforms that allow users to post content containing links, which they can target for link spam. It is very difficult to fight this on a technical level because there is an entire industry built around blackhat SEO, with all kinds of softwares and services dedic…
Earlier quoted context omitted.
Put a CloudFlare web application firewall at the front of the site and then use its rate limited / CAPTCHA features to throttle traffic. It is the easiest way to get rid of parasitic scraping and API abuse. Cost is $0.
Yeah, that's essentially what I've done, except I'm paying for their cheapest non-free tier to have a bit more control over it. I really wish I didn't have to route all my traffic through an untrusted a 3rd party like that, but I guess we can't have nice things on the Internet anymore.
There a rub here, in that people expect to search things without being logged in. But then if you don't log in people, anyone can come calling, including bots. This then causes you to do things like get a third party to filter the data, which then affects the users by having to reroute their traffic to someone else to get rid of some of the visits you don't want from the bots.
And round and round.
Simple authentication to the site with tokens might solve the problem. If an IP comes calling that does so with out authentication, or payment, then hang the connection.
Earlier quoted context omitted.
> Wrong. I surf with JS off because of sites that use JS to collect information about me. Plausible doesn't collect information about you, but the site's usage. Do you also object to physical stores putting up cameras? Here's their own instance, open to public. https://plausible.io/plausible.io > If it's available on the server, then sure that might be considered fair game. But using javascript (or any other client-s…
> Plausible doesn't collect information about you, but the site's usage. Do you also object to physical stores putting up cameras? The difference is that the cameras don't get attached to my physical body, doesn't have any ability to monitor my actions after I have left the presence of the physical store, and can't force me to take any physical item or action. Javascript, on the other hand, has the capability to beco…
What a coincidence, Plausible doesn't either.
This made me curious to try that search engine so I typed "electronic music box" (first thing that came to mind). As far as I can tell none or the 10+ pages of results include all those 3 words. I mean, you might not have any relevant sites in your database (likely if there are only 1000 sites or so as another of your blog posts imply), and I understand you want to show some result to the user, but if I want irreleva…
What the heck is an 'electronic music box'? I personally wouldn't expect those three words to show up on any sites served by a small search engine.
Earlier quoted context omitted.
I can only agree with my sister comment. I find this industrialized web more and more shallow and taxing to use. While professionally I need to help (smaller, local) clients to reach their audiences I become more and more weary. It is like walking through a supermarket with industrialized fast convenience food shouting in bright colors and advertising while ultimately not nourishing me like slow, real food could. I a…
> I am still looking for this digital slow food movement. https://digitalvegan.net Please read it, and if you enjoy it please suggest it to friends.