381–385 of 385 posts
Earlier quoted context omitted.
https://blog.cloudflare.com/ai-labyrinth/ A bit like this? ( iocaine is newer)
If I think about it, I find it awful. The fact that we need to put junk in our own stuff just for crawlers does not sit well with me.
Any functioning society would deal with the offenders directly and had this stopped before it became an issue for most sites.
Earlier quoted context omitted.
This is one reason why most crawlers ignore robots.txt now. The other reason is that bandwidth/bots are cheap enough now that they don't need web admins to help them optimize their crawlers
> This is one reason why most crawlers ignore robots.txt now. I don't buy that for a second. Those not obeying robots.txt were doing so either because they were malicious (they wanted everything and wouldn't be told “please don't plough through these bits”) or stupid (not knowing any better) or both. Anyone who was obeying robots.txt isn't going to start ignoring it because we've put honeypots there. Why would they t…
Actually they often do and that's one of the original purposed of robots.txt - to get search engines to stop wasting time on indexing worthless crap like endless dynamically generated pages. It's only relatively recently that most crawlers had a hostile relationship with website operators.
Earlier quoted context omitted.
My own git server has been hit severely by scrapers. They're scraping everything . Commits, comparisons between commits, api calls for files, everything. And pretty much all of them, ByteDance, OpenAI, AWS, Claude, various I couldn't recognize. I basically just had to block all of them to get reasonable performance for a server running on a mini-pc. I was going to move to codeberg at some point, but they had downtime…
Anyone actually scraping git repos would probably just do a 'git clone'. Crawling git hosts is extremely expensive, as git servers have always been inadvertent crawler traps. They generate a URL for every version of every file on every commit and every branch and tag, and if that wasn't enough, n(n+1)/2 git diffs for every file on every commit it has exited on. Even a relatively small git repo with a few hundred file…
And you also don't need to host a version of each file for each commit - those should just be HTTP redirects to a unique URL for that version of the file, e.g. to the commit that last changed it - or just don't provide it at all since most people are only going to be interested in branches anyway and others can clone the repo.
The same goes for many other expensive operations that other websites (including blogs and forums) do that cause the website to go down when a bad crawler finds it. It's almost all self-inflicted pain that doesn't even provide meaningful features to real users compared to a better designed website with a finite number of pages that you can even host statically if you want.
> The by far nastiest part is CI. GitHub has done an excellent job luring people in with free macOS runners and infinite capacity for public repos Yup and this is where I pass on anything other than GitHub.
E
E
Earlier quoted context omitted.
I find CI very valuable even on my solo projects. > what is CI (in practice today, not in theory)? It's just a set of commands that run on a remote machine and then the output of those commands is displayed in the browser and it also influences what other commands may or may not run. What exactly is the big deal here? The key is hermetically/reproducibly - you don't want to run commands on some random machine, you wa…
These days the reproducibility part is trivial with a Docker container, as much as it's a mess of a technology.