21–30 of 216 posts
For reference, I picked Frankenstein, Alice in wonderland and Moby dick as sources and I think they might be larger than necessary as they take some time to load. But they still work fine.
There also seems to be a bug in babble.c in the thread handling? I did "fix" it as gcc suggested by changing pthread_detach(&thread) to pthread_detach(thread).. I probably broke something but it compiles and runs now :)
I have yet to see any bots figure out how to get past the Basic Auth protecting all links on my (zero traffic) website. Of course, any user following a link will be stopped by the same login dialog (I display the credentials on the home page). The solution is to make the secrets public. ALL websites could implement the same User/Pass credentials: User: nobots Pass: nobots Can bot writers overcome this if they know th…
> Can bot writers overcome this if they know the credentials? Yes, instead of doing just a HTTP request, do a HTTP request with authentication, trivial really. Probably the reason they "can't" do that now is because they haven't came across "public content behind Basic Auth with known correct credentials", so the behavior hasn't been added. But it's literally loading http://username:password@example.com instead of ht…
I want to redirect all LLM-crawlers to that site.
My initial reaction was that running something like this is still a loss, because it probably costs you as much or more than it costs them in terms of both network bytes and CPU. But then I realised two things: 1. If they are using residential IPs, each byte of network bandwidth is probably costing them a lot more than it's costing you. Win. 2. More importantly, if this became a thing that a large fraction of all web…
Not to mention they have to store the data after they download it. In theory storing garbage data is costly to them. However I have a nagging feeling that the attitude of these scrapers is they get paid the same amount per gigabyte whether it's nonsense or not.
Eh? That's the speed of an old-school spinning hard disk.
Earlier quoted context omitted.
Sure, it’s a crime for the bots, but it would also be a crime for the ordinary users that you want to access the website. Or if you make it clear that they’re allowed, I’m not sure you can stop the bots then.
I don't think it'd be illegal for anyone. The (theoretical) scenario is: There is a website (example.com) that publishes the correct credentials, and tells users to go to example.com/authenticate and put those there. At no point is a user (or bot) bypassing anything that was meant to stop them, they're following what the website is telling them publicly.
The user's approach would work only if bots can accurately even be classified, but this is impossible. The end result is that the action is user's site is now nothing but markov garbage. Not only will bots desert it but humans will too.
I had to follow a link to see an example: "A glass is not impossible to make the file and so deepen the original cut. Now heat a small spot on the glass, and a candle flame to a clear singing note. — context_length = 2. The source material is a book on glassblowing."
The user's approach would work only if bots can accurately even be classified, but this is impossible. The end result is that the action is user's site is now nothing but markov garbage. Not only will bots desert it but humans will too.