I guess we'll just throw containerized headless browsers at you those like you then. It'll only cost you more.
Sounds like it'll also cost you much more, though.
True, but if you are doing AI, money is mostly free.
How I block all 26M of your curl requests
41–50 of 74 posts
Re: How I block all 26M of your curl requests
#42Earlier quoted context omitted.
Sounds like it'll also cost you much more, though.
True, but if you are doing AI, money is mostly free.
Eh, we'll revert to the mean at some point. Taking a loss on each unit and making up for it in volume won't scale forever.
Unless headless browsers become cheap enough for that base cost to go to effectively zero too, of course, but I trust web bloat to continue pushing out that intersection point for a bit more.
Re: How I block all 26M of your curl requests
#43Do you actually use this? $ md5 How\ I\ Block\ All\ 26\ Million\ Of\ Your\ Curl\ Requests.html MD5 (How I Block All 26 Million Of Your Curl Requests.html) = e114898baa410d15f0ff7f9f85cbcd9d (downloaded with Safari) $ curl https://foxmoss.com/blog/packet-filtering/ | md5sum e114898baa410d15f0ff7f9f85cbcd9d - I'm aware of curl-impersonate https://github.com/lwthiker/curl-impersonate which works around these kinds of th…
He does use it (I verified it from curl on a recent Linux distro). But he probably blocked only some fingerprints. And the fingerprint depends on the exact OpenSSL and curl versions, as different version combinations will send different TLS ciphers and extensions.
Re: How I block all 26M of your curl requests
#44Good news for curl users: https://github.com/mandatoryprogrammer/thermoptic
Oh great /s
In a month or two, I can be annoyed when I see some vibe-coded AI startup's script making five million requests a day to work's website with this.
They'll have been ignoring the error responses:
{"All data is public and available for free download": "https://example.edu/very-large-001.zip"}
Then I will spend more time fighting this shit, and less time improving the public data system.
Re: How I block all 26M of your curl requests
#45I guess we'll just throw containerized headless browsers at you those like you then. It'll only cost you more.
Yes, it's essentially a cat and mouse game, with the ultimate conclusion that the game itself is destroyed - in the current case, the open internet.
Re: How I block all 26M of your curl requests
#46There are also HTTP fingerprints. I believe it's named after akamai or something. All of it is fairly easy to fake. JavaScript is the only thing that poses any challenge and what challenge it poses is in how you want to do it with minimal performance impact. The simple truth is that a motivated adversary can interrogate and match every single minor behavior of the browser to be bit-perfect and there is nothing anyone…
This is exactly right, and it's why I believe we need to solve this problem in the human domain, with laws and accountability. We need new copyrights that cover serving content on the web, and gives authors control over who gets to access that content, WITHOUT requiring locked down operating systems or browser monopolies.
Laws are only enforceable in their own country, and possibly some friendly countries.
If that means blocking foreign access, the problem is solved anyway.
Re: How I block all 26M of your curl requests
#47There are also HTTP fingerprints. I believe it's named after akamai or something. All of it is fairly easy to fake. JavaScript is the only thing that poses any challenge and what challenge it poses is in how you want to do it with minimal performance impact. The simple truth is that a motivated adversary can interrogate and match every single minor behavior of the browser to be bit-perfect and there is nothing anyone…
This is exactly right, and it's why I believe we need to solve this problem in the human domain, with laws and accountability. We need new copyrights that cover serving content on the web, and gives authors control over who gets to access that content, WITHOUT requiring locked down operating systems or browser monopolies.
Laws only work in domestic scenarios.
If laws appear, the entire planet, all nations must agree and ensure prosecuting on that law. I cannot imagine that happening. It hasn't with anything compute yet.
So it'll just move off shore, and people will buy the resulting data.
Also is your nick and response sarcasm?
Re: How I block all 26M of your curl requests
#48Very interesting
Re: How I block all 26M of your curl requests
#49Earlier quoted context omitted.
The problem you usually attempt to alleviate by using Anubis is that you get hit by load generated by aggressive AI scrappers that are otherwise indistinguishable from real users. As soon as the bot is polite enough to identify as some kind of a bot, the problem's gone, as you can apply your regular measures for rate limiting and access control now. (yes, there are also people who use it as an anti-AI statement, but…
> As soon as the bot is polite enough to identify as some kind of a bot, the problem's gone, as you can apply your regular measures for rate limiting and access control now. Very interesting, so we're about to come full circle? Can't wait to have to mask myself as a (paying?) AI scraper to bypass annoying captchas when accessing "bot protected" websites...
"War, war never changes"
Re: How I block all 26M of your curl requests
#50I'm curious about why the user-agent he described can bypass Anubis, since it contains "Mozilla", sounds like a bug to me.
Edit: Nevermind, I see part of the default config is allowing Googlebot, so this is literally intended. Seems like people who criticize Anubis often don't understand what the opinionated default config is supposed to accomplish (only punish bots/scrapers pretending to be real browsers).