Good news for curl users: https://github.com/mandatoryprogrammer/thermoptic
People like you are why independent sites can’t afford to run on the internet anymore.
They can't? I've run many free independent sites for years, that's news to me.
How I block all 26M of your curl requests
61–70 of 74 posts
Re: How I block all 26M of your curl requests
#62Earlier quoted context omitted.
> so there's literally no evidence of mechanical means. Keystroke dynamics and mouse movement analysis are pretty fun ways to tackle more advanced bots: https://research.roundtable.ai/proof-of-human/ But of course, it is a game of cat and mouse and there are ways to simulate it.
I don't think that mouse movement analysis is used anywhere. But it was reportedly used 10 years ago by Google's captcha. This is a client side check than can trivially be bypassed
The majority of attackers are unsophisticated and just deploying scripts they found or tools they bought on russian forums against whatever endpoint they can find.
The simple tools still work remarkably well.
There are very very effective services for bot detection that still rely heavily on keyboard and mouse behavior.
Re: How I block all 26M of your curl requests
#63Re: How I block all 26M of your curl requests
#64Good news for curl users: https://github.com/mandatoryprogrammer/thermoptic
Oh great /s In a month or two, I can be annoyed when I see some vibe-coded AI startup's script making five million requests a day to work's website with this. They'll have been ignoring the error responses: {"All data is public and available for free download": "https://example.edu/very-large-001.zip"} — a message we also write in the first line of every HTML page source. Then I will spend more time fighting this shi…
Feel free to read the README, this was already an ability that startups could pay for using private premium proxy services before thermoptic.
Having an open source version allows regular people to do scraping and not just those rich in capital.
Much of the best data services on the internet all start with scraping, the README lists many of them.
Re: How I block all 26M of your curl requests
#65" There’s ways to get around TLS signatures but it’s much harder and requires a lot more legwork to get working " I wouldn't call it "much harder". All you need to bypass the signature is to choose random ciphers (list at https://curl.se/docs/ssl-ciphers.html ) and you mash them up in a random order separated by colons in curl's --ciphers option. If you pick 15 different ciphers in a random order, there are over a tr…
This works for the ten minute period it takes to switch from a blacklist to a whitelist
Re: How I block all 26M of your curl requests
#66btw you opensourced also your website ~$ curl https://foxmoss.com/.git/config [core] repositoryformatversion = 0 filemode = true bare = false logallrefupdates = true [remote "origin"] url = https://github.com/FoxMoss/PersonalWebsite fetch = +refs/heads/ :refs/remotes/origin/ [branch "master"] remote = origin merge = refs/heads/master
The git seems to only contain the build of the website with no source code. The author is probably using git to push the content to the hosting server as an rsync alternative, but there does not seem to be much leaked information, apart from the url of the private repository.
It exposed their committer email (I know its already public on the site, but still)
You can wget the whole .git folder and look through the commit history, so if at any point something had been pushed which should not have been its available
Re: How I block all 26M of your curl requests
#67Do you actually use this? $ md5 How\ I\ Block\ All\ 26\ Million\ Of\ Your\ Curl\ Requests.html MD5 (How I Block All 26 Million Of Your Curl Requests.html) = e114898baa410d15f0ff7f9f85cbcd9d (downloaded with Safari) $ curl https://foxmoss.com/blog/packet-filtering/ | md5sum e114898baa410d15f0ff7f9f85cbcd9d - I'm aware of curl-impersonate https://github.com/lwthiker/curl-impersonate which works around these kinds of th…
Claude was scraping my cgit at around 12 requests per second, but in bursts here or there. My VPS could easily handle this, even being a free tier e2-micro on Google Cloud/Compute Engine, but they used almost 10GB of my egress bandwidth in just a few days, and ended up pushing me over the free tier.
Granted it wasn't a whole lot of money spent, but why waste money and resources so "claude" can scrape the same cgit repo over and over again?
>(1) root@gentoo-server ~ # grep 'claude' /var/log/lighttpd/access.log | wc -l
>1099323Re: How I block all 26M of your curl requests
#68Earlier quoted context omitted.
> so there's literally no evidence of mechanical means. Keystroke dynamics and mouse movement analysis are pretty fun ways to tackle more advanced bots: https://research.roundtable.ai/proof-of-human/ But of course, it is a game of cat and mouse and there are ways to simulate it.
I don't think that mouse movement analysis is used anywhere. But it was reportedly used 10 years ago by Google's captcha. This is a client side check than can trivially be bypassed
> But it was reportedly used 10 years ago by Google's captcha.
It sucks so bad. If I solve the captchas by moving the mouse too quickly, Google asks me to try again. If I'm deliberately slow and erratic with my movements as I click on pictures, it almost always lets me through on the first click. Been manually A/B testing this for years and remains true today.
Re: How I block all 26M of your curl requests
#69Earlier quoted context omitted.
I don't think that mouse movement analysis is used anywhere. But it was reportedly used 10 years ago by Google's captcha. This is a client side check than can trivially be bypassed
> But it was reportedly used 10 years ago by Google's captcha. It sucks so bad. If I solve the captchas by moving the mouse too quickly, Google asks me to try again. If I'm deliberately slow and erratic with my movements as I click on pictures, it almost always lets me through on the first click. Been manually A/B testing this for years and remains true today.
I've been doing this for many years and not even once tried to solve a captcha at my normal speed. Will have to check myself.