Good news for curl users: https://github.com/mandatoryprogrammer/thermoptic
How I block all 26M of your curl requests
11–20 of 74 posts
Re: How I block all 26M of your curl requests
#12I got exactly this far: uint8_t *data = (void *)(long)ctx->data; before I stopped reading. I had to go look up the struct xdp_md [1], it is declared like this: struct xdp_md { __u32 data; __u32 data_end; __u32 data_meta; /* ... further fields elided ... */ }; So clearly the `data` member is already an integer. The sane way to cast it would be to cast to the actual desired destination type, rather than first to some o…
Your code emits a compiler warning about casting an integer to a pointer. Changing the cast to void* emits a slightly different warning about the size of integer being cast to a pointer being smaller than the pointer type. Casting to a long and then a void* avoids both of these warnings.
Sorry, all that stuff might be true but this whole process is nuts.
The code segment containing that code looks like a no-op.
The rest of the post seems sane and well informed, so my theory is that this is a C / packet filtering idiom I’m not aware of, working far from that field.
Otherwise I’m already freaked out by treating a 32 bit field as a pointer… even if you extend it to first.
Re: How I block all 26M of your curl requests
#13Earlier quoted context omitted.
Your code emits a compiler warning about casting an integer to a pointer. Changing the cast to void* emits a slightly different warning about the size of integer being cast to a pointer being smaller than the pointer type. Casting to a long and then a void* avoids both of these warnings.
Sorry, all that stuff might be true but this whole process is nuts. The code segment containing that code looks like a no-op. The rest of the post seems sane and well informed, so my theory is that this is a C / packet filtering idiom I’m not aware of, working far from that field. Otherwise I’m already freaked out by treating a 32 bit field as a pointer… even if you extend it to first.
Yeah it's freaky. It's C code but it targets the eBPF virtual machine.
Re: How I block all 26M of your curl requests
#14Earlier quoted context omitted.
The problem you usually attempt to alleviate by using Anubis is that you get hit by load generated by aggressive AI scrappers that are otherwise indistinguishable from real users. As soon as the bot is polite enough to identify as some kind of a bot, the problem's gone, as you can apply your regular measures for rate limiting and access control now. (yes, there are also people who use it as an anti-AI statement, but…
Yeah that makes sense. Bad players will try to look like a regular browser, good players will have no problems revealing they’re a bot.
[deleted]
Re: How I block all 26M of your curl requests
#15Earlier quoted context omitted.
Your code emits a compiler warning about casting an integer to a pointer. Changing the cast to void* emits a slightly different warning about the size of integer being cast to a pointer being smaller than the pointer type. Casting to a long and then a void* avoids both of these warnings.
Sorry, all that stuff might be true but this whole process is nuts. The code segment containing that code looks like a no-op. The rest of the post seems sane and well informed, so my theory is that this is a C / packet filtering idiom I’m not aware of, working far from that field. Otherwise I’m already freaked out by treating a 32 bit field as a pointer… even if you extend it to first.
> Otherwise I’m already freaked out by treating a 32 bit field as a pointer… even if you extend it to first.
The cast from a 32 bit pointer to a 64 bit pointer is in fact an eBPF oddity. So what's happening here is that the virtual machine is just giving us a fake memory address just to use in the program and when the read actually needs to happen the kernel just rewrites the virtual addresses to the real ones. I'm assuming this is just a byproduct of the memory separation that eBPF does to prevent filters from accidentally reading kernel memory.
Also yes the double cast is just to keep the compiler from throwing a warning.
Re: How I block all 26M of your curl requests
#16Those bots would be really naive not to use curl-impersonate. I basically use it for any request I make even if I don’t expect to be blocked because why wouldn’t I.
A lot of the bots are compromised servers (eg hacked Wordpress sites), with limited control over what the TLS fingerprints look like.
Re: How I block all 26M of your curl requests
#17Good news for curl users: https://github.com/mandatoryprogrammer/thermoptic
Work like this is incredible. I did not know this existed. Thank you.
Re: How I block all 26M of your curl requests
#18I guess we'll just throw containerized headless browsers at you those like you then. It'll only cost you more.
Re: How I block all 26M of your curl requests
#19Do you actually use this?
$ md5 How\ I\ Block\ All\ 26\ Million\ Of\ Your\ Curl\ Requests.html(downloaded with Safari)
$ curl https://foxmoss.com/blog/packet-filtering/ | md5sum
e114898baa410d15f0ff7f9f85cbcd9d -
Also: If you serve some Anubis code without actually running the anubis script in the page, you'll get some answers back so there's at least one anubis-simulator running on the Internet that doesn't bother to actually run the JavaScript it's given.
Also also: 26M requests daily is only 300 requests per second and Apache could handle that easily over 15 years ago. Why worry about something as small as that?
Re: How I block all 26M of your curl requests
#20Good news for curl users: https://github.com/mandatoryprogrammer/thermoptic
> NOTE: Due to many WAFs employing JavaScript-level fingerprinting of web browsers, thermoptic also exposes hooks to utilize the browser for key steps of the scraping process. See this section for more information on this.
This reminds me of how Stripe does user tracking for fraude detection https://mtlynch.io/stripe-update/ I wonder if thermoptic could handle that.