I would add cloudflare, but set it to cache only mode *without* the bot blocking features.
No point, they are all unique requests.
Creepy Crawlies
281–290 of 694 posts
Re: Creepy Crawlies
#282> Why is git.kernel.org “interesting” to crawlers I think the post underestimates just how little thought and effort is put into these bots. I also run a cgit instance with far less interesting projects, and am not spared from the deluge of HTTP requests. The explanation I could come up with is that they try to crawl all links regardless of how much sense it makes or how much load it causes. cgit being cgit, this mea…
Exactly my observation as well. They devour absolutely everything, no exceptions. No matter how stupid it might be to digest a source code repository via HTTP. They probably don't even recognize what's inside those pages and that there's an easier way to obtain the same result.
Which, as the post notes, it's incredibly stupid. So much for artificial "intelligence"
Re: Creepy Crawlies
#283Why is no one filing lawsuits over this yet?
Against what person or entity?
Sometimes the first step of a lawsuit is discovering who you're suing. It's not unusual and there are processes for it. You could bring something like an access log to a court and receive an order for all ISPs involved to unmask the corresponding users.
Re: Creepy Crawlies
#284Earlier quoted context omitted.
Indeed. The solution isn't technical but legal. It's clearly abusive of - really stealing - other people's resources; there's no question about it. For some reason, like with fraud via email, text, and phone, we don't do anything about it. All this brazen crime and government does nothing; we don't even imagine government doing anything.
The brazen crime of asking for a resource and receiving it. You have the power, and should exercise it, to rate limit bad actors
How do you know who the bad actors are, if you don't use the legal process to unmask them?
Re: Creepy Crawlies
#285Earlier quoted context omitted.
It's a good point, but in practice it depends on how easy those customizations are to implement / maintain, and how much money and effort you save. At some point the compute cost can disrupt even the nicest scale story. I think the path forward is that websites offer one path for humans, and another for scrapers. But the huge catch is the path for scrapers must be _genuinely_ and _reliably_ the more economical and sc…
Why aren't we litigating against scrapers, anyway? DDoS is a felony.
Largely because they're residential botnets in places like Brazil (a real example from one of my sites that was crawled to near-destruction). Someone could probably do something about this, but it's out of reach for individual site owners.
Re: Creepy Crawlies
#286Given the nature of git, wouldn't all that HTML be highly cacheable? I get that's not free either, but it's got to be a lot less intensive than having cgit generate it every single time.
i think its only 7gb of data total, therefore there should be a strategy to fully optimize for this deterministic scenario, but instead of caching instead with maybe compiled streaming renderer(s)... but he said its ok for now.. and anyway why should they need to fix anything, the slurp is what sucks. pun intended
anyway, I have solved this exact scenario many times in the last 30 years... thanks for the dv.. have a hackathon to solve for this. OP says it's not a problem for him right now but if we extrapolate what he's talking about it's definitely a problem aaaaaaand It's totally solvable, Even with all of the crazy combinations he's talking about it's still solvable. And it's already been solved using patterns we see in streaming services. This is completely hackathonable. but why do we even need to bother with this? The slurpers are the cause of this, and they can cause this problem because of Murphy's Law. well you can only account for Murphy's Law with good architecture or something like that or whatever. Ha ha hackathon.
Re: Creepy Crawlies
#287Earlier quoted context omitted.
Against what person or entity?
Bright Data et al.
Bright Data's business is legal, but they could be subpoenaed to find out which customer is making these requests, but first you would have to prove they were actually involved, because there are many residential proxy providers.
Re: Creepy Crawlies
#288Earlier quoted context omitted.
These crawlers (in contrast to e.g. googlebot and similar better behaved crawlers) are not very smart: they seem to make very little effort to avoiding crawling useless deep trees of generated pages. About the only thing they seem to put a lot of effort into is avoiding blocking. (I'll note that while these are generally attributed to AI data gathering because of the timing of when they took off, it's not actually ob…
> it's not actually obvious who's running these bots This is fascinating to me. It's a large enough phenomenon that it's affecting the entire Internet and yet nobody seems to know yet who's actually doing it. Which isn't surprising, of course, it's hard to trace back to a source through all these proxies and it's probably a bunch of distinct groups anyways, but still! Personally I have to wonder how much of it is "sc…
Especially since if any crawler IP is in your country, you can get a subpoena to find out who it is.
Re: Creepy Crawlies
#289> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge. This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices. I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my…
Memory-hard hash functions maybe? Like, you must dedicate 4gb of ram to compute the function. Not a problem for a one-off, but is a problem when reading lots of pages at once.
Or… the site will serve a random seed and the device must compute 4gb of pseudo-random data, then supply a value at a random server-demanded offset.
Re: Creepy Crawlies
#290Earlier quoted context omitted.
These crawlers (in contrast to e.g. googlebot and similar better behaved crawlers) are not very smart: they seem to make very little effort to avoiding crawling useless deep trees of generated pages. About the only thing they seem to put a lot of effort into is avoiding blocking. (I'll note that while these are generally attributed to AI data gathering because of the timing of when they took off, it's not actually ob…
Meta is an exception. They intentionally ignore robots.txt rules. (No second chances, I blocked their whole IP range.)
Everyone ignores robots.txt because every robots.txt blocks everyone.