Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I use zip bombs to protect my server

idiallo.com

Post reply on HN
Print view View original

191–200 of 467 posts

Re: I use zip bombs to protect my server

#191

Post by vitus »

tom_ wrote: post #144
fracus wrote: post #114

I'm curious why a 10GB file of all zeroes would compress only to 10MB. I mean theoretically you could compress it to one byte. I suppose the compression happens on a stream of data instead of analyzing the whole, but I'd assume it would still do better than 10MB.

It has to cater for any possible input. Even with special case handling for this particular (generally uncommon) case of vast runs of the same value: the compressed data will probably be packetized somehow, and each packet can reproduce only so many repeats, so you'll need to repeat each packet enough times to reproduce the output. With 10 GB, it mounts up. I tried this on my computer with a couple of other tools, af…

The reason why the discussion in this thread centers around gzip (and brotli / zstd) is because those are standard compression schemes that HTTP clients will generally support (RFCs 1952, 7932, and 8478).

As far as I can tell, the biggest amplification you can get out of zstd is 32768 times: per the standard, the maximum decompressed block size is 128KiB, and the smallest compressed block is a 3-byte header followed by a 1-byte block (e.g. run-length-encoded). Indeed, compressing a 1GiB file of zeroes yields 32.9KiB of output, which is quite close to that theoretical maximum.

Brotli promises to allow for blocks that decompress up to 16 MiB, so that actually can exceed the compression ratios that bzip2 gives you on that particular input. Compressing that same 1 GiB file with `brotli -9` gives an 809-byte output. If I instead opt for a 16 GiB file (dd if=/dev/zero of=/dev/stdout bs=4M count=4096 | brotli -9 -o zeroes.br), the corresponding output is 12929 bytes, for a compression ratio of about 1.3 million; theoretically this should be able to scale another 2x, but whether that actually plays out in practice is a different matter.

(The best compression for brotli should be available at -q 11, which is the default, but it's substantially slower to compress compared to `brotli -9`. I haven't worked out exactly what the theoretical compression ratio upper bound is for brotli, but it's somewhere between 1.3 and 2.8 million.)

Also note that zstd provides very good compression ratios for its speed, so in practice most use cases benefit from using zstd.

Re: I use zip bombs to protect my server

#192

Post by tga_d »

There was an incident a little while back where some Tor Project anti-censorship infrastructure was run on the same site as a blog post about zip bombs.[0] One of the zip files got crawled by Google, and added to their list of malicious domains, which broke some pretty important parts of Tor's Snowflake tool. Took a couple weeks to get it sorted out.[1]

[0] https://www.bamsoftware.com/hacks/zipbomb/ [1] https://www.bamsoftware.com/hacks/zipbomb/#safebrowsing

Re: I use zip bombs to protect my server

#193

Post by tom_ »

vitus wrote: post #191
tom_ wrote: post #144

Earlier quoted context omitted.

It has to cater for any possible input. Even with special case handling for this particular (generally uncommon) case of vast runs of the same value: the compressed data will probably be packetized somehow, and each packet can reproduce only so many repeats, so you'll need to repeat each packet enough times to reproduce the output. With 10 GB, it mounts up. I tried this on my computer with a couple of other tools, af…

The reason why the discussion in this thread centers around gzip (and brotli / zstd) is because those are standard compression schemes that HTTP clients will generally support (RFCs 1952, 7932, and 8478). As far as I can tell, the biggest amplification you can get out of zstd is 32768 times: per the standard, the maximum decompressed block size is 128KiB, and the smallest compressed block is a 3-byte header followed…

That's a good point, thanks - I was thinking of this from the point of view of the client downloading a file and then trying to examine it, but of course you'd be much better off fucking up their shit at an earlier stage in the pipeline.

Re: I use zip bombs to protect my server

#194

Post by hxtk »

hxtk wrote: post #173
dagi3d wrote: post #119

Earlier quoted context omitted.

I get your point(and have no idea why it isn't compressed more), but is the theoretical value of 1 byte correct? With just one single byte, how does it know how big should the file be after being decompressed?

In general, this theoretical problem is called the Kolmogorov Complexity of a string: the size of the smallest program that outputs a the input string, for some definition of "program", e.g., an initial input tape for a given universal turing machine. Unfortunately, Kolmogorov Complexity in general is incomputable, because of the halting problem. But a gzip decompressor is not turing-complete, and there are no gzip s…

I'm an idiot, of course the search space is too large. It outgrows what I can brute force by the heat death of the universe by the time it gets to 16 bytes, even if the "test" is a no-op.

I would need to selectively generate grammatically valid zstd streams for this to be tractable at all.

Re: I use zip bombs to protect my server

#195

Post by MoonGhost »

palijer wrote: post #171
MoonGhost wrote: post #79

Earlier quoted context omitted.

Try content labyrinth. I.e. infinitely generated content with a bunch of references to other generated pages. It may help against simple wget and till bots adapt. PS: I'm on the bots side, but don't mind helping.

This doesn't work if you pay bandwidth and CPU usage for your servers though.

That will be your contribution. If others join scrapping will become very pricey. Till bots become smarter. But then they will not download much of generated crap. Which makes it cheaper for you.

Anyway, from bots perspective labyrinths aren't the main problem. Internet is being flooded with quality LLM-generated content.

Re: I use zip bombs to protect my server

#196

Post by bluocms »

aaronbaugher wrote: post #89
carlosjobim wrote: post #78

Earlier quoted context omitted.

There's a lot of essential functionality missing from WordPress, meaning you have to install plugins. Depending on what you need to do. But it's such a bad platform that there really isn't any reason for anybody to use WordPress for anything. No matter your use case, there will be a better alternative to WordPress.

Can you recommend an alternative for a non-technical organization, where there's someone who needs to be able to edit pages and upload documents on a regular basis, so they need as user-friendly an interface as possible for that? Especially when they don't have a budget for it, and you're helping them out as a favor? It's so easy to spin up Wordpress for them, but I'm not a fan either. I've tried Drupal in the past f…

We’re developing https://bluocms.com/

- very hard to hack because we pre render all assets to a Cloudflare kv store

- public website and CMS editor are on different domains

Basically very hard to hack. Also as a bonus is much more reliable as it will only go down when Cloudflare does.

Re: I use zip bombs to protect my server

#197

Post by vitus »

suid wrote: post #182
fracus wrote: post #114

I'm curious why a 10GB file of all zeroes would compress only to 10MB. I mean theoretically you could compress it to one byte. I suppose the compression happens on a stream of data instead of analyzing the whole, but I'd assume it would still do better than 10MB.

Good question. The "ultimate zip bomb" looks something like https://github.com/iamtraction/ZOD - this produces the infamous "42.zip" file, which is about 42KiB, but expands to 3.99 PiB (!). There's literally no machine on Earth today that can deal with that (as a single file, I mean).

> There's literally no machine on Earth today that can deal with that (as a single file, I mean).

Oh? Certainly not in RAM, but 4 PiB is about 125x 36TiB drives (or 188x 24TiB drives). (You can go bigger if you want to shell out tens of thousands per 100TB SSD, at which point you "only" need 45 of those drives.)

These are numbers such that a purpose-built server with enough SAS expanders could easily fit that within a single rack, for less than $100k (based on the list price of an Exos X24 before even considering any bulk discounts).

Re: I use zip bombs to protect my server

#198

Post by MaysonL »

opan wrote: post #166
bobmcnamara wrote: post #158

Earlier quoted context omitted.

I made a 64kx64k JPEG once by feeding the encoder the same line of macro blocks until it produce the entire image. Years later I was finally able to open it.

I had a ton of trouble opening a 10MB or so png a few weeks back. It was stitched together screenshots forming a map of some areas in a game, so it was quite large. Some stuff refused to open it at all as if the file was invalid, some would hang for minutes, some opened blurry. My first semi-success was Fossify Gallery on my phone from F-Droid. If I let it chug a bit, it'd show a blurry image, a while longer it'd foc…

It loaded after 10-15 seconds on myiPad Pro M1, although it did start reloading after I looked around in it.

Re: I use zip bombs to protect my server

#199

Post by j16sdiz »

nophunphil wrote: post #190
carlosjobim wrote: post #108

Earlier quoted context omitted.

By the sound of your question I will guess you want to make a website for a small or medium sized organization? jQuery is probably the only "framework" you should need. If they are selling anything on their website, it's probably going to be through a cloud hosted third party service and then it's just an embedded iframe on their website. If you're making an entire web shop for a very large enterprise or something of…

jQuery hasn’t been necessary for many years. Vanilla JS equivalents of jQuery code are well-supported. https://youmightnotneedjquery.com/

The vanilla one is so much longer.

Re: I use zip bombs to protect my server

#200

Post by gkbrk »

3np wrote: post #48
jerf wrote: post #39

Earlier quoted context omitted.

You want to consider the ratio of your resource consumption to their resource consumption. If you trickle bytes from /dev/random, you are holding open a TCP connection with some minimal overhead, and that's about what they are doing too. Let's assume they are bright enough to use any of the many modern languages or frameworks that can easily handle 10K/100K connections or more on a modern system. They aren't all that…

Also might open up a new DoS vector on entropy consumed by /dev/random so it can be worse than 1:1.

Entropy doesn't really get "consumed" on modern systems. You can read terabytes from /dev/random without running out of anything.
Post reply on HN