Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A better zip bomb (2019)

bamsoftware.com

Post reply on HN
Print view View original

51–60 of 60 posts

Re: A better zip bomb (2019)

#51

Post by integralid »

10000truths wrote: post #17
542458 wrote: post #4

Okay, so I know back in the day you could choke scanning software (ie email attachment scanners) by throwing a zip bomb into them. I believe the software has gotten smarter these days so it won’t simply crash when that happens - but how is this done; How does one detect a zip bomb?

For any compression algorithm in general, you keep track of A = {uncompressed bytes processed} and B = {compressed bytes processed} while decompressing, and bail out when either of the following occur: 1. A exceeds some unreasonable threshold 2. A/B exceeds some unreasonable threshold

In practice one of the things that happens very often is that you compress a file filled with null bytes. Such files compress extremely well, and would trigger your A/B threshold.

On the other hand, zip bomb described in this blog post relies on decompressing the same data multiple times - so it wouldn't trigger your A/B heuristics necessarily.

Finally, A just means "you can't compress more than X bytes with my file format", right? Not a desirable property to have. If deflate authors had this idea when they designed the algorithm, I bet files larger than "unreasonable" 16MB would be forbidden.

Re: A better zip bomb (2019)

#52

Post by brabel »

st_goliath wrote: post #26
arjie wrote: post #14

The fact that ZIP files include the catalog/directory at the end is such nostalgia fever. Back in the day it meant that if you naïvely downloaded the file, a partial download would be totally useless. Fortunately, in the early 2000s, we got HTTP's Range and a bunch of zip-aware downloaders that would fetch the catalog first so that you could preview a zip you were downloading and even extract part of a file! Good tim…

> ... a partial download would be totally useless ... no, not totally . The directory at the end of the archive points backwards to local headers, which in turn include all the necessary information, e.g. the compressed size inside the archive, compression method, the filename and even a checksum. If the archive isn't some recursive/polyglot nonsense as in the article, it's essentially just a tightly packed list of c…

> the directory at the end is really just for quick access.

No, its purpose was to allow multi floppy disks archives. You would insert the last disk, then the other ones, one by one…

Re: A better zip bomb (2019)

#53

Post by 10000truths »

integralid wrote: post #51
10000truths wrote: post #17

Earlier quoted context omitted.

For any compression algorithm in general, you keep track of A = {uncompressed bytes processed} and B = {compressed bytes processed} while decompressing, and bail out when either of the following occur: 1. A exceeds some unreasonable threshold 2. A/B exceeds some unreasonable threshold

In practice one of the things that happens very often is that you compress a file filled with null bytes. Such files compress extremely well, and would trigger your A/B threshold. On the other hand, zip bomb described in this blog post relies on decompressing the same data multiple times - so it wouldn't trigger your A/B heuristics necessarily. Finally, A just means "you can't compress more than X bytes with my file…

> In practice one of the things that happens very often is that you compress a file filled with null bytes. Such files compress extremely well, and would trigger your A/B threshold.

Sure, if you expect to decompress files with high compression ratios, then you'll want to adjust your knobs accordingly.

> On the other hand, zip bomb described in this blog post relies on decompressing the same data multiple times - so it wouldn't trigger your A/B heuristics necessarily.

If you decompress the same data multiple times, then you increment A multiple times. The accounting still works regardless of whether the data is same or different. Perhaps a better description of A and B in my post would be {number of decompressed bytes written} and {number of compressed bytes read}, respectively.

> Finally, A just means "you can't compress more than X bytes with my file format", right? Not a desirable property to have. If deflate authors had this idea when they designed the algorithm, I bet files larger than "unreasonable" 16MB would be forbidden.

The limitation is imposed by the application, not by the codec itself. The application doing the decompression is supposed to process the input incrementally (in the case of DEFLATE, reading one block at a time and inflating it), updating A and B on each iteration, and aborting if a threshold is violated.

Re: A better zip bomb (2019)

#54

Post by st_goliath »

brabel wrote: post #52
st_goliath wrote: post #26

Earlier quoted context omitted.

> ... a partial download would be totally useless ... no, not totally . The directory at the end of the archive points backwards to local headers, which in turn include all the necessary information, e.g. the compressed size inside the archive, compression method, the filename and even a checksum. If the archive isn't some recursive/polyglot nonsense as in the article, it's essentially just a tightly packed list of c…

> the directory at the end is really just for quick access. No, its purpose was to allow multi floppy disks archives. You would insert the last disk, then the other ones, one by one…

That literally is quick access, it does the same thing in both cases, trying to get rid of the linear scan and having to plow through data unnecessarily.

If the archive is on a hard disk, the program reads the directory at the end and then seeks to the local header, rather than doing a linear scan. Or the floppy motor, if it is a small archive on a single floppy.

If you have multiple floppies, you insert the last one, the program reads the header and then tells you what floppy to insert, rather than having to go through them one by one, which you know, would be slower.

In one case, a hard disk arm, or the floppy motor, does the seeking, in the other case, your hands do the seeking. But it's still the same algorithm, doing the same thing, for the same reason.

Re: A better zip bomb (2019)

#55

Post by RGamma »

From the bottom of the page

> A final plea

It's time to put an end to Facebook. Working there is not ethically neutral: every day that you go into work, you are doing something wrong. If you have a Facebook account, delete it. If you work at Facebook, quit.

And let us not forget that the National Security Agency must be destroyed.

Re: A better zip bomb (2019)

#58

Post by measurablefunc »

shakna wrote: post #48
measurablefunc wrote: post #16

Decompression is equivalent to executing code for a specialized virtual machine. It should be possible to automate this process of finding "small" programs that generate "large" outputs. Could even be an interesting AI benchmark.

Many of them already do this. [0] It is a much easier problem to solve than you would expect. No need to drag in a data centre when heuristics can get you close enough. [0] https://sources.debian.org/patches/unzip/6.0-29/23-cve-2019-...

I meant it should be possible to take a specialized virtual machine that is equivalent to decompressing some compressed bitstream & figure out how to write programs for it that are small but generate large outputs, not that it should be possible to do static analysis & figure out whether the given small program will generate a large output although that is also an interesting problem to solve & would also be an interesting AI benchmark.

Re: A better zip bomb (2019)

#59

Post by thunderfork »

halapro wrote: post #35
arjie wrote: post #14

The fact that ZIP files include the catalog/directory at the end is such nostalgia fever. Back in the day it meant that if you naïvely downloaded the file, a partial download would be totally useless. Fortunately, in the early 2000s, we got HTTP's Range and a bunch of zip-aware downloaders that would fetch the catalog first so that you could preview a zip you were downloading and even extract part of a file! Good tim…

I hate that the most common video container on the web does this too. Most non-"stream-ready" mp4 files lack even the basic information such as height/width until the file has completed loading.[1] [1]: https://forum.videohelp.com/threads/393096-Fixing-Partially-Download-MP4-Files

This is also a big issue if you're using mp4 as your container for a real-time capture (e.g. livestreaming and saving the stream to disk) - if the capture software crashes without writing the MOOV atom, it's a real pain to recover the video

Re: A better zip bomb (2019)

#60

Post by [deleted] »

brabel wrote: post #52
st_goliath wrote: post #26

Earlier quoted context omitted.

> ... a partial download would be totally useless ... no, not totally . The directory at the end of the archive points backwards to local headers, which in turn include all the necessary information, e.g. the compressed size inside the archive, compression method, the filename and even a checksum. If the archive isn't some recursive/polyglot nonsense as in the article, it's essentially just a tightly packed list of c…

> the directory at the end is really just for quick access. No, its purpose was to allow multi floppy disks archives. You would insert the last disk, then the other ones, one by one…

[deleted]
Post reply on HN