Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Biggest image in the smallest space

bamsoftware.com

Post reply on HN
Print view View original

41–50 of 106 posts

Re: Biggest image in the smallest space

#41

Post by Filligree »

DanBC wrote: post #8

That's impressive. Here are some other compression curiosities. http://www.maximumcompression.com/compression_fun.php A 24 byte file that uncompresses to 5 MB; another file with good compression under RAR but almost no compression under ZIP; and a compressed file that decompresses to itself.

I'll add mine: https://brage.info/hello

It's a 1MB file that decompresses to 261 tredecillion bytes of "Hello, World".

No terribly clever stream manipulation; it's a perfectly normal gzip file, other than the size. The generation script is here: http://sprunge.us/VhFc, but see if you can figure it out without peeking.

Re: Biggest image in the smallest space

#43

Post by dahart »

Having dealt with and printed a lot of very large images, e.g., 60k x 60k pixels, I have been on the lookout for image processing software that never decompresses the entire image into ram, but instead works on blocks or scan lines or blocks of scan lines, but stays in constant memory and streams to and from disk. For example, the ImageMagick fork GraphicsMagick does a much better job of this than ImageMagick. What other software is out there that can handle these kinds of images?

Re: Biggest image in the smallest space

#44

Post by MatthewWilkes »

andrewstuart wrote: post #13

Is there a way to check for decompression bombs? I'd like my software to be able to unzip zip files safely.

A python example:

    def decompress(data, maxsize=262144):

        dec = zlib.decompressobj()
        data = dec.decompress(data, maxsize)
        if dec.unconsumed_tail:
            raise ValueError("Possible zip Bomb")
        del dec

        return data

Re: Biggest image in the smallest space

#45

Post by dingaling »

fizgig wrote: post #19
andersthue wrote: post #12

Reminds me of how you could crash a fido node by sending them some big empty files, so when they got automatically unzipped the filled of the harddrive :)

I think this kind of thing was common even a few years ago in DoS'ing mail gateways that uncompressed and scanned various archive formats. Things like really huge files when uncompressed or ridiculously deep nested directory structures. I think most software these days is immune to such tricks, or at least has tunables to reduce the chance of such tricks causing harm.

There was also the trick of infinitely recursive zips that kept decompressing to a copy of themselves.

Zip-bombing was such a problem for our corporate network in the late 1990s that inbound e-mail attachments were deliberately discarded for a while. Chaos ensured.

Re: Biggest image in the smallest space

#46

Post by phkahler »

jerf wrote: post #30
michaelmior wrote: post #2

Actually, it decompresses to a 5.8MB PNG. However, many graphics programs may choose to use three bytes per pixel when rendering the image and because it has incredibly large dimensions, this representation would take up 141GB of RAM.

One of the rules of secure programming is that any program that is used in an even remotely security-sensitive context, and anything displaying a Portable Network Graphic is likely to be used in such a context, must be able to specify resource usage limits. In this case that could be dimensions or a limit on the total RAM allowed to be used. Limits need not be hard, either, but could produce a query, for instance, th…

Ultimately it's going to require a malloc to get the space for all those pixels. That is where things should fail. If not, how is one to specify what the image size limits should be? Ever try to open the Blue Marble images from NASA? In a web browser? Back in 2001?

Re: Biggest image in the smallest space

#47

Post by kuschku »

Filligree wrote: post #41
DanBC wrote: post #8

That's impressive. Here are some other compression curiosities. http://www.maximumcompression.com/compression_fun.php A 24 byte file that uncompresses to 5 MB; another file with good compression under RAR but almost no compression under ZIP; and a compressed file that decompresses to itself.

I'll add mine: https://brage.info/hello It's a 1MB file that decompresses to 261 tredecillion bytes of "Hello, World". No terribly clever stream manipulation; it's a perfectly normal gzip file, other than the size. The generation script is here: http://sprunge.us/VhFc , but see if you can figure it out without peeking.

Actually, it decompresses into a 1.4MiB file, which decompresses into another 1.4MiB file, recursively.

Re: Biggest image in the smallest space

#48

Post by rwmj »

DanBC wrote: post #8

That's impressive. Here are some other compression curiosities. http://www.maximumcompression.com/compression_fun.php A 24 byte file that uncompresses to 5 MB; another file with good compression under RAR but almost no compression under ZIP; and a compressed file that decompresses to itself.

This isn't a decompression bomb, but here are some fun virtual disk images I found using AFL fuzzer. One of the files is 329 bytes, but causes qemu to consume 4 GB of heap trying to process it. This has interesting consequences for the public cloud, where people can upload any old stuff and it is usually processed immediately by 'qemu-img'.

https://bugs.launchpad.net/qemu/+bug/1462949

(I have a big collection of these, but most of the bugs have now been fixed in qemu)

Re: Biggest image in the smallest space

#49

Post by digi_owl »

fizgig wrote: post #19
andersthue wrote: post #12

Reminds me of how you could crash a fido node by sending them some big empty files, so when they got automatically unzipped the filled of the harddrive :)

I think this kind of thing was common even a few years ago in DoS'ing mail gateways that uncompressed and scanned various archive formats. Things like really huge files when uncompressed or ridiculously deep nested directory structures. I think most software these days is immune to such tricks, or at least has tunables to reduce the chance of such tricks causing harm.

Zip bombs, a relative of the fork bomb.

https://en.wikipedia.org/wiki/Zip_bomb

The billion laughs XML attack is also lovely in its simplicity.

https://en.wikipedia.org/wiki/Billion_laughs

Re: Biggest image in the smallest space

#50

Post by phkahler »

dahart wrote: post #43

Having dealt with and printed a lot of very large images, e.g., 60k x 60k pixels, I have been on the lookout for image processing software that never decompresses the entire image into ram, but instead works on blocks or scan lines or blocks of scan lines, but stays in constant memory and streams to and from disk. For example, the ImageMagick fork GraphicsMagick does a much better job of this than ImageMagick. What o…

The key is not to store it in raster form in RAM. Either tiles (like GIMP) or I prefer Z-ordering. Then a user can zoom in and pan around easily - you let the system swap and it won't be bad at all. If they zoom out though, you probably want to store MIP maps of it.

Swap works well for this as long as your data has good locality. huge raster images don't.

But no, I'm not aware of any software that handles stuff like that well - except the GIMPs tiling, but that's not going to help when zoomed out.

Post reply on HN