Live data from Hacker News

Biggest image in the smallest space

bamsoftware.com

81–90 of 106 posts

Re: Biggest image in the smallest space

#83
post #60

Photoshop was able to show it: http://i.imgur.com/7EdBySv.png (Macbook Pro, 16GB RAM)

Photoshop is an example of a graphics program that doesn't attempt to read the entire image into memory. How much RAM did it actually use?

Difficult to say. I don't completely understand the activity monitor RAM column: http://i.imgur.com/QS3NPQQ.png Looking at the activity monitor details we see that it uses something in the order of 2.54 GB of real memory. I suspect the rest is mostly compression.

Re: Biggest image in the smallest space

#84
post #8

That's impressive. Here are some other compression curiosities. http://www.maximumcompression.com/compression_fun.php A 24 byte file that uncompresses to 5 MB; another file with good compression under RAR but almost no compression under ZIP; and a compressed file that decompresses to itself.

I'll add mine: https://brage.info/hello It's a 1MB file that decompresses to 261 tredecillion bytes of "Hello, World". No terribly clever stream manipulation; it's a perfectly normal gzip file, other than the size. The generation script is here: http://sprunge.us/VhFc , but see if you can figure it out without peeking.

Is that short or long scale tredecillion?

Re: Biggest image in the smallest space

#85
post #47

Earlier quoted context omitted.

I'll add mine: https://brage.info/hello It's a 1MB file that decompresses to 261 tredecillion bytes of "Hello, World". No terribly clever stream manipulation; it's a perfectly normal gzip file, other than the size. The generation script is here: http://sprunge.us/VhFc , but see if you can figure it out without peeking.

Actually, it decompresses into a 1.4MiB file, which decompresses into another 1.4MiB file, recursively.

I sized it to fit on a floppy disk. :-3

But, no. There's a finite number of levels, and it blows up pretty quickly.

Re: Biggest image in the smallest space

#86
post #8

That's impressive. Here are some other compression curiosities. http://www.maximumcompression.com/compression_fun.php A 24 byte file that uncompresses to 5 MB; another file with good compression under RAR but almost no compression under ZIP; and a compressed file that decompresses to itself.

I'll add mine: https://brage.info/hello It's a 1MB file that decompresses to 261 tredecillion bytes of "Hello, World". No terribly clever stream manipulation; it's a perfectly normal gzip file, other than the size. The generation script is here: http://sprunge.us/VhFc , but see if you can figure it out without peeking.

May I suggest for the merely curious:

    curl https://brage.info/hello | od -c | head -40

Re: Biggest image in the smallest space

#88
post #76

Earlier quoted context omitted.

Better graphics programs will not attempt to put the whole image into RAM, but only decompress the pieces needed for processing it. I remember working with multi-megapixel images on systems with far less than 1MB of RAM, many years ago. Perhaps this is a good example of how more hardware resources can lead to them being wasted - the fact that RAM has grown so much that most images fit completely in it, has also meant…

What you describe sounds a bit like demand paging of a memory-mapped file. The problem with implementing it for a 2d image is that a given rectangular region doesn't map to a contiguous area in memory. It's easy to construct a long thin image that would cause problems for a line-based demand paging strategy. For example, ten pixels high by a billion pixels wide. Edit: skipping sections of the line to get to the regio…

It is a bit like demand paging, yes. There are formats for which rectangular regions map to contiguous block of memory.

TIFF has specs for tiles, strips, subfiles, layers, etc. AFIAK, hardly anyone uses those, but they certainly exist.

JPEG is also composed of 8x8 squares and is easy to stream. The API has per-scanline read & write callbacks IIRC. But you're right; a very very wide one might be a case it can't deal with.

Re: Biggest image in the smallest space

#89
post #19

Earlier quoted context omitted.

I think this kind of thing was common even a few years ago in DoS'ing mail gateways that uncompressed and scanned various archive formats. Things like really huge files when uncompressed or ridiculously deep nested directory structures. I think most software these days is immune to such tricks, or at least has tunables to reduce the chance of such tricks causing harm.

Zip bombs, a relative of the fork bomb. https://en.wikipedia.org/wiki/Zip_bomb The billion laughs XML attack is also lovely in its simplicity. https://en.wikipedia.org/wiki/Billion_laughs

I was doing a presentation about various bombs last year and crashed PowerPoint by copy-pasting billion laughs in a slide. Simple but extremely effective.

Re: Biggest image in the smallest space

#90
post #43

Having dealt with and printed a lot of very large images, e.g., 60k x 60k pixels, I have been on the lookout for image processing software that never decompresses the entire image into ram, but instead works on blocks or scan lines or blocks of scan lines, but stays in constant memory and streams to and from disk. For example, the ImageMagick fork GraphicsMagick does a much better job of this than ImageMagick. What o…

Nuke works in scanlines like this, and can process a whole tree of operations only loading the input lines necessary for the current output row. The SDK docs explain the architecture somewhat: https://www.thefoundry.co.uk/products/nuke/developers/90/ndk...
Post reply on HN