The title was changed and is now more opaque and less descriptive.
Yeah, I agree. The original title was a lot more descriptive.
Biggest image in the smallest space
91–100 of 106 posts
Re: Biggest image in the smallest space
#92Neat. I needed to make very large PNG bombs recently and toyed with the idea of doing it "manually." In the end I decided to take the lazy route and use libpng[1]. [1]: https://bitbucket.org/tetrep/pngbomb/src/03dfc95065d78562c15...
This works wonderfully! With an image size 123456x123456, I made this happen: http://i.imgur.com/2Dgrazj.png
I killed it at about 25GB memory usage, who knows how high it would have climbed otherwise.
Re: Biggest image in the smallest space
#93Earlier quoted context omitted.
Better graphics programs will not attempt to put the whole image into RAM, but only decompress the pieces needed for processing it. I remember working with multi-megapixel images on systems with far less than 1MB of RAM, many years ago. Perhaps this is a good example of how more hardware resources can lead to them being wasted - the fact that RAM has grown so much that most images fit completely in it, has also meant…
This! Command line programs have no excuse, they should never need to decompress the entire file to memory. GUI image editors and web browsers probably generally do need to, but there definitely are such options for dealing with more pixels than you can display. Anyway, do you know some of these "better" graphics programs that actually behave this way, especially command line processing? I am interested in finding mo…
Virtual memory management saves your butt, almost all the time.
It's ridiculous when the compression ratios are so absurd, like in this case.
Re: Biggest image in the smallest space
#94Re: Biggest image in the smallest space
#95If you follow the "related reading" link on the bottom of TFA, you come to a page by Glenn Randers-Pehrson discussing how libpng deals with decompression bombs. On the bottom of that page you find the following curious note; anyone know what to make of it? """ [Note for any DHS people who have stumbled upon this site, be aware that this is a cybersecurity issue, not a physical security issue. Feel free to contact me…
He's presumably had problems with people confusing "decompression bombs" with the blowy-up kind and sending him panicky e-mails.
Another possibly apocryphal case of linguistic collisions resulting in governmental interest: When the MIT Media Lab started doing work on intelligent kitchen counters, they found that a lot of shadowy government agencies wanted to talk to them about their research into "counter intelligence".
Re: Biggest image in the smallest space
#96Earlier quoted context omitted.
You're overdoing it a bit. I believe the most popular API/library for server-side manipulation of images is ImageMagick, and it has a few options for specifying limits that will easily protect against decompression bombs. That being said, even with these limits, it's undeniable that something like ImageMagick still has a very large attack surface (especially since it uses many third-party libraries), so it should run…
I should have added my "no credit for showing the existence of limits if your code doesn't use them" criterion in advance. Now it looks like I'm moving the goal posts. Oh well; you can see a similar point in my linked post from several months ago which at least adds credence to the idea that this isn't a new reaction of mine. As you can see by reading that post as well, I'd also contend it really ought to be in the c…
I think anyone that accepts scanned documents on their site will add these limits sooner or later out of necessity. At my last job we had them, because occasionally users would upload a 6000 dpi letter-sized page as a 1-bit tiff, or other such nonsense.
I'm actually about to add the same limits at my current job. It is an online slideshow builder, and most of the slides are scanned photographs. Last week someone uploaded a 11k x 17k JPG which hit us during a time of peak load and it caused quite a bit of trouble while the server was trying to build a thumbnail of it.
> I'd also contend it really ought to be in the core API, not an optional thing that defaults to no limits.
These limits really ought to be at a higher level, such as a ulimit on each apache process. In a large code base it requires too much effort to protect against every single avenue of potential accidental denial of service.
For example, at a previous job we had a lot of reports where the date range could be customized. This was fine for a long time, until a large client came along and ran his reports for the past five years. Since it was written in perl, it'd use up the RAM available on the system to generate the report, locking up the only webserver we had.
Re: Biggest image in the smallest space
#97Earlier quoted context omitted.
Yeah, I agree. The original title was a lot more descriptive.
The original title is the one the author gave it. The HN guidelines ask you to not to change that unless it is linkbait or misleading.
Ah yes, of course. Thanks, I'll keep that in mind for the future!
Re: Biggest image in the smallest space
#98Earlier quoted context omitted.
What does Z-ordering mean in this context? I definitely want to avoid swap at all costs and find things that are designed to tile & stream instead. The difference between GraphicsMagick resizing an image by streaming and ImageMagick resizing an image that hits swap is orders of magnitude - seconds versus hours.
>> What does Z-ordering mean in this context? You divide the image into quarters and store each quarter as a continuous block of memory. Do this recursively. Normally we'd index into the pixel data using pixel[x,y]. You can get Z-ordering by using pixel[interleave(x,y)] where the function interleave(x,y) interleaves the bits of the two parameters. This works fantastically well when the image is a square power of two,…
> You can get Z-ordering by using pixel[interleave(x,y)] where the function interleave(x,y) interleaves the bits of the two parameters.
That is pretty cool (also the raytracing application), but most surprising to me, is I remember the interleave operator from reading the INTERCAL[0] docs ... I never even considered the function could actually be applied to something useful :-)
[0] one of the early esoteric programming languages, designed to be "most unlike any other language". It also features a "COME FROM" statement (because GOTO is considered harmful), which iirc also actually has a parallel in some modern programming paradigm.
Re: Biggest image in the smallest space
#99Everyone's focusing on this being a PNG problem but actually if my server unzips a 420 byte file into a 5M file of any kind, I'd say that's the first red flag. Assuming some sort of streaming decompression, you could write an output filter that shuts off the decompressor when it's seen a factor of X bytes. A reasonable factor would be 10 - which in this case would have halted bzip decompression at 4kB. This would pro…
The real problem is image handling libraries that blindly render images into too-large objects where unnecessary. While full-res uncompressed images are very convenient under the hood, the image library should inherently handle anything "too big" gracefully. Instead we're often prone to apps crashing when someone feeds in a ridiculously large image.
A 420B > 5MB expansion should not be a "red flag" because there is nothing about it (including the subsequent attempt to process a 141GB uncompressed image) which cannot be handled appropriately in software. Flagging such ratio limits is arbitrary, and setting an arbitrary limit is usually a sign the software is incorrect, not the data.
Re: Biggest image in the smallest space
#100Actually, it decompresses to a 5.8MB PNG. However, many graphics programs may choose to use three bytes per pixel when rendering the image and because it has incredibly large dimensions, this representation would take up 141GB of RAM.
One of the rules of secure programming is that any program that is used in an even remotely security-sensitive context, and anything displaying a Portable Network Graphic is likely to be used in such a context, must be able to specify resource usage limits. In this case that could be dimensions or a limit on the total RAM allowed to be used. Limits need not be hard, either, but could produce a query, for instance, th…
I'm inclined to agree. As legitimate image sizes increase, there's more need to sanely limit the resources thrown at such images.
While on vacation last week, I finally grokked that my relatively cheap Nikon camera is producing 6000x4000 images...that's about 100MB uncompressed. As a mobile app developer, I'm becoming painfully aware how images breaking the 25MB uncompressed line are breaking apps, with some still-in-use 256MB RAM iOS devices crashing when memory fills under normal usage plus a few instances of such large images (1-2 vacation photos can easily overwhelm available memory).