We Need A Standard Layered Image Format • Hacker News Board

Re: We Need A Standard Layered Image Format

#91

Post by blacksqr » Wed, May 01, 2013, 5:18 AM UTC

peatmoss wrote: post #81

blacksqr wrote: post #57

Earlier quoted context omitted.

I just used Spatialite to create a tool for managing large stacks of images on the pixel level; i.e., each image gets its own database file, each pixel is stored as an XYZ coordinate and an RGB value. I can use Spatialite's built-in functions to rotate, translate, scale, etc. images; and use SQL to pull out sub-volumes of the stack, edit and composite them. My biggest stack is almost 2000 images, over 90GB uncompress…

Do you do anything in particular to improve access speed to that image data? I've been working with big matrices that get spit out of traffic assignment software that we use in travel modeling. Every vendor seems to have their own proprietary format. We ended up using HDF5 as a container due to its somewhat awesome speed characteristics. I'd initially tried SQLite for that matrix data, but couldn't squeeze the same k…

Nothing beyond the standard advice for SQLite performance. I found that keeping the data in individual per-image database files worked a lot better than trying to create one big database for the whole stack. With that done, I expect the big performance win to come with parallelization of the image transformation tasks.

Re: We Need A Standard Layered Image Format

#92

Post by shuzchen » Wed, May 01, 2013, 5:19 AM UTC

est wrote: post #82

IMHO SQLite is horrible format. using Matroska (.mkv) would be suitable for anything, vectors, bitmaps, audios and so on. It's essentially a container format like XML but for binary WebM is based on mkv as well.

Could you elaborate on what specific benefits using Matroska would have? One downside I can think of is that there are no programming environments (that I can think of) that ship a Matroska parser, whereas sqlite readers/writers are found in many standard libraries.

Re: We Need A Standard Layered Image Format

#93

Post by mceachen » Wed, May 01, 2013, 5:23 AM UTC

TIFF, http://en.wikipedia.org/wiki/Tagged_Image_File_Format has been around since 1993. It supports layers. It supports compression. It supports paged(!) images. It has well-defined supported image encoding formats. It's already supported by all the image editing software you've ever used (from this millennia).

We have a Standard Layered Image Format. HURRAY. NEXT PROBLEM, PLEASE.

Re: We Need A Standard Layered Image Format

#94

Post by sneak » Wed, May 01, 2013, 5:29 AM UTC

mceachen wrote: post #93

TIFF, http://en.wikipedia.org/wiki/Tagged_Image_File_Format has been around since 1993. It supports layers. It supports compression. It supports paged(!) images. It has well-defined supported image encoding formats. It's already supported by all the image editing software you've ever used (from this millennia). We have a Standard Layered Image Format. HURRAY. NEXT PROBLEM, PLEASE.

http://xkcd.com/927/

Re: We Need A Standard Layered Image Format

#95

Post by eridius » Wed, May 01, 2013, 5:31 AM UTC

chc wrote: post #55

eridius wrote: post #41

Earlier quoted context omitted.

But it doesn't do anything to force the scrollbars either, or to provide a visual treatment that indicates that this is an independent scrollable area. We've had iOS around for 6 years now, with its hidden-scrollbars approach, and OS X has followed the hidden-scrollbars approach for some time now as well. It is no longer acceptable to assume that scrollbars are necessarily going to be visible.

Hacker News does everything it is supposed to according to the CSS standard to indicate to your browser that the area is scrollable. In fact, it is obvious that the site is doing enough because the browser does in fact render the area as scrollable. The fact that your browser doesn't do anything to visually indicate when a region is an independent scrollable area is entirely Apple's fault . You can argue that website…

Hacker News does everything correctly to _functionally_ indicate that the area is scrollable. It does absolutely nothing to _visually_ indicate this. And the CSS spec does not require that the browser visually display scrollbars.

So no, Hacker News is in the wrong here. They're relying on the assumption that the user is using a browser that renders scrollbars, and providing a sub-par experience for any browser that does not. This is somewhat analogous to a site that only renders correctly in IE.

Re: We Need A Standard Layered Image Format

#96

Post by cachvico » Wed, May 01, 2013, 5:32 AM UTC

robbles wrote: post #51

groby_b wrote: post #43

"I want a filesystem in a single file. I'll pick the one I'm most comfortable with and pretend that it's much better than separate files". Meanwhile, we'll ignore all the semantic issues around the data. What exactly _is_ a layer? Any given image file? What if some tools only handle e.g. alpha-layers? What if a layer is actually a filter kernel? How are the layers composited? How are they ordered? Is it a linear orde…

Couldn't someone come up with a relatively sane schema for storing the same information that a PSD would contain in a SQLite database? That way, you could standardize on the semantics that are already documented and familiar to other developers, but avoid the difficulties of the file format , which seem to be the issue here. * disclaimer: I've never had to write a PSD-parser, I'm just going on hearsay.

How about just strip PSD of all its crud, Adobe specifies it openly and gives it to the commons. I have a dream..

Re: We Need A Standard Layered Image Format

#97

Post by sneak » Wed, May 01, 2013, 5:32 AM UTC

SQLite wrote: post #53

aidos wrote: post #46

Earlier quoted context omitted.

This is surely one of the greatest responses I've ever seen on the internet. What was the intention with the unused space originally? Did you foresee this moment coming?

When designing a file format, it is always a good idea to plan for enhancements. There were originally 36 bytes of unused space in the 100-byte header of the SQLite 3.0.0 file format, back in 2004 - bytes set aside specifically to deal with unforeseen needs. Over the years, 12 of those bytes have been allocated to various improvements. 24 bytes remain. (I'd prefer not to use them up all at once or on a whim, obviousl…

Thank heaven for people like you.

Re: We Need A Standard Layered Image Format

#98

Post by sneak » Wed, May 01, 2013, 5:40 AM UTC

cachvico wrote: post #96

robbles wrote: post #51

Earlier quoted context omitted.

Couldn't someone come up with a relatively sane schema for storing the same information that a PSD would contain in a SQLite database? That way, you could standardize on the semantics that are already documented and familiar to other developers, but avoid the difficulties of the file format , which seem to be the issue here. * disclaimer: I've never had to write a PSD-parser, I'm just going on hearsay.

How about just strip PSD of all its crud, Adobe specifies it openly and gives it to the commons. I have a dream..

Keep dreaming: PSD is not a single file format, but two decades of cruft upon cruft:

http://www.jwz.org/blog/2012/11/psd/

Re: We Need A Standard Layered Image Format

#99

Post by jbarham » Wed, May 01, 2013, 5:48 AM UTC

hiker wrote: post #50

OpenEXR solved that problem years ago and it's supported by virtually all content creation tools. An open source implementation with a well designed C++ api and bindings to several other languages. For reasons that I don't understand, it's adoption is weak outside of the VFX industry.

Yeah, OpenEXR (http://www.openexr.com/) is a very nice format, well designed and documented, and with an open-source implementation.

Apart from VFX and post-production, OpenEXR has also become the standard file format in HDR applications.

IMHO it isn't mainstream yet because most people don't know what HDR imaging is and why current image formats are inadequate for HDR. For still image editing Photoshop is overwhelming dominant, whereas in VFX pipelines there's a lot of custom tools so you need an open format.

Re: We Need A Standard Layered Image Format

#100

Post by Groxx » Wed, May 01, 2013, 6:17 AM UTC

This is kind of like saying "We don't need PNG/JPEG/ETC, we just need to put image data in an XML file! It can do everything! Standardization at last!"

Which is true. Except for that little "implementation" detail.

I like the idea of using SQLite as a complex-data-in-single-file format. Hadn't really thought of that for cases like this. But it does nothing to solve the problem of standardization.