Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Moving away from HDF5

cyrille.rossant.net

Post reply on HN
Print view View original

41–50 of 67 posts

Re: Moving away from HDF5

#41

Post by ipunchghosts »

nippoo wrote: post #37
ipunchghosts wrote: post #33

Reading all these comments that bash HDF5 makes me want to tell how HDF5 has really worked for my group. Although the spec is huge, there is plenty of sample code online to get it working. You do actually have to read it though to understand slabs, hyperslabs, strides etc. Once you get though, its really versitle. As far as speed, we used it to replace our propriatary data format. We would have to provide readers to…

The author (my colleague, and probably the most talented developer I know) isn't replacing HDF5 with a 'proprietary binary format': in fact, the transition is as simple as replacing "HDF5 group" with "folder in a filesystem", "HDF5 dataset" with "binary file on the filesystem" (ie you store each array item sequentially on disk, exactly as HDF5 or any other format will store it, which you can memmap trivially with any…

Filesystems are the worst! Try telling a customer to tar up a directory and send it to you -- things get lost so easily! Most of our customers dont even know what "tar" means. I think you are asking for trouble going with a filesystem as a storage mechanism.

HDFfView is not the only viewer in town. There are several viewers.

I've used HDF5 weekly for the last 5 years and so have my associates and its been wonderful. MATLAB even uses it to store its .mat files these days.

I think you are misinformed and Im sticking to my story !

Re: Moving away from HDF5

#42

Post by [deleted] »

nippoo wrote: post #37
ipunchghosts wrote: post #33

Reading all these comments that bash HDF5 makes me want to tell how HDF5 has really worked for my group. Although the spec is huge, there is plenty of sample code online to get it working. You do actually have to read it though to understand slabs, hyperslabs, strides etc. Once you get though, its really versitle. As far as speed, we used it to replace our propriatary data format. We would have to provide readers to…

The author (my colleague, and probably the most talented developer I know) isn't replacing HDF5 with a 'proprietary binary format': in fact, the transition is as simple as replacing "HDF5 group" with "folder in a filesystem", "HDF5 dataset" with "binary file on the filesystem" (ie you store each array item sequentially on disk, exactly as HDF5 or any other format will store it, which you can memmap trivially with any…

[deleted]

Re: Moving away from HDF5

#43

Post by ipunchghosts »

skynetv2 wrote: post #4

* High risks of data corruption - HDF is not a simple flat file. Its a complex file format with a lot of in memory structures. A crash may result in corruption but there is no high risk of corruption. More over, if your app crashed, what good is the data? How can you make sense of the partial file? if you just need a flat file which can be parsed and data recovered, then you didnt need HDF in the first place. So wron…

Good post!

Re: Moving away from HDF5

#44

Post by ipunchghosts »

skynetv2 wrote: post #13
x0x0 wrote: post #8

Earlier quoted context omitted.

A common pattern (that my scientific software used) was this: an initial file is created from the raw data pulled from the sequencer. After sequencing, you could run all sorts of analyses. Sometimes the analyses themselves, and sometimes intermediate results, where very slow to compute and hence cached in the file. I think it's reasonable to be very upset if you have a container file and adding new named chunks to th…

Good example. definitely a problem. but that limitation exists now, so the programer would to work around it. hopefully journalling support will appear soon.

Agreed, HDF could benefit from journaling. I have to ask though, why not just make a cached file of your data and onces its done, integrate into the final HDF file?

Re: Moving away from HDF5

#45

Post by jbandela1 »

gaze wrote: post #17
ycmbntrthrwaway wrote: post #9

Earlier quoted context omitted.

The problem with binary storage is endianess. If you want portable format, you either have to use text, specify endianess in the format specification or store endianess in the file. For highest speed you need to convert data to your endianess before first use, update information about endianess inside the file and then work with memory-mapped file.

What big endian platforms do you have to support? Got an SGI Indigo somewhere in your lab?

Exactly. All the major processors are little endian now. Just standardize on little endian. If you are working on some esoteric platform that is big endian you will know it

Re: Moving away from HDF5

#46

Post by ipunchghosts »

x0x0 wrote: post #8
skynetv2 wrote: post #4

* High risks of data corruption - HDF is not a simple flat file. Its a complex file format with a lot of in memory structures. A crash may result in corruption but there is no high risk of corruption. More over, if your app crashed, what good is the data? How can you make sense of the partial file? if you just need a flat file which can be parsed and data recovered, then you didnt need HDF in the first place. So wron…

A common pattern (that my scientific software used) was this: an initial file is created from the raw data pulled from the sequencer. After sequencing, you could run all sorts of analyses. Sometimes the analyses themselves, and sometimes intermediate results, where very slow to compute and hence cached in the file. I think it's reasonable to be very upset if you have a container file and adding new named chunks to th…

This is true, but anything can happen while futzing with a file.

Our method is to make a copy of the file with a .tmp extension, make your mods, then rename the file and delete the old one.

HDF5 was never intended be be a container used to stream data into -- it was meant for sharing data.

Re: Moving away from HDF5

#47

Post by superbatfish »

I liked the post (well, as an HDF5 user, I found it depressing...).

My main qualm with it was the claim about 100x worse performance than just using numpy.memmap(). To the author's credit, he posted his benchmarking code so we could try it ourselves. (Much appreciated.) But as it turned out, there were problems with his benchmark. A fair comparison shows a mixed picture -- hdf5 is faster in some cases, and numpy.memmap is faster in other cases. (You can read my back-and-forth about the benchmarking code in the blog's comments.)

One minor complaint about presentation: Once the benchmarking claims were shown to be bogus, the author should have removed that section from the post, or added an inline "EDIT:" comment. Instead, he merely revised the text to remove any specific numbers, and he didn't add any inline text indicating that the post had been edited.

I think the rest of the post (without performance complaints) is strong enough to stand on its own. After all, performance isn't everything. In fact, I'd say it's a minor consideration compared to the other points.

When it comes to performance, I think the main issue is this: When you have to "roll your own" solution, you become intimately aware of the performance trade-offs you're making. HDF5 is so configurable and yet so opaque that it's tough to understand why you're not seeing the performance you expect.

Re: Moving away from HDF5

#48

Post by superbatfish »

superbatfish wrote: post #47

I liked the post (well, as an HDF5 user, I found it depressing...). My main qualm with it was the claim about 100x worse performance than just using numpy.memmap(). To the author's credit, he posted his benchmarking code so we could try it ourselves. (Much appreciated.) But as it turned out, there were problems with his benchmark. A fair comparison shows a mixed picture -- hdf5 is faster in some cases, and numpy.memm…

And one last point. In the blog comments, I wrote this, which I think sums up my view of the performance discussion:

...it's worth noting that many of the tricky things about tuning hdf5 performance are not unique to HDF5. For storing ND data, there will always be decisions to make about when to load data into RAM vs. accessing it on demand, whether or not to store the data in "chunks", what the size of those chunks should be (based on your anticipated access patterns), whether/how to compress the data, etc. These are generally hard problems; we can't blame HDF5 for all of them.

Re: Moving away from HDF5

#49

Post by nippoo »

x0x0 wrote: post #5

The problem -- and I've been burned on both sides of this -- is that you need either a container file, or you need users to understand that a directory is essentially a file. Not only does this complicate users lives when they want to move what they, quite reasonably, view as a single file between different computers or back it up, but they can and will remove individual pieces. Or copy it around and have some of the…

This is a double-edged sword. For our (reasonably savvy) users, being able to duplicate and easily modify individual datasets/files is a feature, not a bug: people can symlink the contents of an entire folder but modify a single array and easily run their analysis on this slightly different dataset, for example.

While it's true that you lose atomicity with this, it can both burn you and help you: you can track specific parts of your dataset in revision control, email subsets of it back and forth, combine datasets easily, or even store it across several servers and manage it with symlinks, for example.

Our users are aware of this and it isn't really a problem for our use-case. But if you're worried, there's always the option of having your whole dataset as a ZIP/TAR file (like all of Microsoft Office file formats are - XML files within a .ZIP); tools for modifying folders within ZIP are much more well-established than HDFView, and most modern programming languages provide libraries to read and modify files within archives without unzipping them; you could make your program agnostic to the files being within an archive (high portability, lower performance) or directly within the FS (loss of atomicity, easier/faster to use).

Re: Moving away from HDF5

#50

Post by echlebek »

nippoo wrote: post #37
ipunchghosts wrote: post #33

Reading all these comments that bash HDF5 makes me want to tell how HDF5 has really worked for my group. Although the spec is huge, there is plenty of sample code online to get it working. You do actually have to read it though to understand slabs, hyperslabs, strides etc. Once you get though, its really versitle. As far as speed, we used it to replace our propriatary data format. We would have to provide readers to…

The author (my colleague, and probably the most talented developer I know) isn't replacing HDF5 with a 'proprietary binary format': in fact, the transition is as simple as replacing "HDF5 group" with "folder in a filesystem", "HDF5 dataset" with "binary file on the filesystem" (ie you store each array item sequentially on disk, exactly as HDF5 or any other format will store it, which you can memmap trivially with any…

Funny enough, we started out with a file-based system like the one you described, and moved to an HDF5-based system.

Are you not concerned about i-node consumption with the file-system based approach?

Post reply on HN