Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Moving away from HDF5

cyrille.rossant.net

Post reply on HN
Print view View original

51–60 of 67 posts

Re: Moving away from HDF5

#51

Post by nippoo »

ipunchghosts wrote: post #41
nippoo wrote: post #37

Earlier quoted context omitted.

The author (my colleague, and probably the most talented developer I know) isn't replacing HDF5 with a 'proprietary binary format': in fact, the transition is as simple as replacing "HDF5 group" with "folder in a filesystem", "HDF5 dataset" with "binary file on the filesystem" (ie you store each array item sequentially on disk, exactly as HDF5 or any other format will store it, which you can memmap trivially with any…

Filesystems are the worst! Try telling a customer to tar up a directory and send it to you -- things get lost so easily! Most of our customers dont even know what "tar" means. I think you are asking for trouble going with a filesystem as a storage mechanism. HDFfView is not the only viewer in town. There are several viewers. I've used HDF5 weekly for the last 5 years and so have my associates and its been wonderful.…

It depends who your users are and what needs they have. Our users are mostly MATLAB/Python users, pretty tech-savvy, and being able to edit individual bits of the dataset with other applications or write their own code to analyse them is an often-used feature.

It's rare we ever need the whole dataset - in fact, it's really great to be able to say to the user "don't send us your 100GB dataset: just go to the "acquisition" subfolder and send me the 10MB file called "oscilloscope.dat"". With HDF5 this is difficult enough that it's almost always easier to send 99.9% of useless data (i.e. the whole file) when all you want is a single array within it.

If your users will rarely need to do this, you could just store the entire folder hierarchy in a .zip and access it using standard tools that most programming languages have. It's worth noting that the new Microsoft Office formats do exactly this - in their case, a bunch of XML files inside a .ZIP. (Rename a .docx to .zip and you'll see!).

MATLAB has moved from their own custom binary format to HDF5, which is the lesser of two evils.

Re: Moving away from HDF5

#52

Post by mturmon »

ycmbntrthrwaway wrote: post #23
bhouston wrote: post #12

Earlier quoted context omitted.

Endianness generally is fast to convert, just pick a standard for the file format and detect on the platform. You can convert endianness with just shifts in C if you need to or with assembly instructions that I believe are one tick on most platforms. It is much much faster to convert endianness (again at least 10x faster) than to parse text. It looks like you can just use these functions to convert between endianness…

It is right that converting endianness is faster than reading text, but it still requires more code than calling printf and scanf. I am not into big data really and my program spends most of the time computing instead of reading and writing so 10x speedup in I/O is just not worth it. As I said, HDF5 was an overkill in my case.

The OP, and any other file format spec, needs to be able to read/write large numeric datasets. Printf is very poorly suited to this case.

Your comment is true for your use case, but it's not really responsive to the technical issue here.

Re: Moving away from HDF5

#54

Post by ipunchghosts »

nippoo wrote: post #51
ipunchghosts wrote: post #41

Earlier quoted context omitted.

Filesystems are the worst! Try telling a customer to tar up a directory and send it to you -- things get lost so easily! Most of our customers dont even know what "tar" means. I think you are asking for trouble going with a filesystem as a storage mechanism. HDFfView is not the only viewer in town. There are several viewers. I've used HDF5 weekly for the last 5 years and so have my associates and its been wonderful.…

It depends who your users are and what needs they have. Our users are mostly MATLAB/Python users, pretty tech-savvy, and being able to edit individual bits of the dataset with other applications or write their own code to analyse them is an often-used feature. It's rare we ever need the whole dataset - in fact, it's really great to be able to say to the user "don't send us your 100GB dataset: just go to the "acquisit…

Your usecase doesnt seem like HDF5 would be a good fit.

Yes, I do know about docx and zip.

I'm happy with what matlab has done. People send me .mat files and I happily process them in python. And my plots usually look much nicer also. :)

Re: Moving away from HDF5

#55

Post by Dylan16807 »

ycmbntrthrwaway wrote: post #23
bhouston wrote: post #12

Earlier quoted context omitted.

Endianness generally is fast to convert, just pick a standard for the file format and detect on the platform. You can convert endianness with just shifts in C if you need to or with assembly instructions that I believe are one tick on most platforms. It is much much faster to convert endianness (again at least 10x faster) than to parse text. It looks like you can just use these functions to convert between endianness…

It is right that converting endianness is faster than reading text, but it still requires more code than calling printf and scanf. I am not into big data really and my program spends most of the time computing instead of reading and writing so 10x speedup in I/O is just not worth it. As I said, HDF5 was an overkill in my case.

  for (i=start; i
That's not more code than calling printf/scanf.

Re: Moving away from HDF5

#56

Post by hzhou321 »

In the old days, researchers measure the charge of electron with oil drop, and figure out what is gravity with pen and paper. I guess nowadays researchers have to spend million dollar on a electron microscope to look at anything and have to depend on HDF5 to deal with any data.

Re: Moving away from HDF5

#57

Post by hzhou321 »

ycmbntrthrwaway wrote: post #3

One reason to use binary formats like HDF5 is to avoid precision loss when storing floating-point values. I started using HDF5 once exactly for this reason and it was overkill. HDFView requires Java installed and HDF library with single implementation and complex API is a problem too. For simple uses I now use '%a' printf specifier. It is specifically designed to avoid losing a single bit of informaiton. And you can…

I deal with binary data all the time and I just use fwrite in C and pack in Perl, and a text readme in vim explaining the (simple) format. I don't use Matlab but often need exchange the data with Matlab user; I simply search help page and send the one or two lines of matlab code to them along with the data. I'll do the same thing if my fellow colleague switch to numpy tomorrow (it is surprising that they would read HDF5 tutorial but won't read my readme).

Research data are almost always specific. The idea of a general format serving all research just sounds silly to me.

Re: Moving away from HDF5

#58

Post by ngoldbaum »

superbatfish wrote: post #47

I liked the post (well, as an HDF5 user, I found it depressing...). My main qualm with it was the claim about 100x worse performance than just using numpy.memmap(). To the author's credit, he posted his benchmarking code so we could try it ourselves. (Much appreciated.) But as it turned out, there were problems with his benchmark. A fair comparison shows a mixed picture -- hdf5 is faster in some cases, and numpy.memm…

You can also see h5py maintainer Andrew Collette's response here: https://gist.github.com/rossant/7b4704e8caeb8f173084#gistcom...

Re: Moving away from HDF5

#59

Post by saboot »

batbomb wrote: post #21
iheartmemcache wrote: post #15

Earlier quoted context omitted.

It's also really useful if you have a lot of numerical data streaming in that you want to store and use at a later date. CERN results I'm sure use something similar to HDF5, nearly all of HFT algo trading uses HDF5 for securities they are going to explore down the road but don't want to waste KDB+ licenses on, Google File System's chunking scheme seems to be somewhat similar to it as well. _"Third, most files are mut…

CERN uses ROOT.

ROOT has been making progress with regards to one of the points addressed in the post. Regarding needing a special program to view an HDF5 file, root has developed a JavaScript viewer which can display data and stored plots and graphs. It's pretty cool, HDF5 could do something similar if it doesn't already exist. https://root.cern.ch/js/

Re: Moving away from HDF5

#60

Post by nimrody »

echlebek wrote: post #50
nippoo wrote: post #37

Earlier quoted context omitted.

The author (my colleague, and probably the most talented developer I know) isn't replacing HDF5 with a 'proprietary binary format': in fact, the transition is as simple as replacing "HDF5 group" with "folder in a filesystem", "HDF5 dataset" with "binary file on the filesystem" (ie you store each array item sequentially on disk, exactly as HDF5 or any other format will store it, which you can memmap trivially with any…

Funny enough, we started out with a file-based system like the one you described, and moved to an HDF5-based system. Are you not concerned about i-node consumption with the file-system based approach?

It depends on whether you have many small files or a small number of large files. In the former case, one file approach (HDF5) makes sense. In the latter case, you don't have a problem with i-node consumption and you gain the ability to easily access only the data you need without bringing the entire data-set into memory.

I kind of the like the approach suggested above - if you have many small files store them in a zip archive and use some library to access the data directly.

Post reply on HN