Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Moving away from HDF5

cyrille.rossant.net

Post reply on HN
Print view View original

21–30 of 67 posts

Re: Moving away from HDF5

#21

Post by batbomb »

iheartmemcache wrote: post #15
ycmbntrthrwaway wrote: post #3

One reason to use binary formats like HDF5 is to avoid precision loss when storing floating-point values. I started using HDF5 once exactly for this reason and it was overkill. HDFView requires Java installed and HDF library with single implementation and complex API is a problem too. For simple uses I now use '%a' printf specifier. It is specifically designed to avoid losing a single bit of informaiton. And you can…

It's also really useful if you have a lot of numerical data streaming in that you want to store and use at a later date. CERN results I'm sure use something similar to HDF5, nearly all of HFT algo trading uses HDF5 for securities they are going to explore down the road but don't want to waste KDB+ licenses on, Google File System's chunking scheme seems to be somewhat similar to it as well. _"Third, most files are mut…

CERN uses ROOT.

Re: Moving away from HDF5

#22

Post by iraikov »

This is a very interesting article, thanks for sharing. I attempted several times to understand the HDF5 C API and create a custom format for storing connectivity data for neuroscience models, but each time I found the API exceedingly complex and bizarre. I am quite impressed that the author managed to write a substantial piece of software based around HDF and relieved to read the sections on the excessive complexity and fragility of the format.

Re: Moving away from HDF5

#23

Post by ycmbntrthrwaway »

bhouston wrote: post #12
ycmbntrthrwaway wrote: post #9

Earlier quoted context omitted.

The problem with binary storage is endianess. If you want portable format, you either have to use text, specify endianess in the format specification or store endianess in the file. For highest speed you need to convert data to your endianess before first use, update information about endianess inside the file and then work with memory-mapped file.

Endianness generally is fast to convert, just pick a standard for the file format and detect on the platform. You can convert endianness with just shifts in C if you need to or with assembly instructions that I believe are one tick on most platforms. It is much much faster to convert endianness (again at least 10x faster) than to parse text. It looks like you can just use these functions to convert between endianness…

It is right that converting endianness is faster than reading text, but it still requires more code than calling printf and scanf. I am not into big data really and my program spends most of the time computing instead of reading and writing so 10x speedup in I/O is just not worth it. As I said, HDF5 was an overkill in my case.

Re: Moving away from HDF5

#24

Post by batbomb »

I've done more than my fair share of fucking with FITS and ROOT files, HDF5, SQLite, proprietary struct-based things, etc...

It's easy to get a file format working on one system. It's Herculean getting it working on all systems bug free. It's nearly impossible to get something to work portably and performant across many systems.

As for simplicity, people start wanting metadata and headers and this and that, and before you know it you need HDF5 or ROOT again and it's no longer simple. Maybe if you're lucky you can stick with something that looks like FITS. If it's tabular, SQLite still can't be beat. Maybe Parquet would work fine too.

I'd vehemently oppose anyone in the projects I work on from trying to standardize on a new in-house format. I'd maybe be okay if they were just building on top of MessagePack or Cap'n Proto/thrift etc... but nearly every disadvantage the OP references about HDF5 will undoubtedly be in anything they cook up themselves. For example, a "simpler format" that works well on distributed architectures, well... now you're going to go back to the single implementation problem.

Re: Moving away from HDF5

#25

Post by mydpy »

We used HDF5 and NETCDF at NASA and it was a constant struggle. I remember when someone dropped the specification on my desk and said, "should be a good read. Enjoy!" Glad you found a more suitable alternative.

Re: Moving away from HDF5

#26

Post by ycmbntrthrwaway »

DiThi wrote: post #20
ycmbntrthrwaway wrote: post #9

Earlier quoted context omitted.

The problem with binary storage is endianess. If you want portable format, you either have to use text, specify endianess in the format specification or store endianess in the file. For highest speed you need to convert data to your endianess before first use, update information about endianess inside the file and then work with memory-mapped file.

IEEE 754 floats don't have different endianess. And as the other comment says, swapping endianess is fast (if ever needed, since most platforms are little endian or hybrid).

> IEEE 754 floats don't have different endianess.

The standard does not specify endianness [1]. From the standard point of view float is a sequence of bits. But if you memory-map the file and store float on little-endian machine, you get bytes reversed compared to how it is written in the standard. On big-endian machine sign is stored in the first byte, and on little-endian machine sign is stored in the last byte.

[1] https://en.wikipedia.org/wiki/Endianness#Floating-point

Re: Moving away from HDF5

#27

Post by iheartmemcache »

batbomb wrote: post #21
iheartmemcache wrote: post #15

Earlier quoted context omitted.

It's also really useful if you have a lot of numerical data streaming in that you want to store and use at a later date. CERN results I'm sure use something similar to HDF5, nearly all of HFT algo trading uses HDF5 for securities they are going to explore down the road but don't want to waste KDB+ licenses on, Google File System's chunking scheme seems to be somewhat similar to it as well. _"Third, most files are mut…

CERN uses ROOT.

Huh! Good to know. ftp://root.cern.ch/root/doc/11InputOutput.pdf[1] The spec for anyone interested. For comparison:https://www.hdfgroup.org/projects/hdf5_aip/aip15.gif to page 6 on CERN's PDF.

Re: Moving away from HDF5

#28

Post by iheartmemcache »

batbomb wrote: post #24

I've done more than my fair share of fucking with FITS and ROOT files, HDF5, SQLite, proprietary struct-based things, etc... It's easy to get a file format working on one system. It's Herculean getting it working on all systems bug free. It's nearly impossible to get something to work portably and performant across many systems. As for simplicity, people start wanting metadata and headers and this and that, and befor…

I've done one format which was append-only (so a pretty easy problem to solve) on one homogeneous system with metadata that had above-average performance (compared to the commercial and open-source alternatives available at the time) but it sounds you have me well-beaten. These are the war-stories that I love to hear. What was your problem domain, what were the recurring implementation problems, where were the bugs primarily?

Re: Moving away from HDF5

#29

Post by mynewtb »

ycmbntrthrwaway wrote: post #7

> You can't use standard Unix/Windows tools like awk, wc, grep, Windows Explorer, text editors, and so on, because the structure of HDF5 files is hidden in a binary blob that only the standard libhdf5 understands. HDF provides command-line tools like h5dump and h5diff, so you can dump HDF5 file to text and pipe it into standard and non-standard unix tools [1]. [1] https://www.hdfgroup.org/products/hdf5_tools/index.ht…

The submission talks about terabytes of data. Copying/transforming is not viable in such situations.

Re: Moving away from HDF5

#30

Post by ycmbntrthrwaway »

mynewtb wrote: post #29
ycmbntrthrwaway wrote: post #7

> You can't use standard Unix/Windows tools like awk, wc, grep, Windows Explorer, text editors, and so on, because the structure of HDF5 files is hidden in a binary blob that only the standard libhdf5 understands. HDF provides command-line tools like h5dump and h5diff, so you can dump HDF5 file to text and pipe it into standard and non-standard unix tools [1]. [1] https://www.hdfgroup.org/products/hdf5_tools/index.ht…

The submission talks about terabytes of data. Copying/transforming is not viable in such situations.

With h5dump you can specify which datasets you want to dump. I am sure nobody is going to use awk, grep and wc to process terabytes of data. As for sanity checks, like checking that probabilities stored in a dataset sum up to 1.0 and things like that, dumping one dataset and processing it with awk should be ok.
Post reply on HN