Moving away from HDF5 • Hacker News Board

Re: Moving away from HDF5

#61

Post by zvrba » Fri, Jan 08, 2016, 7:02 AM UTC

In my previous job we were evaluating HDF5 for implementing a data-store a couple of years ago. We had some strict requirements about data corruption (e.g., if the program crashes amid a write operation in another thread, the old data must be left intact and readable), as well as multithreaded access. HDF5 supports parallelism from distinct programs, but its multithreaded story was (is?) very weak.

I ended up designing a transactional file format (a kind of log-structured file system with data versioning and garbage collection, all stored in a single file) from scratch to match our requirements. Works flawlessly with terabytes of data.

Re: Moving away from HDF5

#62

Post by srean » Fri, Jan 08, 2016, 8:51 AM UTC

zvrba wrote: post #61

In my previous job we were evaluating HDF5 for implementing a data-store a couple of years ago. We had some strict requirements about data corruption (e.g., if the program crashes amid a write operation in another thread, the old data must be left intact and readable), as well as multithreaded access. HDF5 supports parallelism from distinct programs, but its multithreaded story was (is?) very weak. I ended up designi…

Might one take a peek at that ? In other words was it open sourced or plans to that effect exists ?

Re: Moving away from HDF5

#63

Post by srean » Fri, Jan 08, 2016, 9:07 AM UTC

Leaving a few links here. From the discussion that has taken place it seems these two would be of interest.

http://cscads.rice.edu/workshops/summer-2012/slides/datavis/... Extreme IO scaling with HDF5

http://algoholic.eu/sec2j-journalling-for-hdf5/ HDF5 with a journal

Re: Moving away from HDF5

#64

Post by zvrba » Fri, Jan 08, 2016, 10:19 AM UTC

srean wrote: post #62

zvrba wrote: post #61

In my previous job we were evaluating HDF5 for implementing a data-store a couple of years ago. We had some strict requirements about data corruption (e.g., if the program crashes amid a write operation in another thread, the old data must be left intact and readable), as well as multithreaded access. HDF5 supports parallelism from distinct programs, but its multithreaded story was (is?) very weak. I ended up designi…

Might one take a peek at that ? In other words was it open sourced or plans to that effect exists ?

No and no. Strictly proprietary technology which gives a real competitive advantage. Fun thing is, if you choose your data structures wisely, it's not even that hard to write; it ended up being under 2k lines of C++ code.

GC was offline though; it was performed at the time the container was "upgraded" from RO to RW access. I don't think it'd be difficult to make it online, but there was no need for that.

Re: Moving away from HDF5

#65

Post by DiThi » Fri, Jan 08, 2016, 9:08 PM UTC

ycmbntrthrwaway wrote: post #26

DiThi wrote: post #20

Earlier quoted context omitted.

IEEE 754 floats don't have different endianess. And as the other comment says, swapping endianess is fast (if ever needed, since most platforms are little endian or hybrid).

> IEEE 754 floats don't have different endianess. The standard does not specify endianness [1]. From the standard point of view float is a sequence of bits. But if you memory-map the file and store float on little-endian machine, you get bytes reversed compared to how it is written in the standard. On big-endian machine sign is stored in the first byte, and on little-endian machine sign is stored in the last byte. [1…

Damn, you're right... therefore I have no idea how a simple float quantization algorithm I did could even work.

Re: Moving away from HDF5

#66

Post by ajbonkoski » Sat, Jan 09, 2016, 6:59 AM UTC

"we have a particular use-case where we have a large contiguous array with, say, 100,000 lines and 1000 columns"

This is where they lost me. This is NOT a lot of data. Should we be surprised that memory-mapping works well here?

Below about 100-200 GB you can do everything in memory. You simply don't need fancy file-systems. These systems are for actual big data sets where you have several terabytes to several petabytes.

Don't try to use a chainsaw to cut a piece of paper and then complain that scissors work better. Of course they do...

Re: Moving away from HDF5

#67

Post by rossant » Sat, Jan 09, 2016, 11:45 AM UTC

ajbonkoski wrote: post #66

"we have a particular use-case where we have a large contiguous array with, say, 100,000 lines and 1000 columns" This is where they lost me. This is NOT a lot of data. Should we be surprised that memory-mapping works well here? Below about 100-200 GB you can do everything in memory. You simply don't need fancy file-systems. These systems are for actual big data sets where you have several terabytes to several petabyt…

Unfortunately our users can't afford fancy computers with hundreds of GB of RAM. They often need to process entire datasets on laptops with 16GB of RAM but 1TB+ GB of disk space. Of course with 200 GB of RAM with have no problem at all...

Also, as I said, the 100,000 x 1000 example is a quite optimistic one, we do have cases now with 100,000,000 x 10,000 arrays, and this is only going to increase in the months to come with the new generation of devices.