Wondering why they chose row-major storage. I think it's far more common to only care about a subset of columns than a subset of rows.
BTables: A fast, compact format for Machine Learning
11–16 of 16 posts
Re: BTables: A fast, compact format for Machine Learning
#12Wondering why they chose row-major storage. I think it's far more common to only care about a subset of columns than a subset of rows.
FTA:
"First, we knew we only cared about row-by-row access over the entire file; we do not need things like random row or column reads."
It sounds like they don't care about subsets of either columns or rows, and are looking to optimize table size and time for full table scans.
Re: BTables: A fast, compact format for Machine Learning
#13HDF5 isn't perfect but it does this kind of job pretty well. The C, C++, and HDF5 APIs are definitely not fun to use, but there are wonderful and intuitive APIs available in some languages---I'm thinking of Python's h5py here. Let me add that the OP's experience that HDF5 files were less space efficient than comparable CSV files suggest that something was grossly amiss in his use of HDF5.
(I'm the author of the library) I'm almost certain you're correct - we (thought) we had compression enabled on our feature builder and never found the root cause, but regardless we're happy with how BTables ended up for the other reasons detailed. For future use cases we'll definitely be re-evaluating HDF5!
Re: BTables: A fast, compact format for Machine Learning
#14HDF5 isn't perfect but it does this kind of job pretty well. The C, C++, and HDF5 APIs are definitely not fun to use, but there are wonderful and intuitive APIs available in some languages---I'm thinking of Python's h5py here. Let me add that the OP's experience that HDF5 files were less space efficient than comparable CSV files suggest that something was grossly amiss in his use of HDF5.
HDF5 is actually a pleasure to work with in python due to h5py - and it's quite an efficient data format.
Re: BTables: A fast, compact format for Machine Learning
#15interesting how it jumps from csv to rewrite stuff without just doing SQL and be done with it. since csv did the job almost good enough, it seem like SQL would just fine and dandy while easier to manage and implement (minutes, literally)
note: after reading a little more I suspect SQL would be faster, in fact.
Re: BTables: A fast, compact format for Machine Learning
#16HDF5 isn't perfect but it does this kind of job pretty well. The C, C++, and HDF5 APIs are definitely not fun to use, but there are wonderful and intuitive APIs available in some languages---I'm thinking of Python's h5py here. Let me add that the OP's experience that HDF5 files were less space efficient than comparable CSV files suggest that something was grossly amiss in his use of HDF5.
Another interesting HDF wrapper in Rust: https://github.com/aldanor/hdf5-rs