Efficient Tabular Storage
matthewrocklin.com
Efficient Tabular Storage
1–10 of 10 posts
Re: Efficient Tabular Storage
#2How do databases like MySQL store data efficiently for querying? It seems like something like protobuf would do well here, though you'd need to generate code for each dataset.
Re: Efficient Tabular Storage
#3How do databases like MySQL store data efficiently for querying? It seems like something like protobuf would do well here, though you'd need to generate code for each dataset.
Typically they use row-oriented binary storage, optionally with individual columns or subsets of columns duplicated into indices for fast querying. Have you tried protobufs? How many hundreds of megs per second do you get? I think it is remarkably slow on the scales we're talking about here.
Re: Efficient Tabular Storage
#4This is super interesting, thanks!
Re: Efficient Tabular Storage
#5Vertica has all those performance enhancements, great DB can't recommend because of pricing :(
Re: Efficient Tabular Storage
#6Great share
Re: Efficient Tabular Storage
#7How do databases like MySQL store data efficiently for querying? It seems like something like protobuf would do well here, though you'd need to generate code for each dataset.
Traditional DBMS's get performance by optimizing storage down to the physical layout of the data on the hardware. So MySQL makes a lot of assumptions based on the mechanics if spinning disks and buffers tailored to their physics. Database Systems: The Complete Book is a good text on the subject and the second half is all about the hardware and software used in implementing traditional systems.
Re: Efficient Tabular Storage
#8It looks like the NYCTaxi dataset is here:
http://www.andresmh.com/nyctaxitrips/
Some background on this data:
http://chriswhong.com/data-visualization/taxitechblog1/
And data for 2014 directly from the city:
Re: Efficient Tabular Storage
#9How is it different from HDF5 (h5py and pytables)?
Re: Efficient Tabular Storage
#10Vertica has all those performance enhancements, great DB can't recommend because of pricing :(
kdb+ answers same description.
And it's a 300KB executable with no dependencies (other than glib/MSVCRT).