Feather: A Fast On-Disk Format for Data Frames for R and Python • Hacker News Board

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#51

Post by shoyer » Wed, Mar 30, 2016, 5:53 AM UTC

jzwinck wrote: post #19

HDF5 is supported by many languages including C, C++, R, and Python. It has compression built in. It can read slices easily. It is battle-tested, stable, and used in production for many years by thousands of people. Pandas even has integrated support for DataFrames stored in HDF5. What's the advantage of Feather over HDF5? Couldn't the Feather libraries be written with the same API but HDF5 as the storage format, if…

HDF5 is a clunky file format and dependency. There's a whole host of usual complaints, many of which have already been mentioned: http://cyrille.rossant.net/moving-away-hdf5/

My biggest personal annoyance is that HDF5 isn't thread safe^, so it only supports parallel reading and writing via multiple processes. This makes parallel computing a pain.

This is especially annoying when using HDF5's built-in compression, which hogs a lot of CPU. Inter-process communication is slower than reading from SSDs, so that isn't a great alternative: http://matthewrocklin.com/blog/work/2015/12/29/data-bandwidt...

There's a lot to be said for file formats that you can simply memory map, and that's exactly what Feather/Arrow are. Building out-of-core workflows on top of should be a joy.

Wes -- does the Python library for Feather already release the GIL?

^ you can use and/or compile HDF5 with a global lock, but the underlying library still isn't thread safe.

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#52

Post by IndianAstronaut » Wed, Mar 30, 2016, 6:25 AM UTC

wesm wrote: post #29

jzwinck wrote: post #19

HDF5 is supported by many languages including C, C++, R, and Python. It has compression built in. It can read slices easily. It is battle-tested, stable, and used in production for many years by thousands of people. Pandas even has integrated support for DataFrames stored in HDF5. What's the advantage of Feather over HDF5? Couldn't the Feather libraries be written with the same API but HDF5 as the storage format, if…

Hi, Wes here. HDF5 is a really great piece of software -- I wrote the first implementation of pandas's HDF5 integration (pandas.HDFStore) and Jeff Reback really went to town building out functionality and optimizing it for many different use cases. But the HDF5 C libraries are very heavy dependency. Feather by comparison is an extremely small amount of code ( There is also the Apache Arrow factor -- integration betwe…

How does this contrast with the new Dask library in Python?

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#53

Post by zzleeper » Wed, Mar 30, 2016, 6:27 AM UTC

This looks amazing!

Hadley, Wes, what are your thoughts on how to implement compression? I recall some open source columnar datastores (e.g. infobright) that achieved very VERY fast compression rates with just a few tricks: https://news.ycombinator.com/item?id=8354416

In particular, compression is extremely fast for columnar datastores (its the same type one after the other). Since a lot of times the data is sorted by some ID (date, individual, etc.), you should see large improvements in both speed and disk space.

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#54

Post by sandGorgon » Wed, Mar 30, 2016, 7:24 AM UTC

peatmoss wrote: post #48

sandGorgon wrote: post #43

Earlier quoted context omitted.

Could you talk about why? Other than convenience factor (and R already does it), could you talk about why. Is it stemming from a fundamental aspect of the data format - for example can you save two data frames to the same file? Because if you can save two - why not save two hundred.

It thwarts reproducibility. By saving your workspace, it drags a lot of state from session to session that isn't accounted for. If you share code with someone else, their workspace space won't be the same, and thus the code may not function the same.

Point taken. But we are again delving dangerously close to thou-shall-not . from my perspective, it is a quick and convenient way to save all the data frames in my code. It's a boon for productivity.

If not this, then I pray for Feather to be able to save multiple data frames innone file.

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#55

Post by jabl » Wed, Mar 30, 2016, 8:14 AM UTC

roywiggins wrote: post #28

oddthink wrote: post #26

Earlier quoted context omitted.

Type information is one. CSV is slow, since it has to parse everything on load. CSV has no random access, since rows can be arbitrary length. CSV takes up a lot of disk space, since a 8-byte double gets expanded into a 15+ digit string.

Also there's no single, official CSV standard.

There's RFC 4180. Problem is, there is no universal acceptance in actually following it.. :)

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#56

Post by roel_v » Wed, Mar 30, 2016, 8:52 AM UTC

phillc73 wrote: post #49

rspeer wrote: post #25

Earlier quoted context omitted.

CSV parsing is relatively slow.

Not necessarily. Importing CSV data from disk in R can be pretty quick with 'fread', from the data.table package.[1] Having said that I'm pretty excited about feather and where development will lead. I use RDS files quite heavily, mostly because of the compression which allows much smaller file sizes for distribution.[2] However, there's a trade off with parsing spread and also interoperability between languages. Loo…

Anything that needs to parse text into numbers will be slow. Your links just compare various slow methods and conclude that one isn't quite as slow as the others.

Of course it all depends on what you call 'slow'. Reading a few hundreds of megabytes of megabytes of csv's isn't going to be 'really' slow on modern hardware even if it was fgetc'd character by character.

Either way: anything that represents data as text will become a bottleneck when the size of the dataset grows.

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#57

Post by joelthelion » Wed, Mar 30, 2016, 8:57 AM UTC

It seems like C++ (C?) is also supported. Why don't you advertise it? In my (limited) experience, exchanging data with C/C++ is even more painful than between Python and R.

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#58

Post by phillc73 » Wed, Mar 30, 2016, 9:17 AM UTC

roel_v wrote: post #56

phillc73 wrote: post #49

Earlier quoted context omitted.

Not necessarily. Importing CSV data from disk in R can be pretty quick with 'fread', from the data.table package.[1] Having said that I'm pretty excited about feather and where development will lead. I use RDS files quite heavily, mostly because of the compression which allows much smaller file sizes for distribution.[2] However, there's a trade off with parsing spread and also interoperability between languages. Loo…

Anything that needs to parse text into numbers will be slow. Your links just compare various slow methods and conclude that one isn't quite as slow as the others. Of course it all depends on what you call 'slow'. Reading a few hundreds of megabytes of megabytes of csv's isn't going to be 'really' slow on modern hardware even if it was fgetc'd character by character. Either way: anything that represents data as text w…

The original comment was, "CSV parsing is relatively slow."

Relative to what?

The links I provided show that, using fread from data.table in R, parsing CSV data can be quicker than returning the same data from, for example, an SQLite database.

I believe the links do show CSV parsing relative to a couple of other common data parsing methods in R. Whether they're all "slow methods" is an entirely different question.

If you'd like to compare parsing CSV data relative to a "fast method", I'd very much like to read the analysis.

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#59

Post by andrioni » Wed, Mar 30, 2016, 10:42 AM UTC

wallstquant wrote: post #39

Now if only there was a Julia package for this too

It shouldn't be hard, given that most of the work is being done by a shared C++ library.

(I'll probably take a stab at writing one this weekend, though)

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#60

Post by masklinn » Wed, Mar 30, 2016, 10:56 AM UTC

shoyer wrote: post #51

jzwinck wrote: post #19

HDF5 is supported by many languages including C, C++, R, and Python. It has compression built in. It can read slices easily. It is battle-tested, stable, and used in production for many years by thousands of people. Pandas even has integrated support for DataFrames stored in HDF5. What's the advantage of Feather over HDF5? Couldn't the Feather libraries be written with the same API but HDF5 as the storage format, if…

HDF5 is a clunky file format and dependency. There's a whole host of usual complaints, many of which have already been mentioned: http://cyrille.rossant.net/moving-away-hdf5/ My biggest personal annoyance is that HDF5 isn't thread safe^, so it only supports parallel reading and writing via multiple processes. This makes parallel computing a pain. This is especially annoying when using HDF5's built-in compression, whi…

> Wes -- does the Python library for Feather already release the GIL?

Checking out the code: https://github.com/wesm/feather/blob/master/python/feather/l...

So the actual reading from and writing to disk should be with a released GIL if I'm reading this correctly. The conversion to and from arrays or dataframes holds the GIL.