Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Feather: A Fast On-Disk Format for Data Frames for R and Python

blog.rstudio.org

Post reply on HN
Print view View original

31–40 of 78 posts

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#31

Post by wesm »

eximius wrote: post #24

Ignorant question: What does this solve that a csv doesn't? Type information?

CSVs, in addition to being slow to parse (in practice, 40-100MB/s is the typical window, see some benchmarks here http://wesmckinney.com/blog/pandas-and-apache-arrow/), drop types (which have to be inferred), sometimes in a way that cannot be recovered (factor / category levels).

By comparison, Feather performs very close to disk performance. So speeds exceeding 500 MB/s (versus < 100MB/s for CSV) are common.

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#32

Post by wesm »

data_scientist wrote: post #6
hadley wrote: post #2

Both Wes and I (project authors) will be tracking this thread in case you have questions!

Great idea! Some questions: - Both R and Python support strings, factors, and complex objects in a dataframe. What is NOT supported by feather? - Feather is "not for long term data storage". Will it be standardize in a distant future? - Do you plan to integrate it into Pandas?

Feather doesn't support a number of pandas features (but these are features that R doesn't really support, either): https://github.com/wesm/feather/blob/master/python/README.md (not comprehensive)

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#33

Post by wesm »

peatmoss wrote: post #9

My heart is palpitating. I love where this is going. Jake Vanderplas talked about the desire for a common data frame lib to unite the warring tribes in his PyCon keynote a year or so ago, and I couldn't have agreed more. This appears to be "only" a serialization format ("oh, my unicorn only lays golden eggs"). I really hope this is the start of some common library infrastructure that can be used for all aspects of in…

This is precisely the goal of the Apache Arrow project http://arrow.apache.org/ -- and I've been working very hard to bring together diverse groups of data system developers to work on this problem together. Exciting road ahead!

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#34

Post by jacobolus »

wesm wrote: post #29
jzwinck wrote: post #19

HDF5 is supported by many languages including C, C++, R, and Python. It has compression built in. It can read slices easily. It is battle-tested, stable, and used in production for many years by thousands of people. Pandas even has integrated support for DataFrames stored in HDF5. What's the advantage of Feather over HDF5? Couldn't the Feather libraries be written with the same API but HDF5 as the storage format, if…

Hi, Wes here. HDF5 is a really great piece of software -- I wrote the first implementation of pandas's HDF5 integration (pandas.HDFStore) and Jeff Reback really went to town building out functionality and optimizing it for many different use cases. But the HDF5 C libraries are very heavy dependency. Feather by comparison is an extremely small amount of code ( There is also the Apache Arrow factor -- integration betwe…

Is there a plan to add other language implementations (or C implementation wrappers)?

I’d love to see a nice format of this type that can easily be written/read from Javascript in a browser [e.g. to get the data into a D3 visualization] and from Matlab, in addition to Python.

I looked into trying to implement an HDF5 codec in Javascript, but that looked like a large task for one person unfamiliar with the format.

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#35

Post by hadley »

jacobolus wrote: post #34
wesm wrote: post #29

Earlier quoted context omitted.

Hi, Wes here. HDF5 is a really great piece of software -- I wrote the first implementation of pandas's HDF5 integration (pandas.HDFStore) and Jeff Reback really went to town building out functionality and optimizing it for many different use cases. But the HDF5 C libraries are very heavy dependency. Feather by comparison is an extremely small amount of code ( There is also the Apache Arrow factor -- integration betwe…

Is there a plan to add other language implementations (or C implementation wrappers)? I’d love to see a nice format of this type that can easily be written/read from Javascript in a browser [e.g. to get the data into a D3 visualization] and from Matlab, in addition to Python. I looked into trying to implement an HDF5 codec in Javascript, but that looked like a large task for one person unfamiliar with the format.

It's just a matter of someone implementing the protocol. It's not a huge amount of work for an experienced js/matlab programmer.

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#37

Post by jacobolus »

hadley wrote: post #35
jacobolus wrote: post #34

Earlier quoted context omitted.

Is there a plan to add other language implementations (or C implementation wrappers)? I’d love to see a nice format of this type that can easily be written/read from Javascript in a browser [e.g. to get the data into a D3 visualization] and from Matlab, in addition to Python. I looked into trying to implement an HDF5 codec in Javascript, but that looked like a large task for one person unfamiliar with the format.

It's just a matter of someone implementing the protocol. It's not a huge amount of work for an experienced js/matlab programmer.

Is there a spec somewhere, or is the existing implementation the spec?

Edit: https://github.com/wesm/feather/blob/master/doc/FORMAT.md

Seems a bit sparse/incomplete still (as would be expected for a brand new project).

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#38

Post by sandGorgon »

Quick request - could you hook into save.image in R and give the ability to save the entire workspace in R? That would be awesome.

Incidentally I had filed a bug request for a functionality to save the entire workspace in Pandas...but was rejected as being unpythonic. Oh and the devs claimed Apache Arrow was vaporware !

https://github.com/pydata/pandas/issues/12381

Post reply on HN