Are there any plans to support larger than RAM datasets? Like hdf5 or bcolz does.
The format already supports larger than RAM data, but we don't yet have an API for creating those files or just extracting slices. That will come in the future.
Feather: A Fast On-Disk Format for Data Frames for R and Python
11–20 of 78 posts
Re: Feather: A Fast On-Disk Format for Data Frames for R and Python
#12what are some practical uses for this if living in a pure R world? Is this like BigMemory but for data frames? https://cran.r-project.org/web/packages/bigmemory/index.html Thanks.
It's often much faster than rds. And in the long long term there will be tools for computing on feather files that don't require loading it into memory. (In the short term I'll add ways to pull in slices of the full dataset)
Re: Feather: A Fast On-Disk Format for Data Frames for R and Python
#13Both Wes and I (project authors) will be tracking this thread in case you have questions!
In discussing Apache Arrow in the context of Python and R, we wanted to see if we could use the insights from feather to design a very fast file format for storing data frames that could be used by both languages. Thus, the Feather format was born. Is the feather in insights from feather the right word? It reads awkwardly to me, which could just be me lacking context.
Should be insights from _arrow_
Re: Feather: A Fast On-Disk Format for Data Frames for R and Python
#14Both Wes and I (project authors) will be tracking this thread in case you have questions!
As one of many people flicking between R and python/pandas, do you feel there are other areas that have the potential for collaborative tools between the two communities?
I mentioned it elsewhere in the thread, but libdataframe.c seems like a pretty natural continuation to me.
EDIT: What I DON'T know is how much libdataframe would look than libsqlite.
Re: Feather: A Fast On-Disk Format for Data Frames for R and Python
#15Both Wes and I (project authors) will be tracking this thread in case you have questions!
As one of many people flicking between R and python/pandas, do you feel there are other areas that have the potential for collaborative tools between the two communities?
Yes! Anything that is mostly C/C++ for performance could be shared between R and Python now that we have an easy interchange format.
Re: Feather: A Fast On-Disk Format for Data Frames for R and Python
#16what are some practical uses for this if living in a pure R world? Is this like BigMemory but for data frames? https://cran.r-project.org/web/packages/bigmemory/index.html Thanks.
It's often much faster than rds. And in the long long term there will be tools for computing on feather files that don't require loading it into memory. (In the short term I'll add ways to pull in slices of the full dataset)
Faster because it isn't (currently) using compression (which rds uses by default) or faster period?
Either way, the idea of mixed Python/R pipelines with feather file intermediates input/outputs is pretty sweet. Learn in scikit, save to feather, plot in ggplot2... using Make to tie the pieces together?
Re: Feather: A Fast On-Disk Format for Data Frames for R and Python
#17Earlier quoted context omitted.
It's often much faster than rds. And in the long long term there will be tools for computing on feather files that don't require loading it into memory. (In the short term I'll add ways to pull in slices of the full dataset)
Faster because it isn't (currently) using compression (which rds uses by default) or faster period? Either way, the idea of mixed Python/R pipelines with feather file intermediates input/outputs is pretty sweet. Learn in scikit, save to feather, plot in ggplot2... using Make to tie the pieces together?
It's usually faster than either compressed or uncompressed RDS - but if you have heavily duplicated data, compressed RDS can be faster than feather (depending on some tradeoff between compression speed and disk speed). Feather will probably gain compression support eventually.
Re: Feather: A Fast On-Disk Format for Data Frames for R and Python
#18If anyone wants to get this running on Windows, I've made a start. In cpp\thirdparty, you can run the .sh scripts with cygwin, but in build_thirdparty you'll need to add to the appropriate section:
elif [[ "$OSTYPE" == "cygwin"* ]]; then
PARALLEL=$NUMBER_OF_PROCESSORS
if [[ "$OSTYPE" == "cygwin"* ]]; then
msbuild gtest.sln /p:configuration=release
Re: Feather: A Fast On-Disk Format for Data Frames for R and Python
#19HDF5 is supported by many languages including C, C++, R, and Python. It has compression built in. It can read slices easily. It is battle-tested, stable, and used in production for many years by thousands of people. Pandas even has integrated support for DataFrames stored in HDF5.
What's the advantage of Feather over HDF5? Couldn't the Feather libraries be written with the same API but HDF5 as the storage format, if the Feather API is preferable?
Re: Feather: A Fast On-Disk Format for Data Frames for R and Python
#20Are there any plans to support larger than RAM datasets? Like hdf5 or bcolz does.
The format already supports larger than RAM data, but we don't yet have an API for creating those files or just extracting slices. That will come in the future.
Thanks for working on this. Really a great effort. Hope your guys can get good inspiration from projects like h5df, bcolz, pytables having the option of incrementally adding features and maintaining an open spec.