Feather: A Fast On-Disk Format for Data Frames for R and Python
blog.rstudio.org
Feather: A Fast On-Disk Format for Data Frames for R and Python
1–10 of 78 posts
Re: Feather: A Fast On-Disk Format for Data Frames for R and Python
#2Both Wes and I (project authors) will be tracking this thread in case you have questions!
Re: Feather: A Fast On-Disk Format for Data Frames for R and Python
#3what are some practical uses for this if living in a pure R world?
Is this like BigMemory but for data frames?
https://cran.r-project.org/web/packages/bigmemory/index.html
Thanks.
Re: Feather: A Fast On-Disk Format for Data Frames for R and Python
#4Both Wes and I (project authors) will be tracking this thread in case you have questions!
As one of many people flicking between R and python/pandas, do you feel there are other areas that have the potential for collaborative tools between the two communities?
Re: Feather: A Fast On-Disk Format for Data Frames for R and Python
#5Both Wes and I (project authors) will be tracking this thread in case you have questions!
In discussing Apache Arrow in the context of Python and R, we wanted to see if we could use the insights from feather to design a very fast file format for storing data frames that could be used by both languages. Thus, the Feather format was born.
Is the feather in insights from feather the right word? It reads awkwardly to me, which could just be me lacking context.
Re: Feather: A Fast On-Disk Format for Data Frames for R and Python
#6Both Wes and I (project authors) will be tracking this thread in case you have questions!
Great idea! Some questions:
- Both R and Python support strings, factors, and complex objects in a dataframe. What is NOT supported by feather?
- Feather is "not for long term data storage". Will it be standardize in a distant future?
- Do you plan to integrate it into Pandas?
Re: Feather: A Fast On-Disk Format for Data Frames for R and Python
#7Are there any plans to support larger than RAM datasets? Like hdf5 or bcolz does.
Re: Feather: A Fast On-Disk Format for Data Frames for R and Python
#8I can see that Wes is a reporter for Spark issue (https://issues.apache.org/jira/browse/SPARK-13534), what are the plans (if any) for tighter Spark/Python/R DataFrames integration?
Re: Feather: A Fast On-Disk Format for Data Frames for R and Python
#9My heart is palpitating. I love where this is going. Jake Vanderplas talked about the desire for a common data frame lib to unite the warring tribes in his PyCon keynote a year or so ago, and I couldn't have agreed more.
This appears to be "only" a serialization format ("oh, my unicorn only lays golden eggs"). I really hope this is the start of some common library infrastructure that can be used for all aspects of in- and out-of-memory data frames.
Great work, and I hope it is a harbinger of good things to come. Also, I'll treat this as tangible evidence that the "language wars" are stupid.
Re: Feather: A Fast On-Disk Format for Data Frames for R and Python
#10Both Wes and I (project authors) will be tracking this thread in case you have questions!
Great idea! Some questions: - Both R and Python support strings, factors, and complex objects in a dataframe. What is NOT supported by feather? - Feather is "not for long term data storage". Will it be standardize in a distant future? - Do you plan to integrate it into Pandas?
Feather currently doesn't support recursive/hierarchical data structures, like lists in R. That'll be added in the future though. We'll definitely standardise in the future so you can feel confident using it in the long term.
I have no plans to integrate it with pandas, but I'm sure Wes does ;)