Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Feather: A Fast On-Disk Format for Data Frames for R and Python

blog.rstudio.org

Post reply on HN
Print view View original

71–78 of 78 posts

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#71

Post by infinite8s »

hadley wrote: post #10
data_scientist wrote: post #6

Earlier quoted context omitted.

Great idea! Some questions: - Both R and Python support strings, factors, and complex objects in a dataframe. What is NOT supported by feather? - Feather is "not for long term data storage". Will it be standardize in a distant future? - Do you plan to integrate it into Pandas?

Feather currently doesn't support recursive/hierarchical data structures, like lists in R. That'll be added in the future though. We'll definitely standardise in the future so you can feel confident using it in the long term. I have no plans to integrate it with pandas, but I'm sure Wes does ;)

Will you take a storage approach for nested data similar to Parquet?

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#72

Post by rspeer »

roel_v wrote: post #62
phillc73 wrote: post #58

Earlier quoted context omitted.

The original comment was, "CSV parsing is relatively slow." Relative to what? The links I provided show that, using fread from data.table in R, parsing CSV data can be quicker than returning the same data from, for example, an SQLite database. I believe the links do show CSV parsing relative to a couple of other common data parsing methods in R. Whether they're all "slow methods" is an entirely different question. If…

I guess in re-reading my comment, I phrased it in a tone that was more combative than I meant, so don't take the GP as an attack. My point was, all conversions from text into numbers (I'm assuming here that the target use case is reading large amounts of numeric data) are slow and the slow part isn't the IO, but the conversion from text to numbers. In that light, it isn't a surprise that sqlite isn't much faster than…

The fact that storing data in binary format gives you more speed is exactly why you'd want to use Feather. It means you don't have to translate things into an unspecified .bin format. This is the answer to the original question about "why not use CSV".

I don't know about R, but in Python, operations on objects such as NumPy arrays and Pandas DataFrames are all implemented using fast C code, and so is Feather. You can be concerned about speed.

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#73

Post by rspeer »

phillc73 wrote: post #58
roel_v wrote: post #56

Earlier quoted context omitted.

Anything that needs to parse text into numbers will be slow. Your links just compare various slow methods and conclude that one isn't quite as slow as the others. Of course it all depends on what you call 'slow'. Reading a few hundreds of megabytes of megabytes of csv's isn't going to be 'really' slow on modern hardware even if it was fgetc'd character by character. Either way: anything that represents data as text w…

The original comment was, "CSV parsing is relatively slow." Relative to what? The links I provided show that, using fread from data.table in R, parsing CSV data can be quicker than returning the same data from, for example, an SQLite database. I believe the links do show CSV parsing relative to a couple of other common data parsing methods in R. Whether they're all "slow methods" is an entirely different question. If…

> Relative to what?

Relative to HDF5, and relative to Feather if it's doing its job right.

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#75

Post by vannevar »

hadley wrote: post #68
th0ma5 wrote: post #45

Any plans to support a pipe-aware POSIX command, or get this into PostgreSQL?

Not by us, but we expect that many other projects will add feather support now that it's used by both R and Python.

Is there any functional distinction between a data frame and a relational table? Could you implement a persistent frame just by wrapping sqlite?

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#76

Post by roel_v »

rspeer wrote: post #72
roel_v wrote: post #62

Earlier quoted context omitted.

I guess in re-reading my comment, I phrased it in a tone that was more combative than I meant, so don't take the GP as an attack. My point was, all conversions from text into numbers (I'm assuming here that the target use case is reading large amounts of numeric data) are slow and the slow part isn't the IO, but the conversion from text to numbers. In that light, it isn't a surprise that sqlite isn't much faster than…

The fact that storing data in binary format gives you more speed is exactly why you'd want to use Feather. It means you don't have to translate things into an unspecified .bin format. This is the answer to the original question about "why not use CSV". I don't know about R, but in Python, operations on objects such as NumPy arrays and Pandas DataFrames are all implemented using fast C code, and so is Feather. You can…

Yes, of course, and I'd much rather use something like it; and when it will support matrices with more than 2 dimensions, I will (or at least, I will look into it). In cases where I need more robust storage I already use HDF5 or NetCDF but they're a PITA to work with.

Of course in-memory operations can be implemented efficiently, R does that too. But Python needs to parse CSV into numbers just like everybody else, and even if it's done in C underneath, it'll still be 'slow' (for some values of that word).

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#77

Post by hadley »

sandGorgon wrote: post #70
hadley wrote: post #66

Earlier quoted context omitted.

Basically because you should be encoding state in code, not data. If you store data between sessions, it's easy to lose the code that you use to create it and then later on you can't recreate it. It is convenient to save your complete workspace but I've seen too many cases where it's contributed to lack of reproducibility to spend my time working on it.

So there's a use case difference. I create models from remote data sources - this is incremental on a daily basis and takes quite a bit of time. So I snapshot the workspace after I do a run and do some experiments. Now - for me, saving the workspace is a convenience feature, NOT a programming feature. This is what I mean by thou-shall-not. My use case is very well defined and I'm not stupid. And I completely knows th…

I hope I didn't come across as "thou shalt not" - it's just never going to be high in my priority list.

(And even for your use case I would think you'd be better off keeping the models in a list and saving that. Then other random stuff in your evn won't get carried along for the ride)

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#78

Post by sandGorgon »

hadley wrote: post #77
sandGorgon wrote: post #70

Earlier quoted context omitted.

So there's a use case difference. I create models from remote data sources - this is incremental on a daily basis and takes quite a bit of time. So I snapshot the workspace after I do a run and do some experiments. Now - for me, saving the workspace is a convenience feature, NOT a programming feature. This is what I mean by thou-shall-not. My use case is very well defined and I'm not stupid. And I completely knows th…

I hope I didn't come across as "thou shalt not" - it's just never going to be high in my priority list. (And even for your use case I would think you'd be better off keeping the models in a list and saving that. Then other random stuff in your evn won't get carried along for the ride)

Oh no you did not! That was polite musing. Thank you for the reply - I still hope you change your mind. Because people do have genuine, but different needs ;)
Post reply on HN