Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Feather: A Fast On-Disk Format for Data Frames for R and Python

blog.rstudio.org

Post reply on HN
Print view View original

41–50 of 78 posts

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#41

Post by hadley »

sandGorgon wrote: post #38

Quick request - could you hook into save.image in R and give the ability to save the entire workspace in R? That would be awesome. Incidentally I had filed a bug request for a functionality to save the entire workspace in Pandas...but was rejected as being unpythonic. Oh and the devs claimed Apache Arrow was vaporware ! https://github.com/pydata/pandas/issues/12381

I think saving your entire workspace is a bad idea too, sorry!

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#42

Post by hadley »

baldeagle wrote: post #36

Any luck in getting it to work with sas... Maybe in a 10.12 future release? My daily grind is SAS to csv to R, because not everyone else has seen the light.

If SAS wanted to support it, it would be super easy for them to implement it.

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#43

Post by sandGorgon »

hadley wrote: post #41
sandGorgon wrote: post #38

Quick request - could you hook into save.image in R and give the ability to save the entire workspace in R? That would be awesome. Incidentally I had filed a bug request for a functionality to save the entire workspace in Pandas...but was rejected as being unpythonic. Oh and the devs claimed Apache Arrow was vaporware ! https://github.com/pydata/pandas/issues/12381

I think saving your entire workspace is a bad idea too, sorry!

Could you talk about why? Other than convenience factor (and R already does it), could you talk about why.

Is it stemming from a fundamental aspect of the data format - for example can you save two data frames to the same file?

Because if you can save two - why not save two hundred.

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#44

Post by th0ma5 »

eximius wrote: post #24

Ignorant question: What does this solve that a csv doesn't? Type information?

After reading through it, the spec seems to promise superior data interchange in pretty much every way while sacrificing about the only thing good about CSV... That most things can be coerced to produce or consume CSV with a simple read or print.

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#46

Post by andreasvc »

nhatbui wrote: post #23
andreasvc wrote: post #21

Earlier quoted context omitted.

I found it quite frustrating to use HDF5. It does not handle variable-length strings well (very common). In Pandas, categoricals and MultiIndex are not supported. I found that settling for CSV and pickle is more reliable & robust. Also, HDF5 basically implements a hierarchical file system, which is overengineering IMO.

Are you talking about a different type of MultiIndex? http://pandas.pydata.org/pandas-docs/stable/advanced.html

That's the one. Don't recall the specifics, but combinations of MultiIndex as index and columns and the other features were not supported with HDF5.

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#47

Post by peatmoss »

wesm wrote: post #33
peatmoss wrote: post #9

My heart is palpitating. I love where this is going. Jake Vanderplas talked about the desire for a common data frame lib to unite the warring tribes in his PyCon keynote a year or so ago, and I couldn't have agreed more. This appears to be "only" a serialization format ("oh, my unicorn only lays golden eggs"). I really hope this is the start of some common library infrastructure that can be used for all aspects of in…

This is precisely the goal of the Apache Arrow project http://arrow.apache.org/ -- and I've been working very hard to bring together diverse groups of data system developers to work on this problem together. Exciting road ahead!

Ah jeez, I read the feather announce, but not the arrow docs. How did I not know about this!?

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#48

Post by peatmoss »

sandGorgon wrote: post #43
hadley wrote: post #41

Earlier quoted context omitted.

I think saving your entire workspace is a bad idea too, sorry!

Could you talk about why? Other than convenience factor (and R already does it), could you talk about why. Is it stemming from a fundamental aspect of the data format - for example can you save two data frames to the same file? Because if you can save two - why not save two hundred.

It thwarts reproducibility. By saving your workspace, it drags a lot of state from session to session that isn't accounted for. If you share code with someone else, their workspace space won't be the same, and thus the code may not function the same.

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#49

Post by phillc73 »

rspeer wrote: post #25
eximius wrote: post #24

Ignorant question: What does this solve that a csv doesn't? Type information?

CSV parsing is relatively slow.

Not necessarily. Importing CSV data from disk in R can be pretty quick with 'fread', from the data.table package.[1]

Having said that I'm pretty excited about feather and where development will lead. I use RDS files quite heavily, mostly because of the compression which allows much smaller file sizes for distribution.[2] However, there's a trade off with parsing spread and also interoperability between languages. Looks like feather already has the interoperability part sorted, just waiting for compression now. Reading slices direct from disk is pretty exciting too.

[1] http://www.starkingdom.co.uk/faster-csv-import-with-r/

[2] http://www.starkingdom.co.uk/faster-import-with-r-redux/

Post reply on HN