Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Feather: A Fast On-Disk Format for Data Frames for R and Python

blog.rstudio.org

Post reply on HN
Print view View original

21–30 of 78 posts

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#21

Post by andreasvc »

jzwinck wrote: post #19

HDF5 is supported by many languages including C, C++, R, and Python. It has compression built in. It can read slices easily. It is battle-tested, stable, and used in production for many years by thousands of people. Pandas even has integrated support for DataFrames stored in HDF5. What's the advantage of Feather over HDF5? Couldn't the Feather libraries be written with the same API but HDF5 as the storage format, if…

I found it quite frustrating to use HDF5. It does not handle variable-length strings well (very common). In Pandas, categoricals and MultiIndex are not supported. I found that settling for CSV and pickle is more reliable & robust. Also, HDF5 basically implements a hierarchical file system, which is overengineering IMO.

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#22

Post by chubot »

jzwinck wrote: post #19

HDF5 is supported by many languages including C, C++, R, and Python. It has compression built in. It can read slices easily. It is battle-tested, stable, and used in production for many years by thousands of people. Pandas even has integrated support for DataFrames stored in HDF5. What's the advantage of Feather over HDF5? Couldn't the Feather libraries be written with the same API but HDF5 as the storage format, if…

HDF5 apparently only has one implementation, which is ridiculously bloated:

https://news.ycombinator.com/item?id=10858189

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#23

Post by nhatbui »

andreasvc wrote: post #21
jzwinck wrote: post #19

HDF5 is supported by many languages including C, C++, R, and Python. It has compression built in. It can read slices easily. It is battle-tested, stable, and used in production for many years by thousands of people. Pandas even has integrated support for DataFrames stored in HDF5. What's the advantage of Feather over HDF5? Couldn't the Feather libraries be written with the same API but HDF5 as the storage format, if…

I found it quite frustrating to use HDF5. It does not handle variable-length strings well (very common). In Pandas, categoricals and MultiIndex are not supported. I found that settling for CSV and pickle is more reliable & robust. Also, HDF5 basically implements a hierarchical file system, which is overengineering IMO.

Are you talking about a different type of MultiIndex? http://pandas.pydata.org/pandas-docs/stable/advanced.html

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#26

Post by oddthink »

eximius wrote: post #24

Ignorant question: What does this solve that a csv doesn't? Type information?

Type information is one. CSV is slow, since it has to parse everything on load. CSV has no random access, since rows can be arbitrary length. CSV takes up a lot of disk space, since a 8-byte double gets expanded into a 15+ digit string.

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#27

Post by deadmutex »

jzwinck wrote: post #19

HDF5 is supported by many languages including C, C++, R, and Python. It has compression built in. It can read slices easily. It is battle-tested, stable, and used in production for many years by thousands of people. Pandas even has integrated support for DataFrames stored in HDF5. What's the advantage of Feather over HDF5? Couldn't the Feather libraries be written with the same API but HDF5 as the storage format, if…

HDF5 is not thread-efficient. https://www.hdfgroup.org/hdf5-quest.html#tsafe

So, python packages like h5py do not even try to release the GIL.

This makes working with HDF5 very annoying in python (when using multiple threads).

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#28

Post by roywiggins »

oddthink wrote: post #26
eximius wrote: post #24

Ignorant question: What does this solve that a csv doesn't? Type information?

Type information is one. CSV is slow, since it has to parse everything on load. CSV has no random access, since rows can be arbitrary length. CSV takes up a lot of disk space, since a 8-byte double gets expanded into a 15+ digit string.

Also there's no single, official CSV standard.

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#29

Post by wesm »

jzwinck wrote: post #19

HDF5 is supported by many languages including C, C++, R, and Python. It has compression built in. It can read slices easily. It is battle-tested, stable, and used in production for many years by thousands of people. Pandas even has integrated support for DataFrames stored in HDF5. What's the advantage of Feather over HDF5? Couldn't the Feather libraries be written with the same API but HDF5 as the storage format, if…

Hi, Wes here.

HDF5 is a really great piece of software -- I wrote the first implementation of pandas's HDF5 integration (pandas.HDFStore) and Jeff Reback really went to town building out functionality and optimizing it for many different use cases.

But the HDF5 C libraries are very heavy dependency. Feather by comparison is an extremely small amount of code (There is also the Apache Arrow factor -- integration between the Arrow memory representation and R and Python tools will have a lot of ecosystem benefits, so one of the goals of Feather is to reconcile Python's and R's metadata requirements with the "official" Arrow metadata so that we can move around data frames with very low overhead.

Re: Feather: A Fast On-Disk Format for Data Frames for R and Python

#30

Post by wesm »

mziel wrote: post #8

I can see that Wes is a reporter for Spark issue ( https://issues.apache.org/jira/browse/SPARK-13534 ), what are the plans (if any) for tighter Spark/Python/R DataFrames integration?

I'd like to see SPARK-13534 completed so that we can bring Spark to Python/R data access performance reach a level we can deem "acceptable". I dug into this issue a bit here: http://wesmckinney.com/blog/pandas-and-apache-arrow/
Post reply on HN