Both Wes and I (project authors) will be tracking this thread in case you have questions!
[deleted]
Feather: A Fast On-Disk Format for Data Frames for R and Python
61–70 of 78 posts
Re: Feather: A Fast On-Disk Format for Data Frames for R and Python
#62Earlier quoted context omitted.
Anything that needs to parse text into numbers will be slow. Your links just compare various slow methods and conclude that one isn't quite as slow as the others. Of course it all depends on what you call 'slow'. Reading a few hundreds of megabytes of megabytes of csv's isn't going to be 'really' slow on modern hardware even if it was fgetc'd character by character. Either way: anything that represents data as text w…
The original comment was, "CSV parsing is relatively slow." Relative to what? The links I provided show that, using fread from data.table in R, parsing CSV data can be quicker than returning the same data from, for example, an SQLite database. I believe the links do show CSV parsing relative to a couple of other common data parsing methods in R. Whether they're all "slow methods" is an entirely different question. If…
I guess in re-reading my comment, I phrased it in a tone that was more combative than I meant, so don't take the GP as an attack.
My point was, all conversions from text into numbers (I'm assuming here that the target use case is reading large amounts of numeric data) are slow and the slow part isn't the IO, but the conversion from text to numbers. In that light, it isn't a surprise that sqlite isn't much faster than csv, because sqlite doesn't have 'strong typing' itself. Any storage format that is concerned with speed will store data in binary format. But of course text formats are a lot easier to work with, and to interface between programs. I sometimes (when I have a lot of data that I know need multiple passes of reading) build quick and dirty 'caches' where I read file.csv and do what is basically a memory dump of the parsed data into file.csv.bin. My read functions can then check if that file exists and skip the parsing step. In my experience, this can be easily an order of magnitude (10x) faster. It's not portable or even elegant, of course.
Apart from that - when speed is of importance, one wouldn't use R in the first place, of course (I say that as someone who likes R for what it is and for the things it does well).
I don't do write ups of speed analyses, nor do I know of any, so I have to cop out on that one.
Re: Feather: A Fast On-Disk Format for Data Frames for R and Python
#63Earlier quoted context omitted.
It thwarts reproducibility. By saving your workspace, it drags a lot of state from session to session that isn't accounted for. If you share code with someone else, their workspace space won't be the same, and thus the code may not function the same.
Point taken. But we are again delving dangerously close to thou-shall-not . from my perspective, it is a quick and convenient way to save all the data frames in my code. It's a boon for productivity. If not this, then I pray for Feather to be able to save multiple data frames innone file.
I don't see it as a thou-shalt-not. As a file format, feather is is lightweight. If they turned it into a container format it would be expanding the scope. If they instrumented it to comb objects in the global namespace and serialize them to the new container format, it would be heavier still--all to support a feature that the authors view as an anti-pattern. That's less a thou shalt not than it is a prioritization of their own vision.
If you're looking for a container to store lots of tabular data in one file, I'd suggest SQLite. Using dplyr, you can save those dataframes very easily. Plus, you can join tables and perform efficient aggregations on datasets too large to keep in memory.
In a lot of ways, I don't understand what limitations prevent SQLite from becoming the defacto common data.frame format. There probably are some, I just don't understand the tradeoffs (especially given how much SQLite gives you for free)!
Re: Feather: A Fast On-Disk Format for Data Frames for R and Python
#64Earlier quoted context omitted.
Point taken. But we are again delving dangerously close to thou-shall-not . from my perspective, it is a quick and convenient way to save all the data frames in my code. It's a boon for productivity. If not this, then I pray for Feather to be able to save multiple data frames innone file.
I don't see it as a thou-shalt-not. As a file format, feather is is lightweight. If they turned it into a container format it would be expanding the scope. If they instrumented it to comb objects in the global namespace and serialize them to the new container format, it would be heavier still--all to support a feature that the authors view as an anti-pattern. That's less a thou shalt not than it is a prioritization o…
Actually this is interesting - whhy Feather vs sqlite. I would love to know the answer!
But coming back to the anti-pattern : well, obviously the authors have the power to not spend time on something. But I'm trying to figure out why it's an anti-pattern in general. Snapshotting execution state is probably the ideal goal, but saving intermediate data structures is a decent convenience feature.
Now if that's restricted by the limitations of the format itself (no multiple frames in a single file), then we are back to thinking that HDF5/sqlite may indeed be the better format.
Re: Feather: A Fast On-Disk Format for Data Frames for R and Python
#65This looks amazing! Hadley, Wes, what are your thoughts on how to implement compression? I recall some open source columnar datastores (e.g. infobright) that achieved very VERY fast compression rates with just a few tricks: https://news.ycombinator.com/item?id=8354416 In particular, compression is extremely fast for columnar datastores (its the same type one after the other). Since a lot of times the data is sorted b…
Compression is on the long term to do list - Wes knows a lot more about it than me.
Re: Feather: A Fast On-Disk Format for Data Frames for R and Python
#66Earlier quoted context omitted.
I don't see it as a thou-shalt-not. As a file format, feather is is lightweight. If they turned it into a container format it would be expanding the scope. If they instrumented it to comb objects in the global namespace and serialize them to the new container format, it would be heavier still--all to support a feature that the authors view as an anti-pattern. That's less a thou shalt not than it is a prioritization o…
Actually this is interesting - whhy Feather vs sqlite. I would love to know the answer! But coming back to the anti-pattern : well, obviously the authors have the power to not spend time on something. But I'm trying to figure out why it's an anti-pattern in general. Snapshotting execution state is probably the ideal goal, but saving intermediate data structures is a decent convenience feature. Now if that's restricte…
Basically because you should be encoding state in code, not data. If you store data between sessions, it's easy to lose the code that you use to create it and then later on you can't recreate it.
It is convenient to save your complete workspace but I've seen too many cases where it's contributed to lack of reproducibility to spend my time working on it.
Re: Feather: A Fast On-Disk Format for Data Frames for R and Python
#67Earlier quoted context omitted.
Hi, Wes here. HDF5 is a really great piece of software -- I wrote the first implementation of pandas's HDF5 integration (pandas.HDFStore) and Jeff Reback really went to town building out functionality and optimizing it for many different use cases. But the HDF5 C libraries are very heavy dependency. Feather by comparison is an extremely small amount of code ( There is also the Apache Arrow factor -- integration betwe…
How does this contrast with the new Dask library in Python?
Dask is a compute framework. So you could use dask to create lots of Feather files, then perform computations.
Re: Feather: A Fast On-Disk Format for Data Frames for R and Python
#68Any plans to support a pipe-aware POSIX command, or get this into PostgreSQL?
Not by us, but we expect that many other projects will add feather support now that it's used by both R and Python.
Re: Feather: A Fast On-Disk Format for Data Frames for R and Python
#69HDF5 is supported by many languages including C, C++, R, and Python. It has compression built in. It can read slices easily. It is battle-tested, stable, and used in production for many years by thousands of people. Pandas even has integrated support for DataFrames stored in HDF5. What's the advantage of Feather over HDF5? Couldn't the Feather libraries be written with the same API but HDF5 as the storage format, if…
HDF5 is a clunky file format and dependency. There's a whole host of usual complaints, many of which have already been mentioned: http://cyrille.rossant.net/moving-away-hdf5/ My biggest personal annoyance is that HDF5 isn't thread safe^, so it only supports parallel reading and writing via multiple processes. This makes parallel computing a pain. This is especially annoying when using HDF5's built-in compression, whi…
Concurrent writes would not be possible but parallel reads definitely are. Haven't put any multithreading work into the library yet but it shouldn't be a large task.
Re: Feather: A Fast On-Disk Format for Data Frames for R and Python
#70Earlier quoted context omitted.
Actually this is interesting - whhy Feather vs sqlite. I would love to know the answer! But coming back to the anti-pattern : well, obviously the authors have the power to not spend time on something. But I'm trying to figure out why it's an anti-pattern in general. Snapshotting execution state is probably the ideal goal, but saving intermediate data structures is a decent convenience feature. Now if that's restricte…
Basically because you should be encoding state in code, not data. If you store data between sessions, it's easy to lose the code that you use to create it and then later on you can't recreate it. It is convenient to save your complete workspace but I've seen too many cases where it's contributed to lack of reproducibility to spend my time working on it.
So there's a use case difference. I create models from remote data sources - this is incremental on a daily basis and takes quite a bit of time.
So I snapshot the workspace after I do a run and do some experiments. Now - for me, saving the workspace is a convenience feature, NOT a programming feature.
This is what I mean by thou-shall-not. My use case is very well defined and I'm not stupid. And I completely knows the pitfalls of what you talk about - but a philosophical opposition is what hurts me (and lots of devs like me)