I'm surprised that data.table is so fast, and that pandas is so slow relative to it. It does explain why I've occasionally had memory issues on ~2GB data files when performing moderately complex functions. (to be fair, it's a relatively old Xeon w/ 12GB ram) I'll have to learn the nuances of data.table syntax now.
I wrote one of the fastest DataFrame libraries
51–60 of 146 posts
Re: I wrote one of the fastest DataFrame libraries
#52Earlier quoted context omitted.
Me too: I've tended to let the database do a lot of heavy lifting before I bring data in. Maybe I don't actually need to do that.
There’s really no harm in doing that, and it’s still a pretty good idea. I generally try and get my data sources as far as possible with the database, then leave framework/language specific things to the last step, means that-if nothing else-someone else picking up your dataset in a different language/framework toolset doesn’t need to pick up yours as a dependency, and you’re not spending time re-implementing what a…
The only downside to letting the database do some of the pre-processing is that I don't have a full raw data set to work with within either R or Python. If I decide I need a an existing measure aggregated up to a different level, or a new measure, I've got to go back to the database and then bring in an additional query. So I have less flexibility within the R or Python environment. But you make a good point: there's trade offs either way, and keeping the dataset as something like a materialized view on the database makes it a little more open to others' usage.
Re: I wrote one of the fastest DataFrame libraries
#53It seems like DataFrames.jl still has a ways to go before Julia can close the gap on R/data.table. I don't think these benchmarks include compilation time either?
I started using Julia in December, DataFrames are in a sort of weird place because they're so much less necessary compared to e.g. Python. In Julia, you could just use a dict of arrays and get most of the benefits, thanks to libraries like Query.jl and Tables.jl. Thus the ecosystem is a lot more spread out. I actually use DataFrames much less than I used to in Python. This is mostly good, because you can apply the sa…
Those arguments apply to Python as well. There is nothing special about Julia that warrants your arguments.
Re: I wrote one of the fastest DataFrame libraries
#54[note: see more nuanced comment below]
The Julia benchmark two links deep at https://github.com/h2oai/db-benchmark doesn't follow even the most basic performance tips listed at https://docs.julialang.org/en/v1/manual/performance-tips/.
Re: I wrote one of the fastest DataFrame libraries
#55Not sure if anything exists but I wish something would do in memory compression + smart disk spillover. Sometimes I want to work with 5-10GB compressed data sets (usually log files) and decompressed that ends up being 10x (plus add data structure overhead). There's stuff like Apache Drill but it's more optimized for multi node than running locally
I think ClickHouse does that.
Re: I wrote one of the fastest DataFrame libraries
#56I'm surprised that data.table is so fast, and that pandas is so slow relative to it. It does explain why I've occasionally had memory issues on ~2GB data files when performing moderately complex functions. (to be fair, it's a relatively old Xeon w/ 12GB ram) I'll have to learn the nuances of data.table syntax now.
I'm convinced that data.table is wizardry.
For anyone who's turned off by dt[i, j, by=k], Andrew Brooks has a good set of examples at http://brooksandrew.github.io/simpleblog/articles/advanced-d.... Data Camp's Cheat Sheet is also a good resource https://s3.amazonaws.com/assets.datacamp.com/blog_assets/dat....
Re: I wrote one of the fastest DataFrame libraries
#57Earlier quoted context omitted.
I started using Julia in December, DataFrames are in a sort of weird place because they're so much less necessary compared to e.g. Python. In Julia, you could just use a dict of arrays and get most of the benefits, thanks to libraries like Query.jl and Tables.jl. Thus the ecosystem is a lot more spread out. I actually use DataFrames much less than I used to in Python. This is mostly good, because you can apply the sa…
Those arguments apply to Python as well. There is nothing special about Julia that warrants your arguments.
I've been using Python a lot longer than I've been using Julia, and this isn't really true. Python tends towards much larger packages where everything is bundled together, and there are fairly deep language-level reasons for that. Python doesn't have major alternatives to pandas the way Julia has half a dozen alternatives to DataFrames. There is nothing like Query.jl that applies to all table-like structures in Python.
In pandas, you'll see things like exponentially weighted moving averages, while DataFrames.jl is pretty much just the data structure.
The centralization of the Python ecosystem and extra attention that pandas has gotten has made it much better in several ways – for example, pandas's indexing makes filtering significantly faster. These optimizations might make it to DataFrames.jl eventually, but I doubt you'll ever see the same level of centralization.
Re: I wrote one of the fastest DataFrame libraries
#58I would love to learn the details of building a Python wrapper on Rust code like you did with pypolars.
I just did that this weekend with py03 (pyo3?). The github page has some working examples.
Re: I wrote one of the fastest DataFrame libraries
#59If this will read a csv that has columns with mixed integers and nulls without converting all of the numbers to float by default, it will replace pandas in my life. 99% of my problems with pandas arise from ints being coerced into floats when a bull shows up.
Re: I wrote one of the fastest DataFrame libraries
#60Earlier quoted context omitted.
If you're not afraid of exotic languages, I encourage you to have a look at the APL ecosystem, and especially J and it's integrated columnar store Jd. I have just embarked on an adventure to do just what you describe in... Racket. But it's nowhere to be seen yet. I'm an epidemiologist and I've been wanting to make my own tools for a while, now. It'll be interesting to see how far I can go with Racket, which already i…
Would you mind pointing me at a resource that explains how J/Jd handle it by comparison?
https://code.jsoftware.com/wiki/Jd/Overview
Jd only packs int vectors, though. So if you're hoping for string compression then I don't know of any free solution. Jd heavily leverages SIMD and mmap. Larger-than RAM columns can be easily processed by ftable. I use Jd for data wrangling before making models in R. Of course, the J language is not for the faint of heart but it's really well-suited to the task.