Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I wrote one of the fastest DataFrame libraries

ritchievink.com

Post reply on HN
Print view View original

31–40 of 146 posts

Re: I wrote one of the fastest DataFrame libraries

#31

Post by bobbylarrybobby »

nijave wrote: post #3

Not sure if anything exists but I wish something would do in memory compression + smart disk spillover. Sometimes I want to work with 5-10GB compressed data sets (usually log files) and decompressed that ends up being 10x (plus add data structure overhead). There's stuff like Apache Drill but it's more optimized for multi node than running locally

I wonder if Arrow (https://arrow.apache.org/faq/) would work for this:

"The Arrow IPC mechanism is based on the Arrow in-memory format, such that there is no translation necessary between the on-disk representation and the in-memory representation. Therefore, performing analytics on an Arrow IPC file can use memory-mapping, avoiding any deserialization cost and extra copies."

Re: I wrote one of the fastest DataFrame libraries

#32

Post by orhmeh09 »

rcthompson wrote: post #28
orhmeh09 wrote: post #13

Earlier quoted context omitted.

I use it a lot but it really breaks the tidyverse, which makes using R actually enjoyable. Why aren’t these other libraries (not in R; I’m talking the others in the benchmark) consistently as fast as data.table? Are the programmers of data.table just that much better?

dplyr and related packages use the existing R data frame class. (A "tibble" is just a regular R data frame under the hood.) This means that it inherits all the performance characteristics of regular R data frames. data.table is a completely separate implementation of a data structure that is functionally similar to a data frame but designed from the ground up for efficiency, though with some compromises, such as esch…

Oh, I know that, I use it daily and I’ve read some of its source code. I’m just astonished that the best-performing data frame library in the world is developed in R and it outperforms engines written with million/billion dollar companies behind it.

Re: I wrote one of the fastest DataFrame libraries

#33

Post by FridgeSeal »

whoevercares wrote: post #30

Now any potential this become next DataBricks?

That’s a bit like asking when Python/Pandas will become the next SaaS product.

This is analogous to Pandas, Databricks is a commercial offering of managed Spark, Ballista is a new project that a Rust/“modern??” Rewrite of Spark.

Re: I wrote one of the fastest DataFrame libraries

#34

Post by jpgvm »

mhh__ wrote: post #23
The_rationalist wrote: post #10

Earlier quoted context omitted.

What is your question? ARM supports SIMD through NEON

You can implement the intel intrinsics on the arm end, but NEON and AVX aren't exactly the same thing, so there's usually performance to be found. I'm also not aware of any free SIMD profilers that work on ARM that hold a candle to vTune.

No profilers really hold a candle to vTune is the problem in general. I love the new AMD chips but uProf isn't in the same class as vTune and that is sad. I'm certain with better tools the AMD chips could be demolishing Intel by an even greater margin.

Re: I wrote one of the fastest DataFrame libraries

#35

Post by FridgeSeal »

cinntaile wrote: post #6
nijave wrote: post #3

Not sure if anything exists but I wish something would do in memory compression + smart disk spillover. Sometimes I want to work with 5-10GB compressed data sets (usually log files) and decompressed that ends up being 10x (plus add data structure overhead). There's stuff like Apache Drill but it's more optimized for multi node than running locally

What do you mean with smart disk spillover?

When the library attempts to load something from disk that doesn’t fit into memory, it’s transparently, and (usually) without extra intervention from the user, swaps to memory-mapping and chunking through the file(s).

Particularly useful for when you’ve got a bunch of data that doesn’t fit in memory, but setting up a whole cluster is not worth the overhead (operationally or otherwise) and/or if you’ve already got a processing pipeline written in a language/framework and you can’t/don’t-want to go through rewriting it for something distributed.

Re: I wrote one of the fastest DataFrame libraries

#36

Post by FridgeSeal »

ineedasername wrote: post #18
danielecook wrote: post #11

Pretty impressed with the data.table benchmarks. The syntax is a little weird and takes getting used to but once you have the basics it’s a great tool.

Me too: I've tended to let the database do a lot of heavy lifting before I bring data in. Maybe I don't actually need to do that.

There’s really no harm in doing that, and it’s still a pretty good idea.

I generally try and get my data sources as far as possible with the database, then leave framework/language specific things to the last step, means that-if nothing else-someone else picking up your dataset in a different language/framework toolset doesn’t need to pick up yours as a dependency, and you’re not spending time re-implementing what a database can already do (and can do more portably).

Re: I wrote one of the fastest DataFrame libraries

#37

Post by mumblemumble »

staticautomatic wrote: post #8

What’s up with all the Dask benchmarks saying “internal error”? I expected at least some explanation in the post.

If you click through to the detailed benchmarks page (https://h2oai.github.io/db-benchmark/). A lot of them are that it's running out of memory, a few of them are features that haven't been implemented yet.

Inefficient use of memory is a problem I've seen with several projects that focus on scale out. All else being equal, they tend to use a lot more memory. This happens for various reasons, but a lot of it is the simple fact that all the mechanisms you need to support distributed computing, and make it reliable, add a lot of overhead.

Re: I wrote one of the fastest DataFrame libraries

#38

Post by ninjin »

clircle wrote: post #26

It seems like DataFrames.jl still has a ways to go before Julia can close the gap on R/data.table. I don't think these benchmarks include compilation time either?

Not that I am a heavy DataFrame user, but I have felt more at home with the comparatively light-weight TypeTables [1]. My understanding is that the rather complicated DataFrame ecosystem in Julia [2] mostly stems from whether tables should be immutable and/or typed. As far as I am aware there has not been any major push at the compiler level to speed up untyped code yet – although there should be plenty of room for improvements – which I suspect would benefit DataFrames greatly.

[1]: https://github.com/JuliaData/TypedTables.jl

[2]: https://typedtables.juliadata.org/stable/man/table/#datafram...

Re: I wrote one of the fastest DataFrame libraries

#39

Post by SatvikBeri »

clircle wrote: post #26

It seems like DataFrames.jl still has a ways to go before Julia can close the gap on R/data.table. I don't think these benchmarks include compilation time either?

I started using Julia in December, DataFrames are in a sort of weird place because they're so much less necessary compared to e.g. Python. In Julia, you could just use a dict of arrays and get most of the benefits, thanks to libraries like Query.jl and Tables.jl. Thus the ecosystem is a lot more spread out. I actually use DataFrames much less than I used to in Python.

This is mostly good, because you can apply the same operations on DataFrames, Streams, Time Series data, Differential Equations Results, etc., but it does mean that some of the specialized optimizations haven't made it into DataFrames.jl

Re: I wrote one of the fastest DataFrame libraries

#40

Post by jrumbut »

clircle wrote: post #17
orhmeh09 wrote: post #13

Earlier quoted context omitted.

I use it a lot but it really breaks the tidyverse, which makes using R actually enjoyable. Why aren’t these other libraries (not in R; I’m talking the others in the benchmark) consistently as fast as data.table? Are the programmers of data.table just that much better?

I dropped dplyr in favor of data.table and never looked back. https://github.com/eddelbuettel/gsir-te

Vanilla R got a bad name but once you understand the fundamentals it's quite good, fewer footguns than used to be there, and I find it easier to reason about than tidyverse.
Post reply on HN