Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I wrote one of the fastest DataFrame libraries

ritchievink.com

Post reply on HN
Print view View original

21–30 of 146 posts

Re: I wrote one of the fastest DataFrame libraries

#21

Post by mhh__ »

Probably worth pointing out that the sections on microarchitecture stopped being representative with the pentium pro in the mid 90s. The processor still has a pipeline, but it's much harder to stall it like that.

http://www.lighterra.com/papers/modernmicroprocessors/ is a good guide (Agner Fog's reference isn't really a book so I don't recommend it for the uninitiated)

Re: I wrote one of the fastest DataFrame libraries

#23

Post by mhh__ »

The_rationalist wrote: post #10
Keyframe wrote: post #7

Arrow and SIMD, how would it work on Arm? I've had quite a success with Gravitons recently.

What is your question? ARM supports SIMD through NEON

You can implement the intel intrinsics on the arm end, but NEON and AVX aren't exactly the same thing, so there's usually performance to be found.

I'm also not aware of any free SIMD profilers that work on ARM that hold a candle to vTune.

Re: I wrote one of the fastest DataFrame libraries

#27

Post by timClicks »

natemcintosh wrote: post #24

I'm guessing Polars and Ballista ( https://github.com/ballista-compute/ballista ) have different goals, but I don't know enough about either to say what those might be. Does anyone know enough about either to explain the differences?

Ballista is distributed. Its author, Andy Grove is the author of the Rust implementation of Arrow though so there will be similarities between the two projects.

Re: I wrote one of the fastest DataFrame libraries

#28

Post by rcthompson »

orhmeh09 wrote: post #13
danielecook wrote: post #11

Pretty impressed with the data.table benchmarks. The syntax is a little weird and takes getting used to but once you have the basics it’s a great tool.

I use it a lot but it really breaks the tidyverse, which makes using R actually enjoyable. Why aren’t these other libraries (not in R; I’m talking the others in the benchmark) consistently as fast as data.table? Are the programmers of data.table just that much better?

dplyr and related packages use the existing R data frame class. (A "tibble" is just a regular R data frame under the hood.) This means that it inherits all the performance characteristics of regular R data frames. data.table is a completely separate implementation of a data structure that is functionally similar to a data frame but designed from the ground up for efficiency, though with some compromises, such as eschewing R's typical copy-on-modify paradigm. There are other more subtle reasons for the differences, but that's the absolute simplest explanation.

Supposedly you can use data.tables with dplyr, but I haven't experimented with it in depth.

Re: I wrote one of the fastest DataFrame libraries

#29

Post by sgtnoodle »

nijave wrote: post #3

Not sure if anything exists but I wish something would do in memory compression + smart disk spillover. Sometimes I want to work with 5-10GB compressed data sets (usually log files) and decompressed that ends up being 10x (plus add data structure overhead). There's stuff like Apache Drill but it's more optimized for multi node than running locally

How is that different from, say, opening up a gzipped file with a reader object in python?
Post reply on HN