I wrote one of the fastest DataFrame libraries • Hacker News Board

Re: I wrote one of the fastest DataFrame libraries

#131

Post by rcthompson » Sun, Mar 14, 2021, 9:54 PM UTC

kkoncevicius wrote: post #130

rcthompson wrote: post #129

Earlier quoted context omitted.

Thank you for the correction. I knew that tibbles were essentially just data frames with an extra class attribute, but for some reason I didn't realize this was also true of data.table. I think assumed that data.table's reference semantics couldn't be implemented on top of the existing data frame class, but I guess I'm wrong about that. Unfortunately it's too late for me to edit my original comment.

Tibbles are not just data frames with extra class attribute. For one - they don't have row names. Second, consider this example, demonstrating how treating tibbles as data frames can be dangerous: df_iris 3 nunique(tb_iris, "Species") > 1 R-devel mailing list had a long discussion about this too: https://stat.ethz.ch/pipermail/r-package-devel/2017q3/001896...

Ok, fine, to be more precise, tibbles and data frames and data tables are all implemented as R lists whose elements are vectors which form the columns of the table. And also `is.data.frame` currently returns TRUE for all of them, whether or not that is ultimately correct.

Re: I wrote one of the fastest DataFrame libraries

#132

Post by celrod » Sun, Mar 14, 2021, 11:52 PM UTC

tkoolen wrote: post #73

SatvikBeri wrote: post #61

Earlier quoted context omitted.

What specifically are you thinking of? The non-const global variables stand out to me, but I'm not experienced enough tell whether that would make a large difference.

Non-const globals could be an issue, but it's possible it doesn't matter too much for this particular benchmark. I'm a little worried about taking compilation time (apart from precompilation) into account (would that also be done for C++ code?). But I must confess I maybe posted my comment a bit too soon, partially because of the time of day, partially because of the semicolons at the end of each line in the code, wh…

I often end every line with a semicolon, so that it doesn't flood a REPL if I run it there.

IIRC, groupby hasn't been optimized in DataFrames.jl yet.

Re: I wrote one of the fastest DataFrame libraries

#133

Post by orhmeh09 » Mon, Mar 15, 2021, 1:18 AM UTC

lordgroff wrote: post #112

orhmeh09 wrote: post #49

Earlier quoted context omitted.

There are almost 200 magrittr-related issues in GitHub and I have had a bad time pairing data.table with tidyverse packages (and others because of e.g. IDate). DT code is like line noise to me, but I prefer to write things in it directly — the only reason I use it is because it’s fast, and guessing how it’s going to interact with tidy stuff and NSE (especially when using in place methods) is counterproductive to that…

19 of those are open and most of them not terribly relevant. Considering the ubiquity of the package, I'd say the total number of issues is shockingly low. As for NSE, DT uses NSE as well, but differently of course. I guess it all comes to what we "mean" by tidyverse. If we mean integration with the cast majority of packages, then yeah, it will work, but of course certain things are out of bounds. If you just want to…

I counted closed issues intentionally — this isn’t some one-off matter that’s easily resolved, as clearly hundreds of people have struggled with these issues over the years, and this should not be dismissed.

It’s far better aesthetically than Python. It’s just too different from the other libraries I use to disrupt my cognitive flow. You might say there are too many ways to do something, too, which makes it that much harder to figure out what code written by someone else (or myself three months ago) does. I also severely dislike seeing calls to eval or unevaluated code within the main body of my program —- quoted code looks awful and I trust it less.

It’d be interesting to see DT repackaged as its own tool with its own syntax. As it stands, it’s constrained by R and it has no comparable ecosystem to the tidyverse around it.

Re: I wrote one of the fastest DataFrame libraries

#134

Post by nijave » Mon, Mar 15, 2021, 3:14 AM UTC

Keyframe wrote: post #5

nijave wrote: post #3

Not sure if anything exists but I wish something would do in memory compression + smart disk spillover. Sometimes I want to work with 5-10GB compressed data sets (usually log files) and decompressed that ends up being 10x (plus add data structure overhead). There's stuff like Apache Drill but it's more optimized for multi node than running locally

gzip, zcat, parallel... That sort of thing?

Yeah but starts to fall apart for anything beyond basic search/filtering (like grouping, computed fields)

Re: I wrote one of the fastest DataFrame libraries

#135

Post by nijave » Mon, Mar 15, 2021, 3:17 AM UTC

hermitcrab wrote: post #90

FridgeSeal wrote: post #35

Earlier quoted context omitted.

When the library attempts to load something from disk that doesn’t fit into memory, it’s transparently, and (usually) without extra intervention from the user, swaps to memory-mapping and chunking through the file(s). Particularly useful for when you’ve got a bunch of data that doesn’t fit in memory, but setting up a whole cluster is not worth the overhead (operationally or otherwise) and/or if you’ve already got a p…

How is that different to virtual memory?

In memory representation also tends to have some data structure overhead so 5GB compressed -> 50GB uncompressed -> 100-250GB as a data structure (seems 2-5x is pretty normal). It starts out seeming pretty innocuous but quickly explodes. In addition, some things like Drill can do some automatic indexing/metadata recording which can reduce the amount of data it needs to access

Re: I wrote one of the fastest DataFrame libraries

#136

Post by sterlinm » Mon, Mar 15, 2021, 4:16 AM UTC

sandGorgon wrote: post #65

"Polars is based on the Rust native implementation Apache Arrow. Arrow can be seen as middleware software for DBMS, query engines and DataFrame libraries. Arrow provides very cache-coherent data structures and proper missing data handling." This is super cool. Anyone know if Pandas is also planning to adopt Arrow ?

I could be misremembering but I seem to recall Wes McKinney saying in some talk that rather than rewrite Pandas to be Arrow-backed it will probably eventually be replaced by newer Arrow-backed libraries some of which might have pandas-like apis. I think the idea was that pandas API is too large and the library too widely used for it to be practical to correct some of the design problems people have mentioned. He'd sketched out a vision for Pandas 2.0 at one point and I think he said that basically that would probably just be a new library.

There's a lot of related discussion in this post on his blog.

https://wesmckinney.com/blog/apache-arrow-pandas-internals/

Re: I wrote one of the fastest DataFrame libraries

#137

Post by DangitBobby » Mon, Mar 15, 2021, 5:51 AM UTC

em500 wrote: post #63

DangitBobby wrote: post #59

If this will read a csv that has columns with mixed integers and nulls without converting all of the numbers to float by default, it will replace pandas in my life. 99% of my problems with pandas arise from ints being coerced into floats when a bull shows up.

Pass dtype = {"colX":"Int64"} for the columns that you want to read as a Nullable integer type: https://pandas.pydata.org/pandas-docs/stable/user_guide/inte...

The problem is not that it can't be done, it's that I'll read one dataset and write the script that behaves as expected (using `head` here and there to check things as out the script progresses), then come back to it later after I get a new dataset, that now has nulls mixed with numbers. It starts behaving differently or is broken in a subtle way, and it's not always obvious why. After lots of experience, I have learned to check for int mangling each time a new Dataframe is read or two Dataframes are merged together. It is enough of a frustration that I am willing to look for a viable alternative, because I think it's a bit absurd that Int64 isn't the default for columns that are clearly meant to integers mixed with nulls, or that I can't set a flag to tell it to stop int mangling.

Re: I wrote one of the fastest DataFrame libraries

#138

Post by em500 » Mon, Mar 15, 2021, 10:04 AM UTC

DangitBobby wrote: post #137

em500 wrote: post #63

Earlier quoted context omitted.

Pass dtype = {"colX":"Int64"} for the columns that you want to read as a Nullable integer type: https://pandas.pydata.org/pandas-docs/stable/user_guide/inte...

The problem is not that it can't be done, it's that I'll read one dataset and write the script that behaves as expected (using `head` here and there to check things as out the script progresses), then come back to it later after I get a new dataset, that now has nulls mixed with numbers. It starts behaving differently or is broken in a subtle way, and it's not always obvious why. After lots of experience, I have lear…

It's not absurd that Int64 isn't the default, because:

1. nullable Int64 was only implemented recently, still experimental, and changing defaults can break lots of existing code

2. implementing nullable Int64 was a very non-trivial exercise, because pandas was mostly built on top of numpy which didn't (and still doesn't) have nullable integer arrays

Re: I wrote one of the fastest DataFrame libraries

#139

Post by mb7733 » Mon, Mar 15, 2021, 1:57 PM UTC

qeternity wrote: post #87

ohazi wrote: post #76

Earlier quoted context omitted.

I think it's mostly a nod to the fact that R's data.table blows everybody else out of the water by such a ridiculously wide margin. It's like a factor of 2 faster than the next fastest... So if you're writing a dataframe library as a hobby project, it's far less demotivating to use "all the other implementations" as your basis for comparison, at least initially.

I think a hobby project written in a general purpose language being the second fastest dataframe library is a hell of an accomplishment.

Sure, but data.table also fits those criteria.

Re: I wrote one of the fastest DataFrame libraries

#140

Post by nijave » Tue, Mar 16, 2021, 2:25 PM UTC

f311a wrote: post #55

nijave wrote: post #3

Not sure if anything exists but I wish something would do in memory compression + smart disk spillover. Sometimes I want to work with 5-10GB compressed data sets (usually log files) and decompressed that ends up being 10x (plus add data structure overhead). There's stuff like Apache Drill but it's more optimized for multi node than running locally

I think ClickHouse does that.

I actually started looking at Clickhouse a couple weeks ago but got a bit side tracked trying to grok how distributed tables work. It looks promising but there's a bit of a learning curve (seems some of the performance also comes from its use of arrays but best I can tell my use case should just use regular tables)