Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I wrote one of the fastest DataFrame libraries

ritchievink.com

Post reply on HN
Print view View original

141–146 of 146 posts

Re: I wrote one of the fastest DataFrame libraries

#141

Post by nijave »

sgtnoodle wrote: post #29
nijave wrote: post #3

Not sure if anything exists but I wish something would do in memory compression + smart disk spillover. Sometimes I want to work with 5-10GB compressed data sets (usually log files) and decompressed that ends up being 10x (plus add data structure overhead). There's stuff like Apache Drill but it's more optimized for multi node than running locally

How is that different from, say, opening up a gzipped file with a reader object in python?

It's not but implementing intelligent data access each time can become pretty tedious (sure you can write libraries and tools, but I'm basically asking if those already exist)

For something like AWS CloudTrail logs, 5GB is 40k 100-130kb gzipped json files so hit single-core CPU bounds almost immediately (just reading/decompressing/json parsing off an SSD). CPU scale out model in Python is processes so now you're copying data between processes if you want to parallelize it so now you hit IPC bottlenecks just using the standard library multiprocessing/concurrent futures stuff

5GB compressed /probably/ won't fit in memory so now you have to deal with that, too unless you have a way to keep it compressed (which would come at the cost of additional CPU usage)

tldr; it's non-trivial to actually fully use the hardware

Re: I wrote one of the fastest DataFrame libraries

#142

Post by hodgesrm »

nijave wrote: post #140
f311a wrote: post #55

Earlier quoted context omitted.

I think ClickHouse does that.

I actually started looking at Clickhouse a couple weeks ago but got a bit side tracked trying to grok how distributed tables work. It looks promising but there's a bit of a learning curve (seems some of the performance also comes from its use of arrays but best I can tell my use case should just use regular tables)

ClickHouse performance is principally due to column storage, compression, and ability to parallelize processing. Arrays can improve performance in some specific cases but are more commonly used to help deal with semi-structured data or perform custom processing on values within groups.

If your data maps cleanly to tables, that's in fact the best case with the easiest options for performance enhancement.

Re: I wrote one of the fastest DataFrame libraries

#143

Post by sgtnoodle »

nijave wrote: post #141
sgtnoodle wrote: post #29

Earlier quoted context omitted.

How is that different from, say, opening up a gzipped file with a reader object in python?

It's not but implementing intelligent data access each time can become pretty tedious (sure you can write libraries and tools, but I'm basically asking if those already exist) For something like AWS CloudTrail logs, 5GB is 40k 100-130kb gzipped json files so hit single-core CPU bounds almost immediately (just reading/decompressing/json parsing off an SSD). CPU scale out model in Python is processes so now you're copy…

Stepping back a bit, if you're CPU bound in a single thread decompressing data off an SSD, does copying the compressed data into memory first actually buy you anything?

If you truly need the dataset fully loaded into memory for performance reasons, then it's presumably because of the need to do lots of random accesses, where the read latency would otherwise harm you. The tricky bit is the fact that it's generally hard to randomly access compressed streams of data. You need to compress the data in a way that makes random access possible, likely to the detriment of compression ratio. Unless you also use the same compression format to store the data on disk, then you're back to having to decompress (and recompress) the whole file sequentially anyway in order to build the data structure in RAM.

I've seen purpose-made compressed log formats that support efficient seeking. I've never seen them loaded into RAM in their raw compressed form, though. Generally they do have a corresponding library to make accessing the log data easy.

Re: I wrote one of the fastest DataFrame libraries

#144

Post by ColFrancis »

juancb wrote: post #101
ColFrancis wrote: post #97

Earlier quoted context omitted.

R is from ~2000, while pandas started in 2011. Is it possible that the lack of compute power had an effect on the required performance characteristics?

R is much older than 2000, it's from 1993.

Thank you, my brief research led to a list of versions that had R 1.0 as 2000, but it appears that v0 lasted a good many years. Pandas as well was in v0 for many years so it is the better comparison to use like-for-like.

Re: I wrote one of the fastest DataFrame libraries

#145

Post by DangitBobby »

em500 wrote: post #138
DangitBobby wrote: post #137

Earlier quoted context omitted.

The problem is not that it can't be done, it's that I'll read one dataset and write the script that behaves as expected (using `head` here and there to check things as out the script progresses), then come back to it later after I get a new dataset, that now has nulls mixed with numbers. It starts behaving differently or is broken in a subtle way, and it's not always obvious why. After lots of experience, I have lear…

It's not absurd that Int64 isn't the default, because: 1. nullable Int64 was only implemented recently, still experimental, and changing defaults can break lots of existing code 2. implementing nullable Int64 was a very non-trivial exercise, because pandas was mostly built on top of numpy which didn't (and still doesn't) have nullable integer arrays

I disagree that those things make it not absurd. The current behavior is a surprise when you discover it and continues to bite you long after. It shouldn't be changed to a default now; the current behavior should never have existed.

I understood the technical reasons since I've researched them myself. It does literally nothing to change the frustration or convince me not to look for an alternative.

Re: I wrote one of the fastest DataFrame libraries

#146

Post by gthompson1 »

Pandas does seem to be on the out if I'm being honest, and thats coming from someone who has invested heavily in it (backend for my project http://gluedata.io/). JMO

I would happily adopt Polars if the feature set is expansive enough.

Pandas is great because its so ubiquitous but I have always felt that it was slow (especially coming from R).

One thing that is weirdly terrible in pandas is data types. The coupling with numpy is awkward. Its so dependent on numpy and if pandas isn't moving fast numpy isn't moving at all. I'd be curious to see how Polars handles this. e.g. Null values, datatimes etc.

Post reply on HN