Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Polars: Fast DataFrame library for Rust and Python

pola.rs

Post reply on HN
Print view View original

101–110 of 131 posts

Re: Polars: Fast DataFrame library for Rust and Python

#101

Post by wodenokoto »

the_biot wrote: post #91

I've never seen the term "dataframe" used as it is on this webste, and the commenters here seem to all use it. Judging by the examples it seems to just refer to a "row" from e.g. a CSV or SQL query. So is that all it is, or am I missing something?

A data frame is one of the basic, built-in data structures in R, which was released in 1993. And R was based on an even older S.

So it’s not a new thing.

If you don’t work in computational statistics / data science it might not be a well known term, though.

Re: Polars: Fast DataFrame library for Rust and Python

#102

Post by civilized »

RcrdBrt wrote: post #62
vatican_banker wrote: post #56

Earlier quoted context omitted.

In what way data.table trumps dplyr? Genuinely interested in knowing. While data.table is faster than dplyr, data manipulations with data.table are difficult to read/understand/maintain. dplyr also grew into a full-fledge list of libraries to work on data-related projects (the tidyverse). These libraries are _very_ well thought out and enables productivity with minimal learning curve [anecdotal]

Anecdotal data: I found that data.table ingestion speed with fread() trumps absolutely everything else

This observation is pretty widely shared.

Re: Polars: Fast DataFrame library for Rust and Python

#103

Post by wodenokoto »

unixhero wrote: post #71

What makes Pandas so bad and what makes Dplyr so great? I have used Pandas a lot for data analysis and for data integration duct tape scenarios. For me it has been a low bar for achieving a lot.

For some people pandas seems to click. Good for you. I always struggle with google and the manual to get even simple things done.

I can never figure out if I am gonna get a series or a data frame out of an operation. It seems to edit rows when I think it’ll edit columns and I constantly have to explicitly reset the index not to get into problems.

I think dplyr is easy to read and write. It does get longer than other alternatives, but the readability is imho so good at it doesn’t feel verbose.

Re: Polars: Fast DataFrame library for Rust and Python

#104

Post by otsaloma »

civilized wrote: post #13

In my world, anything that isn't "identical to R's dplyr API but faster" just isn't quite worth switching for. There's absolutely no contest: dplyr has the most productive API and that matters to me more than anything else. But I'm glad to see Polars moves away from the kludgey sprawl of the Pandas API towards the perfection of dplyr... while also being blazingly fast! Now just mix in a bit of DSL so people aren't ob…

Agreed, dplyr is great.

I built my own data frame implementation on top of NumPy specifically trying to accomplish a better API, similar to dplyr. It's not exactly the same naming or operations, but should feel familiar and much simpler and consistent than Pandas. And no indexes or axes.

Having done this, a couple notes on what will unavoidably differ in Python

* It probably makes more sense in Python to use classes, so method chaining instead of function piping. I wish one could syntactically skip enclosing parantheses in Python though, method chains look a bit verbose.

* Python doesn't have R's "non-standard evaluation", so you end up needing lambda functions for arguments in method chains and group-wise aggregation etc. I'd be interested if someone has a better solution.

* NumPy (and Pandas) is still missing a proper missing value (NA). It's a big pain to try to work around that.

https://github.com/otsaloma/dataiter

Re: Polars: Fast DataFrame library for Rust and Python

#106

Post by BiteCode_dev »

civilized wrote: post #13

In my world, anything that isn't "identical to R's dplyr API but faster" just isn't quite worth switching for. There's absolutely no contest: dplyr has the most productive API and that matters to me more than anything else. But I'm glad to see Polars moves away from the kludgey sprawl of the Pandas API towards the perfection of dplyr... while also being blazingly fast! Now just mix in a bit of DSL so people aren't ob…

You don't need to write "import pandas; pandas.bla()", you can do "from pandas import *; anything_in_pandas()" if you want quick and dirty.

Re: Polars: Fast DataFrame library for Rust and Python

#108

Post by adgjlsfhk1 »

rscho wrote: post #85
ritchie46 wrote: post #82

Earlier quoted context omitted.

Note that the compile times of julia are not included in the benchmarks. If you read the website, you'd seen that the grapsh show the first (excluding the compilation) and the second run (with hot cache). Also in the second run, julia is not the fastest. Julia would not be faster than Rust, its got a garbage collector. This is what you see in the join benchmarks that really push the allocator. Next to that, the datab…

> Julia would not be faster than Rust, its got a garbage collector. Having a garbage collector does not intrinsically make things slower. Especially so outside of the benchmarking microcosm.

that said, Julia currently has a slow GC so it does hurt. GC performance is being worked on though. I have high hopes for a year or 2.

Re: Polars: Fast DataFrame library for Rust and Python

#109

Post by otsaloma »

unixhero wrote: post #71

What makes Pandas so bad and what makes Dplyr so great? I have used Pandas a lot for data analysis and for data integration duct tape scenarios. For me it has been a low bar for achieving a lot.

If you use Pandas daily, maybe get used to it and can ignore the issues, but for anyone using Pandas occasionally, it's every time a huge pain trying to figure out how to use it. The API is not intuitive and the documentation is very verbose and unclear. And stackoverflow top answers are often the "old way" of doing something when yet another way of doing the same thing has been added to the API.

Re: Polars: Fast DataFrame library for Rust and Python

#110

Post by ritchie46 »

gpderetta wrote: post #97

From the python docs: > No Index > They are not needed. Not having them makes things easier. Convince me otherwise Agree completely. first class indices in pandas just complicate everything by having a specially blessed column that can't be manipulated consistently. Secondary indices should be "just" an optimization, while primary indices are a constraint on the whole table (not a single column). The library in gener…

> (as usual project is called select...)

Yeah.. this confusion is in the API as well (you can pass projection to IO readers). we used `select` because SQL. In the logical plan we make the correct distinction between selection and projection, but you don't see that very much in the API.

Post reply on HN