Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scikit-Learn Version 1.0

scikit-learn.org

Post reply on HN
Print view View original

71–80 of 106 posts

Re: Scikit-Learn Version 1.0

#71

Post by baron_harkonnen »

lysecret wrote: post #34
disgruntledphd2 wrote: post #19

Earlier quoted context omitted.

I'm with you on sklearn, the DL libraries and Numpy, but Pandas and Matplotlib are poor, poor relations of the tools available in the R ecosystem (dplyr/ggplot etc).

Hehe used to do R IMO you are right about ggplot but I strongly disagree about pandas. I fing love it. Would love to understand you troubles with it though, after using it for 4 years daily mabye I can offer some perspective ;)

Pandas indexing system is overly complex and I've never personally benefited from that. To start with there are __getitem__, loc and iloc approaches to accessing values. If your library constantly has to warn users that "you might being something wrong, read the docs!" that should be a warning sign that you don't have the correct level of abstraction. R has a much more sane api and assumptions about when you want to access a value by reference (which is almost always) and by value.

Then when doing basic operations like "group by" you end up excessively elaborate indexes that are in my experience useless and always need to be manually squashed to something coherent.

It's a common joke for me that whenever even a seasoned Pandas user cries out "gaarrr! why isn't this working!?" I just reply "have you tried reset_index?"... this works in a frighteningly large number of cases.

Re: Scikit-Learn Version 1.0

#72

Post by baron_harkonnen »

pantsforbirds wrote: post #50
disgruntledphd2 wrote: post #19

Earlier quoted context omitted.

I'm with you on sklearn, the DL libraries and Numpy, but Pandas and Matplotlib are poor, poor relations of the tools available in the R ecosystem (dplyr/ggplot etc).

I'm surprised you dont like pandas. I've found it to be a pretty easy to use and useful tool and you can almost always use something like DASK (or if youre lucky CUDF from rapidsai) if you need better performance. I will say that my very first "real" programming experience was Matlab at a research internship, so maybe i just got used to working in vectors and arrays for computational tasks.

> i just got used to working in vectors and arrays for computational tasks.

Have you worked with R? R, like matlab, natively supports vector based operations. In fact, all values in R are vectors. Many of the problems with Pandas ultimately boil down to the fact that you have to replicate this experience without truly being in a vector based language.

Re: Scikit-Learn Version 1.0

#73

Post by lr1970 »

Early on, pandas made some unfortunate design decisions that are still biting hard. For example, the choice of datetime (pandas.Timestamp) represented by a 64-bit int with a fixed nanosecond resolution. This choice gives dynamic range of +- 292 years around 1970-01-01 (the epoch). This range is too small to represent the works of William Shakespeare, never mind human history. Using pandas in these areas becomes a royal pain in the neck, for one constantly needs to work around pandas datetime limitations.

OTOH, in numpy one can choose time resolution units (anything from attosecond to a year) tailoring time resolution to your task (from high energy physics all way to astronomy). Panda's choice is only good for high-frequency stock traders, though.

Re: Scikit-Learn Version 1.0

#74

Post by Bostonian »

lr1970 wrote: post #73

Early on, pandas made some unfortunate design decisions that are still biting hard. For example, the choice of datetime (pandas.Timestamp) represented by a 64-bit int with a fixed nanosecond resolution. This choice gives dynamic range of +- 292 years around 1970-01-01 (the epoch). This range is too small to represent the works of William Shakespeare, never mind human history. Using pandas in these areas becomes a roy…

Pandas was started by a quant working for AQR Capital, so it's not surprising if "Panda's choice is only good for high-frequency stock traders".

Re: Scikit-Learn Version 1.0

#75

Post by lr1970 »

Bostonian wrote: post #74
lr1970 wrote: post #73

Early on, pandas made some unfortunate design decisions that are still biting hard. For example, the choice of datetime (pandas.Timestamp) represented by a 64-bit int with a fixed nanosecond resolution. This choice gives dynamic range of +- 292 years around 1970-01-01 (the epoch). This range is too small to represent the works of William Shakespeare, never mind human history. Using pandas in these areas becomes a roy…

Pandas was started by a quant working for AQR Capital, so it's not surprising if "Panda's choice is only good for high-frequency stock traders".

An illustrative example of how reasonable short-term and narrow-scope considerations can be really bad in long-term and/or at a larger scope.

Re: Scikit-Learn Version 1.0

#76

Post by nojito »

lr1970 wrote: post #75
Bostonian wrote: post #74

Earlier quoted context omitted.

Pandas was started by a quant working for AQR Capital, so it's not surprising if "Panda's choice is only good for high-frequency stock traders".

An illustrative example of how reasonable short-term and narrow-scope considerations can be really bad in long-term and/or at a larger scope.

Why should he care about other use-cases?

It’s not his responsibility to make sure his package is as wide as possible before opensourcing.

Re: Scikit-Learn Version 1.0

#77

Post by nojito »

boringg wrote: post #32
disgruntledphd2 wrote: post #19

Earlier quoted context omitted.

I'm with you on sklearn, the DL libraries and Numpy, but Pandas and Matplotlib are poor, poor relations of the tools available in the R ecosystem (dplyr/ggplot etc).

Wait how many companies are actually using R in the wild? As I understand it, R is born of academia, great for statistics/analysis but breaks down on data manipulation and isn't used in production/data engineering. Maybe my understanding is dated though?

R with data.table and collapse blow away the competition in terms of tabular data wrangling.

Both in terms of conciseness and performance

https://h2oai.github.io/db-benchmark/

Re: Scikit-Learn Version 1.0

#78

Post by civilized »

mrtranscendence wrote: post #70
MichaelRazum wrote: post #59

Earlier quoted context omitted.

Got it. Regarding fast you have something like Vaex on python side (but not sure how fast it realy is). For me I had with pandas the most issues using it's multiindex.

> For me I had with pandas the most issues using it's multiindex. Yessss. I loathe indices, and have never been in a situation where I was better off with them than without them. > Regarding fast you have something like Vaex on python sid I've never used Vaex, but I've used datatable ( https://github.com/h2oai/datatable ) and polars ( https://github.com/pola-rs/polars ). Polars is my favorite API, but datatable was f…

Pandas is the PHP of data science. Pretty badly designed, but immensely popular because it got there first and had no real competition (in Python) for years.

Re: Scikit-Learn Version 1.0

#79

Post by nojito »

MichaelRazum wrote: post #30
bllguo wrote: post #26

Earlier quoted context omitted.

maybe we are casualties of the vague definition of "data science," but in my experience numpy is too low-level for most of what I consider DS, and pandas/matplotlib are _much_ more central than sklearn or pytorch. Even if your definition only encompasses deep learning research, surely plotting is still indispensable? I'll also add my vote for the superiority of data.table and ggplot2 to any Python alternatives. the b…

Just curious. In which way is data.table superior to pandas? Really interested about it! From my personal experience pandas is just sometimes a bit slow.

Data.table is Faster to write and faster to perform

https://h2oai.github.io/db-benchmark/

Re: Scikit-Learn Version 1.0

#80

Post by lr1970 »

nojito wrote: post #76
lr1970 wrote: post #75

Earlier quoted context omitted.

An illustrative example of how reasonable short-term and narrow-scope considerations can be really bad in long-term and/or at a larger scope.

Why should he care about other use-cases? It’s not his responsibility to make sure his package is as wide as possible before opensourcing.

The problem is not with the Wes' original decision but with the fact that it was never revisited even when pandas took off at much larger scope. Should had been fixed before 1.0 release.
Post reply on HN