Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scikit-Learn Version 1.0

scikit-learn.org

Post reply on HN
Print view View original

91–100 of 106 posts

Re: Scikit-Learn Version 1.0

#91

Post by lr1970 »

nxpnsv wrote: post #86
lr1970 wrote: post #73

Early on, pandas made some unfortunate design decisions that are still biting hard. For example, the choice of datetime (pandas.Timestamp) represented by a 64-bit int with a fixed nanosecond resolution. This choice gives dynamic range of +- 292 years around 1970-01-01 (the epoch). This range is too small to represent the works of William Shakespeare, never mind human history. Using pandas in these areas becomes a roy…

Most data is not 300 years old or in the distance future, in fact ranges 1970+-292 years are very common. That is to say, panda's choice is good for lots of people, including outside high-frequency stock traders.

> Most data is not 300 years old or in the distance future, in fact ranges 1970+-292 years are very common.

In what domains? Astronomy, geology, history call for larger time range. Laser and High Energy physics need femtosecond rather than nanosecond resolution. My point is that a fixed time resolution, whatever it is, is a bad choice. Numpy explicitly allows to select time resolution unit and this is the right approach. BTW, numpy is pandas dependency and predates it by several years.

Re: Scikit-Learn Version 1.0

#92

Post by [deleted] »

minsc__and__boo wrote: post #83
lr1970 wrote: post #75

Earlier quoted context omitted.

An illustrative example of how reasonable short-term and narrow-scope considerations can be really bad in long-term and/or at a larger scope.

This assumes that all projects should be built with the larger scope in mind. Sometimes you just need a shovel, not a Bagger 288.

[deleted]

Re: Scikit-Learn Version 1.0

#93

Post by anigbrowl »

lr1970 wrote: post #80
nojito wrote: post #76

Earlier quoted context omitted.

Why should he care about other use-cases? It’s not his responsibility to make sure his package is as wide as possible before opensourcing.

The problem is not with the Wes' original decision but with the fact that it was never revisited even when pandas took off at much larger scope. Should had been fixed before 1.0 release.

I'm glad you posted about this because I didn't know, but my reflexive response was 'well guess that won't work for [project idea], guess I'll roll my own or just use the NumPy version.'

I personally don't mind the lack of one-size-fits-all. If Pandas were to be part of the Python Standard Library I think you'd have a stronger argument, since the unspoken premise of a SL is that you can leave for a desert island with only that and your IDE and still get things done.

Re: Scikit-Learn Version 1.0

#94

Post by [deleted] »

baron_harkonnen wrote: post #68
boringg wrote: post #32

Earlier quoted context omitted.

Wait how many companies are actually using R in the wild? As I understand it, R is born of academia, great for statistics/analysis but breaks down on data manipulation and isn't used in production/data engineering. Maybe my understanding is dated though?

Of the many companies I've done data science with I can only think of a few, rare exceptions where R wasn't used as much as if not more than Python. If you're mostly dealing with Neural Nets you won't see much R, but for anything really statistical in nature R is a much better tool than Python. For anything that ends up in a report R is much better than Python (a lot of very valuable data science work ends up being a…

[deleted]

Re: Scikit-Learn Version 1.0

#95

Post by bttger »

baron_harkonnen wrote: post #63
disgruntledphd2 wrote: post #19

Earlier quoted context omitted.

I'm with you on sklearn, the DL libraries and Numpy, but Pandas and Matplotlib are poor, poor relations of the tools available in the R ecosystem (dplyr/ggplot etc).

I used to very strongly agree with you re: matplotlib, but I've recently switched from using almost exclusively ggplot2 to almost exlusively Matplotlib and my realization is that they are very different tools serving very different purposes. ggplot2 is obviously fantastic and makes beautiful plots, and very easily at that. However it is definitely a "convention over configuration" tool. For 99% of the typical plot yo…

> Fully agree on Pandas. R's native data frame + tidyverse is world's easier. Pandas' overly complex indexing system is a persistent source of annoyance no matter how much I use that library.

Is it just the syntax/readability that annoys you, or are there actually problems that need like n steps more to do the same with Pandas?

Re: Scikit-Learn Version 1.0

#96

Post by kzrdude »

infimum wrote: post #21

scikit-learn (next to numpy) is the one library I use in every single project at work. Every time I consider switching away from python I am faced with the fact that I'd lose access to this workhorse of a library. Of course it's not all sunshine and rainbows - I had my fair share of rummaging through its internals - but its API design is a de-facto standard for a reason. My only recurring gripe is that the serializat…

What's a typical task you do with sklearn? Just trying to get inspired about what it can do

Re: Scikit-Learn Version 1.0

#97

Post by crimsoneer »

laichzeit0 wrote: post #82

Great that they finally added quantile regression. This was sorely missed. I’m still hoping for a mixed-effects model implementation someday, like lme4 in R. The statsmodels implementation can only do predictions on fixed effects, which limits it greatly. I’ve always wondered why mixed effect type models are not more popular in the ML world.

Preach. The statsmodels implement sucks.

Re: Scikit-Learn Version 1.0

#98

Post by chaps »

bttger wrote: post #95
baron_harkonnen wrote: post #63

Earlier quoted context omitted.

I used to very strongly agree with you re: matplotlib, but I've recently switched from using almost exclusively ggplot2 to almost exlusively Matplotlib and my realization is that they are very different tools serving very different purposes. ggplot2 is obviously fantastic and makes beautiful plots, and very easily at that. However it is definitely a "convention over configuration" tool. For 99% of the typical plot yo…

> Fully agree on Pandas. R's native data frame + tidyverse is world's easier. Pandas' overly complex indexing system is a persistent source of annoyance no matter how much I use that library. Is it just the syntax/readability that annoys you, or are there actually problems that need like n steps more to do the same with Pandas?

I spend more time working around panda's strange isms than it takes me to write vanilla python that does the same thing. The index problems are not just a small annoyances, and sometimes can waste hours because of its awkward defaults. For example, its default in df.to_csv to write an index (without a column name..)! It doesn't make any sense to me whatsoever that reading a csv, then writing the csv would add a new column. I'm really tired of rerunning pandas code after I forget to turn that stupid default index setting off. Is that a small thing? Sure. But it had tons of small things like that.

Re: Scikit-Learn Version 1.0

#99

Post by _Wintermute »

chaps wrote: post #98
bttger wrote: post #95

Earlier quoted context omitted.

> Fully agree on Pandas. R's native data frame + tidyverse is world's easier. Pandas' overly complex indexing system is a persistent source of annoyance no matter how much I use that library. Is it just the syntax/readability that annoys you, or are there actually problems that need like n steps more to do the same with Pandas?

I spend more time working around panda's strange isms than it takes me to write vanilla python that does the same thing. The index problems are not just a small annoyances, and sometimes can waste hours because of its awkward defaults. For example, its default in df.to_csv to write an index (without a column name..)! It doesn't make any sense to me whatsoever that reading a csv, then writing the csv would add a new c…

It's funny you complain about the index being saved in csv files, which is the default behaviour in R.

Re: Scikit-Learn Version 1.0

#100

Post by abacadaba »

dachryn wrote: post #47
boringg wrote: post #32

Earlier quoted context omitted.

Wait how many companies are actually using R in the wild? As I understand it, R is born of academia, great for statistics/analysis but breaks down on data manipulation and isn't used in production/data engineering. Maybe my understanding is dated though?

R is everywhere, especially when you need to visualize stuff. It is primarily used in teams who are trying to get rid of SAS in my experience. You are right in the sense that R is typically not used end-to-end as far as I can tell, but already tries to start with a data connection to some sort of dump or datalake, or datawarehouse. Many people in my team use Python for modelling, but grab ggplot in whatever way to ma…

Ah SAS, the one language that makes R look good.
Post reply on HN