Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scikit-Learn Version 1.0

scikit-learn.org

Post reply on HN
Print view View original

51–60 of 106 posts

Re: Scikit-Learn Version 1.0

#51

Post by andreareina »

lysecret wrote: post #34
disgruntledphd2 wrote: post #19

Earlier quoted context omitted.

I'm with you on sklearn, the DL libraries and Numpy, but Pandas and Matplotlib are poor, poor relations of the tools available in the R ecosystem (dplyr/ggplot etc).

Hehe used to do R IMO you are right about ggplot but I strongly disagree about pandas. I fing love it. Would love to understand you troubles with it though, after using it for 4 years daily mabye I can offer some perspective ;)

I run into pandas edge cases all the time. pd.concat() failing on empty sequences (just let me specify a default for that case please); .squeeze() not letting me say, "squeeze down to a series but not a scalar"; .groupby().apply() returning different types depending on how many groups/rows per group there are... it's fine when you know exactly what you have but it's hard using it in a pipeline with that needs to be agnostic about whether there's zero, one, or many data (datums?).

Re: Scikit-Learn Version 1.0

#52

Post by mrtranscendence »

dachryn wrote: post #47
boringg wrote: post #32

Earlier quoted context omitted.

Wait how many companies are actually using R in the wild? As I understand it, R is born of academia, great for statistics/analysis but breaks down on data manipulation and isn't used in production/data engineering. Maybe my understanding is dated though?

R is everywhere, especially when you need to visualize stuff. It is primarily used in teams who are trying to get rid of SAS in my experience. You are right in the sense that R is typically not used end-to-end as far as I can tell, but already tries to start with a data connection to some sort of dump or datalake, or datawarehouse. Many people in my team use Python for modelling, but grab ggplot in whatever way to ma…

There are Python ports of ggplot (e.g. plotnine (https://github.com/has2k1/plotnine)), but agreed, Python is behind here. I'm not the best at data viz, but I can usually piece together a way to make ggplot do what I want it to do without that much trouble or looking at documentation.

Matplotlib, though ... that's a harder beast to internalize. I know it's possible to make high-quality matplotlib plots, but it's much harder for me. Like pandas, it's a library that I don't want to denigrate because I know people put lots of effort into it, but I can't lie -- I'm not a fan.

Re: Scikit-Learn Version 1.0

#53

Post by dagw »

boringg wrote: post #32
disgruntledphd2 wrote: post #19

Earlier quoted context omitted.

I'm with you on sklearn, the DL libraries and Numpy, but Pandas and Matplotlib are poor, poor relations of the tools available in the R ecosystem (dplyr/ggplot etc).

Wait how many companies are actually using R in the wild? As I understand it, R is born of academia, great for statistics/analysis but breaks down on data manipulation and isn't used in production/data engineering. Maybe my understanding is dated though?

Wait how many companies are actually using R in the wild?

Depends on your definition. While not very often 'deployed' in 'production'. I know lots places in all kinds of industries where people reach for R as soon as they have to look at some new data.

Re: Scikit-Learn Version 1.0

#54

Post by disgruntledphd2 »

mrtranscendence wrote: post #49
disgruntledphd2 wrote: post #37

Earlier quoted context omitted.

> Wait how many companies are actually using R in the wild? As I understand it, R is born of academia, great for statistics/analysis but breaks down on data manipulation and isn't used in production/data engineering. It depends, I've worked in some places where R was the core part of their data infrastructure. Data manipulation (of non text) is far, far better in R. Integrating with other systems can be tricky though…

It's unrelated to your main point, but: > Additionally, R is a very, very flexible language (like Python) I'd argue that R is much more flexible than Python syntactically. There's a reason that every attempt at recreating dplyr in Python ends in a bit of a mess (IMO) -- Python just doesn't allow the sort of metaprogramming you'd require for a really nice port. Something as simple as a general pipe operator can't be d…

> I'd argue that R is much more flexible than Python syntactically. There's a reason that every attempt at recreating dplyr in Python ends in a bit of a mess (IMO) -- Python just doesn't allow the sort of metaprogramming you'd require for a really nice port. Something as simple as a general pipe operator can't be defined in Python, to say nothing of how dplyr scopes column names within verbs.

Well that's just lazy evaluation of function arguments, which can't be done in Python. But if take a look at the Python data model, it does seem super, super flexible. You'll still need strings for column names in any dplyr port though, because of the function argument issue.

Like, both Python/R derive from the CLOS approach (Art of the Metaobject Protocol), but R retains a lot more of the lispy goodness (but Python's implementation is easier to use).

Re: Scikit-Learn Version 1.0

#55

Post by mrtranscendence »

Tarq0n wrote: post #46
boringg wrote: post #32

Earlier quoted context omitted.

Wait how many companies are actually using R in the wild? As I understand it, R is born of academia, great for statistics/analysis but breaks down on data manipulation and isn't used in production/data engineering. Maybe my understanding is dated though?

It's mostly the "in production" part that determines whether R is suitable for a business or not. It's much more complicated to avoid runtime errors or do proper testing in R, whereas it shines for interactive use, or generating reports. That said having used both the DSL's for plotting and data wrangling in the R package ecosystem are vastly superior to pandas and python plotting libraries. For modeling I actually l…

> It's much more complicated to avoid runtime errors or do proper testing in R

It's not that much harder. There's no pytest, but testthat works well enough. I've developed a few packages internally in R and wouldn't say it was that much harder to ensure correctness than for the corresponding Python packages. (We used to keep them in sync, before basically moving everything to Python.)

Re: Scikit-Learn Version 1.0

#56

Post by jfarina »

pantsforbirds wrote: post #50
disgruntledphd2 wrote: post #19

Earlier quoted context omitted.

I'm with you on sklearn, the DL libraries and Numpy, but Pandas and Matplotlib are poor, poor relations of the tools available in the R ecosystem (dplyr/ggplot etc).

I'm surprised you dont like pandas. I've found it to be a pretty easy to use and useful tool and you can almost always use something like DASK (or if youre lucky CUDF from rapidsai) if you need better performance. I will say that my very first "real" programming experience was Matlab at a research internship, so maybe i just got used to working in vectors and arrays for computational tasks.

Numpy has a better api than pandas if we're strictly talking about vectors and arrays.

Pandas indexing makes sense once you get it, but it does seem to require a lot more words than equivalent statements in R.

My primary language is python, but I have been picking up some R.

Re: Scikit-Learn Version 1.0

#57

Post by mrtranscendence »

mrtranscendence wrote: post #49
disgruntledphd2 wrote: post #37

Earlier quoted context omitted.

> Wait how many companies are actually using R in the wild? As I understand it, R is born of academia, great for statistics/analysis but breaks down on data manipulation and isn't used in production/data engineering. It depends, I've worked in some places where R was the core part of their data infrastructure. Data manipulation (of non text) is far, far better in R. Integrating with other systems can be tricky though…

It's unrelated to your main point, but: > Additionally, R is a very, very flexible language (like Python) I'd argue that R is much more flexible than Python syntactically. There's a reason that every attempt at recreating dplyr in Python ends in a bit of a mess (IMO) -- Python just doesn't allow the sort of metaprogramming you'd require for a really nice port. Something as simple as a general pipe operator can't be d…

(Replying to disgruntledphd2)

> Well that's just lazy evaluation of function arguments, which can't be done in Python.

"Just lazy evaluation"! :) It's a pretty big deal. This is three-fifths of the way to a macro system.

> But if take a look at the Python data model, it does seem super, super flexible.

Sure, you can have a lot of control over the behavior of Python objects (some techniques of which remain obscure to me even after using Python for many years). But you don't have anything like syntactic macros. You can define a pipe operator with macropy, though -- it's pretty easy. But macropy is basically dead now I think (and a total hack).

> You'll still need strings for column names in any dplyr port though, because of the function argument issue.

This is major, though, because you can't do this:

    mutate(df, x="y" + "z")
You have to do something like what dfply does, defining an object that defines addition, subtraction, etc.

    mutate(df, x=X.y + X.z)
But that hits corner cases quickly. What if you want to call a regular Python function that expects numeric arguments? This won't work:

    mutate(df, x=f(X.y))
etc. Granted, this only really works in R because it's easy to define functions that accept and return vectors. So in that sense it's kind of a leaky abstraction. But you couldn't even get that far in Python, because X.y isn't a vector ... it's a kind of promise to substitute a vector.

Give Python macros, I say! To hell with the consequences!

Re: Scikit-Learn Version 1.0

#58

Post by dragonwriter »

mrtranscendence wrote: post #57
mrtranscendence wrote: post #49

Earlier quoted context omitted.

It's unrelated to your main point, but: > Additionally, R is a very, very flexible language (like Python) I'd argue that R is much more flexible than Python syntactically. There's a reason that every attempt at recreating dplyr in Python ends in a bit of a mess (IMO) -- Python just doesn't allow the sort of metaprogramming you'd require for a really nice port. Something as simple as a general pipe operator can't be d…

(Replying to disgruntledphd2) > Well that's just lazy evaluation of function arguments, which can't be done in Python. "Just lazy evaluation"! :) It's a pretty big deal. This is three-fifths of the way to a macro system. > But if take a look at the Python data model, it does seem super, super flexible. Sure, you can have a lot of control over the behavior of Python objects (some techniques of which remain obscure to…

> Sure, you can have a lot of control over the behavior of Python objects (some techniques of which remain obscure to me even after using Python for many years). But you don't have anything like syntactic macros.

Not yet, but there’s a PEP for that:

https://www.python.org/dev/peps/pep-0638/

Re: Scikit-Learn Version 1.0

#59

Post by MichaelRazum »

mrtranscendence wrote: post #36
MichaelRazum wrote: post #30

Earlier quoted context omitted.

Just curious. In which way is data.table superior to pandas? Really interested about it! From my personal experience pandas is just sometimes a bit slow.

I'm more a dplyr man myself, but data.table is much faster than pandas, most noticeably IMO when reading large files. It's also extremely succinct if you're into that sort of thing (though I find it a bit obfuscated). pandas is a lot of things, but "fast" and "concise" are not two of them.

Got it. Regarding fast you have something like Vaex on python side (but not sure how fast it realy is). For me I had with pandas the most issues using it's multiindex.

Re: Scikit-Learn Version 1.0

#60

Post by mindv0rtex »

mrtranscendence wrote: post #52
dachryn wrote: post #47

Earlier quoted context omitted.

R is everywhere, especially when you need to visualize stuff. It is primarily used in teams who are trying to get rid of SAS in my experience. You are right in the sense that R is typically not used end-to-end as far as I can tell, but already tries to start with a data connection to some sort of dump or datalake, or datawarehouse. Many people in my team use Python for modelling, but grab ggplot in whatever way to ma…

There are Python ports of ggplot (e.g. plotnine ( https://github.com/has2k1/plotnine )), but agreed, Python is behind here. I'm not the best at data viz, but I can usually piece together a way to make ggplot do what I want it to do without that much trouble or looking at documentation. Matplotlib, though ... that's a harder beast to internalize. I know it's possible to make high-quality matplotlib plots, but it's muc…

Speaking of what's possible in matplotlib, I am very much looking forward to reading this book: https://github.com/rougier/scientific-visualization-book
Post reply on HN