Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scikit-Learn Version 1.0

scikit-learn.org

Post reply on HN
Print view View original

41–50 of 106 posts

Re: Scikit-Learn Version 1.0

#41

Post by disgruntledphd2 »

mrtranscendence wrote: post #38
lysecret wrote: post #34

Earlier quoted context omitted.

Hehe used to do R IMO you are right about ggplot but I strongly disagree about pandas. I fing love it. Would love to understand you troubles with it though, after using it for 4 years daily mabye I can offer some perspective ;)

I don't mean to disparage pandas, which is a library that does a lot of things fairly well. But as an API for data manipulation I find it very verbose and it doesn't mesh with a "functional" way of thinking about applying transformations. Generally, I've even preferred Spark to pandas, though it's hardly less verbose. Coming from R, it's much slower than data.table and nowhere near as slick and discoverable as dplyr.…

Agreed on your major points.

That being said: > I hate finding CSVs that other data scientists have created from pandas, because they invariably include the index ...

This is also default in R, with row numbers (like I have ever needed them). To be fair, it's gotten better since people stopped putting important information in rownames.

Polars looks interesting, thanks for the recommendation!

Re: Scikit-Learn Version 1.0

#42

Post by disgruntledphd2 »

kzrdude wrote: post #23
disgruntledphd2 wrote: post #19

Earlier quoted context omitted.

I'm with you on sklearn, the DL libraries and Numpy, but Pandas and Matplotlib are poor, poor relations of the tools available in the R ecosystem (dplyr/ggplot etc).

They need seaborn too, whoch makes the python side a lot stronger

plotnine FTW! You'll pry ggplot2 from my cold, dead hands.

Re: Scikit-Learn Version 1.0

#43

Post by disgruntledphd2 »

pletnes wrote: post #27
civilized wrote: post #22

Earlier quoted context omitted.

As a working data scientist I'd say it's completely the opposite: a good tabular data manipulation package is the single most valuable tool in my tool box. And R's packages (either data.table or dplyr) are definitely way better than pandas. There's no comparison. I would be hard-pressed to find a working data scientist whose definition of data science is "that thing you do with sklearn, Deep Learning and Numpy".

Tabular data is great for many usecases, but saying that image, audio, and video analysis is not data science seems like a weird variant of gatekeeping to me.

> Tabular data is great for many usecases, but saying that image, audio, and video analysis is not data science seems like a weird variant of gatekeeping to me.

Most problems are mostly tabular, IME.

I completely agree that text, images and video are much, much better handled by Python (that's why I use and know both).

Re: Scikit-Learn Version 1.0

#44

Post by disgruntledphd2 »

pc86 wrote: post #20
disgruntledphd2 wrote: post #19

Earlier quoted context omitted.

I'm with you on sklearn, the DL libraries and Numpy, but Pandas and Matplotlib are poor, poor relations of the tools available in the R ecosystem (dplyr/ggplot etc).

If you're doing data science aren't sklearn, DL, and numpy getting you 90% of the way there anyway? Even if R has better "versions" of pandas/matplotlib (not conceding that point) it's not exactly central to the job of data science.

> If you're doing data science aren't sklearn, DL, and numpy getting you 90% of the way there anyway?

Not really, tbh. Most of my jobs (even when the primary output was models) require spending a _lot_ of time data wrangling and plotting. R is much, much better for this kind of exploratory work.

But if I need to integrate with bigger systems (as I normally do), there's a stronger push for Python to reduce complexity and make it easier for SE's to understand and maintain (some of) the code.

Re: Scikit-Learn Version 1.0

#46

Post by Tarq0n »

boringg wrote: post #32
disgruntledphd2 wrote: post #19

Earlier quoted context omitted.

I'm with you on sklearn, the DL libraries and Numpy, but Pandas and Matplotlib are poor, poor relations of the tools available in the R ecosystem (dplyr/ggplot etc).

Wait how many companies are actually using R in the wild? As I understand it, R is born of academia, great for statistics/analysis but breaks down on data manipulation and isn't used in production/data engineering. Maybe my understanding is dated though?

It's mostly the "in production" part that determines whether R is suitable for a business or not. It's much more complicated to avoid runtime errors or do proper testing in R, whereas it shines for interactive use, or generating reports.

That said having used both the DSL's for plotting and data wrangling in the R package ecosystem are vastly superior to pandas and python plotting libraries. For modeling I actually like the better namespacing of Python which helps keep things more legible when there are a ton of model options to choose from, assuming you don't need cutting edge statistics.

Re: Scikit-Learn Version 1.0

#47

Post by dachryn »

boringg wrote: post #32
disgruntledphd2 wrote: post #19

Earlier quoted context omitted.

I'm with you on sklearn, the DL libraries and Numpy, but Pandas and Matplotlib are poor, poor relations of the tools available in the R ecosystem (dplyr/ggplot etc).

Wait how many companies are actually using R in the wild? As I understand it, R is born of academia, great for statistics/analysis but breaks down on data manipulation and isn't used in production/data engineering. Maybe my understanding is dated though?

R is everywhere, especially when you need to visualize stuff. It is primarily used in teams who are trying to get rid of SAS in my experience.

You are right in the sense that R is typically not used end-to-end as far as I can tell, but already tries to start with a data connection to some sort of dump or datalake, or datawarehouse.

Many people in my team use Python for modelling, but grab ggplot in whatever way to make their presentations and visuals (they all use different methods, usually something messy like mixing python and R in a notebook or so). GGPlots also has a vast library of super high quality plugins.

Python is far far behind in the viz space

Re: Scikit-Learn Version 1.0

#48

Post by optimalsolver »

disgruntledphd2 wrote: post #19
lysecret wrote: post #16

Excellent library for train_test_split. Jokes aside. This next to Numpy, Pandas Jupyter and Matplotlib + the DL libraries are the reason Python is the powerhouse it is for Data Science.

I'm with you on sklearn, the DL libraries and Numpy, but Pandas and Matplotlib are poor, poor relations of the tools available in the R ecosystem (dplyr/ggplot etc).

Of possible interest, a C++ replacement for Pandas:

https://github.com/hosseinmoein/DataFrame

Re: Scikit-Learn Version 1.0

#49

Post by mrtranscendence »

disgruntledphd2 wrote: post #37
lysecret wrote: post #35

Earlier quoted context omitted.

used to work in insurance and heavily used it.

> Wait how many companies are actually using R in the wild? As I understand it, R is born of academia, great for statistics/analysis but breaks down on data manipulation and isn't used in production/data engineering. It depends, I've worked in some places where R was the core part of their data infrastructure. Data manipulation (of non text) is far, far better in R. Integrating with other systems can be tricky though…

It's unrelated to your main point, but:

> Additionally, R is a very, very flexible language (like Python)

I'd argue that R is much more flexible than Python syntactically. There's a reason that every attempt at recreating dplyr in Python ends in a bit of a mess (IMO) -- Python just doesn't allow the sort of metaprogramming you'd require for a really nice port. Something as simple as a general pipe operator can't be defined in Python, to say nothing of how dplyr scopes column names within verbs.

Arguably this does allow you to go crazy in a way that ends up being detrimental to readability, but I'd say overall it's a net benefit to R over Python. I really miss this stuff and have spent an undue amount of time thinking of the best way to emulate it (only to come up with ideas that just disappoint).

> Finally, when you need to hand over stuff to software engineers, they vastly tend to prefer Python

Indeed, this is maybe 50% of the reason my organization has pushed R to the sidelines over the past few years. We used to be very heavily into R but now it has "you can use it, but don't expect support" status.

Re: Scikit-Learn Version 1.0

#50

Post by pantsforbirds »

disgruntledphd2 wrote: post #19
lysecret wrote: post #16

Excellent library for train_test_split. Jokes aside. This next to Numpy, Pandas Jupyter and Matplotlib + the DL libraries are the reason Python is the powerhouse it is for Data Science.

I'm with you on sklearn, the DL libraries and Numpy, but Pandas and Matplotlib are poor, poor relations of the tools available in the R ecosystem (dplyr/ggplot etc).

I'm surprised you dont like pandas. I've found it to be a pretty easy to use and useful tool and you can almost always use something like DASK (or if youre lucky CUDF from rapidsai) if you need better performance.

I will say that my very first "real" programming experience was Matlab at a research internship, so maybe i just got used to working in vectors and arrays for computational tasks.

Post reply on HN