Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scikit-Learn Version 1.0

scikit-learn.org

Post reply on HN
Print view View original

61–70 of 106 posts

Re: Scikit-Learn Version 1.0

#61

Post by zeec123 »

infimum wrote: post #21

scikit-learn (next to numpy) is the one library I use in every single project at work. Every time I consider switching away from python I am faced with the fact that I'd lose access to this workhorse of a library. Of course it's not all sunshine and rainbows - I had my fair share of rummaging through its internals - but its API design is a de-facto standard for a reason. My only recurring gripe is that the serializat…

There is so much wrong with the api design of sklearn (how can one think "predict_proba" is a good function name?). I can understand this, since most of it was probably written by PhD students without the time and expertise to come up with a proper api; many of them without a CS background.[1]

[1] https://www.reddit.com/r/haskell/comments/7brsuu/machine_lea...

Re: Scikit-Learn Version 1.0

#63

Post by baron_harkonnen »

disgruntledphd2 wrote: post #19
lysecret wrote: post #16

Excellent library for train_test_split. Jokes aside. This next to Numpy, Pandas Jupyter and Matplotlib + the DL libraries are the reason Python is the powerhouse it is for Data Science.

I'm with you on sklearn, the DL libraries and Numpy, but Pandas and Matplotlib are poor, poor relations of the tools available in the R ecosystem (dplyr/ggplot etc).

I used to very strongly agree with you re: matplotlib, but I've recently switched from using almost exclusively ggplot2 to almost exlusively Matplotlib and my realization is that they are very different tools serving very different purposes.

ggplot2 is obviously fantastic and makes beautiful plots, and very easily at that. However it is definitely a "convention over configuration" tool. For 99% of the typical plot you might want to create, ggplot is going to be easier and look nicer.

However matplotlib lib really shines when you want to make very custom plots. If you have a plot in your mind that you want to see on paper, matplotlib will be the better tool for helping you create exactly what you are looking for.

For certain projects I've done, where I want to do a bunch of non-standard visualizations, especially ones that tend to be fairly dense, I prefer matplotlib. For day to day analytics ggplot2 is so much better it's ridiculous. The real issue is that Python doesn't really offer anything in the same league as ggplot2 for "convention over configuration" type plotting.

Fully agree on Pandas. R's native data frame + tidyverse is world's easier. Pandas' overly complex indexing system is a persistent source of annoyance no matter how much I use that library.

Re: Scikit-Learn Version 1.0

#64

Post by disgruntledphd2 »

mrtranscendence wrote: post #55
Tarq0n wrote: post #46

Earlier quoted context omitted.

It's mostly the "in production" part that determines whether R is suitable for a business or not. It's much more complicated to avoid runtime errors or do proper testing in R, whereas it shines for interactive use, or generating reports. That said having used both the DSL's for plotting and data wrangling in the R package ecosystem are vastly superior to pandas and python plotting libraries. For modeling I actually l…

> It's much more complicated to avoid runtime errors or do proper testing in R It's not that much harder. There's no pytest, but testthat works well enough. I've developed a few packages internally in R and wouldn't say it was that much harder to ensure correctness than for the corresponding Python packages. (We used to keep them in sync, before basically moving everything to Python.)

I actually quite like R's error handling. It's as good as Common Lisp's which is often held up as the epitome of this.

You also have the dump.frames option, which will save your workspace on failure, which is incredibly useful when running R stuff remotely/in a distributed fashion.

Re: Scikit-Learn Version 1.0

#65

Post by disgruntledphd2 »

mrtranscendence wrote: post #57
mrtranscendence wrote: post #49

Earlier quoted context omitted.

It's unrelated to your main point, but: > Additionally, R is a very, very flexible language (like Python) I'd argue that R is much more flexible than Python syntactically. There's a reason that every attempt at recreating dplyr in Python ends in a bit of a mess (IMO) -- Python just doesn't allow the sort of metaprogramming you'd require for a really nice port. Something as simple as a general pipe operator can't be d…

(Replying to disgruntledphd2) > Well that's just lazy evaluation of function arguments, which can't be done in Python. "Just lazy evaluation"! :) It's a pretty big deal. This is three-fifths of the way to a macro system. > But if take a look at the Python data model, it does seem super, super flexible. Sure, you can have a lot of control over the behavior of Python objects (some techniques of which remain obscure to…

I'm totally with you on these points, and it's one of the places where R's genesis as a scheme program has lead to really, really good consequences.

Fundamentally though, both DS Python and R are abstractions over well-tested Fortran linear algebra routines (I'm sortof kidding, but only sortof).

Re: Scikit-Learn Version 1.0

#66

Post by mrtranscendence »

dragonwriter wrote: post #58
mrtranscendence wrote: post #57

Earlier quoted context omitted.

(Replying to disgruntledphd2) > Well that's just lazy evaluation of function arguments, which can't be done in Python. "Just lazy evaluation"! :) It's a pretty big deal. This is three-fifths of the way to a macro system. > But if take a look at the Python data model, it does seem super, super flexible. Sure, you can have a lot of control over the behavior of Python objects (some techniques of which remain obscure to…

> Sure, you can have a lot of control over the behavior of Python objects (some techniques of which remain obscure to me even after using Python for many years). But you don't have anything like syntactic macros. Not yet, but there’s a PEP for that: https://www.python.org/dev/peps/pep-0638/

Nice, I'd love for this to see the light of day. I suspect it'll see some resistance (even pattern matching caused conflict, and I thought that was terribly innocuous).

(Why can I reply at this level of nesting now, whereas before I couldn't?)

Re: Scikit-Learn Version 1.0

#67

Post by _Wintermute »

civilized wrote: post #29
jstx1 wrote: post #24

Earlier quoted context omitted.

> "Data science is that thing where you do sklearn, Deep Learning and Numpy" is not a working data scientist's perspective. It could be. It's such a broad job title and it looks so different across different companies and teams that the main tool for one data scientist might be something that another data scientist never has to touch. Different data science jobs prioritise different tools, that's all.

Right, so defining data science as 90% sklearn+DL+numpy is just as silly as saying that it's 90% table manipulation. That's exactly my point. Still, if anyone here has managed to find a data science job in which tabular data management is not a sizable piece of what you do, I'd like to know some details!

I worked as a datascientist for a couple of years and tabular data was a very small part of my job. I spent far more time with image-analysis and JSON, both of which I found R sucks at.

Re: Scikit-Learn Version 1.0

#68

Post by baron_harkonnen »

boringg wrote: post #32
disgruntledphd2 wrote: post #19

Earlier quoted context omitted.

I'm with you on sklearn, the DL libraries and Numpy, but Pandas and Matplotlib are poor, poor relations of the tools available in the R ecosystem (dplyr/ggplot etc).

Wait how many companies are actually using R in the wild? As I understand it, R is born of academia, great for statistics/analysis but breaks down on data manipulation and isn't used in production/data engineering. Maybe my understanding is dated though?

Of the many companies I've done data science with I can only think of a few, rare exceptions where R wasn't used as much as if not more than Python.

If you're mostly dealing with Neural Nets you won't see much R, but for anything really statistical in nature R is a much better tool than Python. For anything that ends up in a report R is much better than Python (a lot of very valuable data science work ends up being a report to non-technical people).

> breaks down on data manipulation

This is very outdated. The tidyverse eco-system has bumped R back into being first in class for data manipulation now. This becomes less true as you get further and further from having your data in a matrix/df (I can't imagine doing Spark queries in R), but if you already have a basic data frame, manipulation from there is very easy.

Even for things that end up in production, whether you're in R or Python, whatever your first pass is should always be a prototype and will have to be reworked before you get close to moving it to production.

Re: Scikit-Learn Version 1.0

#69

Post by deshpand »

mrtranscendence wrote: post #38
lysecret wrote: post #34

Earlier quoted context omitted.

Hehe used to do R IMO you are right about ggplot but I strongly disagree about pandas. I fing love it. Would love to understand you troubles with it though, after using it for 4 years daily mabye I can offer some perspective ;)

I don't mean to disparage pandas, which is a library that does a lot of things fairly well. But as an API for data manipulation I find it very verbose and it doesn't mesh with a "functional" way of thinking about applying transformations. Generally, I've even preferred Spark to pandas, though it's hardly less verbose. Coming from R, it's much slower than data.table and nowhere near as slick and discoverable as dplyr.…

Ideally you should be using the parquet format which will use the binary format, preserve column types and indexes [df.to_parquet(); df = pd.read_parquet()]

You can get away from a lot of problems by simply avoiding text files

Re: Scikit-Learn Version 1.0

#70

Post by mrtranscendence »

MichaelRazum wrote: post #59
mrtranscendence wrote: post #36

Earlier quoted context omitted.

I'm more a dplyr man myself, but data.table is much faster than pandas, most noticeably IMO when reading large files. It's also extremely succinct if you're into that sort of thing (though I find it a bit obfuscated). pandas is a lot of things, but "fast" and "concise" are not two of them.

Got it. Regarding fast you have something like Vaex on python side (but not sure how fast it realy is). For me I had with pandas the most issues using it's multiindex.

> For me I had with pandas the most issues using it's multiindex.

Yessss. I loathe indices, and have never been in a situation where I was better off with them than without them.

> Regarding fast you have something like Vaex on python sid

I've never used Vaex, but I've used datatable (https://github.com/h2oai/datatable) and polars (https://github.com/pola-rs/polars). Polars is my favorite API, but datatable was faster at reading data (Polars was faster in execution). I'll have to give Vaex a try at some point.

Post reply on HN