Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python is not a great language for data science

blog.genesmindsmachines.com

Post reply on HN
Print view View original

191–200 of 339 posts

Re: Python is not a great language for data science

#191

Post by a_bonobo »

jakobnissen wrote: post #27

Excellent article - except that the author probably should have gated their substantiation of the claim behind a cliffhanger, as other commenters have mentioned. The author's priorities are sensible, and indeed with that set of priorities, it makes sense to end up near R. However, they're not universal among data scientists. I've been a data scientist for eight years, and have found that this kind of plotting and dat…

>I find there is usually also some file juggling, parsing, [...]

I'd say I'm 50/50 Python/R for exactly this reason: I write Python code on HPC or a server to parse many, many files, then I get some kind of MB-scale summary data I analyse locally in R.

R is not good at looping over hundreds of files in the gigabytes, Python is not good at making pretty insights from the summary. A tool for every task.

Re: Python is not a great language for data science

#192

Post by habinero »

OkayPhysicist wrote: post #130
throwaway2037 wrote: post #127

Earlier quoted context omitted.

> There's a number of structures that I think are missing in our major programming languages. Tables are one. Matrices are another. I disagree. Most programmers will go their entire career and never need a matrix data structure. Sure, they will use libraries that use matrices, but never use them directly themselves. It seems fine that matrices are not a separate data type in most modern programming languages.

Unless you think "most programmers" === "shitty webapp developers", I strongly disagree. Matrices are first class, important components in statistics, data analysis, graphics, video games, scientific computing, simulation, artificial intelligence and so, so much more. And all of those programmers are either using specialized languages, (suffering problems when they want to turn their program into a shitty web app, fo…

I don't see why the majority of engineers need to cater to your niche use cases. It's a programming language, you can just make the library if it doesn't exist. Nobody's stopping you.

Plus, plenty of third party projects have been incorporated into the Python standard library.

Re: Python is not a great language for data science

#193

Post by rjzzleep »

lmm wrote: post #157
LtWorf wrote: post #100

Earlier quoted context omitted.

The success of python is due to not needing a broader ecosystem for A LOT of things. They are of course now abandoning this idea.

> The success of python is due to not needing a broader ecosystem for A LOT of things. I honestly think that was a coincidence. Perl and Ruby had other disadvantages, Python won despite having bad package management and a bloated standard library, not because of it.

It's because Ruby captured the web market and Python everything else, and I get everything is more timeless than a single segment.

Re: Python is not a great language for data science

#194

Post by Surac »

I at the moment try to learn python as a hobby language. I use c c++ and c# to earn my money. MY biggest problem is finding good examples that are up to date. I spent a whole day learning that there a four (I think) ways to do formatting strings. This „bloat“ in syntax makes even a simple print very heavy to digest. I don’t even bother using v2 python only v3. Also using whitespaces to block things together sounds appealing but in reality you need to use editors that can indent and unindent whole blocks or I never get it right

Re: Python is not a great language for data science

#196

Post by m_mueller »

RobinL wrote: post #16

I think a lot of this comes down to the question: Why aren't tables first class citizens in programming languages? If you step back, it's kind of weird that there's no mainstream programming language that has tables as first class citizens. Instead, we're stuck learning multiple APIs (polars, pandas) which are effectively programming languages for tables. R is perhaps the closest, because it has data.frame as a 'firs…

Fortran gives you that and more, it has first class multidimensional arrays, including matrix operations.

Re: Python is not a great language for data science

#197

Post by genidoi »

RobinL wrote: post #16

I think a lot of this comes down to the question: Why aren't tables first class citizens in programming languages? If you step back, it's kind of weird that there's no mainstream programming language that has tables as first class citizens. Instead, we're stuck learning multiple APIs (polars, pandas) which are effectively programming languages for tables. R is perhaps the closest, because it has data.frame as a 'firs…

This is an interesting observation. One possible explanation for a lack of robust first class table manipulation support in mainstream languages could be due to the large variance in real-world table sizes and the mutually exclusive subproblems that come with each respective jump in order-of-magnitude row size.

The problems that one might encounter in dealing with a 1m row table are quite different to a 1b row table, and a 1b row table is a rounding error compared to the problems that a 1t row table presents. A standard library needs to support these massive variations at least somewhat gracefully and that's not a trivial API surface to design.

Re: Python is not a great language for data science

#198

Post by lenkite »

Surac wrote: post #194

I at the moment try to learn python as a hobby language. I use c c++ and c# to earn my money. MY biggest problem is finding good examples that are up to date. I spent a whole day learning that there a four (I think) ways to do formatting strings. This „bloat“ in syntax makes even a simple print very heavy to digest. I don’t even bother using v2 python only v3. Also using whitespaces to block things together sounds ap…

15 years ago, Python programmers used to mock Perl by quoting the Zen of Python: "There should be one - and preferably only one - obvious way to do it.". This was in stark contrast to Perl's TIMTOWTDI motto: "There Is More Than One Way To Do It."

The Zen of Python is sadly now an absolute lie.

Re: Python is not a great language for data science

#200

Post by willvarfar »

My experience was that data science was doable but clunky and ugly with pandas. It got slightly better with polars. Only really slightly better. Then, for me at least, it jumped lightyears ahead with duckdb.

These days I run some big query on an OLAP database and download the results to parquet stored on the local disk of a cloud notebook VM and then mine it to bits with duckdb reading straight from these parquet files.

The notebooks end up with very clear SQL queries and results (most notebook servers support SQL cells with highlighting and completion etc), and small pockets of python cells for doing those corner case things that an imperative language makes easier.

So when I get to the bottom of the article where it shows the difference between Python and R, I'm screaming "wouldn't that look better in SQL?!" :)

Post reply on HN