Live data from Hacker News

Python is not a great language for data science

blog.genesmindsmachines.com

131–140 of 339 posts

Re: Python is not a great language for data science

#131
post #91
post #85

The pure Python code in the last example is more verbose than it needs to be. groups = {} for row in filtered: key = (row['species'], row['island']) if key not in groups: groups[key] = [] groups[key].append(row['body_mass_g']) can be rewritten as: groups = collections.defaultdict(list) for row in filtered: groups[(row['species'], row['island'])].append(row['body_mass_g']) and variance = sum((x - mean) ** 2 for x in v…

Disagree. In the first instance, the original code is readable and tells me exactly what's what. In your example, you're sacrificing readability for being clever. Clear code(even if verbose) is better than being clever.

I would keep the explicit key= assignment since it's more than just a single literal but otherwise the second version is more idiomatic and readable.

Re: Python is not a great language for data science

#133
So I've been writing Python for around 20 years now, and doing data science/ML work for around 15. Despite being a Python programmer first I spent a good 5 years using R exclusively. There's a lot of things I genuinely love about R and I strongly believe that R is unfairly maligned by devs... but there's a good reason I have written exclusively Python for DS work for the last 5 years.

> Python is pretty good for deep learning. There’s a reason PyTorch is the industry standard. When I’m talking about data science here, I’m specifically excluding deep learning.

I've written very little deep learning code over my career, but made very frequent use of the GPU and differentiable programming for non-deep learning specific tasks. In general Python is much easier to write quantitative programs that make use of the hardware, and you have a lot more options when your problem doesn't fit into RAM.

> I have been running a research lab in computational biology for over two decades.

I've been working nearly exclusively in industry for these two decades and a major reason I find Python just better is it's much, much easier to interface with other parts of engineering when you're a using truly general purpose PL. I've actually never worked for a pure Python shop, but it's generally much easier to get production ML/DS solutions into prod when working with Python.

> Data science as I define it here involves a lot of interactive exploration of data and quick one-off analyses or experiments

This re-iterates the previous difference. In my experience I would call this "step one" in all my DS related work. The first step is to understand the problem and de-risk. But the vast majority of code and work is related to delivering a scalable product.

You can say that's not part of "data science", but if you did you'd have a hard time finding a job on most of the teams I've worked on.

All that said, my R vs Python experience has boiled down to: If your end result is a PDF report, R is superior. If your end result is shipping a product, then Python is superior. And my experience has been that, outside of university labs, there aren't a lot of jobs out there for DS folks who only want to deliver PDFs.

Re: Python is not a great language for data science

#134

Earlier quoted context omitted.

I can't speak for Julia - never used it; never used Common Lisp for analyzing data (I don't think it's very "data-oriented" for the modern age and the shape of data), but Clojure is really not "obscure" - it only looks weird for the first fifteen minutes or so; once you start using it - it is one of the most straightforward and reasonable languages out there - it is in fact simpler than Python and Javascript. Immutab…

I tried to get into Clojure, but a lot of the JVM hosted languages require some Java experience. Same thing with Scala and Kotlin or F# on .NET. The early tooling was also pretty dependent on Vim or Emacs. Maybe it's all easier now with VSCode or something like that.

None of this even remotely true. I've gotten into Clojure without knowing jackshit about Java, almost ten years later, after tons of things successfully built and deployed, still don't know jackshit about Java. Mia, co-host of 'Clojure apropos' podcast was my colleague, we've worked together on multiple teams, she learned Clojure as her very first PL. Later she tried learning some Java and she was shocked how impossibly weird it looked compared to Clojure. Besides, you can use Clojure without any JVM - e.g., with nbb. I use it for things like browser automation with Playwright.

The tooling story is also very solid - I use Emacs, but many of my friends and colleagues use IntelliJ, Vim, Sublime and VSCode, and some of them migrated to it from Atom.

Re: Python is not a great language for data science

#135
post #91

Earlier quoted context omitted.

Disagree. In the first instance, the original code is readable and tells me exactly what's what. In your example, you're sacrificing readability for being clever. Clear code(even if verbose) is better than being clever.

Using a very common utility in the standard library is to avoid reinventing the wheel is not "clean code"? defaultdict is ubiquitous in modern python, and is far from a complicated concept to grasp.

I don't think that's the right metaphor to use here, it exists at a different level than what I would consider "reinventing the wheel". That to me is more some attempt to make a novel outward-facing facet of the program when there's not much reason to do so. For example, reimplementing shared memory using a custom kernel driver as your IPC mechanism, despite it not doing anything that shared memory doesn't already do.

The difference between the examples is so trivial I'm not really sure why the parent comment felt compelled to complain.

Re: Python is not a great language for data science

#136
post #80

Earlier quoted context omitted.

Python is nothing without it’s batteries.

I hear this so much from Python people -- almost like they are paid by the word to say it. Is it different from Perl, Ruby, Java, or C# (DotNet)? Not in my experience, except people from those communities don't repeat that phrase so much. The irony here: We are talking about data science. 98% of "data science" Python projects start by creating a virtual env and adding Pandas and NumPy which have numerous (really: squ…

Someone correct me if I'm completely wrong, but by default (i.e. precompiled wheels) numpy has 0 dependencies and pandas has 5, one of which is numpy. So not really "squillions" of dependencies.

pandas==2.3.3

├── numpy [required: >=1.22.4, installed: 2.2.6]

├── python-dateutil [required: >=2.8.2, installed: 2.9.0.post0]

│ └── six [required: >=1.5, installed: 1.17.0]

├── pytz [required: >=2020.1, installed: 2025.2]

└── tzdata [required: >=2022.7, installed: 2025.2]

Re: Python is not a great language for data science

#137

I'm not sure what that last example is meant to be other than an anti-Python caricature. If you're implementing calculating things like standard deviations by hand, that's not real-world coding, that's the undergraduate harassment package which should end with a STEM bachelor's. Of course there's a bunch of loops and things; you're exposing what has to happen in both R and Python under the hood of all those packages.

> that's not real-world coding

It's pretty clear the post is focused on the context of work being done in an academic research lab. In that context I think most of the points are pretty valid, but most of the real world benefit I've experience from using Python is being able to work more closely with engineering (even on non-Python teams).

I shipped R code to a production environment once over my career and it felt incredibly fragile.

R is great for EDA, but really doesn't work well for iteratively building larger software projects. R is has a great package system, but it's not so great when you need abstraction in between.

Re: Python is not a great language for data science

#138

Earlier quoted context omitted.

Python has a list of issues fundamentally broken in the language, and relies heavily on integrated library bindings to operate at reasonable speeds/accuracy. Julia allows embedding both R and Python code, and has some very nice tools for drilling down into datasets: https://www.queryverse.org/ It is the first language I've seen in decades that reduces entire paradigms into single character syntax, often outperforming…

Deeply ironic for a Julia proponent to smear a popular language as "fundamentally broken" without evidence. https://yuri.is/not-julia/

Python threading and computational errata issues go back a long time. It is a popular integration "glue" language, but is built on SWiG wrappers to work around its many unresolved/unsolvable problems.

Not a "smear", but rather a well known limitation of the language. Perhaps your environment context works differently than mine.

It is bizarre people get emotionally invested in something so trivial and mundane. Julia is at v1.12.2 so YMMV, but Queryverse is a lot of fun =3

Re: Python is not a great language for data science

#139

Earlier quoted context omitted.

I can't speak for Julia - never used it; never used Common Lisp for analyzing data (I don't think it's very "data-oriented" for the modern age and the shape of data), but Clojure is really not "obscure" - it only looks weird for the first fifteen minutes or so; once you start using it - it is one of the most straightforward and reasonable languages out there - it is in fact simpler than Python and Javascript. Immutab…

Common Lisp fan here, but not a data scientist. Why do you say to avoid CL for data analysis? Not trying to flame or anything, just curious about your experience with it.

I don't have great experience of using CL for analyzing data, because of "why?", if I already have another Lisp that is simply amazing for data.

Clojure, unlike lists in traditional Lisps, based on composable, unified abstraction for its collections, they are lazy by default and literal readable data structures, they are far easier to introspect and not so "opaque" compared to anything - not just CL (even Python), they are superb for dealing with heterogeneous data. Clojure's cohesive data manipulation story is where Common Lisp's lists-and-symbols just can't match.

Re: Python is not a great language for data science

#140
post #72

> Examples include converting boxplots into violins or vice versa, turning a line plot into a heatmap, plotting a density estimate instead of a histogram, performing a computation on ranked data values instead of raw data values, and so on. Most of this is not about Python, it’s about matplotlib. If you want the admittedly very thoughtful design of ggplot in Python, use plotnine > I would consider the R code to be sl…

R is more of a statistical software than a programming language. So, if you are a so-called "statistician," then R will feel familiar to you
Post reply on HN