I think a lot of this comes down to the question: Why aren't tables first class citizens in programming languages? If you step back, it's kind of weird that there's no mainstream programming language that has tables as first class citizens. Instead, we're stuck learning multiple APIs (polars, pandas) which are effectively programming languages for tables. R is perhaps the closest, because it has data.frame as a 'firs…
APL Is great
Python is not a great language for data science
211–220 of 339 posts
Re: Python is not a great language for data science
#212It lives in a sterile, idealized world.
Python is a great language for data science in practice because it turns out data science is also:
- gluing a lot of data sources
- cleaning up a ton of terribly shaped data
- validation and error handling
- I/O, networking, and format conversion
- emboarding non-programmers into programming
- wrapping a lot of compiled languages' libs or plugging system
- prototyping stuff and exposing that prototype to some people
- turning prototypes into more permanent projects
And it turns out Python and its ecosystem are good at those while remaining decent at the other things.There are other languages excellent at some of those, or some of the other things, but rarely good at most. And because humanity is vast, diverse, and constantly renewing, being the second best at those is eventually always winning.
Because whoever you are, you will be annoyed at not having the best experience at task X. But you would be mortified if you had the worst experience at doing task Y and Z. And task X, Y, and Z change depending on who you ask.
And you want to get things done, while days have 24 hours.
As usual, to understand the Python phenomenon, you have to see the whole picture. Not your little corner of the bubble. Not the ideal world in your head either. Life is not a maths problem with a clearly laid out premise and an elegant answer.
That's the same debate about why PHP won the web in 2000 no matter the size of the spaghetti plate, why Windows stayed used for so long despite it being terrible, why people keep using iphones after all the abuses, etc. There is more to it than the use case you have every day. People have needs you don't haven't thought about.
So it's not "let the language war begin". It's, "dude, get more experience, go work with accountants, ngos, govs and logistic chains, go work in china, africa and south america, go from a startup to schools to corporate, satisfy the geeks, the artists and the business people, than we'll talk".
Re: Python is not a great language for data science
#213I think a lot of this comes down to the question: Why aren't tables first class citizens in programming languages? If you step back, it's kind of weird that there's no mainstream programming language that has tables as first class citizens. Instead, we're stuck learning multiple APIs (polars, pandas) which are effectively programming languages for tables. R is perhaps the closest, because it has data.frame as a 'firs…
Re: Python is not a great language for data science
#214My experience was that data science was doable but clunky and ugly with pandas. It got slightly better with polars. Only really slightly better. Then, for me at least, it jumped lightyears ahead with duckdb. These days I run some big query on an OLAP database and download the results to parquet stored on the local disk of a cloud notebook VM and then mine it to bits with duckdb reading straight from these parquet fil…
Re: Python is not a great language for data science
#215> Examples include converting boxplots into violins or vice versa, turning a line plot into a heatmap, plotting a density estimate instead of a histogram, performing a computation on ranked data values instead of raw data values, and so on. Most of this is not about Python, it’s about matplotlib. If you want the admittedly very thoughtful design of ggplot in Python, use plotnine > I would consider the R code to be sl…
In R it's often that things for which there's a ready made libraries and recipes are easy, but when those don't exist, things become extremely hard. And the usual approach is that if something is not easy with a library recipe, it just is not done.
Re: Python is not a great language for data science
#216I was all hyped up, ready to see the amazing examples and arguments that would convince me to pick up R, and it gave me absolutely nothing (except quotes and brackets..).
Disappointing.
Re: Python is not a great language for data science
#217My experience was that data science was doable but clunky and ugly with pandas. It got slightly better with polars. Only really slightly better. Then, for me at least, it jumped lightyears ahead with duckdb. These days I run some big query on an OLAP database and download the results to parquet stored on the local disk of a cloud notebook VM and then mine it to bits with duckdb reading straight from these parquet fil…
Re: Python is not a great language for data science
#218Earlier quoted context omitted.
> The success of python is due to not needing a broader ecosystem for A LOT of things. I honestly think that was a coincidence. Perl and Ruby had other disadvantages, Python won despite having bad package management and a bloated standard library, not because of it.
The bloated standard library is the only reason I kept using python in spite of the packaging nightmare. I can do most things with no dependencies, or with one dependency I need over and over like matplotlib If python had been lean and needed packages to do anything useful, while still having a packaging nightmare, it would have been unusable
Re: Python is not a great language for data science
#219Python is a pretty bad language for tabular data analysis and plotting, which seems to be the actual topic of this post. R is certainly better, hell Tableau, Matlab, JMP, Prism and even Excel are all better in many cases. Pandas+seaborn has done a lot, but seaborn still has frustrating limits. And pandas is essentially a separate programming language. If your data is already in a table, and you’re using Python, you’r…
I'd say dplyr/tidyverse is a lot more a separate programming language to R than pandas is to Python.
Re: Python is not a great language for data science
#220This was underwhelming. I work with Python and Pandas, and I can show examples of much clumsier workflows I run into. The most often, you get dataframe[(dataframe.column1 == something) & ~dataframe.column2.isna()] constucts, which show that python syntax falls short here, and isn't suitable for such manipulations. Unfortunately, there's no alternative, and I don't see R as much easier, there are plenty of ugly things…
> Unfortunately, there's no alternative, and I don't see R as much easier, there are plenty of ugly things as well there. Have you tried Polars? It really discourages the inefficient creation of intermediate boolean arrays such as in the code that you are showing. > There's Julia -- it has serious drawbacks, like slow cold start if you launch a Julia script from the shell, which makes it unsuitable for CLI workflows.…
The problem is not usually inefficiency, but syntactic noise. Polars does remove that in some cases, but in general gets even more verbose (apparently by design), which gets annoying fast when doing explorative data analysis.