Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python is not a great language for data science

blog.genesmindsmachines.com

Post reply on HN
Print view View original

151–160 of 339 posts

Re: Python is not a great language for data science

#151

Post by aethor »

whyenot wrote: post #29

What makes Python a great language for data science, is that so many people are familiar with it, and that it is an easy language to read. If you use a more obscure language like Clojure, Common Lisp, Julia, etc., many people will not be familiar with the language and unable to read or review your code. Peer review is fundamental to the scientific endeavor. If you only optimize on what is the best language for the ta…

Peer review is fundamental to scientific endeavor but... in ML fields, reviewers almost never check the code and Python package management is hardly reproducible. So clearly we are not there, Python or not.

Re: Python is not a great language for data science

#152

Post by plaidfuji »

Python is a pretty bad language for tabular data analysis and plotting, which seems to be the actual topic of this post. R is certainly better, hell Tableau, Matlab, JMP, Prism and even Excel are all better in many cases. Pandas+seaborn has done a lot, but seaborn still has frustrating limits. And pandas is essentially a separate programming language.

If your data is already in a table, and you’re using Python, you’re doing it because you want to learn Python for your next job. Not because it’s the best tool for your current job. The one thing Python has on all those other options is $$$. You will be far more employable than if you stick to R.

And the reason for that is because Python is one of the best languages for data and ML engineering, which is about 80% of what a data science job actually entails.

Re: Python is not a great language for data science

#153

Post by rtaylorgarlock »

robot-wrangler wrote: post #129
aorist wrote: post #72

> Examples include converting boxplots into violins or vice versa, turning a line plot into a heatmap, plotting a density estimate instead of a histogram, performing a computation on ranked data values instead of raw data values, and so on. Most of this is not about Python, it’s about matplotlib. If you want the admittedly very thoughtful design of ggplot in Python, use plotnine > I would consider the R code to be sl…

>> I would consider the R code to be slightly easier to read (notice how many quotes and brackets the Python code needs) Oh god no, do people write R like that, pipes at the end? Elixir style pipe-operators at the beginning is the way. And if you really wanted to "improve" readability by confusing arguments/functions/vars just to omit quotes, python can do that, you'll just need a wrapper object and getattr hacks to…

Upvoted for pipes at the beginning

Re: Python is not a great language for data science

#154

Post by ashdev »

ashdev wrote: post #91
progval wrote: post #85

The pure Python code in the last example is more verbose than it needs to be. groups = {} for row in filtered: key = (row['species'], row['island']) if key not in groups: groups[key] = [] groups[key].append(row['body_mass_g']) can be rewritten as: groups = collections.defaultdict(list) for row in filtered: groups[(row['species'], row['island'])].append(row['body_mass_g']) and variance = sum((x - mean) ** 2 for x in v…

Disagree. In the first instance, the original code is readable and tells me exactly what's what. In your example, you're sacrificing readability for being clever. Clear code(even if verbose) is better than being clever.

Interesting! Thanks for the responses. I'm not python native and haven't worked as extensively with python as some of you here.

That said, I'll change my mind here and agree on using std library, but I'd still have separate 'key' assignment here for more clarity.

Re: Python is not a great language for data science

#155

Post by lock1 »

OkayPhysicist wrote: post #130
throwaway2037 wrote: post #127

Earlier quoted context omitted.

> There's a number of structures that I think are missing in our major programming languages. Tables are one. Matrices are another. I disagree. Most programmers will go their entire career and never need a matrix data structure. Sure, they will use libraries that use matrices, but never use them directly themselves. It seems fine that matrices are not a separate data type in most modern programming languages.

Unless you think "most programmers" === "shitty webapp developers", I strongly disagree. Matrices are first class, important components in statistics, data analysis, graphics, video games, scientific computing, simulation, artificial intelligence and so, so much more. And all of those programmers are either using specialized languages, (suffering problems when they want to turn their program into a shitty web app, fo…

That's needlessly aggressive. Ignoring webapps, you could do gamedev without even knowing what a matrix is.

You don't even need such construction in most native applications, embedded systems, and OS kernel development.

Re: Python is not a great language for data science

#156

Post by sgarland »

IgorPartola wrote: post #111
RobinL wrote: post #16

I think a lot of this comes down to the question: Why aren't tables first class citizens in programming languages? If you step back, it's kind of weird that there's no mainstream programming language that has tables as first class citizens. Instead, we're stuck learning multiple APIs (polars, pandas) which are effectively programming languages for tables. R is perhaps the closest, because it has data.frame as a 'firs…

SQL is not just about a table but multiple tables and their relationships. If it was just about running queries against a single table then basic ordering, filtering, aggregation, and annotation would be easy to achieve in almost any language. Soon as you start doing things like joins, it gets complicated but in theory you could do something like an API of an ORM to do most things. With using just operators you quick…

Every time I see stuff like this (Google’s new SQL-ish language with pipes comes to mind), I am baffled. SQL to me is eminently readable, and flows beautifully.

For reference, I think the same is true of Python, so it’s not like I’m a Perl wizard or something.

Re: Python is not a great language for data science

#157

Post by lmm »

LtWorf wrote: post #100
pphysch wrote: post #89

Earlier quoted context omitted.

The design and success of e.g. Golang is pretty strong support for the idea that you can't and shouldn't separate a language from its broader ecosystem of tooling and packages.

The success of python is due to not needing a broader ecosystem for A LOT of things. They are of course now abandoning this idea.

> The success of python is due to not needing a broader ecosystem for A LOT of things.

I honestly think that was a coincidence. Perl and Ruby had other disadvantages, Python won despite having bad package management and a bloated standard library, not because of it.

Re: Python is not a great language for data science

#158

Post by getnormality »

aorist wrote: post #72

> Examples include converting boxplots into violins or vice versa, turning a line plot into a heatmap, plotting a density estimate instead of a histogram, performing a computation on ranked data values instead of raw data values, and so on. Most of this is not about Python, it’s about matplotlib. If you want the admittedly very thoughtful design of ggplot in Python, use plotnine > I would consider the R code to be sl…

It's not about Python, it's about how R lets you do something Python can't?

Re: Python is not a great language for data science

#159

Post by getnormality »

RobinL wrote: post #16

I think a lot of this comes down to the question: Why aren't tables first class citizens in programming languages? If you step back, it's kind of weird that there's no mainstream programming language that has tables as first class citizens. Instead, we're stuck learning multiple APIs (polars, pandas) which are effectively programming languages for tables. R is perhaps the closest, because it has data.frame as a 'firs…

Saying that SQL is the standard for manipulating tabular data is like saying that COBOL is the standard for financial transactions. It may be true based on current usage, but nobody thinks it's a good idea long term. They're both based on the outdated idea that a programming language should look like pidgin English rather than math.

Re: Python is not a great language for data science

#160

Post by maest »

RobinL wrote: post #16

I think a lot of this comes down to the question: Why aren't tables first class citizens in programming languages? If you step back, it's kind of weird that there's no mainstream programming language that has tables as first class citizens. Instead, we're stuck learning multiple APIs (polars, pandas) which are effectively programming languages for tables. R is perhaps the closest, because it has data.frame as a 'firs…

> Why aren't tables first class citizens in programming languages?

They are in q/kdb and it's glorious. Sql expressions are also first class citizens and it makes it very pleasant to write code

Post reply on HN