Live data from Hacker News

Python is not a great language for data science

blog.genesmindsmachines.com

251–260 of 339 posts

Re: Python is not a great language for data science

#251
post #146

Earlier quoted context omitted.

No, R is a serious general purpose programming language that is great for building almost any type of complex scientific software with. Projects like Bioconductor are a good example.

No, it's not. Even established packages have bugs caused by R weirdness. I like it nevertheless.

Care to give some examples?

Re: Python is not a great language for data science

#252
post #72

> Examples include converting boxplots into violins or vice versa, turning a line plot into a heatmap, plotting a density estimate instead of a histogram, performing a computation on ranked data values instead of raw data values, and so on. Most of this is not about Python, it’s about matplotlib. If you want the admittedly very thoughtful design of ggplot in Python, use plotnine > I would consider the R code to be sl…

I wonder what the last example of "logistics without libraries" would look like in R. Based on my experience of having to do "low-level" R, it's gonna be a true horror show. In R it's often that things for which there's a ready made libraries and recipes are easy, but when those don't exist, things become extremely hard. And the usual approach is that if something is not easy with a library recipe, it just is not don…

Python: easy things are easy, hard things are hard.

R: easy things are hard, hard things are easy.

Re: Python is not a great language for data science

#253
post #124
post #72

> Examples include converting boxplots into violins or vice versa, turning a line plot into a heatmap, plotting a density estimate instead of a histogram, performing a computation on ranked data values instead of raw data values, and so on. Most of this is not about Python, it’s about matplotlib. If you want the admittedly very thoughtful design of ggplot in Python, use plotnine > I would consider the R code to be sl…

> This isn’t about Python, it’s about the tidyverse. > it’s non-standard-evaluation allows packages to extend the syntax in a way Python does not expose Well this is a fundamental difference between Python and R.

The point is that the ability to extend the syntax of R leads to chaos and mess (in general) but when used correctly and effectively in the tidyverse, improves the experience of writing and reading code.

Re: Python is not a great language for data science

#254
post #91
post #85

The pure Python code in the last example is more verbose than it needs to be. groups = {} for row in filtered: key = (row['species'], row['island']) if key not in groups: groups[key] = [] groups[key].append(row['body_mass_g']) can be rewritten as: groups = collections.defaultdict(list) for row in filtered: groups[(row['species'], row['island'])].append(row['body_mass_g']) and variance = sum((x - mean) ** 2 for x in v…

Disagree. In the first instance, the original code is readable and tells me exactly what's what. In your example, you're sacrificing readability for being clever. Clear code(even if verbose) is better than being clever.

Imo, if you read such code the first time, you may prefer the first. If you read it for the 20th time, you may prefer the second. Once you understand what you are doing, often one prefers more concise syntax that helps in handling complexity within a larger project. But it can seem a bit "too clever" in the beginning.

Re: Python is not a great language for data science

#255
post #91

Earlier quoted context omitted.

Disagree. In the first instance, the original code is readable and tells me exactly what's what. In your example, you're sacrificing readability for being clever. Clear code(even if verbose) is better than being clever.

Imo, if you read such code the first time, you may prefer the first. If you read it for the 20th time, you may prefer the second. Once you understand what you are doing, often one prefers more concise syntax that helps in handling complexity within a larger project. But it can seem a bit "too clever" in the beginning.

This happened to me with comprehensions in python, and with JS' love for anonymous/arrow functions.

Once you get used to a language's "quirks" (so long as they're considered idiomatic), they no longer feel quirky, and it's usually pretty quick.

Re: Python is not a great language for data science

#256

Earlier quoted context omitted.

SQL is not just about a table but multiple tables and their relationships. If it was just about running queries against a single table then basic ordering, filtering, aggregation, and annotation would be easy to achieve in almost any language. Soon as you start doing things like joins, it gets complicated but in theory you could do something like an API of an ORM to do most things. With using just operators you quick…

Every time I see stuff like this (Google’s new SQL-ish language with pipes comes to mind), I am baffled. SQL to me is eminently readable, and flows beautifully. For reference, I think the same is true of Python, so it’s not like I’m a Perl wizard or something.

Oh I agree. The problem is that they are two different languages. Inside a Python file, SQL is just a string. No syntax highlighting, no compile time checking, etc. A Kwisatz Haderach of languages that incorporates both its own language and SQL as first class concepts would be very nice but the problem is that SQL is just too different.

For one thing, SQL is not really meant to be dynamically constructed in SQL. But we often need to dynamically construct a query (for example customer applied several filters to the product listing). The SQL way to handle that would be to have a general purpose query with a thousand if/elses or stored procedures which I think takes it from “flows beautifully” to “oh god who wrote this?” Or you could just do string concatenation in a language that handles that well, like Python. Then wrap the whole thing in functions and objects and you get an ORM.

I still have not seen a language that incorporates anything like SQL into it that would allow for even basic ORM-like functionality.

Re: Python is not a great language for data science

#257

Earlier quoted context omitted.

Imo, if you read such code the first time, you may prefer the first. If you read it for the 20th time, you may prefer the second. Once you understand what you are doing, often one prefers more concise syntax that helps in handling complexity within a larger project. But it can seem a bit "too clever" in the beginning.

This happened to me with comprehensions in python, and with JS' love for anonymous/arrow functions. Once you get used to a language's "quirks" (so long as they're considered idiomatic), they no longer feel quirky, and it's usually pretty quick.

You get to the same point with non-considered idiomatic syntax also, the only problem being that it will be only you who understands it.

Re: Python is not a great language for data science

#258
post #16

I think a lot of this comes down to the question: Why aren't tables first class citizens in programming languages? If you step back, it's kind of weird that there's no mainstream programming language that has tables as first class citizens. Instead, we're stuck learning multiple APIs (polars, pandas) which are effectively programming languages for tables. R is perhaps the closest, because it has data.frame as a 'firs…

It makes sense from a historical perspective. Tables are a thing in many languages, just not the ones that mainstream devs use. In fact, if you rank programming languages by usage outside of devs, the top languages all have a table-ish metaphor (SQL, Excel, R, Matlab). The languages devs use are largely Algol derived. Algol is a language that was used to express algorithms, which were largely abstractions over Turing…

As a slight refinement of your point, C does have storage map based N-D arrays/tensors like Fortran, just with the old column-major/row-major difference and a clunky "multiple [][]" syntax. There was just a restriction early on to need compile-time known dimensions to the arrays (up to the final dimension, anyway) because it was a somewhat half-done/half-supported thing - and because that also fit the linear data model well. So, it is also common to see char *argv[] like arrays of pointers or in numerics sometimes libraries which do their own storage map equations from passed dimensions.

Also, the linear memory model itself is not really only because of Algol/Turing machines/theoretical CS/"early" hardware and mechanical sympathy. DRAM has rows & columns internally, but byte addressability leads to hiding that from HW client systems (unless someone is doing a rowhammer attack or something). More random access than tape rewind/fast forward is indeed a huge deal, but I think the actual popularity of linearity just comes from its simplicity as an interface more than anything else. E.g.s, segmented x86 memory with near/far pointers was considered ugly relative to a big 32-bit address space and disk files and other allocation arenas have internally a large linear address/seek spaces. People just want to defer using >1 number until they really need to. People learn univariate-X before they learn multivariate-X where X could be calculus, statistics, etc., etc.

Re: Python is not a great language for data science

#259
post #72

> Examples include converting boxplots into violins or vice versa, turning a line plot into a heatmap, plotting a density estimate instead of a histogram, performing a computation on ranked data values instead of raw data values, and so on. Most of this is not about Python, it’s about matplotlib. If you want the admittedly very thoughtful design of ggplot in Python, use plotnine > I would consider the R code to be sl…

>> I would consider the R code to be slightly easier to read (notice how many quotes and brackets the Python code needs) Oh god no, do people write R like that, pipes at the end? Elixir style pipe-operators at the beginning is the way. And if you really wanted to "improve" readability by confusing arguments/functions/vars just to omit quotes, python can do that, you'll just need a wrapper object and getattr hacks to…

IIRC, putting pipe operator `|>` at end of line prevents the expression from terminating early. Otherwise the newline would terminate it.

Re: Python is not a great language for data science

#260

Earlier quoted context omitted.

There are a number of dynamic languages to choose from where tables/dataframes are truly first-class datatypes: perhaps most notably Q[0]. There are also emerging languages like Rye[1] or my own Lil[2]. I suspect that in the fullness of time, mainstream languages will eventually fully incorporate tabular programming in much the same way they have slowly absorbed a variety of idioms traditionally seen as part of funct…

Nushell is another one with tables built-in: https://www.nushell.sh/book/working_with_tables.html

It's interesting how often there are similarities between Numshell, Rye and Lil, although I think they are from different influences. I guess it's sort of current zeitgeist if you want something light, high level and interactive.
Post reply on HN