Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python is not a great language for data science

blog.genesmindsmachines.com

Post reply on HN
Print view View original

301–310 of 339 posts

Re: Python is not a great language for data science

#301

Post by timbit42 »

RobinL wrote: post #16

I think a lot of this comes down to the question: Why aren't tables first class citizens in programming languages? If you step back, it's kind of weird that there's no mainstream programming language that has tables as first class citizens. Instead, we're stuck learning multiple APIs (polars, pandas) which are effectively programming languages for tables. R is perhaps the closest, because it has data.frame as a 'firs…

The 3rd edition of Dartmouth BASIC, back in the 1960's, had a MAT command for dealing with matrices.

Re: Python is not a great language for data science

#302

Post by redwall_hp »

RobinL wrote: post #46
CivBase wrote: post #31

Earlier quoted context omitted.

What is a table other than an array of structs?

I would argue that's about how the data is stored. What I'm trying to express is the idea of the programming language itself supporting high level tabular abstractions/transformations such as grouping, aggregation, joins and so on.

Map/filter/reduce are idiomatic Java/Kotlin/Scala.

SELECT thing1, thing2 FROM things WHERE thing2 != 2;

val thingMap = things.map { it.thing2 to it.thing2 }.filter { it.thing2 !=2 }

Then you've got distinct(), sorting methods, take/drop for limits, count/sumOf/average/minOf/maxOf.

There are set operations, so you can do unions and differences, check for presence, etc.

Joins are the hard part, but map() and some lambda work can pull it off.

Re: Python is not a great language for data science

#303

Post by forgotpwd16 »

puzzlingcaptcha wrote: post #246
forgotpwd16 wrote: post #3

Article is well written but fails to address its own thesis by postponing it to a sequel article. At its current state only alludes that Python is not great because requires specialized packages. (And counterexample is R for which also used a package.)

The 'sequel' is also online: https://blog.genesmindsmachines.com/p/python-is-not-a-great-...

Thanks! In such serial articles usually there's link to the end pointing to the next one so, since there wasn't any, thought next one hadn't been written. This one indeed addresses the thesis. The TL;DR, taken directly from the article,

>The core problems I see with Python as a language for data science are call-by-reference semantics, lack of built-in concepts of missing values, lack of built-in vectorization, and lack of non-standard evaluation.

Re: Python is not a great language for data science

#304

Post by steine65 »

northlondoner wrote: post #298
blubber wrote: post #241

Earlier quoted context omitted.

No, it's not. Even established packages have bugs caused by R weirdness. I like it nevertheless.

Yes, R is a proper general purpose programming language. Turing complete, functional, procedural, object oriented.../

Just in case someone reads this far and sees blubber's confident "No." Blubber is definitely wrong here. I used to do all of my programming in R. Throw the question into an LLM if you're wondering if R has a package like ___ in python.

Re: Python is not a great language for data science

#305

Post by iLemming »

7thaccount wrote: post #164
iLemming wrote: post #134

Earlier quoted context omitted.

None of this even remotely true. I've gotten into Clojure without knowing jackshit about Java, almost ten years later, after tons of things successfully built and deployed, still don't know jackshit about Java. Mia, co-host of 'Clojure apropos' podcast was my colleague, we've worked together on multiple teams, she learned Clojure as her very first PL. Later she tried learning some Java and she was shocked how impossi…

It might not be a problem for you, but it has been for many. I did start by reading through 3 Clojure books. The repl and the basic stuff like using lists is all easy of course, but the tooling was pretty poor compared to what I was used to (I like lisp, but Emacs is a commitment). Also, a lot of tutorials at the time definitely assumed java familiarity, especially with debugging java stack traces.

> It might not be a problem for you, but it has been for many

Do you have a habit of referring to yourself in plural, or do you typically like to generalize things based on your personal experiences?

I personally know many Clojurists who never had problems you're describing - hundreds of people. Sure, that could be the case of survivorship bias, perhaps I just don't befriend people who struggled with getting into Clojure specifically in a way you're describing. But like they say: "Those who are willing to make the effort will find the solutions. Those who aren't will find the excuses."

Clojure undeniably had challenges in the past, and still has some today. But not the things you're talking about. This is literally not an exaggeration - it's as easy as installing Calva extention for VSCode - that's all one needs to mess around with Clojure.

Re: Python is not a great language for data science

#306

Post by ZhiqiangWang »

rossdavidh wrote: post #114

Speaking as a python programmer who has occasionally done work in R: yes, of course. Python is not a great language for anything; it's a pretty good language for just about anything. That is, and always has been, its strength. If you're doing data science all day, you should learn R, even if it's so weird at first (for somebody coming from a C-style language) that it seems way harder; R is made for the way statistici…

Agree, the argument is well made in sklearn API design paper https://arxiv.org/abs/1309.0238

Re: Python is not a great language for data science

#308

Post by iLemming »

dreamcompiler wrote: post #171
iLemming wrote: post #139

Earlier quoted context omitted.

I don't have great experience of using CL for analyzing data, because of "why?", if I already have another Lisp that is simply amazing for data. Clojure, unlike lists in traditional Lisps, based on composable, unified abstraction for its collections, they are lazy by default and literal readable data structures, they are far easier to introspect and not so "opaque" compared to anything - not just CL (even Python), th…

Homework assignments notwithstanding, very few serious Common Lisp programs use lists and symbols as their primary data structures. This has been true since around 1985. Common Lisp has O[1] vectors, multidimensional arrays, hash-tables (what Clojure calls maps), structs, and objects. It has set operations too but it doesn't enforce membership uniqueness. It also has bignums, several sizes of floats, infinite-precisi…

I wasn't trying to denigrate Common Lisp, I'm sorry if I hurt your feelings. It does have comprehensive support for all kinds of data structures. I wasn't talking it being limited to "list processing". SBCL is great for many things, but from many practical points Clojure actually much better suited for data analysis.

You're saying: "hash-tables (what Clojure calls maps)" not only inaccurate, you're hand-waving Clojure's core design philosophy (immutability, structural sharing, lazy sequences) as orthogonal. But those aren't cosmetic differences - they're the reason why Clojure's data structures are fundamentally better for data analysis. I think you're confusing "having equivalent data types" with "solving the same problem the same way"

Post reply on HN