Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python is not a great language for data science

blog.genesmindsmachines.com

Post reply on HN
Print view View original

321–330 of 339 posts

Re: Python is not a great language for data science

#321

Post by northlondoner »

blubber wrote: post #320
steine65 wrote: post #304

Earlier quoted context omitted.

Just in case someone reads this far and sees blubber's confident "No." Blubber is definitely wrong here. I used to do all of my programming in R. Throw the question into an LLM if you're wondering if R has a package like ___ in python.

I know people who used Visual Basic for all of their programming. I'd say No either way unless people explained to me without bursting out into laughter that they also have extensive experience with, e.g., Kotlin, Rust, C#, Java etc. and still prefer VB or R for non-trivial programs.

Of course R isn't a complied language and probably not the same category as C/Rust as systems language but is not in the same category as VB. R is a serious scientific programming language used in non-trivial programs for industrial applications. See Posit's customers. I suggest John Chambers ( https://en.wikipedia.org/wiki/John_Chambers_(statistician) ) book, he explain how he designed S language, R's grandfather so to speak, Software for Data Analysis ( https://link.springer.com/book/10.1007/978-0-387-75936-4 ).

Re: Python is not a great language for data science

#322

Post by willvarfar »

mettamage wrote: post #214
willvarfar wrote: post #200

My experience was that data science was doable but clunky and ugly with pandas. It got slightly better with polars. Only really slightly better. Then, for me at least, it jumped lightyears ahead with duckdb. These days I run some big query on an OLAP database and download the results to parquet stored on the local disk of a cloud notebook VM and then mine it to bits with duckdb reading straight from these parquet fil…

Huh, as a frequent polars user, I'll try duckdb.

I think you can query and join polars dataframes if you call to_arrow() on them too.

Re: Python is not a great language for data science

#323

Post by drtc »

I work as a data scientist where I do lots of exploratory work using Python+Pandas/Polars+Jupyter notebooks, I have to say that I agree with the title of the article.

I feel like I'm using Python more and more in a way that is just not native to it. Strict typing is simply necessary at some point in time, but I don't get any of the performance benefits or compile-time warnings that other languages provide. But more than Pandas/Polars/DuckDB, I think it is the plotting ecosystem that keeps me in the Python universe. Seaborn, altair, plotnine all take ggplot's GoG and bring it to Python and I'm really grateful for that.

I don't see an alternative to Python for someone who needs to: 1) Work with data that fits in memory and is mostly tabular (thus Pandas/Python), 2) needs to visualize this data often, 3) does exploratory work (Jupyter notebooks).

Re: Python is not a great language for data science

#324

Post by Cosi1125 »

blubber wrote: post #320
steine65 wrote: post #304

Earlier quoted context omitted.

Just in case someone reads this far and sees blubber's confident "No." Blubber is definitely wrong here. I used to do all of my programming in R. Throw the question into an LLM if you're wondering if R has a package like ___ in python.

I know people who used Visual Basic for all of their programming. I'd say No either way unless people explained to me without bursting out into laughter that they also have extensive experience with, e.g., Kotlin, Rust, C#, Java etc. and still prefer VB or R for non-trivial programs.

I do.

And I'm still waiting for your examples of "established R packages with bugs caused by R weirdness".

Re: Python is not a great language for data science

#325

Post by fnord77 »

hekkle wrote: post #102
fnord77 wrote: post #93

Python is not a great language

Not great at what? I agree that Python is not great at anything specifically, but it is good at almost everything, and that's what makes it great.

which is nicer?

    float(len(str))
or

    str.len().float()
Also, there's numerous ways of doing the same thing, despite that being contrary to python's philosophy

Re: Python is not a great language for data science

#326

Post by cmontella »

fluorinerocket wrote: post #313
OkayPhysicist wrote: post #120

Earlier quoted context omitted.

There's a number of structures that I think are missing in our major programming languages. Tables are one. Matrices are another. Graphs, and relatedly, state machines are tools that are grossly underused because of bad language-level support. Finally, not a structure per se, but I think most languages that are batteries-included enough to included a regex engine should have a a full-fledged PEG parsing engines. Most…

Would love to see a language in which hierarchical state machines, math/linear algebra, I/O to sensors and actuators, and time/timing were first class citizens. Mainly for programming control systems for robotics and aerospace applications

You might be interested in following my project, as I'm trying to support exactly that nexus of features for those applications!

https://github.com/mech-lang/mech https://www.hytradboi.com/2022/i-tried-rubbing-a-database-on...

Not all features are implemented yet but we are getting there!

Re: Python is not a great language for data science

#328

Post by _yb2s »

3eb7988a1663 wrote: post #165
_yb2s wrote: post #145

Doing computational biology for several decades in about a dozen languages, I do think R is a much better language for data science, but in practice I end up using Python almost every time because it has more libraries, and it’s easier to find software engineers and collaborators to work on Python. However, R makes for much simpler cleaner code, less silent errors, and the 1 indexing makes dealing with biological seq…

Pardon? Less silent errors? R has quite a few foot guns around permissively parsing user intention. Which does make it handy for exploratory analysis, but a lot more fragile when you want production code. Just a simple one that can get you, R is 1-indexed. Yet if you have a vector, accessing myvec[0] is not an error. Alternatively, if you had say, a vector length of 3 and do myvec[10] that gets NA (an otherwise legal…

All languages have things like that, but from my experience developing a number of widely used scientific software packages for both Python and R, those sort of bugs happen a lot more in Python. Rs copy on write, for example makes it a much more functional language without unintended side effects- Python is very inconsistent in that regard, with different variable types and data structures behaving entirely different in identical situations.

Re: Python is not a great language for data science

#329

Post by 7thaccount »

iLemming wrote: post #305
7thaccount wrote: post #164

Earlier quoted context omitted.

It might not be a problem for you, but it has been for many. I did start by reading through 3 Clojure books. The repl and the basic stuff like using lists is all easy of course, but the tooling was pretty poor compared to what I was used to (I like lisp, but Emacs is a commitment). Also, a lot of tutorials at the time definitely assumed java familiarity, especially with debugging java stack traces.

> It might not be a problem for you, but it has been for many Do you have a habit of referring to yourself in plural, or do you typically like to generalize things based on your personal experiences? I personally know many Clojurists who never had problems you're describing - hundreds of people. Sure, that could be the case of survivorship bias, perhaps I just don't befriend people who struggled with getting into Clo…

I've had this discussion here on HN several times over the years. Lots of comments from others have pointed out similar experiences. I'm guessing your experience was more positive and that's great to hear.

I did point out that maybe things had changed a good bit (literally said maybe VSCode made that easier now as it has for other tools) and tried to make it clear that my experience was a bit dated.

As far as excuses go, I don't see how that's relevant. I just pointed out I had issues with a steep learning curve when I was seriously considering it many years ago along with other languages that are hosted on the JVM (Scala, Kotlin) or .NET (F#). Nothing against those languages, but all the tutorials and even many of the books at the time would frequently borrow from the host language in weird ways. Like I'd have to use some random Java library and when it didn't work, had no idea how to troubleshoot why it wasn't there and I didn't want to have to go learn Java first.

I own at least two books on F# and talked with some prominent authors personally and they admitted it was really geared towards intermediate or greater C# users who wanted to move over to functional programming. I could have stuck with it, but decided to stick with other tools.

Clojure certainly is nice and I wanted to take advantage of it...it just ended up not being as ergonomic for my needs as I had hoped.

Post reply on HN