From Python to NumPy (2017) • Hacker News Board

Re: From Python to NumPy (2017)

#31

Post by nemothekid » Fri, Jun 10, 2022, 5:05 AM UTC

z9znz wrote: post #23

boberoni wrote: post #19

Earlier quoted context omitted.

> Python has a great/dope community Python's community is one of the greatest and worst parts of the language. Python might not be the best language for anything, but it's the second best language for everything. I've dabbled in many programming languages, a lot of which are backed by a single large company or consultancy: - Java (Oracle, at least initially) - TypeScript, C# (Microsoft) - Golang, Dart (Google) - Elix…

> Python's community is one of the greatest and worst parts of the language. If you come from a Ruby background, and you are now doing Python, you may find yourself on Stackoverflow looking up how to do X idomatically in Python (something you did all the time in Ruby). Or you may ask colleagues who know Python but do not know Ruby. A common answer from the Pythonista community is, "Why would you ever want to do that?…

I'm not a Pythonista by any means, but these complaints are banal. Ternary operators? Plenty of new popular languages don't have a ternery operator (Go, Rust, Elixir, Kotlin, Nim).

My list of python gripes are long, but language constructs are not on the list.

Re: From Python to NumPy (2017)

#32

Post by z9znz » Fri, Jun 10, 2022, 5:13 AM UTC

analog31 wrote: post #29

z9znz wrote: post #25

Earlier quoted context omitted.

> Knowledge of a language or code quality is rarely the hard or important part. That is true. The problem is, that code in Python of questionable quality (but real utility) is now turned into production code; and it is done so by building Python stuff around it. So now the insignificant implementation detail which was the language choice is now driving the entire project. Also, a PhD data scientist/mathematician can…

To me, this seems like the wrong way of doing it. I'm a scientist, and I code in Python, but my code is never directly incorporated into the production code base. (Similar story for the portions of my work involving electronics, mechanics, etc). A decent generalization is that once I solve a problem, and demonstrate a result using Python, the actual amount of code that needs to be understood and translated into the p…

If you run a company, then you might make better choices (or at least allow for proper discovery of what approach to take to building a large system).

But there seems to be an accepted rule that "if you're doing data science, you are a Python shop". And if you are trying to pitch a data science company to investors, they know Python for data science.

New companies start all the time using a technology because "that's what you use", without knowing why or why not. In fact, (and not that I idolize him, which I really do not), Paul Graham's http://www.paulgraham.com/avg.html essay talks about this. Most companies follow whatever is the accepted practice for doing things. It may "work", but it definitely may not be optimal. And the companies that choose a different path based on more consideration or more awareness may have an easier time.

I mean after all, there was a time when everyone wrote C (because C was so much better than COBOL or whatever). And believe it or not, there was a time when it was impossible to convince a company to try Java. Or as the old saying went, "nobody got fired for buying IBM". After all, it was the safe path. Substitute Oracle, or C, or C++, or Java, or Python, or PHP, or Windows (there was a time when people didn't trust Linux and couldn't imagine why you would ever try to run your business software on Linux machines).

Your reasoning makes sense, but in practice in the software world, it is not as well planned or thought out as you might expect.

Re: From Python to NumPy (2017)

#33

Post by z9znz » Fri, Jun 10, 2022, 5:25 AM UTC

nemothekid wrote: post #31

z9znz wrote: post #23

Earlier quoted context omitted.

> Python's community is one of the greatest and worst parts of the language. If you come from a Ruby background, and you are now doing Python, you may find yourself on Stackoverflow looking up how to do X idomatically in Python (something you did all the time in Ruby). Or you may ask colleagues who know Python but do not know Ruby. A common answer from the Pythonista community is, "Why would you ever want to do that?…

I'm not a Pythonista by any means, but these complaints are banal. Ternary operators? Plenty of new popular languages don't have a ternery operator (Go, Rust, Elixir, Kotlin, Nim). My list of python gripes are long, but language constructs are not on the list.

The point is about inconsistency.

You can say "x = 1 if expr else 2", but you cannot say "x = 1 if expr". You do a two line

if expr:

    x = 1

You cannot say "x = 1 if expr else return".

[1] is a list with one item, 1.

{1} is a set with one item, 1.

(1) is a number, 1, with pointless parenthesis. Perhaps you mean (1,), a tuple containing only one value.

Or how about mutable default values in function paramters? That makes a lot of sense...

I don't need to keep offering examples. You can search the internet and find many places where people list the warts, inconsistencies, and misfeatures of Python.

Re: From Python to NumPy (2017)

#34

Post by wodenokoto » Fri, Jun 10, 2022, 5:30 AM UTC

I feel like I never use directly numpy anymore. All my data is in Pandas, and I mostly import numpy for the odd `df.transform(np.somefunction)`

I remember it being quite fun to build these vectorized numpy functions (in the crosswords and puzzles are fun, kind of way)

Figuring out how to make a transformation in Pandas is always frustrating, when it isn't straightforward.

Re: From Python to NumPy (2017)

#35

Post by pbronez » Fri, Jun 10, 2022, 5:30 AM UTC

z9znz wrote: post #32

analog31 wrote: post #29

Earlier quoted context omitted.

To me, this seems like the wrong way of doing it. I'm a scientist, and I code in Python, but my code is never directly incorporated into the production code base. (Similar story for the portions of my work involving electronics, mechanics, etc). A decent generalization is that once I solve a problem, and demonstrate a result using Python, the actual amount of code that needs to be understood and translated into the p…

If you run a company, then you might make better choices (or at least allow for proper discovery of what approach to take to building a large system). But there seems to be an accepted rule that "if you're doing data science, you are a Python shop". And if you are trying to pitch a data science company to investors, they know Python for data science. New companies start all the time using a technology because "that's…

For machine learning companies, Julia is probably the dark horse language that could give you a competitive advantage. Like always, the problem with using something more niche is the ecosystem strength. You need to train people more, build more tooling yourself. Still, there is performance to be had if you work for it.

Re: From Python to NumPy (2017)

#36

Post by oivey » Fri, Jun 10, 2022, 5:46 AM UTC

z9znz wrote: post #25

oivey wrote: post #18

Earlier quoted context omitted.

The “luck” Python ran into was that some people came together to write NumPy. The alternatives at the time were Matlab and R. Matlab is not a great general programming language and is expensive. R has esoteric syntax and also isn’t a great general purpose language. This shift didn’t happen so long ago that it can’t be remembered. It certainly didn’t happen because it was installed by default by some Linux distros. Fo…

> Knowledge of a language or code quality is rarely the hard or important part. That is true. The problem is, that code in Python of questionable quality (but real utility) is now turned into production code; and it is done so by building Python stuff around it. So now the insignificant implementation detail which was the language choice is now driving the entire project. Also, a PhD data scientist/mathematician can…

Data scientists are hired for a certain set of skills, and software engineers are hired for a different set. That’s the reality of specialization. I don’t often see data scientists complaining that software engineers are bad at statistics. I’m not sure why so many software engineers feel the need to complain that data scientists are bad at software engineering.

At my job I work both sides of the fence. Sometimes I write research code in Python/Julia/Matlab. Sometimes I write production code in C++/CUDA. The choice of production language is primarily due to performance needs rather than code quality, though.

I’ll repeat myself but slightly differently: the main reason Python is used over, say, Ruby is because Ruby lacks a comparatively good library for working with arrays. It subsequently didn’t develop all the ecosystem for ML and scientific computing (SciPy, Matplotlib, sklearn, skimage, Torch, etc). It’s basically never going to catch up. And when it comes down to it, Ruby is basically as ill-suited as Python, Clojure, Elixir, and most other languages when it comes to doing something like automatic differentiation on a GPU. And again, even so, Python is still the best choice among those because so many resources have been poured into PyTorch, Matplotlib, NumPy, etc.

Re: From Python to NumPy (2017)

#37

Post by z9znz » Fri, Jun 10, 2022, 6:07 AM UTC

oivey wrote: post #36

z9znz wrote: post #25

Earlier quoted context omitted.

> Knowledge of a language or code quality is rarely the hard or important part. That is true. The problem is, that code in Python of questionable quality (but real utility) is now turned into production code; and it is done so by building Python stuff around it. So now the insignificant implementation detail which was the language choice is now driving the entire project. Also, a PhD data scientist/mathematician can…

Data scientists are hired for a certain set of skills, and software engineers are hired for a different set. That’s the reality of specialization. I don’t often see data scientists complaining that software engineers are bad at statistics. I’m not sure why so many software engineers feel the need to complain that data scientists are bad at software engineering. At my job I work both sides of the fence. Sometimes I wr…

I'm not attacking data scientists. I have no objection to how they solve their problems.

My point is that their choice of Python (because of the libraries they use, or the training they got in school?) bleeds out and causes Python to be used in places where it doesn't fit so well.

If the libraries you mention were written for Ruby (and they absolutely could have been... probably with fewer defects and fewer lines of code), then we wouldn't be having this discussion.

There is no use of any of those libraries that someone could demonstrate which could not be equally or better demonstrated in one of the other languages you mention. Therefore, the only argument for using Python is that it's what those libraries are written in.

In fact, with a little effort an API could be written and wrapped around those libraries to make them available to other languages, decoupling the bigger projects from the library-specific logic. This is how it should be done at any reasonable company anyway. The data science wizards build their specific systems in whatever language they need, and that gets built into a minimal service. But things often start highly coupled, meaning the special code in Python gets business logic and persistence stuff built around it also in Python.

Re: From Python to NumPy (2017)

#38

Post by coderenegade » Fri, Jun 10, 2022, 6:14 AM UTC

z9znz wrote: post #21

boredemployee wrote: post #13

Earlier quoted context omitted.

Sorry, that discussion is so overrated and elitist. I'll never understand people complaining about one technology/language/tool that made it possible to dozens of people to join to the - once restrict - small club which was computer science and which is now making it possible to a lot of people to get a job with an easy to learn and less verbose language. One can argue about speed/memory/better languages out there to…

Use any language you like to build a proof of concept. But one you need to take that POC and build a real product which needs to be supported, extended, and maintained, then you must choose tools which fit that long term need. Many of use first experienced school in kindergarten. Kindergarten was an important step in our education and socialization. But we didn't refuse to move on to the next level, no matter how muc…

The issue with a lot of the gripes with Python is that there's no other language that combines mature linear algebra libraries, scientific computing libraries, and industry standard computer vision libraries with easy to use basic tools like csv reading, dataframes, json parsing, etc., while also providing easy syntax and fast iteration time. I work in C++ and Python, and things that are completely trivial in Python have a lead time measured in days, if not weeks when moving to C++. This is before even getting into the weeds of writing bug-free C++ code; you need to be familiar with the nitty-gritty details of the library you're working with, because things you might take for granted may not work the way you think.

Sure, maybe we should be rewriting POC code in a language that's more fit-for-purpose if we're talking about a large project, but for that you need mature libraries, or a team with some fairly serious chops (i.e. decent math background, senior coders, expensive to hire, and hard to replace). There's a lot to be said for a language that makes it easy to be productive on technically demanding problems, while also making it easy to find new talent. As someone who's written code relying on linear algebra and CV in Rust and C#, and I'd honestly pick Python every time for those types of jobs. It's not perfect, but that's not the benchmark.

Re: From Python to NumPy (2017)

#39

Post by oivey » Fri, Jun 10, 2022, 7:00 AM UTC

z9znz wrote: post #37

oivey wrote: post #36

Earlier quoted context omitted.

Data scientists are hired for a certain set of skills, and software engineers are hired for a different set. That’s the reality of specialization. I don’t often see data scientists complaining that software engineers are bad at statistics. I’m not sure why so many software engineers feel the need to complain that data scientists are bad at software engineering. At my job I work both sides of the fence. Sometimes I wr…

I'm not attacking data scientists. I have no objection to how they solve their problems. My point is that their choice of Python (because of the libraries they use, or the training they got in school?) bleeds out and causes Python to be used in places where it doesn't fit so well. If the libraries you mention were written for Ruby (and they absolutely could have been... probably with fewer defects and fewer lines of…

> My point is that their choice of Python (because of the libraries they use, or the training they got in school?) bleeds out and causes Python to be used in places where it doesn't fit so well.

They use those libraries because there aren’t alternatives in the languages you prefer. Millions of man hours went into developing the Python ecosystem.

> If the libraries you mention were written for Ruby (and they absolutely could have been... probably with fewer defects and fewer lines of code), then we wouldn't be having this discussion.

This probably isn’t true because the core of NumPy is C. A theoretical NumRs would be the same. The issue with both languages is that they’re slow, so you have to resort to C wrappers. NumRs might be incrementally better at best. It wouldn’t be a revolution.

> There is no use of any of those libraries that someone could demonstrate which could not be equally or better demonstrated in one of the other languages you mention. Therefore, the only argument for using Python is that it's what those libraries are written in.

But no one wrote these libraries in Ruby. Someone had to do it, and no one did. Someone did it in Python. It would be impossible to justify recreating NumPy in Ruby today. If I want to do some simple image processing, Fourier transforms, simple optimization, and plotting in Python it’s trivial. If I want to do it in Ruby it is a ton of work.

> In fact, with a little effort an API could be written and wrapped around those libraries to make them available to other languages, decoupling the bigger projects from the library-specific logic. This is how it should be done at any reasonable company anyway. The data science wizards build their specific systems in whatever language they need, and that gets built into a minimal service. But things often start highly coupled, meaning the special code in Python gets business logic and persistence stuff built around it also in Python.

This isn’t a company scale problem. The Python numerical computing stack has developed over 20 years with enormous investment from companies and universities. Code could maybe be shared via an API, but someone would have to do it. It would have to be done very carefully to guarantee performance. If it were easy you’d expect to see significant code reuse between, for example, Ruby and Java web frameworks. I guess you could say this API does exist, and it’s called C FFI interfaces. Sane companies do use this, but it still sucks pretty bad.

Python’s lead isn’t unassailable, but any language that is just another C wrapper isn’t going to unseat it. Whatever does it will need to be compiled. I’d bet on it being dynamic with type inference, too, because static type checking almost never helps with correctness issues in numerical code while adding significant cognitive overhead. If it isn’t Julia, Julia is at least the blueprint.

Re: From Python to NumPy (2017)

#40

Post by pen2l » Fri, Jun 10, 2022, 7:04 AM UTC

z9znz wrote: post #8

Numpy is a great thing, and also a terrible thing (because it is built on Python). Python is a terrible language compared to many others, not the least of which is Ruby (and I would include Java, Clojure, Elixir, and even C++ and probably C# and almost certainly F# if I knew them well). To make matters worse, there's a common mentality amongst Pythonistas of being aggressively complacent. A typical response to a ques…

It's a joy to work with, it's perfect for glue work, it reads like pseudo-code which makes it accessible for the non-programmers and children, it has a bustling community which means it's easy to find solutions should one encounter problems, it has a flourishing library, it's a great gateway language. For these reasons and more I posit that Python was perhaps one of the great milestones in the history of c̶o̶m̶p̶u̶t̶e̶r̶s̶ mankind, and Guido probably deserves a Turing award.