Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Array Programming with NumPy

nature.com

Post reply on HN
Print view View original

61–70 of 114 posts

Re: Array Programming with NumPy

#61

Post by throwaway-wroc »

jofer wrote: post #28

Don't underestimate the impact this has on getting funding or even just tenure/etc recognition for working on numpy. I'm in industry these days, but coming from the academic side, it's _really_ hard to get recognized for building the underlying infrastructure that tons of people use. I've built and maintained libraries that are used in a ton of publications, but was always told my work was "utterly and completely use…

props to your work and similar to numpy, i assume it has been immensely useful for loads of people.

but 'building the underlying infrastructure that tons of people use' is not science. in my department we had to fail a phd student because 90% of his work was just implementing bunch of existing methods as a python library. useful, yes; science, no. wasn't his fault, had a shitty supervisor, but making useful tools is not the same as undertaking scientific research.

Re: Array Programming with NumPy

#62

Post by uryga »

ktpsns wrote: post #56
mjburgess wrote: post #52

Earlier quoted context omitted.

Er... array programming and statistical programming languages pre-date both by decades. APL from 1966, I believe, is the key lang for array programming. And statistical languages like S from 1976 come to mind: https://en.wikipedia.org/wiki/S_(programming_language) At a quick glance, it seems PDL is just a variation on S.

R ( https://en.wikipedia.org/wiki/R_(programming_language) ) is kind of the successof of S. In certain communities (not only statistics, but for instance also biogenetics), there is quite some concurrency between R and (scientific) Python for data science. For my understanding, the numpy syntax most closely resembles what would be possible in matlab. And matlab again seems to have roots from Fortran. Thanks to that,…

> there is quite some concurrency

i think you meant "competition" here :)

(in polish, my native language, it's "konkurencja", but it's a "false friend of the translator"; i'm guessing you're in a similar boat)

Re: Array Programming with NumPy

#63

Post by sradman »

throrthaway wrote: post #6

Ah, just the right time to publish about numpy - right when everyone is moving over to Julia because of numpy's warts.

Julia is a fine platform for data science but so are Python/NumPY, R, and Matlab. Few of us have the luxury of building greenfield projects that are independent of our previous choices and real-world constraints. Understanding NumPY within this larger context is important, even if you are committed to the Julia ecosystem.

Re: Array Programming with NumPy

#64

Post by MayeulC »

clickok wrote: post #15
lumost wrote: post #10

Earlier quoted context omitted.

There's an interesting trend of what content gets published in peer-reviewed journals vs. blogs/github/etc. I suspect there is an audience segment that strongly values peer reviewed pieces that are equivalent content wise to introductory material in a variety of formats. I wonder if github should add a "Review" feature to provide a similar content authoring experience.

It would be nice if citing repositories were easier-- either for generating a reference for my own code or acknowledging when I've used someone else's code in my research. There's tons of math and physics blogs that contain useful results that the author wanted to make available but didn't manage to incorporate into a paper. I wonder if there'd be any interest in a sort of GitHub for proofs? It could even use git, si…

Owning to the distributed nature of git, and the properties of the hashes it uses, it is probably enough to put a full commit id in a paper to securely reference a software project, regardless of its hosting platform.

We'd just need a dedicated search engine, and a way to automatically extract those from papers, to clone and archive repos.

Re: Array Programming with NumPy

#65

Post by scottlocklin »

jofer wrote: post #28

Don't underestimate the impact this has on getting funding or even just tenure/etc recognition for working on numpy. I'm in industry these days, but coming from the academic side, it's _really_ hard to get recognized for building the underlying infrastructure that tons of people use. I've built and maintained libraries that are used in a ton of publications, but was always told my work was "utterly and completely use…

I know Travis' and Paul Dubois[0] work at Livermore was immediately recognized as of towering importance almost immediately. I was down the highway at LBNL porting shitty Mathematica, IDL[1] and Fortran Diffraction Grating code to Numeric or whatever they called Numpy back then almost as soon as it was released. People probably don't remember their history, but pretty much the only open source intepreters of the day were things like Perl (whose math capabilities at the time were pretty lousy). Scientists paid for a shitload of Maple, Mathematica, IDL, Matlab and Igor[2] licenses; and there still weren't enough licenses to share code with your friends, because nobody had licenses for them all.

Python 1.5 was the first non-mentat tier open source interpreter available that didn't get in your way as a scientist, and Numeric/Numpy was the first and still the most elementary piece that made it usable to science and numerics people. Might not have been letters to Nature tier back then, but Nature ain't what it used to be anyhow.

[0] Since a lot of folks have actually forgotten Paul: http://www.pfdubois.com/bio.html

[1] https://en.wikipedia.org/wiki/IDL_(programming_language)

[2] https://en.wikipedia.org/wiki/IGOR_Pro

Re: Array Programming with NumPy

#67

Post by ivirshup »

coliveira wrote: post #58
Random_ernest wrote: post #46

Earlier quoted context omitted.

I recall a story where a friend was unable to publish a paper in which he wrote an alternative to a very commonly used commercial tool (that virtually everybody used) with roughly 10 times better performance. He open sourced it and all, it was extremely useful, but there was no new methodology, it was simply very well implemented. At a talk of his it lead to a very heated discussion where an older professor accused h…

It is obvious that we need good software, however from the point of view of science the old professor may have reason. If you are receiving a grant, you're not being paid to write software, in the same way that an engineer is not paid to write novels. As useful as the software may be, the person in question should be spending time on research (by definition new subjects), not writing again an existing software.

> If you are receiving a grant, you're not being paid to write software

In my (albeit limited) experience, software is a pretty common deliverable from a grant, at least in computational biology. This has also been my experience with more alternative funding sources like CZI and DARPA.

Taken more broadly, I think there is a huge disconnect between what academics are paid to do, and what takes most of their time. Review is unpaid. Grants are not dependent on which journal the results go into, but time could be saved by aiming lower. A salary can be payed from a research grant, while the investigator still has to teach.

Re: Array Programming with NumPy

#68

Post by sandgiant »

jofer wrote: post #28

Don't underestimate the impact this has on getting funding or even just tenure/etc recognition for working on numpy. I'm in industry these days, but coming from the academic side, it's _really_ hard to get recognized for building the underlying infrastructure that tons of people use. I've built and maintained libraries that are used in a ton of publications, but was always told my work was "utterly and completely use…

I wholeheartedly agree.

I think journal editors have a responsibility here too in promoting references to software libraries used in the articles they publish. I almost never see these in my field (astrophysics), even though they are readily available and very easy to include.

Re: Array Programming with NumPy

#69

Post by throrthaway »

edgyquant wrote: post #49
srean wrote: post #32

Earlier quoted context omitted.

Well that could have been said (and was said) about Numpy/Scipy when it started, "oh R has so many more packages, what numpy can do I can do in MATLAB ...", yet here we are.

I just don't like the BASIC derived syntax of Julia (and Ruby.) I wish there was a language that was typed, had python like classes, subroutines and lambdas but JS like anonymous functions that was fast like Julia or at least close to numpy in number crunching without needing a module written in C.

OCaml?

Re: Array Programming with NumPy

#70

Post by DiogenesKynikos »

coliveira wrote: post #58
Random_ernest wrote: post #46

Earlier quoted context omitted.

I recall a story where a friend was unable to publish a paper in which he wrote an alternative to a very commonly used commercial tool (that virtually everybody used) with roughly 10 times better performance. He open sourced it and all, it was extremely useful, but there was no new methodology, it was simply very well implemented. At a talk of his it lead to a very heated discussion where an older professor accused h…

It is obvious that we need good software, however from the point of view of science the old professor may have reason. If you are receiving a grant, you're not being paid to write software, in the same way that an engineer is not paid to write novels. As useful as the software may be, the person in question should be spending time on research (by definition new subjects), not writing again an existing software.

What if that piece of software increases research output across the entire field? Often, a good piece of scientific software advances research more than what you're calling "research."
Post reply on HN