Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Array Programming with NumPy

nature.com

Post reply on HN
Print view View original

51–60 of 114 posts

Re: Array Programming with NumPy

#51

Post by mjburgess »

austincheney wrote: post #34
teej wrote: post #30

Earlier quoted context omitted.

> my work was "utterly and completely useless I can never understand the arrogance of folks who would say something like this

I used to maintain a code beautifier/diff tool and hear the same things about how the idea was useless only to see those same people shortly there after use my tool or a close competitor. Once you see that pattern a few times you learn to ignore it in it’s entirety. It’s hard to tell why people behave like that. I presume it’s because many people have a great fear of originality and require social validation.

Asymmetry between complexity of principle and power.

Ie., we are easily persuaded that something very complex will be very powerful (eg., a smart phone) -- but we intuitively regard something simple (eg., a hammer) as under-powered.

Hard to say how well this actually holds, but I'd guess in both cases we arent really enumerating use-cases in our head, we're just using explanatory complexity as a guide to practical power.

This is probably more extreme in cases where people have a specific notion of complexity in mind, eg., in academic environments where "tool A" is as simple as "tool B" if they use the same theoretical basis.

ie., Tool C is worthwhile if it includes a more complex theory, as therefore it is more powerful.

Re: Array Programming with NumPy

#52

Post by mjburgess »

hoytech wrote: post #35

Let's not forget to give at least some credit to Perl Data Langauge (PDL). It pioneered a lot of these ideas 10 years before NumPy existed, and is still a pretty great tool today: http://pdl.perl.org/index.php?page=FirstSteps

Er... array programming and statistical programming languages pre-date both by decades.

APL from 1966, I believe, is the key lang for array programming.

And statistical languages like S from 1976 come to mind: https://en.wikipedia.org/wiki/S_(programming_language)

At a quick glance, it seems PDL is just a variation on S.

Re: Array Programming with NumPy

#53

Post by westurner »

clickok wrote: post #15
lumost wrote: post #10

Earlier quoted context omitted.

There's an interesting trend of what content gets published in peer-reviewed journals vs. blogs/github/etc. I suspect there is an audience segment that strongly values peer reviewed pieces that are equivalent content wise to introductory material in a variety of formats. I wonder if github should add a "Review" feature to provide a similar content authoring experience.

It would be nice if citing repositories were easier-- either for generating a reference for my own code or acknowledging when I've used someone else's code in my research. There's tons of math and physics blogs that contain useful results that the author wanted to make available but didn't manage to incorporate into a paper. I wonder if there'd be any interest in a sort of GitHub for proofs? It could even use git, si…

You can get a free DOI for and archive a tag of a Git repo with FigShare or Zenodo.

If you have repo2docker REES dependency scripts (requirements.txt, environment.yml, postInstall,) in your repo, a BinderHub like https://mybinder.org can build and cache a container image and launch a (free) instance in a k8s cloud.

Journals haven't yet integrated with BinderHub.

Putting the suggested citation and DOI URI/URL in your README and cataloging citations in an e.g. wiki page may increase the crucial frequency of citation.

A Linked Data format for presenting well-formed arguments with #StructuredPremises would help to realize the potential of the web as a graph of resources which may satisfy formal inclusion criteria for #LinkedMetaAnalyses.

Re: Array Programming with NumPy

#54

Post by pjmlp »

1vuio0pswjnm7 wrote: post #33

BLAS and LAPACK were originally written for FORTRAN, and FORTRAN is particularly suited to numerical computing and array programming. https://modelingguru.nasa.gov/docs/DOC-1762

And quite modern with recent language revisions, including OOP, generics, modules and first class support for C interoperability.

Re: Array Programming with NumPy

#55

Post by yig »

dr_zoidberg wrote: post #47
Random_ernest wrote: post #46

Earlier quoted context omitted.

I recall a story where a friend was unable to publish a paper in which he wrote an alternative to a very commonly used commercial tool (that virtually everybody used) with roughly 10 times better performance. He open sourced it and all, it was extremely useful, but there was no new methodology, it was simply very well implemented. At a talk of his it lead to a very heated discussion where an older professor accused h…

Been there. A few years back I got a government scholarship for my PhD (which is still in progress, due to my follow up work). I basically built the foundation upon which to establish a new field for my university, and the region where I live. There are some professor who think that scholarship (and the little money it gave me) was wasted on my because I chose to build all of that from the ground up, instead of rushi…

For every story like this, I believe there are many more in which the student simply writes their own implementation due to not invented here syndrome or engineering as a form of procrastination.

Re: Array Programming with NumPy

#56

Post by ktpsns »

mjburgess wrote: post #52
hoytech wrote: post #35

Let's not forget to give at least some credit to Perl Data Langauge (PDL). It pioneered a lot of these ideas 10 years before NumPy existed, and is still a pretty great tool today: http://pdl.perl.org/index.php?page=FirstSteps

Er... array programming and statistical programming languages pre-date both by decades. APL from 1966, I believe, is the key lang for array programming. And statistical languages like S from 1976 come to mind: https://en.wikipedia.org/wiki/S_(programming_language) At a quick glance, it seems PDL is just a variation on S.

R (https://en.wikipedia.org/wiki/R_(programming_language) ) is kind of the successof of S. In certain communities (not only statistics, but for instance also biogenetics), there is quite some concurrency between R and (scientific) Python for data science.

For my understanding, the numpy syntax most closely resembles what would be possible in matlab. And matlab again seems to have roots from Fortran. Thanks to that, young folks nowadays can switch so easily between Fortran and Numpy, the syntax and call structures can easily be made to almost fit to each other.

Re: Array Programming with NumPy

#57

Post by sradman »

1vuio0pswjnm7 wrote: post #33

BLAS and LAPACK were originally written for FORTRAN, and FORTRAN is particularly suited to numerical computing and array programming. https://modelingguru.nasa.gov/docs/DOC-1762

Indeed, FORTRAN has traditionally been used for vector and matrix math and NumPY adds the equivalent functionality to Python. The simple linear algebra abstractions for manipulating these structures are fundamental to each of the popular data science platforms.

MATLAB was created as an interface to LINPACK/EISPACK without having to learn FORTRAN. The importance of this comment is the emphasis on the core fundamentals shared by all the data science platforms rather than the different tradeoffs inherent in each ecosystem.

Re: Array Programming with NumPy

#58

Post by coliveira »

Random_ernest wrote: post #46
jofer wrote: post #28

Don't underestimate the impact this has on getting funding or even just tenure/etc recognition for working on numpy. I'm in industry these days, but coming from the academic side, it's _really_ hard to get recognized for building the underlying infrastructure that tons of people use. I've built and maintained libraries that are used in a ton of publications, but was always told my work was "utterly and completely use…

I recall a story where a friend was unable to publish a paper in which he wrote an alternative to a very commonly used commercial tool (that virtually everybody used) with roughly 10 times better performance. He open sourced it and all, it was extremely useful, but there was no new methodology, it was simply very well implemented. At a talk of his it lead to a very heated discussion where an older professor accused h…

It is obvious that we need good software, however from the point of view of science the old professor may have reason. If you are receiving a grant, you're not being paid to write software, in the same way that an engineer is not paid to write novels. As useful as the software may be, the person in question should be spending time on research (by definition new subjects), not writing again an existing software.

Re: Array Programming with NumPy

#59

Post by coliveira »

yig wrote: post #55
dr_zoidberg wrote: post #47

Earlier quoted context omitted.

Been there. A few years back I got a government scholarship for my PhD (which is still in progress, due to my follow up work). I basically built the foundation upon which to establish a new field for my university, and the region where I live. There are some professor who think that scholarship (and the little money it gave me) was wasted on my because I chose to build all of that from the ground up, instead of rushi…

For every story like this, I believe there are many more in which the student simply writes their own implementation due to not invented here syndrome or engineering as a form of procrastination.

Yes, it is easy to be sidetracked on writing software. Not that software is a bad thing, but research is something different.

Re: Array Programming with NumPy

#60

Post by jcelerier »

teej wrote: post #30
jofer wrote: post #28

Don't underestimate the impact this has on getting funding or even just tenure/etc recognition for working on numpy. I'm in industry these days, but coming from the academic side, it's _really_ hard to get recognized for building the underlying infrastructure that tons of people use. I've built and maintained libraries that are used in a ton of publications, but was always told my work was "utterly and completely use…

> my work was "utterly and completely useless I can never understand the arrogance of folks who would say something like this

as a former phd student, it's pretty systemic - labs sometimes get funding as a direct mapping with how many papers get published by that lab. If you do cool work but that does not land you a paper, you are literally wasting the budget that your employer spent on you. Once you solve that (and thanks to things like this numpy papers, things may be starting to evolve !)
Post reply on HN