Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Array Programming with NumPy

nature.com

Post reply on HN
Print view View original

91–100 of 114 posts

Re: Array Programming with NumPy

#91

Post by V1ndaar »

yig wrote: post #55
dr_zoidberg wrote: post #47

Earlier quoted context omitted.

Been there. A few years back I got a government scholarship for my PhD (which is still in progress, due to my follow up work). I basically built the foundation upon which to establish a new field for my university, and the region where I live. There are some professor who think that scholarship (and the little money it gave me) was wasted on my because I chose to build all of that from the ground up, instead of rushi…

For every story like this, I believe there are many more in which the student simply writes their own implementation due to not invented here syndrome or engineering as a form of procrastination.

If you talked to me about my PhD for a few minutes you would surely put me into your "had to reinvent the wheel for no reason" category.

As indeed, I wrote an analysis framework for my data (of a gaseous detector used for axion search) [0] instead of using an existing framework used by my predecessor. However, things are always more complicated than they seem. Many of those not talked about students who rewrite stuff probably have reasons!

In my case the existing framework [1] was a monster that was bent to allow it to work with the kind of data we have in the first place. In my case my detector had several additional features, which fit _even less_ into the existing framework. It would have been a hack and still a significant amount of work to make it work well.

To be fair, when I started this I expected it to be less work than it ended up being. But that's the story of software development.

The advantages now are significant of course. I know the whole codebase. It does exactly what I want. I can extend it easily as I see fit.

That doesn't mean I didn't also partly procrastinate writing software. Far from it. Hell, there was no reason to write a freaking plotting library (a sort of port of ggplot2 for Nim) [3]. But again, this means my thesis will have plots created natively using a TikZ backend while at the same time provide links to Vega-Lite plots for each and every plot in my thesis (which of course will include the data for each plot!).

Finally, the most important point: A university / professor who only pays me for 20h a week does not get to tell me how I do my PhD.

[0]: https://github.com/Vindaar/TimepixAnalysis [1]: https://ilcsoft.desy.de/portal/software_packages/marlintpc/ [2]: https://github.com/Vindaar/ggplotnim

Re: Array Programming with NumPy

#92

Post by wjnc »

throwaway-wroc wrote: post #87
wjnc wrote: post #75

Earlier quoted context omitted.

That's quite the wrong way of approaching science. The scientific method is based on building on the shoulders of giants. Those giants aren't the professors in the direct vicinity nor are they only the papers you cite. The whole of the process is science and if we need further specialization for building better tools (hey, maths and statistics are scientific tools as well) I would classify that as science to a large…

> The scientific method is based on building on the shoulders of giants. > The whole of the process is science no. scientific research is proposing a useful model of an observable phenomenon. this is what you train for during a phd, at least in natural/life sciences: you learn how to test a hypothesis, not an easy skill. refactoring code or transforming bunch of C++ into a python library is useful, but it's not scien…

I'm trained as an economist so might have a different view. But what I think I know from physics is that, say, the people actively involved in engineering things like matter collidors do get authorship or at least appreciation for their role in furthering science.

For me, our discussion is mainly in where to draw the line around "the process of science". The chair, laptop and coffee machines aren't science. The statistical methods, papers and engineering are. You seem to cut parts of the engineering out, namely the non-novel parts. There's a lot to say for that. But a PhD is proof of apprenticeship as well. I wouldn't grant someone a PhD if all of his work is 'mere retooling'. But in a mainly research papers based PhD-application I wouldn't feel some retooling couldn't be allowed. One could demonstrate scientific craftsmanship in retooling.

Re: Array Programming with NumPy

#93

Post by martingoodson »

throwaway-wroc wrote: post #61
jofer wrote: post #28

Don't underestimate the impact this has on getting funding or even just tenure/etc recognition for working on numpy. I'm in industry these days, but coming from the academic side, it's _really_ hard to get recognized for building the underlying infrastructure that tons of people use. I've built and maintained libraries that are used in a ton of publications, but was always told my work was "utterly and completely use…

props to your work and similar to numpy, i assume it has been immensely useful for loads of people. but 'building the underlying infrastructure that tons of people use' is not science. in my department we had to fail a phd student because 90% of his work was just implementing bunch of existing methods as a python library. useful, yes; science, no. wasn't his fault, had a shitty supervisor, but making useful tools is…

This comment displays a remarkable ignorance of scientific history. Why do you think Ramon y Cajal shared the Nobel prize, for discovering neurons, with Golgi, who 'simply' invented the staining method?

Re: Array Programming with NumPy

#94

Post by mantap »

throwaway-wroc wrote: post #90
mantap wrote: post #89

Earlier quoted context omitted.

Maybe we should take all this "not science" software away from the scientists and see how much science they can do without it. If you write code that allows science to be done that couldn't be done otherwise then that is science. As a high profile example, a large amount of specialist software was developed for the LHC to allow it to process all the events coming from the detectors. It sounds like the refactoring her…

yes, in 2020 you mostly cannot do science without software, electricity, desks and chairs and buildings, printers, pick your own irreplaceable tool. yet building these things to enable research is emphatically not itself scientific research. doing a phd -> training to be a scientist.

Printers? Desks? Take your strawmen somewhere else.

Re: Array Programming with NumPy

#95

Post by mratsim »

yig wrote: post #55
dr_zoidberg wrote: post #47

Earlier quoted context omitted.

Been there. A few years back I got a government scholarship for my PhD (which is still in progress, due to my follow up work). I basically built the foundation upon which to establish a new field for my university, and the region where I live. There are some professor who think that scholarship (and the little money it gave me) was wasted on my because I chose to build all of that from the ground up, instead of rushi…

For every story like this, I believe there are many more in which the student simply writes their own implementation due to not invented here syndrome or engineering as a form of procrastination.

Given the current publish or perish culture, I doubt any student serious about publication can afford to waste time.

Re: Array Programming with NumPy

#96

Post by jefft255 »

martingoodson wrote: post #93
throwaway-wroc wrote: post #61

Earlier quoted context omitted.

props to your work and similar to numpy, i assume it has been immensely useful for loads of people. but 'building the underlying infrastructure that tons of people use' is not science. in my department we had to fail a phd student because 90% of his work was just implementing bunch of existing methods as a python library. useful, yes; science, no. wasn't his fault, had a shitty supervisor, but making useful tools is…

This comment displays a remarkable ignorance of scientific history. Why do you think Ramon y Cajal shared the Nobel prize, for discovering neurons, with Golgi, who 'simply' invented the staining method?

Inventing a method isn’t the same a re-implementing an existing method! Again, useful re-implementations are a good thing that people should be rewarded for somehow, but they are not new science.

Re: Array Programming with NumPy

#97

Post by jefft255 »

yqx wrote: post #84
uryga wrote: post #62

Earlier quoted context omitted.

> there is quite some concurrency i think you meant "competition" here :) (in polish, my native language, it's "konkurencja", but it's a "false friend of the translator" ; i'm guessing you're in a similar boat)

Similar false friends in Dutch: Concurrentie

Or French : concurrence :D

Re: Array Programming with NumPy

#98

Post by Mikhail_K »

Fishysoup wrote: post #26
alanbernstein wrote: post #18

Earlier quoted context omitted.

I'm interested in Julia, but if numpy has warts, I suppose Julia has not yet had time to even grow its skin.

Theoretically Julia is better for scientific computing, the only issue is its package ecosystem isn't as mature as Python's. But it's growing incredibly fast and there have already been libraries available for a few years that would be really impractical to write and maintain at such a level in C++ for Python. I assume that for science at least Julia will catch on a lot.

> its package ecosystem isn't as mature as Python's.

Python packages are either interfacing external libraries (something that is much easier to do in Julia) or if they are pure python, badly designed and buggy.

(and package management in python is broken beyond repair)

Re: Array Programming with NumPy

#99

Post by Mikhail_K »

teorema wrote: post #3

For some reason this struck me as inappropriate for the outlet. It's a nice piece as an introduction to array programming with numpy, but seemed out of place to me.

It also leads the readers down the dead-end. Python is incapable of parallelism, the only way to badly emulate it is to launch several runtimes. Writing any scientific software in a badly design language when better alternatives exist is wasted time and effort.

Re: Array Programming with NumPy

#100

Post by mumblemumble »

enriquto wrote: post #66

Young programmers are slowly but steadily reaching the power that Fortran programmers had 40 years ago. That's good news!

That statement only feels true to me if you interpret the word "power" as an exact synonym for "performance." Which is a definition that is valid, but also just about perfect for leading someone to miss the point.

Numpy approaches the performance you could get with Fortran. Mostly because its core is written in Fortran. What Numpy offers that Fortran never did, though, is leverage. The article mentions, but doesn't really do justice to, the sheer volume of interoperability that Numpy has enabled. It's not just that all these libraries were built on top of Numpy. It's also that their common Numpy substrate makes them all deeply interoperable with each other. And that works both above and below the boundary. You can swap out BLAS and LAPACK for something else - say, CUDA, or a distributed representation - and as long as the replacement also speaks Numpy's language, you can plug it into existing libraries that were originally written against Numpy.

In short: Fortran gets you performance. Numpy gets you that, and also productivity. I would argue that that actually makes Numpy more powerful than what was possible with just Fortran.

Post reply on HN