Why Is NumPy Only Now Getting Funded? • Hacker News Board

Re: Why Is NumPy Only Now Getting Funded?

#101

Post by mattkrause » Wed, Jun 28, 2017, 6:15 PM UTC

jacobolus wrote: post #61

mattkrause wrote: post #58

Earlier quoted context omitted.

On the other hand, one of the other major data analysis platforms went the other way: R is a free implementation of S language. TIBCO still sells S-PLUS, but my impression is that R is much, much more popular. My impression is that Octave always lags a little behind Matlab, both in terms of features and performance, since the matlab language is essentially whatever The Mathworks says it is. R and S-PLUS both have a s…

How much money does R make? Are the main R contributors working as processional academics, do they have corporate jobs that pay them to work part-time on R, or are they funded full time by donations/grants, or is R just a hobby side project for them? I’m not saying there’s anything wrong with Octave, or R, or GIMP, or ..., only that it’s difficult to make money on it. This is especially true for Octave though, becaus…

I'm not sure either, but I would guess it's a mix. It seems like it's most of Hadley Wickham's career, for example, and there's clearly some work being done by the RStudio and Revolution Analytics folks, but a lot of the packages are clearly the product of people working on their academic or corporate jobs. My impression is that this is also true for most matlab toolboxes (though obviously not the core language), which doesn't quite explain why people use Matlab (but not Octave), versus R (and not S-PLUS)

Re: Why Is NumPy Only Now Getting Funded?

#102

Post by [deleted] » Wed, Jun 28, 2017, 6:15 PM UTC

jacobolus wrote: post #61

mattkrause wrote: post #58

Earlier quoted context omitted.

On the other hand, one of the other major data analysis platforms went the other way: R is a free implementation of S language. TIBCO still sells S-PLUS, but my impression is that R is much, much more popular. My impression is that Octave always lags a little behind Matlab, both in terms of features and performance, since the matlab language is essentially whatever The Mathworks says it is. R and S-PLUS both have a s…

How much money does R make? Are the main R contributors working as processional academics, do they have corporate jobs that pay them to work part-time on R, or are they funded full time by donations/grants, or is R just a hobby side project for them? I’m not saying there’s anything wrong with Octave, or R, or GIMP, or ..., only that it’s difficult to make money on it. This is especially true for Octave though, becaus…

[deleted]

Re: Why Is NumPy Only Now Getting Funded?

#103

Post by mattkrause » Wed, Jun 28, 2017, 6:22 PM UTC

in9 wrote: post #13

jordigh wrote: post #9

Also a problem for Octave. Remember this? https://news.ycombinator.com/item?id=13603575 It also stings a little when people say that it's completely obsolete because Matlab itself is "legacy" and we should all be abandoning the language, Octave included... and yet, even though I like numpy and Python and matplotlib and Julia and R, I still find myself reaching for Octave whenever I need a quick visualisation of some…

Really? Why is that? In my mind, it is tough to beat ggplot

Matlab/octave also include everything but the kitchen sink, especially if you're on an academic license. You can certainly replicate this with a python stack, but it involves a fair number of external libraries.

Re: Why Is NumPy Only Now Getting Funded?

#104

Post by mattkrause » Wed, Jun 28, 2017, 6:52 PM UTC

Radim wrote: post #65

Is the idea that "foundational work" (in any field) can be done without "huge sacrifices" widely accepted? It sounds a tad unrealistic to me, unsupported by history. It's as if people want to have it both ways: Create innovative SW, but also don't take risks or make sacrifices. Offer software "for free" (and belligerently oppose even something like GPL), but also get paid (preferably by the government, so the people…

I think you're missing the point.

The authors of NumPy (and other scientific software) have made it possible for many, many people to do better, faster research, and their career trajectory ought to reflect that contribution.

Right now, it does not.

The NSF will certainly give you money to develop asymptotically faster matrix multiplication algorithms. They are much less interested in funding efforts that save an equal amount of researcher time by writing clearer documentation. Tenure committees would rather see 10 papers or 100 citations than a thousand pull requests, even if the last has a much bigger impact on the state of the entire field. If the authors of NumPy were totally rational, they would have written just enough code and documentation to publish something like "NumPy: A Python library for linear algebra" in the Journal of Statistical Software (or something), then moved on to something else entirely. All the work beyond that (assuming it's not supporting a future paper) comes at the expense of their academic careers. People certainly do it anyway--sometimes out of pride, or a sense of helpfulness--but they're certainly not rewarded for it.

They could be though. Funding agencies could give grants for the on-going development and maintenance of software that helps their grantees. Universities could consider contributions to the broader community as part of their hiring and promotion process, and so on.

The odd part is that this would probably be more cost-effective too. Even a $10M/5 year grant is just a drop in the bucket compared to what all NIH/NSF/etc grantees pay for Matlab or Prism licenses.

Re: Why Is NumPy Only Now Getting Funded?

#105

Post by kevin_thibedeau » Wed, Jun 28, 2017, 7:34 PM UTC

dagw wrote: post #72

kevin_thibedeau wrote: post #40

Earlier quoted context omitted.

But you have to run through their interview gauntlet.

Even if you get head hunted?

Ken Thompson didn't (still doesn't ?) have commit access because he hasn't been vetted by Google as a competent C programmer. I don't think Google is going to relax its hiring policies no matter who you are.

Re: Why Is NumPy Only Now Getting Funded?

#106

Post by heavenlyblue » Wed, Jun 28, 2017, 8:15 PM UTC

abousara wrote: post #78

Each country have its own taxes to sustain fundamentals, same thing must apply in software industry. Software engineers might be paid +100k$ while developing over open source languages/frameworks or libraries. That's is not fair, it is like riding a starving horse. A French point of view (after all, France invented VAT...) would suggest to introduce a taxe on software engineers salaries (1% ?) and redistribute this f…

How do you decide which projects receive more funding than others?

Re: Why Is NumPy Only Now Getting Funded?

#107

Post by Radim » Thu, Jun 29, 2017, 3:28 AM UTC

mattkrause wrote: post #104

Radim wrote: post #65

Is the idea that "foundational work" (in any field) can be done without "huge sacrifices" widely accepted? It sounds a tad unrealistic to me, unsupported by history. It's as if people want to have it both ways: Create innovative SW, but also don't take risks or make sacrifices. Offer software "for free" (and belligerently oppose even something like GPL), but also get paid (preferably by the government, so the people…

I think you're missing the point. The authors of NumPy (and other scientific software) have made it possible for many, many people to do better, faster research, and their career trajectory ought to reflect that contribution. Right now, it does not. The NSF will certainly give you money to develop asymptotically faster matrix multiplication algorithms. They are much less interested in funding efforts that save an equ…

With all due respect, the career trajectory of Travis Oliphant (&co) has been nothing but spectacular! He's extremely accomplished, respected and (I hope) compensated.

You're kinda undermining your own point there :-)

In any case: "force other people to finance my hobbies" is not my favourite ideology. I find the idea of some "tenure committee" deciding what I should do with my money mildly disgusting.

You're very cavalier throwing millions around, but you do realize this is money other people had to earn and tax first, right? And that these people might have preferred to contribute and support a different cause instead, perhaps something closer to their own heart and interests?

Re: Why Is NumPy Only Now Getting Funded?

#108

Post by ChrisFoster » Thu, Jun 29, 2017, 9:25 AM UTC

jordigh wrote: post #48

jacobolus wrote: post #43

Earlier quoted context omitted.

Octave’s “business model” is “give away for free an inferior clone of a commercial product”. Why is anyone surprised that that isn’t lucrative? Anyone who has the money to spend on Octave is likely to just pay for Matlab. The defining feature of Octave users is that they want to use Matlab’s library ecosystem but they’re too cheap or broke to pay for a copy of Matlab.

Because Matlab is expensive. Very expensive. Especially if you want to run it on servers or clusters, which is something people want to do. We're talking a single company paying millions of dollars per year for Matlab licenses, versus paying thousands for Octave (this does happen, and some companies do pay for Octave, but very few). My last job consisted in fixing Octave enough to be able to run some classifiers in s…

You're crazy if you're spending millions on matlab licenses. Just buy a copy of the matlab "compiler" mcc (not much of a compiler - more of packaging system after compilation to byte code, as I understand it). It's quite expensive, but you only need one license for your whole cluster.

Re: Why Is NumPy Only Now Getting Funded?

#109

Post by mattkrause » Thu, Jun 29, 2017, 1:33 PM UTC

Radim wrote: post #107

mattkrause wrote: post #104

Earlier quoted context omitted.

I think you're missing the point. The authors of NumPy (and other scientific software) have made it possible for many, many people to do better, faster research, and their career trajectory ought to reflect that contribution. Right now, it does not. The NSF will certainly give you money to develop asymptotically faster matrix multiplication algorithms. They are much less interested in funding efforts that save an equ…

With all due respect, the career trajectory of Travis Oliphant (&co) has been nothing but spectacular! He's extremely accomplished, respected and (I hope) compensated. You're kinda undermining your own point there :-) In any case: "force other people to finance my hobbies" is not my favourite ideology. I find the idea of some "tenure committee" deciding what I should do with my money mildly disgusting. You're very ca…

Travis Elephant has done very well with Enthought and Continuum Analytics. Before that though, he was an assistant prof at BYU. He's written several times about how he essentially gave up his academic career to continue working on NumPy. For example, he alludes to it in this post he wrote http://technicaldiscovery.blogspot.ca/2012/01/transition-to-... that says "[my wife] watched me sacrifice my tenure-track position by writing NumPy instead of more papers."

You agree that NumPy has been a huge boon to the scientific community and that BME (his previous field) has reaped the benefits too, right? Let's say NumPy has made minor (~5%) contributions to, say, 500 papers; this is probably a huge underestimate. If he had instead published (say) 25 papers, he would have a thriving academic career and yet his actual impact on the world would be a lot lower. Academia is really bad at rewarding work that has a broad, diffuse impact. In a better world, administrators and funders would recognize that his work on NumPy was sufficiently valuable that he wouldn't have had to choose.

I'm not sure this is the right article to get into "research funding==financing hobbies" debate, but...given that some research is probably going to be funded anyway, wouldn't you rather it be done in an efficient way? Right now, most academic software is a bit of a waste: it's written so that the authors can write a paper about having written it. There's no incentive to produce code that others can use or reuse, no reason to maintain it or update it, and so on. Individual labs can usually figure this out, but that also costs time and money which is coming out of the same budget one way or another.

Re: Why Is NumPy Only Now Getting Funded?

#110

Post by jordigh » Thu, Jun 29, 2017, 6:55 PM UTC

ChrisFoster wrote: post #108

jordigh wrote: post #48

Earlier quoted context omitted.

Because Matlab is expensive. Very expensive. Especially if you want to run it on servers or clusters, which is something people want to do. We're talking a single company paying millions of dollars per year for Matlab licenses, versus paying thousands for Octave (this does happen, and some companies do pay for Octave, but very few). My last job consisted in fixing Octave enough to be able to run some classifiers in s…

You're crazy if you're spending millions on matlab licenses. Just buy a copy of the matlab "compiler" mcc (not much of a compiler - more of packaging system after compilation to byte code, as I understand it). It's quite expensive, but you only need one license for your whole cluster.

I was unclear, sorry. The companies that I know that are paying millions are more concerned about copies of Matlab that they can give to their engineers (actual engineers that work on hardware, not Silicon Valley "engineers" that work on ads), of which they need hundreds to thousands, and they keep running out of copies to give to their employees.