Why we need to create careers for research software engineers

So universities should hire Software Engineering professionals and have them build tools for the research ecosystem? They can definitely afford it.

I've interviewed for a number of these types of jobs in the last year or so. The pay is just too poor to even think about it. One large university with a super well-funded and growing medical school in a huge mega-city was looking to hire someone with 5+ years of deep experience in both Python and C/C++, as well as some familiarity or past education in statistics and machine learning, to be the software development lead for a set of research groups. They weren't willing to go above $90k/year.

In another case, I interviewed with a school of public health at an Ivy university. It was in a medium-to-large city on the east coast, where the cost of living was certainly higher than national average, but not insanely high. Even still, they were looking to pay in the range of $75k for someone with significant years of experience in the scientific Python stack.

Not only does this fail to compare even with start-ups that aren't located in major tech hubs, but also when compared against salaries offered by established companies in the same cities, it's almost a 2x pay jump (more in some cases), for the same skill set.

To boot, these lab/university environments tend to have many of the same political issues that any other organization has. Sometimes they are slightly more generous with vacation time and work/life balance, but not always. But you are generally also working with a lot of research staff who see "programming" as a nuisance that gets in the way of their research, rather than seeing programming as the medium of expression of their research. So when you lobby for using good tools, good workflow practices, and try to teach people how to write reasonable software, you usually get a lot of pushback that people just don't like doing it that way, and since you're the super low-status programmer, your opinion generally doesn't count for much, even if you can back it up with more substance.

You also carry some employment risk when projects are funded by grants. Even large, well-established, multi-year grants can be suddenly changed or reduced when there is significant political change in the funding organizations. And a lot of the universities, at least, are unwilling to hire these kinds of expensive programmers as permanent university employees.

You'd probably have a much better career if you instead went to work as a programming-literate business analyst in some part of the university's fundraising / business departments.

In the end, these places can only afford developers with very little experience, or people who don't understand the cost of living differential in the city they are moving to, or I guess are independently wealthy and just prefer to be a low-status programmer in one of these labs.

Edit: Another thing that struck me -- in a lot of these university research situations, the programming talent that they need to hire is genuinely just harder to find / harder to replace than even the research talent you would be working for.

Since there's a huge surplus of Ph.D. and post-doc labor, you could staff the domain expert research side of things much more easily than the programmer side -- especially if you need someone who is truly full-stack to do things ranging from migrating legacy research data to a cloud platform all the way up to designing fancy UI demos with d3.

If you look at another set of industries where the lower-level contributors are much harder to employ than the manager -- professional team sports -- you see that the market dictates that the lower level experts (the actual players on a sports team) almost always command more salary than the coach. There can be exceptions, like Phil Jackson (but even he didn't earn more than his best players) and there will be backup players who earn less. But the general idea seems to hold true.

This means that a lot of these scientific labs and universities will eventually have to face the market reality that the domain expert labor is just not worth as much as the software design labor. Or else they can just keep on compromising on the quality of the software and try to hold on to a worldview where scraping by with bottom-of-the-barrel software development is good enough. It might work for them for a while longer, but I doubt it will work for too long.

Working in research as a software engineer can be very frustrating. I wrote the majority of the analysis code for one of the chapters of the IPCC 5th Assessment and yet was told my name couldn't be put anywhere as I wasn't a 'scientist'. I also consistently see people who have never written much code put in charge of projects where the majority of the work/research is writing software. These projects end up with no code repository use, poor quality control, zero testing, limited OO use - all the things that a professional would take for granted. Worse, they don't perceive the difference between a postdoc who can write code and a professional developer with years of experience. So they hire the cheaper option and end up with the usual 'ball of mud'.

Are regular promotions that important? As long as I get keep getting to do interesting work and keep getting reasonable raises I'm quite happy with my job title not changing more than once a decade.

I think I have this sort of career.

I take theoretical paper and turn it into practical implementation. I make money on consulting. Lot of independence and freedom.

I dont think it is possible to do such thing in academia. Software guys I know in astronomy are underfunded and on short contracts. Most guys will leave for commercial company before starting a family.