Live data from Hacker News

Science needs more research software engineers

nature.com

281–290 of 366 posts

Re: Science needs more research software engineers

#281
post #126

This looks great, and I would have loved to see this when I was in the lab. When the software industry says to you: "We will nearly triple your salary, you don't have to work weekends, and you also don't have to feed the mice on a Sunday night." You will 100% take this deal. I was a 10yr+ academic tool maker in biochemistry, built cutting edge microscopes, hardware, and image analysis software. My lab was successful…

> You don't have enough letters after your name to matter, and you will likely be a pet instead of a peer. This is an underrated point. This is the case for programmers in finance as well, and requires a hefty salary premium to put up with.

Agreed- you might appreciate a story that happened recently. I worked at a finance algo trading startup, right before and into the financial crisis. The first CEO/founder gets ousted, new guy is an old school "phones and traders" kind of guy, and didn't know, or even seem to care about tech at all. It was a strange choice, since we were built as a tech first company, but seeing as we were having difficulty getting traction, I think the hope was by getting the old guard type in there, we would have an easier time selling the new thing... anyway, I give this a go for a few months but eventually leave as I just could not stand his hardly contained contempt for technology, you could just see it on his face that he longed for the days of dropping F bombs on the floor, and then going out for expensive steak dinners at night. As I give my resignation, I get screamed at, he is red in the face dropping F bombs on me- "You are F---ing us!" etc... Long story short, I offered them to counter, with a 10x'ing of my equity stake, and even to extend my notice period- at my newly offered salary, but they declined all of it, though practically begged me to stay on for 3 extra months at my current salary. On the week I left, this guy tries to get me to sign all kinds of nasty non-disparagement agreements, which I had not signed previously, and with no additional consideration ($) in exchange, and I just refused, and he literally threw the stack of papers at me at one point. I guess I took all of this because I had literally been there since day 1 and just had a sense of ownership over everything- I also didn't think this guy would last very long.

Anyway, fast forward about 10 years to a few months ago, I get a generic "cold call" type message on LinkedIn from a unicorn data tools company, from the same CEO guy- he bounced around and somehow landed a sales role there. I ignored the first one... he sent a followup, and I was incredulous- did he not remember me? Did he not care? It was something along the lines of "Hey how are you? I am working with xxxx and think you would be interested- can we set up a chat..." and I just replied back saying "I am great, haven't been screamed at or had anyone throw something at me in ten years..." and he still had the balls to right back something like "lol, great. Let me know when we can set up a call..." and I wasn't really sure how to respond, but after about a week just wrote "If your next message isn't a very specific apology for your past behavior, do not contact me. I am surprised with your past attitude you would even work at a place like xxxx." If he was a dick I was going to reach out to their head of sales and possibly CEO explaining his past and how I was disappointed that a firm with their reputation would even hire someone like that. He gave me just a half hearted enough non-specific apology to not do that- followed up immediately with an offering of buying me a beer (so he can pitch me), so I opted not to escalate any further.

I have a few other stories similar to this, where Karma really bit those that were hostile and condescending towards technology and technologists, but this is the most direct. I kept in touch with a few of the old "traders" I used to work with, and used to go out for drinks with them from time to time, and they would invite a larger group of people, and I actually stopped because they were all kind of depressing- they all lost their old jobs, a few pivoted into different decent roles, but mostly they just got drunk pining for the good old days to come back.

While there was a tiny bit of schadenfreude, in the end its just sad.

Re: Science needs more research software engineers

#282
The pay gap is real, and a huge reason why perfectly good software folk (and scientists, data specialists, etc.) move out of academia and into industry.

I have an industry job with good pay with a side of academic work stemming from my PhD. My industry income is literally more than 2x the salary of my academic PI, who holds a senior academic post. My pay, which is probably around the median of most HN readers, is slightly more than the top peg of the academic pay scale in my area - i.e. University senior leadership team. So I do the academic work because I love doing the academic work. I cannot afford to take a full-time academic job, although it's been offered.

I am baffled as to why academic institutions aren't failing to recruit. If they were businesses offering half the going rate, they'd get third-rate engineers, if they had any applications at all. Instead, there seems to be a long queue for academic tenure - and even for non-tenured positions. The only competitive advantage I can see is the sense of academic 'freedom' that these positions confer.

But - it's not freedom. My academic colleagues spend hours each week chasing funding, filling in grant requests, attending meetings. Their teaching load is allocated to them without choice. They're in the office 9-5 same as everyone else. Academic freedom is anything but.

I am lucky that for the research I do, mostly behind a laptop, my industry pay lets me afford the things I need to do it - cloud services, international travel, software licenses, article processing fees. I can 'do science' the way I want to without begging anyone for cash or going to grants committees. And I'm hoping this brand of 'citizen science' becomes the norm, because University-led research is going to get harder and harder the more the gap between industry and academic pay widens.

Re: Science needs more research software engineers

#283
post #23

I've worked at a top research university for the past 15 years as a research software engineer. Many of these comments have some truth but a lot are definitely not universally true. Poor compensation: Yup, I'll cop to that. I could be making way more in "industry" but I make enough to live and I don't build ad-ware and do take pride in my work. Being a PhD but little respect: I actually have the opposite experience.…

Lets be honest. Searching for https://hn.algolia.com/?q=burnout in hn shows so many people suffering from burnouts working in industry. My SO is a research software engineer. 1. As you say, she doesn't build ad-ware and do take pride... 2. relaxed and convenient job timings 3. In EU - close to 31 days paid + usually Professors are very generous about going home early etc... 4. Pension, unions, unlimited contracts 5.…

Obviously "research" and "industry" are not monolithic entities. It's possible to have a relaxing and fulfilling and balanced work environment in industry too.

Re: Science needs more research software engineers

#284
post #126

Earlier quoted context omitted.

> You don't have enough letters after your name to matter, and you will likely be a pet instead of a peer. This is an underrated point. This is the case for programmers in finance as well, and requires a hefty salary premium to put up with.

It certainly echoes my experience having just left a professional dev job in academia after an 11 year stretch. Anybody without academic credentials relevant to the subject matter is "the help" no matter how much you contribute, and it's flat-out demoralizing. I worked on a tech-heavy project large enough to get an NYT feature article covering its launch. For it, I collaborated heavily on the service design and logis…

What you did doesn't matter, and they gave you the right amount of recognition. There are 1000's of imported indentured servants that will happily do your job the moment you leave. Of course, we don't call them indentured servants any more, we use terms like 'academic visa' or such.

I'm sure they also didn't think the electricity company for keeping the lights on, or Microsoft for creating Windows to write their speeches, or the guy that emptied the waste baskets in the office so the PHD guy didn't have to.

Don't carry water for someone else. Enrich yourself. That's all anyone else is doing, all the 'research' is for personal enrichment and prestige. Don't prop up the broken academic industry with less than market wages, let them fail.

Re: Science needs more research software engineers

#285

This looks great, and I would have loved to see this when I was in the lab. When the software industry says to you: "We will nearly triple your salary, you don't have to work weekends, and you also don't have to feed the mice on a Sunday night." You will 100% take this deal. I was a 10yr+ academic tool maker in biochemistry, built cutting edge microscopes, hardware, and image analysis software. My lab was successful…

I have some similar feelings about working for a game company (that was fairly successful and on the “better half” of the distribution of game companies).

Loved the creative control and influence I could have (even as a mid-20s tech lead on a title), loved my colleagues, loved the work, and even enjoyed the satisfaction from shipping a golden master after a multi-week crunch period.

Ultimately, a hedge fund was willing to pay me a multiple of what I was making in games and I decided I’d rather have a house than work on games.

Re: Science needs more research software engineers

#286

Well, then science needs to do more to create an environment where software engineers would want to be an RSE. From my experience "science" does not appreciate RSEs, the compensation is bad, the freedom is bad, and you get all of the big company bureaucracy from the University. It's just a bad environment and to top it off, even if you have a PhD many will see you as "less than."

If RSEs are actually valuable but under-compensated, can someone please disrupt this industry? Would love to see more startup medicine development companies ... Alternatively, would it make sense to provide RSE services as a company, for very high hourly rates, separately from the rest of the university, so status and rules are less of a problem?

The reason they're under-compensated is that they aren't valued, unfortunately. A lot of scientists are self taught programmers and don't know what they don't know. The other issue is that if you say you should spend half your departments budget on hiring much better paid software developers, it looks really bad:

a. You're implying your colleagues are incompetent.

b. You can't use that money for "science".

c. It puts in painful relief that the software guys are much better paid than you are, implying that their work is more valuable than yours. But a lot of scientists put up with low pay because they believe their work is really valuable.

Re: Science needs more research software engineers

#287
post #17

The other side of "We need software engineers in research" is surely the botched covid model from Imperial College way back at the beginning of the pandemic. The group that produced it had the audacity to complain that the software industry didn't make C++ foolproof enough, and it was our fault that the modelling was wrong. This shows the attitude that some folks in these places have - what you do is easy, and rather…

Having worked in both, software is much easier than producing quality scientific discoveries.

OK great, but being good at one doesn’t mean you’re automatically good at the other, or that it’s some ‘lesser’ field’s fault when you screw it up.

In this case, due apparently to arrogance, neither was produced.

Re: Science needs more research software engineers

#288
So much to say. I went through the whole academic cycle and moved to industry... about 15 years ago. I can't put it any other way than: the professors and PIs are abusing their employees. It has worked for some time because there were always enough people ultra-motivated to work in science for the prestige and other perks, but now, the economy is such that people can no longer do that (science job + family + mortgage = unhappy).

The right way to think about this is in terms of long-term politics and the success of nations. If you disincentivize national research, your country will fare worse than its competitors. For me, that's the US- if the US doesn't turn around the incentive systems for doing high quality academic research in the national interest, we will eventually be at an economic disadvantage to China.

Re: Science needs more research software engineers

#289
post #90

Earlier quoted context omitted.

> They call it "codes" They/We have probably been calling it that since before you were alive. I wouldn’t be surprised if it comes from the 60s or 70s. Every group of people has their own terminology for stuff. There is no right or wrong.

I have never heard or read anyone, beside physicists, say codes for software code. It doesn't even make sense. Where have you seen that used?

I've seen it a lot outside of physics.

Obviously, what word they use for it is arbitrary but it's indicative of an actual problem - they're all totally cut off from the software engineering / CS community. Lots of them think programming is like learning to juggle or ride a bike or something, a skill that anyone just learns and then they're done. Then, obviously, they make major mistakes that invalidate their conclusions and nobody notices or cares because it's all student code, all the time.

Re: Science needs more research software engineers

#290
post #118

For this to work academic science would have to start paying significantly more, which many labs simply cannot do, and other labs will refuse to do. My PI flat out said when we were trying to hire a python developer that they simply could not be paid 2x more than anyone else because it would cause major issues with team cohesion. 2x is the bare minimum to even get close to market rate for a full time developer who wo…

let them know that congress needs to provide more money for basic research

That won't work. Academia isn't under-funded, especially not in the USA. The NSF budget last year was $7.7 billion so money is there.

The reason salaries are so low is because academia will generally choose to expand the empire by creating new projects and hiring more postdocs over raising salaries of their existing team. And they can, because lots of people have stars in their eyes about science. They see it as more virtuous than mere product building and are willing to put up with a lot for the associated prestige. Also they may have picked their vocation early on in life when their understandings of real market rates was poor, then get stuck in it.

Post reply on HN