Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Every modeler is supposed to be a great Python programmer

statmodeling.stat.columbia.edu

Post reply on HN
Print view View original

11–20 of 131 posts

Re: Every modeler is supposed to be a great Python programmer

#11

Post by toddm »

This resonates so strongly with me.

I'm a modeler and best described as a lifelong scientific programmer. I'm much, much better at doing the specialty science I was trained to do and have done than write unit tests (for Jupyter notebooks? why?) or struggle through big-O questions (again, why? I write ~100 line programs that are never production code). Like the author of the post, I am not a professional developer and don't pretend to be or want to be. There are people out there way better than me for doing those jobs.

Recently, I interviewed for a role as a computational chemist - this is an ideal fit for a person like me with the domain knowledge, advanced degree in the subject area, passion, and a proven (if dated) track record of publishing in the domain. The idea is to use software, amend what's there if/when needed, and apply my knowledge to the process and what comes out of it.

What did the interview start with? A surprise interactive coding challenge that I wasn't prepared for, and thankfully the interviewer was kind and professional enough to understand that my value-add is in, well, computational chemistry and not in the details covered by a CS major.

I thought Jesus, I bet not a single software engineer at this company got asked to do a simple organic synthesis or even a redox problem during their interview process.

Re: Every modeler is supposed to be a great Python programmer

#12

Post by Fiahil »

Yes, that’s expected of every scientist I am working with. The reason is quite simple and have little to do with Engineering work being more expensive than Scientific one (it’s actually the opposite): get a nice problem to solve or a great model to build, and very soon, you have an ivory tower completely disconnected from the original objective.

To be fair, I don’t ask scientists to understand concurrency issues in programming. Only the basic stuff that is required for delivering a functional program. Yes, Pandas and Scikit-learn belong to the basic stuff.

Re: Every modeler is supposed to be a great Python programmer

#13

Post by [deleted] »

version_five wrote: post #2

This was something that surprised me after I did my PhD as well. I thought that employers would focus on my specialized skills and "someone else" would somehow pick up the pieces and make something out of what I did. Turns out this is completely wrong, and I now see how frustrating it is to work with people that have this kind of attitude. Most of most jobs is a bunch of mundane stuff. I've seen it in software develo…

[deleted]

Re: Every modeler is supposed to be a great Python programmer

#14

Post by feoren »

toddm wrote: post #11

This resonates so strongly with me. I'm a modeler and best described as a lifelong scientific programmer. I'm much, much better at doing the specialty science I was trained to do and have done than write unit tests (for Jupyter notebooks? why?) or struggle through big-O questions (again, why? I write ~100 line programs that are never production code). Like the author of the post, I am not a professional developer and…

> I bet not a single software engineer at this company got asked to do a simple organic synthesis or even a redox problem during their interview process.

Well they should have! One of the most important traits for a good software engineer is to be able to pick up a working understanding of the domain they're programming in. If they can't do that, they're more likely to make negative contributions than any positive ones.

Re: Every modeler is supposed to be a great Python programmer

#15

Post by dollo_7 »

version_five wrote: post #2

This was something that surprised me after I did my PhD as well. I thought that employers would focus on my specialized skills and "someone else" would somehow pick up the pieces and make something out of what I did. Turns out this is completely wrong, and I now see how frustrating it is to work with people that have this kind of attitude. Most of most jobs is a bunch of mundane stuff. I've seen it in software develo…

I totally agree. I work on the private sector, coming from a research position too. I was also focused on the "interesting" side of the problem: the modeling, integrating domain knowledge into the analysis, drawing all sorts of plots... But there were other unavoidable and "uninteresting" needs for the research project, like building a data gathering system with its API and everything. This required my best software engineering abilities. Needless to say my best weren't precisely THE best, so as the project got bigger, the not-so temporary fixes increased, as well as poor design choices (if any). This finally led to a complete reestructure and almost fresh start.

I feel some of it could be avoided, so I learned the hard way that the whole modelling + software engineering process is a subtle craft. It is important to take care on the implications of your code and, specially, on how its done, since it may fall back onto you eventually. This reconciled me with the more technical stuff (my tools) to eventually put up a good work in a more satisfying way.

Re: Every modeler is supposed to be a great Python programmer

#16

Post by jvans »

modeling is 1-2% of the overall work involved in having a model serve customers something useful. There's just so much other work to do that it rarely makes sense to have people who can only model. At some level of scale, building out infrastructure to support people to model full time and constantly run a/b tests make sense but that is not the vast majority of use cases for ML in the real world and building out that support infrastructure is a huge investment

Re: Every modeler is supposed to be a great Python programmer

#17

Post by Moissanite »

polalavik wrote: post #10
version_five wrote: post #2

This was something that surprised me after I did my PhD as well. I thought that employers would focus on my specialized skills and "someone else" would somehow pick up the pieces and make something out of what I did. Turns out this is completely wrong, and I now see how frustrating it is to work with people that have this kind of attitude. Most of most jobs is a bunch of mundane stuff. I've seen it in software develo…

That’s what makes technical interviews even more frustrating. Most jobs day to day are like 5-10% solving deep technical puzzles and 90-95% fiddling with tooling and automating things that are incredibly frustrating or time consuming. So ya I’m not sure how to solve your silly brain teaser, but I have written custom test frameworks to automate the tedium away to save the team hundreds of man hours. It took a deep und…

> So you hire somebody who can Leetcode, but can’t figure out how to fit all the pieces of the actual job together.

Yep - 2021 was a bad time for me, mostly because I was stuck working with "smart" people on 2 different projects who had no idea how to add value. All 3 of us have PhDs, but while they had coasted from there in positions where being "the genius" was enough, I had taken a couple of career tangents. They were constantly mesmerized by my Bash skills and infrastructure knowledge, and in the end my parts of the project got done and theirs were written off due to lack of progress.

Re: Every modeler is supposed to be a great Python programmer

#18

Post by opportune »

Everybody wants to work on the fun stuff. Turns out in software, getting something to actually deliver value (meaning convert it from prototype or proof of concept to something running in production) is 90% not fun stuff. Most people learn that in their first job.

As a software engineer the only way I’d become a “data scientist/trader’s servant” is if I’m getting paid exorbitantly. Otherwise it’s the worst kind of work because someone else is going to get credit for everything that goes right while I take the flak for all the hard stuff.

The converse is that to get a software engineer to be your servant you better also be really good, or else you’re probably going to end up paired with someone who doesn’t have the luxury of taking other jobs and maybe won’t be particularly good at getting your stuff to actually run.

Re: Every modeler is supposed to be a great Python programmer

#19

Post by NHQ »

Tensorflow.js is as good as the Python version, and Node.js is a better production platform, and models can run in browsers. Python is easy though; hiring about a language is a sign management doesn't know enough about programming to trust their own judgement.

https://www.tensorflow.org/js

Re: Every modeler is supposed to be a great Python programmer

#20

Post by notacop31337 »

To be honest I don't believe this to be isolated to just modelling and production code, I feel as though this issue presents itself in most of the job ads I see today, Backend roles with requirements for React, Python roles with requirements for Javascript, Backend roles with requirements for DevOps/SRE.

I don't necessarily have an issue with widely skilled engineers, but I would prefer it's for the right reasons, and I largely believe that it's an exercise in laziness on most companies behalf. They just want to hire less people and have more of their workers do tasks that are outside of their remit.

I have zero interest in writing Javascript, absolutely none, I don't want to do it, and I have pushed my career in directions that mean that for the most part, I don't have too. I'm happy with this decision and have made it willingly.

It's the same with a lot of "DevOps" tasks, having previously been a DevOps Engineer, I now just want to write code, real code, but it feels as though most places now are just not hiring DevOps/Infra people, and just telling their other engineers to do it, which I understand, but it results in a far worse experience for both sides of it. I have to regularly force my hand down from volunteering for things that I have the technical experience to do, and do properly, versus colleagues that don't have the experience, because I'm tired of being shoehorned back into a role that I intentionally left. All of this is because the idea of "cross functional teams" no longer means hiring specialised engineers to do specialised roles, and just getting everyone to do everything, and then being surprised when the context switch penalty actually exists, and it's done to a worse standard than someone who is skilled in that role.

Post reply on HN