Every modeler is supposed to be a great Python programmer
statmodeling.stat.columbia.edu
Every modeler is supposed to be a great Python programmer
1–10 of 131 posts
Re: Every modeler is supposed to be a great Python programmer
#2This was something that surprised me after I did my PhD as well. I thought that employers would focus on my specialized skills and "someone else" would somehow pick up the pieces and make something out of what I did. Turns out this is completely wrong, and I now see how frustrating it is to work with people that have this kind of attitude.
Most of most jobs is a bunch of mundane stuff. I've seen it in software development, and I've seen it in management consulting. The best people, typically, are those that will happily do both, understanding that the fun stuff comes with a lot of baggage.
The "someone else is better at the stuff I don't want to do than me" argument rarely holds up either. The friction that comes from dividing the work along lines like modeling and production and trying to hand off is rarely worth it when one person can do both.
Anyway, I've been where the author is, but personally I think it's wishful thinking, unless maybe you want to start your own shop and structure it around yourself that way.
Re: Every modeler is supposed to be a great Python programmer
#3This was something that surprised me after I did my PhD as well. I thought that employers would focus on my specialized skills and "someone else" would somehow pick up the pieces and make something out of what I did. Turns out this is completely wrong, and I now see how frustrating it is to work with people that have this kind of attitude. Most of most jobs is a bunch of mundane stuff. I've seen it in software develo…
It is not that surprising because software engineers are usually more expensive than scientists. That leads business people to ask serious questions about which staff they really need, how much of that work can be done by lower-paid specialists, and whether they really need the professional code quality when the stuck-together python that scientists tend to write usually also works.
Re: Every modeler is supposed to be a great Python programmer
#4This was something that surprised me after I did my PhD as well. I thought that employers would focus on my specialized skills and "someone else" would somehow pick up the pieces and make something out of what I did. Turns out this is completely wrong, and I now see how frustrating it is to work with people that have this kind of attitude. Most of most jobs is a bunch of mundane stuff. I've seen it in software develo…
I have a name for the sort of mundane-yet-employable programming tasks. "Plumbing work". You're not doing the clever problem solving that once sucked you into programming, you're welding pipes together that other people made.
Re: Every modeler is supposed to be a great Python programmer
#5This was something that surprised me after I did my PhD as well. I thought that employers would focus on my specialized skills and "someone else" would somehow pick up the pieces and make something out of what I did. Turns out this is completely wrong, and I now see how frustrating it is to work with people that have this kind of attitude. Most of most jobs is a bunch of mundane stuff. I've seen it in software develo…
Well, in purely software shops, there's often people dreaming up the 'what to do' and a different group of people actually writing the code. Same with systems design, we have 'architects.'
This should be no different for statistical modelers or other disciplines. Employers are just cheap.
Re: Every modeler is supposed to be a great Python programmer
#6> I’d much rather get something working and then hand it off to someone else who can refactor it for speed and clarity, and have it conform to the desired style conventions, etc. etc.
I've been on the coding end of this - when everyone actually has those fixed roles and goes into it eyes wide open, it goes pretty well! When instead the PhD is assigned to go do the feature, and then the programmer is called in later when it's not implemented well, it tends to go quite poorly and nobody is happy, as everyone's time is wasted.
Re: Every modeler is supposed to be a great Python programmer
#7> every one of them wants Python. I haven’t seen a single one where they’re looking for R or even C++; Python rules this roost.
Tried putting R into production recently? It’s a frustrating and brittle experience. Don’t get me wrong, R is fantastic at what it does - analysis, research, statistics, and arguably the API’s on the R data frame packages are a lot saner than Pandas.
C++ is out for different reasons I suspect. As this touches on, “modellers” (and data scientists/engineers) ought to be decent developers, but a lot of them are not, and in my experience actively refuse to learn any of these skills, the comfort zone is “jupyter notebooks” and that’s it. Getting them to write C++ (disregarding language debates), a language that is unequivocally more difficult and fraught with complexity than Python is basically a non-starter.
Do I wish it was different? Yep. Do I wish there was some more variety in the “language ecosystem” so it’s more than just the “lowest common denominator Python” dominance? Absolutely.
Re: Every modeler is supposed to be a great Python programmer
#8This was something that surprised me after I did my PhD as well. I thought that employers would focus on my specialized skills and "someone else" would somehow pick up the pieces and make something out of what I did. Turns out this is completely wrong, and I now see how frustrating it is to work with people that have this kind of attitude. Most of most jobs is a bunch of mundane stuff. I've seen it in software develo…
To all the specialists out there, if you value job security, become a specialist that can produce. When the company I'm in did layoffs, the area that had the biggest layoffs were in groups that did not actively produce the product we were selling.
Re: Every modeler is supposed to be a great Python programmer
#9As someone who was naively hired as a "data-scientist" without my employer checking if I actually knew anything about modeling I ended up falling into the "person who fixes up the code, maintains tests it, etc." role, while my seniors, who were much less enthused about software engineering, were the ones pushing out models that were admittedly very impressive but complete disasters in terms of code.
We worked very well off of each other, it was interesting to pick at a model from a software engineering perspective, how the code could be structured and improved, where some tradeoffs would need to be made and how we would test and verify if it actually worked for our users. I eventually left because the company was more concerned with getting new models out as soon as possible regardless of their actual performance, but it did ignite my passion for software engineering and devops.
Re: Every modeler is supposed to be a great Python programmer
#10This was something that surprised me after I did my PhD as well. I thought that employers would focus on my specialized skills and "someone else" would somehow pick up the pieces and make something out of what I did. Turns out this is completely wrong, and I now see how frustrating it is to work with people that have this kind of attitude. Most of most jobs is a bunch of mundane stuff. I've seen it in software develo…
That’s what makes technical interviews even more frustrating. Most jobs day to day are like 5-10% solving deep technical puzzles and 90-95% fiddling with tooling and automating things that are incredibly frustrating or time consuming.
So ya I’m not sure how to solve your silly brain teaser, but I have written custom test frameworks to automate the tedium away to save the team hundreds of man hours. It took a deep understanding of VERY specific tools (shout out to ASIC EDA tools). But that doesn’t matter on a technical interview. So you hire somebody who can Leetcode, but can’t figure out how to fit all the pieces of the actual job together.