Live data from Hacker News

Every modeler is supposed to be a great Python programmer

statmodeling.stat.columbia.edu

21–30 of 131 posts

Re: Every modeler is supposed to be a great Python programmer

#21

This was something that surprised me after I did my PhD as well. I thought that employers would focus on my specialized skills and "someone else" would somehow pick up the pieces and make something out of what I did. Turns out this is completely wrong, and I now see how frustrating it is to work with people that have this kind of attitude. Most of most jobs is a bunch of mundane stuff. I've seen it in software develo…

I dunno... I am a software/data engineer who partners with data scientists. I think that comparative advantage here is a real and important. Don't get me wrong, I'm happy when my data scientist partners write good code or show interest in getting better, but I'm more than happy to take their janky code and make it production ready. It often needs to be optimized for scale or refactored for reusability, and a lot of that often falls very solidly in the engineering discipline.

I'll often tell my DS partners, "Don't worry about the code; you get the math right then give it to me. You can go about doing more math, I'll do the engineering." In my experience, this is often a really pragmatic division of labor.

This is also important because often, the data scientists are never on-call, so if something breaks in production, engineering needs to know what is going on.

Re: Every modeler is supposed to be a great Python programmer

#22
post #5

This was something that surprised me after I did my PhD as well. I thought that employers would focus on my specialized skills and "someone else" would somehow pick up the pieces and make something out of what I did. Turns out this is completely wrong, and I now see how frustrating it is to work with people that have this kind of attitude. Most of most jobs is a bunch of mundane stuff. I've seen it in software develo…

Well, in purely software shops, there's often people dreaming up the 'what to do' and a different group of people actually writing the code. Same with systems design, we have 'architects.' This should be no different for statistical modelers or other disciplines. Employers are just cheap.

I've never worked on more painful codebases than when the "architects" don't have to bother writing actual code and so are ignorant of all the edge cases and special-case business rules that turn their pretty pictures into a horrific ball of mud.

Re: Every modeler is supposed to be a great Python programmer

#23
post #12

Yes, that’s expected of every scientist I am working with. The reason is quite simple and have little to do with Engineering work being more expensive than Scientific one (it’s actually the opposite): get a nice problem to solve or a great model to build, and very soon, you have an ivory tower completely disconnected from the original objective. To be fair, I don’t ask scientists to understand concurrency issues in p…

If they wanted to learn, where would you point them?

Re: Every modeler is supposed to be a great Python programmer

#24
If OP wants to make models but never worry about them after doing the fun bits, it sounds like they might enjoy academia. The industry premium salary is in part from doing all the work around the “fun part”. As others have mentioned that’s where a lot of the value is, and nobody wants to be your servant.

Though even in academics, you have to write the paper yourself after doing the fun bits.

Re: Every modeler is supposed to be a great Python programmer

#25
Isn’t this because they are hiring to solve the problem, like, a function that predicts X, and if the model engineer is also the production engineer this is one hire that solves the business problem?

The alternative is hiring two engineers and possibly additional pm/management workflow to make sure they mesh and the prod engineer is not blocked on the model engineer, and the model engineer delivers things that are usable. It’s a bit like when we used to have a “webmaster”, or now I suppose they could be a “full stack consultant” who was in charge of making sure the right pixels appeared on thecompanywebsite.com more or less by any means necessary because that was the business need.

Re: Every modeler is supposed to be a great Python programmer

#26

> every one of them wants Python. I haven’t seen a single one where they’re looking for R or even C++; Python rules this roost. Tried putting R into production recently? It’s a frustrating and brittle experience. Don’t get me wrong, R is fantastic at what it does - analysis, research, statistics, and arguably the API’s on the R data frame packages are a lot saner than Pandas. C++ is out for different reasons I suspec…

I work in an Industrial smelter (I'm a Materials/Chem Eng I look at outputs of models don't have a lot to do with developing them) the way the data pipelines are setup here the models are deployed as something called a "pickle file" I think this is something very specific to Python - maybe the equivalent would be a .dll file in C++ world.

From point of view of the integration process the pickle file is just a black box input data flows in, response flows out of it. If something changes with the model (trained it on a new dataset etc.) then it's just a matter of drop in a new pickle file and it all keeps working.

I don't know if you can do the equivalent with packaging R code like this. All of the tooling is setup around Python and I think this is pretty industry standard we use Azure and I'm pretty sure this is all out of the box stuff.

Re: Every modeler is supposed to be a great Python programmer

#27
I support a research environment for biostatisticians and other researchers and we have Python and R offerings, and R is the overwhelming favorite, or so product tells me. As someone who isn’t a modeler, it’s interesting how this varies by industry.

Re: Every modeler is supposed to be a great Python programmer

#28
What stood out to me even more was the "machine learning" buzzword - even though it doesn't seem like there's any guarantee that training a neural network would actually improve the modeling (and it's just another tool that the modeler should be able to decide on their own to use or not).

Specifically, my advisor just suggested that I ignore this bit, and send in my resumé to those job offers anyway (this being something that we should be able to learn on the fly at our level anyway I guess...)

Re: Every modeler is supposed to be a great Python programmer

#29

This was something that surprised me after I did my PhD as well. I thought that employers would focus on my specialized skills and "someone else" would somehow pick up the pieces and make something out of what I did. Turns out this is completely wrong, and I now see how frustrating it is to work with people that have this kind of attitude. Most of most jobs is a bunch of mundane stuff. I've seen it in software develo…

That’s what makes technical interviews even more frustrating. Most jobs day to day are like 5-10% solving deep technical puzzles and 90-95% fiddling with tooling and automating things that are incredibly frustrating or time consuming. So ya I’m not sure how to solve your silly brain teaser, but I have written custom test frameworks to automate the tedium away to save the team hundreds of man hours. It took a deep und…

The way I think of it is that in a business you often need to get a certain fixed number of things right before you could even start to make money.

Indeed, 95% of those things are mundane and tractable (maybe you have to be fast and careful), but the remaining 5% are challenges that can only be solved by specialized knowledge or some innovation.

If you hire someone good at solving mundane problems, they can contribute to the 95%. But if 95% of the company is hired like that, the pool of people you can draw on to do the remaining 5% is Another approach is to hire, say 50% to do mundane work and 50% to do both mundane work and more special work. This costs more, but if done right it can make the company move faster, and have better chance to survive than any case of the former.

First of all, it's often (but not always) the case that people who can do special work can be trained to do the mundane things better. Second, having many people who can solve one-of-a-kind problems is especially important if for each project the 5% can depend on a different set of things; it's often also the case that solutions to the innovative work is closely related to experience dealing the more mundane parts, so this structure basically recognizes that innovation can come from rank and file rather than a specialized "ideas guy" that just flies by to solve problems. So what happens in certain small but profitable companies is that they try to find people who are happy to do dirty work, have a business mindset, but at the same time show signs of being innovative.

This is not to say I agree leetcoding is a good test. I think Google and Facebook don't test enough whether someone is capable of or interested in identifying technical priorities with a business mind. Although Google has some data to connect leetcode performance to job performance, I'm skeptical of their performance evaluation methods.

Re: Every modeler is supposed to be a great Python programmer

#30

> every one of them wants Python. I haven’t seen a single one where they’re looking for R or even C++; Python rules this roost. Tried putting R into production recently? It’s a frustrating and brittle experience. Don’t get me wrong, R is fantastic at what it does - analysis, research, statistics, and arguably the API’s on the R data frame packages are a lot saner than Pandas. C++ is out for different reasons I suspec…

I work in an Industrial smelter (I'm a Materials/Chem Eng I look at outputs of models don't have a lot to do with developing them) the way the data pipelines are setup here the models are deployed as something called a "pickle file" I think this is something very specific to Python - maybe the equivalent would be a .dll file in C++ world. From point of view of the integration process the pickle file is just a black b…

Pickles and .dlls solve completely different purposes. A .dll, which stands for "dynamic link library", is code - it's how you package parts of your implementation in certain environments (Windows). A pickle file is data - it's a way of serializing out the contents of the state of a data structure, using some magic built into Python reflection, and then loading it back later.

Pickling is very convenient for its purpose. It's also... very bad in production. The problem is that the serialization format is derived from the implementation and not very stable, so if the implementation changes - sometimes in subtle ways like upgrading your Python version - the serialized data isn't readable anymore. You're almost always better off either using an explicit data serialization format (JSON with schemas, protocol buffers, etc.) and writing your own logic for saving and restoring to/from it.

> the pickle file is just a black box input data flows in, response flows out of it

There's 100% more to it than that - there's got to be something that's evaluating the data in the pickle file and actually using it to compute the response. That substrate layer probably doesn't change often, but it should have its own versioning and deployment story.

Post reply on HN