Live data from Hacker News

Every modeler is supposed to be a great Python programmer

statmodeling.stat.columbia.edu

41–50 of 131 posts

Re: Every modeler is supposed to be a great Python programmer

#41
My dad got me into computers, but he's not a programmer by trade. He's a mechanical engineer, and he used BASIC on a TRS-80 to do engine simulations to characterize the mechanical forces on the crankshaft for different crank types. Back then, BASIC was basically what MATLAB is today for engineers. They just slammed some code in there to do the math they needed to do and ran it. They didn't care about things we care about: testability, maintainability, observability.

Similarly, Judge Alsup of Oracle v. Google fame writes astronomy software in QBasic. He doesn't give a shit about best practices, if it helps him aim his telescope correctly it's all good.

Welcome to a world with citizen programmers. A world of terrible code that does the job. I frickin' love it.

Re: Every modeler is supposed to be a great Python programmer

#42

I loved Python. It was easy to learn, very powerful, has libraries for everything... Then I started supporting researchers and scientists who wrote "python code" to run simulations etc. Most of it's pretty basic, install some scientific code published by some research group. They chuck their data in and run it. But then they started abusing virtual environments, writing their own code, cutting and pasting, commenting…

The situations such as improving package management in python is a problem but fundamentally in any software stack you will eventually find bad code. Especially as the code base gets larger and people move on.

Instead you can try to fight it by teaching the rest of the group how to write more idiomatic code but that only goes so far. Also training and documentation can go out the window when the objective is to ship something that works once.

Re: Every modeler is supposed to be a great Python programmer

#43
The modeler who can write code unlike you will get hired first.

Okay, now that I said the provocative thing to kind of drive home how real and serious this point is, I will say that I am not a statistician, but I run more computational physics simulations, so less statistical modelling but modelling of experiments and systems based on the PDEs. The one thing I observe is that there really is only this patience for this lack of understanding your own tools for theorists. Experimentalists can fix their own tools, they can open up the casing and resolder the boards if they need to, heck most of them can fix their own cars. But you have no idea how many computational scientists just load up Lumerical or Ansys and just click around but really have no concept or idea how it works under the hood beyond just things they show on intro slides to talks. Some know how to script say Meep or something if they're good but they've never implemented a DE solver themselves unless it was in a class in college or first year grad school then they forgot it all.

You really only have this disconnect from your own tools for theorists. Programming is your breadboard, your substrate. Code is the material you use to do your work. I don't understand why it is okay for theorists of all stripes to slide on never really understanding how their on research actually works on a computer whereas every experimentalist I've ever known could recreate their entire experimental apparatus from scratch if they were paid to do so. But that's okay, because that means as long as too many theorists can only write equations and then have to have someone to hand hold them so they actually do the things they've written down, I will be valuable and have job opportunities for myself. It would however make life easier for myself and lessen the many headaches I've been subject to, and heck, may be science could move forward a little better, yadda yadda.

I probably shouldn't encourage my competition like that especially when they're injuring themselves but that "move science forward and lessen my headaches" vibe does make me want to share the sentiment so that theorists at least understood on some level how the libraries they import work sometimes.

Re: Every modeler is supposed to be a great Python programmer

#44

> every one of them wants Python. I haven’t seen a single one where they’re looking for R or even C++; Python rules this roost. Tried putting R into production recently? It’s a frustrating and brittle experience. Don’t get me wrong, R is fantastic at what it does - analysis, research, statistics, and arguably the API’s on the R data frame packages are a lot saner than Pandas. C++ is out for different reasons I suspec…

I believe that is one of the points the author of the article is trying to make: that the data scientists ought to be able to work in whatever language lets them develop the model most quickly and then hand it over to a software developer to make it work well in code, so that each can focus on doing what they do best and both aspects can be done very well.

Re: Every modeler is supposed to be a great Python programmer

#45

Earlier quoted context omitted.

I dunno... I am a software/data engineer who partners with data scientists. I think that comparative advantage here is a real and important. Don't get me wrong, I'm happy when my data scientist partners write good code or show interest in getting better, but I'm more than happy to take their janky code and make it production ready. It often needs to be optimized for scale or refactored for reusability, and a lot of t…

Why do you want them to write code at all, then? Why not just task them with writing user stories around the parameterized functions they need, and just let you figure out how to implement it all?

You usually have to evaluate the model as you develop it, it's not something that a story driven model will fit

Re: Every modeler is supposed to be a great Python programmer

#46

This was something that surprised me after I did my PhD as well. I thought that employers would focus on my specialized skills and "someone else" would somehow pick up the pieces and make something out of what I did. Turns out this is completely wrong, and I now see how frustrating it is to work with people that have this kind of attitude. Most of most jobs is a bunch of mundane stuff. I've seen it in software develo…

I dunno... I am a software/data engineer who partners with data scientists. I think that comparative advantage here is a real and important. Don't get me wrong, I'm happy when my data scientist partners write good code or show interest in getting better, but I'm more than happy to take their janky code and make it production ready. It often needs to be optimized for scale or refactored for reusability, and a lot of t…

> You can go about doing more math, I'll do the engineering

I'd say most of the teams I supported as a backend/DevOps/infrastructure engineer followed that pattern.

Generally the evolution I saw was start with a handful of sciency folks working in R or python, get more grant money as work becomes more important, have some SE's thrown in the mix to re-write in python and/or C/Cpp.

A lot of those SE's would float between teams with one of us infra folks to performance tune as they scaled.

> the data scientists are never on-call,

Yeah, but I'll keep my on-call rotation over their constant 11th hour shenanigans trying to get a paper out the door.

Re: Every modeler is supposed to be a great Python programmer

#47

I loved Python. It was easy to learn, very powerful, has libraries for everything... Then I started supporting researchers and scientists who wrote "python code" to run simulations etc. Most of it's pretty basic, install some scientific code published by some research group. They chuck their data in and run it. But then they started abusing virtual environments, writing their own code, cutting and pasting, commenting…

[deleted]

Re: Every modeler is supposed to be a great Python programmer

#48

I loved Python. It was easy to learn, very powerful, has libraries for everything... Then I started supporting researchers and scientists who wrote "python code" to run simulations etc. Most of it's pretty basic, install some scientific code published by some research group. They chuck their data in and run it. But then they started abusing virtual environments, writing their own code, cutting and pasting, commenting…

Conda may suck in some ways but it solved a big problem; installing software without root accesss. It effectively removed os dependencies allowing for operating system changes. HPC operating systems were effectively ossified prior to that.

Notebooks on the other hand are a god damned abomination. Obligatory past discussion;

https://news.ycombinator.com/item?id=17856700

Re: Every modeler is supposed to be a great Python programmer

#49

Earlier quoted context omitted.

I work in an Industrial smelter (I'm a Materials/Chem Eng I look at outputs of models don't have a lot to do with developing them) the way the data pipelines are setup here the models are deployed as something called a "pickle file" I think this is something very specific to Python - maybe the equivalent would be a .dll file in C++ world. From point of view of the integration process the pickle file is just a black b…

Pickles and .dlls solve completely different purposes. A .dll, which stands for "dynamic link library", is code - it's how you package parts of your implementation in certain environments (Windows). A pickle file is data - it's a way of serializing out the contents of the state of a data structure, using some magic built into Python reflection, and then loading it back later. Pickling is very convenient for its purpo…

> There's 100% more to it than that - there's got to be something that's evaluating the data in the pickle file and actually using it to compute the response.

I suspect that is the Microsoft Azure ML part I'm not sure. As I said I don't write the models, I just come along afterwards and review the results. I don't know anything about Python.

Re: Every modeler is supposed to be a great Python programmer

#50

This was something that surprised me after I did my PhD as well. I thought that employers would focus on my specialized skills and "someone else" would somehow pick up the pieces and make something out of what I did. Turns out this is completely wrong, and I now see how frustrating it is to work with people that have this kind of attitude. Most of most jobs is a bunch of mundane stuff. I've seen it in software develo…

> Most of most jobs is a bunch of mundane stuff.

That’s why we call it “work.”

Post reply on HN