Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Every modeler is supposed to be a great Python programmer

statmodeling.stat.columbia.edu

Post reply on HN
Print view View original

31–40 of 131 posts

Re: Every modeler is supposed to be a great Python programmer

#31

Post by goosedragons »

bigger_cheese wrote: post #26
FridgeSeal wrote: post #7

> every one of them wants Python. I haven’t seen a single one where they’re looking for R or even C++; Python rules this roost. Tried putting R into production recently? It’s a frustrating and brittle experience. Don’t get me wrong, R is fantastic at what it does - analysis, research, statistics, and arguably the API’s on the R data frame packages are a lot saner than Pandas. C++ is out for different reasons I suspec…

I work in an Industrial smelter (I'm a Materials/Chem Eng I look at outputs of models don't have a lot to do with developing them) the way the data pipelines are setup here the models are deployed as something called a "pickle file" I think this is something very specific to Python - maybe the equivalent would be a .dll file in C++ world. From point of view of the integration process the pickle file is just a black b…

I think the R equivalent of a pickle file is just an object(s) that's been saved with "save" or "saveRDS" to a file. These can then be loaded and could contain a trained model.

Re: Every modeler is supposed to be a great Python programmer

#32

Post by PaulHoule »

I was talking to somebody at the bus stop the other day who I met back in my physics days and still teaches scientific computing about how I got dragged kicking and screaming into Python (people just kept showing up with work to be done), that I’d almost like to drop Python from my practice because there is no way I can quit Java or JavaScript but libraries keep me in the ecosystem and how we are both shocked that anybody uses R.

Re: Every modeler is supposed to be a great Python programmer

#33

Post by bluedino »

I loved Python. It was easy to learn, very powerful, has libraries for everything...

Then I started supporting researchers and scientists who wrote "python code" to run simulations etc.

Most of it's pretty basic, install some scientific code published by some research group. They chuck their data in and run it.

But then they started abusing virtual environments, writing their own code, cutting and pasting, commenting out random lines because they saw someone else "fix" something that way... and they all want Jupyter notebooks.

Now it's like an eternal September plus I get to deal with annoyingly slow package managers like Conda and rough academic projects with poor documentation and little testing.

Re: Every modeler is supposed to be a great Python programmer

#34

Post by PaulHoule »

mynameisash wrote: post #21
version_five wrote: post #2

This was something that surprised me after I did my PhD as well. I thought that employers would focus on my specialized skills and "someone else" would somehow pick up the pieces and make something out of what I did. Turns out this is completely wrong, and I now see how frustrating it is to work with people that have this kind of attitude. Most of most jobs is a bunch of mundane stuff. I've seen it in software develo…

I dunno... I am a software/data engineer who partners with data scientists. I think that comparative advantage here is a real and important. Don't get me wrong, I'm happy when my data scientist partners write good code or show interest in getting better, but I'm more than happy to take their janky code and make it production ready. It often needs to be optimized for scale or refactored for reusability, and a lot of t…

That’s an unusual style of operation. It may be a really good one, but we don’t really see a lot of teams where people have complimentary skills although really we should.

Re: Every modeler is supposed to be a great Python programmer

#35

Post by mountainriver »

version_five wrote: post #2

This was something that surprised me after I did my PhD as well. I thought that employers would focus on my specialized skills and "someone else" would somehow pick up the pieces and make something out of what I did. Turns out this is completely wrong, and I now see how frustrating it is to work with people that have this kind of attitude. Most of most jobs is a bunch of mundane stuff. I've seen it in software develo…

I think there is value in both and it sort of depends on the organization.

With the commoditization of models, we are seeing the rise of MLEs over data scientists. Engineers that understand enough DS to make things work are wildly proficient in this space.

However, not all models have been commoditized, and there is still a need for new math in many places and that’s where the division of labor makes sense. You can’t be an all star engineer and an all star data scientist it’s just too much for one human

Re: Every modeler is supposed to be a great Python programmer

#36

Post by fbdab103 »

FridgeSeal wrote: post #7

> every one of them wants Python. I haven’t seen a single one where they’re looking for R or even C++; Python rules this roost. Tried putting R into production recently? It’s a frustrating and brittle experience. Don’t get me wrong, R is fantastic at what it does - analysis, research, statistics, and arguably the API’s on the R data frame packages are a lot saner than Pandas. C++ is out for different reasons I suspec…

>Tried putting R into production recently? It’s a frustrating and brittle experience.

Oh man, can I sign onto this rant. While Python has spent a decade+ trying-and-failing to standardize on one of a dozen tools to properly manage dependencies, at least it is trying. R is still global-namespace, no-pinning by default. Sorta-kinda you can squint where renv is going, but still needs a lot of development.

I had some hopes of Julia stealing the R mindshare and righting some of the more egregious wrongs in that ecosystem, but (as an outsider) it feels like Julia has lost a lot of steam.

Re: Every modeler is supposed to be a great Python programmer

#37

Post by lr4444lr »

mynameisash wrote: post #21
version_five wrote: post #2

This was something that surprised me after I did my PhD as well. I thought that employers would focus on my specialized skills and "someone else" would somehow pick up the pieces and make something out of what I did. Turns out this is completely wrong, and I now see how frustrating it is to work with people that have this kind of attitude. Most of most jobs is a bunch of mundane stuff. I've seen it in software develo…

I dunno... I am a software/data engineer who partners with data scientists. I think that comparative advantage here is a real and important. Don't get me wrong, I'm happy when my data scientist partners write good code or show interest in getting better, but I'm more than happy to take their janky code and make it production ready. It often needs to be optimized for scale or refactored for reusability, and a lot of t…

Why do you want them to write code at all, then? Why not just task them with writing user stories around the parameterized functions they need, and just let you figure out how to implement it all?

Re: Every modeler is supposed to be a great Python programmer

#38

Post by Tcepsa »

bluedino wrote: post #33

I loved Python. It was easy to learn, very powerful, has libraries for everything... Then I started supporting researchers and scientists who wrote "python code" to run simulations etc. Most of it's pretty basic, install some scientific code published by some research group. They chuck their data in and run it. But then they started abusing virtual environments, writing their own code, cutting and pasting, commenting…

That sounds similar to my experience with the Python ecosystem (of which I have a profound dislike at the moment). If you haven't tried Mamba yet I encourage you to give it a shot; for me it has had a pretty significant impact on the time I have to spend sitting around waiting for Conda packages to install.

Re: Every modeler is supposed to be a great Python programmer

#40

Post by Tcepsa »

lr4444lr wrote: post #37
mynameisash wrote: post #21

Earlier quoted context omitted.

I dunno... I am a software/data engineer who partners with data scientists. I think that comparative advantage here is a real and important. Don't get me wrong, I'm happy when my data scientist partners write good code or show interest in getting better, but I'm more than happy to take their janky code and make it production ready. It often needs to be optimized for scale or refactored for reusability, and a lot of t…

Why do you want them to write code at all, then? Why not just task them with writing user stories around the parameterized functions they need, and just let you figure out how to implement it all?

I expect that it makes sense to have the modelers do at least enough coding to test the common use cases/expected inputs for the model to validate it before handing it off to be "properly" implemented.
Post reply on HN