Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Every modeler is supposed to be a great Python programmer

statmodeling.stat.columbia.edu

Post reply on HN
Print view View original

81–90 of 131 posts

Re: Every modeler is supposed to be a great Python programmer

#81

Post by Hasnep »

FridgeSeal wrote: post #79
fbdab103 wrote: post #74

Earlier quoted context omitted.

If the cure for R is that I need to jump fully into the Nix ecosystem, I am not sure that is a solution. Disappointing about Julia. Only played with it a small bit, but I had assumed it had identified dependency management as a huge problem that needed to be addressed.

Julia’s package management is miles ahead of R, and bests Python’s on a large number of factors. However the last time I used it, it still required you to issue commands into the repl to setup your packages. However, that might have been resolved by now? The language and tooling clips along at a pretty good pace.

You can now set up a Julia environment non-interactively from the command line by running:

    julia --project=. -e 'import Pkg; Pkg.instantiate()'
Not as clean as running

    python -m pip install -r requirements.txt
but it's something.

Re: Every modeler is supposed to be a great Python programmer

#82

Post by wwilim »

Isn't what they mean just "be able to export the model in a form that a) is portable enough to be taken and integrated by someone else, b) has a reasonable runtime performance, and then c) explain how to use it to an application developer"? Those are high demands already. There are companies with excellent modelers who never actually get anything deployed because there is no one to bridge the technological gap between what they're doing and production services.

Re: Every modeler is supposed to be a great Python programmer

#83

Post by roflyear »

version_five wrote: post #2

This was something that surprised me after I did my PhD as well. I thought that employers would focus on my specialized skills and "someone else" would somehow pick up the pieces and make something out of what I did. Turns out this is completely wrong, and I now see how frustrating it is to work with people that have this kind of attitude. Most of most jobs is a bunch of mundane stuff. I've seen it in software develo…

I love boring stuff because often it allows me to do something really well. Hard problems are fun but hard to solve well by their nature. Programming is fun because I get to do both of these things.

Re: Every modeler is supposed to be a great Python programmer

#84

Post by isoprophlex »

bluedino wrote: post #33

I loved Python. It was easy to learn, very powerful, has libraries for everything... Then I started supporting researchers and scientists who wrote "python code" to run simulations etc. Most of it's pretty basic, install some scientific code published by some research group. They chuck their data in and run it. But then they started abusing virtual environments, writing their own code, cutting and pasting, commenting…

Having been in the same situation as you, you have to bring some technical leadership if you want to change things. And some empathy, these people are stuck in a local maximum, they try their best probably with limited tools and knowledge.

Conda is a shitshow for sure: teach them how things could be better.

Notebooks are a reproducibility nightmare: show them how nice eg. Pycharm is

People are unqualified to write O(n) code in less than O(n^3) time: identify skill gaps and get their management to sign off on some software development / python training.

If you can make a convincing case why your approach is better, you can make everyone's life better.

Re: Every modeler is supposed to be a great Python programmer

#85

Post by abdullahkhalids »

mattpallissard wrote: post #48
bluedino wrote: post #33

I loved Python. It was easy to learn, very powerful, has libraries for everything... Then I started supporting researchers and scientists who wrote "python code" to run simulations etc. Most of it's pretty basic, install some scientific code published by some research group. They chuck their data in and run it. But then they started abusing virtual environments, writing their own code, cutting and pasting, commenting…

Conda may suck in some ways but it solved a big problem; installing software without root accesss. It effectively removed os dependencies allowing for operating system changes. HPC operating systems were effectively ossified prior to that. Notebooks on the other hand are a god damned abomination. Obligatory past discussion; https://news.ycombinator.com/item?id=17856700

> Notebooks on the other hand are a god damned abomination.

I completely disagree. Notebooks are an excellent way of doing literate programming, because they go beyond plain text, and integrate richly formatted text, tables, figures, videos with code. This is excellent where the human is following a line of reasoning, and offloading the computation to the computer. The reasoning is explained via the non-code, made concrete via the code cells, and its effects shown via the code cells.

Of course, people can abuse the conveniences of the medium to write really bad code, and follow some really bad practices. But that doesn't mean the medium is bad. In fact, I think if you forced people to stop using notebooks, their productivity would drop by quite a bit. Notebooks are tool for thought [1].

The solution is to teach them better practices, and provide better tools within the notebook interface that will help people write better code.

[1] https://numinous.productions/ttft/

Re: Every modeler is supposed to be a great Python programmer

#86

Post by dragonwriter »

mynameisash wrote: post #21
version_five wrote: post #2

This was something that surprised me after I did my PhD as well. I thought that employers would focus on my specialized skills and "someone else" would somehow pick up the pieces and make something out of what I did. Turns out this is completely wrong, and I now see how frustrating it is to work with people that have this kind of attitude. Most of most jobs is a bunch of mundane stuff. I've seen it in software develo…

I dunno... I am a software/data engineer who partners with data scientists. I think that comparative advantage here is a real and important. Don't get me wrong, I'm happy when my data scientist partners write good code or show interest in getting better, but I'm more than happy to take their janky code and make it production ready. It often needs to be optimized for scale or refactored for reusability, and a lot of t…

> I think that comparative advantage here is a real and important

It is.

But so are transaction costs, and all handoffs have them.

Re: Every modeler is supposed to be a great Python programmer

#87

Post by nevermore »

> Python doesn’t yet have anything remotely close to ggplot for rapidly making exploratory graphics, for example.

Plug for plotnine (https://plotnine.readthedocs.io/en/stable/). I don't know R but use ggplot indirectly through this library for exploratory data analysis, and comparing the experience to any other python plotting library, I understand why R folks are usually so sad to be using Python.

Re: Every modeler is supposed to be a great Python programmer

#88

Post by kgwgk »

bigger_cheese wrote: post #26
FridgeSeal wrote: post #7

> every one of them wants Python. I haven’t seen a single one where they’re looking for R or even C++; Python rules this roost. Tried putting R into production recently? It’s a frustrating and brittle experience. Don’t get me wrong, R is fantastic at what it does - analysis, research, statistics, and arguably the API’s on the R data frame packages are a lot saner than Pandas. C++ is out for different reasons I suspec…

I work in an Industrial smelter (I'm a Materials/Chem Eng I look at outputs of models don't have a lot to do with developing them) the way the data pipelines are setup here the models are deployed as something called a "pickle file" I think this is something very specific to Python - maybe the equivalent would be a .dll file in C++ world. From point of view of the integration process the pickle file is just a black b…

> something called a "pickle file" I think this is something very specific to Python - maybe the equivalent would be a .dll file in C++ world.

The equivalent would be a .dat file :-)

Re: Every modeler is supposed to be a great Python programmer

#89

Post by kgwgk »

c54 wrote: post #24

If OP wants to make models but never worry about them after doing the fun bits, it sounds like they might enjoy academia. The industry premium salary is in part from doing all the work around the “fun part”. As others have mentioned that’s where a lot of the value is, and nobody wants to be your servant. Though even in academics, you have to write the paper yourself after doing the fun bits.

> nobody wants to be your servant.

Most people in IT industry are servants of others.

It’s not just an industry/academia divide. The issue here is independent contractor / large company.

Re: Every modeler is supposed to be a great Python programmer

#90

Post by disgruntledphd2 »

nevermore wrote: post #87

> Python doesn’t yet have anything remotely close to ggplot for rapidly making exploratory graphics, for example. Plug for plotnine ( https://plotnine.readthedocs.io/en/stable/ ). I don't know R but use ggplot indirectly through this library for exploratory data analysis, and comparing the experience to any other python plotting library, I understand why R folks are usually so sad to be using Python.

Plotnine makes Python possible for EDA stuff. I mean pandas is still worse than the tidyverse, but at least you use ggplot.
Post reply on HN