Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Every modeler is supposed to be a great Python programmer

statmodeling.stat.columbia.edu

Post reply on HN
Print view View original

71–80 of 131 posts

Re: Every modeler is supposed to be a great Python programmer

#71

Post by mattpallissard »

bluedino wrote: post #60
mattpallissard wrote: post #48

Earlier quoted context omitted.

Conda may suck in some ways but it solved a big problem; installing software without root accesss. It effectively removed os dependencies allowing for operating system changes. HPC operating systems were effectively ossified prior to that. Notebooks on the other hand are a god damned abomination. Obligatory past discussion; https://news.ycombinator.com/item?id=17856700

Couldn't you just do that with pip?

No conda is for dependencies of your python environment. You need python itself, shell utilities, and some dependencies are dynamically linked. Think package manager for the `--prefix` compilation flag AND `LD_LIBRARY_PATH`

Re: Every modeler is supposed to be a great Python programmer

#72

Post by lukego »

fbdab103 wrote: post #36
FridgeSeal wrote: post #7

> every one of them wants Python. I haven’t seen a single one where they’re looking for R or even C++; Python rules this roost. Tried putting R into production recently? It’s a frustrating and brittle experience. Don’t get me wrong, R is fantastic at what it does - analysis, research, statistics, and arguably the API’s on the R data frame packages are a lot saner than Pandas. C++ is out for different reasons I suspec…

>Tried putting R into production recently? It’s a frustrating and brittle experience. Oh man, can I sign onto this rant. While Python has spent a decade+ trying-and-failing to standardize on one of a dozen tools to properly manage dependencies, at least it is trying. R is still global-namespace, no-pinning by default. Sorta-kinda you can squint where renv is going, but still needs a lot of development. I had some hop…

I find it really painless to put bits of R into production using the Nix support: https://nixos.org/manual/nixpkgs/stable/#r

The same framework can provide an identical-to-production R or RStudio environment for development.

I have a harder time with Julia since its tooling is more opinionated. I don't know a better solution than wrapping it in a Docker container.

Re: Every modeler is supposed to be a great Python programmer

#73

Post by jvans »

mynameisash wrote: post #21
version_five wrote: post #2

This was something that surprised me after I did my PhD as well. I thought that employers would focus on my specialized skills and "someone else" would somehow pick up the pieces and make something out of what I did. Turns out this is completely wrong, and I now see how frustrating it is to work with people that have this kind of attitude. Most of most jobs is a bunch of mundane stuff. I've seen it in software develo…

I dunno... I am a software/data engineer who partners with data scientists. I think that comparative advantage here is a real and important. Don't get me wrong, I'm happy when my data scientist partners write good code or show interest in getting better, but I'm more than happy to take their janky code and make it production ready. It often needs to be optimized for scale or refactored for reusability, and a lot of t…

> often needs to be optimized for scale or refactored for reusability

It's so easy to make subtle assumptions when you redo their code that completely invalidates the work they've done. ML completely collapses on extremely small errors. Handing something off to someone else to refactor is a dangerous step in the process that risks everyone wasting their time

Re: Every modeler is supposed to be a great Python programmer

#74

Post by fbdab103 »

lukego wrote: post #72
fbdab103 wrote: post #36

Earlier quoted context omitted.

>Tried putting R into production recently? It’s a frustrating and brittle experience. Oh man, can I sign onto this rant. While Python has spent a decade+ trying-and-failing to standardize on one of a dozen tools to properly manage dependencies, at least it is trying. R is still global-namespace, no-pinning by default. Sorta-kinda you can squint where renv is going, but still needs a lot of development. I had some hop…

I find it really painless to put bits of R into production using the Nix support: https://nixos.org/manual/nixpkgs/stable/#r The same framework can provide an identical-to-production R or RStudio environment for development. I have a harder time with Julia since its tooling is more opinionated. I don't know a better solution than wrapping it in a Docker container.

If the cure for R is that I need to jump fully into the Nix ecosystem, I am not sure that is a solution.

Disappointing about Julia. Only played with it a small bit, but I had assumed it had identified dependency management as a huge problem that needed to be addressed.

Re: Every modeler is supposed to be a great Python programmer

#75

Post by musicale »

> II. Modelers have to be coders

Since you can basically use numpy like a large calculator, it seems like a potentially useful tool to have under your belt. And matplotlib is good for making graphs. Python/numpy/etc. seems like a reasonable alternative to matlab (etc.) in many cases.

Symbolic math tools like Mathematica are useful as well.

It's certainly helpful if modelers can understand the code that implements the model and spot obvious errors in the code as well as in the results. It's also extremely beneficial if whoever is writing, testing, and using the model code has a very good understanding of the model itself.

A potential step toward this happen is implementing the model as a standalone library of very straightforward code that everyone on the project can understand.

Re: Every modeler is supposed to be a great Python programmer

#76

Post by programmarchy »

Is there anything like Blender Nodes, but for data modeling? It's an amazingly powerful system. [1] The learning curve is still fairly steep in terms of having to learn all the nodes and how they interact, but you can't make a syntax error.

[1] https://www.youtube.com/watch?v=7EeIsUErzLE #nodevember - Simon Thommes - Procedural Shader Showreel (Blender)

Re: Every modeler is supposed to be a great Python programmer

#77

Post by djha-skin »

> If a company says that they need excellent Python skills, and they mean it, then I’m not the right person for that job.

Eh, let the company decide that. That's one of the biggest things I tell my buddy who thinks he's bad at programming and really doesn't want to have to look for a job. He's convinced he's horrible but I don't think he is. He just doesn't want to get rejected, so he just avoids interviews as much as possible and tries to stay at the company he's at.

I would imagine that most companies are okay with python code that is rough and ready as long as it can be integrated by the rest of the team into the production code. Most data programmers are probably expected to be better at the data than the programming.

Re: Every modeler is supposed to be a great Python programmer

#78

Post by ramblerman »

djha-skin wrote: post #77

> If a company says that they need excellent Python skills, and they mean it, then I’m not the right person for that job. Eh, let the company decide that. That's one of the biggest things I tell my buddy who thinks he's bad at programming and really doesn't want to have to look for a job. He's convinced he's horrible but I don't think he is. He just doesn't want to get rejected, so he just avoids interviews as much a…

This coincides with grandad's dating advice.

"She will find a bunch of things wrong with you, your job is not to help her out"

Re: Every modeler is supposed to be a great Python programmer

#79

Post by FridgeSeal »

fbdab103 wrote: post #74
lukego wrote: post #72

Earlier quoted context omitted.

I find it really painless to put bits of R into production using the Nix support: https://nixos.org/manual/nixpkgs/stable/#r The same framework can provide an identical-to-production R or RStudio environment for development. I have a harder time with Julia since its tooling is more opinionated. I don't know a better solution than wrapping it in a Docker container.

If the cure for R is that I need to jump fully into the Nix ecosystem, I am not sure that is a solution. Disappointing about Julia. Only played with it a small bit, but I had assumed it had identified dependency management as a huge problem that needed to be addressed.

Julia’s package management is miles ahead of R, and bests Python’s on a large number of factors. However the last time I used it, it still required you to issue commands into the repl to setup your packages. However, that might have been resolved by now? The language and tooling clips along at a pretty good pace.

Re: Every modeler is supposed to be a great Python programmer

#80

Post by myhf »

mattpallissard wrote: post #71
bluedino wrote: post #60

Earlier quoted context omitted.

Couldn't you just do that with pip?

No conda is for dependencies of your python environment. You need python itself, shell utilities, and some dependencies are dynamically linked. Think package manager for the `--prefix` compilation flag AND `LD_LIBRARY_PATH`

conda can also be used to manage python packages, but its dependency solver is very slow and sometimes completely fails.

Once you have created a conda environment, it is usually much more practical to install packages with pip or mamba.

Post reply on HN