Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Every modeler is supposed to be a great Python programmer

statmodeling.stat.columbia.edu

Post reply on HN
Print view View original

101–110 of 131 posts

Re: Every modeler is supposed to be a great Python programmer

#101

Post by zem »

opportune wrote: post #18

Everybody wants to work on the fun stuff. Turns out in software, getting something to actually deliver value (meaning convert it from prototype or proof of concept to something running in production) is 90% not fun stuff. Most people learn that in their first job. As a software engineer the only way I’d become a “data scientist/trader’s servant” is if I’m getting paid exorbitantly. Otherwise it’s the worst kind of wo…

everyone differs on what the "fun stuff" is. i would find doing data modelling and statistics unbearably tedious, but quite enjoy taking working code and refactoring it to be performant, maintainable and well tested.

Re: Every modeler is supposed to be a great Python programmer

#102

Post by pjmlp »

p1necone wrote: post #59
majormajor wrote: post #22

Earlier quoted context omitted.

I've never worked on more painful codebases than when the "architects" don't have to bother writing actual code and so are ignorant of all the edge cases and special-case business rules that turn their pretty pictures into a horrific ball of mud.

The "architects" I've known also had a habit of giving you their grand perfect plan in a meeting that lasts no more than an hour or so, disappearing for the next 6 months without communicating with the people building the thing at all, and then being surprised when the final codebase looks nothing like the perfect system they'd designed in their head. Architects have to be involved in building the thing they're archi…

Having some times taken architect like roles, I think the big problem is that architects is a very loaded term, like everything in IT.

You have proper architects, those that design nice diagrams, have proper technical knowledge, and also code, even if small portions when compared with the rest of the team. These I would call proper architects, and tend to explicit mention Technical Architects to make the point.

Then you have the "architects" that do diagrams, spend the time in meetings with customers, plan features per sprint, delegate activities, and so forth. This ones I call managers and always double check if the company isn't using architec as synomim for managers.

At least some shops are more honest, by making use of business analyst or solution architect as synomim for high level management work.

Re: Every modeler is supposed to be a great Python programmer

#103

Post by lysecret »

I know exactly what you feel like and I used to be exactly like you. Even down to preferring R to Python. All I can say is: Bite the bullet. Learn Python, forget about R. R is nice and all but there is nothing that couldn't also be done in Python.

What helped me is: Learn to appreciate the beauty in actual coding, in deployments in environments in well structured, maintainable code. In scaling issues, databases etc.. There is an endless world out there which is extremely fascinating as soon as you get over the "all I want to do is modeling" mindset.

Good luck. You can definitely do it, because I did it as well.

Re: Every modeler is supposed to be a great Python programmer

#104

Post by pjmlp »

andy800 wrote: post #93
FridgeSeal wrote: post #7

> every one of them wants Python. I haven’t seen a single one where they’re looking for R or even C++; Python rules this roost. Tried putting R into production recently? It’s a frustrating and brittle experience. Don’t get me wrong, R is fantastic at what it does - analysis, research, statistics, and arguably the API’s on the R data frame packages are a lot saner than Pandas. C++ is out for different reasons I suspec…

> wish there was some more variety Kotlin is coming up on the outside. Late to the party, but the data science ecosystem is progressing nicely.

CERN started diving into Python back in 2002, I was part of one of the first trainings that were taking place.

Same applies to many other research institutions.

Good luck catching up with a 20 years old ecosystem, with a language that only gets used mainly thanks to Google's sponsorhip for Android apps.

Re: Every modeler is supposed to be a great Python programmer

#105

Post by LAC-Tech »

c54 wrote: post #24

If OP wants to make models but never worry about them after doing the fun bits, it sounds like they might enjoy academia. The industry premium salary is in part from doing all the work around the “fun part”. As others have mentioned that’s where a lot of the value is, and nobody wants to be your servant. Though even in academics, you have to write the paper yourself after doing the fun bits.

You're always somebody's servant in programming.

If you're doing frontend, you're a servant for the user. Backend, you're a servant for the frontend.

I feel like a specialist who can communicate well and respects what other people do - is incredibly valuable, if a bit rare.

Re: Every modeler is supposed to be a great Python programmer

#106

Post by koliber »

People who are experts at their domain are not always good at explaining things to others. This means that if a data scientist does not know how to code, they need to partner with a programmer and be able to explain things to them. Same goes for a bunch of other professions. For some people, it is easier to learn to code than it is to learn to communicate their ideas with people.

If you are not going to be able to implement something, whether in code, or with a saw and hammer, you must be able to explain things really well. If you can not do either, you will have limited ability to apply your craft.

Re: Every modeler is supposed to be a great Python programmer

#107

Post by eftychis »

subradios wrote: post #95
FridgeSeal wrote: post #65

Earlier quoted context omitted.

I mean, this is probably how things should be done. Having the specialists of each field appropriately resourced, focused and aligned streamlines everyone’s life. The problem comes in that most places either get the scientists to pull double duty and we end up in the scenario scientists are pushing messy, subpar code out the door that nobody else wants to touch because they’re unsupported and operating out of their d…

The big thing is prestige, it takes some pretty big brains to write good, clean production infrastructure that doesn't fall over every five minutes. Those big brains are absolutely capable of learning the math and doing the creative work of modeling. They want to do the creative work. Which means that if you create a prestige gradient and don't let your engineers do interesting things they themselves might get a pape…

This. There is no clean application of comparative advantage here. The great individual that will rewrite the code and knows all the details to get the model scalable, robust and production ready, can also do the data science "creative" work.

If you are not willing to grow to become that person you are only damaging yourself long term. And if you don't attribute that individual's work: a) this is unethical b) they will, and should leave.

P.S. Seen this firsthand and second hand.

Re: Every modeler is supposed to be a great Python programmer

#108

Post by eftychis »

Heads up to the author of the article:

a) writing is hard

b) nobody likes writing production code

c) nobody is good at production without trying

d) most people I have seen that have claimed they are good at modeling, but not good at writing that for production, have actually not reached their "I am good at modeling" state yet.

e) (Most) People don't write code to write code. Like fiction writers do not write lines of text to write text. Writing is a means to an end. It is part of the idea birth process.

TL;DR: Write code for production; you will be great at it relatively soon.

P.S. Curious if people have had the opposite experience or counterexamples. Edit: stylistic.

Re: Every modeler is supposed to be a great Python programmer

#109

Post by Tade0 »

jvans wrote: post #73
mynameisash wrote: post #21

Earlier quoted context omitted.

I dunno... I am a software/data engineer who partners with data scientists. I think that comparative advantage here is a real and important. Don't get me wrong, I'm happy when my data scientist partners write good code or show interest in getting better, but I'm more than happy to take their janky code and make it production ready. It often needs to be optimized for scale or refactored for reusability, and a lot of t…

> often needs to be optimized for scale or refactored for reusability It's so easy to make subtle assumptions when you redo their code that completely invalidates the work they've done. ML completely collapses on extremely small errors. Handing something off to someone else to refactor is a dangerous step in the process that risks everyone wasting their time

> It's so easy to make subtle assumptions when you redo their code that completely invalidates the work they've done.

It's equally easy to test whether those assumptions actually break anything. Especially when minor errors can be catastrophic.

I'm part of a team that was tasked with producing a web app from something that was originally a piece of Matlab code.

We considered just running the Matlab code in a container but ultimately IEEE 754 is IEEE 754 regardless of platform/language, so creating a 1:1 implementation in C++ proved possible.

Re: Every modeler is supposed to be a great Python programmer

#110

Post by sundarurfriend »

FridgeSeal wrote: post #79
fbdab103 wrote: post #74

Earlier quoted context omitted.

If the cure for R is that I need to jump fully into the Nix ecosystem, I am not sure that is a solution. Disappointing about Julia. Only played with it a small bit, but I had assumed it had identified dependency management as a huge problem that needed to be addressed.

Julia’s package management is miles ahead of R, and bests Python’s on a large number of factors. However the last time I used it, it still required you to issue commands into the repl to setup your packages. However, that might have been resolved by now? The language and tooling clips along at a pretty good pace.

I find the flexibility of being able to setup and switch environments from within the REPL very enabling, and the Python approach of using the command-line shell feels kinda messy and cobbled together to me. (I felt this way about the Python approach before I even came across Julia, by the way.)

That said, there is a command-line utility jlpkg [1] that makes package management available from the shell. It's not very widely used, but maybe it suits your needs.

Although, the way you phrased

> it still required you to issue commands into the repl to setup your packages

makes me think maybe you didn't know about the Pkg mode in the REPL, and assumed you had to do everything with commands like `Pkg.add("DataFrames"); Pkg.update()` and such? If so, there's a package management mode to the REPL you can access with the ] key, which is kind of its own subshell within the REPL where you can do `add DataFrames`, `update`, and such instead.

[1] https://github.com/fredrikekre/jlpkg

Post reply on HN