Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Every modeler is supposed to be a great Python programmer

statmodeling.stat.columbia.edu

Post reply on HN
Print view View original

111–120 of 131 posts

Re: Every modeler is supposed to be a great Python programmer

#111

Post by onos »

noobermin wrote: post #43

The modeler who can write code unlike you will get hired first. Okay, now that I said the provocative thing to kind of drive home how real and serious this point is, I will say that I am not a statistician, but I run more computational physics simulations, so less statistical modelling but modelling of experiments and systems based on the PDEs. The one thing I observe is that there really is only this patience for th…

Computation is not theory. Theorists use paper and pencil and sometimes mathematica, not simulation. As for not being able to rebuild a computer, that’s truly ridiculous.

Re: Every modeler is supposed to be a great Python programmer

#112

Post by Bootvis »

fbdab103 wrote: post #36
FridgeSeal wrote: post #7

> every one of them wants Python. I haven’t seen a single one where they’re looking for R or even C++; Python rules this roost. Tried putting R into production recently? It’s a frustrating and brittle experience. Don’t get me wrong, R is fantastic at what it does - analysis, research, statistics, and arguably the API’s on the R data frame packages are a lot saner than Pandas. C++ is out for different reasons I suspec…

>Tried putting R into production recently? It’s a frustrating and brittle experience. Oh man, can I sign onto this rant. While Python has spent a decade+ trying-and-failing to standardize on one of a dozen tools to properly manage dependencies, at least it is trying. R is still global-namespace, no-pinning by default. Sorta-kinda you can squint where renv is going, but still needs a lot of development. I had some hop…

Just use renv (successor of packrat). I had multiple versions of R working with multiple versions of Shiny running in production.

Re: Every modeler is supposed to be a great Python programmer

#113

Post by plastiquebeech »

opportune wrote: post #18

Everybody wants to work on the fun stuff. Turns out in software, getting something to actually deliver value (meaning convert it from prototype or proof of concept to something running in production) is 90% not fun stuff. Most people learn that in their first job. As a software engineer the only way I’d become a “data scientist/trader’s servant” is if I’m getting paid exorbitantly. Otherwise it’s the worst kind of wo…

One of the first lessons that a junior developer learns is:

>My job is not to write code. My job is to solve business problems efficiently, using computers as a medium.

Sounds like data scientists have the same problem. Sometimes, to solve business problems efficiently, you have to step out of your comfort zone and learn to do things that seem ridiculous.

Why just the other day, I had to write a report for humans to read. Using words and pictures! What am I, a technical writer?

Re: Every modeler is supposed to be a great Python programmer

#114

Post by Lyngbakr »

I'm a data engineer who works in an R-centric engineering team, which is quite unusual. Our experience has been that R works well for our use cases and lets us work closely with our analysts and data scientists who are all R users. There are no silos due to different teams speaking different languages. That said, am I still writing Python whenever possible? Damn straight. I'm well aware of how peculiar our team is and, like the post says about modelling, Python (+ SQL) is the default language of the data engineering world. If I want another job, I need Python.

Re: Every modeler is supposed to be a great Python programmer

#115

Post by darepublic »

FridgeSeal wrote: post #65
mynameisash wrote: post #21

Earlier quoted context omitted.

I dunno... I am a software/data engineer who partners with data scientists. I think that comparative advantage here is a real and important. Don't get me wrong, I'm happy when my data scientist partners write good code or show interest in getting better, but I'm more than happy to take their janky code and make it production ready. It often needs to be optimized for scale or refactored for reusability, and a lot of t…

I mean, this is probably how things should be done. Having the specialists of each field appropriately resourced, focused and aligned streamlines everyone’s life. The problem comes in that most places either get the scientists to pull double duty and we end up in the scenario scientists are pushing messy, subpar code out the door that nobody else wants to touch because they’re unsupported and operating out of their d…

Good software dev should be able to refactor the subpar code

Re: Every modeler is supposed to be a great Python programmer

#116

Post by hermitcrab »

From the article: >R is extremely slow at a lot of tasks, for one thing, even more than Python.

Base R is quite slow. R + data.table is faster than Python + Pandas in a benchmark that I did recently.

For a 1 million row CSV file, Read + Sort + self-Join + Write took on a Windows box:

Base R: 47.56s

Python + Pandas: 6.44s

R + data.table: 2.99s

More details at:

https://www.easydatatransform.com/data_wrangling_etl_tools.h...

Re: Every modeler is supposed to be a great Python programmer

#117

Post by carlmr »

version_five wrote: post #2

This was something that surprised me after I did my PhD as well. I thought that employers would focus on my specialized skills and "someone else" would somehow pick up the pieces and make something out of what I did. Turns out this is completely wrong, and I now see how frustrating it is to work with people that have this kind of attitude. Most of most jobs is a bunch of mundane stuff. I've seen it in software develo…

>The "someone else is better at the stuff I don't want to do than me" argument rarely holds up either. The friction that comes from dividing the work along lines like modeling and production and trying to hand off is rarely worth it when one person can do both.

This was always my attitude. Every time you split something you add coordination overhead. This overhead gets worse the more times you split.

Of course there is specialization that can make someone else sufficiently more effective that you shouldn't do everything.

But every gain in specialization has to be weighed against increased communication costs.

Add to that a lot of problems don't need a lot of specialization but touch on many different disciplines.

This is why I think, contrary to the trend of specialization, we need generalists that can cover most bases at once and decrease the communication overhead considerably.

It's often still good to have specialists, but you should mostly employ generalists and only a few specialists in key technologies that set your company apart from others.

Re: Every modeler is supposed to be a great Python programmer

#118

Post by laichzeit0 »

hermitcrab wrote: post #116

From the article: >R is extremely slow at a lot of tasks, for one thing, even more than Python. Base R is quite slow. R + data.table is faster than Python + Pandas in a benchmark that I did recently. For a 1 million row CSV file, Read + Sort + self-Join + Write took on a Windows box: Base R: 47.56s Python + Pandas: 6.44s R + data.table: 2.99s More details at: https://www.easydatatransform.com/data_wrangling_etl_tools…

Reading the data is hardly the problem. Most R libraries seem to be inherently single threaded. Unlike sklearn where just about everything has a n_jobs parameter. Even stuff like xgboost is insanely parallelizable. Try fitting a really big mixed effects model with LMER, you will cry (I have submitted multiple fixes and performance improvements to LMER and Mertools). Cool I have a model. Now I want to serve predictions in real time to downstream APIs. Hello single threaded R runtime my old friend.

Re: Every modeler is supposed to be a great Python programmer

#119

Post by azmodeus »

darepublic wrote: post #115
FridgeSeal wrote: post #65

Earlier quoted context omitted.

I mean, this is probably how things should be done. Having the specialists of each field appropriately resourced, focused and aligned streamlines everyone’s life. The problem comes in that most places either get the scientists to pull double duty and we end up in the scenario scientists are pushing messy, subpar code out the door that nobody else wants to touch because they’re unsupported and operating out of their d…

Good software dev should be able to refactor the subpar code

Sure but it's less fun to be a refactor assistant full time
Post reply on HN