Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is “data scientist” the new “programmer”?

blogs.harvard.edu

Post reply on HN
Print view View original

101–110 of 246 posts

Re: Is “data scientist” the new “programmer”?

#101

Post by mlthoughts2018 »

saltcured wrote: post #89
mlthoughts2018 wrote: post #14

Earlier quoted context omitted.

Where I work, the data scientists are more educated and experienced on containerization, CI tooling, unit testing, profiling tools, web service prototyping including API validation tools, caching layers, queues, GPU systems programming, etc. etc. We are constantly thwarted by infrastructure teams that use superficial policy basically so they can whine and complain that they don’t want to have to provide support for t…

I feel like you are roughly describing research programmers versus system administrators or operators in academic computing environments. I think a big difference between research programmers and production/ops people is that as researchers we often chase a transient goal. Build some complex and horrible integration to compute a result or put something in a paper. We used to call these Rube Goldberg machines rather t…

I’m not referring to transient research prototypes, but to robust long-lived systems needed for experimentation and reproducible results tracking, and services that are directly customer facing.

We are often required to create new services and functionality because it is how our company can grow, and we have to have ease of access to experimental working space, with freedom to do things like custom compilations of ML frameworks, using programming languages that haven’t been widely used in the company yet to gain access to an important library, define complex assumption-breaking deployment constraints relative to GPU runtimes or containerized notebook servers, etc.

I think people who see how these things grow out of prototypes and wrongly conclude it was designed with transient concerns and thus isn’t robust in some way, they are rushing to judgment and discounting the fact that that ML expert who also wrote the web service layer and who also wrote the Jenkinsfile and who also wrote the container definition abd who also knows how to tune indices in the database, etc., really made their choices for serious, pragmatic engineering reasons that solve the business problem efficiently, and that they already anticipated and accounted for the shallow tradeoffs and caveats that IT will use as potshots to try to circumvent the responsibility to help maintain it.

Re: Is “data scientist” the new “programmer”?

#102

Post by opportune »

natalyarostova wrote: post #90
minimaxir wrote: post #50

Earlier quoted context omitted.

A trend I've been noticing (especially as ML/AI tooling becomes more accessible) is that people believe the quality of data science code and workflows is proportionate to its complexity/LOC (since complex problems require complex code, right?). It's a toxic perspective that ignores recent and pragmatic innovations in the field.

I often see careers built more by complex non maintainable models that show fancy math than more simple ones.

Lol, reminds me of the nightmarishly complicated first NLP model I wrote. I would classify this under "resume driven development" which you see a lot on the software side too with fancy new frameworks.

Funnily enough, although the first place that let me work with Hadoop and Spark didn't need to be using Hadoop and Spark, I probably wouldn't have worked there if they didn't let me learn them, so maybe this isn't as wasteful as it seems at first glance

Re: Is “data scientist” the new “programmer”?

#103

Post by telchar »

mythrwy wrote: post #59
xaranke wrote: post #54

Earlier quoted context omitted.

Can you elaborate more on the "algorithms in JS" bit? What libraries/tools are they using? Why JS versus Python or JVM languages?

Seconded. "Data scientists" and "being most comfortable writing in JS" just sounds strange to me.

I don't want to pile on but that sentence strongly reminded me of the oldish saw about "a data scientist is a programmer who lives in SF". I've never heard of anyone using JS for data science. What might it have that's anywhere comparable to the tidyverse or numpy/scipy/sklearn?

Re: Is “data scientist” the new “programmer”?

#104

Post by mlthoughts2018 »

dasil003 wrote: post #55
mlthoughts2018 wrote: post #14

Earlier quoted context omitted.

Where I work, the data scientists are more educated and experienced on containerization, CI tooling, unit testing, profiling tools, web service prototyping including API validation tools, caching layers, queues, GPU systems programming, etc. etc. We are constantly thwarted by infrastructure teams that use superficial policy basically so they can whine and complain that they don’t want to have to provide support for t…

If the data scientists know more about CI, unit testing, profiling, and caching than the engineers then they are better engineers and I'd wonder a bit about their math/stats chops and whether their role was just re-branded "data scientist" to keep up with trends.

It’s quite common to start out with a PhD / masters in math / stats, with deep specialization in fields like NLP, computer vision, MCMC sampling, and then to become an experienced expert in GPU computing, containerization, web service layers, etc., while working on implementations of ML models.

This was true for me anyway. The main thing I do is deep learning for computer vision and image search, but I think it’s fair to say I have significant experience with Docker, GPU architectures, various CI tooling, linux system programming, deep internals of CPython, internals of MySQL and Postgres, lots of frustrating performance tradeoffs with py4j in the pyspark world, as well as all the usual crap with pandas, sklearn, data visualization tools, and a lot more.

I’d say almost all data scientists I’ve worked with are just like me, just with maybe different specialization areas, except possibly for very young data scientists right out of undergrad.

Re: Is “data scientist” the new “programmer”?

#105

Post by peatmoss »

110285591136 wrote: post #95
EdwardDiego wrote: post #42

Earlier quoted context omitted.

The fact that R has such buy-in despite being a rather awful programming language (a friend of mine worked on the next Lisp-like version of R under Ross Ihaka, and the next version is based on the fact that current R is a bit awful) is precisely because it offers such convenience to non-programmers. In my sister company, they have data scientists, and data engineers. The data scientists write their algorithms in the…

> ... despite being a rather awful programming language (...) it offers such convenience to non-programmers I've heard people say similar things about MATLAB - that it's a poorly designed language, but many that people (mostly non-CS folk) use it out of convenience. Can someone with experience using R explain what makes it so appealing to non-programmers? It seems like these two factors, "poorly designed" and "easy t…

Eh, it’s not as bad as people like to whinge that it is. There are indeed warts, but they’re pretty overblown. If you are comfortable with functional idioms R mostly does what you want without a great deal of fuss. If you’re predisposed to procedural idioms, then you’re going to be fighting the language.

I started learning R about the time I started reading How to Design Computer Programs, and I found it pretty easy to transfer that model of thinking to R. And I find Clojure, Racket, and Scheme to also be somewhat comfortable after a short reacclimation period.

Some of the convenience bits have to do with most functions working on vectors without needing to explicitly iterate most of the time. Also libraries. If you want to estimate a linear regression, or make some exploratory plots, or try some rando statistical method that your graduate advisor suggests, you don’t have to worry about whether it’s already been implemented for you in R.

You can do a lot of heavy lifting by cribbing off of example code because most code is short. You just get heaps of leverage by using R.

Look, I like to do things the hard way a lot. My whole life is pretty much a string of highest friction path choices. For data science R is easy because all the work has been done for you. It's the difference between writing GUI apps against Cocoa APIs vs, I dunno, XLib or Motif.

Re: Is “data scientist” the new “programmer”?

#106

Post by peatmoss »

minimaxir wrote: post #45
EdwardDiego wrote: post #42

Earlier quoted context omitted.

The fact that R has such buy-in despite being a rather awful programming language (a friend of mine worked on the next Lisp-like version of R under Ross Ihaka, and the next version is based on the fact that current R is a bit awful) is precisely because it offers such convenience to non-programmers. In my sister company, they have data scientists, and data engineers. The data scientists write their algorithms in the…

Base R is annoying, but IMO the tidyverse alone makes the language worthwhile.

It’s a great set of DSLs that show off some of the pretty decent meta programming facilities of R. I think those DSLs could be created elsewhere, but I also don’t think it's an accident that it happened in R.

Re: Is “data scientist” the new “programmer”?

#107

Post by peatmoss »

nsriv wrote: post #96
EdwardDiego wrote: post #42

Earlier quoted context omitted.

The fact that R has such buy-in despite being a rather awful programming language (a friend of mine worked on the next Lisp-like version of R under Ross Ihaka, and the next version is based on the fact that current R is a bit awful) is precisely because it offers such convenience to non-programmers. In my sister company, they have data scientists, and data engineers. The data scientists write their algorithms in the…

R is cumbersome, but then you look at Matlab and Stata and try to explain that to social science graduate students that just managed to grok LaTex, and R starts looking like a streamlined vision of the future.

Totally. I’ve taught intro R to mostly social scientists, and intro Python to mostly physical scientists. Every workshop, the R group is coding circles around the Python group with the same number of hours of instruction. And the intro R curriculum that we use was largely a ported version of the Python curriculum.

Having seen brand new scientific programmers tackle two “beginner friendly” languages, man the differences after a day or two are stark.

Re: Is “data scientist” the new “programmer”?

#108

Post by telchar »

sixdimensional wrote: post #80

This post states that a data scientist uses compact languages such as SQL and R. Genuine question - do people really believe that being able to write and understand complex SQL makes you a data scientist? I ask because, I've been writing some of the nastiest, most difficult looking SQL around for probably at least 15 years. And yet, I would NOT call myself a data scientist because I know and can work with data and us…

I've been working as a data scientist for several years and have written some pretty gnarly looking SQL myself. I have a background in math and hard science so I have some understanding of the scientific method as well. While I respect our DBAs I wouldn't call any of them qualified to be data scientists.

While I have been able to hold my own in this job I went back to school to pursue a graduate degree (partly) because being in the field has shown me how much more there is to know. While it's easy enough to train a simple model in R there are so many ways to fool yourself and produce an invalid analysis and so many variations on otherwise-simple problems.

It seems this field has a lot of variation. A glorified report writer might get the DS title but they're not going to get the really cool jobs.

If you're interested in data science try out a kaggle competition and try to place high. The variety of methods and tricks people try to improve their entries can be illuminating, I think.

Post reply on HN