Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is “data scientist” the new “programmer”?

blogs.harvard.edu

Post reply on HN
Print view View original

131–140 of 246 posts

Re: Is “data scientist” the new “programmer”?

#131

Post by portal_narlish »

Data scientist, programmer and software engineer are different things. They are not disjoint by any means, but this guy is conflating them in a way that's totally wrong.

Software engineers have to engineer things. They deal with production applications, distributed systems, concurrency, build systems, microservices... coding is sometimes only a small part of the job.

Data scientists nowadays do programming in interest of research, modeling and data visualization. But they are not only programmers - they are usually supposed to have an applied statistics or research background. Some also do software engineering, especially at companies serving data science/ML in their products.

A programmer is actually someone like a data analyst or business systems developer. They don't have to build systems themselves, they just write loosely structured code against existing systems. Like writing SQL queries for dashboards, or drop-in code for things like Salesforce. This is probably the closest thing to what he's describing as the "70s archetype". Minus the deep optimization stuff.

Re: Is “data scientist” the new “programmer”?

#132

Post by joycian »

EdwardDiego wrote: post #42
danso wrote: post #36

This is such a bizarre post. The reason why people use a language like R is because it is easy to learn and use (and install, via RStudio) for data analysis without having to be a well-trained programmer. I can’t recall ever hearing from anyone who has relied on R, doing so because it was computationally efficient. The point of the language is convenience — particularly with how easy it is to create attractive graphi…

The fact that R has such buy-in despite being a rather awful programming language (a friend of mine worked on the next Lisp-like version of R under Ross Ihaka, and the next version is based on the fact that current R is a bit awful) is precisely because it offers such convenience to non-programmers. In my sister company, they have data scientists, and data engineers. The data scientists write their algorithms in the…

> The data scientists write their algorithms in the language they're most comfortable with (typically JS)

This seems weird to me, I associate JS with web dev (and even more with the front end side of it)

Re: Is “data scientist” the new “programmer”?

#133

Post by kosei »

portal_narlish wrote: post #131

Data scientist, programmer and software engineer are different things. They are not disjoint by any means, but this guy is conflating them in a way that's totally wrong. Software engineers have to engineer things . They deal with production applications, distributed systems, concurrency, build systems, microservices... coding is sometimes only a small part of the job. Data scientists nowadays do programming in intere…

I think you're taking this analogy too literally.

Re: Is “data scientist” the new “programmer”?

#134

Post by yvrev »

telchar wrote: post #108
sixdimensional wrote: post #80

This post states that a data scientist uses compact languages such as SQL and R. Genuine question - do people really believe that being able to write and understand complex SQL makes you a data scientist? I ask because, I've been writing some of the nastiest, most difficult looking SQL around for probably at least 15 years. And yet, I would NOT call myself a data scientist because I know and can work with data and us…

I've been working as a data scientist for several years and have written some pretty gnarly looking SQL myself. I have a background in math and hard science so I have some understanding of the scientific method as well. While I respect our DBAs I wouldn't call any of them qualified to be data scientists. While I have been able to hold my own in this job I went back to school to pursue a graduate degree (partly) becau…

I'll preface this with I've not had a look at any Kaggle competition, but I always assumed Kaggle competitions was on par with programming competitions in terms of how the skills transfer professionally. A great programmer is not necessarily great at programming competitions after all.

Am I way off here?

Re: Is “data scientist” the new “programmer”?

#135

Post by portal_narlish »

skybrian wrote: post #69
thidr0 wrote: post #41

Something about this article strikes me as a thinly-veiled complaint about poorly designed object-oriented systems. Take, for example this comment by the author: >Even if the money were half of what today’s coder gets paid it might still be a better job because one is spared the tedium of looking at millions of lines of Java that do almost nothing! What all those millions of lines of code are is abstractions, decoupl…

All of today's software is built on millions of lines of code. This isn't much of a problem when it's hidden away behind a good abstraction. Being written in another language forces the API to be documented well enough so you don't need to go deeper: "native code", "kernel code", "part of the browser". Crappy million-line Java apps are generally crappy not due to raw line count but rather due to leaky abstractions an…

I sometimes wonder if programmers instinctively overcomplicate things in interest of collective job security. Some of the stuff I've seen in (particularly awful) Java code bases is perplexing to the point where it seems intentional.

Re: Is “data scientist” the new “programmer”?

#136

Post by tanilama »

I don't think so.

Per my observation, the most 'interesting' part of a data scientist's job is story telling, that is using data analysis to draft a theory to push forward for product direction. Some of the ML engineers works under Data Scientist umbrella, but since the DL thing happens, they are now putting under even fancier titles like AI Engineers or such.

So data scientists are really product manager/owner with analysis skills. Is this job interesting? For sure, when it follows this definition. Interesting? That only depends on the problem domain, not the title, IMHO.

Re: Is “data scientist” the new “programmer”?

#137

Post by disgruntledphd2 »

yvrev wrote: post #134
telchar wrote: post #108

Earlier quoted context omitted.

I've been working as a data scientist for several years and have written some pretty gnarly looking SQL myself. I have a background in math and hard science so I have some understanding of the scientific method as well. While I respect our DBAs I wouldn't call any of them qualified to be data scientists. While I have been able to hold my own in this job I went back to school to pursue a graduate degree (partly) becau…

I'll preface this with I've not had a look at any Kaggle competition, but I always assumed Kaggle competitions was on par with programming competitions in terms of how the skills transfer professionally. A great programmer is not necessarily great at programming competitions after all. Am I way off here?

Nope. Kaggle just covers the modelling part, which is normally much easier than figuring out how to solve business problems using data.

Re: Is “data scientist” the new “programmer”?

#138

Post by dnautics »

vinayms wrote: post #127

The take down on abstraction and software engineers (by using Java as an example) is similar to saying "back in the day to find a prime number we would simply use a sieve, but today it is a tedium, what with all the pi's and e's and thetas that get in the way, and what are geometry and polynomials doing here, and what in the god's name is this i , I just want to count the prime numbers which are nice round whole numb…

I'm a scientist (wet lab) by training, a programmer (back end) by profession, and a data scientist by hobby (I have a machine learning project that I'm working on), and most of "data science" is not really stats... There will be a bit of stats at the end product but really the bulk of the necessary work is data curation. Annoying stuff like making sure my data fit into the right buckets.

I did have to debug a memory leak that only showed up when I deployed my data pipeline on 22 cores.

Re: Is “data scientist” the new “programmer”?

#140

Post by PeterisP »

portal_narlish wrote: post #135
skybrian wrote: post #69

Earlier quoted context omitted.

All of today's software is built on millions of lines of code. This isn't much of a problem when it's hidden away behind a good abstraction. Being written in another language forces the API to be documented well enough so you don't need to go deeper: "native code", "kernel code", "part of the browser". Crappy million-line Java apps are generally crappy not due to raw line count but rather due to leaky abstractions an…

I sometimes wonder if programmers instinctively overcomplicate things in interest of collective job security. Some of the stuff I've seen in (particularly awful) Java code bases is perplexing to the point where it seems intentional.

It's more likely the natural entropy of code - it's easy to add stuff to a system in a way that makes it more messy; and if the system already is a big mess, then it's much harder to do non-messy additions and the bigger mess it is, the harder it is to start cleaning it up.
Post reply on HN