Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don't need data scientists, we need data engineers

mihaileric.com

Post reply on HN
Print view View original

111–120 of 367 posts

Re: We don't need data scientists, we need data engineers

#111

Post by SirSourdough »

slt2021 wrote: post #47
alexpetralia wrote: post #38

Earlier quoted context omitted.

These are incredibly disparate skill sets. Of course anyone would want to hire someone like this, and far more would claim to possess such a broad skill set, but in practice it is extremely rare. You'd need someone with excellent communication skills (presentation, memo writing, teamwork), project management skills (identifying & overcoming workflow bottlenecks), professional skills (timely responses, political savvy…

exactly, these are characteristics of a unicorn and I think most of these skills are trivial to build up over time through practice and self-learning and these skills can yield great benefits both for employers and employees

I guess in a vacuum each of those skills is easy to build up through practice and self-learning (which, lets remember, many people struggle with to begin with). However, I think the fact that you refer to people possessing all of them as "unicorns" should be telling as far as how trivial it actually is to build all these skills beyond a simply passable level.

Re: We don't need data scientists, we need data engineers

#112

Post by lordnacho »

My experience is in quant hedge funds, where sometimes you get some guys who develop the strategy and some guys who put it into production.

Yes, I do admit there can be some specialization in terms of time spent on science vs engineering.

But you really need people who understand both. Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a way that's detrimental to the rest of the team. You get people who just don't appreciate that there's other work to do than finding models, and that models depend on that other work to function.

You also get a huge prestige gap, because inevitably management will think that there's a magician and a blacksmith. One guy needs to be paid a lot, and the other guy needs to be paid enough.

These effects feed each other. Magician will say "where's my data" and expect blacksmith to make it, promptly. He won't do it himself, because spending time on mundane stuff makes the magic disappear. And not doing it yourself, or taking the time to understand it, will eventually lead to problems with the magic.

Re: We don't need data scientists, we need data engineers

#113

Post by marcinzm »

alexpetralia wrote: post #38
slt2021 wrote: post #35

Full Stack Data Scientist (data janitor + data engineer + ML engineer + ML Ops + Business Analyst) is the future

These are incredibly disparate skill sets. Of course anyone would want to hire someone like this, and far more would claim to possess such a broad skill set, but in practice it is extremely rare. You'd need someone with excellent communication skills (presentation, memo writing, teamwork), project management skills (identifying & overcoming workflow bottlenecks), professional skills (timely responses, political savvy…

I, interestingly enough, have that skill set (mostly) and probably a broader set of technical skills than you're imagining. I use it to hire a team of specialists under me and interact with other specialized teams (ie: I speak their language) since I lack depth in too many areas. I wouldn't ever imagine hiring a clone of myself except in cases where I can't build out a larger team for a long period of time.

Re: We don't need data scientists, we need data engineers

#114

Post by civilized »

mynameisash wrote: post #83
AndrewKemendo wrote: post #3

Preach! The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and engineers to do everything that empowers Data Science and ML. We need more Data Engineers involved at time zero in projects to help: 1. Plan out what data should be produced/captured by the product 2. Instrument systems to actually generate data consistently and effec…

> What ends up happening is you have a bunch of Data Scientists just handed a pg_dump or flat file from some ops team Not to disparage the amazing data scientists I've worked with, but I've been on teams where this is very much the approach to operationalizing models. It's basically, "Here's the sklearn model and some fragile featurization scripts we built. Can you take this to prod ASAP?" The problem I've seen is th…

This is 100% my experience as a data scientist. The engineering support we get is restricted to submitting a ticket for database access or moving data from one system to another. Wouldn't dream of involving an engineer in a data science project team, because I have no evidence that they have any experience or expertise in anything other than tickets to move data around.

Re: We don't need data scientists, we need data engineers

#115

Post by listenallyall »

alexfromapex wrote: post #30
AndrewKemendo wrote: post #3

Preach! The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and engineers to do everything that empowers Data Science and ML. We need more Data Engineers involved at time zero in projects to help: 1. Plan out what data should be produced/captured by the product 2. Instrument systems to actually generate data consistently and effec…

I agree, if anything the data engineers (folks with engineering backgrounds) should be doing the applied work while a department of data scientists works on the theoretical or novel data analysis methods. Right now our product has accumulated a lot of technical debt on the data validation side because data scientists designed the test code in a way that dramatically slows the development process.

> novel data analysis methods

Many "data scientists" (not all, but many) have little to no ability to do anything other than apply "recipes" of algorithms or classification methods or logistic regressions, etc. Asking them to develop a "novel" method would be fruitless. Asking them to clean and scrub the source data set is like telling an amateur pie-baker the store was out of pie crusts, you'll have to make your own from scratch -- it's not going to happen, they just don't have that skill, the instructions on the box don't account for that possibility. As soon as the task diverges from the simple step 1, step 2, step 3 that they were originally taught, you realize they have very little ability to adapt. YMMV of course.

Re: We don't need data scientists, we need data engineers

#116

Post by valarauko »

slt2021 wrote: post #47
alexpetralia wrote: post #38

Earlier quoted context omitted.

These are incredibly disparate skill sets. Of course anyone would want to hire someone like this, and far more would claim to possess such a broad skill set, but in practice it is extremely rare. You'd need someone with excellent communication skills (presentation, memo writing, teamwork), project management skills (identifying & overcoming workflow bottlenecks), professional skills (timely responses, political savvy…

exactly, these are characteristics of a unicorn and I think most of these skills are trivial to build up over time through practice and self-learning and these skills can yield great benefits both for employers and employees

I think the point is that these skills are not trivial to build up over time

Re: We don't need data scientists, we need data engineers

#117

Post by names_are_hard »

nerdponx wrote: post #50
mgh2 wrote: post #6

Earlier quoted context omitted.

Who will do the proper cleaning then?

It doesn't matter, as long as you don't make the person with the PhD in biostatistics spend their time writing ETL pipelines, which is a wildly inefficient use of a very expensive resource.

Do people with PhDs in biostatistics earn significantly more than programmers? I honestly know nothing about the market for biostatisticians, but my impression was that advanced degrees in the natural sciences don't really pay that well compared to software engineers, especially given that they're much more educated.

Re: We don't need data scientists, we need data engineers

#119

Post by hankchinaski »

the article is long overdue. so many times i have seen data scientists glue together non production ready code and libraries expecting someone else to finish the job and put the project to production. in various places i worker at they have soon realised that we mostly and exclusively needed data engineers - glorified DevOps people that make production ready data pipelines - more than we needed data scientists to do “modelling”

Re: We don't need data scientists, we need data engineers

#120

Post by chadash »

lordnacho wrote: post #112

My experience is in quant hedge funds, where sometimes you get some guys who develop the strategy and some guys who put it into production. Yes, I do admit there can be some specialization in terms of time spent on science vs engineering. But you really need people who understand both. Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a…

I worked in investment banking (as an analyst, not an engineer), so very different part of finance, but this was my take as well. Companies might love to talk about how important engineers are, but at the end of the day, if you can't directly link someone to revenue, they get viewed as a cost center and take on second tier status in the organization. Then the same companies complain that they can't find enough (or retain) engineering talent. Not many places get the balance right. Silicon valley treats engineers well because for the most part, the value they bring is more obvious (and also, they don't threaten the existing hierarchy in the company). Curious to hear if anyone has had the opposite experience.
Post reply on HN