Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don't need data scientists, we need data engineers

mihaileric.com

Post reply on HN
Print view View original

241–250 of 367 posts

Re: We don't need data scientists, we need data engineers

#241

Post by mywittyname »

mushbino wrote: post #228
chadash wrote: post #120

Earlier quoted context omitted.

I worked in investment banking (as an analyst, not an engineer), so very different part of finance, but this was my take as well. Companies might love to talk about how important engineers are, but at the end of the day, if you can't directly link someone to revenue, they get viewed as a cost center and take on second tier status in the organization. Then the same companies complain that they can't find enough (or re…

Engineers get paid well in SV because they are in demand, have lots of employment opportunities, and therefore are more difficult to retain.

And because their contributions can be tied back to revenue. You need both, demand for talent, as well as the ability & justification to pay for it.

Engineers are in high demand all over the world. But most companies do not profit enough from technology to justify similar paying SV salaries.

Re: We don't need data scientists, we need data engineers

#242

Post by mgh2 »

AndrewKemendo wrote: post #25
alex_anglin wrote: post #10

Earlier quoted context omitted.

The aspiration that GP was getting to was that less cleaning is required as a result of better data engineering, I believe.

Correct. If you build your instrumentation correctly, then you don't really need to do any "cleaning." Doesn't mean you might not need to do transformation for different uses but ideally wouldn't need to, for example change data types like turning a bool into an int.

Do data engineers have good analysis skills? Do business analysts have good engineering skills? I don't think either of them can fill the data scientist role.

The scientific training and mindset (scientific method, hypothesis, experiment setup, etc.) to even create an accurate model is an undervalued skill here no? Even if data cleaning is automated, these skills cannot be easily learned.

There is a reason why so many PhDs get into the field, because they were trained in the exploratory/research mindset that no engineering or analytics skills can fill. Correct me if I am wrong.

Re: We don't need data scientists, we need data engineers

#243

Post by ramraj07 »

NikolaNovak wrote: post #168
noodlenotes wrote: post #154

Earlier quoted context omitted.

Also, a lot of data scientists find the science fun and the engineering boring. But they have overlapping skill sets - if you aren't good at one, you're probably not good at the other either. Somebody who shows up to a team with the goal of only modeling and pushing all the dirty engineering work to their teammates is basically a worst case scenario because 1) They probably aren't going to produce good models since t…

Here's the tricky thing: I love your post; I agree with your post; but it takes a 90 degree turn at the end: "My number one advice to entry level data scientists is to not be this guy. " Everything most people are saying here indicates it's GREAT to be that guy. You're paid, you're respected, you get the fun parts, you love your job and it's pretty safe. It just happens to suck for everybody else including team and b…

If you're that guy and you have a secure job it means you write models no one ever sees in a company which doesn't know or respect data, or you work in some data science factory as a small cog of a fairly well oiled team. The latter does happen from time to time, but it's often the former.

In every other place, your job is on the line to be erased because people will soon realize no one wants a wise-ass who doesn't actually contribute much to the bottom-line in the end.

Re: We don't need data scientists, we need data engineers

#244

Post by aqme28 »

nitrogen wrote: post #148
aqme28 wrote: post #87

Earlier quoted context omitted.

> you have a bunch of Data Scientists just handed a pg_dump or flat file from some ops team. I feel seen. At a previous job, our output after some cleaning and transforming was a pg_dump for the data scientists to load. We had little visibility of what they did to that database once they got it.

I suspect in rare cases this is by design, because engineers would object to the behavior of the Business Intelligence department on ethical grounds.

If anything, we would have objected to the quality of the code they were writing.

Re: We don't need data scientists, we need data engineers

#245

Post by ramraj07 »

noodlenotes wrote: post #154
lordnacho wrote: post #112

My experience is in quant hedge funds, where sometimes you get some guys who develop the strategy and some guys who put it into production. Yes, I do admit there can be some specialization in terms of time spent on science vs engineering. But you really need people who understand both. Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a…

Also, a lot of data scientists find the science fun and the engineering boring. But they have overlapping skill sets - if you aren't good at one, you're probably not good at the other either. Somebody who shows up to a team with the goal of only modeling and pushing all the dirty engineering work to their teammates is basically a worst case scenario because 1) They probably aren't going to produce good models since t…

It's hard for most people entering this field because the incentives are perverted - there's this perception that DS is sexy and you actually don't need to know coding that much (just enough to scikit learn). Thus people with pipe dreams of tweaking model hyperparameters to spin gold come in and get a rude awakening. Not a lot unlike people flocking to become actors to LA.

Re: We don't need data scientists, we need data engineers

#246

Post by donkeyd »

This is interesting. I was the technical founder for a data startup that used NLP, elastic and some other stuff for analysis. It's still active, growing and approaching profitability, is used by fortune 500 companies and has had some media attention. However, I've never been approached for a related role and have never been invited after applying for similar roles.

Maybe my resume is bad, maybe my experience doesn't really fit anywhere, but I thought it was an interesting observation in light of this article.

Re: We don't need data scientists, we need data engineers

#247

Post by mywittyname »

urthor wrote: post #209
NikolaNovak wrote: post #168

Earlier quoted context omitted.

Here's the tricky thing: I love your post; I agree with your post; but it takes a 90 degree turn at the end: "My number one advice to entry level data scientists is to not be this guy. " Everything most people are saying here indicates it's GREAT to be that guy. You're paid, you're respected, you get the fun parts, you love your job and it's pretty safe. It just happens to suck for everybody else including team and b…

The flipside is that there are 4x the job posts for data engineering as there are for "that guy". Companies understand that you can't hire five of that guy and get things done. If you have 5-8 years of experience as a technical product manager/data science combo then you are very happy as the magician. But very few magicians are being hired out of college, and a lot of "software engineers in data"

Pretty soon companies are going to start realizing that the 4x DEs can largely replace that 1 DS, and they will be more than happy to do so.

I went into DE because I was kind of forced into the space, but I'd strongly prefer doing full-stack DE. Anymore, I still have the opportunity to build models, they just aren't client-facing stuff, but instead are kind of Data Plumber Bots that help me do my job better so I can waste more time building other fun bots that I can't otherwise be paid for.

Seems like a waste of resources, but my manager could have another DS tomorrow, but my role would take months to fill.

Re: We don't need data scientists, we need data engineers

#248

Post by ramraj07 »

nerdponx wrote: post #39
ziml77 wrote: post #32

Having to deal with data scientists, I absolutely agree. The thing that I've seen that lands in the "lab" vs production distinction is that these people expect their data to be pristine. They flip out when the world isn't as perfect as their models want. Leads to me as just a normal software developer having to do the data analysis and figure out how to clean it up. I also end up having to be the one to talk to data…

Instead of sneering at "having to deal with" data scientists, consider that the data scientists themselves would often much rather have data engineers and dev ops people involved in the process. Data scientists like to quip that 80% of the job is data cleaning, with the remaining 20% divided up arbitrarily among other tasks as suited the joke. In some shops nowadays, it's more like 45% data cleaning, 45% data enginee…

If the DE tooling was shit and you couldn't hire more fast enough, why didn't your team members start addressing these problems? Surely spending half the time cleaning up the pipes would increase the value of what you do with the other half?

Re: We don't need data scientists, we need data engineers

#250

Post by mumblemumble »

jnwatson wrote: post #206
wpietri wrote: post #201

Earlier quoted context omitted.

> Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a way that's detrimental to the rest of the team. My god, this. These people make me bonkers. Especially because I feel like I have a bit of this tendency myself, the desire just to think big thoughts and do no actual work. Happily, I long ago learned that ideas were approximately wort…

"I'm the idea guy" out of someone's mouth is the stark red-flag warning that their net contribution is 0.

Or even negative. I've seen situations where the idea person is so busy being Mr. Toad that everyone around them is regularly scrambling to clean up messes and it ends up being a constant distraction from actually pushing projects through to completion.
Post reply on HN