Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don't need data scientists, we need data engineers

mihaileric.com

Post reply on HN
Print view View original

151–160 of 367 posts

Re: We don't need data scientists, we need data engineers

#151

Post by bart_spoon »

jariel wrote: post #9
avs733 wrote: post #2

I teach engineers for a living. I struggle to see how this is not just a straw man argument based on colloquial usage of terms. It is just inferences drawn based on job ads that are rarely written by people doing the job and instead are effectively human-as-seo-optimized so the best candidates can find the job they hopefully fit for and not be too confused to apply for it.

So I think that the delineation between the scientist working with the content, and the Engineers who actually provide the mechanics for it is very fair. If there is a question mark here - it's really how much value are we deriving from all of these data people? Where is all the ML that's changing our lives? Search, Alexa and TikTok, I can see it. In the future obviously vision systems for autonomous cars etc.. But I…

The vast majority of applications of machine learning that is changing the world isn't happening on a consumer level. Its happening in factories, warehouses, farms, logistics chains, etc.

Re: We don't need data scientists, we need data engineers

#152

Post by LeifCarrotson »

lordnacho wrote: post #112

My experience is in quant hedge funds, where sometimes you get some guys who develop the strategy and some guys who put it into production. Yes, I do admit there can be some specialization in terms of time spent on science vs engineering. But you really need people who understand both. Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a…

I really like your magician/blacksmith analogy.

I'm in industrial automation, but it's much the same. Projects where someone developed a strategy but has never been involved in the details of a machine are doomed to failure (or at best to be unreliable and producing low quality parts). Projects built by machine fabricators are over-engineered, frequently late, and sometimes unprofitable, but damn if they don't work well.

The main trouble, I think, is that when a shiny new contraption is brought to the king, it's too often the magicians doing the talking - whether they're speaking words of power or Common, their job is to talk. Meanwhile, the blacksmith is probably busy at in his workshop some ornate scroll work for the next thing, or repairing the previous gizmo, because he'd rather be hammering away at his anvil than talking.

The higher you go in an org chart, the fewer the number of people who understand the work their company actually does, and the more voices you have between the workers and the decision-makers to take some of the credit for work as it passes up the chain.

Re: We don't need data scientists, we need data engineers

#153

Post by superhuzza »

listenallyall wrote: post #115
alexfromapex wrote: post #30

Earlier quoted context omitted.

I agree, if anything the data engineers (folks with engineering backgrounds) should be doing the applied work while a department of data scientists works on the theoretical or novel data analysis methods. Right now our product has accumulated a lot of technical debt on the data validation side because data scientists designed the test code in a way that dramatically slows the development process.

> novel data analysis methods Many "data scientists" (not all, but many) have little to no ability to do anything other than apply "recipes" of algorithms or classification methods or logistic regressions, etc. Asking them to develop a "novel" method would be fruitless. Asking them to clean and scrub the source data set is like telling an amateur pie-baker the store was out of pie crusts, you'll have to make your own…

>Asking them to clean and scrub the source data set...it's not going to happen, they just don't have that skill

I think you've been working with conmen/conwomen. I've never seen a data science project that doesn't involve data cleaning or wrangling of some sort.

Re: We don't need data scientists, we need data engineers

#154

Post by noodlenotes »

lordnacho wrote: post #112

My experience is in quant hedge funds, where sometimes you get some guys who develop the strategy and some guys who put it into production. Yes, I do admit there can be some specialization in terms of time spent on science vs engineering. But you really need people who understand both. Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a…

Also, a lot of data scientists find the science fun and the engineering boring. But they have overlapping skill sets - if you aren't good at one, you're probably not good at the other either. Somebody who shows up to a team with the goal of only modeling and pushing all the dirty engineering work to their teammates is basically a worst case scenario because

1) They probably aren't going to produce good models since they're not sensitive to data nuances, but now they've taken over ALL the modeling work.

2) They bring down the job satisfaction of everyone else on the team who would like to be doing at least some modeling.

3) They're sucking up the prestige that should be distributed over the entire team and management thinks they should be paid more for work that it turns out everybody thinks is more fun anyway.

My number one advice to entry level data scientists is to not be this guy. Don't give your interviewers the impression that you won't do your own engineering work because they won't want someone who brings negative value to the team.

Re: We don't need data scientists, we need data engineers

#155

Post by tchalla »

astrophysician wrote: post #68
ziml77 wrote: post #32

Having to deal with data scientists, I absolutely agree. The thing that I've seen that lands in the "lab" vs production distinction is that these people expect their data to be pristine. They flip out when the world isn't as perfect as their models want. Leads to me as just a normal software developer having to do the data analysis and figure out how to clean it up. I also end up having to be the one to talk to data…

Just want to say that while the data science profession definitely includes a wide range of people and skillsets, a good data scientist should be practical and able to work with the available data in whatever state it's in. No good data scientist should ever expect data to be pristine. And a good data scientist, even if they don't have quite the engineering chops necessary to build a production-quality ETL, should kn…

A good data scientist should also be good at science. Otherwise, you can simply hire people with engineering skills - you don't need scientists. If you hire scientists and then are surprised they aren't good at engineering, the hiring process needs a reality check.

Re: We don't need data scientists, we need data engineers

#156

Post by hef19898 »

LeifCarrotson wrote: post #152
lordnacho wrote: post #112

My experience is in quant hedge funds, where sometimes you get some guys who develop the strategy and some guys who put it into production. Yes, I do admit there can be some specialization in terms of time spent on science vs engineering. But you really need people who understand both. Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a…

I really like your magician/blacksmith analogy. I'm in industrial automation, but it's much the same. Projects where someone developed a strategy but has never been involved in the details of a machine are doomed to failure (or at best to be unreliable and producing low quality parts). Projects built by machine fabricators are over-engineered, frequently late, and sometimes unprofitable, but damn if they don't work w…

That seems to be true in every field I can think of. The smaller the gap, or rather the more practical experience the strategy people have, the better a given org seems to be.

One common issue I run into is that when the blacksmiths start talking, nobody listens.

Re: We don't need data scientists, we need data engineers

#157

Post by simo7 »

tchalla wrote: post #149
listenallyall wrote: post #115

Earlier quoted context omitted.

> novel data analysis methods Many "data scientists" (not all, but many) have little to no ability to do anything other than apply "recipes" of algorithms or classification methods or logistic regressions, etc. Asking them to develop a "novel" method would be fruitless. Asking them to clean and scrub the source data set is like telling an amateur pie-baker the store was out of pie crusts, you'll have to make your own…

> Many "data scientists" (not all, but many) have little to no ability to do anything other than apply "recipes" of algorithms or classification methods or logistic regressions, etc. This is because they rarely hire people with scientific thinking ability. They just hire people who can code and program from set recipes. Once you hire such people you can not expect them to do non-recipe work. If you don't want recipe…

Precisely.

Re: We don't need data scientists, we need data engineers

#158

Post by oivey »

lordnacho wrote: post #112

My experience is in quant hedge funds, where sometimes you get some guys who develop the strategy and some guys who put it into production. Yes, I do admit there can be some specialization in terms of time spent on science vs engineering. But you really need people who understand both. Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a…

I think this insight exists across a lot of fields. Basically, if you want to be a really excellent magician you also better be a decent blacksmith. More concretely in this case, if you’re unable to do the data “engineering” yourself then it will close a lot of doors for interesting and novel work on the “science” side. Beyond that, if the scientist’s job just involves gluing sklearn models together I think that job is more on the engineering side of things than the supposed scientist usually wants to admit.

Re: We don't need data scientists, we need data engineers

#159

Post by thecolorgreen »

pricci wrote: post #144
C4stor wrote: post #18

I can't recommend the Data Engineer career enough for junior developers. It's how I started and what I pursued for 6 years (and I would love doing it again), and I feel like it gave me such an incredible foundation for future roles : - Actually big data (so, not something you could grep...) will trigger your code in every possible way. You quickly learn that with trillions of input, the probabily to reach a bug is ei…

And where would you recommend someone to start a data engineering path. Any book, learning source?

I have the same question and I believe the answer is in the same vein as someone who asks about software engineering. Books/courses are great for the concepts, but your goal should be to build something ASAP since that's where actual learning will come from.

Re: We don't need data scientists, we need data engineers

#160

Post by curiousgal »

inthewoods wrote: post #129
lordnacho wrote: post #112

My experience is in quant hedge funds, where sometimes you get some guys who develop the strategy and some guys who put it into production. Yes, I do admit there can be some specialization in terms of time spent on science vs engineering. But you really need people who understand both. Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a…

That's interesting - I just completed book on Jim Simon/Renaissance (The Man Solved The Market). One of their early advantages was having a person who was just focused on acquiring and cleaning data. I expect that advantage has largely gone away at this point due to wide availability of market data but I thought it was interesting in the context of this article.

Same for CFM too, they have an entire team working on alternative data and they feed it to a modeling team.
Post reply on HN