We don't need data scientists, we need data engineers • Hacker News Board

Re: We don't need data scientists, we need data engineers

#201

Post by wpietri » Thu, Jan 14, 2021, 8:02 PM UTC

lordnacho wrote: post #112

My experience is in quant hedge funds, where sometimes you get some guys who develop the strategy and some guys who put it into production. Yes, I do admit there can be some specialization in terms of time spent on science vs engineering. But you really need people who understand both. Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a…

> Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a way that's detrimental to the rest of the team.

My god, this. These people make me bonkers. Especially because I feel like I have a bit of this tendency myself, the desire just to think big thoughts and do no actual work. Happily, I long ago learned that ideas were approximately worthless without labor, and that I anyway had much better ideas when laboring because it forced me to engage with the details.

And yes, those people can poison a team. My best working experiences have all been with people who a) all valued actual work and b) believed that everybody could have good ideas.

Re: We don't need data scientists, we need data engineers

#202

Post by wpietri » Thu, Jan 14, 2021, 8:05 PM UTC

hntrader wrote: post #199

Karrot_Kream wrote: post #198

Earlier quoted context omitted.

> 1/3 statistician, 1/3 developer and 1/3 trader How is being a trader different from being a statistician? Curious as I've never worked in finance before.

By trader, I mean domain knowledge about the markets. Statistics is the toolbox that this domain expert uses to test their hypotheses and turn them into a profitable model. But if the person isn't a domain expert and only knows statistics, their ideas about what to test won't be good.

And to knowledge, I'd add disposition. It's been years since I've been in finance, but the best traders I worked with were all very driven to succeed, to dominate, to win. Markets were really interesting to me, but I never cared much about that part.

Re: We don't need data scientists, we need data engineers

#203

Post by superhuzza » Thu, Jan 14, 2021, 8:06 PM UTC

listenallyall wrote: post #188

superhuzza wrote: post #153

Earlier quoted context omitted.

>Asking them to clean and scrub the source data set...it's not going to happen, they just don't have that skill I think you've been working with conmen/conwomen. I've never seen a data science project that doesn't involve data cleaning or wrangling of some sort.

Have you read through the comment thread? Did you read the article? Most everyone is in agreement that projects require a lot of cleaning & wrangling and a lot more -- the point is that data scientists are generally not doing that stuff, they expect academic-quality, pre-processed, pristine data, so it's data engineers who are stuck preparing the data, and who are in high demand.

Yes I read both the article and the comments.

I meant a data science project in terms of a project completed by data scientists. In my experience, all data scientists are accustomed to doing extensive cleaning etc.

Re: We don't need data scientists, we need data engineers

#204

Post by theflyinghorse » Thu, Jan 14, 2021, 8:07 PM UTC

C4stor wrote: post #18

I can't recommend the Data Engineer career enough for junior developers. It's how I started and what I pursued for 6 years (and I would love doing it again), and I feel like it gave me such an incredible foundation for future roles : - Actually big data (so, not something you could grep...) will trigger your code in every possible way. You quickly learn that with trillions of input, the probabily to reach a bug is ei…

I wonder how: 1. one finds organizations that have data engineering 2. gets hired to said organization with software engineering background.

Re: We don't need data scientists, we need data engineers

#205

Post by wpietri » Thu, Jan 14, 2021, 8:10 PM UTC

coding123 wrote: post #21

A couple of us inherited a machine learning project a while back. The code was horrible. Riddled with copy pasta (nearly half of the entire thing was copy paste and no code reuse). We basically refactored everything, standardized input and output file names. We put up a small Flask service to allow outside services hit it easily and wrapped it up in a Docker container so it was ultimately easy to deploy. Yes it was a…

The phrase "If you can't dazzle them with brilliance, baffle them with bullshit" comes to mind.

Re: We don't need data scientists, we need data engineers

#206

Post by jnwatson » Thu, Jan 14, 2021, 8:11 PM UTC

wpietri wrote: post #201

lordnacho wrote: post #112

My experience is in quant hedge funds, where sometimes you get some guys who develop the strategy and some guys who put it into production. Yes, I do admit there can be some specialization in terms of time spent on science vs engineering. But you really need people who understand both. Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a…

> Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a way that's detrimental to the rest of the team. My god, this. These people make me bonkers. Especially because I feel like I have a bit of this tendency myself, the desire just to think big thoughts and do no actual work. Happily, I long ago learned that ideas were approximately wort…

"I'm the idea guy" out of someone's mouth is the stark red-flag warning that their net contribution is 0.

Re: We don't need data scientists, we need data engineers

#207

Post by Dumblydorr » Thu, Jan 14, 2021, 8:11 PM UTC

Dumblydorr wrote: post #102

I'm a scientist first, data person second. I'll use the label data scientist if that's what the employer wants. I'll use data analyst, scientist, researcher, or any other term, so long as they pay me. What matters is: are we contributing to the betterment of society with data-driven decision making?

Another aspect: so many PhDs and postdocs in the sciences do similar work but get paid 1/3 or less of the wages. We shouldn't be surprised they snap up industry jobs in data science, it's way more remuneration than academics, with less stress IMO

Re: We don't need data scientists, we need data engineers

#208

Post by theflyinghorse » Thu, Jan 14, 2021, 8:13 PM UTC

baron_harkonnen wrote: post #88

ziml77 wrote: post #32

Having to deal with data scientists, I absolutely agree. The thing that I've seen that lands in the "lab" vs production distinction is that these people expect their data to be pristine. They flip out when the world isn't as perfect as their models want. Leads to me as just a normal software developer having to do the data analysis and figure out how to clean it up. I also end up having to be the one to talk to data…

The data science field has been flooded with PhDs with nowhere else to go that have no background in engineering, and sadly often have a very poor understanding of both machine learning and statistics. Companies were in a rush hire "data scientists" and boot camps like Insight were more than happy to pump out very impressive PhDs with just enough understanding to build a Keras model. I've worked in industry awhile do…

I work at a place with a very high count of PhDs. Some of them write code. All of them view writing code as something menial and unimportant and its shows in the resulting work, which from my experience is atrocious.

Of course I understand that YMV, but I will forever be skeptical of anyone writing code with a PhD after working here.

Re: We don't need data scientists, we need data engineers

#209

Post by urthor » Thu, Jan 14, 2021, 8:14 PM UTC

NikolaNovak wrote: post #168

noodlenotes wrote: post #154

Earlier quoted context omitted.

Also, a lot of data scientists find the science fun and the engineering boring. But they have overlapping skill sets - if you aren't good at one, you're probably not good at the other either. Somebody who shows up to a team with the goal of only modeling and pushing all the dirty engineering work to their teammates is basically a worst case scenario because 1) They probably aren't going to produce good models since t…

Here's the tricky thing: I love your post; I agree with your post; but it takes a 90 degree turn at the end: "My number one advice to entry level data scientists is to not be this guy. " Everything most people are saying here indicates it's GREAT to be that guy. You're paid, you're respected, you get the fun parts, you love your job and it's pretty safe. It just happens to suck for everybody else including team and b…

The flipside is that there are 4x the job posts for data engineering as there are for "that guy".

Companies understand that you can't hire five of that guy and get things done. If you have 5-8 years of experience as a technical product manager/data science combo then you are very happy as the magician. But very few magicians are being hired out of college, and a lot of "software engineers in data"

Re: We don't need data scientists, we need data engineers

#210

Post by hntrader » Thu, Jan 14, 2021, 8:14 PM UTC

wpietri wrote: post #202

hntrader wrote: post #199

Earlier quoted context omitted.

By trader, I mean domain knowledge about the markets. Statistics is the toolbox that this domain expert uses to test their hypotheses and turn them into a profitable model. But if the person isn't a domain expert and only knows statistics, their ideas about what to test won't be good.

And to knowledge, I'd add disposition. It's been years since I've been in finance, but the best traders I worked with were all very driven to succeed, to dominate, to win. Markets were really interesting to me, but I never cared much about that part.

Yeah, it's a performance discipline like any other (competitive gaming, athletics, etc) where only the top few % can succeed. If someone isn't very driven then they won't make it.