Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don't need data scientists, we need data engineers

mihaileric.com

Post reply on HN
Print view View original

301–310 of 367 posts

Re: We don't need data scientists, we need data engineers

#301

Post by red_hare »

hntrader wrote: post #199
Karrot_Kream wrote: post #198

Earlier quoted context omitted.

> 1/3 statistician, 1/3 developer and 1/3 trader How is being a trader different from being a statistician? Curious as I've never worked in finance before.

By trader, I mean domain knowledge about the markets. Statistics is the toolbox that this domain expert uses to test their hypotheses and turn them into a profitable model. But if the person isn't a domain expert and only knows statistics, their ideas about what to test won't be good.

This is almost precisely the thought process behind how my company hires data scientists who build user-facing analysis.

1/3 statistician 1/3 engineer 1/3 product person who can learn the user's domain-specific needs

Re: We don't need data scientists, we need data engineers

#302

Post by baron_harkonnen »

jhap wrote: post #290
baron_harkonnen wrote: post #88

Earlier quoted context omitted.

The data science field has been flooded with PhDs with nowhere else to go that have no background in engineering, and sadly often have a very poor understanding of both machine learning and statistics. Companies were in a rush hire "data scientists" and boot camps like Insight were more than happy to pump out very impressive PhDs with just enough understanding to build a Keras model. I've worked in industry awhile do…

> The data science field has been flooded with PhDs with nowhere else to go that have no background in engineering, and sadly often have a very poor understanding of both machine learning and statistics. I am a PhD student in a non-engineering field. I've been taking as many math and stats courses as I can, but what other courses should I be trying to take if I want to excel as a data scientist? Software engineering…

My question is: "Why are you pursuing a PhD if you want to end up as a data scientist?"

I've known a surprisingly large number of people that are mid-phd thinking about data science as a career. Don't pursue 5+ years of learning to master the world of academic research if your goal is to help people sell t-shirts or whatever.

Certainly there are some people pursuing specific PhDs, such as those in computer vision and nlp where there are some industry options that might offer more challenging/interesting research than academia. It makes sense if you're a PhD at NYU or Stanford in CS fields related to neural networks to go work for Yann Lecun at Facebook or Geoffrey Hinton at Google.

But if you're, say a biologist that wants to sell clothes online... why spend 6 years working in academia to do that? Is your dream really to optimize clothing sales? If so don't be a biologist. If your dream is biology, why in the world would you set your course on selling clothes?

I get it if your dream is biology but you can't find a tenure track job and so you pivot to industry... but if you are mid-phd, what are you doing there? If you love your subject, try to find a way to work in that and if you don't, don't waste your time.

Data Science is not a glamorous job, and the vast majority of companies it is literally bullshit. The people solving mind-bendingly hard problems are already in programs specializing in those problems because that's what they are passionate about. On top of that DS is way over indexed at most companies. If you're mid-phd now I would expect a serious contraction in DS jobs in the next 5 years. DS will be a niche job after the next market "correction"

Re: We don't need data scientists, we need data engineers

#303

Post by baron_harkonnen »

lordgrenville wrote: post #258
theflyinghorse wrote: post #208

Earlier quoted context omitted.

I work at a place with a very high count of PhDs. Some of them write code. All of them view writing code as something menial and unimportant and its shows in the resulting work, which from my experience is atrocious. Of course I understand that YMV, but I will forever be skeptical of anyone writing code with a PhD after working here.

Are they CS/EE PhDs?

Think back to your own CS professors, were any of them particular good software engineers?

I've found that Physics PhDs tend to have the highest probability of being good coders since a certain subset of them get bit by the software bug when they need to write non-trivial amounts of code to solve research problems.

Re: We don't need data scientists, we need data engineers

#304

Post by baron_harkonnen »

boterham wrote: post #133
baron_harkonnen wrote: post #88

Earlier quoted context omitted.

The data science field has been flooded with PhDs with nowhere else to go that have no background in engineering, and sadly often have a very poor understanding of both machine learning and statistics. Companies were in a rush hire "data scientists" and boot camps like Insight were more than happy to pump out very impressive PhDs with just enough understanding to build a Keras model. I've worked in industry awhile do…

Do you have any suggestions for where to start looking for good places to apply that don't suffer from this?

I personally have given up and turned mercenary. Even if you're passionate about statistics, machine learning, or any ds related discipline don't think of work as being a bigger part of your identity than the average star bucks barista does. Find a team/company that pays you well and isn't too opinionated, with low ego (if possible). Don't look for challenging work, the few places it exists already have the people they need, whereas the companies that pretend they have challenging problems tend to be insufferable. Look for a team where you can check-in and check-out without too much stress and get paid well.

Re: We don't need data scientists, we need data engineers

#305

Post by bigger_cheese »

AndrewKemendo wrote: post #3

Preach! The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and engineers to do everything that empowers Data Science and ML. We need more Data Engineers involved at time zero in projects to help: 1. Plan out what data should be produced/captured by the product 2. Instrument systems to actually generate data consistently and effec…

>The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and engineers to do everything that empowers Data Science and ML.

This matches my observations as well. I'm an engineer (The non software kind) at an Industrial plant, I have noticed similar in my involvement with data scientists.

I think in a lot of cases it needs to be acknowledged that data scientists are not domain subject matter experts. Very often the data scientists we have worked with lack knowledge I take for granted as an engineer such as knowledge of basic chemistry, physics etc. I can sanity check plant data almost instantly. For example I will know if a material reacts in an endothermic or exothermic manner and can verify that its effect on a temperature prediction model make sense.

As a result I often feel like Data Scientists are not empowered to bring their full expertise to bear, they don't understand our process fully and lack a lot "engineering" knowledge to make value added inferences about what their models are demonstrating. Often they can deliver a model and show that a particular term is significant but they have a very shallow understanding of what the term actually represents and can't provide concrete recommendations as to how we could modify our plant to benefit from what their model is demonstrating.

Sometimes I feel like we need an additional translator sitting between who can speak both "Data science" and "Engineer" I don't think this is quite what "Data Engineer" as suggested by parent article is but possibly the role could be expanded to incorporate this.

Re: We don't need data scientists, we need data engineers

#306

Post by Threeve303 »

eftychis wrote: post #296
mbeex wrote: post #293

Earlier quoted context omitted.

> Ideas are so cheap and easy. I doubt this.

It is true however. Consider it was easy to bring an idea in this world, and the hard part was the initial first thought; writing a paper/article painstakingly rigorously would be unnecessary. Writing a book would be a breeze and no author would ever go through more than a single draft. The idea was born beforehand, was complete correct and perfect, so putting everything down with words is just a matter of transcribi…

The best way I've heard this described is... Imagine the best painting you can come up with or have ever seen. Now go paint it.

Re: We don't need data scientists, we need data engineers

#307

Post by analog31 »

baron_harkonnen wrote: post #303
lordgrenville wrote: post #258

Earlier quoted context omitted.

Are they CS/EE PhDs?

Think back to your own CS professors, were any of them particular good software engineers? I've found that Physics PhDs tend to have the highest probability of being good coders since a certain subset of them get bit by the software bug when they need to write non-trivial amounts of code to solve research problems.

I got my physics PhD in the early 90s. Physics has had a tradition of interest in programming that goes back decades. We've always had "big data," meaning big relative to the tools available at any given moment. We ran out of problems that could be solved by pencil and paper in the 1930s.

Every physics student at my college had to take FORTRAN, plus programming was assumed in many of the other courses, and we also took an electronics course that included digital techniques. And maybe the main thing was simply that programming was interesting and fun.

We've also had a tradition of learning to do everything ourselves, for better or worse. I had no access to a professional programmer.

Re: We don't need data scientists, we need data engineers

#308

Post by kyawzazaw »

lordnacho wrote: post #112

My experience is in quant hedge funds, where sometimes you get some guys who develop the strategy and some guys who put it into production. Yes, I do admit there can be some specialization in terms of time spent on science vs engineering. But you really need people who understand both. Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a…

How does compensation tends to differ? And the education levels?

Re: We don't need data scientists, we need data engineers

#309

Post by xkcd-sucks »

mbeex wrote: post #293
the-pigeon wrote: post #270

Earlier quoted context omitted.

Ideas are so cheap and easy. Implementation is a long hard road. And where you learn your idea was vague enough that it had almost no value. And only through painstaking iteration can you turn it into something with value.

> Ideas are so cheap and easy. I doubt this.

"Pure" ideas are cheap and easy; good ideas require a very thorough knowledge of implementation which is usually achieved through experience

Re: We don't need data scientists, we need data engineers

#310

Post by test-account »

proverbialbunny wrote: post #170
noodlenotes wrote: post #154

Earlier quoted context omitted.

Also, a lot of data scientists find the science fun and the engineering boring. But they have overlapping skill sets - if you aren't good at one, you're probably not good at the other either. Somebody who shows up to a team with the goal of only modeling and pushing all the dirty engineering work to their teammates is basically a worst case scenario because 1) They probably aren't going to produce good models since t…

Back in the day (3 years ago and earlier) at every company I was at we used the term 'productionization' to describe someone making a model aka a proof of concept, and then someone else, a machine learning engineer or some kind of engineer rewriting it to work on a server. This process is horrible, and not just because it doubles the work, but because it introduces bugs. When the version up in the cloud does not work…

Exactely why I left Dolby. Didn't want to be a part of that process.
Post reply on HN