Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don't need data scientists, we need data engineers

mihaileric.com

Post reply on HN
Print view View original

171–180 of 367 posts

Re: We don't need data scientists, we need data engineers

#171

Post by jimbokun »

oivey wrote: post #166
listenallyall wrote: post #115

Earlier quoted context omitted.

> novel data analysis methods Many "data scientists" (not all, but many) have little to no ability to do anything other than apply "recipes" of algorithms or classification methods or logistic regressions, etc. Asking them to develop a "novel" method would be fruitless. Asking them to clean and scrub the source data set is like telling an amateur pie-baker the store was out of pie crusts, you'll have to make your own…

Yep. The key is really software skills. If you’re unable to even filter the data yourself, you’re also probably unlikely to be able do implement novel analysis techniques, especially if the analysis algorithm has many complicated steps or is computationally expensive. In all fairness, it’s basically impossible for a new grad to have those skills. 4 years of a bachelors in any field isn’t enough to cover such a wide a…

The hope is that the 4 year degree gave you the ability to quickly pick up those skills on your own.

If your four year degree didn't give you the ability to learn and expand your knowledge on your own, its a colossal waste of your time and money.

Re: We don't need data scientists, we need data engineers

#172

Post by zzzeek »

lordnacho wrote: post #112

My experience is in quant hedge funds, where sometimes you get some guys who develop the strategy and some guys who put it into production. Yes, I do admit there can be some specialization in terms of time spent on science vs engineering. But you really need people who understand both. Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a…

maybe hedge funds would be able to find more people if they didn't only hire "guys".

Re: We don't need data scientists, we need data engineers

#173

Post by oivey »

proverbialbunny wrote: post #163
twic wrote: post #147

Earlier quoted context omitted.

I'm not sure about crappy quants. Some people of the "quantitatively inclined trader who has learned Python" variety are never going to be good at the engineering side - it takes years to learn to be a good software engineer, and that's not a good use of time, for them, or for their employer. But they can still do useful work. The trick is to figure out how to work effectively with those people. Build infrastructure…

If they're using a dynamically typed language to do monetary calculations, it's not going to be ideal. Researchers do not need to have deep programming experience, but they have to be comfortable enough to use an environment that can lend itself itself to the problem at hand. On the quant side, unlike on the data science side, the barrier of entry on the programming side is a bit higher. To solve this problem many fi…

This is dogmatism swung too far in the other direction, IMO. There are many, many successful production code bases written in dynamic languages. In my own experience as a vision scientist/engineer, there is tremendous value in being able to quickly whip up a concept in Python and then being able to easily visualize the results. Doing this exploration in C++ is wasteful. Implementation takes much longer, the correctness brought by static typing is dubious since the code isn’t in prod, and the canned CV/visualization libraries are fewer and frequently suck in at least some way. That said, there is also tremendous value in understanding how to map your Python prototype into production code, too. Someone strong in this field can do both.

Re: We don't need data scientists, we need data engineers

#174

Post by proverbialbunny »

NikolaNovak wrote: post #168
noodlenotes wrote: post #154

Earlier quoted context omitted.

Also, a lot of data scientists find the science fun and the engineering boring. But they have overlapping skill sets - if you aren't good at one, you're probably not good at the other either. Somebody who shows up to a team with the goal of only modeling and pushing all the dirty engineering work to their teammates is basically a worst case scenario because 1) They probably aren't going to produce good models since t…

Here's the tricky thing: I love your post; I agree with your post; but it takes a 90 degree turn at the end: "My number one advice to entry level data scientists is to not be this guy. " Everything most people are saying here indicates it's GREAT to be that guy. You're paid, you're respected, you get the fun parts, you love your job and it's pretty safe. It just happens to suck for everybody else including team and b…

It sucks being that guy because everyone else ends up hating you.

Depending on the work environment it's not a stretch to see software engineers complaining to management, sometimes going as far to create rumors to get the jr data scientist fired.

So, no the grass is not greener. It's best to not be that person. This is why I go out of my way to prevent that scenario when I lead a team.

Re: We don't need data scientists, we need data engineers

#175

Post by a_zaydak »

My view is from a small startup with little to no room for single purpose employees.

When I first started hiring and working with data scientist my view was this: If you can only manipulate data and run it through pipelines to generate models then you can't do enough to be highly valuable. You either need to have a strong enough background in CS to build the pipelines / tools or a strong enough mathematics background to be able to propose cutting edge new ideas. From my experience it is hard to find someone who has one of these skill just from a University "data science" program. At a small company (at least ones that I have worked with) being only proficient in R and basic Python isn't enough. That being said, I have met and handful of Data Scientist who were very smart and self motivated enough to pick up on the lacking skills when given the chance.

My question to HN is this; are there rolls at these larger companies for a Data Scientist who who primarily just crunches data in R and Python without the ability to actually build the pipelines / tools or conduct research?

Re: We don't need data scientists, we need data engineers

#176

Post by [deleted] »

twic wrote: post #147
hntrader wrote: post #141

Earlier quoted context omitted.

To add, quants that can't do the data engineering work are always crappy quants. I haven't seen a counter-example to that. Profitable models aren't going to be delivered on a silver platter. They need to be able to process pretty low level data effectively and build ad-hoc custom tools and data pipelines around that to test out their ideas. Otherwise they're constrained to the tools others have built and that massive…

I'm not sure about crappy quants. Some people of the "quantitatively inclined trader who has learned Python" variety are never going to be good at the engineering side - it takes years to learn to be a good software engineer, and that's not a good use of time, for them, or for their employer. But they can still do useful work. The trick is to figure out how to work effectively with those people. Build infrastructure…

[deleted]

Re: We don't need data scientists, we need data engineers

#177

Post by proverbialbunny »

oivey wrote: post #173
proverbialbunny wrote: post #163

Earlier quoted context omitted.

If they're using a dynamically typed language to do monetary calculations, it's not going to be ideal. Researchers do not need to have deep programming experience, but they have to be comfortable enough to use an environment that can lend itself itself to the problem at hand. On the quant side, unlike on the data science side, the barrier of entry on the programming side is a bit higher. To solve this problem many fi…

This is dogmatism swung too far in the other direction, IMO. There are many, many successful production code bases written in dynamic languages. In my own experience as a vision scientist/engineer, there is tremendous value in being able to quickly whip up a concept in Python and then being able to easily visualize the results. Doing this exploration in C++ is wasteful. Implementation takes much longer, the correctne…

This was addressed in the previous comment

>On the quant side, *unlike on the data science side*,

Vision scientist is on the data science side. You're not dealing with monetary values where floating point error compounds on itself to the point your models become garbage. Quant work is it's own unique field with its own unique prerequisites.

Re: We don't need data scientists, we need data engineers

#178

Post by noodlenotes »

NikolaNovak wrote: post #168
noodlenotes wrote: post #154

Earlier quoted context omitted.

Also, a lot of data scientists find the science fun and the engineering boring. But they have overlapping skill sets - if you aren't good at one, you're probably not good at the other either. Somebody who shows up to a team with the goal of only modeling and pushing all the dirty engineering work to their teammates is basically a worst case scenario because 1) They probably aren't going to produce good models since t…

Here's the tricky thing: I love your post; I agree with your post; but it takes a 90 degree turn at the end: "My number one advice to entry level data scientists is to not be this guy. " Everything most people are saying here indicates it's GREAT to be that guy. You're paid, you're respected, you get the fun parts, you love your job and it's pretty safe. It just happens to suck for everybody else including team and b…

Specifically, this is my advice to ENTRY level data scientists who are trying to find a job and compete against a flood of candidates hot off the bootcamps. I guess once you get your foot in the door, you can be that guy if you want. It seems to be a successful strategy at companies without technical leadership.

Re: We don't need data scientists, we need data engineers

#179

Post by carabiner »

proverbialbunny wrote: post #170
noodlenotes wrote: post #154

Earlier quoted context omitted.

Also, a lot of data scientists find the science fun and the engineering boring. But they have overlapping skill sets - if you aren't good at one, you're probably not good at the other either. Somebody who shows up to a team with the goal of only modeling and pushing all the dirty engineering work to their teammates is basically a worst case scenario because 1) They probably aren't going to produce good models since t…

Back in the day (3 years ago and earlier) at every company I was at we used the term 'productionization' to describe someone making a model aka a proof of concept, and then someone else, a machine learning engineer or some kind of engineer rewriting it to work on a server. This process is horrible, and not just because it doubles the work, but because it introduces bugs. When the version up in the cloud does not work…

How do you maintain notebooks in production? You use papermill? What about versioning?

Re: We don't need data scientists, we need data engineers

#180

Post by AndrewKemendo »

hinkley wrote: post #164
AndrewKemendo wrote: post #3

Preach! The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and engineers to do everything that empowers Data Science and ML. We need more Data Engineers involved at time zero in projects to help: 1. Plan out what data should be produced/captured by the product 2. Instrument systems to actually generate data consistently and effec…

This is a systemic problem. We ask non software engineers to write code, and then we expect them to apply a level of robustness and long term planning that even we have difficulty achieving. Not because we're being picky, but because we know the failure modes that are likely, and we know that people convince themselves that they aren't. We've been through this with installer writers, database admins, test automation,…

It's a two way street. SWE need to learn some data practices and data folks need to learn some SWE practices.
Post reply on HN