Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don't need data scientists, we need data engineers

mihaileric.com

Post reply on HN
Print view View original

191–200 of 367 posts

Re: We don't need data scientists, we need data engineers

#191

Post by nerdponx »

baron_harkonnen wrote: post #88
ziml77 wrote: post #32

Having to deal with data scientists, I absolutely agree. The thing that I've seen that lands in the "lab" vs production distinction is that these people expect their data to be pristine. They flip out when the world isn't as perfect as their models want. Leads to me as just a normal software developer having to do the data analysis and figure out how to clean it up. I also end up having to be the one to talk to data…

The data science field has been flooded with PhDs with nowhere else to go that have no background in engineering, and sadly often have a very poor understanding of both machine learning and statistics. Companies were in a rush hire "data scientists" and boot camps like Insight were more than happy to pump out very impressive PhDs with just enough understanding to build a Keras model. I've worked in industry awhile do…

Not to mention the dark pattern of giving data scientist candidates an unsolved industry problem as their interview take-home task, and then telling them to only spend 4 hours on it. Data science hiring often feels like a competition where the winner is the one who has the most free time and willingness to do other people's work without compensation.

It's kind of a fucked up field right now.

Re: We don't need data scientists, we need data engineers

#192

Post by RobRivera »

proverbialbunny wrote: post #174
NikolaNovak wrote: post #168

Earlier quoted context omitted.

Here's the tricky thing: I love your post; I agree with your post; but it takes a 90 degree turn at the end: "My number one advice to entry level data scientists is to not be this guy. " Everything most people are saying here indicates it's GREAT to be that guy. You're paid, you're respected, you get the fun parts, you love your job and it's pretty safe. It just happens to suck for everybody else including team and b…

It sucks being that guy because everyone else ends up hating you. Depending on the work environment it's not a stretch to see software engineers complaining to management, sometimes going as far to create rumors to get the jr data scientist fired. So, no the grass is not greener. It's best to not be that person. This is why I go out of my way to prevent that scenario when I lead a team.

easy to ignore hate when youre pulling a 300k bonus at comp season and can jet to st. barts to go deep sea fishing and drink claws.

Re: We don't need data scientists, we need data engineers

#193

Post by singhrac »

proverbialbunny wrote: post #163
twic wrote: post #147

Earlier quoted context omitted.

I'm not sure about crappy quants. Some people of the "quantitatively inclined trader who has learned Python" variety are never going to be good at the engineering side - it takes years to learn to be a good software engineer, and that's not a good use of time, for them, or for their employer. But they can still do useful work. The trick is to figure out how to work effectively with those people. Build infrastructure…

If they're using a dynamically typed language to do monetary calculations, it's not going to be ideal. Researchers do not need to have deep programming experience, but they have to be comfortable enough to use an environment that can lend itself itself to the problem at hand. On the quant side, unlike on the data science side, the barrier of entry on the programming side is a bit higher. To solve this problem many fi…

> If they're using a dynamically typed language to do monetary calculations, it's not going to be ideal.

I think this is an inaccurate take. No one in finance is doing accounting or model estimation using Python's floats; they are using numpy's float32 (or float64) type instead. I think a more accurate version of what you're saying is that static type checking is useful when modeling complicated contracts; this might be true, but I think it's not that important, as those things aren't that liquid anyway.

Jane Street's decision to use OCaml is almost as much about hiring and history as it is about language features.

Re: We don't need data scientists, we need data engineers

#194

Post by anonymousDan »

chadash wrote: post #184
proverbialbunny wrote: post #163

Earlier quoted context omitted.

If they're using a dynamically typed language to do monetary calculations, it's not going to be ideal. Researchers do not need to have deep programming experience, but they have to be comfortable enough to use an environment that can lend itself itself to the problem at hand. On the quant side, unlike on the data science side, the barrier of entry on the programming side is a bit higher. To solve this problem many fi…

> "To solve this problem many firms have their own internal programming language." Any examples other than Jane Street?

Goldman Sachs (Slang)

Re: We don't need data scientists, we need data engineers

#195

Post by proverbialbunny »

RobRivera wrote: post #192
proverbialbunny wrote: post #174

Earlier quoted context omitted.

It sucks being that guy because everyone else ends up hating you. Depending on the work environment it's not a stretch to see software engineers complaining to management, sometimes going as far to create rumors to get the jr data scientist fired. So, no the grass is not greener. It's best to not be that person. This is why I go out of my way to prevent that scenario when I lead a team.

easy to ignore hate when youre pulling a 300k bonus at comp season and can jet to st. barts to go deep sea fishing and drink claws.

Data scientists do not pull that kind of bonus. Today many of them get paid less than the data engineers do.

Re: We don't need data scientists, we need data engineers

#196

Post by twic »

singhrac wrote: post #193
proverbialbunny wrote: post #163

Earlier quoted context omitted.

If they're using a dynamically typed language to do monetary calculations, it's not going to be ideal. Researchers do not need to have deep programming experience, but they have to be comfortable enough to use an environment that can lend itself itself to the problem at hand. On the quant side, unlike on the data science side, the barrier of entry on the programming side is a bit higher. To solve this problem many fi…

> If they're using a dynamically typed language to do monetary calculations, it's not going to be ideal. I think this is an inaccurate take. No one in finance is doing accounting or model estimation using Python's floats; they are using numpy's float32 (or float64) type instead. I think a more accurate version of what you're saying is that static type checking is useful when modeling complicated contracts; this might…

> No one in finance is doing accounting or model estimation using Python's floats

We are. When your input data only has five significant figures, and probably less than that of real information, numerical accuracy is the least of your worries.

Re: We don't need data scientists, we need data engineers

#197

Post by Fellshard »

jimbokun wrote: post #169
bitcharmer wrote: post #49

Earlier quoted context omitted.

This same sentiment (which I personally agree with) applies to software engineering. As in: engineers deliver more practical value than comp scientists. Now you can down-vote me to oblivion.

I think generally, Computer Science is a degree and Software Engineer is a job description. So many people get Computer Science degrees, then have a career as a Software Engineer. Yes, there are Software Engineering degrees. But I think a minority of Software Engineers have a Software Engineering degree. What this means in practice, is that Computer Science majors need to learn the engineering skills on the job or on…

Anecdotal: University of Washington considers (considered?) them two separate degrees, holding CS as more theory and research-driven, and CE as more practice and career-driven.

Re: We don't need data scientists, we need data engineers

#198

Post by Karrot_Kream »

hntrader wrote: post #141
lordnacho wrote: post #112

My experience is in quant hedge funds, where sometimes you get some guys who develop the strategy and some guys who put it into production. Yes, I do admit there can be some specialization in terms of time spent on science vs engineering. But you really need people who understand both. Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a…

To add, quants that can't do the data engineering work are always crappy quants. I haven't seen a counter-example to that. Profitable models aren't going to be delivered on a silver platter. They need to be able to process pretty low level data effectively and build ad-hoc custom tools and data pipelines around that to test out their ideas. Otherwise they're constrained to the tools others have built and that massive…

> 1/3 statistician, 1/3 developer and 1/3 trader

How is being a trader different from being a statistician? Curious as I've never worked in finance before.

Re: We don't need data scientists, we need data engineers

#199

Post by hntrader »

Karrot_Kream wrote: post #198
hntrader wrote: post #141

Earlier quoted context omitted.

To add, quants that can't do the data engineering work are always crappy quants. I haven't seen a counter-example to that. Profitable models aren't going to be delivered on a silver platter. They need to be able to process pretty low level data effectively and build ad-hoc custom tools and data pipelines around that to test out their ideas. Otherwise they're constrained to the tools others have built and that massive…

> 1/3 statistician, 1/3 developer and 1/3 trader How is being a trader different from being a statistician? Curious as I've never worked in finance before.

By trader, I mean domain knowledge about the markets. Statistics is the toolbox that this domain expert uses to test their hypotheses and turn them into a profitable model. But if the person isn't a domain expert and only knows statistics, their ideas about what to test won't be good.

Re: We don't need data scientists, we need data engineers

#200

Post by Karrot_Kream »

pricci wrote: post #144
C4stor wrote: post #18

I can't recommend the Data Engineer career enough for junior developers. It's how I started and what I pursued for 6 years (and I would love doing it again), and I feel like it gave me such an incredible foundation for future roles : - Actually big data (so, not something you could grep...) will trigger your code in every possible way. You quickly learn that with trillions of input, the probabily to reach a bug is ei…

And where would you recommend someone to start a data engineering path. Any book, learning source?

A lot of these are just "garden variety" (distributed) systems problems. Dealing with systems with differing latency distributions, recovering from failure, acceptable tradeoffs between speed and accuracy, etc
Post reply on HN