Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don't need data scientists, we need data engineers

mihaileric.com

Post reply on HN
Print view View original

131–140 of 367 posts

Re: We don't need data scientists, we need data engineers

#131

Post by isolli »

chadash wrote: post #120
lordnacho wrote: post #112

My experience is in quant hedge funds, where sometimes you get some guys who develop the strategy and some guys who put it into production. Yes, I do admit there can be some specialization in terms of time spent on science vs engineering. But you really need people who understand both. Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a…

I worked in investment banking (as an analyst, not an engineer), so very different part of finance, but this was my take as well. Companies might love to talk about how important engineers are, but at the end of the day, if you can't directly link someone to revenue, they get viewed as a cost center and take on second tier status in the organization. Then the same companies complain that they can't find enough (or re…

Yes, quite a few developers left our investment bank and went to work for our suppliers (of trading software), stating they'd rather work somewhere where they're seen as value creators rather than a cost center.

Re: We don't need data scientists, we need data engineers

#132

Post by lottin »

Ansil849 wrote: post #95

Genuine question: why is there so much pure teeming hatred for data scientists in this comment thread? Almost every comment comes off as full of snark and vitriol against data scientists.

Yes, the tide is turning now... who came up with the term "data scientist" anyway? It's a made up profession. If you need someone who understands statistics, get a statistician, or maybe a mathematician. If you need someone that designs and writes computer programs, get a computer programmer. But a "data scientist"? No, thanks.

Re: We don't need data scientists, we need data engineers

#133

Post by boterham »

baron_harkonnen wrote: post #88
ziml77 wrote: post #32

Having to deal with data scientists, I absolutely agree. The thing that I've seen that lands in the "lab" vs production distinction is that these people expect their data to be pristine. They flip out when the world isn't as perfect as their models want. Leads to me as just a normal software developer having to do the data analysis and figure out how to clean it up. I also end up having to be the one to talk to data…

The data science field has been flooded with PhDs with nowhere else to go that have no background in engineering, and sadly often have a very poor understanding of both machine learning and statistics. Companies were in a rush hire "data scientists" and boot camps like Insight were more than happy to pump out very impressive PhDs with just enough understanding to build a Keras model. I've worked in industry awhile do…

Do you have any suggestions for where to start looking for good places to apply that don't suffer from this?

Re: We don't need data scientists, we need data engineers

#134

Post by dumb1224 »

AndrewKemendo wrote: post #3

Preach! The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and engineers to do everything that empowers Data Science and ML. We need more Data Engineers involved at time zero in projects to help: 1. Plan out what data should be produced/captured by the product 2. Instrument systems to actually generate data consistently and effec…

In specific research areas such as biomedical science it is certainly tricky to get involved because of the data governance / confidentiality issue... so we have to do both roles to some extent

Re: We don't need data scientists, we need data engineers

#136

Post by marcinzm »

lordnacho wrote: post #112

My experience is in quant hedge funds, where sometimes you get some guys who develop the strategy and some guys who put it into production. Yes, I do admit there can be some specialization in terms of time spent on science vs engineering. But you really need people who understand both. Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a…

As I see it you need people who have shallow knowledge of many areas and deep knowledge of one area. That lets you have a group of experts but ones that know enough about other areas of expertise to work with those other experts.

Re: We don't need data scientists, we need data engineers

#137

Post by darksaints »

alexpetralia wrote: post #94
C4stor wrote: post #18

I can't recommend the Data Engineer career enough for junior developers. It's how I started and what I pursued for 6 years (and I would love doing it again), and I feel like it gave me such an incredible foundation for future roles : - Actually big data (so, not something you could grep...) will trigger your code in every possible way. You quickly learn that with trillions of input, the probabily to reach a bug is ei…

The other thing I'd emphasize here is dealing with "state". Data is effectively state. As application engineers build increasingly "stateless" code (e.g. pure functions, serverless deployments, etc), that state gets pushed elsewhere. Someone has to manage the queues, file versions/locations, logs, databases, configurations and so on. That is all "data". State management is a tricky problem even in a single-threaded a…

> As application engineers build increasingly "stateless" code (e.g. pure functions, serverless deployments, etc), that state gets pushed elsewhere.

Exactly. You can't magically make a stateful problem stateless, you can merely move that state around. Sometimes moving state around means moving it somewhere that is appropriate and capable of expertly handling that data. But if you make those choices wrong, it makes every aspect of your application more complex.

UI programming tried going down this idea of stateless programming, and for a while it was trendy to do so stuff like redux. The problem is that UIs are state machines. That's not an analogy, that is a literal statement. And it is true of all UI's...it's just as true of the transmission lever in your car as it is for your saas dashboard. You can't program stateless UIs...they would cease to be a UI. So at best, you can move that state around. And with most of these solutions (eg. redux), you end up pushing that state into a massive global singleton, where even simple things like the state of a single radio button needs to be fed through dozens of tightly coupled components in order to "statelessly" render. And even worse, you lose the extremely helpful distinction between UI state and domain state, mixing them both together into a gigantic shit stew.

Re: We don't need data scientists, we need data engineers

#138

Post by Mauricebranagh »

civilized wrote: post #114
mynameisash wrote: post #83

Earlier quoted context omitted.

> What ends up happening is you have a bunch of Data Scientists just handed a pg_dump or flat file from some ops team Not to disparage the amazing data scientists I've worked with, but I've been on teams where this is very much the approach to operationalizing models. It's basically, "Here's the sklearn model and some fragile featurization scripts we built. Can you take this to prod ASAP?" The problem I've seen is th…

This is 100% my experience as a data scientist. The engineering support we get is restricted to submitting a ticket for database access or moving data from one system to another. Wouldn't dream of involving an engineer in a data science project team, because I have no evidence that they have any experience or expertise in anything other than tickets to move data around.

That's first line support not engineering

Re: We don't need data scientists, we need data engineers

#139

Post by marcinzm »

nlbrown wrote: post #135

What are the typical entry level Data Science/Engineering positions like? Are they available to fresh college or bootcamp graduates?

From everything I'm hearing Data Science is right now flooded with entry level applicants to the point where they're applying for even pure Data Analyst jobs. Data Engineering a lot less so.
Post reply on HN