Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don't need data scientists, we need data engineers

mihaileric.com

Post reply on HN
Print view View original

271–280 of 367 posts

Re: We don't need data scientists, we need data engineers

#271

Post by Avalaxy »

coding123 wrote: post #21

A couple of us inherited a machine learning project a while back. The code was horrible. Riddled with copy pasta (nearly half of the entire thing was copy paste and no code reuse). We basically refactored everything, standardized input and output file names. We put up a small Flask service to allow outside services hit it easily and wrapped it up in a Docker container so it was ultimately easy to deploy. Yes it was a…

This is 100% my experience. I got hired as a ML engineer to bring a data scientists models into production. I did the same as you by tearing the whole thing apart and engineering it properly. I also look at the models, and oh boy... that data scientist had no idea what he was doing. Couldn't explain why he chose the model, didn't have any performance metrics (or even knew what metric to use to measure the performance) and just generally did not understand the basic concepts of his fields. I had to try really hard to drag answers out of him, but in the end I came out dissatisfied.

Re: We don't need data scientists, we need data engineers

#272

Post by analog31 »

jimbokun wrote: post #169
bitcharmer wrote: post #49

Earlier quoted context omitted.

This same sentiment (which I personally agree with) applies to software engineering. As in: engineers deliver more practical value than comp scientists. Now you can down-vote me to oblivion.

I think generally, Computer Science is a degree and Software Engineer is a job description. So many people get Computer Science degrees, then have a career as a Software Engineer. Yes, there are Software Engineering degrees. But I think a minority of Software Engineers have a Software Engineering degree. What this means in practice, is that Computer Science majors need to learn the engineering skills on the job or on…

Indeed, I think it's possible that the majority of people with degrees, have jobs that are not identical to their college major.

Certainly, most of us with unemployable majors. ;-)

Another phenomenon is employers applying the "engineer" title to any technical worker, such as designers, programmers, technicians, and so forth.

Part of it is that businesses evolve towards having caste systems. When this happens, then folks in the lower castes will try to rearrange their job titles to resemble the upper castes, or change jobs.

Re: We don't need data scientists, we need data engineers

#273

Post by sgt101 »

legerdemain wrote: post #267
1vuio0pswjnm7 wrote: post #263

Earlier quoted context omitted.

What must be communicated to management: It is easy to find other magicians. It is not easy to find another blacksmith. Without the right blacksmith, there can be no magic. Magicians will be magicians, always hustling (bullshitting), but they will never have the value and job security of the blacksmith. The blacksmith can see the fruits of her own labour, whilst the magician must lie to herself and others in order to…

> It is easy to find other magicians. It is not easy to find another blacksmith. Without the right blacksmith, there can be no magic. What? That runs counter to my experience at every company where I've either seen data engineers or worked as one. My observations of how management treats the two groups is this: Data engineers ("blacksmiths"): Blacksmiths are paid less. People think of them as less highly educated. Th…

Good Data Engineering is what enables good Data Science. With a good infrastructure you can go 100 times faster. Getting rid of Data Engineers means killing Data Science.

Re: We don't need data scientists, we need data engineers

#274

Post by Aeolun »

jnwatson wrote: post #206
wpietri wrote: post #201

Earlier quoted context omitted.

> Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a way that's detrimental to the rest of the team. My god, this. These people make me bonkers. Especially because I feel like I have a bit of this tendency myself, the desire just to think big thoughts and do no actual work. Happily, I long ago learned that ideas were approximately wort…

"I'm the idea guy" out of someone's mouth is the stark red-flag warning that their net contribution is 0.

That’s much better than the alternative. Where their ideas are crap, and their contribution is a direct detriment.

Re: We don't need data scientists, we need data engineers

#275

Post by hawktheslayer »

jnwatson wrote: post #206
wpietri wrote: post #201

Earlier quoted context omitted.

> Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a way that's detrimental to the rest of the team. My god, this. These people make me bonkers. Especially because I feel like I have a bit of this tendency myself, the desire just to think big thoughts and do no actual work. Happily, I long ago learned that ideas were approximately wort…

"I'm the idea guy" out of someone's mouth is the stark red-flag warning that their net contribution is 0.

It's similar to the expression I hear frequently at my company--"I don't have all the answers, but I have all the questions".

Re: We don't need data scientists, we need data engineers

#276

Post by legerdemain »

sgt101 wrote: post #273
legerdemain wrote: post #267

Earlier quoted context omitted.

> It is easy to find other magicians. It is not easy to find another blacksmith. Without the right blacksmith, there can be no magic. What? That runs counter to my experience at every company where I've either seen data engineers or worked as one. My observations of how management treats the two groups is this: Data engineers ("blacksmiths"): Blacksmiths are paid less. People think of them as less highly educated. Th…

Good Data Engineering is what enables good Data Science. With a good infrastructure you can go 100 times faster. Getting rid of Data Engineers means killing Data Science.

Sure. So in a few years, an all-in-one 80% solution like Palantir Foundry will come along and there will suddenly be a lot less demand for data engineers.

Anecdotally, the former head of QA for Palantir UK is now the head of data engineering for Palantir UK, and Palantir does have an out-of-the-box, end-to-end, it-just-works product that handles 80% of ML workflows. You're betting your career that they won't put it in a box and sell it at commodity software prices?

Re: We don't need data scientists, we need data engineers

#277

Post by nelsondev »

prionassembly wrote: post #181

I'm late to the comment party, but: this is classic "commoditize your complement". This guy would have you believe that Pytorch has Solved the entire, vast field of data analysis as inherited from Newton, de Moivre, Laplace, Bayes, Fisher, Neyman, Pearson, Wald, Savage, Jaynes, Breiman, Pearl. This is a lot like saying that photography has Solved art, and now we need people who can climb ladders and glue the posters…

It’s about diminishing returns.

E.g. getting a model from 0% accuracy to 70% accuracy might be a couple Pytorch library calls that any dummy who watched Andrew Ng’s course can do. But getting that same model from 70% to 75% accuracy might be deeply mathematical and require the latest and greatest mathmeticians, statisticians.

But in this hypothetical example, an engineer who stands up the 70% model and keeps it running 99.9% of the time, with high uptime; is more valuable to the bottom line of the business than the 75% accuracy model hacked together with scripts with 50% uptime.

Re: We don't need data scientists, we need data engineers

#278

Post by bumby »

legerdemain wrote: post #267
1vuio0pswjnm7 wrote: post #263

Earlier quoted context omitted.

What must be communicated to management: It is easy to find other magicians. It is not easy to find another blacksmith. Without the right blacksmith, there can be no magic. Magicians will be magicians, always hustling (bullshitting), but they will never have the value and job security of the blacksmith. The blacksmith can see the fruits of her own labour, whilst the magician must lie to herself and others in order to…

> It is easy to find other magicians. It is not easy to find another blacksmith. Without the right blacksmith, there can be no magic. What? That runs counter to my experience at every company where I've either seen data engineers or worked as one. My observations of how management treats the two groups is this: Data engineers ("blacksmiths"): Blacksmiths are paid less. People think of them as less highly educated. Th…

>reduced their QA or eliminated it wholesale

This is usually the case only for companies that work on low risk applications (I.e. not safety related or critical industries) or have been lulled into complacency (sometimes, ironically, “we haven’t had a major issue so obviously QA isn’t needed” when strong QA is precisely why they didn’t see issues)

Re: We don't need data scientists, we need data engineers

#279

Post by Rury »

1vuio0pswjnm7 wrote: post #263
lordnacho wrote: post #112

My experience is in quant hedge funds, where sometimes you get some guys who develop the strategy and some guys who put it into production. Yes, I do admit there can be some specialization in terms of time spent on science vs engineering. But you really need people who understand both. Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a…

What must be communicated to management: It is easy to find other magicians. It is not easy to find another blacksmith. Without the right blacksmith, there can be no magic. Magicians will be magicians, always hustling (bullshitting), but they will never have the value and job security of the blacksmith. The blacksmith can see the fruits of her own labour, whilst the magician must lie to herself and others in order to…

>Without the right blacksmith, there can be no magic. Magicians will be magicians, always hustling (bullshitting)

>management may consult the blacksmith in the selection of magicians

I mean when you put it like that, why hire a magician (bullshitter), if the magic relies on the blacksmith?

And, if management needs to consult someone (a blacksmith) on hiring (another blacksmith or for whatever reason a magician), then arguably management is made of magicians.

Don't get me wrong, I agree with the point your making. It's just the problem is with BS and BS is rampant, or like you say: a dime-a-dozen.

Re: We don't need data scientists, we need data engineers

#280

Post by lumost »

sgt101 wrote: post #273
legerdemain wrote: post #267

Earlier quoted context omitted.

> It is easy to find other magicians. It is not easy to find another blacksmith. Without the right blacksmith, there can be no magic. What? That runs counter to my experience at every company where I've either seen data engineers or worked as one. My observations of how management treats the two groups is this: Data engineers ("blacksmiths"): Blacksmiths are paid less. People think of them as less highly educated. Th…

Good Data Engineering is what enables good Data Science. With a good infrastructure you can go 100 times faster. Getting rid of Data Engineers means killing Data Science.

The way the evolution in software went, platforms became more capable and allowed individuals to automate more common tasks. QA/DevOps/SRE teams were consolidated and replaced with smaller platform teams which empowered internal engineers to quickly write scalable and well tested services. If data management, instrumentation, and ML tooling become sufficient then perhaps the data engineers will be replaced by a science platform team.

Caveat is that many scientists are expected to publish novel research to advance their career. "Infrastructure" and "data management" do not tend to produce the kinds of sexy projects which are attractive to publish.

Post reply on HN