Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don't need data scientists, we need data engineers

mihaileric.com

Post reply on HN
Print view View original

121–130 of 367 posts

Re: We don't need data scientists, we need data engineers

#121

Post by brd »

As an industry we're letting history repeat itself and making all the same mistakes.

There are different kinds of developers. At it's most base form, you have systems focused developers and algorithmic focused developers. Sure there is a grey area but I think those two buckets are pretty defensible.

In the data science world you have an exact parallel. Those who build the systems and those who optimize the thing the system supports.

In the ML world you have another parallel. Those who build the systems and those who optimize and pioneer the model architectures and parameters.

We never reached consensus on the titles for different kinds of developer/programmer/computer scientists. And we're failing now to reach consensus on sane titles for ML and DS.

Re: We don't need data scientists, we need data engineers

#122

Post by hardtke »

itsoktocry wrote: post #105

My first job as a "Data Scientist" (it wasn't called that, but the work was the same) was for a small gaming shop, around 2011. It involved applying econometric analysis and doing simple statistical testing on the player data sets. I realized quickly that knowing how to do statistical testing was only a very small portion of what it took to create value in such a role. At the time, I didn't even know (but learned) SQ…

> The most valuable people in the data chain will be those that can take idea to near-production.

Having hired many data scientists/ML engineers over the years, people that build robust automated intelligence directly into products are extremely rare. I've estimated a maximum of 10K people in the entire world. They also command the highest salaries, not coincidentally. Very few people have both the statistics and engineering backgrounds as well as temperament to be successful, particularly when the problem requires new data sources or new types of models. There are some real simple practical hurdles such as the need to implement robust tracking that allows data snapshotting at the time when a decision needs to be made without affecting product performance, as well as figuring out how to gather data on users/situations that are actually important for moving the needle (first time users, casual users, etc.) There is also a mismatch between the best frameworks for prototyping/research and implementation which (at least at the companies I've worked for) can be summarized as "Java is good for application development, not ML. Python is good for ML, not application development."

Re: We don't need data scientists, we need data engineers

#123

Post by Enginerrrd »

mgh2 wrote: post #6
AndrewKemendo wrote: post #3

Preach! The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and engineers to do everything that empowers Data Science and ML. We need more Data Engineers involved at time zero in projects to help: 1. Plan out what data should be produced/captured by the product 2. Instrument systems to actually generate data consistently and effec…

Who will do the proper cleaning then?

Not to worry, corporate will just outsource to firm which hires Data Janitors

Re: We don't need data scientists, we need data engineers

#124

Post by tfehring »

One annoying thing about being a generalist is that domain experts in any given area that you need familiarity with can't help but complain about how little you know about that domain, ignoring the fact that your job requires equally deep knowledge of several other domains simultaneously.

In the case of data scientists, I think the business folks that want them to understand the business domain better generally have the strongest argument, followed by the statisticians - good data scientists need to personally understand both of those things well, while the engineering and ops stuff that data scientists are also expected to do is easier to compartmentalize on other teams. So I agree that we should have more data engineers, but apparently for the opposite reason as most people in this thread.

Re: We don't need data scientists, we need data engineers

#125

Post by jaegerpicker »

spacemanmatt wrote: post #78

Data engineer, here. Or at least, that has been my title a couple of times. Some data is inherently trash but a huge part of the data quality problem is sources who are allowed to produce trash that everyone else has to clean up, when it would be way more efficient for them to quit producing trash. Not to pick on any one institution, but SOAP seems to be a read flag that the service will also deliver some screwy data…

Every time I see SOAP in a ticket/task I die a little inside. Without fail every SOAP related project I've dealt with in the last 10 years has been a shit show. I've actually and unfortunately gotten pretty good at dealing with it but I so hate it.

Re: We don't need data scientists, we need data engineers

#126

Post by avs733 »

antipaul wrote: post #29
avs733 wrote: post #2

I teach engineers for a living. I struggle to see how this is not just a straw man argument based on colloquial usage of terms. It is just inferences drawn based on job ads that are rarely written by people doing the job and instead are effectively human-as-seo-optimized so the best candidates can find the job they hopefully fit for and not be too confused to apply for it.

The article is so true, my latest mantra at work is “engineering is more important than data science”. Everyone is buzzing about the latter, and few even realize what is the former.

eh...I think this can be analogized to what we already see in code...

You need architecture, you need backends, you need a front end, you need product design...all with data.

Why are computer scientists computer scientists not engineers? Why is computer science about the code side? Why did computer engineering end up being more on the hardware end of the spectrum?

Words, especially newly coined terms are pointers to meaning. That meaning is socially mediated, it is not inherent.

You're saying this (adn I think the author is too) because there is a need for this group of people to look beyond titles to skillsets, and the existing titles carry linguistic baggage of the difference between science and engineering that has existed for decades.

Re: We don't need data scientists, we need data engineers

#127

Post by fock »

baron_harkonnen wrote: post #88
ziml77 wrote: post #32

Having to deal with data scientists, I absolutely agree. The thing that I've seen that lands in the "lab" vs production distinction is that these people expect their data to be pristine. They flip out when the world isn't as perfect as their models want. Leads to me as just a normal software developer having to do the data analysis and figure out how to clean it up. I also end up having to be the one to talk to data…

The data science field has been flooded with PhDs with nowhere else to go that have no background in engineering, and sadly often have a very poor understanding of both machine learning and statistics. Companies were in a rush hire "data scientists" and boot camps like Insight were more than happy to pump out very impressive PhDs with just enough understanding to build a Keras model. I've worked in industry awhile do…

I do an introductory Python lab course at my university. It's targeted at engineers who still create graphs from Excel and then normally level up to MATLAB, if things get complicated (think insets, ...). I guess about 30% of the people previously did at least some of the YT/Udemy "courses" on datascience. It's really horrifying for me (not being an engineer myself, but imo having a relatively engineering-like mindset) to see these people horrified at simple tasks like writing a variadic function. "What do I need this for?". Well, it's using the programming environment. And then let them code up a simple version of Levenberg-Marquardt. The level of "why do I need to do this" is astonishing again...

Re: We don't need data scientists, we need data engineers

#128

Post by danbrooks »

Ansil849 wrote: post #95

Genuine question: why is there so much pure teeming hatred for data scientists in this comment thread? Almost every comment comes off as full of snark and vitriol against data scientists.

I would guess that it's a reaction to job title hype.

There's a huge variety in DS responsibility and background between companies.

Re: We don't need data scientists, we need data engineers

#129

Post by inthewoods »

lordnacho wrote: post #112

My experience is in quant hedge funds, where sometimes you get some guys who develop the strategy and some guys who put it into production. Yes, I do admit there can be some specialization in terms of time spent on science vs engineering. But you really need people who understand both. Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a…

That's interesting - I just completed book on Jim Simon/Renaissance (The Man Solved The Market). One of their early advantages was having a person who was just focused on acquiring and cleaning data. I expect that advantage has largely gone away at this point due to wide availability of market data but I thought it was interesting in the context of this article.

Re: We don't need data scientists, we need data engineers

#130

Post by andrewnc »

Ansil849 wrote: post #95

Genuine question: why is there so much pure teeming hatred for data scientists in this comment thread? Almost every comment comes off as full of snark and vitriol against data scientists.

I came to this thread interested in the discussion, but I feel now like the homer simpson meme retreating into the hedge.

Maybe I'll come back in a few hours, but for now I'll stay away.

Post reply on HN