Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don't need data scientists, we need data engineers

mihaileric.com

Post reply on HN
Print view View original

51–60 of 367 posts

Re: We don't need data scientists, we need data engineers

#51

Post by glitchc »

ziml77 wrote: post #32

Having to deal with data scientists, I absolutely agree. The thing that I've seen that lands in the "lab" vs production distinction is that these people expect their data to be pristine. They flip out when the world isn't as perfect as their models want. Leads to me as just a normal software developer having to do the data analysis and figure out how to clean it up. I also end up having to be the one to talk to data…

This implies a lack of rigorous training. In the physical sciences, one wouldn't become an applied scientist without conducting an experiment to test a phenomenon, and the teeth gnashing that goes with making that experiment work.

Those who have been fed pristine data without having to undergo the trials and tribulations of actually having to collect the data have missed a crucial part of scientific training. Like you, I find this lack of rigour is rather common among data scientists. Not all, but quite a few.

Re: We don't need data scientists, we need data engineers

#52

Post by nerdponx »

mlthoughts2018 wrote: post #27
coding123 wrote: post #21

A couple of us inherited a machine learning project a while back. The code was horrible. Riddled with copy pasta (nearly half of the entire thing was copy paste and no code reuse). We basically refactored everything, standardized input and output file names. We put up a small Flask service to allow outside services hit it easily and wrapped it up in a Docker container so it was ultimately easy to deploy. Yes it was a…

I am curious what your take is on things like this article: https://managingml.substack.com/p/the-myth-that-machine-lear... It has been my experience too. Basically, ML / DS engineers are thrown under the bus for being poor general software engineers, but in practice it’s totally the opposite.

The problem is that ML engineers are not the people who wrote GP's garbage code. Data scientists wrote it, and I know at least a few of my very intelligent, high-functioning data scientist colleagues who are alarmingly, astoundingly bad programmers.

Re: We don't need data scientists, we need data engineers

#53

Post by bonniemuffin »

hn_throwaway_99 wrote: post #5
avs733 wrote: post #2

I teach engineers for a living. I struggle to see how this is not just a straw man argument based on colloquial usage of terms. It is just inferences drawn based on job ads that are rarely written by people doing the job and instead are effectively human-as-seo-optimized so the best candidates can find the job they hopefully fit for and not be too confused to apply for it.

It's not a straw man, I've seen it clear as day in several companies. When it comes to data science, it's "garbage in, garbage out". I've seen companies do lots of "data science" with a bunch of data scientists skilled in python and jupyter notebooks, only to discover a ton of work was useless because the incoming event data was tagged incorrectly due to a bug. The actual process of collecting, aggregating, cleaning…

I suspect this may actually be an issue of school vs real world rather than scientist vs engineer.

Data in the classroom setting is pristine and beautiful; data in the real world is messy and buggy. You have to get burned by buggy data a few times (or maybe a bunch of times) in the real world to learn to look for bad data smells -- I don't think schools effectively teach this kind of intuition, regardless of whether the students are training as data engineers or data scientists.

If data scientists are spending more time in school getting advanced degrees, they're not getting as much exposure to buggy data, whereas data engineers with a BS and a few years of industry experience would already have built up this skill.

Re: We don't need data scientists, we need data engineers

#54

Post by slt2021 »

esyir wrote: post #41
slt2021 wrote: post #35

Full Stack Data Scientist (data janitor + data engineer + ML engineer + ML Ops + Business Analyst) is the future

Ugh, all this gets you is being mediocre at all of this.

yes, if you need to roll-up everything on your own from scratch.

NO, if you use right amount of automation and software (usable data science workbench with MLOps built in, usable and scalable ETL/ELT framework, usable AutoML, etc, etc.)

Re: We don't need data scientists, we need data engineers

#55

Post by nerdponx »

basseq wrote: post #15

The problem that I've seen is often "data scientists" are expected to be the equivalent of full-stack engineers (or maybe more accurately: one-man CTO shops)—to understand data architecture, understand business architecture, ensure data quality, build data into product, build dashboards, derive insights, posit hypotheses, set strategy, and drive business value. Thus many "data scientists" are juiced-up report-builder…

This is uncharitable.

In my experience, this is true:

> "data scientists" are expected to be the equivalent of full-stack engineers (or maybe more accurately: one-man CTO shops)—to understand data architecture, understand business architecture, ensure data quality, build data into product, build dashboards, derive insights, posit hypotheses, set strategy, and drive business value.

But this is not:

> Thus many "data scientists" are juiced-up report-builders who can't analyze their way out of a paper bag.

Rather, the data scientists are trained in only two of the requirements you mentioned: derive insights, posit hypotheses. The rest is all self-study and on-the-job experience. This means that we are putting unrealistic expectations on data scientist and/or their training is insufficient, not that data scientists are somehow morons.

Re: We don't need data scientists, we need data engineers

#56

Post by tristor »

Agreed. In my decently long career the types of data problems I've seen be most impactful on the business are not head-in-the-clouds ML issues, but more mundane yet more far-reaching:

1. Appropriately identifying what data needs to be captured from a product to correctly operationalize it.

2. Understanding and modeling data structures in internal applications to identify and tune backend data storage mechanisms (including DBMS). Inclusive in this is helping the application development team pick the correct structure and implement it correctly.

3. Validating implementation of instrumentation within the application so that data cleaning isn't necessary and that telemetry can be appropriately reported on. Building said reports.

4. Doing ETL and taking care of out of band data management to link disparate systems within the business to help build holistic views of the business overall.

5. Be a safeguard against the over-collection of data, because data engineers understand that data isn't an asset, it's a liability that increase costs and risks as a business or product scales, and when there's not a specific need that can be articulated clearly for that data, collecting it is a user/customer-hostile action.

My experience has been that data is a crucial element to understand the health and state of the business with both breadth and depth at a given point in time and identify trends. However, it's mostly used by folks in management as a crutch to try to de-risk decision making, or worse as a political tool to give a faux support to a decision that's already been made but not yet publicized. Decisions carry inherent risk, including the decision to do nothing, you cannot eliminate this, it's one of the components of decision trade-offs. This sort of broken use of data by management is supported by "Data Scientists" that see the field as a cash-cow they can milk while they work on pie-in-the-sky ML strategies which are often unnecessary, even when they actually work.

Done correctly a strong data culture in a company can increase decision velocity, empower engineers, and reduce overhead on management to understand the business. Done improperly, data culture in a business can easily destroy decision velocity, empower dysfunctional politics, and increase engineering overhead to understand systems. Getting it right is the main test for businesses in the new era.

Re: We don't need data scientists, we need data engineers

#57

Post by vbtemp »

bitcharmer wrote: post #49
AndrewKemendo wrote: post #3

Preach! The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and engineers to do everything that empowers Data Science and ML. We need more Data Engineers involved at time zero in projects to help: 1. Plan out what data should be produced/captured by the product 2. Instrument systems to actually generate data consistently and effec…

This same sentiment (which I personally agree with) applies to software engineering. As in: engineers deliver more practical value than comp scientists. Now you can down-vote me to oblivion.

You and me both, friend. Except I normally get down-voted to oblivion for saying the opposite.

Re: We don't need data scientists, we need data engineers

#59

Post by ct0 »

superbcarrot wrote: post #34
ziml77 wrote: post #32

Having to deal with data scientists, I absolutely agree. The thing that I've seen that lands in the "lab" vs production distinction is that these people expect their data to be pristine. They flip out when the world isn't as perfect as their models want. Leads to me as just a normal software developer having to do the data analysis and figure out how to clean it up. I also end up having to be the one to talk to data…

I think that this is more of a problem with the specific people that you have worked with and it isn't inherent to the role of a data scientist.

Doesn't sound like a modern Data Scientist, sounds more like a statistician with 30+ years of experience.

Re: We don't need data scientists, we need data engineers

#60

Post by adamisom »

vbtemp wrote: post #57
bitcharmer wrote: post #49

Earlier quoted context omitted.

This same sentiment (which I personally agree with) applies to software engineering. As in: engineers deliver more practical value than comp scientists. Now you can down-vote me to oblivion.

You and me both, friend. Except I normally get down-voted to oblivion for saying the opposite.

I guess depends on what valuable means. I imagine most comp scientists are less replaceable than most software engineers, so point for compsci.
Post reply on HN