Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don't need data scientists, we need data engineers

mihaileric.com

Post reply on HN
Print view View original

21–30 of 367 posts

Re: We don't need data scientists, we need data engineers

#21

Post by coding123 »

A couple of us inherited a machine learning project a while back. The code was horrible. Riddled with copy pasta (nearly half of the entire thing was copy paste and no code reuse). We basically refactored everything, standardized input and output file names. We put up a small Flask service to allow outside services hit it easily and wrapped it up in a Docker container so it was ultimately easy to deploy. Yes it was all the plumbing. However we also looked at the code, and the ML strategies, and while there was "some" level of competence, it was nothing more than word2vec add and divide. Totally horrible for actually finding key phrases that matter to the subject we're matching. So we started tackling that too with LSTM but our time got cut short and shifted off to another area. So not only was the "scientist" they hired completely crappy at the engineering, they weren't really helpful in the ML either.

This is obviously of lesser value to the topic at hand, and more about making sure you hire good people I think.

Re: We don't need data scientists, we need data engineers

#22

Post by zeku »

langitbiru wrote: post #11
zeku wrote: post #7

I'm a data engineer for most of my day right now, and a lot of it is done with ruby/python/shell scripts into postgres DBs. What learning path should I go down? I'm a solo actor at work with a lot of agency to decide my workflows. I see myself building small to medium size data collections over the next year or two at my job. Can someone point me to some learning? I have a CS degree etc. and my title in software engi…

A couple of days ago, there was a thread about "How to become a data engineer in 2021": https://news.ycombinator.com/item?id=25728198

thanks!

Re: We don't need data scientists, we need data engineers

#23

Post by zeku »

ageofwant wrote: post #14
zeku wrote: post #7

I'm a data engineer for most of my day right now, and a lot of it is done with ruby/python/shell scripts into postgres DBs. What learning path should I go down? I'm a solo actor at work with a lot of agency to decide my workflows. I see myself building small to medium size data collections over the next year or two at my job. Can someone point me to some learning? I have a CS degree etc. and my title in software engi…

Adopt a good Workflow tool, like apache airflow. Easiest is to rent a service from AWS https://aws.amazon.com/managed-workflows-for-apache-airflow/

thanks!

Re: We don't need data scientists, we need data engineers

#24

Post by avs733 »

finnthehuman wrote: post #20
hn_throwaway_99 wrote: post #5

Earlier quoted context omitted.

It's not a straw man, I've seen it clear as day in several companies. When it comes to data science, it's "garbage in, garbage out". I've seen companies do lots of "data science" with a bunch of data scientists skilled in python and jupyter notebooks, only to discover a ton of work was useless because the incoming event data was tagged incorrectly due to a bug. The actual process of collecting, aggregating, cleaning…

>The actual process of collecting, aggregating, cleaning and verifying data is a hugely important skill, and not one I've really seen typical data scientists possess. Then they are not scientists. They have a label "scientist" but lack of rigor of actual science. I don't see why changing the label to "engineer" would suddenly make them have rigor.

Right?!

This is sort of the meta failure of the argument. They are arguing that people's data skillsets are wrong. To make that argument they are analyzing based on the wrong variable in a data set.

Re: We don't need data scientists, we need data engineers

#25

Post by AndrewKemendo »

alex_anglin wrote: post #10
mgh2 wrote: post #6

Earlier quoted context omitted.

Who will do the proper cleaning then?

The aspiration that GP was getting to was that less cleaning is required as a result of better data engineering, I believe.

Correct. If you build your instrumentation correctly, then you don't really need to do any "cleaning."

Doesn't mean you might not need to do transformation for different uses but ideally wouldn't need to, for example change data types like turning a bool into an int.

Re: We don't need data scientists, we need data engineers

#26

Post by jimsparkman »

Keeping in mind DE can mean different things at different companies, I spend a lot of time working on infrastructural components to just get at data reliably. Working in a product company with disparate generators of data, I’m often building out network connectivity (VPC peering, VPNs, etc.), subnets, ACLs, firewalls and load balancers across our visualization tools, managing job flows, controlling AWS costs, building read replicas for production databases, yadda yadda. There might be a ton of hoops to jump through before I can even start to process data and it’s the type of work that wouldn’t make sense to hand off to my DS counterparts.

Re: We don't need data scientists, we need data engineers

#27

Post by mlthoughts2018 »

coding123 wrote: post #21

A couple of us inherited a machine learning project a while back. The code was horrible. Riddled with copy pasta (nearly half of the entire thing was copy paste and no code reuse). We basically refactored everything, standardized input and output file names. We put up a small Flask service to allow outside services hit it easily and wrapped it up in a Docker container so it was ultimately easy to deploy. Yes it was a…

I am curious what your take is on things like this article:

https://managingml.substack.com/p/the-myth-that-machine-lear...

It has been my experience too. Basically, ML / DS engineers are thrown under the bus for being poor general software engineers, but in practice it’s totally the opposite.

Re: We don't need data scientists, we need data engineers

#28

Post by antipaul »

AndrewKemendo wrote: post #3

Preach! The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and engineers to do everything that empowers Data Science and ML. We need more Data Engineers involved at time zero in projects to help: 1. Plan out what data should be produced/captured by the product 2. Instrument systems to actually generate data consistently and effec…

I have a dream - and it looks like this!

Re: We don't need data scientists, we need data engineers

#29

Post by antipaul »

avs733 wrote: post #2

I teach engineers for a living. I struggle to see how this is not just a straw man argument based on colloquial usage of terms. It is just inferences drawn based on job ads that are rarely written by people doing the job and instead are effectively human-as-seo-optimized so the best candidates can find the job they hopefully fit for and not be too confused to apply for it.

The article is so true, my latest mantra at work is “engineering is more important than data science”.

Everyone is buzzing about the latter, and few even realize what is the former.

Re: We don't need data scientists, we need data engineers

#30

Post by alexfromapex »

AndrewKemendo wrote: post #3

Preach! The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and engineers to do everything that empowers Data Science and ML. We need more Data Engineers involved at time zero in projects to help: 1. Plan out what data should be produced/captured by the product 2. Instrument systems to actually generate data consistently and effec…

I agree, if anything the data engineers (folks with engineering backgrounds) should be doing the applied work while a department of data scientists works on the theoretical or novel data analysis methods.

Right now our product has accumulated a lot of technical debt on the data validation side because data scientists designed the test code in a way that dramatically slows the development process.

Post reply on HN