We don't need data scientists, we need data engineers
mihaileric.com
We don't need data scientists, we need data engineers
1–10 of 367 posts
Re: We don't need data scientists, we need data engineers
#2I teach engineers for a living. I struggle to see how this is not just a straw man argument based on colloquial usage of terms. It is just inferences drawn based on job ads that are rarely written by people doing the job and instead are effectively human-as-seo-optimized so the best candidates can find the job they hopefully fit for and not be too confused to apply for it.
Re: We don't need data scientists, we need data engineers
#3Preach!
The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and engineers to do everything that empowers Data Science and ML.
We need more Data Engineers involved at time zero in projects to help:
1. Plan out what data should be produced/captured by the product
2. Instrument systems to actually generate data consistently and effectively
3. Build ETL pipelines and data management systems
4. Manage enterprise data sharing and resiliency
etc...
What ends up happening is you have a bunch of Data Scientists just handed a pg_dump or flat file from some ops team. That is typically missing data or poorly formatted and they spend 90% of their time cleaning it up then running some basic regression with numpy or whatever.
Need better understanding of the data lifecycle by organizations and investment in instrumentation and data management.
Re: We don't need data scientists, we need data engineers
#4In other words: we need plumbers.
Re: We don't need data scientists, we need data engineers
#5I teach engineers for a living. I struggle to see how this is not just a straw man argument based on colloquial usage of terms. It is just inferences drawn based on job ads that are rarely written by people doing the job and instead are effectively human-as-seo-optimized so the best candidates can find the job they hopefully fit for and not be too confused to apply for it.
It's not a straw man, I've seen it clear as day in several companies. When it comes to data science, it's "garbage in, garbage out". I've seen companies do lots of "data science" with a bunch of data scientists skilled in python and jupyter notebooks, only to discover a ton of work was useless because the incoming event data was tagged incorrectly due to a bug.
The actual process of collecting, aggregating, cleaning and verifying data is a hugely important skill, and not one I've really seen typical data scientists possess.
Re: We don't need data scientists, we need data engineers
#6Preach! The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and engineers to do everything that empowers Data Science and ML. We need more Data Engineers involved at time zero in projects to help: 1. Plan out what data should be produced/captured by the product 2. Instrument systems to actually generate data consistently and effec…
Who will do the proper cleaning then?
Re: We don't need data scientists, we need data engineers
#7I'm a data engineer for most of my day right now, and a lot of it is done with ruby/python/shell scripts into postgres DBs.
What learning path should I go down? I'm a solo actor at work with a lot of agency to decide my workflows.
I see myself building small to medium size data collections over the next year or two at my job.
Can someone point me to some learning?
I have a CS degree etc. and my title in software engineer etc. etc.
End users of my data usually like their data as a CSV that is then read using R or Python. However there is also a use case where I will build an app to view my data in a simple way.
All of this is completely doable with my current knowledge/workflow but I can't help feel like I do a data engineering job with very different tools than i see "data engineers" speaking about online.
Re: We don't need data scientists, we need data engineers
#8In other words: we need plumbers.
Or, if the data is food, we have chefs making incredible plates, but we need wait staff to get it to the people who want to eat it.
I would love to be on that wait staff; as an infradev I feel like the process is very close to me but I am struggling to break into it.
Re: We don't need data scientists, we need data engineers
#9I teach engineers for a living. I struggle to see how this is not just a straw man argument based on colloquial usage of terms. It is just inferences drawn based on job ads that are rarely written by people doing the job and instead are effectively human-as-seo-optimized so the best candidates can find the job they hopefully fit for and not be too confused to apply for it.
So I think that the delineation between the scientist working with the content, and the Engineers who actually provide the mechanics for it is very fair.
If there is a question mark here - it's really how much value are we deriving from all of these data people?
Where is all the ML that's changing our lives? Search, Alexa and TikTok, I can see it.
In the future obviously vision systems for autonomous cars etc..
But I'm really wary about the heavily decreasing marginal returns after that.
It will surely change the world, but I think in specific areas. Most of the entire field seems like an optimization on something rather than anything new.
Washing Machines feed up immense amount of labour and toil. Alexa telling me the weather is not.
Re: We don't need data scientists, we need data engineers
#10Preach! The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and engineers to do everything that empowers Data Science and ML. We need more Data Engineers involved at time zero in projects to help: 1. Plan out what data should be produced/captured by the product 2. Instrument systems to actually generate data consistently and effec…
Who will do the proper cleaning then?
The aspiration that GP was getting to was that less cleaning is required as a result of better data engineering, I believe.