[deleted]
We don't need data scientists, we need data engineers
101–110 of 367 posts
Re: We don't need data scientists, we need data engineers
#102I'm a scientist first, data person second. I'll use the label data scientist if that's what the employer wants. I'll use data analyst, scientist, researcher, or any other term, so long as they pay me.
What matters is: are we contributing to the betterment of society with data-driven decision making?
Re: We don't need data scientists, we need data engineers
#103Just my personal experience, but the people at my company titled "Data Engineers" basically can only be trusted to (very slowly) move data around, while the people titled "Data Scientists" have to do all the cleaning work to make the data suitable for analysis and modeling, in addition to doing that analysis and modeling.
Is the point here that data scientists are doing too much of the work that should be handled by data engineers? If so I agree, but there are some org barriers. For one thing, our data engineers are not accountable to any particular project. All we can do is send them tickets to move data around, and it already takes them weeks to move a table from one database to another. I can't imagine the headaches if we gave them anything nontrivial to do.
Re: We don't need data scientists, we need data engineers
#104Why is data scientist a profession in IT but not for example computer scientist? Many IT professionals studied computer science but they don't call themselves scientists in their line of work.
"Computer science" is a well-known misnomer as it is not about computers nor is it science. It's generally reserved for the academic study of computing. Data science is much closer to science.
Re: We don't need data scientists, we need data engineers
#105My first job as a "Data Scientist" (it wasn't called that, but the work was the same) was for a small gaming shop, around 2011. It involved applying econometric analysis and doing simple statistical testing on the player data sets. I realized quickly that knowing how to do statistical testing was only a very small portion of what it took to create value in such a role. At the time, I didn't even know (but learned) SQL. Everything I wanted to do involved teaming up with a developer, which wasn't efficient in a small operation. So I learned to program. I continue to enjoy skilling-up, most recently learning cloud-tech to enable me to deploy data tools I develop.
The most valuable people in the data chain will be those that can take idea to near-production. Running ML libraries over clean datasets is overrated. The fact is, 80% of the value of "Data Science" comes from KPIs and basic stuff.
Re: We don't need data scientists, we need data engineers
#106workera.ai has a very interesting approach to measuring, categorizing, and visualizing where you are on the skill spectrum and what employers are looking for on that spectrum.
Re: We don't need data scientists, we need data engineers
#107Genuine question: why is there so much pure teeming hatred for data scientists in this comment thread? Almost every comment comes off as full of snark and vitriol against data scientists.
I'm guessing just venting personal frustrations due to their own experiences, plus maybe poor hiring and guidance of data scientists in their own teams? I have definitely seen DS people in my experience that fit some of the descriptions here, but I think it's a mistake to trivialize the DS position itself. A good DS is a valuable asset, but depending on your company/data, maybe not worth the cost. Plus there is no "single" DS candidate or role, a lot of these roles (data engineer, DS, analyst, swe) blend together at times, and it's about finding the right balance of skills.
Sometimes I think a company (not having the DS experience themselves) mistakenly over-hire DS roles in today's hype of "AI" when their data is mostly run-of-the-mill and only requires simple linear models that can be architected an understood by a stats/math-savvy engineer. Even then, a good DS is still useful (even linear models can be complex: e.g. what priors do you want to use? Do you want a multi-task solution? etc.), but maybe not worth the cost.
Re: We don't need data scientists, we need data engineers
#108Earlier quoted context omitted.
Ugh, all this gets you is being mediocre at all of this.
yes, if you need to roll-up everything on your own from scratch. NO, if you use right amount of automation and software (usable data science workbench with MLOps built in, usable and scalable ETL/ELT framework, usable AutoML, etc, etc.)
You still get someone mediocre at everything just they cover up the gaps for a bit longer. Eventually things they don't understand will interact in ways they don't understand and cause production issues. It's okay to be a generalist, one should however understand the blind spots a generalist has.
Re: We don't need data scientists, we need data engineers
#109Preach! The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and engineers to do everything that empowers Data Science and ML. We need more Data Engineers involved at time zero in projects to help: 1. Plan out what data should be produced/captured by the product 2. Instrument systems to actually generate data consistently and effec…
I agree, if anything the data engineers (folks with engineering backgrounds) should be doing the applied work while a department of data scientists works on the theoretical or novel data analysis methods. Right now our product has accumulated a lot of technical debt on the data validation side because data scientists designed the test code in a way that dramatically slows the development process.
This feels especially true when you have access to things like BigQuery ML.
It's very easy for an average engineer (like me) to start using ML using these tools, but a lot harder to explain how it works, or exactly which type of models to use.
In my mind a DS would be really useful to just point us in the right direction and check work. Like a super specialist QA...
Re: We don't need data scientists, we need data engineers
#110Genuine question: why is there so much pure teeming hatred for data scientists in this comment thread? Almost every comment comes off as full of snark and vitriol against data scientists.
Probably just people blowing off steam and the target de jour are data scientists. If people had to sit down with their company's data scientists to air their grievances face to face I doubt they would be so condescending.