How to Become a Data Engineer in 2021
khashtamov.com
How to Become a Data Engineer in 2021
1–10 of 142 posts
Re: How to Become a Data Engineer in 2021
#22021? More like 2010. Hadoop is getting deprecated rapidly and more companies split their write and read workloads. Separated storage and compute is also popular. Scala is not used that much, I think it is not worth the time investment. More and more companies go for Kotlin instead of Java when these want to tap into the Java ecosystem.
Re: How to Become a Data Engineer in 2021
#32021? More like 2010. Hadoop is getting deprecated rapidly and more companies split their write and read workloads. Separated storage and compute is also popular. Scala is not used that much, I think it is not worth the time investment. More and more companies go for Kotlin instead of Java when these want to tap into the Java ecosystem.
Hadoop is still widely used in enterprises (especially in banks), if you have experience working with Hadoop ecosystems it is a big plus anyway.
Re: How to Become a Data Engineer in 2021
#4I think learning Scala is a bit of a waste of time, but I don’t know everyone’s stack. Maybe it’s a west coast bubble, but serverless seems to be the most popular choice for new ETL stacks even if the rest the cloud tech stack isn’t serverless. AWS tools like kinesis, glue (pyspark), step functions, pipelines, lambdas, etc.
If you are working in that domain, being able to use the CDK in TypeScript becomes way more important than being able to build a Hadoop cluster from scratch using Scala.
Re: How to Become a Data Engineer in 2021
#5I've been in this space last 6 yrs or so and my scala usuage has gone down to zero. Not worth learning scala.
Re: How to Become a Data Engineer in 2021
#6...and nothing of basic statistics? Data Science people want to know about your data pipeline and have some quantification of the quality of that data. Also, monitoring data pipelines for data integrity often relies upon a statistical test. You don't need to go as far as Bayesian but you do need to understand when a median goes way off or if it bi-modal, etc.
Re: How to Become a Data Engineer in 2021
#7...and nothing of basic statistics? Data Science people want to know about your data pipeline and have some quantification of the quality of that data. Also, monitoring data pipelines for data integrity often relies upon a statistical test. You don't need to go as far as Bayesian but you do need to understand when a median goes way off or if it bi-modal, etc.
a good catch, thanks!
Re: How to Become a Data Engineer in 2021
#8[deleted]
Re: How to Become a Data Engineer in 2021
#9...and nothing of basic statistics? Data Science people want to know about your data pipeline and have some quantification of the quality of that data. Also, monitoring data pipelines for data integrity often relies upon a statistical test. You don't need to go as far as Bayesian but you do need to understand when a median goes way off or if it bi-modal, etc.
That should be assumed in the "engineer" part of the role.
Re: How to Become a Data Engineer in 2021
#10Nice article. From experience I would say the SQL knowledge should be advanced though. Not intermediate.