Live data from Hacker News

How to Become a Data Engineer in 2021

khashtamov.com

141–142 of 142 posts

Re: How to Become a Data Engineer in 2021

#141

Earlier quoted context omitted.

experienced data engineers should graduate to data architects/ML engineers and this way they can get on par with SWE, pls correct me if I am wrong.

You're right, but most companies do not have those positions formalized and therefore you're expected to do those as part of your job, but not gain the financial benefits. Also, there is a big disparity in these titles and what the duties entail, which inherently again feeds into the problem.

> You're right, but most companies do not have those positions formalized and therefore you're expected

To move on to somewhere that does, until the laggards get the message.

Re: How to Become a Data Engineer in 2021

#142
post #131

2021? More like 2010. Hadoop is getting deprecated rapidly and more companies split their write and read workloads. Separated storage and compute is also popular. Scala is not used that much, I think it is not worth the time investment. More and more companies go for Kotlin instead of Java when these want to tap into the Java ecosystem.

What does your typical stack look like?

Depends. Just some random mixture of stacks: PrestoDB, S3, Airflow, Luigi, Dremio, Athena, Hive LLAP, EMC Isilon, Kafka.

My favorite so far is S3 + PrestoDB with either ORC or Parquet files. It is a solid DWH solution for most enterprises on the cloud. (Cloud or not is a different discussion). It works for small scale (50TB) to really high scale (50PB). There are some (very few) gotchas and moving parts as opposed to Hadoop + co. You can combine it with Kafka for streaming data and you got yourself a pretty solid data solution.

Post reply on HN